Hi! You might want to look at the Curvature Map parameters of the baker. Increasing the sampling radius and adjusting the tone mapping might help. Other than that could also check out if AO or thickness maps are any use to build a mask.
This is literally curvature of the surface, so it does show you the polygons. In case of subdiv HP models, I prefer to increase iterations up to 3 to bump up an overall amount of polys. In case of sculpted stuff either play with the sampling radius or decimate less if using decimation master in ZBrush.
Replies
Ideally share the meshes, so people can reproduce the issue.
High and low poly model attached