The one on the left is baked in Marmoset Toolbag 3 and the one on the right is baked in Handplane. The two inner rim lines are actually floaters but every time i try to bake in Marmost toolbag it always comes out horribly like this. It seems to be just floaters that this occurs on, I've tried a range of models and floating…
A lip is generally helpful in these cases. Making the floaters a bit thicker could help as well. One other thing, have you pressed the bake button? What you see in the viewport while editing the cage or other features is a quick preview, so you may not be seeing the final quality rendered bake until your press bake.
Adding a lip to the geometry seems to have mitigated most of the harsh edges but still doesn't look quite as good as the handplane ones. The rest of the bake looks fantastic though and its way faster as its on the GPU. I make use of floaters quite a lot and can foresee where adding lips to everything might become…