The one on the left is baked in Marmoset Toolbag 3 and the one on the right is baked in Handplane. The two inner rim lines are actually floaters but every time i try to bake in Marmost toolbag it always comes out horribly like this. It seems to be just floaters that this occurs on, I've tried a range of models and floating meshes in exactly the same positions as in Handplane but the Marmoset ones come out looking terrible with jagged edges no matter what resolution the texture is.
Tried all settings under the sun in the baker object in Marmoset too and the samples are as high as it will go (64x64).
The softening option doesn't seem like its the best thing to do either as it applies softening to the whole normal map and destroy it. The cage is fine too and everything seems to check out, even tried importing the same custom cage I used in Handplane.
Any help would be appreciated. Here's some more pictures to show what I mean. They look great in Handplane but I've paid for Marmoset and i want to get it to work for me.
Heres the nice smooth one from Handplane.
One other thing, have you pressed the bake button? What you see in the viewport while editing the cage or other features is a quick preview, so you may not be seeing the final quality rendered bake until your press bake.
And yeah, I have pressed bake. Those initial comparison images are actually with the textures applied inside Maya. Thanks for the tip though.
I make use of floaters quite a lot and can foresee where adding lips to everything might become problematic. Whats the reason for the differences between the baking methods here? I don't seem to get these problems in XNormal either even without any filtering applied.