Is there a specific way i should be baking floaters in Marmoset?

Glacierfox

The one on the left is baked in Marmoset Toolbag 3 and the one on the right is baked in Handplane. The two inner rim lines are actually floaters but every time i try to bake in Marmost toolbag it always comes out horribly like this. It seems to be just floaters that this occurs on, I've tried a range of models and floating meshes in exactly the same positions as in Handplane but the Marmoset ones come out looking terrible with jagged edges no matter what resolution the texture is.

Tried all settings under the sun in the baker object in Marmoset too and the samples are as high as it will go (64x64).
The softening option doesn't seem like its the best thing to do either as it applies softening to the whole normal map and destroy it. The cage is fine too and everything seems to check out, even tried importing the same custom cage I used in Handplane.

Any help would be appreciated. Here's some more pictures to show what I mean. They look great in Handplane but I've paid for Marmoset and i want to get it to work for me.



Heres the nice smooth one from Handplane.

EarthQuake

what does the high poly mesh look like in this area?

Glacierfox

Just two high poly band things, tried upping and lowing the poly count, doesnt seem to make a difference. I've tried pulling them out further, pushing them in and even pushing them into the mesh to no avail. Would it make sense to add some sort of 'lip' to either side level with the surface of the low poly tyre so it doesnt have to deal with such a harsh transition?

EarthQuake

A lip is generally helpful in these cases. Making the floaters a bit thicker could help as well.

One other thing, have you pressed the bake button? What you see in the viewport while editing the cage or other features is a quick preview, so you may not be seeing the final quality rendered bake until your press bake.

Glacierfox

EarthQuake said:

A lip is generally helpful in these cases. Making the floaters a bit thicker could help as well.

One other thing, have you pressed the bake button? What you see in the viewport while editing the cage or other features is a quick preview, so you may not be seeing the final quality rendered bake until your press bake.

Yes I'll give it a try with a lip and see if its any better.

And yeah, I have pressed bake. Those initial comparison images are actually with the textures applied inside Maya. Thanks for the tip though.

Glacierfox

Adding a lip to the geometry seems to have mitigated most of the harsh edges but still doesn't look quite as good as the handplane ones. The rest of the bake looks fantastic though and its way faster as its on the GPU. 
I make use of floaters quite a lot and can foresee where adding lips to everything might become problematic. Whats the reason for the differences between the baking methods here? I don't seem to get these problems in XNormal either even without any filtering applied.

EarthQuake

Its not clear what the reason for the float jaggedness is. If you can send a copy of your bake scene to support@marmoset.co with a brief description of the problem and a link to this thread, we'll look into it further. Thanks!

Glacierfox

Will do, thanks for your help!