Confused about tangent spaces Marmo/SP -> Unity
polycounter lvl 7
I'm looking into integrating Marmoset baking into my workflow but I seem to be running into some issues from going to app to app. I have my preferences in Marmoset set to Mikk / xNormal as default.
Just for testing, this mesh is a single smooth group, triangulated. The first example on the top is baked inside of Marmoset (looks great), the bottom baked in Substance Painter and brought into Marmoset (see the shading errors)
So the same mesh / normal maps put into Unity. The top version is using the Marmoset normal map (bad shading), the bottom Substance Painter (looks good).
OR should I be adding doing the hard edges on UV splits (this is what I generally do) which in this image looks good. The Marmoset bake being the green version, the yellow baked in Substance.
In any case this was a test scenario to see if having 1 smoothing group on everything would work consistently between apps. Does this simply mean you should still be adding hard edges on uv splits / 90 degree angles to keep things consistent?
Just for testing, this mesh is a single smooth group, triangulated. The first example on the top is baked inside of Marmoset (looks great), the bottom baked in Substance Painter and brought into Marmoset (see the shading errors)
So the same mesh / normal maps put into Unity. The top version is using the Marmoset normal map (bad shading), the bottom Substance Painter (looks good).
OR should I be adding doing the hard edges on UV splits (this is what I generally do) which in this image looks good. The Marmoset bake being the green version, the yellow baked in Substance.
In any case this was a test scenario to see if having 1 smoothing group on everything would work consistently between apps. Does this simply mean you should still be adding hard edges on uv splits / 90 degree angles to keep things consistent?
Replies
I believe that Unity does not calculate bi-normals per-pixel, while Toolbag (and Unreal) does. Currently we do not have a setting to disable this behavior (it should match with Unreal, but not with Unity).
It looks like there is an option in Painter called "Compute tangent space per fragment" when this is on it should match the Toolbag result. When it is off, it should match what Unity expects. If you try to bake from Painter with that setting on, what happens when you load it in Toolbag and Unity?
Assuming this is the cause, the discrepancy typically should not be noticeable, but if you've got a complex mesh with harsh shading and it's on a single smoothing group, it may become apparent.
In any case it is a good idea to use hard edges along your UV splits. This way your normal map has to do less of the heavy lifting, and will handle texture compression and mip-mapping better. There's no benefit to not hardening edges along UV seams as well, the in-game vertex count will be the same, as any seam splits double the verts along that edge, so there isn't any reason not to do it.
as Victor mentions, Painter bakes look fine in Unity without hard edges.
it also looks fine in our proprietary engine without any hard edges.
CTSF on
I work in a pipeline where we have artists baking in Marmoset, taking those maps into Painter then exporting into Unity. We also have other artists who bake directly in Painter. For those using Marmoset for baking do you have any suggestions / tips to maintaining consistency going from Marmo / Painter / Unity? Or if your mesh is prepped correctly the differences shouldn't be noticeable? I am talking strictly hard surface assets.
For now, hardening edges at UV seams is your best bet.
https://polycount.com/discussion/198435/marmoset-toolbag-3-normal-baking-to-substance-painter-normals-not-displaying-properly
looks like you have to change normal map format in painter to OpenGL.