Weird Triangulation when baking Normal Map (Substance Painter > Marmoset)
Hello, I'm having very odd triangulation issues when exporting normal maps from Substance Painter.
After making my materials in Painter and transferring to Marmoset, I'm seeing some odd normal map triangulation that wasn't present in Painter. The issues also seem to be in the baked Normal Map as well. I've tried importing into Marmoset Mikk space and flipping the Y normal map channel, but the issue persists.
Fwiw I'm following Simon Fuch's Military Radio tutorial extremely closely and he wasn't having these issues.
I'd like to get the model looking in Marmoset like it does in Painter, since it looks correct there. I've spent hours browsing forums trying to debug this with no success, so any help is appreciated!
Replies
what tangentspace is your asset set up inside marmoset toolbag?
that it shoes in the lowpoly is to be expected. i am a bit surprised that your maya topology shot is facetted.
In the marmoset view there appears to be hard edges, you can see it all over the dials and curved surfaces obviously,
I expect that the nomral map will be borked on that model shown in Maya as well. I think the low poly in substance you are using to bake is a smoothed and face weighted normals maya would give you, but you are viewing it on a model with different vert normals, ie hard edges all over it.
I think what is happening is, in terms of vertex normals, that you are not using the normal map on the same model you baked it to. I half expect it to look fine if you match the vert normals and triangulation on the model when baked to when viewed in marmoset.
In Marmoset I had set it to
Default Tangent Space: Mikk
Default Tangent Orientation: Right-Handed
Strangely though, if I toggle these options in Marmoset (in the Preferences and in the mesh) I see no difference whatsoever in my viewport, which is confusing. They seem to have no effect.
it's not Substance issue. It's a matter of surface shading, vertex normals , proper split/hard edges placement . First apply any shiny/ polished material in Maya to check if you see same issues there. There shouldn't be triangle like gradients or reflection distortions . Neither in a first next LOD too at least.
You may just not seeing them with default Lambert material. ( or whatever, I haven't open Maya for decade already) Those cylindrical parts also shouldn't look faceted . In a word vertex normals based shading should be ok BEFORE normal maps .
While a normal map could compensate shading gradients for Substance Painter viewport or Marmoset one if you baked there by doing counter gradients in the normal map I have yet to see a game engine that would do it good enough with typical texture compression, MikkT or not. Better is not having those gradients at all.
If it's totally ok In Maya with reflective material then it's something in fbx export. There might be different options. I usually export with normals checked in and smooth groups unchecked.
Also you could try check in 'compute tangent space per fragment ' in Painter's open dialog and export triangulated mesh together with textures. It sometimes helps with random normals quirks
Thanks for the comments everyone, I think it is the triangulation coming from Maya. I do wonder how mine ended up so different from the tutorial. I have a lot of issues with hard edges as well that Simon was not getting.
The low poly mesh imported to Substance had hard edges on all the UV seams so I wasn't really seeing this triangulation in Maya, even with a shiny blinn material. I triangulated before exporting.
An interesting quirk- if I set Texture Set Settings in Substance to a very low resolution (256) I can see the triangulation, but it goes away at high resolutions and looks great. According to what gnoop said though, this might be a benefit of Substance that you wouldn't necessarily get in a game engine?
I've been manually 'Set Vertex Normal' on some verts in Maya with limited success. Is there a better tool for fixing these things?
You'd get the same in a game engine with low res normal maps that contain heavy gradients like the one shown. The reason why you see triangulation in very low resolution normal map here, is because there aren't enough pixels in those gradients to compensate for the shading on the low poly model. It wouldn't matter where you viewed it if the res was 256, UE, substance, Maya etc, you'd get shadind errors.
But baiscally you should in Maya>
Reset/unlock the normals
Smooth the whole mesh
find a script that selects the UV borders and sets them hard. (probably there would be some manual tweaks to setting some things hard or soft that makes the most sense)
Then export a triangulated copy of the mesh and bake it.
From then on, the normal map and the triangulated mesh work. At any point you go and change the vertex normals/shading of the low poly etc, you will have to rebake.
Your Maya screen clearly shows faceted shading . So something is definitely wrong . Check shading in Maya and make sure you have split/hard edges in those areas in your exported model. Import fbx back in Maya and check .
We also need to see your UV to figure out if you doing it right .
The hard/split edges shouldn't be set after your UV . You should first set hard/split edges and vertex normals to reach gradient-less shading , sometimes just auto shading based n angle threshold and only AFTER that do your UV .