Issue with baking my normal map

hijal100

Hello. I have an issue with artifacts on my normal map. The low-poly model was created in 3D Max, and I’m baking it in Marmoset Toolbag. The hard edges follow the UV seams. However, in Substance Painter, I’m getting dark spots around the holes in the model. I’ve attached screenshots of the low-poly topology, UV layout, and the issue in Substance Painter. I suspect the problem might be related to the topology. Could you please suggest possible solutions to this issue?

gnoop

Looks like 3d max . Make sure you have different smooth group  inside the hole .   If it's there anyway  find  that handle before you cut  the hole in  and transfer vertex normals  to vertexes where you see shading gradients  from a mesh  without them  ( before the hole)      using  normal theft   script   or just manually by copy/paste .

Or as a second option you can make proper  sub-D   topology, quads only  etc  , edge loops around holes etc .

hijal100

gnoop said:

Or as a second option you can make proper  sub-D   topology, quads only  etc  , edge loops around holes etc .

But then it will create unnecessary triangles and vertices, and can the same approach used for sub-d modeling be applied to a game model?

poopipe

for a game model do the smoothing group thing

if that doesn't completely fix it, it may help to put an extra loop surrounding the hole and use it  to even out the distribution of vertices. This would allow you to make the triangles leading from the hole less long and thin. 
Long thin triangles/quads are more prone to producing shading artefacts on normal shaded meshes in a similar way that they cause issues on subdivided meshes. 


A little extra geometry like that is generally not a problem on high-lod meshes - in a game you'll remove that geometry as you move through the lod chain and the artefacts stop being noticeable

hijal100

I managed to solve the problem. I adjusted the topology and added a Weighted Normals modifier with the Smoothing Groups option. Without it, dark edges around the hole were visible on the normal map. I wanted to ask if using the Weighted Normals modifier during normal map baking is a proper workflow. Also, is it worth triangulating the geometry before exporting for baking using the Turn to Poly modifier?

Noren

"Without it, dark edges around the hole were visible on the normal map."
Just for the record, the normal map should show color changes (if there is smoothing to counteract). Obviously, it has an easier job when the smoothing is more uniform. 

The mesh you use for baking (generally) should be the mesh you export, at least in regards to shading.

On the first screenshots, the dark shading doesn't seem to go around the corner. The new topology is better either way, but my suspicion is that the first unwrap wasn't properly relaxed because some of the polygons in the more dense areas weren't triangulated and some edges ended up undercutting others in the unwrap (hard to see with the resolution of the images, though).
Alternatively (probably more likely come to think of it), the triangulation might have been changed during export/import to Marmoset.

gnoop

hijal100 said:

gnoop said:

Or as a second option you can make proper  sub-D   topology, quads only  etc  , edge loops around holes etc .

But then it will create unnecessary triangles and vertices, and can the same approach used for sub-d modeling be applied to a game model?

If for some reason you still want to get rid of unnecessary loops.  For example in  case of mobile game   or if your game show first lod on replays  and cut scenes only.  You still  can manipulate the normals responsible for shading "corners" directly .  Sometimes it's enough just copy normal from a vertex outside the hole to one sitting on the split/hard edge     using normals modifier in 3d max.       

 Weighted normals is not universal solution but they do help  . they transfer/ re-distribute   shading gradients to smaller  polygons  where they are less visible.  

Triangulating before export is a general rule indeed.

Neox

tbh the issues in first post look like a triangulation mismatch between baker and renderer to me. put that turn to poly limited to 3 on top of your stack and use that to bake and rendering (in engine, or max) and as long as the tangentspace is synced (ie. MikkT) it should not happen like that.

its a bit odd as it looks like you preview in substance painter, so i assume you also baked there, but these issues certainly look like the triangulation isnt static

"I wanted to ask if using the Weighted Normals modifier during normal map baking is a proper workflow."

depends on your pipeline really. many if not most productions these days are maya based, so flushing the mesh through maya is a must for triangulation AND normals. out of the box, Mayas normals are angle and area weighted. In max and blender you use a modifier to achieve this, in maya it is just ON by default and you have to turn it off, if you dont want it. but to be honest unweighted normals are shit compared to weighted ones, so if you can use them in your workflow, it will release shading/stress from your normalmaps, which is better for many things, mipmapping, compression, possible LODs