Triangles after baking normal map

DC74

Hi everybody! 

I've got the issue with triangles showing on the model after baking high poly to low poly. I try to bake the model in Substance Painter, and in the viewport everything seems to be ok, but in Iray the triangles are visible. 
I found a similar issue in this topic: Triangles showing on bake/normal map and tried to follow the advice. Checked the edges, tried to modify UV shells (I added hard edges, cut more shells, but it did not help). Probably I am missing something important. 

Can you please help me: what shall I check/fix? 

Here is the model in Maya:

Here are the UV shells: 

This is how it looks in Substance Painter's viewport (when the edges are not highlighted with red the model looks ok, without artifacts): 

And this the result in Iray. The triangles' edges are visible. 

You help will be much appreciated! 

Fabi_G

Hi! What does the lowpolys shading look like, without normal map? Attach your meshes if you want them looked at.

DC74

Fabi_G said:

Hi! What does the lowpolys shading look like, without normal map? Attach your meshes if you want them looked at.

You mean shadin in Maya, right? It looks like this. As for me, I do not see problems here. If there are - please, let me know!

P.S. I am experimenting with the edges/shells, so the meshes below are slightly different from the ones I have attached in the first message of the topic. 

DC74

Fabi_G said:

Hi! What does the lowpolys shading look like, without normal map? Attach your meshes if you want them looked at.

If you mean shading of the low poly in Subtance Painter - there are no artifacts. 

Fabi_G

Hm, from what I can tell, lowpolys shading looks fine. Also made a quick test and could re-produce the issue. I wonder then, does the same happen for you when rendering with something else than Iray, like Toolbag, Eevee or Unreal Engine? It rendered fine for me in Toolbag, with and without RT, so maybe it's Painters Iray that has some caveats here. To simplify things, I would primarily check with the renderer the asset is going to be presented with, to make sure it looks correct there.

DC74

Fabi_G said:

Hm, from what I can tell, lowpolys shading looks fine. Also made a quick test and could re-produce the issue. I wonder then, does the same happen for you when rendering with something else than Iray, like Toolbag, Eevee or Unreal Engine? It rendered fine for me in Toolbag, with and without RT, so maybe it's Painters Iray that has some caveats here. To simplify things, I would primarily check with the renderer the asset is going to be presented with, to make sure it looks correct there.

Thanks! Unfortunately, I do not have other renderers. I also added bevels for the low poly mesh and modified the UV shells a little bit more. For some reason now it renders ok (not perfect, but at least there are no triangles). 

poopipe

it is most likely that the bevel fixed it because it insulates the awkward topology on the top surface from the 90 degree corners. 
you could probably achieve the same result by insetting all the top faces. 

DC74

poopipe said:

it is most likely that the bevel fixed it because it insulates the awkward topology on the top surface from the 90 degree corners. 
you could probably achieve the same result by insetting all the top faces. 

Hi! 
What is "insetting"? Do you mean to add additional faces?
Not sure that I understand it correctly, but I selected the top faces and tried to Extrude > Offset them (highlighted with pink in the screenshot below). Before exporting the mesh I applied triangulation as well.  

The result looks ok: 

If that's what you meant: is it a common way to deal with such triangle artifacts? 

gnoop

The shading depends on vertex normals.   Flat thing should have them same directed .   Typically  3d packages   control vertex normal  on their own   but with lots of tiny long triangles  errors usually happen .   insetting  or beveling or face weighting   are just  ways  to help automatic algorithm.   

 You can do same manually too.    Like splitting edges  and  copy vertex normal  on one  vertex  and paste into others or using attribute transfer  to borrow normals directions from an object below.

   

DC74

gnoop said:

The shading depends on vertex normals.   Flat thing should have them same directed .   Typically  3d packages   control vertex normal  on their own   but with lots of tiny long triangles  errors usually happen .   insetting  or beveling or face weighting   are just  ways  to help automatic algorithm.   

 You can do same manually too.    Like splitting edges  and  copy vertex normal  on one  vertex  and paste into others or using attribute transfer  to borrow normals directions from an object below.

   

I checked the normals, and they look like this: 

The normals of the faces on the flat surface looked quite uniformed. 

gnoop

DC74 said:

gnoop said:

The shading depends on vertex normals.   Flat thing should have them same directed .   Typically  3d packages   control vertex normal  on their own   but with lots of tiny long triangles  errors usually happen .   insetting  or beveling or face weighting   are just  ways  to help automatic algorithm.   

 You can do same manually too.    Like splitting edges  and  copy vertex normal  on one  vertex  and paste into others or using attribute transfer  to borrow normals directions from an object below.

   

I checked the normals, and they look like this: 

The normals of the faces on the flat surface looked quite uniformed. 

While the face normals (the one from a center of each triangle ) looks perfectly up     those normal are derivative as a median of vertex normals in triangles corners.   The  bottom left corner  for example , clearly not 90 degree up . it's what  creates shading gradients.  

  Face weighting,  or direct normals editing  could fix it .    As well as selecting  those vertexes and using  attribute transfer to borrow normal direction from the surface beneath  this object.  

 In Maya face weighting   ( when vertex normal gets  perpendicular to bigger neighboring face ) is default option as far as i read .   But I am not Maya user .  Face weighting works better on quads  usually so maybe triangulation and shape of object  messed it .  I am not perfectly sure why it didn't work.

    

 "inset"  fixes it  because it creates another supporting   edge loop where  faces  on both sides are perfectly up  . But for games such  extra loops are a waste of geometry  since it could be fixed  either by normals editing. or when face weighting really works. and at next LOD you would want to  get rid of them anyway.

From your screen it looks like you use hard edge with split normals  AND bevel   with  the hard edge  in the middle . It's a sort of double waste and what makes you those problems too.

DC74

gnoop said:

DC74 said:

gnoop said:

The shading depends on vertex normals.   Flat thing should have them same directed .   Typically  3d packages   control vertex normal  on their own   but with lots of tiny long triangles  errors usually happen .   insetting  or beveling or face weighting   are just  ways  to help automatic algorithm.   

 You can do same manually too.    Like splitting edges  and  copy vertex normal  on one  vertex  and paste into others or using attribute transfer  to borrow normals directions from an object below.

   

I checked the normals, and they look like this: 

The normals of the faces on the flat surface looked quite uniformed. 

While the face normals (the one from a center of each triangle ) looks perfectly up     those normal are derivative as a median of vertex normals in triangles corners.   The  bottom left corner  for example , clearly not 90 degree up . it's what  creates shading gradients.  

  Face weighting,  or direct normals editing  could fix it .    As well as selecting  those vertexes and using  attribute transfer to borrow normal direction from the surface beneath  this object.  

 In Maya face weighting   ( when vertex normal gets  perpendicular to bigger neighboring face ) is default option as far as i read .   But I am not Maya user .  Face weighting works better on quads  usually so maybe triangulation and shape of object  messed it .  I am not perfectly sure why it didn't work.

    

 "inset"  fixes it  because it creates another supporting   edge loop where  faces  on both sides are perfectly up  . But for games such  extra loops are a waste of geometry  since it could be fixed  either by normals editing. or when face weighting really works. and at next LOD you would want to  get rid of them anyway.

From your screen it looks like you use hard edge with split normals  AND bevel   with  the hard edge  in the middle . It's a sort of double waste and what makes you those problems too.

I returned to the mesh without bevels, selected the top faces, and applied Mesh Display > Set to face. Now, all the normals are aligned with the faces:

Thus, I managed to get rid of extra geometry (bevels), and the final result seems to be fine: 

I totally missed the tools for adjusting the normals. Also I tried to "break" the vertices on purpose and "fix" them back with the Set to face tool.
Thank you!  

poopipe

DC74 said:

If that's what you meant: is it a common way to deal with such triangle artifacts? 

yep, that's what I meant.  gnoop explained the why quite well
It is a way to deal with this sort of problem. I do it often when extruding awkward shapes for meshes where I'm not planning to use baked normals, 

DC74

Thanks, @gnoop and @poopipe!