Wireframe affecting shading in substance painter

TannedBatman

I feel like this is probably a fairly common issue but I cant seem to find anything online so im probably googling poorly. Im sure the answer is probably how the triangulation is done but im not sure how else to do it, its the exact same topology and issue on the other side also. It was baked in marmoset and it looked alright in there although it was untextured. Thanks.

Fabi_G

Hi! Did you triangulate the mesh on export? That would be a way to make sure its shading is consistent between applications.

Also ensure the Normalmap handedness (y/green-channel direction) is consistent between baking and texturing project.

Happy to take a look if you share files (zip + attach).

TannedBatman

Hey thanks, couldnt include the substance project because it was too large, either way I hope I have included everything.   the "_Low" is the exploded and the assembled is made from that one so the topology should be the same.

gnoop

Your problem is on modelling side.  I see same huge shading gradients on your mesh too.   And while normal map can theoretically counter-measure  them  in a perfectly "synced" workflow  you would never get it looking  perfectly right in actual game anyway .   8 bit normal map with compression just doesn't have enough precision  for that.

So you should get rid  of those  shading  artifacts  on your mesh first .    Using hard/split edges  , face weighted normals , transferred/projected   vertex normals   or just hit auto -smooth  button  and then have your UV seams where hard edges are.       UV seam is a split geometry edge for a game engine  anyway .

Your UV is not good either  BTW.  Many uv islands have no  space in between them while other split up to evry single triangle .   

Smooth all / single smooth group   might work for  some objects but for boxy shaped things you really need to use split/hard edges  for good shading .  

Sometimes you may get  those shading gradients/artifacts when you triangulate   your quad only or booleans made   model .   Make sure  your triangulator keeps vertex normals intact .   Blenders  "triangulate" modifier has  "keep normals" checkbox  for example .

Fabi_G

Hi! Checking the lowpoly in Blender with the Normalmap applied, there are similar shading errors as you show. How does the lowpoly + Normalmap look in Toolbag? Did the shading of the lowpoly change by any chance since the bake?

That said, I agree with gnoop on improving the lowpolys' shading. Hard edges at steep angles being the most logical option (considering you split UVs at fitting edges anyways) to remove the shading gradients from the lowpoly and subsequently from the Normalmap. The old Making sense of hard edges, uvs, normal maps and vertex counts thread contains some good information. Edit: Unfortunately images in the thread seem to be broken

Here is a quick test with the back body part, previewed with a smooth, metallic surface:
A - making use of hard edges at steep angles (+ split UVs)
B - smooth shaded

Attached the files used.

TannedBatman

Yeah there are areas which are quite broken and im not sure how that happened tbh, but those areas are small and unnoticeable, unless there are areas that are broken that I dont know about. I just wanted to get it over with because this project was annoying me so I chose to ignore those areas. You mentioned my uv island spacing and that was because I used auto layout I guess. For most of the uvs I believe they are split at hard edges, I also used smooth edges on the whole thing before exporting.

TannedBatman

Unless im confused, wouldnt hardening the edges make it so less information is acquired when baking? I was under the impression you want to smooth the entire mesh so the normals dont miss any areas.

Marmoset bake:

Fabi_G

I suppose you could select edges by UV seams and set them to hard which should remove some mayor shading gradients. Then check if the result is improved when baking anew.

poopipe

using smooth normals works when everything is perfect but at no other time. 

in practice, if the model looks good in an untextured, shaded view  then you're in a good place to start baking. 

it's not a guarantee that hardening all your shell edges will look the best but it's a good starting point

TannedBatman

I just baked a segment with edges given the 'harden edge' option in maya and the normal map came out much better, im slightly confused however but after watching some videos if I understand correctly I should smooth all the edges except the hard ones? I was under the impression you wanted to smooth the entire mesh.

gnoop

Smoothing entire mesh works ok for organic  subjects  or characters.    Anything  boxy shaped or just having close to 90 degree  angle in between polygons   would need  either  split/hardened edge   or a geometry bevel/ chamfer  with vertex normals  slightly rotated  to be perpendicular to adjacent bigger faces . It's called face weighting .    Maya does it by default as far as I read but you still need that bevel in the geometry or shading supporting loops.  

 

 This supporting loops or bevels   might  go away on a next LOD (level of details)  . A simplified  model to switch  at some distance away.  And your normal map wouldn't be  matching  geometry anymore  creating a sudden  shading pop .   So  more common practice is just to keep hardened  edges on boxy things  because  the shading would stay same through all LODs

Your unwrap isn't good because  you have all your edges  having harden attribute or something while  actual vertex normals are all fixed/custom and set to be smooth .    So your auto unwrapper  see this hardened attribute  and split every  cylinder  to each triangle .   While cylinders  should stay smooth  except their   cap/bottom  faces.  

AI is getting a mess and  returns the mess .   

 Proper surface shading  by vertex normals    is a part of modelling.   But every 3d package   have  ways to do it automatically  more or less.    I don't remember already how it works in Maya . in 3d max it's smooth groups   and auto-smooth groups button  .     In Blender it's auto smooth  either but you first need to  get rid of  locked  vertex normals  by "clear custom split normals data" button .    Blender calls  vertex normals  "split normals" .  

My guess  Maya should have something like that too.  So I assume  you need  to clean all custom shading data you accidentally created  from  your geometry first   then do auto smoothing .

TannedBatman

Ive changed the smoothing to be hard edged on steep angles and I think the whole thing looks pretty good now, or atleast alot better than it did before.

with metal texture: