Manipulating normals manually to get better bakes

mogorodnik

hey guys,
recently, i've encountered this little video, which made me realize that i'm missing quite big chunks of understanding about normals and baking.
it's about controlling shading and getting better normal bakes
it was accompanied by an article, but they didn't get into it all

can anybody explain to me what is actually going on in that video or link any other videos/articles/threads that go in-depth about manipulating normals manually?

thanks!

gnoop

It's very right approach imo ,  same what I usually do.    Consider it as baking your normal map for lod 02  where you  couldn't  have all those  extra geo  to weed out triangle shading artifacts on something cylindrical+ a hole.        And rather use  vertex normals projected from something perfectly  shaded or just received  from moi3d .   So no smooth groups at all .     All the shading is from custom vertex normals .   They are not manually edited  actually but rather projected .

Sometimes you have to make it perfect for lod02  because it what you see most of a time  and get lod01  seen only  in replays .    Then  your lod01 would have  all vertex normals projected from lod 02  like on that flat plane detail  in the beginning of the video, It actually explains it all.

This approach has  drawbacks  although.     You have to make  your lods manually most of a time.  i'e   project vertex normals from one lod to another,  keep them consistent through lods . Think and care about it . Tell people  they wouldn't break them  accidentally in 3d max  with minor editing something  etc.       But as a plus  you usually have less vertexes with custom normals and less shading glitches on switching lods.    So it's  time vs quality issue. 

And 3d max vs Blender /MAya  too maybe.   Since  MAx still doesn't  have a way to  project vertex normals  non-destructively.   

poopipe

I hardly ever agree 100% with gnoop but this is a time where I do agree with gnoop 😁  

It's labour intensive compared to Palm smashing autounwrap, running a random fwn script and running the result through an autolodder but you get out what you put in and this will give you the best quality and most efficient result at all Lod levels

gnoop

My bet once  3dMAx copy  normals projecting /data transfer  modifier from  Blender   it may became a mainstream approach  and everybody forget those  smooth groups riddles  you have to deal with .       

mogorodnik

thanks for the replies guys, but i'm not sure i know what you mean. I'm mainly interested in the pros/cons of manually tweaking normals pre-bake. Like, is it mostly for specific use cases or can it be used generally, for most assets? If i understood you right, one of the cons is that you have to make lods manually, right? 

poopipe

It can be used generally

Pros 
Your bakes look better

Cons. 
It is fragile - edits to your mesh can break your normals which means you end up doing them again. 
It's not always obvious to another artist that normals have been tweaked and therefore it is more likely that things will get broken when other team mbers work on your meshes.



The process  gnoop describes is baking to the Lod you see most often - not to the top Lod

For this to work you have to project the lower lod normals up to higher lods so that your baked normal map reacts correctly. 
The necessitates manual creation - or at least edits - to your lods

It's not a con to my mind - you get better looking, more efficient results if you build your assets properly

gnoop

mogorodnik said:

one of the cons is that you have to make lods manually, right? 

if you use U4 which makes lods automatically,   yes.     Last time I tried it can't cope with custom  normals. but it was a while ago .  Have no idea if it "understands"  them now .

if you have to do lods manually  in  your 3d package  it could be also semi automatic  but not in 3d MAx probably( I actually have not tried it there for a while too).   In Blender it's not that much of a pain  with shrinkwrap modifier that can project  vertexes on surface of previous lod  and  data transfer modifier that projects vertex normal direction from some hidden perfect  thing, hi res model  for example from Z brush.   And triangulation modifier keeps custom normals too.

Here is a picture of some lod02 model .  Notice absence  of shading artifacts in triangles around the hole .   In 3d max you could never do so with smooth groups only  and you would  probably be suggested  to  make more geo and support loops to workaround such artifacts .   

in Blender you have to apply split modifier  on hard edges although  and  merge back  in the end.    I do it for unwrap anyway .   So it's  not perfectly non-destructive too.     Still in a next lod you just  remove a few edges  so not that much of a pain.

So in my experience the main issue with custom normals  is persuading  MAx users to be careful and not break them  accidentally while  do some  model  revisions.    Some people are not even aware  of what they are.

here is how lods switch works  with consisted normals projected to both  lods :

No shading glitches.   And that would be impossible with smooth groups.

mogorodnik

@gnoop @poopipe thanks a lot for clearing it up guys 👍

Kanni3d

I understand how/why this is important, but I'd love to have some light shed on a couple things. Firstly, you'd do your LODs first before you bake, and you mention to bake for LOD2 (to ensure that one looks the best, since obviously LOD1 and LOD0 will only look the same bake wise since no geo/vertices will be removed).

Do you set up custom normals for LOD2, and then have lod 1 and 2 copy it? 3 and 4 and so on will likely have issues since there's far less vertices/tangents to match the same shading, right?

gnoop

Think about it  in terms  of dx11 tessellation shader   .   You have  your not so hi poly model and the shader tessellates  it first by creating  new vertexes  , than  extrudes  those new vertexes in direction extrapolated from vertex normals and  using height map as  a distance to move a vertex alone normal.

  That way the shader do  a hi res model with more vertexes   but it's still perfectly  in match with a normal map  you baked for initial low res model.  Because  vertex normals stay looking in same direction  . 

 

You can  recreate it  in 3d package  .     just try Data transfer  modifier in Blender . or Attribute transfer in MAya or Normal theft script in Max   

       You  can copy/paste normals  from one mesh to another  but  data transfer modifier is  more convenient .  It can do it on the fly  automatically

As of lod03, lod04  etc  you can't obviously project normals  backward  to no vertex at all.   So  shading switches are obviously unavoidable  but those are too far away usually  so not so critical.

Same way as normals  you can project UV  from one object to another   or edit  a copy of object while data transfer copes with fixing UV   ( not reliably  across uv seams  although)

poopipe

Kanni3d said:

I understand how/why this is important, but I'd love to have some light shed on a couple things. Firstly, you'd do your LODs first before you bake, and you mention to bake for LOD2 (to ensure that one looks the best, since obviously LOD1 and LOD0 will only look the same bake wise since no geo/vertices will be removed).

Do you set up custom normals for LOD2, and then have lod 1 and 2 copy it? 3 and 4 and so on will likely have issues since there's far less vertices/tangents to match the same shading, right?

you are correct about lods lower than the one you bake to

For this approach you want to bake to the lowest lod where it won't knacker the appearance of the higher ones.  if you've got something fairly simple like a doorway or a cliff face then it'll probably be the lowest but for something more ornate there tends to be stuff missing on the low lods which makes baking a bit tricky.
I've been known to bake to a hybrid of lod meshes that doesn't go anywhere near the game to support this sort of thing.