Manipulating normals manually to get better bakes

mogorodnik · Sep 2021

hey guys,
recently, i've encountered this little video, which made me realize that i'm missing quite big chunks of understanding about normals and baking.
it's about controlling shading and getting better normal bakes
it was accompanied by an article, but they didn't get into it all

can anybody explain to me what is actually going on in that video or link any other videos/articles/threads that go in-depth about manipulating normals manually?

thanks!

gnoop · Sep 2021

It's very right approach imo , same what I usually do. Consider it as baking your normal map for lod 02 where you couldn't have all those extra geo to weed out triangle shading artifacts on something cylindrical+ a hole. And rather use vertex normals projected from something perfectly shaded or just received from moi3d . So no smooth groups at all . All the shading is from custom vertex normals . They are not manually edited actually but rather projected .

Sometimes you have to make it perfect for lod02 because it what you see most of a time and get lod01 seen only in replays . Then your lod01 would have all vertex normals projected from lod 02 like on that flat plane detail in the beginning of the video, It actually explains it all.

This approach has drawbacks although. You have to make your lods manually most of a time. i'e project vertex normals from one lod to another, keep them consistent through lods . Think and care about it . Tell people they wouldn't break them accidentally in 3d max with minor editing something etc. But as a plus you usually have less vertexes with custom normals and less shading glitches on switching lods. So it's time vs quality issue.

And 3d max vs Blender /MAya too maybe. Since MAx still doesn't have a way to project vertex normals non-destructively.

poopipe · Sep 2021

I hardly ever agree 100% with gnoop but this is a time where I do agree with gnoop 😁

It's labour intensive compared to Palm smashing autounwrap, running a random fwn script and running the result through an autolodder but you get out what you put in and this will give you the best quality and most efficient result at all Lod levels

gnoop · Sep 2021

My bet once 3dMAx copy normals projecting /data transfer modifier from Blender it may became a mainstream approach and everybody forget those smooth groups riddles you have to deal with .

mogorodnik · Sep 2021

thanks for the replies guys, but i'm not sure i know what you mean. I'm mainly interested in the pros/cons of manually tweaking normals pre-bake. Like, is it mostly for specific use cases or can it be used generally, for most assets? If i understood you right, one of the cons is that you have to make lods manually, right?

poopipe · Sep 2021

It can be used generally

Pros
Your bakes look better

Cons.
It is fragile - edits to your mesh can break your normals which means you end up doing them again.
It's not always obvious to another artist that normals have been tweaked and therefore it is more likely that things will get broken when other team mbers work on your meshes.

The process gnoop describes is baking to the Lod you see most often - not to the top Lod

For this to work you have to project the lower lod normals up to higher lods so that your baked normal map reacts correctly.
The necessitates manual creation - or at least edits - to your lods

It's not a con to my mind - you get better looking, more efficient results if you build your assets properly

gnoop · Sep 2021

mogorodnik said:

one of the cons is that you have to make lods manually, right?

if you use U4 which makes lods automatically, yes. Last time I tried it can't cope with custom normals. but it was a while ago . Have no idea if it "understands" them now .

if you have to do lods manually in your 3d package it could be also semi automatic but not in 3d MAx probably( I actually have not tried it there for a while too). In Blender it's not that much of a pain with shrinkwrap modifier that can project vertexes on surface of previous lod and data transfer modifier that projects vertex normal direction from some hidden perfect thing, hi res model for example from Z brush. And triangulation modifier keeps custom normals too.

Here is a picture of some lod02 model . Notice absence of shading artifacts in triangles around the hole . In 3d max you could never do so with smooth groups only and you would probably be suggested to make more geo and support loops to workaround such artifacts .

in Blender you have to apply split modifier on hard edges although and merge back in the end. I do it for unwrap anyway . So it's not perfectly non-destructive too. Still in a next lod you just remove a few edges so not that much of a pain.

So in my experience the main issue with custom normals is persuading MAx users to be careful and not break them accidentally while do some model revisions. Some people are not even aware of what they are.

here is how lods switch works with consisted normals projected to both lods :

Image: https://us.v-cdn.net/5021068/uploads/editor/gd/dku6pksj7bky.gif

No shading glitches. And that would be impossible with smooth groups.

mogorodnik · Sep 2021

@gnoop @poopipe thanks a lot for clearing it up guys 👍

Kanni3d · Sep 2021

I understand how/why this is important, but I'd love to have some light shed on a couple things. Firstly, you'd do your LODs first before you bake, and you mention to bake for LOD2 (to ensure that one looks the best, since obviously LOD1 and LOD0 will only look the same bake wise since no geo/vertices will be removed).

Do you set up custom normals for LOD2, and then have lod 1 and 2 copy it? 3 and 4 and so on will likely have issues since there's far less vertices/tangents to match the same shading, right?

gnoop · Sep 2021

Think about it in terms of dx11 tessellation shader . You have your not so hi poly model and the shader tessellates it first by creating new vertexes , than extrudes those new vertexes in direction extrapolated from vertex normals and using height map as a distance to move a vertex alone normal.

That way the shader do a hi res model with more vertexes but it's still perfectly in match with a normal map you baked for initial low res model. Because vertex normals stay looking in same direction .

You can recreate it in 3d package . just try Data transfer modifier in Blender . or Attribute transfer in MAya or Normal theft script in Max

You can copy/paste normals from one mesh to another but data transfer modifier is more convenient . It can do it on the fly automatically

As of lod03, lod04 etc you can't obviously project normals backward to no vertex at all. So shading switches are obviously unavoidable but those are too far away usually so not so critical.

Same way as normals you can project UV from one object to another or edit a copy of object while data transfer copes with fixing UV ( not reliably across uv seams although)

poopipe · Sep 2021

Kanni3d said:

I understand how/why this is important, but I'd love to have some light shed on a couple things. Firstly, you'd do your LODs first before you bake, and you mention to bake for LOD2 (to ensure that one looks the best, since obviously LOD1 and LOD0 will only look the same bake wise since no geo/vertices will be removed).

Do you set up custom normals for LOD2, and then have lod 1 and 2 copy it? 3 and 4 and so on will likely have issues since there's far less vertices/tangents to match the same shading, right?

you are correct about lods lower than the one you bake to

For this approach you want to bake to the lowest lod where it won't knacker the appearance of the higher ones. if you've got something fairly simple like a doorway or a cliff face then it'll probably be the lowest but for something more ornate there tends to be stuff missing on the low lods which makes baking a bit tricky.
I've been known to bake to a hybrid of lod meshes that doesn't go anywhere near the game to support this sort of thing.

Manipulating normals manually to get better bakes

Replies