# Understanding averaged normals and ray projection/Who put waviness in my normal map?

mod
Offline / Send Message
mod
Of interest to most who will read this thread, I wrote an extensive tutorial that covers this topic and many other common baking issues. It's geared towards baking in Toolbag, but most of the Basics and Best Results sections apply universally. Check it out on the Marmoset site: https://www.marmoset.co/posts/toolbag-baking-tutorial/

---

I keep seeing this question come up time and time again, what causes waviness? How do we fix it? I feel the reason it keeps coming up, is a fundamental lack of understanding.

To get to the root of the problem, we have to look at how normals are averaged, and thus, how the actual rays that are doing the "baking" are cast, and in what direction.

I whipped up a quick test scene here, to show a couple different types of "waviness".

First off, the highpoly, notice that the right two meshes are the same, and the left 6 are also the same, the big difference being a slope instead of a hard 90 degree angle at the ridges.

Now, we'll take a look at the lowpoly meshes. From the averaged vertex normals here we can see the beginnings of our problems, the vertex normals here are far from flat, so any and all differences between the high and the low will become exaggerated. You should be able to learn to spot potential ray trace problems, "waviness" or "skewed details" simply by looking at the lowpoly mesh normals.

And the resulting bake;

Quickly we see that even a gentle slope with our ridged details produces a much different affect, and much more acceptable waviness. A1 and B1 are really worst case examples, and you should design and model to avoid these sort of shapes, unless you have the geometry budget to accurately match the shape of the high.

A3 shows how we can account for "skewing" type details, by adding control loops on either side of the shape, B4 shows what happens when we do not do this.

B1-B4 basically show that we simply do not have enough geometry, and unless this object was going to be seen very small on screen, it simply looks terrible. No amount of tweaks or repainting or cage hacks will fix these meshes, they are hopeless. At least from the side view, from 3/4s view, the waviness actually helps to make certain areas more presentable.

As we break down what the mesh normals are doing, and as a result, the direction the rays are "looking" when we bake the mesh, we see the problem. Along these hard angles the rays are being cast out at a 45 degree angle from the surface.

Now when we look at the difference between the high and the low, the problem becomes very obvious. The greater the difference in curvature, the more problems you're going to have.

Now, the other common problem, skewed detail; is really the exact same problem, just with a different type of geometry.

Here we see that adding these supporting loops flattens out the mesh normals along the cylinder, resulting in a nice, even bake. Without the loops, the normals are averaged along the length of the mesh, and the projection is variable, resulting in "skewed" details along the length.

So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?

Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:

A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.

Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.

## Replies

• Offline / Send Message
Brilliant. Clear and concise explanation. Sticky it, wiki it, put it on the News page. Your next installment needs to be MOAR GEO, then 90% of the newb questions will be answered.

To reiterate what fletch said in another thread, they really need to pay you for this shit.
• Offline / Send Message
mod
Artifice wrote: »
Brilliant. Clear and concise explanation. Sticky it, wiki it, put it on the News page. Your next installment needs to be MOAR GEO, then 90% of the newb questions will be answered.

To reiterate what fletch said in another thread, they really need to pay you for this shit.

Thanks, maybe I'll condense everything I know and write a book someday.
• Offline / Send Message
polycounter lvl 9
3 Point video tutorials! Between yourself and Perna and Vahl you'd make a fortune Plus it would be good information.. there is a lot of mediocre half correct training material floating about these days.
• Offline / Send Message
polycounter lvl 13
r_fletch_r wrote: »
3 Point video tutorials! Between yourself and Perna and Vahl you'd make a fortune Plus it would be good information.. there is a lot of mediocre half correct training material floating about these days.

I agree with this!

Thanks for the brief explanation on those, they're always a bitch.
• Offline / Send Message
polycounter lvl 11
EQ be my valentine

seriously thanks for this man, been going through the wavy cylinder thing recently and that has just helped me so much, I concur that you 3 point folks should get some video tutorials out there.
• Offline / Send Message
polycounter lvl 9
Registered just to say thanks for this. I'm new to 3D modeling, this post condenses info I had sort of figured out on my own but didn't have 100%. Definitely going keep it handy for next time I have bake issues.
• Offline / Send Message
polycounter lvl 11
Good stuff.
• Offline / Send Message
polycounter lvl 11
EarthQuake wrote: »

That's hot. :poly136:

Nice thread, EQ. Thanks for your guidance.
Why not show the flats or at least the UVs, since you mention them?
• Offline / Send Message
mod
Sure, normals:

So, the A method has 5 uv islands as apposed to 4 for B/C. If you had to use hard edges to account for the normals on method A, you'd probably be up to about 10 uv islands, maybe more. If you had to use any hard edges on B/C, you'd simply use them on the natural seams, no big deal.

 Doing the math, if you had to send A into an engine/app that wasn't properly synced as far as tangent basis goes, you'de have to have a uv seam at every 90 degree angle, which would = 13 uv islands. Ralph
• Offline / Send Message
polycounter lvl 12
Thread full of win, once again.

I'm always so tempted to slope cylinders like this on real world copies (on weapons for instance), but usually end up deciding to go for reference accuracy instead, and add moar geo to make up for it... Could probably slope the suckers and no one would tell the difference though.

On a related note, do you ever modify your vertex normals? I do it alot these days and I really like what I can achieve with some simple changes (cleaner bakes, more easily reusable normal maps for modular pieces etc), mostly useful for env stuff though, less so for characters/weapons.
• Offline / Send Message
mod
I dont do anything but basic hard edges, no manual vertex editing, and even then I try to just use scripts that assign hard edges to uv borders, super fast. I switch between modo/max/maya a lot so I just try to keep it basic.
• Offline / Send Message
polycounter lvl 11
One question though, can´t we just paint the wavy lines straight in photoshop?

Not that i´m bitching about waviness, just curious.
• Offline / Send Message
mod
SpeCter wrote: »
One question though, can´t we just paint the wavy lines straight in photoshop?

Not that i´m bitching about waviness, just curious.

This pretty much sums it up:
EarthQuake wrote: »
So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?

Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:

A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.

Certainly you can paint it out in photoshop, or tweak your cage(max) or render multiple bakes and composite them in photoshop, but be aware of the drawbacks. Its better to fully understand the problem, and try to fix it in the mesh first, as painting it out for instance becomes very difficult in more complex situations.

If you've got a simple cylindrical mesh, you know you'll never make any changes to it ever again, you can really do whatever you want. I never like to give advice that isn't sound technically and applicable to a wide variety of situations however.

As shown in a few other threads, painting out waviness can also result in a worse looking asset in some cases as well, depending on the viewing angle. I might do some more examples or just dig up those threads later.

The point of the thread isnt so much to say "ALWAYS DO THIS!" or "NEVER DO THAT!" but to share information and understanding, and challenge the way we work and think, so that you can come to your own conclusion as to when it is or isn't appropriate to do extra tweaks or whatever.
• Offline / Send Message
polycounter lvl 11
Excellent explanation.
• Offline / Send Message
polycounter lvl 11
Argh sry Earthquake , so you meantioned it
I just read through your text while doing something else, so i missed that part, my bad.
I know that it can happen, that it looks worse in some situations if you paint it straight.

But in this example i think it could get rid of waviness without screwing up,
but like you said there is no exact way which can be applied generally and it´s just an example
• Offline / Send Message
polycounter lvl 13
Great write up EQ. Extremely informative and your write ups are always especially great because its a never a you should do this, or you shouldnt do that. Its always just great, solid information for people to learn from and understand why somethings happens or differing ways that an issue or problem can be tackled.

I for one second the book idea ha
• Offline / Send Message
greentooth
• Offline / Send Message
mod
SpeCter wrote: »
Argh sry Earthquake , so you meantioned it
I just read through your text while doing something else, so i missed that part, my bad.
I know that it can happen, that it looks worse in some situations if you paint it straight.

But in this example i think it could get rid of waviness without screwing up,
but like you said there is no exact way which can be applied generally and it´s just an example

Its a question that was bound to be asked, and certainly a good question to ask as well, as i'm sure many will be wondering the same.
• Offline / Send Message
polycounter lvl 11
There was something else what you could do to get rid of the waviness at the end of the barrel(it a cage tweak again though, but can be applied with not much work involved)

If you bevel the egdes at the end to get rid of 90 degree angles you should/could tweak the cage to go straight(let the beveled part have the same diameter as the rest i guess)
And the wobblyness in this part should be nearly gone in many cases.
• Offline / Send Message
polycounter lvl 10
What exactly is a "welded cage"?

Edit - Nevermind, just read the wiki:poly136:
• Offline / Send Message
mod
SpeCter wrote: »
There was something else what you could do to get rid of the waviness at the end of the barrel(it a cage tweak again though, but can be applied with not much work involved)

If you bevel the egdes at the end to get rid of 90 degree angles you should/could tweak the cage to go straight(let the beveled part have the same diameter as the rest i guess)
And the wobblyness in this part should be nearly gone in many cases.

Cage tweaks can be used in some situations, however they do not work the same in all apps, need to be redone when editing the mesh/doing revisions as mentioned about. I'm not terribly interested in discussing band-aids to fix the symptoms of the problem, moreso the actual problem itself.

I think most people understand you can edit your normals in photoshop, or you can tweak your cage, or whatever other hack you want to do to get to the end result. However the mentality that this is the way to fix things, and that these problems are just because of "the way it works", this mentality that people seem to have is a huge part of the problem. And is a big reason why we see over and over, people asking the same question, how do I avoid X in X situation, because someone hasn't told them the exact little hack they should do there....

When we thoroughly understand the situation, critical thinking can be used to find a working method for any situation. When we rely on doing "X" for "Z" situation because we read it on "Y", or because "Steve" told us that is the way to do it, it becomes much more difficult to problem solve.
• Offline / Send Message
polycounter lvl 11
Yeah right, giving beginners a foundation why exactly these problems happen should be the first thing to do

I just mentioned this fix/trick because in most cases it´s just like selecting the
"end"parts of the cage and scale them up which takes very little time to none.
But like you said that only works for programs where the cage also represents the direction where the ray is coming from not just the distance)
• Offline / Send Message
polycounter lvl 10
EarthQuake wrote: »
Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.

I did not know this, but it makes complete sense, and explains the strange offsetting in a lot of my hard edges.
• Offline / Send Message
polycounter lvl 11
:poly124: Ah, thanks for this. Shored up a lot of things for me.
• Offline / Send Message
mod
Enodmi: Looks like you answered your own question before I could.
• Offline / Send Message
mod
I approve of this thread!
• Offline / Send Message
godlike master sticky
The funny thing about it all is that the 'angled vs straight' thing is not only better for baking, it is better for in-game readability in general, since it will catch highlights much better, even at a distance.
• Offline / Send Message
mod
pior wrote: »
The funny thing about it all is that the 'angled vs straight' thing is not only better for baking, it is better for in-game readability in general, since it will catch highlights much better, even at a distance.

Funny, this is what Per said when I showed him this thread. Slopes = more interesting highlights. It makes a lot of sense, because we have more variation in surface direction, instead of just the boring up down, left right, etc. You can see it clearly even with my simple little barrel test mesh, on A the highlight is always 1. on the thinly beveled edges or 2. on the "flat" cylindrical pieces uniformly. On B we've got much more variation in the direction, so much more interesting highlights.
• Offline / Send Message
Excellent.
• Offline / Send Message
polycounter lvl 10
Well I just wanted to say thanks to you EQ for not only posting this but all of your technical advice you give out. I always read your replies in other threads & their very thorough and informative.
• Offline / Send Message
polycounter lvl 10
As posted in jaxellers vehicle thread http://www.polycount.com/forum/showthread.php?t=81116, I just want to drop this in here, as it seems relevant. If you don't need to turbosmooth your cylindrical shapes, then dont't! you're not getting anything out of baking a highres cylinder to a low res cylinder, other than wobbly shapes in your normal maps edges, if all you need is the chamfers, edgeloops and ridges of your cylinder then just add those and don't change the number of sides of the cylinder.

Example:
• Offline / Send Message
polycounter lvl 10
Except for the fact that the above example will look like a smoothed octagon from all angles, where the other will read like a cylinder depending on the angle.
• Offline / Send Message
polycounter lvl 12
Yeah I agree with ENODMI, I'd rather have my cylinder look like a cylinder sometimes than never, I used to do this a while back, but realised it was more trouble for a less pleasing result.
• Offline / Send Message
mod
Right, just like you can paint your lines straight in photoshop, this only actually makes your asset look better from one very specific angle, a straight on side view. If your asset will always be viewed like that, then sure what the hell. However, if you'll view the asset from multple angles, or a different fixed angle, like a 3/4th view, it will virtually always look worse.

 Few threads, this thread is basically an expanded version of the same stuff I keep posting in these threads...

This is an image that I think is very important as well, for those who think beveling edges is the solution to waviness
• Offline / Send Message
Awesome.

Get this on the wiki you old dog!
• Offline / Send Message
mod
NEVER
• Offline / Send Message
polycounter lvl 11
Don´t take away his pleasure answering the same Questions over and over again!:D
• Offline / Send Message
polycounter lvl 12
• Offline / Send Message
polycounter lvl 10
Holy moly. If it was my thread that helped inspire this then I'm glad I asked. Didn't expect to get so much help.
• Offline / Send Message
polycounter lvl 6
*throws his bra on stage*
• Offline / Send Message
polycounter lvl 6
EarthQuake wrote: »
This pretty much sums it up:

Certainly you can paint it out in photoshop, or tweak your cage(max) or render multiple bakes and composite them in photoshop, but be aware of the drawbacks. Its better to fully understand the problem, and try to fix it in the mesh first, as painting it out for instance becomes very difficult in more complex situations.

If you've got a simple cylindrical mesh, you know you'll never make any changes to it ever again, you can really do whatever you want. I never like to give advice that isn't sound technically and applicable to a wide variety of situations however.

As shown in a few other threads, painting out waviness can also result in a worse looking asset in some cases as well, depending on the viewing angle. I might do some more examples or just dig up those threads later.

The point of the thread isnt so much to say "ALWAYS DO THIS!" or "NEVER DO THAT!" but to share information and understanding, and challenge the way we work and think, so that you can come to your own conclusion as to when it is or isn't appropriate to do extra tweaks or whatever.

Yeah. I learned a lot of this stuff from making horrible bakes. And having to paint things out and composite, then realizing my mesh was terrible and having to start over on parts, then re-composite and paint.

I'm pretty done with reworking things. Why not do it right from the start? Reworking your art really, really sucks the fun out of it.
• Offline / Send Message
Great write ups! Lots of help for future work. Thanks.
• Offline / Send Message
mod
Yeah. I learned a lot of this stuff from making horrible bakes. And having to paint things out and composite, then realizing my mesh was terrible and having to start over on parts, then re-composite and paint.

I'm pretty done with reworking things. Why not do it right from the start? Reworking your art really, really sucks the fun out of it.

Exactly, this is pretty much the evolution of my methods as well. When I started doing normal map stuff, oh about 7 years ago, it was an absolute mess. I had no idea what I was doing, no concept of how any of it worked and I would just throw random stuff together and hope it worked, painfully editing the results at the end when it didn't. The more you work the more you learn, and you realize that all of these problems are caused.... By you! Not because the software is bad(with the exception of smoothing errors, this is bad software 99% of the time). So, you can develop a wide array of methods to work, most of them taking the same amount of time, and producing a better result, and often cleaner and more efficient meshes as well.

Rework is my biggest enemy, and if I do need to make changes(as I do fairly often) I want it to be as painless as possible. So I always try to model "bake it and forget" types of meshes, that require little/no cleanup/rework. To me, when I bake the mesh, I'm done! Aside from tweaking AO wierdness or other such things.
• Offline / Send Message
polycounter lvl 7
EarthQuake wrote: »
Right, just like you can paint your lines straight in photoshop, this only actually makes your asset look better from one very specific angle, a straight on side view. If your asset will always be viewed like that, then sure what the hell. However, if you'll view the asset from multple angles, or a different fixed angle, like a 3/4th view, it will virtually always look worse.

 Few threads, this thread is basically an expanded version of the same stuff I keep posting in these threads...

This is an image that I think is very important as well, for those who think beveling edges is the solution to waviness

Ugh, been looking through simlar normal map threads to figure out what the hell is going on whenever I have to bake stuff; this sums it up nicely. I do have a question about this specific example though. Is it safe to say that the waviness is less in the far right cylinder because shape of the cylinder matches closer to the high poly? The top edge also looks nicer than I've ever been able to achieve; is that only from averaging the normals of the cage and having more sides?
• Offline / Send Message
polycounter lvl 10
Full of information I needed to know. Thank you so much EarthQuake!
• Offline / Send Message
polycounter lvl 8
This was some great information. I am still a rookie at modeling, high to low poly modeling in particular and I noticed that you mentioned creating a "model for your low", which I assume is a thread similar to this or perhaps a tutorial. Regardless, I'm having difficulty finding it. Could you post a link please? I would love to read it.
• Offline / Send Message
polycounter
Usually when it's straight, it' just messes up the bi-normal directions, and emulating a hard edge in the low poly.

That can sometimes be a solution, but usually there's aliasing issues... particularly on consoles.

I think in Far Cry 2, the first person guns don't even have Normal Maps. They just used additional geometry.
• Offline / Send Message
polycounter lvl 10
Bal wrote: »
Yeah I agree with ENODMI, I'd rather have my cylinder look like a cylinder sometimes than never, I used to do this a while back, but realised it was more trouble for a less pleasing result.

Personally i dislike the dark edges you get on the rim of the object more where the baked normals thats shouldn't be visible are visible, but thats a matter of taste i guess.

Also getting round _enough_ rims on cars or barrels on guns in the first lod hasn't really been an issue in our engine on this generation.
• Offline / Send Message
polycounter lvl 10
@Killnpc, I wasn't trying to derail EQs awesome thread but I suppose your right about that so I'll relocate.