Understanding averaged normals and ray projection/Who put waviness in my normal map?
mod
Of interest to most who will read this thread, I wrote an extensive
tutorial that covers this topic and many other common baking issues.
It's geared towards baking in Toolbag, but most of the Basics and Best
Results sections apply universally. Check it out on the Marmoset site: https://www.marmoset.co/posts/toolbag-baking-tutorial/
---
I keep seeing this question come up time and time again, what causes waviness? How do we fix it? I feel the reason it keeps coming up, is a fundamental lack of understanding.
To get to the root of the problem, we have to look at how normals are averaged, and thus, how the actual rays that are doing the "baking" are cast, and in what direction.
I whipped up a quick test scene here, to show a couple different types of "waviness".
First off, the highpoly, notice that the right two meshes are the same, and the left 6 are also the same, the big difference being a slope instead of a hard 90 degree angle at the ridges.
Now, we'll take a look at the lowpoly meshes. From the averaged vertex normals here we can see the beginnings of our problems, the vertex normals here are far from flat, so any and all differences between the high and the low will become exaggerated. You should be able to learn to spot potential ray trace problems, "waviness" or "skewed details" simply by looking at the lowpoly mesh normals.
And the resulting bake;
Quickly we see that even a gentle slope with our ridged details produces a much different affect, and much more acceptable waviness. A1 and B1 are really worst case examples, and you should design and model to avoid these sort of shapes, unless you have the geometry budget to accurately match the shape of the high.
A3 shows how we can account for "skewing" type details, by adding control loops on either side of the shape, B4 shows what happens when we do not do this.
B1-B4 basically show that we simply do not have enough geometry, and unless this object was going to be seen very small on screen, it simply looks terrible. No amount of tweaks or repainting or cage hacks will fix these meshes, they are hopeless. At least from the side view, from 3/4s view, the waviness actually helps to make certain areas more presentable.
As we break down what the mesh normals are doing, and as a result, the direction the rays are "looking" when we bake the mesh, we see the problem. Along these hard angles the rays are being cast out at a 45 degree angle from the surface.
Now when we look at the difference between the high and the low, the problem becomes very obvious. The greater the difference in curvature, the more problems you're going to have.
Now, the other common problem, skewed detail; is really the exact same problem, just with a different type of geometry.
Here we see that adding these supporting loops flattens out the mesh normals along the cylinder, resulting in a nice, even bake. Without the loops, the normals are averaged along the length of the mesh, and the projection is variable, resulting in "skewed" details along the length.
So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?
Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:
A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.
Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.
---
I keep seeing this question come up time and time again, what causes waviness? How do we fix it? I feel the reason it keeps coming up, is a fundamental lack of understanding.
To get to the root of the problem, we have to look at how normals are averaged, and thus, how the actual rays that are doing the "baking" are cast, and in what direction.
I whipped up a quick test scene here, to show a couple different types of "waviness".
First off, the highpoly, notice that the right two meshes are the same, and the left 6 are also the same, the big difference being a slope instead of a hard 90 degree angle at the ridges.
Now, we'll take a look at the lowpoly meshes. From the averaged vertex normals here we can see the beginnings of our problems, the vertex normals here are far from flat, so any and all differences between the high and the low will become exaggerated. You should be able to learn to spot potential ray trace problems, "waviness" or "skewed details" simply by looking at the lowpoly mesh normals.
And the resulting bake;
Quickly we see that even a gentle slope with our ridged details produces a much different affect, and much more acceptable waviness. A1 and B1 are really worst case examples, and you should design and model to avoid these sort of shapes, unless you have the geometry budget to accurately match the shape of the high.
A3 shows how we can account for "skewing" type details, by adding control loops on either side of the shape, B4 shows what happens when we do not do this.
B1-B4 basically show that we simply do not have enough geometry, and unless this object was going to be seen very small on screen, it simply looks terrible. No amount of tweaks or repainting or cage hacks will fix these meshes, they are hopeless. At least from the side view, from 3/4s view, the waviness actually helps to make certain areas more presentable.
As we break down what the mesh normals are doing, and as a result, the direction the rays are "looking" when we bake the mesh, we see the problem. Along these hard angles the rays are being cast out at a 45 degree angle from the surface.
Now when we look at the difference between the high and the low, the problem becomes very obvious. The greater the difference in curvature, the more problems you're going to have.
Now, the other common problem, skewed detail; is really the exact same problem, just with a different type of geometry.
Here we see that adding these supporting loops flattens out the mesh normals along the cylinder, resulting in a nice, even bake. Without the loops, the normals are averaged along the length of the mesh, and the projection is variable, resulting in "skewed" details along the length.
So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?
Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:
A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.
Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.
Replies
To reiterate what fletch said in another thread, they really need to pay you for this shit.
Thanks, maybe I'll condense everything I know and write a book someday.
I agree with this!
Thanks for the brief explanation on those, they're always a bitch.
seriously thanks for this man, been going through the wavy cylinder thing recently and that has just helped me so much, I concur that you 3 point folks should get some video tutorials out there.
First the high:
You'll see A is similar to the worst case example again, but is a very common type of model to see. Lots of straight edges, little regard given to how well this would translate to a low. B and C are the same(to show two different low methods), you can see here lots of low-friendly slopes, on the problem areas like the front of the barrel that we would likely add extra geometry to help keep it from skewing I've modeled a nice fat bevel there, so our extra geo will actually go towards the quality of the model.
Now, onto the low. A cool thing about the "more friendly" model is that it is actually less geometry, and is more interesting. We can add a little more geo to the larger cylindrical shapes(which again will help waviness) and STILL come out under the tri count of A. For everyone who always thinks you've gotta use 2x more geometry to get good bakes, noooo, you just need to understand the issues and plan accordingly.
Bakes, besides the obvious stuff, take note of how the high and low "bullet hole" shape is model in the front of the barrel, in the high we have a nice fat bevel in the center, and in the low we've got that nice support bevel as well, this results in a much better bake, in the A example, where the high has a simple indent and no supporting geo in the low, this area barely shows up.
Also notable, while B/C are clearly better, both of these look fine at most angles aside from a completely straight side view, so, try not to be so anal about minor waviness.
Ok, so we see that modifying our design enables us to get better bakes with less geometry, but not only in regards to waviness.
The "friendly" mesh(s) here:
1. Look better
2. Have less waviness
3. Have less skewed details
4. Use a simpler uv map with less uv island, this easier to work with + better performance
5. Use less extreme normals, thus resulting in less potential smoothing errors, which again allows us to have a simpler uv map, as we do not need to detach as many uv islands to work well with hard edges - This one is pretty huge.
So, even if all things were equal, and you could get the same/better results as far as waviness on A as we do on B/C, it simply isn't worth your time making models like this, for the variety of other advantages that proper understanding and planing can bring to your mesh.
That's hot. :poly136:
Nice thread, EQ. Thanks for your guidance.
Why not show the flats or at least the UVs, since you mention them?
So, the A method has 5 uv islands as apposed to 4 for B/C. If you had to use hard edges to account for the normals on method A, you'd probably be up to about 10 uv islands, maybe more. If you had to use any hard edges on B/C, you'd simply use them on the natural seams, no big deal.
[Edit] Doing the math, if you had to send A into an engine/app that wasn't properly synced as far as tangent basis goes, you'de have to have a uv seam at every 90 degree angle, which would = 13 uv islands. Ralph
I'm always so tempted to slope cylinders like this on real world copies (on weapons for instance), but usually end up deciding to go for reference accuracy instead, and add moar geo to make up for it... Could probably slope the suckers and no one would tell the difference though.
On a related note, do you ever modify your vertex normals? I do it alot these days and I really like what I can achieve with some simple changes (cleaner bakes, more easily reusable normal maps for modular pieces etc), mostly useful for env stuff though, less so for characters/weapons.
Not that i´m bitching about waviness, just curious.
This pretty much sums it up:
Certainly you can paint it out in photoshop, or tweak your cage(max) or render multiple bakes and composite them in photoshop, but be aware of the drawbacks. Its better to fully understand the problem, and try to fix it in the mesh first, as painting it out for instance becomes very difficult in more complex situations.
If you've got a simple cylindrical mesh, you know you'll never make any changes to it ever again, you can really do whatever you want. I never like to give advice that isn't sound technically and applicable to a wide variety of situations however.
As shown in a few other threads, painting out waviness can also result in a worse looking asset in some cases as well, depending on the viewing angle. I might do some more examples or just dig up those threads later.
The point of the thread isnt so much to say "ALWAYS DO THIS!" or "NEVER DO THAT!" but to share information and understanding, and challenge the way we work and think, so that you can come to your own conclusion as to when it is or isn't appropriate to do extra tweaks or whatever.
I just read through your text while doing something else, so i missed that part, my bad.
I know that it can happen, that it looks worse in some situations if you paint it straight.
But in this example i think it could get rid of waviness without screwing up,
but like you said there is no exact way which can be applied generally and it´s just an example
I for one second the book idea ha
Its a question that was bound to be asked, and certainly a good question to ask as well, as i'm sure many will be wondering the same.
If you bevel the egdes at the end to get rid of 90 degree angles you should/could tweak the cage to go straight(let the beveled part have the same diameter as the rest i guess)
And the wobblyness in this part should be nearly gone in many cases.
Edit - Nevermind, just read the wiki:poly136:
Cage tweaks can be used in some situations, however they do not work the same in all apps, need to be redone when editing the mesh/doing revisions as mentioned about. I'm not terribly interested in discussing band-aids to fix the symptoms of the problem, moreso the actual problem itself.
I think most people understand you can edit your normals in photoshop, or you can tweak your cage, or whatever other hack you want to do to get to the end result. However the mentality that this is the way to fix things, and that these problems are just because of "the way it works", this mentality that people seem to have is a huge part of the problem. And is a big reason why we see over and over, people asking the same question, how do I avoid X in X situation, because someone hasn't told them the exact little hack they should do there....
When we thoroughly understand the situation, critical thinking can be used to find a working method for any situation. When we rely on doing "X" for "Z" situation because we read it on "Y", or because "Steve" told us that is the way to do it, it becomes much more difficult to problem solve.
I just mentioned this fix/trick because in most cases it´s just like selecting the
"end"parts of the cage and scale them up which takes very little time to none.
But like you said that only works for programs where the cage also represents the direction where the ray is coming from not just the distance)
I did not know this, but it makes complete sense, and explains the strange offsetting in a lot of my hard edges.
Funny, this is what Per said when I showed him this thread. Slopes = more interesting highlights. It makes a lot of sense, because we have more variation in surface direction, instead of just the boring up down, left right, etc. You can see it clearly even with my simple little barrel test mesh, on A the highlight is always 1. on the thinly beveled edges or 2. on the "flat" cylindrical pieces uniformly. On B we've got much more variation in the direction, so much more interesting highlights.
Example:
[edit] Few threads, this thread is basically an expanded version of the same stuff I keep posting in these threads...
http://www.polycount.com/forum/showthread.php?t=65790
http://www.polycount.com/forum/showthread.php?t=67479
http://www.polycount.com/forum/showthread.php?p=999702
This is an image that I think is very important as well, for those who think beveling edges is the solution to waviness
Get this on the wiki you old dog!
Exactly, this is pretty much the evolution of my methods as well. When I started doing normal map stuff, oh about 7 years ago, it was an absolute mess. I had no idea what I was doing, no concept of how any of it worked and I would just throw random stuff together and hope it worked, painfully editing the results at the end when it didn't. The more you work the more you learn, and you realize that all of these problems are caused.... By you! Not because the software is bad(with the exception of smoothing errors, this is bad software 99% of the time). So, you can develop a wide array of methods to work, most of them taking the same amount of time, and producing a better result, and often cleaner and more efficient meshes as well.
Rework is my biggest enemy, and if I do need to make changes(as I do fairly often) I want it to be as painless as possible. So I always try to model "bake it and forget" types of meshes, that require little/no cleanup/rework. To me, when I bake the mesh, I'm done! Aside from tweaking AO wierdness or other such things.
Ugh, been looking through simlar normal map threads to figure out what the hell is going on whenever I have to bake stuff; this sums it up nicely. I do have a question about this specific example though. Is it safe to say that the waviness is less in the far right cylinder because shape of the cylinder matches closer to the high poly? The top edge also looks nicer than I've ever been able to achieve; is that only from averaging the normals of the cage and having more sides?
That can sometimes be a solution, but usually there's aliasing issues... particularly on consoles.
I think in Far Cry 2, the first person guns don't even have Normal Maps. They just used additional geometry.
Personally i dislike the dark edges you get on the rim of the object more where the baked normals thats shouldn't be visible are visible, but thats a matter of taste i guess.
Also getting round _enough_ rims on cars or barrels on guns in the first lod hasn't really been an issue in our engine on this generation.
You get dark edges in your 3D viewer, but once in game with a proper environement and lighting, it's rarely noticeable in my opinion, but yeah matter of taste.