Home Technical Talk

Understanding averaged normals and ray projection/Who put waviness in my normal map?

13456711
Of interest to most who will read this thread, I wrote an extensive tutorial that covers this topic and many other common baking issues. It's geared towards baking in Toolbag, but most of the Basics and Best Results sections apply universally. Check it out on the Marmoset site: https://www.marmoset.co/posts/toolbag-baking-tutorial/

---

I keep seeing this question come up time and time again, what causes waviness? How do we fix it? I feel the reason it keeps coming up, is a fundamental lack of understanding.

To get to the root of the problem, we have to look at how normals are averaged, and thus, how the actual rays that are doing the "baking" are cast, and in what direction.

I whipped up a quick test scene here, to show a couple different types of "waviness".

First off, the highpoly, notice that the right two meshes are the same, and the left 6 are also the same, the big difference being a slope instead of a hard 90 degree angle at the ridges.



Now, we'll take a look at the lowpoly meshes. From the averaged vertex normals here we can see the beginnings of our problems, the vertex normals here are far from flat, so any and all differences between the high and the low will become exaggerated. You should be able to learn to spot potential ray trace problems, "waviness" or "skewed details" simply by looking at the lowpoly mesh normals.



And the resulting bake;



Quickly we see that even a gentle slope with our ridged details produces a much different affect, and much more acceptable waviness. A1 and B1 are really worst case examples, and you should design and model to avoid these sort of shapes, unless you have the geometry budget to accurately match the shape of the high.

A3 shows how we can account for "skewing" type details, by adding control loops on either side of the shape, B4 shows what happens when we do not do this.

B1-B4 basically show that we simply do not have enough geometry, and unless this object was going to be seen very small on screen, it simply looks terrible. No amount of tweaks or repainting or cage hacks will fix these meshes, they are hopeless. At least from the side view, from 3/4s view, the waviness actually helps to make certain areas more presentable.

As we break down what the mesh normals are doing, and as a result, the direction the rays are "looking" when we bake the mesh, we see the problem. Along these hard angles the rays are being cast out at a 45 degree angle from the surface.



Now when we look at the difference between the high and the low, the problem becomes very obvious. The greater the difference in curvature, the more problems you're going to have.



Now, the other common problem, skewed detail; is really the exact same problem, just with a different type of geometry.

Here we see that adding these supporting loops flattens out the mesh normals along the cylinder, resulting in a nice, even bake. Without the loops, the normals are averaged along the length of the mesh, and the projection is variable, resulting in "skewed" details along the length.



So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?

Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:

A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.



Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.









Replies

  • Artifice
    Brilliant. Clear and concise explanation. Sticky it, wiki it, put it on the News page. Your next installment needs to be MOAR GEO, then 90% of the newb questions will be answered.

    To reiterate what fletch said in another thread, they really need to pay you for this shit.
  • EarthQuake
    Artifice wrote: »
    Brilliant. Clear and concise explanation. Sticky it, wiki it, put it on the News page. Your next installment needs to be MOAR GEO, then 90% of the newb questions will be answered.

    To reiterate what fletch said in another thread, they really need to pay you for this shit.

    Thanks, maybe I'll condense everything I know and write a book someday. =)
  • r_fletch_r
    Offline / Send Message
    r_fletch_r polycounter lvl 9
    3 Point video tutorials! Between yourself and Perna and Vahl you'd make a fortune :) Plus it would be good information.. there is a lot of mediocre half correct training material floating about these days.
  • [HP]
    Offline / Send Message
    [HP] polycounter lvl 13
    r_fletch_r wrote: »
    3 Point video tutorials! Between yourself and Perna and Vahl you'd make a fortune :) Plus it would be good information.. there is a lot of mediocre half correct training material floating about these days.

    I agree with this! :)

    Thanks for the brief explanation on those, they're always a bitch.
  • seth.
    Offline / Send Message
    seth. polycounter lvl 14
    EQ be my valentine :)


    seriously thanks for this man, been going through the wavy cylinder thing recently and that has just helped me so much, I concur that you 3 point folks should get some video tutorials out there.
  • Internet Friend
    Offline / Send Message
    Internet Friend polycounter lvl 9
    Registered just to say thanks for this. I'm new to 3D modeling, this post condenses info I had sort of figured out on my own but didn't have 100%. Definitely going keep it handy for next time I have bake issues.
  • PredatorGSR
    Offline / Send Message
    PredatorGSR polycounter lvl 14
  • EarthQuake
    Alright, so I took a little time to do a more practical example. You guys have probably seen me write "model for your low" before, so I'm going to show what I mean by that.

    First the high:
    You'll see A is similar to the worst case example again, but is a very common type of model to see. Lots of straight edges, little regard given to how well this would translate to a low. B and C are the same(to show two different low methods), you can see here lots of low-friendly slopes, on the problem areas like the front of the barrel that we would likely add extra geometry to help keep it from skewing I've modeled a nice fat bevel there, so our extra geo will actually go towards the quality of the model.



    Now, onto the low. A cool thing about the "more friendly" model is that it is actually less geometry, and is more interesting. We can add a little more geo to the larger cylindrical shapes(which again will help waviness) and STILL come out under the tri count of A. For everyone who always thinks you've gotta use 2x more geometry to get good bakes, noooo, you just need to understand the issues and plan accordingly.



    Bakes, besides the obvious stuff, take note of how the high and low "bullet hole" shape is model in the front of the barrel, in the high we have a nice fat bevel in the center, and in the low we've got that nice support bevel as well, this results in a much better bake, in the A example, where the high has a simple indent and no supporting geo in the low, this area barely shows up.

    Also notable, while B/C are clearly better, both of these look fine at most angles aside from a completely straight side view, so, try not to be so anal about minor waviness.




    Ok, so we see that modifying our design enables us to get better bakes with less geometry, but not only in regards to waviness.

    The "friendly" mesh(s) here:
    1. Look better
    2. Have less waviness
    3. Have less skewed details
    4. Use a simpler uv map with less uv island, this easier to work with + better performance
    5. Use less extreme normals, thus resulting in less potential smoothing errors, which again allows us to have a simpler uv map, as we do not need to detach as many uv islands to work well with hard edges - This one is pretty huge.

    So, even if all things were equal, and you could get the same/better results as far as waviness on A as we do on B/C, it simply isn't worth your time making models like this, for the variety of other advantages that proper understanding and planing can bring to your mesh.



  • Vrav
    Offline / Send Message
    Vrav polycounter lvl 11
    EarthQuake wrote: »


    That's hot. :poly136:

    Nice thread, EQ. Thanks for your guidance.
    Why not show the flats or at least the UVs, since you mention them?
  • EarthQuake
    Sure, normals:



    So, the A method has 5 uv islands as apposed to 4 for B/C. If you had to use hard edges to account for the normals on method A, you'd probably be up to about 10 uv islands, maybe more. If you had to use any hard edges on B/C, you'd simply use them on the natural seams, no big deal.

    [Edit] Doing the math, if you had to send A into an engine/app that wasn't properly synced as far as tangent basis goes, you'de have to have a uv seam at every 90 degree angle, which would = 13 uv islands. Ralph
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    Thread full of win, once again.

    I'm always so tempted to slope cylinders like this on real world copies (on weapons for instance), but usually end up deciding to go for reference accuracy instead, and add moar geo to make up for it... Could probably slope the suckers and no one would tell the difference though.

    On a related note, do you ever modify your vertex normals? I do it alot these days and I really like what I can achieve with some simple changes (cleaner bakes, more easily reusable normal maps for modular pieces etc), mostly useful for env stuff though, less so for characters/weapons.
  • EarthQuake
    I dont do anything but basic hard edges, no manual vertex editing, and even then I try to just use scripts that assign hard edges to uv borders, super fast. I switch between modo/max/maya a lot so I just try to keep it basic.
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    One question though, can´t we just paint the wavy lines straight in photoshop?

    Not that i´m bitching about waviness, just curious.
  • EarthQuake
    SpeCter wrote: »
    One question though, can´t we just paint the wavy lines straight in photoshop?

    Not that i´m bitching about waviness, just curious.

    This pretty much sums it up:
    EarthQuake wrote: »
    So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?

    Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:

    A. The work must be redone every time we change the mesh.
    B. The work must be redone every time your boss requests you to change the mesh.
    C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.

    Certainly you can paint it out in photoshop, or tweak your cage(max) or render multiple bakes and composite them in photoshop, but be aware of the drawbacks. Its better to fully understand the problem, and try to fix it in the mesh first, as painting it out for instance becomes very difficult in more complex situations.

    If you've got a simple cylindrical mesh, you know you'll never make any changes to it ever again, you can really do whatever you want. I never like to give advice that isn't sound technically and applicable to a wide variety of situations however.

    As shown in a few other threads, painting out waviness can also result in a worse looking asset in some cases as well, depending on the viewing angle. I might do some more examples or just dig up those threads later.

    The point of the thread isnt so much to say "ALWAYS DO THIS!" or "NEVER DO THAT!" but to share information and understanding, and challenge the way we work and think, so that you can come to your own conclusion as to when it is or isn't appropriate to do extra tweaks or whatever.
  • jocose
    Offline / Send Message
    jocose polycounter lvl 11
    Excellent explanation.
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    Argh sry Earthquake , so you meantioned it :)
    I just read through your text while doing something else, so i missed that part, my bad.
    I know that it can happen, that it looks worse in some situations if you paint it straight.

    But in this example i think it could get rid of waviness without screwing up,
    but like you said there is no exact way which can be applied generally and it´s just an example :)
  • Autocon
    Offline / Send Message
    Autocon polycounter lvl 15
    Great write up EQ. Extremely informative and your write ups are always especially great because its a never a you should do this, or you shouldnt do that. Its always just great, solid information for people to learn from and understand why somethings happens or differing ways that an issue or problem can be tackled.

    I for one second the book idea ha :)
  • metalliandy
    Offline / Send Message
    metalliandy interpolator
    Awesome thread EQ.
    :D
  • EarthQuake
    SpeCter wrote: »
    Argh sry Earthquake , so you meantioned it :)
    I just read through your text while doing something else, so i missed that part, my bad.
    I know that it can happen, that it looks worse in some situations if you paint it straight.

    But in this example i think it could get rid of waviness without screwing up,
    but like you said there is no exact way which can be applied generally and it´s just an example :)

    Its a question that was bound to be asked, and certainly a good question to ask as well, as i'm sure many will be wondering the same.
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    There was something else what you could do to get rid of the waviness at the end of the barrel(it a cage tweak again though, but can be applied with not much work involved)

    If you bevel the egdes at the end to get rid of 90 degree angles you should/could tweak the cage to go straight(let the beveled part have the same diameter as the rest i guess)
    And the wobblyness in this part should be nearly gone in many cases.
  • ENODMI
    Offline / Send Message
    ENODMI polycounter lvl 14
    What exactly is a "welded cage"?

    Edit - Nevermind, just read the wiki:poly136:
  • EarthQuake
    SpeCter wrote: »
    There was something else what you could do to get rid of the waviness at the end of the barrel(it a cage tweak again though, but can be applied with not much work involved)

    If you bevel the egdes at the end to get rid of 90 degree angles you should/could tweak the cage to go straight(let the beveled part have the same diameter as the rest i guess)
    And the wobblyness in this part should be nearly gone in many cases.

    Cage tweaks can be used in some situations, however they do not work the same in all apps, need to be redone when editing the mesh/doing revisions as mentioned about. I'm not terribly interested in discussing band-aids to fix the symptoms of the problem, moreso the actual problem itself. =)

    I think most people understand you can edit your normals in photoshop, or you can tweak your cage, or whatever other hack you want to do to get to the end result. However the mentality that this is the way to fix things, and that these problems are just because of "the way it works", this mentality that people seem to have is a huge part of the problem. And is a big reason why we see over and over, people asking the same question, how do I avoid X in X situation, because someone hasn't told them the exact little hack they should do there....

    When we thoroughly understand the situation, critical thinking can be used to find a working method for any situation. When we rely on doing "X" for "Z" situation because we read it on "Y", or because "Steve" told us that is the way to do it, it becomes much more difficult to problem solve.
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    Yeah right, giving beginners a foundation why exactly these problems happen should be the first thing to do :)

    I just mentioned this fix/trick because in most cases it´s just like selecting the
    "end"parts of the cage and scale them up which takes very little time to none.
    But like you said that only works for programs where the cage also represents the direction where the ray is coming from not just the distance)
  • ENODMI
    Offline / Send Message
    ENODMI polycounter lvl 14
    EarthQuake wrote: »
    Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.


    I did not know this, but it makes complete sense, and explains the strange offsetting in a lot of my hard edges.
  • Jeremy Tabor
    Offline / Send Message
    Jeremy Tabor polycounter lvl 13
    :poly124: Ah, thanks for this. Shored up a lot of things for me.
  • EarthQuake
    Enodmi: Looks like you answered your own question before I could. =)
  • Pedro Amorim
    I approve of this thread!
  • pior
    Online / Send Message
    pior grand marshal polycounter
    The funny thing about it all is that the 'angled vs straight' thing is not only better for baking, it is better for in-game readability in general, since it will catch highlights much better, even at a distance.
  • EarthQuake
    pior wrote: »
    The funny thing about it all is that the 'angled vs straight' thing is not only better for baking, it is better for in-game readability in general, since it will catch highlights much better, even at a distance.

    Funny, this is what Per said when I showed him this thread. Slopes = more interesting highlights. It makes a lot of sense, because we have more variation in surface direction, instead of just the boring up down, left right, etc. You can see it clearly even with my simple little barrel test mesh, on A the highlight is always 1. on the thinly beveled edges or 2. on the "flat" cylindrical pieces uniformly. On B we've got much more variation in the direction, so much more interesting highlights.
  • rasmus
  • ajr2764
    Offline / Send Message
    ajr2764 polycounter lvl 10
    Well I just wanted to say thanks to you EQ for not only posting this but all of your technical advice you give out. I always read your replies in other threads & their very thorough and informative.
  • imperator_dk
    Offline / Send Message
    imperator_dk polycounter lvl 10
    As posted in jaxellers vehicle thread http://www.polycount.com/forum/showthread.php?t=81116, I just want to drop this in here, as it seems relevant. If you don't need to turbosmooth your cylindrical shapes, then dont't! you're not getting anything out of baking a highres cylinder to a low res cylinder, other than wobbly shapes in your normal maps edges, if all you need is the chamfers, edgeloops and ridges of your cylinder then just add those and don't change the number of sides of the cylinder.

    Example:
    Normalmapbakes.jpg
  • ENODMI
    Offline / Send Message
    ENODMI polycounter lvl 14
    Except for the fact that the above example will look like a smoothed octagon from all angles, where the other will read like a cylinder depending on the angle.
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    Yeah I agree with ENODMI, I'd rather have my cylinder look like a cylinder sometimes than never, I used to do this a while back, but realised it was more trouble for a less pleasing result.
  • EarthQuake
    Right, just like you can paint your lines straight in photoshop, this only actually makes your asset look better from one very specific angle, a straight on side view. If your asset will always be viewed like that, then sure what the hell. However, if you'll view the asset from multple angles, or a different fixed angle, like a 3/4th view, it will virtually always look worse.


    [edit] Few threads, this thread is basically an expanded version of the same stuff I keep posting in these threads...

    http://www.polycount.com/forum/showthread.php?t=65790
    http://www.polycount.com/forum/showthread.php?t=67479
    http://www.polycount.com/forum/showthread.php?p=999702

    This is an image that I think is very important as well, for those who think beveling edges is the solution to waviness
    cylinder_sides.jpg
  • Eric Chadwick
    Awesome.

    Get this on the wiki you old dog!
  • EarthQuake
  • SpeCter
    Offline / Send Message
    SpeCter polycounter lvl 14
    Don´t take away his pleasure answering the same Questions over and over again!:D
  • Racer445
  • LRoy
    Offline / Send Message
    LRoy polycounter lvl 10
    Holy moly. If it was my thread that helped inspire this then I'm glad I asked. Didn't expect to get so much help.
  • killnpc
    Offline / Send Message
    killnpc polycounter
    *throws his bra on stage*
  • jimmypopali
    Great write ups! Lots of help for future work. Thanks.
  • EarthQuake
    Yeah. I learned a lot of this stuff from making horrible bakes. And having to paint things out and composite, then realizing my mesh was terrible and having to start over on parts, then re-composite and paint.

    I'm pretty done with reworking things. Why not do it right from the start? Reworking your art really, really sucks the fun out of it.

    Exactly, this is pretty much the evolution of my methods as well. When I started doing normal map stuff, oh about 7 years ago, it was an absolute mess. I had no idea what I was doing, no concept of how any of it worked and I would just throw random stuff together and hope it worked, painfully editing the results at the end when it didn't. The more you work the more you learn, and you realize that all of these problems are caused.... By you! Not because the software is bad(with the exception of smoothing errors, this is bad software 99% of the time). So, you can develop a wide array of methods to work, most of them taking the same amount of time, and producing a better result, and often cleaner and more efficient meshes as well.

    Rework is my biggest enemy, and if I do need to make changes(as I do fairly often) I want it to be as painless as possible. So I always try to model "bake it and forget" types of meshes, that require little/no cleanup/rework. To me, when I bake the mesh, I'm done! Aside from tweaking AO wierdness or other such things.
  • Tickwomp
    Offline / Send Message
    Tickwomp polycounter lvl 7
    EarthQuake wrote: »
    Right, just like you can paint your lines straight in photoshop, this only actually makes your asset look better from one very specific angle, a straight on side view. If your asset will always be viewed like that, then sure what the hell. However, if you'll view the asset from multple angles, or a different fixed angle, like a 3/4th view, it will virtually always look worse.


    [edit] Few threads, this thread is basically an expanded version of the same stuff I keep posting in these threads...

    http://www.polycount.com/forum/showthread.php?t=65790
    http://www.polycount.com/forum/showthread.php?t=67479
    http://www.polycount.com/forum/showthread.php?p=999702

    This is an image that I think is very important as well, for those who think beveling edges is the solution to waviness
    cylinder_sides.jpg

    Ugh, been looking through simlar normal map threads to figure out what the hell is going on whenever I have to bake stuff; this sums it up nicely. I do have a question about this specific example though. Is it safe to say that the waviness is less in the far right cylinder because shape of the cylinder matches closer to the high poly? The top edge also looks nicer than I've ever been able to achieve; is that only from averaging the normals of the cage and having more sides?
  • Rurouni Strife
    Offline / Send Message
    Rurouni Strife polycounter lvl 10
    Full of information I needed to know. Thank you so much EarthQuake!
  • LightningShard5
    Offline / Send Message
    LightningShard5 polycounter lvl 8
    This was some great information. I am still a rookie at modeling, high to low poly modeling in particular and I noticed that you mentioned creating a "model for your low", which I assume is a thread similar to this or perhaps a tutorial. Regardless, I'm having difficulty finding it. Could you post a link please? I would love to read it.
  • JacqueChoi
    Offline / Send Message
    JacqueChoi polycounter
    Usually when it's straight, it' just messes up the bi-normal directions, and emulating a hard edge in the low poly.

    That can sometimes be a solution, but usually there's aliasing issues... particularly on consoles.




    I think in Far Cry 2, the first person guns don't even have Normal Maps. They just used additional geometry.
  • imperator_dk
    Offline / Send Message
    imperator_dk polycounter lvl 10
    Bal wrote: »
    Yeah I agree with ENODMI, I'd rather have my cylinder look like a cylinder sometimes than never, I used to do this a while back, but realised it was more trouble for a less pleasing result.

    Personally i dislike the dark edges you get on the rim of the object more where the baked normals thats shouldn't be visible are visible, but thats a matter of taste i guess.

    Also getting round _enough_ rims on cars or barrels on guns in the first lod hasn't really been an issue in our engine on this generation.
  • ajr2764
    Offline / Send Message
    ajr2764 polycounter lvl 10
    @Killnpc, I wasn't trying to derail EQs awesome thread but I suppose your right about that so I'll relocate.
  • Bal
    Offline / Send Message
    Bal polycounter lvl 17
    Personally i dislike the dark edges you get on the rim of the object more where the baked normals thats shouldn't be visible are visible, but thats a matter of taste i guess.

    You get dark edges in your 3D viewer, but once in game with a proper environement and lighting, it's rarely noticeable in my opinion, but yeah matter of taste. :)
13456711
Sign In or Register to comment.