A Practical Guide On Normal Mapping For Games

1
polycounter lvl 7
Offline / Send Message
SuperFranky polycounter lvl 7
As we all know, surely, that Normal Mapping isn't easy. There are a lot of problems that can emerge when you try to bake normal maps. I decided to start this guide after being enormously frustrated with the lack of information and the prominence of downright wrong facts on normal mapping. I, as most people, was always confused about this process and had to spend countless hours on searching forums for answers.

With this guide, I tried to encompass everything there is to know about today's process of creating normal maps for games and make it as simple to understand as possible.

The guide was inspired by Joe “EarthQuake” Wilson's work here in the Technical Talk. W

Right now it's Work In Progress, so there's bound to be a lot of mistakes on my part. If there's anything you want to see fixed or added, please inform me and I'll try to do what I can to fix it as soon as possible.


A Practical Guide On Normal Mapping For Games

https://drive.google.com/file/d/0B02lElvs8BcvYllmQWpXUGxod3M/view?usp=sharing

It's a Microsoft Word document, if you download it and open in Word you will get access to Table of Contents, it's very useful.

Hope you like it :)


Changelog:



Jan.13 2015:
Added 3rd method of dealing with slanted details
Added a simple Troubleshooting section

Jan.11 2015:
Expanded a bit on Baking
Spelling, formatting

Jan.10 2015:
Added explanation of Tangent Space
Expanded a bit on floaters
Expanded on baking
Added UV overlapping section to Building Low Poly
Expanded on Slanted details
Added Table of Contents
Formatting, spelling

Replies

  • beefaroni
    Offline / Send Message
    beefaroni polycounter
    Deyum this is long. I'm going to have to read through on my next break.

    My first quick crit is that the design could be improved, but I won't go in too muhc depth until it's all finished up!
  • AdvisableRobin
    Offline / Send Message
    AdvisableRobin polycounter lvl 5
    SUUUUUUUUUPER.

    Didn't get a chance to really read over all of it, but I think you could reword the section on '3.Intersecting geometry for combining in low poly' since I didn't totally understand what you were saying until my co-worker explain it to me.

    Also for the skewing section I would add in a link to PeterK's skewmesh trick. Here

    Looking good otherwise!
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    beefaroni, well I'm not a guy capable of designing a proper document, so you'll have to forgive me on it looking pretty bad :)

    AdvisableRobin, will try to add everything in the next revision
  • pior
    Offline / Send Message
    pior insane polycounter
    Lots of great tips in there, but I would be careful about the whole "fear of gradients" thing. Making people think that gradients in a normalmap are bad just leads to misunderstandings and sometimes bad practices down the road.

    Now don't get me wrong, I can tell by reading the document that you know what you are talking about, and you are just basically taking a bit of a shortcut for the sake of things being easier to follow. But still, I feel like there should be at least a note or a special in-depth section explaining further why and how these "gradients" are totally fine when the pipeline is properly setup and synchronized.

    The reason why I am pointing this out is simply because I personally had to deal with artists that needed to "unlearn" their fear of normalmap gradients, and that's just a bit of a waste of time really ... especially when the pipeline *is* properly synchronized to begin with :/
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    pior wrote: »
    Lots of great tips in there, but I would be careful about the whole "fear of gradients" thing. Making people think that gradients in a normalmap are bad just leads to misunderstandings and sometimes bad practices down the road.

    Now don't get me wrong, I can tell by reading the document that you know what you are talking about, and you are just basically taking a bit of a shortcut for the sake of things being easier to follow. But still, I feel like there should be at least a note or a special in-depth section explaining further why and how these "gradients" are totally fine when the pipeline is properly setup and synchronized.

    The reason why I am pointing this out is simply because I personally had to deal with artists that needed to "unlearn" their fear of normalmap gradients, and that's just a bit of a waste of time really ... especially when the pipeline *is* properly synchronized to begin with :/

    That's what I tried to tell. I said many times in the guide that gradients aren't a bad thing at all and you can get away with most of it if you use a synchronised workflow.

    Maybe I should rephrase it a bit, because I never wanted to tell that gradients are bad...
  • AdvisableRobin
    Offline / Send Message
    AdvisableRobin polycounter lvl 5
    I thought gradients also caused problems in the compression and mipping department? or is that only a problem when you have serious gradients?
  • pior
    Offline / Send Message
    pior insane polycounter
    SuperFranky : totally man, I get what you are getting at. I guess my point is that I think that it would be more educational to show a perfectly synced workflow first (where even the funkiest gradients are fine, because things are just working as they should without the need of extra user input), and *then* move on to the hacks ; as opposed to starting from the hacks and not showing the best case scenario at all.

    Robin : it depends ! Some engines are pretty liberal with that sort of stuff and go all hardcore on the compression, which impacts surface quality. Some have specific compression flags for normalmaps. And then there's also everything else : like, using high color depth source normalmaps saved to EXR to get the most accurate data as possible before a well optimized compression routine.

    On a side note : Frank, thank you for mentioning the triangulation issue. I really, really wish more people knew/understood this :)
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    pior wrote: »
    SuperFranky : totally man, I get what you are getting at. I guess my point is that I think that it would be more educational to show a perfectly synced workflow first

    I always assumed that Xnormal is synced with Marmoset 2. I guess it isn't?
  • pior
    Offline / Send Message
    pior insane polycounter
    No idea ! You'd have to ask both the Marmoset and Xnormal guys. I don't remember hearing definitive statements from either of them on that subject though.

    As far as I am aware, Source has a perfectly synced solution through Handplane, and IDTech is perfectly synced with itself as it has its own, internal baking tool (not sure if this is still the case tho, but it certainly was back in the IDtech4 days). As far as other engines are concerned, it's a case by case basis really. Both Cryengine and Unreal used to have strong problems in that department for years, but these might have been addressed by now ... maybe ? One thing I can say for sure is that working with a perfectly synced engine really is a huge deal. It simplifies the processes alot, thus saves a lot of time, and it looks good !

    Some interesting details being discussed here :

    [ame]


    Good luck man - this document of yours is already a goldmine of information for anyone starting out. It's very cool of you to put all this together, as it will without a doubt benefit a lot of people !
  • EarthQuake
    TB2 supports the following tangent spaces:
    Max
    Maya
    Xnormal/Mikktspace
    Marmoset (our own standard)
    and in 2.07 Unity as well
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    EarthQuake wrote: »
    TB2 supports the following tangent spaces:
    Max
    Maya
    Xnormal/Mikktspace
    Marmoset (our own standard)
    and in 2.07 Unity as well

    So you're saying that Xnormal is perfectly synced with Marmoset 2? Just to clarify.

    What would be a good example of a non-synced workflow, for demonstration purposes?

    Also, does anyone know all possible pairs of Engine-Baker there is that provides synced workflow? I guess that's something that needs to be in the guide...
  • kary
    Offline / Send Message
    kary polycounter lvl 13
    So you're saying that Xnormal is perfectly synced with Marmoset 2? Just to clarify.

    Yes, if you choose to have TB2 displaying Xnormal/Mikktspace with the preference setting or on a per object. It's not set that way by default.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    kary wrote: »
    Yes, if you choose to have TB2 displaying Xnormal/Mikktspace with the preference setting or on a per object. It's not set that way by default.

    Great, that's what I was working with when I wrote this guide. So I guess it's safe to say that all the practices written there apply to the synced workflow.
  • Quack!
    Offline / Send Message
    Quack! polycounter lvl 13
    It's also worth noting, that modern engines will inherit the tangents and binormals from the file imported. That is to say, most modern engines (Substance, Toolbag2, Unreal 4, etc) will be 'synched' to whichever tangent basis the model was created in, without having to be specifically synched to a basis. This works best because there is no golden standard as to a tangent basis, although Mikktspace is the most widely adopted.

    A non-synched workflow would be baking in 3dsmax and displaying in max viewport. This was busted last I checked, but could have been fixed in latest iterations.
  • EarthQuake
    Oh yeah, worth noting as well that if you import and FBX with tangents saved in the file, TB2 will use those automatically. OBJ doesn't support tangents though.
  • ScottMichaelH
    Offline / Send Message
    ScottMichaelH polycounter lvl 9
    I just want to say thanks for the effort. I myself have struggled with normals more than anything else, in part because of the lack of organization to all the information.
  • kodde
    Offline / Send Message
    kodde polycounter lvl 14
    Nice work so far.

    I think it could be an idea to include more in-depth technical info in your "Tech Babble" part in the beginning. Explaining what a vertex normal is, how it affects the resulting lighting, what the normal map actually does in the lighting calculation, etc.
  • LensFlare
    Offline / Send Message
    LensFlare polycounter lvl 7
    This was great, thank you! Definitely going to read and re-read for a while. My only suggestions might be to re organize to present the information in the order that you do things in, if that makes any sense, making it a sort of step by step cheat sheet for the whole process. Again, thank you!
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 5
    Unreal Engine 4 has a perfect sync with Mikk since 4.7 for static meshes. Skeletal meshes are close enough for government work. To bake for Unreal Engine 4 since 4.7 here's what you need to do:
    • go to the MikkTSpace plugin setting in Xnormal by clicking the tacky plug icon in the bottom left
    • make sure "Compute Binormal in Pixel Shader" is unchecked
    • export all .fbx files from your DCC WITHOUT tangents and binormals, otherwise Xnormal will use that tangent basis instead
    • when you import to Unreal, make sure it's set to "Import Normals" which will calculate the tangents

    The big advantage of Mikk over Max/Maya tangent spaces is that if you reuse UV space in any way, whether it's mirroring a normal map or reusing an element, the shading will always be correct, which isn't the case with Max/Maya tangent spaces. Mikk works for this because its tangent space calculation is order-independent; the tangents depend only on the UV layout and the vertex normals. If you've ever wondered why you get a seam in your shading when you mirror a human face (e.g.) in Max, it's because the tangent basis is order-dependent. Use Mikk instead if you have the option.

    Cryengine is still not particularly synced to anything last time I checked, so if you're working with Cryengine it's best to bake in Max and make sure that your vertex normals correctly capture the low frequency details in your high-poly mesh. In fact, it's probably a good idea to do this for Unreal as well, because you'll get less artifacts from texture compression and lower mips.
  • FourtyNights
    Offline / Send Message
    FourtyNights greentooth
    For some reason I've never quite understood what "tangents" .fbx saves when exporting... Nowdays, even Blender has the ability to tick "Tangent Space" when exporting .fbx files out of it. But I don't know what it actually saves inside the file.

    Is it something related to smoothing groups/hard edges/custom normals? Because when I bake normals, I'll use xNormal with triangulated .obj files including exported normals with hard edges (if needed) with UV splits on those hard edges (custom normals is a work in progress in Blender development, so hard edges/"smoothing groups" are supported only).

    EDIT: Sorry for noobish question.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Thanks for all the help and suggestions, guys. I already have a huge list of things to fix and add, so expect more good info soon.

    kodde, the primary goal of this guide is to be as easy to understand as possible, I'm trying to avoid unnecessary technical information... That's why it's a Practical guide. I will probably put some links in there for people who want to know more.
  • dzibarik
    Offline / Send Message
    dzibarik polycounter lvl 6
    This guide is super useful but yes, examples of perfectly sinced normal maps with wavy gradients which also look cool would be very helpful. So far I was able to avoid it using UV splits and hard edges only.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    dzibarik wrote: »
    This guide is super useful but yes, examples of perfectly sinced normal maps with wavy gradients which also look cool would be very helpful. So far I was able to avoid it using UV splits and hard edges only.

    Actually, I used a synced workflow throughout the guide, by baking in Xnormal and displaying in Marmoset 2 with proper tangent space selected. As I see it, a synced baker-engine pair won't save you from the worst possible gradients.
  • pior
    Offline / Send Message
    pior insane polycounter
    Well, the way I see it is as follows :

    Rather than : "a synced baker-engine pair won't save you from the worst possible gradients", it's more like : "a synced baker-engine pair allows you to not even have to worry about what your normalmap looks like".

    Now again I totally understand that not every scenario is the same, and in many cases the "tricks" are indeed necessary because things don't always go as planned. But still, I'll take Source/Dota2 as an example. I see artists working with it jumping through all sorts of hoops and hacks (arbitrarily adding hard edges where there is no need for them to begin with, doing funky things to their normalmaps by hand, and so on) even though there is a perfectly synced workflow available (Handplane conversion) that allows users to not have to worry about any of that, saving hours of headaches in the process.

    This is why I believe that technical information like the one Jed is providing above is super important, as it helps us understand the very root of issues. After all, game dev is quite technical by nature !
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    I guess I'll have to install an engine like Unity or Unreal Engine, establish proper synced workflow and do some tests
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Alright, this is what I've done:

    1. Bake Object Space map from Xnormal
    2. Convert to Unity tangent space using Handplane
    (To be sure, I did that again using Unity tangent space plugin for Xnormal, but that gives the same result)
    OOspiCU.png
    grhz6Ba.png

    The result:

    itz0OZq.png
    moved lighting source a bit
    0TiBKd1.png

    As you can clearly see, I couldn't get away with extreme gradients even using a perfectly synced workflow. I'm pretty sure Xnormal does the same job anyway.
  • pior
    Offline / Send Message
    pior insane polycounter
    Then that means it is time to investigate further in order to help making this particular pipeline better ! It should be up to the engineers to provide reliable tools - not up to the artists to find ways around problems.

    Here is another example related to Source. Many artists are running into the issue highlighted here, which is actually caused by a cool "hack" introduced by Maya engineers and set as default without anyone being made aware of it.

    mayanormalssmoothingoptionswdota2_zps0869d090.jpg~original

    Notice how the normalmap with "strong gradients" is actually the one giving accurately shaded results, as seen at the bottom of the texture. And it just so happens that the part on the middle right has less shading ; but it's irrelevant, because once you know for sure that a given pipeline is accurately synchronized, you don't have to worry about any of that.

    I believe that the best course of action is to always keep digging, in order to make the tools more and more reliable. Of course it's good to know of hacks when there is no other choice available ; but the bigger picture is all about making the life of users easier in the long run :)
  • Farfarer
    Offline / Send Message
    Farfarer Polycount Sponsor
    Pior, that's odd. I use area & angle weighted normals in Modo and Handplane for the conversion and I've had no issues. Although I'm exporting via FBX, rather than OBJ.
    Alright, this is what I've done:

    1. Bake Object Space map from Xnormal
    2. Convert to Unity tangent space using Handplane
    (To be sure, I did that again using Unity tangent space plugin for Xnormal, but that gives the same result)
    <images snipped>
    <images snipped>

    The result:

    <images snipped>
    moved lighting source a bit
    <images snipped>

    As you can clearly see, I couldn't get away with extreme gradients even using a perfectly synced workflow. I'm pretty sure Xnormal does the same job anyway.
    This is an issue with the way Unity uses tangents in forward rendering, not with Handplane.

    There's a couple of ways around it...

    - Use deferred rendering (Pro only).
    - Roll your own shaders and convert from tangent to object space in the pixel shader, then do lighting in object space.
    - Normalizing the view & light vectors in the pixel shader helps to reduce the effect, but not remove it (but it's a bit easier than rolling your own fully custom shaders).
  • pior
    Offline / Send Message
    pior insane polycounter
    Fafarer : very true ! I should totally have used a more "complete" format than obj, since it basically culls out a lot of vital information for baking (it is sometimes a good thing, and sometimes a bad thing). Whenver time allows I will go a bit deeper into investigating the problem.

    Still, the new "cool trick" introduced in Maya can cause a lot of issues - I've seen it happen in productions at the studio, and I see artists struggling because of it online too :/
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Time to call it a day. Update is up! Updated the link in the OP.

    Can't wait to hear what you think about it.
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 5
    You should probably recommend this method for combining normal maps because it's the only method out of those you listed that's mathematically correct and usually gives the best results (depending on what you want to do, of course.)
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 5
    You might also take a look at the skewmesh method for correcting some kinds of projection errors.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    You should probably recommend this method for combining normal maps because it's the only method out of those you listed that's mathematically correct and usually gives the best results (depending on what you want to do, of course.)

    I already listed this method, it's on Page 19 :) Unfortunately, I couldn't get it to work no matter what I tried.
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 5
    Maybe I should rephrase that--I think you should recommend it as the one preferred method that you should always use to combine normal maps if you can, unless you specifically want another specific effect from another method of blending. If the Photoshop plugin doesn't work for you for some reason Farfarer has a standalone program that uses the same algorithm:

    Download "RNM Normal Map Combiner"

    Mari also has this algorithm built-in and of course there's a custom Substance node for it as well here, so there are definitely options that people have when it comes to using this technique.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Maybe I should rephrase that--I think you should recommend it as the one preferred method that you should always use to combine normal maps if you can, unless you specifically want another specific effect from another method of blending. If the Photoshop plugin doesn't work for you for some reason Farfarer has a standalone program that uses the same algorithm:

    Download "RNM Normal Map Combiner"

    Mari also has this algorithm built-in and of course there's a custom Substance node for it as well here, so there are definitely options that people have when it comes to using this technique.

    Oh, I just never heard about these methods before! Thanks for informing me, I'll definitely list them in the guide.
  • Chantel-sky
    Offline / Send Message
    Chantel-sky polycounter lvl 3
    Thanks for taking the time to do this :)
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Thanks for taking the time to do this :)

    You're welcome, I hope you find it useful.
  • Tzur_H
    Offline / Send Message
    Tzur_H polycounter lvl 5
    Thanks for taking the time to do this :)
    +1
    I had no idea overlapping/mirrored UVs need to be offset on normal map, why is that necessary?
    Also, now I'm a bit confused, right now I'm stacking all of the UVs in 0-1 space, no matter the texture map type. Offsetting UVs is needed only when doing the bake for the normal map? not for all of the other types of textures?
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 5
    Mirrored UVs need to be offset in UV space because that way the baker knows exactly what geometry you want to bake to. In some cases it might get different results for different geometry (especially if you modeled the high-poly hurriedly and it isn't 100% symmetrical) and if the baker tries to bake two different spots on the low-poly to two different parts on the texture the results might differ, giving you an incorrect bake with artifacts. It can be especially problematic if the tangent basis is order-dependent, since even if the high-poly and low-poly have identical shapes and normals the tangent basis may be different between the different low-poly pieces, resulting in quite different normal map bakes (but probably pretty similar bakes of other maps.)

    I'd recommend just leaving the UVs offset into UDIM 1002 anyway, unless you're working with a megatexture engine that's capable of rendering multi-tile UVs. But that's a pretty special case these days. If your UVs never overlap in 0..1 UV space the baker can't make mistakes.
  • martinszeme
    Offline / Send Message
    martinszeme polycounter lvl 8
    Just wanted to ask couple of questions as I've followed the guide and normals are still acting up.
    Every time I bake stuff I almost have to figure out these techniques from scratch as nothing works consistently on everything.
    So I've made HP and LP in 3ds max. LP, kas a projection modifier on top of it (edit mesh modifier in between) I export HP and LP as *.SBM.
    LP.SBM has the cage built in, so I turn it on in xnormal and bake it. Bakes visually come out without much artifacts and look fine in PS.
    Yet in toolbag I can't get it to show up correctly (I presume it wouldn't work anywhere with my setup).
    So I've created this image to illustrate what I mean:

    Xhz7Z2Z.jpg

    Glitches showed up randomly even though meshes are absolutely the same.
    Sorry if this is off topic.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Try to export your mesh and cage as .obj
  • martinszeme
    Offline / Send Message
    martinszeme polycounter lvl 8
    Ok, OBJ bake fixes the shading issues and glitches, but hard edges still have that distinctive edging.
    zvYPbRi.jpg

    OvRcucW.jpg


    EDIT: Yet edges near on top are completely fine and smooth. Must be something about the smoothing groups and how I split the UVs.
  • Farfarer
    Offline / Send Message
    Farfarer Polycount Sponsor
    Is the mesh set to use MikkTSpace tangent basis in Marmoset?
  • EarthQuake
    Make sure you have uv splits everywhere you have hard edges.
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    In the next update, I'll probably add a quick troubleshooting/fixing guide.
  • EarthQuake
    Yeah that would be a great idea.
  • martinszeme
    Offline / Send Message
    martinszeme polycounter lvl 8
    Changed it, but it didn't fix it unfortunately. I'll try and fix on my own, don't want to spam the thread. Thanks for the help guys.
  • martinszeme
    Offline / Send Message
    martinszeme polycounter lvl 8
    Quick question. When you split UVs in max for hard edges do you make sure they have some padding around them? So far I've been selecting the hard edges and doing a brake on them in UV Editor and leaving them as they are. So should I nudge these islands a bit to give say 2-4 pixel padding?
  • SuperFranky
    Offline / Send Message
    SuperFranky polycounter lvl 7
    Quick question. When you split UVs in max for hard edges do you make sure they have some padding around them? So far I've been selecting the hard edges and doing a brake on them in UV Editor and leaving them as they are. So should I nudge these islands a bit to give say 2-4 pixel padding?
    Make sure you have as much padding as possible for your current texture resolution. Have at least 8 pixels of padding.
  • EarthQuake
    Yeah you need to physically split the uv islands per smoothing group and move them apart. The problem happens because you have a hard edge with a shared edge in pixel space and the baker can't account for the normals on both sides of that edge along 1 pixel line.
1
Sign In or Register to comment.