I need some clarifications on normal maps and tangent spaces

polycounter lvl 7
Offline / Send Message
huffer polycounter lvl 7
I need some help understanding tangent space, tangent basis and getting a good normal map baking workflow. Please correct me if you notice something wrong! :)

Let's say there's the following scenario: I work in Maya and have to create an asset, the client specifies their engine tangent space is synced to xNormal (Mikk), deliverables are 3dsmax 2013 scene and .fbx file.

Now, my understanding so far was that tangent space (or tangent basis) refers to how the roundness of the geometry is shaded from vertex to vertex, or more precisely, how vertex normals get their orientation for a given object.

Thinking this, I assumed that , for example, the normal map shading artifacts that appear in a mesh with one smoothing group (or all soft edges) baked in Maya and imported in Max are there because Max interpolates the vertex normals differently. Like the difference between angle and area weighted vs unweighted (how 3dsmax would do it):
vertexnormals.jpg

I read some tips on how to avoid shading artifacts in normal maps and achieve greater compatibility between packages, and one solution was to use custom normals - as in the picture. With vertex normals aligned to face, or flattened, one can get rid of extreme gradients in the normal map, but there are engines that don't support custom normals.

Now, the workflow I had in the past was :
1. Make high poly and low inside Maya, always set the low poly to Unweighted (since angle and area weighted means the vertex normals are custom, and Unweighted is closest to how it looks in 3dsmax)
2. Export fbx (smoothing groups, tangents and binormals on) to xnormal and bake
3. Import fbx in 3dsmax, apply normal map, preview.

But now I realise, I had it all wrong! (well almost all :poly124:). Now I understand that tangent basis refers to something different - it's another set of coordinates that determines how each color channel from the normal map is applied to the mesh and chow it hanges per-pixel normals. Only one part it's determined by vertex normals, and the rest are calculated inside the specific software with its own method. This scheme made it clear (from cryengine, vertex normals are gray):

OSTS2.jpg

I thought that using Unweighted in Maya meant you were safest from shading artifacts when importing from Maya to Max, but now I realise it didn't even matter - I imported my fbx custom normals anyway, and it looked great inside 3dsmax, so in the end isn't it better to use all the custom normals you can?

Are there even any modern engines today that can't import custom vertex normals? Or, let me rephrase:

1. engines that don't support custom normals reset every vertex normal (apart from split vertex normals in smoothing groups) and recalculate them when you import a mesh? If so, and if the client demands a scene in 3dsmax, can I assume his engine is synced to 3dsmax, and can I export an obj from Maya to Max, reset normals, and use that obj further in xnormal?

2. .fbx and .obj can import in 3dsmax with changes to vertex normals. When in the past I exported my .fbx to bake in xnormal, the low poly was baked with the exact vertex normals as in Maya? Or were they recalculated inside xnormal?

3. If they were the same as in Maya, why not use angle and area weighted or custom normals all the time to achieve gradient free normal maps and minimise any tangent basis differences? Fbx and even obj support it by default.

4. If they were recalculated, what's the point in importing my .fbx in 3dsmax with different vertex normals from the baking program? I'll only see it properly in engine.

5. More so, wouldn't the ideal preview be just inside xnormal if you don't have acces to the engine?

6. Any shading issues that show up so far are solely from errors in tangent space calculations inside of 3dsmax ( 2013 is buggy in that manner), so isn't it completely unreasonable to demand fixes in the normal map (shading artifacts) just by viewing them in the max viewport? The normal map is in theory perfectly synced with Mikk, so if you change it to look good in max, or maya, you break it for the engine.


THANK YOU

Replies

  • Dylan Brady
    Offline / Send Message
    Dylan Brady polycounter lvl 7
    you cant import custom normals for skeletal meshes in UDK or UE4.
  • Quack!
    Offline / Send Message
    Quack! polycounter lvl 11

    1. engines that don't support custom normals reset every vertex normal (apart from split vertex normals in smoothing groups) and recalculate them when you import a mesh? If so, and if the client demands a scene in 3dsmax, can I assume his engine is synced to 3dsmax, and can I export an obj from Maya to Max, reset normals, and use that obj further in xnormal?

    I would not assume that it is synced to max, no. It could be that their engine can read the tangents and binormals from the fbx file and thus 'act' synced. It should also be noted that most modern engines will just read the tangents and binormals from the fbx file. Therefore, 1 bake from xnormal should look pretty much synced in many modern engines(UE4, Toolbag2, Substance, etc) if using the proper workflow.
    2. .fbx and .obj can import in 3dsmax with changes to vertex normals. When in the past I exported my .fbx to bake in xnormal, the low poly was baked with the exact vertex normals as in Maya? Or were they recalculated inside xnormal?

    Xnormal can read the tangents and binormals of the file, so if those were sent along with it, they would be read and used by xnormal unless otherwise stated. This pertains to later versions of xnormal, I am unsure of when this became a feature.
    4. If they were recalculated, what's the point in importing my .fbx in 3dsmax with different vertex normals from the baking program? I'll only see it properly in engine.

    Correct. Viewing normal maps in 3dsmax is a very bad idea unless they have a custom directx shader they use.
    5. More so, wouldn't the ideal preview be just inside xnormal if you don't have acces to the engine?

    If xnormal displays their normals using the same basis as their calculation, then yes.

    6. Any shading issues that show up so far are solely from errors in tangent space calculations inside of 3dsmax ( 2013 is buggy in that manner), so isn't it completely unreasonable to demand fixes in the normal map (shading artifacts) just by viewing them in the max viewport? The normal map is in theory perfectly synced with Mikk, so if you change it to look good in max, or maya, you break it for the engine.

    A few things could be at play

    1. The company has a custom directx shader for max that can display proper normal maps that are synced.
    2. The company doesn't know that Max's default viewport is a terrible place to view your normal maps.
    3. The company views all normal maps in engine.

    Viewing a normal map in the target engine is by far the best way to work.
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 4
    You should always ask your client whether their pipeline allows custom normals or not. They may want you to avoid custom normals even if their pipeline allows it, because it's easy to destroy custom normals in Max if you're not careful, making it more difficult for them to make changes to your deliverables if they need to. Custom normals can affect how you put your mesh together, especially at lower polygon counts, so you have to ask your client if it's OK to use them. You also need to make sure that both xNormal and 3ds max can import the custom normals; if the normals are different in either xNormal or Max you'll be in trouble. If the client says their tangent space is synced to xNormal, you need to bake in xNormal so that the tangent basis matches up (of course), but you also need to make sure that when you export the low-poly mesh you export it with tangents and binormals OFF so that xNormal calculates tangents and binormals with its own tangent basis calculator. If you export tangents and binormals from your DCC xNormal will use them, which will make your client unhappy because your mesh will be using Maya's tangent space instead of xNormal tangent space, even though you baked it in xNormal, and since they don't have Maya licenses they won't be able to re-bake the normal map. Also, you should probably have "Compute binormal in the pixel shader" checked in the xNormal plugin options, but again this is something you should verify with your client. I know that it has to be checked for UE4, Blender, and Toolbag 2, so the general standard is to have it checked. If you bake using the correct process, you shouldn't have to worry about any tangent basis differences as long as your client is giving you correct information.

    So in summary, ask your client these questions:
    1. Can I use custom normals to smooth out gradients in the normal map bake, leading to fewer artifacts from texture compression and low mips? Or, would you prefer to have the flexibility to change my deliverables that using unaltered normals gives?
    2. Should I check "Compute binormal in the pixel shader" in Xnormal's options for the MikkTSpace plugin to get a correct tangent space sync?
    3. Should I bake with X+Y+Z+ swizzle or X+Y-Z+ swizzle? (The answer will probably be X+Y-Z+ because they use 3ds Max)

    Once you have that information from your client, you'll be able to bake a good normal map that's synced to their engine, and everyone can be happy. :)
  • [PB]Snoelk
    Offline / Send Message
    [PB]Snoelk polycounter lvl 7
    we got some unsynced in out engine as well. this was solved after we changed our import mesh handling. as you stated with xNormal the mikkt tangend base extension.
    there is a c++ sdk available to handle meshes and generate tangends and binormals. After we integrated this in our import procedure our normals looks in synced.
    so this is a very important when importing baked normals. if you generate tangends and binormals with plain directx mesh import you get big problems with mirrored, uv splits, uv mirrors etc.
  • huffer
    Offline / Send Message
    huffer polycounter lvl 7
    Thanks for the replies! Things are way clearer now, here are my conclusions :):

    - If I create my low poly inside Maya, using "Angle and area weighted" or "angle weighted" for vertex normals definitely means custom normals. Even using "Unweighted" might mean custom normals (different from 3dsmax or engine standard), but,

    - it doesn't matter, since obj or fbx have vertex normals stored by default. They do get broken if geometry is added to the low poly, so in order to make sure the vertex normals are "non-custom" is to use and Edit Normals after export in 3dsmax to reset normals and recreate smoothing groups, then use that mesh for baking

    - I'm still not sure what makes a vertex normal a custom vertex normal, or what happens when you import an mesh with edited normals inside an engine that doesn't support custom normals. Even with standard edits in 3dsmax sometimes the vertex normals get wonky. Should I reset them once I'm finished with the low poly and then proceed to bake? If I split a couple of vertex normals to create a hard edge, does that mean they're custom? If an engine doesn't support custom normals, it resets all vertex normals from Maya or Max, except where hard edges are?

    - since I was exporting .fbx with tangents and binormals stored in the mesh, my bakes looked good both in Maya or Max and in the engine on a simple fluke. Baking them in xnormal didn't mean they were synced to Mikk, I just carried tangent basis data over in the fbx. Even when I baked with 3dsmax and it looked good it was still a fluke, I still had tangent basis on export in the fbx from 3dsmax! So now with this knowledge, the request to "bake in Xnormal" should be rephrased to the clearer: "sync your normal map bakes to xnormal/Mikk tangent space". The bake won't look good in either Maya or Max, but in the engine (and Marmoset Toolbag 2), unless there's a custom shader involved on the client part.

    - Are there game engines that use the tangent basis from 3dsmax or Maya, or engines that simply use the tangents and binormals from the fbx and don't perform any tangent basis calculation? Is it viable to use a workaround with baking object space in your program of choice, then use handplane to get a target synced tangent space map?

    This was a huge eye opener, I can't believe I've been working partly wrong all this time! :poly122:
  • JedTheKrampus
    Offline / Send Message
    JedTheKrampus polycounter lvl 4
    The normal map looking good in the engine was probably not a simple fluke. Likely the tangents and binormals that got imported into Xnormal got imported into whatever engine you were using as well, meaning that they would look fine because they're using the same tangents that they were baked with, but would be actually be using the wrong tangent space. This has the potential to cause problems later if your client wants to re-bake the normal map.

    Most engines these days can import custom tangents from a .fbx file, whether they're from Max, Maya, or Blender. There are probably a few engines out there that calculate the tangent space like Max or Maya but they're probably used at big studios that I've never worked for.

    Usually you can count on edges being marked hard/soft (or, in Max, smoothing group splits) to make it through any pipeline, but you must remember to have a UV split anywhere you've got a hard edge. (You don't necessarily have to have a hard edge where you have a UV split, though. Depending on your model it may be better to have a soft edge where you have a particular UV split.)
  • huffer
    Offline / Send Message
    huffer polycounter lvl 7
    Well everything is much clearer now, almost an year later, thanks a bunch. I'm able to get synced bakes 90 percent of the time (testing in extreme cases, using lots of soft edges or weird triangulation).

    However, is this still valid - if I export an FBX with tangents and binormals ON and bake in xNormal, xNormal should not use mikk tangent space, but instead use whatever is in the FBX? If I apply the baked map in 3dmax (from where I exported the FBX) it should look perfect right? I did some tests and it doesn't look good.

    So far I've either baked in 3dsmax and exported to engine FBX with tangents and binormals (it worked) or exported for bake FBX (without tangents and binormals) or OBJ, baked in xNormal (using mikk) and imported in engine (UE4), where I checked recalculate tangents, and it also worked right. 
Sign In or Register to comment.