I need some help understanding tangent space, tangent basis and getting a good normal map baking workflow. Please correct me if you notice something wrong!
Let's say there's the following scenario: I work in Maya and have to create an asset, the client specifies their engine tangent space is synced to xNormal (Mikk), deliverables are 3dsmax 2013 scene and .fbx file.
Now, my understanding so far was that tangent space (or tangent basis) refers to how the roundness of the geometry is shaded from vertex to vertex, or more precisely, how vertex normals get their orientation for a given object.
Thinking this, I assumed that , for example, the normal map shading artifacts that appear in a mesh with one smoothing group (or all soft edges) baked in Maya and imported in Max are there because Max interpolates the vertex normals differently. Like the difference between angle and area weighted vs unweighted (how 3dsmax would do it):
I read some tips on how to avoid shading artifacts in normal maps and achieve greater compatibility between packages, and one solution was to use custom normals - as in the picture. With vertex normals aligned to face, or flattened, one can get rid of extreme gradients in the normal map, but there are engines that don't support custom normals.
Now, the workflow I had in the past was :
1. Make high poly and low inside Maya, always set the low poly to Unweighted (since angle and area weighted means the vertex normals are custom, and Unweighted is closest to how it looks in 3dsmax)
2. Export fbx (smoothing groups, tangents and binormals on) to xnormal and bake
3. Import fbx in 3dsmax, apply normal map, preview.
But now I realise, I had it all wrong
! (well almost all :poly124:). Now I understand that tangent basis refers to something different - it's another set of coordinates that determines how each color channel from the normal map is applied to the mesh and chow it hanges per-pixel normals. Only one part it's determined by vertex normals, and the rest are calculated inside the specific software with its own method. This scheme made it clear (from cryengine, vertex normals are gray):
I thought that using Unweighted in Maya meant you were safest from shading artifacts when importing from Maya to Max, but now I realise it didn't even matter - I imported my fbx custom normals anyway, and it looked great inside 3dsmax, so in the end isn't it better to use all the custom normals you can?
Are there even any modern engines today that can't import custom vertex normals? Or, let me rephrase:
1. engines that don't support custom normals reset every vertex normal (apart from split vertex normals in smoothing groups) and recalculate them when you import a mesh? If so, and if the client demands a scene in 3dsmax, can I assume his engine is synced to 3dsmax, and can I export an obj from Maya to Max, reset normals, and use that obj further in xnormal?
2. .fbx and .obj can import in 3dsmax with changes to vertex normals. When in the past I exported my .fbx to bake in xnormal, the low poly was baked with the exact vertex normals as in Maya? Or were they recalculated inside xnormal?
3. If they were the same as in Maya, why not use angle and area weighted or custom normals all the time to achieve gradient free normal maps and minimise any tangent basis differences? Fbx and even obj support it by default.
4. If they were recalculated, what's the point in importing my .fbx in 3dsmax with different vertex normals from the baking program? I'll only see it properly in engine.
5. More so, wouldn't the ideal preview be just inside xnormal if you don't have acces to the engine?
6. Any shading issues that show up so far are solely from errors in tangent space calculations inside of 3dsmax ( 2013 is buggy in that manner), so isn't it completely unreasonable to demand fixes in the normal map (shading artifacts) just by viewing them in the max viewport? The normal map is in theory perfectly synced with Mikk, so if you change it to look good in max, or maya, you break it for the engine.