Fixed Normal Map Bake Skewing
null
Discovered a new way to fix skewing when baking my normal map detail in Substance Painter and wanted to share for those who have fixed the problem or have ran into the problem of normal map skewing. I used a vertex normals Script for 3DS max and custom edited the normals of the face where the detail is going to bake on. I uploaded 4 images of the main phases of the process. Also not sure yet if it works for all situations so will continue experimenting and seeing how this technique works out in other situations. If there is a more efficient and faster way to achieve this in Substance painter do comment please.
Replies
There's a better trick IMO,
https://forum.allegorithmic.com/index.php?topic=12713.0
For the sake of comparison we're going to look at bakes from 4 lowpoly and a highpoly with repeating details.
A: Default
B: Copy of your weighted + edited vertex normals
C: Edit to A having spent a minute adding a few edge loops and a vert in the middle
D; Duplicate of C with weighted vertex normals
The resulting bakes look like this.
A has skewing as you might expect. B has even harsher skewing in exchange for two sides having acceptable projection. C is flawless. D has some minor issues around the edges of the intrusion that it shares with B due to the face weighting but is otherwise fine.
The trick here is that the above was shown on 4 duplicates of A. They're all identical.
This is where the objectspace normalmap comes in. The original four bakes were done in objectspace and then converted to tangentspace using a different lowpoly with a matching uv.
The setup and conversion is pretty quick. The first lowpoly used for baking is best done after unwrapping the uv but before triangulating. The second lowpoly should preferably be triangulated and using your final topology/vertexnormals/smoothing. The smoothing between the 2 lowpoly does not need to match. The orientation (but not position) of both lowpoly does need to match however.
Next is baking. You don't need to use substance painter for this but the objectspace normalmaps it gives work fine.
If you're using xNormal go into your normalmap bake settings and make sure your swizzle coordinates are X+ Y+ Z+ and that tangent space is unticked.
Next is to go into xNormal > Tools > Object/Tangent space converter. Enter your second lowpoly and your normalmap and convert to tangentspace.
If there's a way to do this is Painter I'm not aware of it.
There. Done.
If you are authoring for an older engine that doesn't use MikkTspace (unreal3 or source for example) convert to tangentspace in Handplane rather than xNormal. Either that or texture in Painter as normal, convert back to objectspace with xNormal (invert the green channel after exporting from Painter), and then run it through Handplane.
Pros:
- All maps only need to be baked once and use the same projection regardless of baker
- Highpoly details are not distorted along edges
- Lowpoly can still be edited after the initial bake (which can allow for some honestly silly vertex normal setups)
Cons:- Additional lowpoly does take time to setup
- Sheers off uv seams
That last one matters a fair bit. Anything not caught in the uv gets discarded when being converted. How precise it is when determining what gets kept depends on the resolution. To get around this you render your objectspace at several times your intended resolution and downscale after your final conversion to tangentspace. Give it a lot of padding to account for the downscaling.