Making Tangent space normals completely override the mesh's normals?

johngloid

I'm new to the world of normal mapping. I'm looking at normal maps to fix some really bad mesh normals on a corrective blend shape. I'm using UE4, but this would apply to any game engine, I imagine.

Tangent space normals haven't worked out initially, since they don't override the underlying mesh normals. Seems more like a blending effect with the mesh normals and the normal map. Object space sounds like what I want, as it'll totally override the mesh's normals, but it's not supposed to be used with deformable meshes. Except for this on Polycount: 

http://wiki.polycount.com/wiki/Normal_Map_Technical_Details

Rainbow colors. Objects can rotate, but usually shouldn't be deformed, unless the shader has been modified to support deformation. Object-space is also called local-space or model-space.

Is there a way to completely override the mesh's normals using a tangent normal map somehow? That seems like the path to least resistance. Or possibly a way to make object space normal maps more deformation friendly?

throttlekitty

Object space would be the way to go, I don't know the details of using it on a deformable mesh, other than that it can be done. These maps are essentially absolute in that they totally define the normals. However, you'd be in a position of either trying to paint or sculpt in such a hacky way that it would be difficult to get a good result. You could potentially paint a correction in a tangent space map, but again it's a bit of guesswork and you're still adding overhead by adding another texture for the shader to blend.

Is it possible to redo the geometry to support the blend shape or could the morph be done better? Is this blend fixing a common motion, or some one-off animation that's only going to be visible for a moment in a cutscene? Garbage in, garbage out.

pior

"Or possibly a way to make object space normal maps more deformation friendly?"

They are. Them being not deformable is just a widespread misconception, it all depends on the implementation.

However ...

Don't bother trying to use OS maps on a character - it's pointless since they are barely supported by anything at this point. Focus on fixing your initial problem (poor surfacing of the original model), which will be dealt with by working with higher density models.

ActionDawg

You should share some images if possible.

I would also recommend working within tangent space rather than going for object, fixing the model to get the shading you need. As for a tangent space that overrides the underlying normals... not possible. A tangent space normal map relies on the low poly normals, essentially storing the difference between the high poly and low poly normal at any given point.

musashidan

Agreed with the above. Fix the mesh normals. Anything else is hacky and a bad habit.

johngloid

Thanks all for the quick replies and explanations! I really appreciate it.

I'm going to go back to the drawing board and figure another way out to avoid the bad normals, as suggested. Cheers!

ActionDawg

if you are going between two extreme poses then, without knowing exactly what you're working on, a good idea may be to make the high and low poly meshes modeled and unwrapped halfway between those two poses and bake textures from that. This way you will get a good amount of pixels in the normal map to work with for both poses.

An example of this being eyelids which are best authored with the initial pose looking sleepy so the final textured shading is nice and clean when the eyes are wide open or totally closed.