The Wonders of Normal Map Baking

Rooster128

So I've done a bit of testing with normal maps in Unreal to see how optimal I could make my results, after testing different things for years. I've known about smoothing groups causing issues for normal maps for years. But only recently learned about how different engines use different tangent spaces and not all normal maps are created equal when going from 3D Package to Engine.

I know this has been talked to death on the polycount forums, but I thought I'd share my own personal results from experimenting. Note, I probably could have done more tests to control for more (triangulation, smoothing groups) but I was just going for the best results at the time.



Keeping your entire low poly mesh smooth and just baking out from a 3D package or xNormal wont always work. In my mind, this is the best workflow:

1. Triangulate your mesh by hand. This removes any discrepancy between how different exporters/importers triangulate models.
2. Crease the concave edges of the mesh (the inwards-facing corners)
3. Finally, export the meshes. Your high poly, your low poly, and your game mesh. When exporting, make sure smoothing groups, smooth mesh, and export normals are on. Do not export bitangents/normals. Export out a bake cage if you want that level of control of your bake- I usually let xNormal do this for me and fix any problems on the normal map by hand.
4. Open xNormal. Import the low and high poly models. Scale by 10. Use the raytrace calculator to make an approximate cage- or import in your own cage. Bake out an object-spaaace normal map.
5. Open up Handplane 3D, target your low poly mesh, the baked out object-spaaace map, and specify the engine you will ultimately be using. Bake.
6. Take this resulting normal map and bring it into your engine. Also import your mesh. In the case of Unreal, if its a static mesh, turn on Explicit Normals, so that they are read from the mesh.

In my mind, these are the best practices for generating good normal maps (no matter the engine- works well in Unity too).


If you've baked out tangent-spaaace normals from xNormal and have been using those, don't worry! You can still create corrected normal maps out of those by loading up xNormal, and using the Object/Tangent spaaace converter tool to make an object-spaaace normal map out of your tangent spaaace map, and running that through HandPlane3D with the same steps mentioned above.

I'll probably add more clarification/pictures a bit later. I'm hoping this helps somebody- I know its not a fun process to work through. Any questions, let me know.

Obscura

Is is that necessary to use baked normals on this simple shapes? As you also can see, none of your results are perfect. One way to get better result would be to use hard edges, the second way would be to use chamfers (don't worry, its ok in 2014). Anyways, you would avoid a lot of headache if you would simply go with custom normals insead of trying to do tricks to get proper shading with baked normalmaps when a this simple shape not must require it. Here is what I mean:

The first mesh is the same as yours, and its smoothed. Looks really bad, for sure. The second if the same mesh as your, just with a chamfer. The smoothed shading is better, but still far from perfect. The third mesh is the same as the previous, but with custom normals, and looks perfect to me even without a normalmap. And I'm not sure about adding a baked normalmap to this mesh would make it any better.

Edit: Here is the same example with the worst case mesh:


Hope this helps!

WarrenM

Right, but he's not talking about that specific mesh. He's talking about workflow. The mesh is irrelevant, it's just an example.

Obscura

Right, the mesh is irrelevant, I just wanted to show that chamfers and custom normals helps a lot to achieve perfect result.

EarthQuake

UDK or UE4? I think UE4 can load synced normals baked from XN now, can't it?

Rooster128

Oh- not sure. I was talking about the workflow for UDK. But if UE4 has synced normals from XN now, that would surely cut down on the work!

Obscura, it's not about the shape, it's about the workflow for getting the normal to read correctly in a game engine. To be fair, I do agree beveling edges can help a lot with normal map baking too.

Obscura

I know that its about the workflow, I just wanted to say your workflow still not gives perfect shading(slight gradients are still visible). And if you would use chamfers/custom normals, then you could get the perfect shading. It just wanted to be a tip about how you can improve the workflow to get better result

repete

UE4 should be synced to XN as they both use mikktspace as default.

I wish lux would add the mikktspace normals as an option as it's not rocket science ! They are looking into it however :poly121:

EarthQuake wrote: »

UDK or UE4? I think UE4 can load synced normals baked from XN now, can't it?

Rooster128

Again Obscura, I think you are missing the point- but I do agree my method is not perfect and some gradients do exist. I've known about beveling and chamfering- that are quite useful. And custom normals is what I meant by 'creased edges' - I do realize in Maya that is a seperate thing, what I meant was soft/hard normals or smoothing groups (custom normals?)

What I'm interested in is better workflows for baking and maintaining normal maps. But if normal maps look correct straight out of xN for UE4 use, then I can take a huge sigh of relief for sure.

sprunghunt

UDK should also load Xnormal synched normals from a FBX file. You just have to uncheck the checkbox for regenerating normals. Otherwise it overwrites the normals in the FBX with new ones.

Rooster128

Which export option was that, sprunghunt?

sprunghunt

Rooster128 wrote: »

Which export option was that, sprunghunt?

read this:

http://udn.epicgames.com/Three/XNormalWorkflow.html

http://www.polycount.com/forum/showthread.php?t=102838

You shouldn't need to use handplane for any game engine now. Most are compatible with xnormal in some capacity.

Rooster128

Oh man! This is amazing!

How did I not see this before?! Haha- thanks!

Rooster128

I tried what the documentation suggested- it worked for UDK, not for UE4.

Seems like I'm not the only one with this problem either...

https://answers.unrealengine.com/questions/14375/normal-map-workflow-with-xnormal-is-not-working.html

My workflow using Handplane doesn't work with UE4 either. I really think Epic ought to look into implementing derivative maps- because having so many things factor into how light is bent around a surface is crazy.

Rooster128

Actually... just got my workflow I suggested on the first post to work using HandPlane3D, saving the normal map as an 8-bit TGA. I did export out normals and tangents though...

Rooster128

I want to bump this thread again because I have been experimenting with my workflow and the workflow sprunghunt linked, and I'm still finding I'm writing into weird normal map inconsistencies. Not huge ones- the normal maps seem to work fine for anything that isn't going to be tiled. But there are inconsistencies I always run into that make it hard for modular pieces read as tileable seamless assets.

I've actually found Obsura's method to work the best for making modular pieces, since it saves me time and saves on resources (you need a lot less geometry to "control a bake"), with none of the weird lighting artifacts.

This all being said, I've definitely seen some games out there use normal map baking for their modular building pieces. There has to be a de facto method out there for producing the perfect normal maps, even for tileable pieces.

sprunghunt

Rooster128 wrote: »

This all being said, I've definitely seen some games out there use normal map baking for their modular building pieces. There has to be a de facto method out there for producing the perfect normal maps, even for tileable pieces.

I have no problems with using xnormal and baking my normal maps in UE4 using the same techniques outlined in the documentation I provided. I use Maya LT, xnormal, and zbrush.

I was having a problem with importing any normals from Maya LT because the Maya LT FBX exporter was too new for UE4. However since version 4.3.1 this has been fixed.

Perhaps you could post pictures of the problems you're having? It may have nothing to do with the normalmaps.

Jesus001

Obscura said:

Right, the mesh is irrelevant, I just wanted to show that chamfers and custom normals helps a lot to achieve perfect result.

what is custom normals?

Obscura

What is love?