Normal Map Baking Issues/ Smoothing
polycounter lvl 11
I'm learning how to bake normal maps for hard surface models. Already read the polycount wiki and watched a ton of videos which taught me a lot. Then I encountered more issues than before. Ignorance is bliss...but I want to make sexy bakes!
The first problem is probably my low-poly topology. I thought I knew what I was doing, but probably not. Just learned about smoothing groups and how you should have UV seams where there are hard edges. So I applied that throughout the model, which caused some pretty nasty smoothing artifacts. Pictures describe better than words:
This is just the cage I used for xNormal and the high poly model on right. Maybe I'm not using the high poly to it's full potential.
I tried a bunch of different methods to bake. All smoothed mesh, adding more bevels on low, playing with the cage, etc. For some reason the normal map messed up some of the surface pieces, so instead of going out they cut in. I checked face normals, the cage, etc, dunno why it's doing that since the cage is covering the high poly.
Maybe my settings are messed up somewhere? Cause I tried at a school computer and the map it spat out was even worse! The UV packing for this model was done using Maya's layout button which put everything in 0,1. I just wanted to test quick normal maps. But when I did it a second time and tried to bake in xnormal, there were some weird stuff going on (in the second normal map). Transfer maps in Maya on the same model seemed to work fine. I also noticed sometimes exporting models from Maya to UDK screwed up the shading and I had to reimort the FBx file and export it out again then it was okay. I check my export settings and it's all good. Weird tech issues..
I'm usually able to solve problems like this with a google search. But I read pretty much everything there is about normal maps and I can't figure this one out...
The first problem is probably my low-poly topology. I thought I knew what I was doing, but probably not. Just learned about smoothing groups and how you should have UV seams where there are hard edges. So I applied that throughout the model, which caused some pretty nasty smoothing artifacts. Pictures describe better than words:
This is just the cage I used for xNormal and the high poly model on right. Maybe I'm not using the high poly to it's full potential.
I tried a bunch of different methods to bake. All smoothed mesh, adding more bevels on low, playing with the cage, etc. For some reason the normal map messed up some of the surface pieces, so instead of going out they cut in. I checked face normals, the cage, etc, dunno why it's doing that since the cage is covering the high poly.
Maybe my settings are messed up somewhere? Cause I tried at a school computer and the map it spat out was even worse! The UV packing for this model was done using Maya's layout button which put everything in 0,1. I just wanted to test quick normal maps. But when I did it a second time and tried to bake in xnormal, there were some weird stuff going on (in the second normal map). Transfer maps in Maya on the same model seemed to work fine. I also noticed sometimes exporting models from Maya to UDK screwed up the shading and I had to reimort the FBx file and export it out again then it was okay. I check my export settings and it's all good. Weird tech issues..
I'm usually able to solve problems like this with a google search. But I read pretty much everything there is about normal maps and I can't figure this one out...
Replies
There are certain types of topology that require a hard edge in order for the vertex normals to properly align. The most common is a 90 degree corner edge. You can either bevel it and keep it soft, and also one UV shell, or you will need to make it a hard edge, and cut the UV shell. Much of what seems to be the problem in your images are surfaces like this, as seen around the drawer handles and the circular parts.
This thread should explain in further detail.
Additionally, in your fourth image down, normal maps don't bake details very well from high poly elements that are perfect 90 degree angles from the low poly. You would want to either make low poly topology to match the high poly, or pinch the high poly element more so the edges can be seen better from a straight-on viewpoint.
This thread will explain averaged normals
Your last image is concerning. I would suggest deleting history and re-exporting your model and see if that happens again.
I'll have to do a lot of work to fix this model, so in the meantime I went to the bare basics of baking just to see if there is a software problem. Turns out there is an issue when I export the mesh as FBX to xNormal, it creates wierd normal/tangent errors. I checked with UDK and it seems to accept the FBX meshes fine, so I'm guessing there is a problem with xNormal. I double checked literally everything in Maya. The only thing I didn't care too much about was paddign the UV shells in the UV map cause they just created black lines on the edges which is easy to fix. But the vertex normals and stuff is just acting strange...
So here is what the mesh looks like in the 3D viewer when I export obj. It works fine.
But when I export as FBX, following the guide from UDN ( http://udn.epicgames.com/Three/XNormalWorkflow.html ), there's all sorts of problems with tagents and normals...
The same FBX mesh using same export settings (but with the obj normals applied) looks fine in UDK.
Did some other real simple tests, sorry if it's redundant. But it proves to me that obj baking in xNormal works. What I'm also thinking is there's not enough texture resolution in the previous mesh to hold all the details of the normal map. I was using a 2048 texture though which seems to be pretty large or standard or simple enviro asset. TBH that asset isn't even important so maybe I should ditch the normal map altogether on something like that and use tiling texture/nDo2 with more bevels?
Going to install a newer version of xNormal later and see if it accepts FBX. Thanks for the help guys. If it weren't for polycount I wouldn't even know about these issue. Hopefully someone knows the FBX problem.
So I guess now I have to work on my modeling skills. ZacD mentioned I had some useless tris. TBH I just made the low poly by using Maya 2014's reduce to lower the poly count and cleaned up some edge loops. But I still left some of the high poly topology in there in case I had to edit later. But Maybe I should forget about high poly topology completely since the low poly is all that matters?
I'm not sure if the one on the right is an improvement since the quads make long tris? seems to smooth better though..
Though there are probably a lot of areas where you can simple merge the meshes into one larger, watertight mesh. This will make baking easier, and save you UV space (so you're not uving/texturing bits that will never be seen).
Also, watch your geometry density, you have cylinders that are all over the board in terms of number of sides/roundness. The larger a round shape is, the more sides it has. You also have some totally unneeded trim elements modeled into your lowpolys, and small bevels and things, but then skimp on the roundness of larger cylinders. Make sure to concentrate your geometry on the silhouette.
Thanks a lot for the tips by the way! I think I saw you reply to every thread on normal mapping, was so helpful.
Another thing to consider is what you can get away with by just using the normal map to convey shapes. Those 4 circles for instance would look a lot rounder with less geometry if the lowpoly was simply flat there and the circle shapes were added with the normal map. You want to make these sort of decisions depending on how close the player would get to the object and the spec (mobile, console, pc, next gen console?).
Oh and yes, the last shot you posted, the right version looks a lot better. Much cleaner with less wasteful edge loops.