Baking (good) normal maps in Max?
polycounter lvl 8
Hi all,
Despite of countless tutorials, I'm still having issues with normal maps.
I have tried several different ways of setting up UV seams for example, but I still don't get clean results.
I'm baking the maps with Max's render to texture and I must be missing something, because I keep getting quite noticeable artifacts - I marked a few on the screenshot: weird diagonals across my quads.
I have also been wondering whether the low poly mesh should completely encompass the high poly. In the tutorials this doesn't seem to be an issue and rather the cage of the projection modifier needs to cover the mesh, but in my tests it seems as if the cage has no effect and I will get ray missed warnings, until I make sure the low poly covers the high poly perfectly?
All tips would be really appreciated!
Despite of countless tutorials, I'm still having issues with normal maps.
I have tried several different ways of setting up UV seams for example, but I still don't get clean results.
I'm baking the maps with Max's render to texture and I must be missing something, because I keep getting quite noticeable artifacts - I marked a few on the screenshot: weird diagonals across my quads.
I have also been wondering whether the low poly mesh should completely encompass the high poly. In the tutorials this doesn't seem to be an issue and rather the cage of the projection modifier needs to cover the mesh, but in my tests it seems as if the cage has no effect and I will get ray missed warnings, until I make sure the low poly covers the high poly perfectly?
All tips would be really appreciated!
Replies
You need to push the cage out so that it encompasses the high poly model. The low poly model should match up to the high poly model's silhouette as closely as possible.
About the ray misses, IIRC there's some sort of "Use cage" option in the render to texture options, you might have that unchecked.
In this case it looks like you should also straighten out your UV shells, leaving them at an angle like that will leave you with lower quality edge definition. Pixels are square so UVs on edges should be straight if you want them to be well defined in your textures.
Hmm, so should I be using smoothing groups then? I tested that too at some point, but currently I'm not using them. Instead I've been separating UV shells pretty liberally, as it seems to help at least in the tight angles (like 60 degrees and over). My setup is pretty basic - there's a low poly object and the high poly is just a turbosmoothed version of that, with a few extra loops to keep the form.
The cage option is on, but I just read the manual page and it said that the low poly should be *smaller* than the high poly, in other words inside it, so I did that and extended the cage out enough to cover the high poly. Still errors but maybe little less.
Yeah, the shells are a mess, I will straighten them as soon as I get rid of the obvious errors - just getting kind of tired to do them properly, because I've been debugging these maps for a couple of days now
Generally speaking you should separate your UV shells whenever there is a smoothing group change. In other words, split the UV shells where there is a hard edge on your model. See this thread for more info about hard edge normal mapping and related issues.
For organic shapes it's usually best to keep the UVs as continuous as possible and use only one smoothing group.
Just because you think your normal map looks bad doesn't mean it is bad. It's a bad way to evaluate your normalmap.
for example here's a normalmap from something I made earlier:
(note the shading and visible polygon lines on some areas)
and here it is on the model:
notice that the areas that look "bad" on the normal map don't look bad when on the model?
I would suggest you always evaluate your work in the context it's going to be viewed.
I should have mentioned that I do test them - in Unity3D, which is where the assets are primarily meant to be used.
I just tried smoothing groups again and now it actually made a huge difference: My normal maps look pretty ok now. I'm aware of the increased vertex count, though. I don't currently have any diffuse textures on my models (which also makes the normal map errors very unforgiving), but once I get the diffuse texture painted, I will see if I'd actually get away without smoothing groups.
Beautiful walls! Did you bake the normal maps in max?
What does the model look like in unity?
If a model is very simple I'll often use a few smoothing groups here and there too. But you could also try chamfering a few of the edges instead. It would depend on the effect you're looking for though.
I tried xNormal as well, but couldn't really get that much better maps out of it, so for now I'll stick with Max. In Max I've been using the cage for now and things really started rolling, when I tried the export/import feature - so much easier to edit the cage when it's a separate object.
I Can't really show the actual model right now, but it's starting to look alright. There are a few seams here and there still, but I'm pretty sure the diffuse texture will help to hide them.
Not really correct, smoothing groups will only increase your vertex count if they are not along uv seams, vertexes are doubled along uv seams and using a hard edge there will not increase them. Also;
1. You should only ever use hard edges along your uv seams, otherwise you will get errors and artifacts
2. The uv borders tend to be the most natural place for hard edges.
You should always use a cage whenever you have any hard edges, or else you will get a "gap" in projection at the hard edge. When using the "offset method" in 3ds Max, your raw mesh normals are used, however when you use cage, the projection is averaged so there will be no gaps.
Be careful when giving contradicting advice, these things snowball when people start repeating the same incorrect information.
This depends on how your exporter handles smoothing groups. In my experience an extra smoothing group means extra verts. Which engine behaves like this?
The point is that it doesn't matter if an edge has broken smoothing, broken UVs or both - all 3 of those circumstances constitute one break in the tri-strip.
So it makes sense to try and sync UV breaks to smoothing breaks as EQ mentioned. That way you're keeping the amount of breaks to a minimum.
This is correct I see what the misunderstanding is.
I am assuming that a model would have a minimum of UV splits already. So using smoothing groups in order to fix the occasional normal error would always increase the number of verts.
for example: I would never split a cube into six UV islands therefore I would never make a cube into six smoothing groups.
Gradients on a normal map aren't a bad thing. The gradients you see on the normal map are what make those faces appear flat in-game.
So what's the bottom line on smoothing groups? On which angles should you use them?
After sprunghunt's post I separated every sharp angle with smoothing groups and then worked backwards removing them and checking the effect. I ended up using them in couple of places. I didn't quite answer those questions with the tests as I hoped for - it seems that sometimes they really improve the result and sometimes they don't, even differently in corners that look similar.
Anyway, end result seems ok-ish. But for next time I need a more intelligent work flow
I watched this video, before I started - the author himself says that the information is inaccurate and that the video needs to be redone, but he doesn't say what exactly is inaccurate. There are some smoothing groups tests in the beginning of the video.
This depends highly on how your workflow is set up, if your baker and render are properly synced up, then you can get away with always using smoothing groups, however it really isn't the best idea in the world even if you can do it. Now, not very many people understand this, and even less have proper workflows to take advantage of this, so unless you know for certain that someone is using a rock-solid workflow, telling them to never use hard edges is simply bad advice. Even in an ideal workflow, hard edges can be helpful, take a look at this;
http://www.polycount.com/forum/showpost.php?p=1217175&postcount=457
Now, there is an argument to be made that it will require more uv seams to get better results, but this is generally a trivial problem in reality, we're talking about a very small amount of extra verts(comparing the verts you will already split by having uvs at all, to the slightly higher % of verts you will use to create a uv layout that is more "hard edge friendly"), and with that comes all sorts of benifits. I dont really follow the "less UV seams are always better!" mentality as it generally sacrifices quality and trades distortion for a slightly easier layout to paint, which if you're setting your seams up well, shouldn't really be an issue in the first place.
Like i said, the areas where you will want hard edges are generally the areas where you naturally will want UV seams, there may be a few exceptions to this on a complex mesh, but we're talking very minor issues here.
The cube example is a generally poor example, as yes, every face on a cube would work best with normals if it had a hard edge. This is just a fact, even if your baker and renderer are perfectly synced up, you'll likely still get errors on a 12 triangle cube, it will mipmap worse, it will lod worse, and it will be harder to derive any sort of "cavity" type detail from, as the severe gradiations will cause artifacts in the output.
Now, a more realistic way to look at the cube example is actually a 4 sided cylinder, where the top and bottom caps are separated in the uv layout. This is going to be much more forgiving technically, rendering wise it will work out better, and you will be able to pack your uvs much better.
What do you lose? You miss out on 2 arbitrarily attached edges making your texture slightttttly harder to texture, but you've still got plenty of seams that you need to deal with, so if your workflow doesn't involve a good way to work with seams, you're shit out of luck no matter what. an extra attached edge or two will not make your life any noticeable amount easier, you still need to be very away of where your seams are and how to deal with them.
Lets look at the pros of both of these methods.
A
1. More flexible UV layout, takes less space to pack, can pack caps in random odd-shaped corners of uv layouts to improve overall uv usage.
2. Common sense approach means hard edges can exist where they naturally should.
3. Caps of cylinder can be aligned correctly on uvs, not rotated 22.5 degrees, which will offer worse detail representation in some situations.
4. Will work with either method, 1 SG or hard edges on uv borders without any issues.(the box still will have errors in standard max, but will work well with almost any other app, and will work perfectly with 3ps quality mode).
5. Less gradiations offer a wide number of advantages already mentioned
B
1. Two arbitrary edges are connected on each layout
2. .... thats it, no other advantages
I agree that two extra verts aren't an issue. 24 verts VS 6 is. That's why I used a cube as an example.