Normal Map Artifacts!
polycounter lvl 14
Hopefully someone here can help me find a solution to my normal mapping problem. For my latest job I have been asked to create a normal mapped prop. So I build the High Poly mesh and the Low Poly mesh. Then I unwrap the low poly and set the entire low poly mesh to one smoothing group. I then render to texture and the resulting normal map on the low poly model has artifacts caused by the low polygon vert shading! In other words the normal map itself has vert shading in the RGB values of the normal map.
For the most part, the normal map generation is successful but in a few areas there are these fugly artifacts.
Another similar and equally mysterious artifact that appears whenever I do a normal map on a low poly mesh is the mysterious UV vert 0 artifact. Aside from the other artifacts that pop up from vert shading, this one is all its own and is 10x worst then the other artifacts. It looks as though the normal for this one vert is facing the opposite way and when I go and check which vert it is in the UVeditor window its vert connection 0.
There seems to be a million tutorials and threads on all art forums worldwide on normal maps and yet it seems like this is a common problem with very little discussion. Does anyone know why I've been plagued with this annoyingness?
I am using 3DS Max8
Replies
Anyways if this isn't annoying enough when I do a render, the shading artifacts become more apparent in other places and the ones previously seen in the view port are miraculously missing from the render.
The thing is, bringing it into a game engine is the only thing that has kept me from going insane. Once in an engine, the viewport artifacts present in max are not visible in the engine, though the ones in scnaline render are. So basically the max viewport artifacts are annoying to all hell and the different ones that show up in engine are equally as frustrating. I would like to avoid both if possible! Any solutions?
If the normalmap creation tool and the game engine use the same tangents, then WYSIWYG (what you see is what you get).
2nd issue, the low-poly is too low. The vertex normals are changing direction too much across the surface of your mesh. If you add a simple chamfer it should decrease these artifacts significantly. Though they won't disappear entirely unless the normalmap caster and the engine are in sync.
The UV index 0 is probably a direct result of this disparity... the tangents at either end of the UV do not match up across the seam, so the normals are being suddenly twisted there.
IIRC, xNormal's coder Jogshy released his tangent-gen src. You might try his tool instead, if you can convince your coder to implement Jogshy's methods.
http://boards.polycount.net/showflat.php?Cat=0&Number=105812&an=&page=&vc=1
2) Try to collapse + XForm the mesh before rendering the normal map/export mesh.
3) I think the problem is that 3dsmax averages the vertex normal when computing the tangent basis(aka it doesn't respect the smoothing groups). Try to tessellate/chamfer a few more the cube and avoid hard edges like Eric said. Also try to apply a "Edit normals" modifier and sure you break the normals so the hard edges will be conserved.
4) To see the tangent-space normal map(exported from 3dsmax) in the 3D engine you need to use the tangent basis that 3dsmax calculates(tell your programmer to see the IGameObject::GetTangent() method in the max's SDK). If uses a custom tangent-space calculation then you will have problems(because the programmer's TB will differ from the MAX one). The problem is that the max SDK don't allow you to import your own TB, only to export.
5) Use an object-space normal map and try. If you get good results with it then is a tangent-basis error.
6) Caution with the 3dsmax2obj exporter. It exports the vertex normals bad sometimes(specially for simple boxes like yours).
7) Try to see if is a model problem or a baking error. Use other program to render your normal map and compare.
Do you get the same results with other programs like xNormal, Mudbox or ZMapper?
Hope it helps
First off, due to the nature of my work, contacting the export coder directly will be pretty much impossible. I have been using UE3 for the most part to preview the work I've done and also Ben clowards's shaders for max.
Increasing the poly count is definitely something I will try to suggest to my Art Lead first thing Monday. I figured this was one of the problems. I am currently at our poly budget for this model though I will try to ask for a little more polies in troublesome areas.
Jogshy, actually I have your latest version of xnormal and been using it lately and following your thread on these forums. Its a very nice app. Last night I attempted running my model through but in the 3d viewer I still got some of the artifacts I received in max leading me to believe its most likely an issue with poly count. We are attempting to avoid using more than one smooth group for our the mesh's though they can be used sparingly. I know how to use smooth groups and what they do but it is a little difficult for me to determine the end result of the normal map when it requires a re bake every time I change them...a little annoying.
Concerning UV's, I heard that it is possible to have UV's facing the wrong direction when unwrapping? I have still yet to understand this fully. Say I make a cube, and unwrap it with the idea of having as little seams as possible. So In this case I choose unfold mapping, works nicely on a cube. Okay so I apply a checker number texture to see how the texture will allign and how it will be baked.
Ben Mathus in one of his tutorials says [ QUOTE ]
"keep in mind that UV chunks can be facing "forward" or "backwards". The way to tell, is to apply a texture with some text on it. If the text displays correctly, it's facing forwards. If the text displays in reverse, it's backwards."
[/ QUOTE ]
The problem with this is that when a box is unfolded, there will be places where the text will appear upside down, sideways or reverse in the max viewport because of how the Uv;s are layed out for best usage of space. So how is it possible to know what is right or apply his method of thinking? Clarification on this would be nice.
And for Xform and having the model collapsed, check and check.
Thanks again for the help so far.
I have heard UE3 doesn't have a problem with this, although apparently it does have a problem with mirrored UVs. This is where you UV your mesh, then mirror the mesh about a centerline, for example to make a bilaterally-symmetrical human. This also mirrors the UVs, which is great for reuse of UVs (the forward-facing UVs share the same UV space as the backward-facing UVs), but not so great for normalmapping in most engines.
One way to get around the mirror-seam problem is by mirroring the mesh, then moving the UV seam off to the side somewhere, where it's less obvious. You do this by selecting some UV faces on one side of the seam, detaching them, then stitching/welding them to the mirror-edge. Here's an example by b1ll, where the face mesh was mirrored, then the right side was stitched, to move the mirror-seam off to the left side of the face (the right cheek and ear are used by both sides):
http://img230.imageshack.us/img230/8671/miniskinningtut28hj.jpg
Smoothing groups are the same vertex cost as a bevel, both cause a single vert to be split into two verts. But bevels are better... they offer more control over the smoothing (wide or tight, you choose), and you can use a single smoothing group for the whole mesh so the normalmap raycasting has less artifacts. Smoothing groups tend to create normalmap artifacts, because the one vert is split into two new verts, which now cast in very different directions.
Whew, hope that makes sense.
You can't actually paint out the mirror seam, unless you make it totally flat in the normalmap. Even then you'll probably still get a seam. The difference in tangent/bitangent vectors across the seam cause a lighting discontinuity, because these uber-vertex-normals suddenly have to twist drastically.
I forgot to mention. If you're simply importing your Max-generated normalmap into xNormal, then you'll likely still see shading artifacts. But if you generate AND display in xNormal, I bet your artifacts will disappear.