How do I fix mirrored UV seams in a normal map?
polycounter lvl 14
Sorry, I'm looking through the sticky and it is giving me a headache. Too much info.
I'm very frustrated. This is setting me back and interrupting my workflow...I just want to figure out a solution and move on. I've browsed and googled. I've come across warby's photoshop trick but I really don't want to monkey around in photoshop that much on just one UV shell, let alone many different UV shells.
I was told xNormal handled UV seams better, but I've had no luck.
Some people talk about a "double baked method" and "move off your UV's". I guess I don't understand this at all and I can't picture in my head what people are talking about? I was wondering if someone could help explain better?
I sure wish someone would be awesome enough to make a video tutorial for the community showing the exact procedure and steps needed to be taken to solve this problem.
I'm very frustrated. This is setting me back and interrupting my workflow...I just want to figure out a solution and move on. I've browsed and googled. I've come across warby's photoshop trick but I really don't want to monkey around in photoshop that much on just one UV shell, let alone many different UV shells.
I was told xNormal handled UV seams better, but I've had no luck.
Some people talk about a "double baked method" and "move off your UV's". I guess I don't understand this at all and I can't picture in my head what people are talking about? I was wondering if someone could help explain better?
I sure wish someone would be awesome enough to make a video tutorial for the community showing the exact procedure and steps needed to be taken to solve this problem.
Replies
http://wiki.polycount.net/Normal_Map#UVCoordinates
http://forbrook.com/allparts.jpg
Moving from left to right...my high poly and half of my low poly cage. Modifier shows my stack before I export to xNormal. The result is my low poly with that annoying "hill-to-valley" seam right down the middle. Then my uv layout of this particular shell.
Then render out your normal map again.
Also I'm wondering why you have an Edit Mesh modifier on top of an Editable Poly object? It seems fairly unnecessary, in fact it would be better not to have it at all. What's it for?
Edit: This might be a 3dsmax display bug. What does the mesh and normal map look like in the XNormal viewer? I am guessing it might look correct. So you may not need to edit your mesh or normals at all, just try a different viewport shader.
Also, I'm editing your post to make the giant image into a link, it's making the "edit reply" field really hard to edit
I notice that your cage is just on one side of the object. That means that if you imagine another cage symetrical to the one you already have, you'll 'see' rays being shot in two different directions where the cages 'meet' - hence, a raycast seam is bound to appear at the mirror line.
I believe some engines can conpensate this in realtime, but it's obviously better to have things right from the start - simply duplicate your lowpoly half to recreate the whole shape, move the duplicate's UVs out of the 0-1 range, merge the to halves together, weld the middle verts and bake.
Once you have the bake you can either move the second halfs UVs back in range, or delete this half and recreate it from the first one.
You can also edit the normalmap by hand, but this works better in the 'bake both halves' scenario anyways.
Good luck, keep us updated
xNormal gives me an error when trying to export to its' .SBM format saying that it wants triangles, not quads, hence the reason for snuggling an "Edit Mesh" into there.
Anywho to answer your question, in the xNormal viewport it still shows that seam.
Your suggestion about applying a different smoothing group confuses me a bit. Won't that create a bunch of hard edged angles/quads in my normal map?
I think what you are describing is the "double-half bake solution" I was refering to in my first post. I feel mentally handicapped here because everytime I read that suggestion I just can't visualize what people are trying to say. I want to give this solution a try but I'm lost.
Moving out of 1-1 range? Can you mark up on my UV layout where the 2nd half should go?
I apologize...this should probably be a simple technical term any 3D artist should know but I am just not familiar with it.
http://wiki.polycount.net/Normal_Map#ShadersAndSeams
If it doesn't, then rebake using some of the suggestions here.
I assume you didn't generate the SBM files using the buggy 3dsmax2obj OBJ exporter that comes with 3dsmax... because that could bug a lot the vertex normals...
I suspect the problem is that 3dsmax tangent basis computation is sightly different than the xNormal default tangent basis calculator one... which could be probably solved using the D3DX tangent basis calculator instead... but try the shaders Eric mentioned before.
Yep, that's normal. The SBM exporter requires the projection modifier to be applied over a triangulated mesh. However, you should be able to export a quad/poly mesh if the "Export cage" is not checked ( but you'll loose the cage's data obviously ).
Thank you very much now I understand.
I tried generating the normal map quickly in 3ds max and while still not perfect, it looks A LOT better now moving that 2nd half out of that 0-1 space. I'm going to give xNormal a try now because it seems to generate a darker shade of blue along the edges in the tangent space normal map.
StandardFX.fx sure sucks. It still shows the crappy seam. But the DirectX material looks much better with the 'hardware map' enabled in viewport when combined with this new "double bake method" I finally understand.
Ben Cloward's shaders don't work well for me...I want specular so when I grab that one, I'm always seeing just black. I'd also like to see him add an option for ambient occlusion.
In a perfect world I would love if someone rewrote the StandardFX.fx (because I like its' setup and it has ambient occlusion) and apply Ben Cloward's code to it if it really helps with seams well. Then give people the option to flip the red/green channels like the DirectX material bump slot > Normal Bump options.
Also StandardFX.fx only displays a very low res map for some reason. The DirectX material matches your map res.
Combine all those elements and we'd have the perfect .fx file to use in my mind.
Are you refering to the global illumination shader? I can't get that to work. Just appears black. Maybe it's not meant to work with max2008?
http://forbrook.com/doubleHalfBake.jpg
StandardFX.fx still looks like ass. Everything else looks pretty good though. You get far enough away and now I can see the edge but maybe I should just take my "padding" up higher.
I really like the Ambient Occlusion in StandardFX.fx, I wish I could pull off the same effect with Ben Cloward. I can't find a Cloward AO shader, only GI?
You've got the right shader on the end of your image there.
Ambient Texture slot is for the "diffusely convolved" cubemap. Easy to make with this tool:
http://ati.amd.com/developer/cubemapgen/index.html
Probably needs to be in DDS cubemap format. I can make a quick one for you if you want.
Lightmap Texture slot is for your AO bitmap.
Here's a really old example of a diffusely convolved cubemap in action. Bottom left is lighting alone, bottom right is cubemap alone, top left is both together, top right is the diffusely convolved cubemap in the OpenGL-cubecross layout format. DDS uses a different layout, but end result is the same.
I wish I could write and understand these .fx files. StandardFX is perfect except it is too low res and the mirror seam in the normal map is visible.
Seriously, don't mess with the fx shaders that ship with Max. They don't use the tangent basis. Why, I don't know. Use Ben's or Joel's.
You could also try mixing the ao into your normalmap, then using that map in a good normalmap/specular shader. If the shader doesn't re-normalize the map, then the ao that you add to the map will essentially shorten the normals in the creases/cracks, which then causes the lighting contribution to be reduced there. If going that route, try multiplying it only on the Blue channel, not Red or Green.
I tell you this because if you generated the normal map using xNormal ( which uses an OpenGL coordinate system ) but you wanna preview it in 3dsmax ( which uses Z up ) probably need to generate the normal map in xNormal using a swizzling of "X+Y-Z+".
Also try to change the default tangent basis calculator in xNormal from the "Default tangent basis calculator" to the D3DX one ( which is the one that 3dsmax uses ).
At any rate, thanks guys, I think I've got mostly everything figured out now. xNormal combined with the "double-half bake" method seems to work the best.
So I guess if the rule of thumb for padding = 4 on a 1024x1024 image, then the padding should = 16 for 4096x4096?
The Cloward GI shader I'm still a bit stumped on. Cloward's NormalMap_Specular seems to be working the best for right now. I just dumped the AO map into the diffuse slot and I'm getting the result I want for the most part. I guess I'll just have to combine the AO with my main color dirt map in photoshop when the time comes and just deal with that for now.
Like I've said before, StandardFX really has the options I want...the normal map code just sucks for that seam and the resolution seems somewhat shoddy. I tried a new StandardFX.fx that came with max2009 but it still looks the same. Great options, poor code. If StandardFX had Cloward's touch it would be perfect.
I suppose I should just use xNormal's viewport but I'm still getting this weird result in the 3D viewport over there. It makes it look like the welding is breaking apart when I know everything is welded in max. Maybe I should try something other than the .SBM format?
Lumonix ShaderFX is on a whole other level of confusing...I hate nodes and linking. Reminds me why I hate texturing in Maya and why I stuck with Max. ;-) I'll try and mess around with it when I get time. I've tried e-mailing Ben a couple days ago regarding some of his .fx, I'm sure he is fairly busy. Joel seems to only work with skin/hair shaders but I suppose I could drop him a line too.
If anyone knows of any other resources/talk on the web where a person could grab some .fx shaders I'd appreciate it.
http://developer.download.nvidia.com/shaderlibrary/webpages/shader_library.html
The "generic image based lighting" shader is pretty cool.
http://www.luxinia.de/index.php/ArtTools/3dsmaxFX
ATI also has their free shader editor, probably has examples.
http://ati.amd.com/developer/rendermonkey/index.html
As for the padding, that should be fine. Try resizing your bitmap (with padding) down to 256x256, and see what happens on the model. Also down to 64x64 or lower. If you see seams, increase the padding on the high-res bitmap. Also always make sure to fill empty areas with an average color (never just black). Whenever the bitmap pixels on the model (called texels, short for texture-pixels) are smaller than the screen pixels, the bitmap gets resized down by the engine.