Stacked UV Shells and Normal Maps

mdeforge

Looks like I've learned the all important lesson today that stacking UV Shells when using a normal map doesn't exactly work. That begs the question though if there's anything I can do about it.

Flipping the normals isn't an acceptable solution in this case because I'm using a mental ray fast sss shader for sub surface scattering. It needs the normals facing the right direction. Not only that, but there's padding near the top of the dressing window (medical dressing, if you can't tell), that will be shown if the normals face the opposite way. Plus, flipping the normals doesn't sound like a good solution in any case.

Short of putting the UV Shell in a unique space, thus reducing my texel size, is there anything else I can do about it? Some magical check box nested deeply inside one of Maya's menu's perhaps? Those are my favorite. Anyways, sage wisdom is welcome here.

See attached for example.

cman2k

You can offset the UVs out of the 0 to 1 UV space. I'm not certain how that works in Maya but it should be possible.

This will allow one set of polys to get the same information, but not interfere with the normal-map baking. The problem with stacking polygons is, when you bake a normal map, it's trying to sample normals from two different places on your model. One side is going to end up being wrong.

Mark Dygert

In max you can either push all the pieces back behind one of the pieces (switch editor to UW instead of UV). That way only one renders on top of all the others. Or you can offset them like cman2k said.

EarthQuake

Mark Dygert wrote: »

In max you can either push all the pieces back behind one of the pieces (switch editor to UW instead of UV). That way only one renders on top of all the others. Or you can offset them like cman2k said.

The offset by U/V method carries a couple other benefits as well:
Works in any app
Works exported to Xnormal etc(haven't actually tested this, so may work fine from max exported with modified W)
Easy to see that you've already offset it

Of course, the big draw back is if you need to edit your layout, you've gotta move it back, edit, and then offset again.

mdeforge

Hey guys, thanks for the responses. :thumbup:

Am I not offsetting this right? I'm still having the same issue. See attached again.

I'm not seeing anything about UVW in Maya's texture editor. Unless I'm blind, it looks like it's just restricted to U and V.

This is the first time I've had to offset but I could have sworn it was going to work. Maybe MR Fast SSS is contributing to the issue?

Warheart

I think this is being caused by the weird Maya tangent space (based on UV winding order). It has been discussed recently in a thread I think you were involved with, Stradigos.

TBH on a single mesh which shares a normal map on two sides I don't know of a "good" solution. I can suggest a couple of dirty hacks if you like though :P.

Flipping normals on one side would be my first suggestion but as you already mentioned that's not going to work with the SSS shader.

Dirty hack #2 would be to make two versions of the shader where the only difference is that one of the channels (probably green) is flipped in the normal map. (you could do this manually in photoshop or just do it in the shader by passing one channel through a "reverse" node).

The "magic hidden checkboxes" related to this problem are in the attributes for the shape node of your object under the "tangent space" rollout. First thing to check should be that UV winding order is set to "detect". If none of those options help then I suspect you'll have to opt for one of the above dirty hacks.

If anyone else has a "good" solution to this problem then I'd also love to hear it

mdeforge

Have you read this? http://wiki.polycount.com/NormalMap#Mirroring

Yeah, I've read it and it's very helpful. However, I'm not baking my UV's from inside Maya. I'm just taking a UV Snapshot (overlapped shells and all) into Photoshop and Crazy Bump and importing the diffuse and normal maps from their respective apps. I thought it was pretty straight forward, but apparently not since the normal map is still struggling.

Am I right in my understanding that when my mesh gets rendered, the mirroring unfolds like a book? Is this what I'm seeing here?

I've considered doing a bake from inside Maya, but I'd have to find a way to create a shit ton of holes in the model. I could use an array and all that jazz, but I just scan in the dressing, make the holes pop via some PS trickery, and then let CB do the work.

@Warheart: Lol, really? I don't recall.

At any rate, I've just decided to suck it up and lose a bit of texel resolution. I doubt the camera will be getting close enough in the animation to really show a difference anyway.

Thanks for all the sage wisdom. I'll check in later to see if there's been anymore discussion. I'm still interested.

m4dcow

I have had this problem with mental ray normal mirroring myself. Under render settings for mental ray, on the options tab and performance dropdown, check the Maya derivatives box.

While I didn't use this with the same shader, try out my soloution and see if it helps.

Rooster128

If you're using mental ray inside Maya, with mirrored shells like this, sometimes going into the Render Settings, into MR features, and then turning on "Maya Derivatives" fixes this. Doing this with the offset shells while transferring/using CrazyBump should help.

If this doesn't work, and you have two shells on top of one another in the UV space for two geometry patches that are side by side (like in your case), sometimes it helps to paint in an adjustment layer. Make a new layer inside photoshop, paint a few lines on the shell border where the two geometry shells meet using 128-128-255 as your color, and then blur/smudge it out to make it seamlessly fade into the normal map. It's kind of tricky to do, but it's the only way I know to make this work.

Overlapping shells can save a lot of UV space and are generally good, but they are not good for when the two shells are actually touching geometry. Try to make sure the overlapping bits are 'islands' seperate from one another when you do it. I hope this helps.

mdeforge

Thanks for the tips. It still didn't work, but it's another lead to pursue. I'll have to research exactly what Maya Derivatives are and see if there's a solution waiting for me there.

Warheart

Using "maya derivatives" means that mental ray will derive most of its settings from maya. As far as I know this is the default behaviour. Normally you'd only uncheck this if you wanted to force mental ray to use its own default to override the normal maya settings.

I don't think that painting the normal map out toward the seam is good practice in this case because this is not simply a seam (as far as I can see). The problem is that the handedness is different on each side because the normals don't match the UV winding order, making the normal map flipped inside out on one side of the model.

Details can be found about this problem here:
http://download.autodesk.com/us/maya/2011help/files/Polygons_nodes_Tangent_Space.htm

m4dcow

@Warheart- Maya derivatives are not enabled by default.

@Stradigos- You said you tried checking Maya derivatives and it didn't work, but I went ahead and did a mockup of what my problem was and how checking maya derivatives fixed it. Also make sure that your actual geometry normals aren't flipped on one side.