Maya 2012 Normal Map Baking Problem

Brygelsmack

Hey folks. I'm currently at school trying to figure this one out...

See, I've been assigned to model a locker. I'm currently working on the door. I've made a high poly model and a low poly with UVs. I baked the normal map yesterday and at first glance it looked alright, but when assigning the diffuse the 3D effect more or less disappeared, so I decided to bake out an AO map as well to fake shadows. This is where I noticed the problem. The normal map and the AO don’t match, so the textures look skewed when used together. The maps were baked at the same time so I can't imagine the two textures being different really. The models and low poly UVs remained the same throughout the process. I think it's an error on the normal map.

It's like the normal map is baked from above, it's hard to explain. I’ll try to explain using pictures.

Okay, so this is the high poly mesh in the side view. This is the point of view I thought I’d get the normal map… but no. The AO is baked in this point of view, just straight in front of the mesh, but the normal map is not.



Here it is from another angle. The part extruded inwards is the handle of the locker door. It’ll be my point of reference. This is the angle I feel I'm getting the normal map, like... from above.



This is the AO. As you can see, looks good.



And this is the normal map. Like I said, it looks like it’s baked from above, not in front of the model.



This is an overlay in PS, showing how the two textures combined looks skewed.



This is the result in Maya. As you can see they don’t match at all, it looks really off.



These are my baking settings.





I hope this explains my issue well enough. I’m fairly new to modeling but I think I’m doing it right. I’m on a deadline so I really need this solved ASAP! Thanks a lot in advance guys!

Scruples

Ambient is being rendered in Mental Ray and Normal is being rendered with Native, they are using different methods. Check your 'Mental Ray Common Output' tab to make sure the settings Match 'Maya Common Output'.

Turn Backface culling on in your Persp viewport to make sure the Normals are all facing the same directions.

Oh, and in the Advanced options Tab there is an option to Match Using> I think usually you get a better result with Surface Normals but you can try Geometry normals too if it was already set to Surface.

Brygelsmack

Thank you so much for the answer! I think I got the settings the way you explained, and I think that's how I first did it, with the exception of having Match using set to Surface Normals. Right now that is my biggest hope.

Here's the settings I'll try now. Is it good?

Jeff Parrott

That top edge is hidden. Grab the face that is inset and scale it down slightly so it is sloped. That will allow the transfer maps plugin to get that edge when it bakes down. It should still look fine when the normal map is applied. It's actually a pretty common normal map issue. Not just with Maya

nathdevlin

Hi,

Along with the other good advice another factor that may cause a normal map "distorting" would be the size of the search envelope. 5.0 is a large number. It only needs to be big enough to cover the high poly mesh.

Have you accidentally moved the envelope ?

Brygelsmack

I know 5 is a fairly high number but it's what covers both models. I might be able to go down to 4.5. I'm running a bake as we speak, we'll see how that turns out. Hopefully won't have to change the HP because it's a lot of polys... Thanks for the advice guys, much appreciated!

Brygelsmack

The normal map turned out more or less like I wanted, but the maps still don't match. See pics:








What could possibly be wrong?

Jeff Parrott

Have you tried doing an ambient bake with the default render and not Mental Ray?

Brygelsmack

Nope, how do I do that? Is that what happens when I untick 'Use mental ray common settings"? It defaults to the Maya rendering?

Jeff Parrott

Yep.

bugo

you probably moved the envelope, I usually don't even use the envelope and just use the distance on the bottom tab called "max search depth" and it works.

Brygelsmack

jeffro wrote: »

Have you tried doing an ambient bake with the default render and not Mental Ray?

I tried that now, it didn't solve my problem. I don't understand why the normal map offsets so much.

bugo wrote: »

you probably moved the envelope, I usually don't even use the envelope and just use the distance on the bottom tab called "max search depth" and it works.

Like I said, I rendered out the maps in the same bake, so that's impossible.

That other thing sounds interesting though. What values do you usually run with?

Scruples

Each grid is 5 units (default), so looking at your grids, a value of 1-2 to start with.

Bugo's solution will likely work, but if it doesn't and you are running out of time
you can always render the maps in Xnormal, http://www.xnormal.net/1.aspx