Baking lighting-Normal mapped object (maya 6)

CheeseOnToast

I'm nearing completion of the modelling/normal mapping bit of my first full character done with normal maps. Before I start on the colour maps I'd really like to be able to bake some (subtle) lighting to a texture, including the info from the normal map. This would at least give me a starting point and some reference for the extra detail. Any ideas how to do this in Maya 6? The usual "convert to file texture" doesn't seem to work, I guess because it's software rendered and doesn't take into account the normal map. Any help would be much appreciated, as this is pretty urgent.

MoP

Can't you used the high poly mesh for generating lighting info and baking that down to the lowpoly mesh in the same way you generated the normal maps? Or did you use a 3rd-party program for generating the normals?

CheeseOnToast

Cheers for the ultra-quick reply. Yeah, I used melody to make the normal map. Maybe there's something in Maya's "transfer surface information" tool that can do it, but I'm pretty much a total noob to this, and I don't know how yet....

MoP

See, if this were Max it'd be a cinch... :P
I'm afraid I don't know enough about rendering in Maya to help you out at this point in time. I expect if Jeffro or Whargoul are around, they could shed some light on the matter.

scanning_factor

mental ray bakes normal maps. hook your normal map up to a shading node then use Lighting/Shading > Batch Bake (mental ray).

Jeff Parrott

Yeah I would create the normal map first then apply it as a color map and then convert to file texture. Or just convert to file tex on a light grey shader and overlay that on the normal map is PS. I don't think there's a way to do what you want in 1 step in Maya though. But Whargoul might know some uberhidden trick.

scanning_factor

actually another trick is to run your normal map through an rgb>luminance node and apply the output of that as a bump map which would work with convert to file texture. . .or you could just use the out alpha of the normal map's file node as a bump map too, not sure if the results would be that much different.

poopinmymouth

I thought melody had an occlusion option?

*edit: forgot I had it installed, nevermind, it doesn't*

CheeseOnToast

Thanks for all the tips. Scanning_factor...can you be more specific about the settings I need to use mental ray for baking? I got something out of it, but it has very harsh, angular shadows and has none of the lights' colours. On the side in shadow it seems to be using vertex normals, causing a very hard edged look.

**edit** Tried using the normal map as a bump map, as suggested, the results were very rough looking. I'm a rendering idiot, and really need some step-by-step instruction here. The batch baking looks like it *will* work, I just can't get the results I'm looking for.

Whargoul

Try looking for the shading network versions of normal mapping, you should be able to bake those out.

The main problem is that the normal maps are in tangent space, you need need the shader to convert that to either world space or camera space at render time (for Maya or mental ray). There's also a node someone wrote that can do it, but I've never tried it myself.

You can get the shaders here:

Shader for mental ray

It also shows you how to modify the shader (remove 2 nodes) to get it to work in the Maya renderer. From there on, it should bake out like any other material.

CheeseOnToast

That worked a treat, thanks Wargoul. One headache gone

Whargoul

Yeah, I checked it out this morning, and you have to use the Mental Ray one and back with MR, I couldn't get the Maya one to work (it bakes out each channel separately - ie the bump gets a baked bump map, the colour it's own, etc - instead of combining them).

But MR does it perfectly