How to bake an ambient occlusion map in Max8?

Jay Evans

Using the exact same process as creating a normal map, I am trying to bake an ambient occlusion map from a high res model onto a low res model. I seem to have run into some unexpected results.

-First thing I did was to create a normal map with a cage and all that. It worked great.
-Next changed the output to Ambient occlusion (MR), and applied the correct materials to my high res.
-**Problem is mental ray is including my low res mesh into the calculations. Basically where ever my low res mesh covers the high res it will render out just black. No matter what I try I can’t seem to be able to get it to disregard the lowres mesh in the baked texture.
-things I’ve tried.. Using the cage, not using the cage, making low res 100% transparent, and randomly clicking almost every option box I can find.
-Just having trouble understanding why it works perfectly for a normal map, but doesn’t for an AO map.

Anyone have any luck with this?

okkun

Which package are you using?

Asthane

I don't know about max, but most renderers should have an option to completely ignore a mesh for renders (Simply setting it to transparancy might not work since the AO shader doesn't take transparancy into account or such, that's a raytracing/caustics thing). I would hunt around the object properties, which is where it's found in Maya. If that doesn't help, well... good luck

Edit: Okkun, see the title of the thread

Kevin Albers

Mental Ray may not work correctly when rendering to texture in Max. I usually add a global Illumination skylight (using light tracer set to defaults), and bake out the "lighting" checkbox in render to texture.

KDR_11k

Can you bake the AO into a lightmap for the highpoly that you then use for rendering to the lowpoly?

MoP

Yeah, do what Kevin Albers said - if you're just doing an ambient occlusion or GI pass, just stick a skylight in set it's colour to white and apply a white material to the highpoly model, turn on advanced lighting and render... then you'll be left with a nice map you can use as an overlay in Photoshop.

Jay Evans

Thanks for the quick replies I'll try them out. My only reason for avoiding the skylight was the speed. But slow is better than nothing.:)

Rick Stirling

I've found a much easier way - bake your normal maps, apply the normals to the low poly, bake the low poly out using gray materials.

I prefer to use a light dome setup rather than a skylight myself.

Jay Evans

there are quite a few different dome light scripts. Do you have any suggestions for ones that are compatible with max 8?

Jay Evans

-Rick,
wow, I just tried the chugnut domelight setup, with my normal map on my low res. I'm pretty impressed by the results although the render takes forever. The biggest benefit of this method is that you are guaranteed that your lightmap will line up exactly with your normal map. Seems to do a bit of weird lighting sometimes as well, but it will serve the purpose. Thanks for the tip, I never would have thought of it. DomeLights seem so "old school" these days. Im waiting for an Ambient Occlusion shader that works with normal maps, that would be great cause AO is soooo fast, and looks great.

Looks like you have to use a domelight btw. I dont think the skylight shows the normal maps. Could be just the settings I had for it, but when using the skylight it just looked like I was baking out the low without normal maps.

Rick Stirling

Max 8 did have a problem with Skylights and normal maps (Mental Ray might have be fine). The problem I always got was nasty artefacts, but this could be fixed by using Global Supersampling.

The nice thing about this method is that it renders much faster than a skylight with bounces and supersampling, so you can play with the light strenths. I hav a saved lightrig that I mreger in, split into 3 sections - 60ish spots, one omni for controlling overall ambience, and a set of key lights for pulling out extra details..

You will find it useful to play with the spec settings of your gray material.

Also, when you do import your AO map into photoshop, experiemnt with multiply, overlay AND screen - you often need to have 2 or 3 of those layers rather than just one. I often use the levels to compress the AO contrast, so that the multiply level nails the shadows and the screen nails the highlights.

MoP

Rick: Just curious, why would you need the AO layer to punch out highlights? Surely the spec map will cover that? Or have you found it gives better final results if you subtly push the highlights in the diffuse too? I tend to just use a single Multiply layer.

headengine

Totally - it just pops a little more Mop... it's very much down to engine specifics how effective the spec map is in any case. If there isn't a subtle highlight in addition to a subtle shadow it can look somewhat flat.

I'm sure Rick would agree

MoP

Aha, sounds good, thanks for the info Mike

Rick Stirling

Yup - I've found you need that little bit in the diffuse, otherwise in soft light everything looks just that little bit too flat.

solar

If you are still wanting a way to bake a MR ambient occlusion map you can either scale down the low poly object or scale up the high poly object. The push modifier works well for this. Ive found that this works quite effectively and eliminates the intersecting low poly objest which causes the black spots. The MR bake is very nice too..nice and accurate compared with a standard skylight bake.

Jay Evans

thanks solar, it was just bugging me that it dosen't use the cage like it should. Scaling works ok, but I dont think it would line up perfectly with a normal map I created using the cage. The way it "should" work is click.. normal map done. change to MR, change material. click AO map done. :P

Delaney King

As far as I know you turn on MR, then apply a self illuminated material with ambient occlusion assigned to the diffuse.
You then render with shadows turned off and bobs your ambient uncle.

A quicker HACK that works well is to drop an omnilight into the scene at exactly 0,0,0 and set it to ambient only. Then in the projection settings drop in an ambient occlusion map. This is a fast way to add it into your scene without comping. Just thought I would share.

Another nice one is to use pits and peaks to create occlusion vertex lighting and a vertex map for render to texture. Nice.