Baking AO with Mental Ray ...... HELL!!

GIS3000

Hi there again!

So I've been able to bake AO using the Skylight method in 3DS max 2012.:)

However, the results are not as good as Mental ray. The skylight method would always leave a light rim in corners using intersecting geometry. MR does not.

I've been experimenting with MR and the big trouble I'm having with it right now is that it isn't very reliable.:(

Sometimes, it works just as expected, but refuses to work properly on other occasions.

Here's an example, I'm trying to bake AO from a high rez model to a low poly model. When I render, the preview window shows red spots (Ray miss check). Despite adjusting the low poly cage, it does not seems to change the situation.
Even the "Hide the low poly model's faces" method did not work.

Now here's the freaky part, scaling the low poly model actually helps! It's as if the baking process no longer listens to the cage but to the geometry itself!

And to add to the mystery, the baked normal map and AO maps, on occasion come out right even if red spots (Ray Miss Check) are seen in the preview window.

The funny thing is that switching backing to the skylight method gives no trouble at all.

The good news is that MR AO baking works pretty well if it's just baking a low poly model onto itself.:\

Anyone care to share his/her thoughts?
I've also heard of an alternative to the cage method, can someone tell me about it?

thanks!

Regards
GIS3000

Ace-Angel

Yeah, Mental Ray is known to be finicky like that, don't bake with Mental Ray, especially in Max, period.

Either bake in scanline or Xnormal for several reasons (at least for AO):

1) Scanline doesn't create surface banding, which happens with MR on low settings.
2) MR requires ridiculous settings = Time needed for bake is ten fold.
3) Padding gets bad streaking.

I think I never once used MR for baking AO. I know I used xNormal, ZBrush and even VRay, but never once MR.

GIS3000

Thanks for the quick reply!

Oh wow, so it does stink that bad.......

Oh well. Thanks for the heads up! How's Vray anyway? And do you have any pointers for creating the best AO bakes for scanline? You know....... the "intersecting geometry" issue?
Lauren's tutorial talked about it, ( www.laurenscorijn.com/articles/ambient-occlusion-baking ) so I'm kinda still looking.

Anyone else?

thanks again!

cptSwing

Ace-Angel wrote: »

2) MR requires ridiculous settings = Time needed for bake is ten fold.

ohmygod yes.

for low-2-low AO i've started using lighttraces since xNormal keeps baking triangle edges into my occlusion maps.

EDIT: for baking High-2-Low with floaters in xNormal, check "discard backface" or similar, within the specific maps' settings.

jaffajuice

Mental ray does suck, XNormal is better but I never got used to its workflow and Gui and setting up complicated scenes became a ball ache. Vray is really nice, but has similar cage issues in my experience when using High to Low projections.

So ... I do occasionally use mental ray for the AO if I'm in a hurry. The trick is to bake the normal maps using scanline first. Apply a standard material with the bump normal applied to your object. Then bake out the AO using Mental ray without the High Res object! No clipping, No cage errors, No Worries and all the normal detail.

Keep all the settings as default with the exception of those within the Render To Texture window. Samples to 100+ and Spread to 1.0; These are very simple settings that will get you something like an AO Map.

You can play with Spread, Max Dist and Falloff further to get better results. Best to test these first with a lower Res output and lower samples till you get something you like. Keep this as a broad AO Layer.

For finer detailed AO I would take the Displacement Map or Normal Map into something like Crazy Bump or again XNormal. Photoshops NDo isn't too bad either.

Blend the two or three in Photoshop.

Before I set off in a mad baking session I always ask myself is it quicker to paint by hand? Sometimes its just is!

GIS3000

That's just what I'm discovering too!

Thanks!

spectre1130

If you bake an ao map from the high res, do you need to have it unwrapped as well??? Or do you just explode both of the models and reverse the render to texture setting to bake from high to low?? I'm just starting to play with this baking stuff and I'm really spent trying to figure it out.

GIS3000

Hi Spectre,

No, you do not need to unwrap the UVs of the high res. Just the low res model.

hope this helps.