Hi-poly AO bake in Maya.

ng.aniki

Hi, I am trying to figure out and understant a full hipoly bake pipeline in Maya (Using Maya 2009 and the Transfer Map tool), I am okay with the normal map bake, the heightmap, but for the Ambiant Occlusion I can't get any good results.

I have something that looks like working, but the bake keeps reaching other faces in their back. I tried 10000 times with different settings, but i am not getting anywhere.

The only setting really making a difference is the Occlusion Max Distance, it lowers the issue but render less and less occlusion if I lower it.

Some pictures:
Here is a part of the model:


In here how is baked the front surface:


Here are the settings

I tried with envelope mode, "both", with different search envelope, with different Search Method and Max search depth, with all the possibles settings in the MentalRay Common Output.
If I lower the Occlusion Max Distance, as intended the faces in the back are not baked anymore, but the whole map is getting completely white.

I don't know what else to do.

MikeF

as far as i know the AO transfer maps is completely broken in maya, using 2008 for years and i was never able to get it to work properly. I think there's quite a few threads about it.

Best bet is to go with xnormal

ng.aniki

MikeF wrote: »

as far as i know the AO transfer maps is completely broken in maya, using 2008 for years and i was never able to get it to work properly. I think there's quite a few threads about it.

Best bet is to go with xnormal

By any chances, did anyone try it in Maya 2012 ? Maybe it has been updated ?

kodde

Just thinking out loud here.

If the AO transfer map feature indeed is broken, could you not use the Polygon Toolset > Color > Batch Bake (bake override and set to Occlusion) to bake AO on your higpoly. Then map the newly created AO texture on a shaders color channel and use the Diffuse transfer map feature to transfer this data from the HP to the LP? You might need to fiddle with the material settings to get it full bright.

Otherwise, does XNormal do a good job with you are looking for? If I'm not mistaking, I think it has that functionality.

A third option could be to use Batch Bake and bake your AO results to the HPs vertices, then export that as a SBM (XNormal native format, can store Vertex Colors). Then finally use the Vertex Colors transfer functionality in XNormal to get the data from the HP to the LP.

throttlekitty

I really can't tell what's going on with the images, or where things are going wrong for you.

AO in Maya works, it's just not always so good, I've used it many times with success. But I'd also recommend using xNormal as well

MikeF

ThrottleKitty, could you post what settings you use generally?

i've tried countless times and wasted many hours of rendering only to get solid white or grainy distorted bakes. If it is possible i'd like to know more, personnaly i'm not a huge fan of xnormal, not because it isnt a good program, i just like to use as few tools as possible.

throttlekitty

Sure, it's been a while, and I mostly used the Mental Ray/Batch Bake.
This tutorial is pretty old, but I remembered it filled in holes in what I knew at the time. It's less straightforward than using Transfer Maps, and I honestly don't remember if one is better over the other.

For graininess, it's mostly a matter of more rays/samples. I've never had one come out 100% perfect in that aspect, there's usually some postwork in Photoshop. Also, use some Antialiasing in the mental ray render options.

For distorted bakes, that's more model specific; uv layout, overlapping geometry, etc.

tehrobster2

AO transfer maps have been working fine with me, What are you setting you Occlusion rays at ??

Fingus

One thing you can try is to use Batch Bake and bake the AO to vertex color, and then use Paint Vertex Color Tool to export the vertex colors as a texture map. Baking to vertex color is super fast even when you use 256 rays, but you may need to add some geometry to your model. You can set the Paint Vertex Color Tool to Smooth and then use Flood to smooth out the gradients if you get harsh results. To export the vertex color as a texture map go into the tool setting of the Paint Vertex Color Tool and find the tab called attribute maps and find the export dialogue.

I can make a video demo if there is any confusion.

kodde

Fingus> Cool. Didn't know you could export Vertex Colors to a Map in Maya with the Paint Vertex Color Tool. Although the map will be based on the same object which has the vertex colors right? Wouldn't this first require some sort of transfer of the vertex colors from the high -> low? Or decent UV set on the High and then Warp Image to the Low?

Since we are talking about getting AO details from the highpoly to the lowpoly's UVs.

Fingus

Hmm, I didn't think of that. I've only used the technique in a lowpoly workflow.

In Transfer Maps you can bake the highpoly diffuse color to a lowpoly, but I'm not sure if it works with vertex color. If it doesn't then a potential workaround is that you just do a quick automatic map on the highpoly and do the whole vertex color to texture thing and then bake the diffuse with the AO texture applied to the highpoly. It's a bit of a hurdle so it might not be worth it though. A real AO bake will make more accurate detail. The advantage of the vertex method is that it's lightning fast, and if your engine can import vertex color it could save you a lot of resources.

Fingus

I was testing if you could bake vertex color with both the Diffuse and Shaded option in Transfer Map, turns out it doesnt. Boo.

kodde

Fingus you can use XNormal as I suggested earlier in this thread. Although at that point you are venturing outside of "doing it all in Maya".

EarthQuake

Whenever I need to do maya bakes, I always use Xnormal for AO. Its really sad that Maya doesn't have a modern AO workflow, but what can you do. Any in-maya workflow is just a massive time sink, might as well just use XN.

Fingus

Oh I agree, xNormal is by far preferable for AO. Although in Maya you can have more control if you use a surface shader with mia_occlusion. Still slow as hell compared to xNormal though.

I was just mentioning the vertex color method as it can give you pretty good and cheap results, and not a lot of people know of it.