Alpha Channel on Ambient Occlusion

Raymond Arnold

I'm working in Maya, doing an AO pass on a single frame. The end product needs to be a layer in photoshop that is transparent except for the AO shadows, rather than black on white. (It's possible that the client I'm working for can do a multiply instead of a normal layer, but so far all the samples he's sent me have the shadows translucent on a separate layer and I figure I should keep it that way).

There's an option in the AO dialogue box in Maya for an Alpha channel, but it only seems to are about what geometry is visible rather than Ao for the shadows.

I feel like I know how to do this and that it's easy but I can't remember how.

renderhjs

I guess you could do that with scripting or do in Photoshop manually. Just create new black bitmap that uses as alpha channel the inverted AO render (white = key).

But honestly solid RGB layers are pretty common compositors, like reflection, light and or shadow maps they are usually used with blend modes in AE, PS or whatever tool.

Raymond Arnold

I'm doing work for an indie game company. His engine, as far as I can tell, uses a translucent layer for shadows whose opacity he adjusts as necessary and a solid layer for the other objects.

I asked him whether he could switch to multiply-layering, haven't heard back yet. But ultimately it's my job to match his workflow, not get him to redo his engine so that he can match mine.

renderhjs

or create a photoshop action and do it just once in Photoshop. After that just batch script stuff inside Photoshop (file > automate > batch processing or something like that once you have your script recorded).

MoP

Set the AO layer as Normal blending mode at 100% opacity in Photoshop, then go to the Channels palette and CTRL+click any of the channels to make that into a selection. Invert the selection.

Add this selection as a layer mask to a pure black solid colour layer (or dark blue, whatever colour you want your shadows).

You can either leave it like that now, or do "Apply Layer Mask" if you want to leave just the transparent/opaque pixels.

Personally I leave it as a layer mask since it's more flexible that way, but it depends what you need to do for that engine, I guess.

Raymond Arnold

Thanks!