Maya alpha channel mask for compositing?

malcolm

So I tick the box, select production quality from the drop down and render hoping to get a mask out so I can composite a skybox in later. The mask is not accurate and there is a 1-2 pixel outline around everything when I use the alpha channel as a mask. What I'm I doing wrong I thought this would work.

bugo

Are you using MentalRay or Maya Software?
In Mentalray there's a premultiply function that if you turn it on will help you out selecting the mask in photoshop without getting any inaccurate background colors from your render. Also you would like to mess around with the material Matte Opacity? Or even creating a background material to hold all the background so you can only grab what you want above that material.

bejkon

Think it's its using the alpha fot aa purposes.

malcolm

The problem is not setting up the mask the problem is the mask is not the same as the render. It's as if the antialiasing is off by 1 pixel. I'm using mental ray maya 2008.

malcolm

I don't have any alpha on the models, basically the problem occurs with a cube on a black background. There must be a way to get this working or you wouldn't be able to composite anything at all in maya?

malcolm

I can't use a surface shader because the skybox would block the procedural environment sky I'm using for the final gather/global illumination.

commander_keen

You are aware that you have to un-premultiply anything with AA when compositing?

malcolm

commander_keen, thanks. Yeah a guy at work also had the answer I had to turn off premultiple and turn on alpha clip in the frame buffer options.

commander_keen

Yeah generally this is done in the compositing app though. You can "undo" the blending with the background that happens in most renders. I guess you could call it Post Divide. I think the math would be (ImageColor.rgb /= ImageColor.a) assuming the background color is black. Most compositing software allows you to import images as "premultiplyed" which will do this automatically.