How to set up alpha masks in Maya?

Cronnix

Maybe I am approaching this completely wrong, but I can't figure it out how to apply alpha masks. When I apply a black and white mask to the "transparency" channel of a material (or a targa with alpha channel), the texture does become "cut out" but it also becomes half transparent everywhere and slightly darker, resulting in the following:

As you can see the leaves are cut out, but they and the trunk also became transparent, so you can see branches behind it, despite mask being black and white.

m4dcow

I'm not sure about the textures getting darker, but if you go to Renderer-Viewport 2.0 (on the viewport menu bar). Change the Transparency algorithm from Object Sorting to Depth Peeling (or something else) it should help.

You might also want padding on edges of your color map, ie: have the color extend past the edges of your black and white alpha so you don't get that white bleeding through around the edges.

Cronnix

Depth peeling solved it! Thank you. 

Eric Chadwick

If this is for game export, you'll want to use a different material for the trunk/branches with no alpha, for faster rendering. Alpha is fill-rate expensive. We have some info about this on the wiki
http://wiki.polycount.com/wiki/TransparencyMap#Sorting_Problems