3ds Max Alpha overlapping

HAWK12HT

Hey hi all, Made a tree from hi poly to use its alpha in low poly version. All works great however problem arises as soon as I bake the low poly tree (RTT and Flatiron) using mental ray. Alpha and diffuse of leaves below is projected on top leaves creating chaos even though all low poly leaves have unique unwrap (no overlaps) using flatiron. 
So far it only happens in Mental ray and not in scanline render. 
Since this is not even happening in Vray my guess is Mental Ray is messing about or I have missed a setting somewhere. So far have checked all advance option in arch design material and render settings and no luck.

HAWK12HT

bump

HAWK12HT

bump bump 

codyaq2

Why are you even using mental ray?

HAWK12HT

Same is with Vray, this does not happen with scanline default render though. 

Eric Chadwick

You're using RTT. Why use MR with it? You just want to bake things into textures, right?

Also, transparency is not going to work well in RTT.

HAWK12HT

Doing arch viz work for gear vr that is why using MR. Tried Vray it still has same outcome, Corona dose not work with RTT. What did work was RGB light setup for making normal maps, its only good for Ivy and planer grass / foliage etc. There must be some way to get it right in the first place. 

Solution that I found but its a hack really:
1. Bake once with alpha applied in diffuse slot (black n white), make sure render element is also Diffuse and not Complete map. No cutout or transparency enabled in material. 
2. Bake complete map render element with alpha cutout, the usual way of making foliage. This will give same result as above but use the diffuse from 1 as alpha in Photoshop. 

However I still need to know how to fix this properly.  

Eric Chadwick

You can't bake transparency properly with RTT. If you're just making texture flats, the best way is to set up a camera and render the geometry into a texture, using the regular rendering path, not RTT. I've done this quite a bit, for baking highres models into textures. 

Solidify C filter for Photoshop works the best for padding the edges outside the alpha channel.

HAWK12HT

Yea guess when it comes to 3d foliage with baked light on it, its a killer of dreams lol. From 3d to 2d yea your method works. Thank you for help mate. Cheers