I'm using Blender btw. I made some leaves that I baked onto planes with alpha, then I put them all together to make a single branch and now I want to bake it into another plane, but I cant seem to save alpha that I initially made for leaves? Whenever I bake it just fills empty spaces with black color, no transparency.
Pretty much what
this post described
Replies
Opacity to Opacity rendering is one of the more computationally complex processes that rendering engines have to preform, so most render engines don't do opacity to opacity that well, if they even attempt it at all, most don't.
The few that do, usually only do it through high quality viewport/camera rendering and aren't set up to bake textures using cages, which is meant to be a quick fast process. So you end up stuck using the long rendering process but only on things like like planes (hair/fur rendered to a plane) and not complex models that require cages.
Personally I would tightly trim around your high poly leaf cards so they no longer use opacity, or create several leaf models that don't use transparency and scatter them on the surface of a branch using a particle system, then bake from the geometry to opacity. But even "geometry to opacity" will be difficult to do with some rendering engines.
Substance Designer is the only one that I know of that will bake Geo to Opacity, which is a pain but it works reliably. All of the rest (xNormal, 3dsmax, Maya and Blender) at best do a "missed ray" color when a ray hits the void and that can be used as a hacky opacity mask. But that's pretty crappy and doesn't give you much control over the edges of the transparency. It also has trouble rendering cards behind cards and will usually be a huge pain in the ass, more trouble than it's worth.
Geometry to Opacity, sort of works but it's dicey with most render engines.
Opacity to Opacity, is a unicorn process that you would think saves a bunch of time but isn't supported by the vast majority of baking options.
Not by baking, but by simply rendering a top/camera view of opacity mapped meshes, saving out to a format with alpha. TGA or TIFF, usually.
http://wiki.polycount.com/wiki/Texture_Baking#Transparency
But doing the same with normals(texture & geometry based), AO, translucency is a nightmare.
Back in the day I remember seeing someone do the normals with a special light setup and camera render. I was able to achieve the same in mental Ray but using different render layer overrides, but it wasnt a very efficient workflow.
Last year I tried to tackle it again, this time using Arnold and AOVs, I made some decent progress but got stumped again and gave up, can't remember the thread but it should be somewhere in my replies.
Ultimately I found it quicker to just make the leaves high poly cutouts not using alpha, and bake that out normally instead if fiddling around.
It was possible in max with mental ray but since that's gone away we're all a bit buggered.
It's worth pointing out that it's in no way a technically impossible problem - especially in a software renderer - so I fail to see why nobody's implemented it
Vray does still work (via RTT and refraction) in Max. (I think there is a version of Vray for Blender?)
Normals don't make sense for non opaque / soft alpha surfaces anyway, as you can't simply mix the normals of two surfaces and get the same effect as seeing the two surfaces through each other.
Generally any renderer who can do a cutout of sorts should theoretically have the potential to bake at least 1 bit alpha.
This is the same map in Opacity (inverted) and Refraction, IOR set to 1.
But that at least works, so turning all textures white (edit: Brainfart. Not even necessary, as the alpha itself is ok. I'm using a background object set to matte, matte for refl/refr and alpha contribution = -1 in the background) for baking the alpha and using a world normal texture (in my case three world space falloff maps blended additively with a composite map, after the bake converted to tangent space with handplane) these can be rendered via the Vray complete map. Still a hassle, of course.
I used the diffuse as a bump map on the HP as a rough example that normal maps / bumps of the HP get baked, too, that way.
Arnold isn't bad, in my opinion, and actually quite straight forward (so far I prefer it over Mental Ray, to be honest). It just does many things a bit different, doesn't offer much options in regards to optimization and is overall still poorly integrated (not gonna lie, quite disappointed after I skipped 2018 to avoid just that). Also the axing of free render nodes makes it obviously a bad option in that regard.