Render shadows on transparent background

madmenno

Hi,

I am creating sprites for a game and I want my object to cast shadows on a transparent background. I am using 3Dmax scanline renderer.

Rendering with just the object gives me a image with a alpha but obviously the shadow does not extend passed the object.

So I have setup a ground plane with a matte material to receive shadows. But I cannot set it so only the shadow gets rendered.

I just want the object on a transparent image that includes the full shadow in alpha. So I can overlay this image (that could be a building) on top of ground images in the game.

Thanks in advance,

Eric Chadwick

I've done this using the mentalray renderer in 3ds Max, their Matte/Shadow/Reflection material works better than the Scanline one.

However alphas don't show up properly in window reflections: objects in shadow are missing from the reflection's RGB, and shadows in a reflection don't show up in the alpha at all.

Here's an example




FWIW, V-Ray does a really good job at generating proper alphas for shadows, refractions, and reflections.

madmenno

I remember doing this is the past with the scanline renderer. I might also need to turn anti aliasing off and "degrade" the renderer for a oldschool look of the final image. MR is overkill to what I am looking for.

I attached a image of what I exactly want. You can see the rendered image in the top right. That is perfect, but as you can see there is no alpha being rendered. The plane below the object has that Matte/Shading material applied to it and it does not have opaque selected. I tried many combinations here but nothing gets me a alpha so I can overlay the image nicely over other images.

Eric Chadwick

Hmm, works fine for me. Which version of Max?

Make sure Affect Alpha is checked in the shadow section!

madmenno

Eric Chadwick wrote: »

Hmm, works fine for me. Which version of Max?

Make sure Affect Alpha is checked in the shadow section!

Max 2012 x64 and yeah I tried checking that too, I tried shadow map (which I prefer for slightly softer shadows and ray traced shadows as well.

Here I deleted the old plane and created a small one and added the matte material again, I also selected "Affect Alpha". See what happens.

Eric Chadwick

Here, see if this works.
http://ericchadwick.com/examples/files/2015-08-16_scanline_matte_shadows.zip

madmenno

Eric Chadwick wrote: »

Here, see if this works.
http://ericchadwick.com/examples/files/2015-08-16_scanline_matte_shadows.zip

Nope, the complete plane is being rendered. Here is the alpha channel I get...

Are you getting a alpha shadow? I remember it was pretty easy to setup what I want but I cannot seem to get it anymore.

Eric Chadwick

Weird. Here's what I get. Maybe you have some funky render settings in your Preferences?

madmenno

Eric Chadwick wrote: »

Weird. Here's what I get. Maybe you have some funky render settings in your Preferences?

That is strange indeed. I am after exactly what you have. These are my render settings, cannot remember I ever touched those but it has been some years I used Max.

I can work around it using MR, sadly MR does not support toon shading, at least it does not render my current setup with the toon shader. So for each image I have to render out an additional mask and that will eventually be a lot of work and perhaps post editing too. It must be possible in the default scanline renderer.

madmenno

I am getting somewhere! I had my environment in a gray color when I turned it to black the matte plane dissipated from the render. Is this normal behavior?

But now my shadow is to strong. I have set my shadow density in the light parameters to 0.33 like you did and in the matte itself I toned it down too. Yet it is still completely black. When I set the environment to a light gray again I can see a soft grayish shadow exactly how I want it but when I put the enivronment back to black to get rid of the plane in alpha the alpha renders solid black. The alpha is not anti aliased at all btw.

Here are both final renders, one with the enivronment on gray rendering the plane in the alpha channel and one with environment on black rendering the shadow but too strong.

Eric Chadwick

"Don't anti-alias against background" is preventing the alpha from being fractional values.

You may just have to post-process the output, if you clamp the alpha to two values that might give you the effect you want.

madmenno

Yes turning that off did the trick. Why was that on by default anyway? I am close to my end result just a bit of fiddling with the correct shadow "thickness" and I can render my assets.

Thank you very much!

Eric Chadwick

Some games avoid using alpha, by disabling antialiasing, using a bright pink bkg, and color-keying away every pixel with that color.

Inugami

If UR trying to do this in vray renderer it will not work. Here is easy tutorial for this:
http://rafikgfx.blogspot.com/2014/09/vray-car-rendering-using-hdri-as.html

If Your problem is different kind, I can not help