Compositing - 3dsmax

MoP · Mar 2006

Hey folks,

I've been working on a simple scene in Max for camera matching and compositing... camera match and making matte objects was easy enough, it's just the lighting and matte rendering itself which is giving me headaches.

I feel like I'm up against a brick wall with this one, I've scoured the Max help files to no avail, also Google seems to be failing me on this point too. I feel like I must have missed something really obvious (heh, that's happened before!) but I can't think where else to go, so I'm asking here!

OK, here's my horrible rendering so far:

As you can see it's pretty basic... here are my two problems:

1. Shadow casting. As you can see, for shadows to be correctly cast onto the 3dsmax ChamferBox test objects, I had to model the shelf and DVD boxes and apply Matte/Shadow materials, so the Chamferboxes receive correct shadows, but the photo shows through on the shelf and DVD boxes.
However I also need the Chamferboxes to cast shadows on the wall (a big Matte/Shadow plane), but this means that the Matte/Shadow shelf and DVD boxes are also casting THEIR shadows onto the wall too, which is no good since it doubles up on top of the real shadows in the photo!
Exclude/Include lists in the lights are no help, since they only allow turning off shadows on a global basis, whereas I need to be able to turn off shadows on a per-object basis. Anyone know how I might be able to do this or otherwise get around the problem?

2. Reflections in the mirror. As you can see, that's just screwed - I've got a Matte/Shadow material with a Flat Mirror map in the Reflection slot on the mirror geometry, but it's making the reflection of the ChamferBoxes really faint and transparent, even with the reflection value set to 100.
I've tried turning on and off pretty much every single variable in the Matte/Shadow and Flat Mirror materials, and nothing gives an accurate reflection (ie. 100% mirrored chamferboxes, with environment showing through 100% in the other areas). All I've been able to get is either a black mirror with correctly reflected boxes, or some weird combination of the environment map overlaid on the background...

So, yeah, that's my dilemma. If anyone knows what I'm doing wrong, or a better way to do this, or even a more appropriate forum to ask on, point me the right way please!

So yeah... per-object shadow exclusion, and correct matte mirroring so I won't have to build a perfect 3d copy of the room behind the camera.

I'd prefer to be able to do this all from inside of max rather than having to run it through an external compositor... and the stuff I want to do doesn't sound so complex that it'd need another program to make it work... right?

Cheers for any replies.

Ninjas · Mar 2006

Are you using Mental Ray?

MoP · Mar 2006

Nope, this is scanline renderer. I tried switching to mental ray and it had exactly the same results.
Why - is the mirror matte meant to be working differently?

Ninjas · Mar 2006

No, I just don't use scanline because I think it sucks, and I'm not sure if any of my advice will work in scanline.

Well, I'm not exacly sure I perfectly understand all of you problems, but for starters, multipass rendering is the way to go. I usually use After Effects to comp everything together. A lot of these problems can be fixed manually by using animated masks in after effects, or rendering off masks to hide areas of layers in After Effects.

Also I think that there are cast/recieve shadow setting somewhere. In the object settings maybe? I need to fire up my computer to check. You can render relection it it's own pass and adjust it to your hearts content... I'm not sure that will fix your problem, but it may be a start.

[edit] yeah. There are setting for casting and recieving shadows under "object properties". Maybe that can help too.

MoP · Mar 2006

Yeah... I was hoping to avoid having to use After Effects, since that would mean going into university and using the labs in there every time I want to check out an update.

You're right that there are cast/recieve shadow settings in the object properties. However again like the Exclude/Include lighting settings, these are global - so if I turned off shadow casting for the Shelf matte object, it'd fix the problem of it duplicating the "real" shadow on the wall, but at the same time it would stop casting shadows onto the chamferboxes, which would diminish the believability.

I'm sure I once saw an option somewhere to set shadow interactions between specific objects, but either I was hallucinating or I don't know where those settings are now.

Looks like I might have to go the multipass route, although it feels like this should have been a simple fix (especially the "semitransparent reflection" problem).

Cheers for the help anyway.

Ninjas · Mar 2006

I'm sorry I couldn't help more. Compositing in my experience is really not that fun

arshlevon · Mar 2006

hrmmm.. if your just mostly using these matte materials to be proxies for shadow casting why dont you try using multiple sets of lights. a set just to handle the background, a shadow casting light and a no shadows light, and use the include/exclude list to choose what objects you want to have shadows and not. and since they are for matte materials you can turn illumination off so as not to mess up your primary lighting. and your primary lighting can exclude all the matte objects casue you will have a seperate set of light affecting them...
might work, i dont know..

MoP · Mar 2006

Thanks for the suggestion Arsh - I did actually try that this evening but didn't have much luck, maybe I just didn't spend enough time setting up include/exclude lists ... I'll probably end up having to use about 10 different lights all in the same location with different Exclude/Include lists... but hey it should work, I think you're right. I'll give it another try.

CrazyButcher · Mar 2006

wasnt there a checkbox per object (in properties) that allows to disable shadow casting. I think per object there is two checkboxes, receive and cast shadows. that would fix the issue with matte objects casting shadows.

about the reflection, no idea, but I guess you are doomed to masking the reflection manually somehow. maybe the object's reflection shows up in the object id buffer ? (just guessing) you could then use that as a mask for the reflection. other than that the renderer doesnt know you want just a single object in the reflection, cause the environment (black) is also a valid reflection.

MoP · Mar 2006

Hi CrazyButcher, thanks for the reply.

As I mentioned earlier, the problem is not just disabling shadow-casting, but specifically to stop objects casting shadows on some meshes, but keep casting shadows on other meshes. I think I will have to do this the way Arshlevon talked about.

You're correct about the reflection, it does show up in the alpha channel so I could use it as a mask, but I've found a fix for it now - I just applied a Raytrace material to the plane of the mirror matte, now it reflects the 3d objects completely solid, while also keeping the environment background true to the photograph - I can't believe I didn't try that earlier!

Back to work...

thomasp · Mar 2006

if i have learnt anything in working with compositing then it is to render everthing and then some in passes and do the tricky bits exclusively in the comp app.

not only is it a great time saver overall and helps to avoid the numerous max render bugs but you'll also never be able to experiment a lot with a dog-slow renderer such as mental ray.

in compositing it's so much quicker to find that final look.
using a compositor with a true 3d workspace helps a great deal as well. no afx experience here though.

MoP · Mar 2006

Heheheh, after a bit of thinking I realised I could just use a cheap hacky method:

Basically just quickly cloned + painted out the shadows in Photoshop, and left everything shadow-casting in the Max scene. Basically it just recreates the original lighting setup.

Since I'm only compositing my animation onto a still frame, this will do

Cheers for the ideas and advice guys!

Ninjas · Mar 2006

Interesting solution! I like it.

Compositing - 3dsmax

Replies