3DSMax RTT - How to bake using alpha thru DIF/NRM/etc.?

ParoXum

Hi, got a bit of a problem with baking objects using alpha channel. Basically what I want to do is bake lots of layers of dead leaves which are just alpha cards scattered on a plane. I'd like to get a usual diffuse/normal/AO bake from this and I'd like 3dsmax to take into account my alpha for the leaves:

PROBLEM: it cant seem to use the alpha channel at all when baking a diffuse or normal map with RTT. Doing a normal render seem to work tho...But I don't want that.

Close up of the result: (you can see leaves overlap each other where it should see through..)


How can I force RTT to not draw transparent pixels in my bakes?

Thanks for any help you could have!

divi

http://www.polycount.com/forum/showpost.php?p=826598&postcount=32
not sure if there is any other way as of yet. RTT just doesn't like transparency.

ParoXum

Yeah I thought of that, but that is so far fetched just to get the thing done. How come there's no quicker work around?

Could Xnormal bake using alpha for example?

Noren

As far as I remember MR can handle this kind of situations better.

ParoXum

I tried mentalray as well, it's no different basically, I had it default tho, maybe some magic setup on mental ray would make this thing work, but again, I don't know how.

cptSwing

ToTex (a RTT script) has an Alpha function. Haven't tested it though

sprunghunt

if you're just doing a flat image have you tried just renderng the top view - or rendering from an orthographic camera?

Also what does your alpha channel look like?

ParoXum

Thanks cptSwing, gonna give this a try sometimes this week! With some luck it can do what I need.

yeah sprunghunt that's basically what earthquake did with the vines. That'll be my last resort I guess.

Noren

I just run some tests with Mental Ray and basically it can do what you want. However it also introduces some more quirks and bugs that you'll have to work around.
First you'll need an alpha texture with decent resolution.
It can't have any anti aliasing, just 100% white or black pixels.
Also the filtering for that texture should be set to none. (Otherwise you'll have white pixels in your rendered maps. )
Use Arch&Design as a material. Put your alpha in the cutout slot.
Now you'll still have the problem that your alpha map won't anti alias properly against the background and you could work around that by putting a plane (or other mesh, in case of a more complicated bake) in the bg and use e.g. the specular slot for getting your alpha. But unfortunately MR also refuses to filter your normal map. So the simplest solution is to render in double resolution or bigger and downscale later.

sprunghunt

ParoXum wrote: »

Thanks cptSwing, gonna give this a try sometimes this week! With some luck it can do what I need.

yeah sprunghunt that's basically what earthquake did with the vines. That'll be my last resort I guess.

another quick workaround is just to render the leaves with a new material that uses the alpha as the diffuse. So you get an image that's just black and white.

ParoXum

sprunghunt wrote: »

another quick workaround is just to render the leaves with a new material that uses the alpha as the diffuse. So you get an image that's just black and white.

This wont work I think since there's lots of alpha cards interpenetrated (not sure of the word), so this would result is an alpha card jumping to another in the final result.

Noren thanks for the advice, I'll try that one first and foremost. I guess the AA thing can be solved by rendering an 8k for a final 1024, should give plenty of AA.

Noren

I guess so, just try out how big a resolution you actually need. .You could also set max samples to 1 per pixel to reduce rendering time.

paulsvoboda

I've run into this problem before and as far as I can tell there is no way around it using RTT. The best solution is to set up a camera where you top viewport is, set your resolution to 2048x2048 (or whatever res you wish to render) and then render out your diffuse, normals, AO, Spec and whatever other maps you may want.
The best way to get Normals is to use this method: http://bencloward.com/tutorials_normal_maps11.shtml - apply a white material with just your alpha assigned to it.

I use this method but have it setup in Maya as that's the workflow I am familiar with but it should work just as well within Max. For the diffuse sometimes I give the material a slight translucency and have an ambient light setup (no shadows) with a very dim key light - this can add some nice touches to the diffuse.

Eric Chadwick

Yep, this is the best way.

A couple more tidbits of info here, but most has been said already.
http://wiki.polycount.com/NormalMap#Baking_Transparency

ParoXum

Allright, thanks every one for the help. I was able to bake the normal map the way I needed finally. Here's a little feedback on what's been said in this thread and if it worked for me in this case:

Default RTT : failed to do what I needed
Mental ray RTT with Arch&Design cutout technique: failed to do what I needed
DirectX FX shader: would have worked if my objects to bake were geometry based, not alpha based.
ToTex script: sorry didn't have time to test that one out.
5 lights setup and ortho camera: worked perfectly. All I had to do was making a no diffuse standard material, with my alpha in opacity and normal bump enabled.

With the light setup and the alpha sorted I was able to get what i wanted:



This sample doesn't tile

Again thx every one for helping me on this one.

Noren

"Mental ray RTT with Arch&Design cutout technique: failed to do what I needed"

What was the problem exactly?
If you want, you can send me a PM and I'll give you my email address so you can send me the files. Just out of curiosity, after all we won't bake from planes all the time, even if the problem at hand is resolved.

ParoXum

well it resulted in the same as the default RTT, rendered thru F9 I can get the diffuse like a standard material but in RTT it gives the same output (alpha not taken into account) and I needed the normal map.

Noren

Perhaps a refraction issue? (I noticed this is turned on by default and the standard refraction is set to 1,4.)

Some pics to show it should be working:
Just a proof of concept, not supposed to be pretty.
No padding for better readability.
Note there are lots of geometry intersections that make some of the leaves seem cut off, but no alpha problems at all.

ParoXum

Would be cool if you could share your render preset file here, if it's a matter of setting up MR correctly

If you were speaking of refraction in the material indeed i did not check that. Also you were speaking of putting 'no filtering' on textures, but I didnt find anywhere an option to set that on a per texture basis.

Noren

You can set that in the material editor where you load the bitmap:


Refraction you can deactivate globally in the render settings:


Here's a preset, but it overwrites some of the settings you might want to adjust for different renderings/scenes. Best you go by the pics and description and understand what's going on.
http://home.arcor.de/Noren/MR_RTT_cutout_preset.rps

ParoXum

Cool thanks for the informations Noren. I'll try and give this a go asap. Completely forgot there was a filtering selection just next to the alpha settings, my bad.

ParoXum

Okay, tried with those settings, I didn't try with the preset yet, the diffuse with the cutout map works perfectly with RTT. But for some reason the normal bake doesn't do as well~~

Noren

hmm, that's really hard to tell without looking at the actual file.
Did you try the preset meanwhile?
The greyish/yellowish color, is that from the diffuse map or perhaps the mesh color of the lowpoly you are rendering from? Or the background?
Try setting distinct colors for those in order to find out.
The white spots might indicate that your alpha is not "hard" (just pure black and white pixels), or is getting filtered after all.
Then I never tried rendering with a bump or normalmap on the "higphpoly". (It doesn't seem to appear in that latest rendering, though and I doubt it's the culprit. But you never know.)

ParoXum

I tried it again today, with your settings and your preset, no material filtering on every texture samples. I didnt even add the normal map so it should only bake like your example. And I still get the same result as the image above.

I'm going to test it in 2010 in a few minutes but that was for 2012.

I'll update this post.


Oookay, so this is gonna be a huge help to all the users of this cutout technique: THIS ONLY WORKS IN 2010 (and maybe previous versions)!

Thank you again Autodesk.

Noren

Well, bummer...
Sorry for not mentioning the version.

ParoXum

No problem

This cutout technique should be added on the wiki I think, unless it's already on it.