3ds Max - Baking Normal Maps with Alpha

Zyth

Is there any way to bake a normal map with alpha transparency in 3ds Max?  Something like this:



I want to create some normal maps that I can use in Substance Painter as extra details.

poopipe

Not natively. 
You don't need the alpha anyway - Painter takes care of it

Mark Dygert

Yep, you'll have to bake in something else, painter or xNormal.

Eric Chadwick

If you want, you can simply render with a camera, instead of using RTT.
http://wiki.polycount.com/wiki/Texture_Baking#Transparency

Noors

If you need some specific alpha shape, you could apply black and white materials to your objects, then bake as diffuse.

Zyth

So I've been experimenting with this and got it to work in xNormal.  The only problem is that it doesn't anti-alias the edges of the shape even with anti-aliasing set to 4x.  I decided to try it in 3ds Max using what I learned in xNormal.  It turns out you can bake a normal map with alpha transparency if you save it as a PNG with Alpha.  You'll also want to set the Padding to 0.  The good thing about 3ds Max is that it anti-aliases the edges unlike xNormal, however, now I have a question about tangent basis.

Since the normal maps are being baked in 3ds Max but will be used in Substance Painter do I have to do any flipping of the channels so that they display correctly in Substance Painter?

poopipe

max is the same orientation as substance directX

Noors

You can also flip the channel in max Render To Texture/Projection Mapping/Options/Normal map space

Thanez

For the Antialiasing issue you can always bake at 2x the res and scale it down in photoshop or pretty much any other software. That'll AA your texture.
Doing AA in post also has it's benefits, as you can near-instantly try out different AA or downscaling algorithms. If you have patterns or a texture in the normals, you might want that.
Also, baking at 2x res without AA in MAX is faster than at 1x with 4x AA.