3Ds Max Render to Texture mentalray no Multisampling

Raildex

I want to Bake Textures with 3Ds Max (Currently to render Hair Plane textures), but unfortunately, Multisampling doesn't seem to work properly with mentalray (or in general)
This is my mentalray render:

And this is the same Render, but with Scanline and Star Super Sampling:

You see it's looks like shit in both cases, although Scanline is slightly better.

My Sampling Settings are like this:

The thing is, my Rendered Frame Window Looks like this :

I have no Idea what I am doing wrong

Eric Chadwick

How are you generating the hair source objects? And which version of Max are you using?

Raildex

I am using 3Ds Max 2017 and the Hair is geometry, i.e. it's an editable Poly

Eric Chadwick

Ok thanks. It looks like your UV is just a flat square. So I would suggest just rendering it from an orthographic camera.

Raildex

And How Should I render Normals and other maps with that?

Mark Dygert

They've made a lot of changes over the last 5-6 versions with regard to rendering, and Render to Texture hasn't been kept up to date. Even 5-6 versions ago it was starting to break down and become buggy especially with, Mental Ray. It always had issues with AA, padding, background color on normal maps, AO would render artifacts, cage projection distortion when the cage is fine, just a ton of wasted time goes into working around RTT.

The last time I used it, I had to use a mix of standard render (which is ancient AF and lacks a ton of features) and metal ray for AO. I finally gave up and I now bake in substance or xNormal (which is also kind of janky). I'll even bake in zBrush before I touch RTT.

Be prepared for an even bigger cluster fuck if you jump up to 2018 or higher, they ripped out metal ray and have replaced it with Arnold, which doesn't really work with RTT either... There are rumors (or maybe hopeful wishes) that they are going to upgrade RTT, fingers crossed...

Raildex

Thank you for the clarification. My initial assumption was that something doesn't work properly with my hardware.

Eric Chadwick

I'm actually using RTT effectively in Max 2018, albeit with V-Ray instead. Working pretty well, considering. 

If you're baking normal maps, better to use newer tech, like Marmoset or Knald or Substance Painter.

Be aware though, as far as i know none of them bake transparency properly. They just render overlapping transparent polygons in UV order, cutting off lower polygons.

HAWK12HT

Oh this issue, well at work I am using Flatiron and ran into same trouble with trees transparency bake. 
Solution although time consuming but get the results! 
What I ended up doing is applied diffuse texture to leaves first and baked normally with transparency and hell break lose, then applied a black and white diffuse (unlit light setup in new scene) I made from leaves texture and merged that diffuse as alpha in Photoshop and exported as tiff. You can do same with png too. 

Now with hair stuff I have no idea how that is working but I am pretty sure you can set your hair to glow white / unlit and rest of meshes pure black and get your nice aplha texture out as diffuse. 

in theory you should be baking just once and you get albedo/ diffuse and alpha as transparency by default but the math behind it seem to not work and render sees the thing behind transparency too and bakes it in alpha creating chaos. 
Note later on I baked without transparency just to get some smoother results when using cutout material in Unity.