Tangent basis convertion Xnormal/3dsmax?

Marco Splash

I was wondering if there is a way to bake in Xnormal and convert the tangent basis for 3ds max's renderers?

I'm preparing a course for my students on lowres modelling and normal maps baking and I want them to see the power of Xnormal. The problem is that we have to make renders in 3ds max and I'm getting the "black edges on UV seams" issue when I do use default scanline, mental ray or the real-time viewport. And I'm even using the fix where you apply the smoothing groups in order to have the hard edges where the seams are. By looking on the net, I came to the conclusion that it is due to a difference in tangent basis between Xnormal and 3ds max. If I use the render to texture included in 3ds max, I resolve the black edge issue, but the normal maps are not as great as what Xnormal can do overall.

Here's some examples...

That's the source highres, it's the simple cube test with extrusions.

That's the render baked in Xnormal (with black edges)

That's the render baked in Xnormal after I averaged the size of the lowres before I baked (seems a bit better on the edges)

This is the render baked in 3dsmax RTT (which is the best one)

The renders are done with default scanline render (mentalray or default real-time viewport isn't showing good results).

Edit Normals modifier was applied.

Xnormal options were mostly checked (ray distance for the cage, edge padding, axis, etc).

I'm not an expert with the tangent space concept, but do you think there is a fix to help me with my problem?

I heard there might be an importer with Xnormal to make better results in 3ds max, but I looked on the net and haven't find anything. DOes anyone know about this?

If you need precision, I can try to make other screenshots.

Tyrone70

you should try chamfering those edges. the way they are now isn't good enough to capture the curved normals. I could have sworn there was an example of this in the wiki, but it seems to have been moved. however, this section should help you out.

http://wiki.polycount.com/NormalMap?action=show&redirect=Normal+Map#Shaders_and_Seams

Marco Splash

Hey Tyrone, i already read everything in there. I don't think I should have to do chamfers. The test I did using 3ds Max RTT and scanline render is already a quality I'm satisfied with. My consern is to be able to bake in Xnormal, render in 3ds max but having the same quality.

Marco Splash

hahaha, i know what you mean. thing is that RTT in 3ds max can be problematic with heavy models and i know xNormal can handle heavier models... thats the power i'm talking about

Tyrone70

ah, i get what you're saying now.
xNormal isn't perfect (yet :P) but you're not trying to burn a diffuse map, maybe you should try decimating your hi poly's in Zbrush then bringing it into max to bake.

another thing, although I'm not sure if it'll help: try importing your low poly as an FBX. I remember hearing somewhere about obj and custom tangents, but i forget exactly what was said

Eric Chadwick

Well you should be decimating the models a bit before bringing them into Max, get rid of the redundant micro-triangles that you'll never see in the resulting normal map.

Some good workflows here
http://wiki.polycount.com/ReTopologyModeling

Also check out
http://wiki.polycount.com/CategoryTools#Decimation_Software

Marco Splash

@ Tyrone: Unfortunately it doesn't solve the issue... I also tried SBM without success. Gawd!!

@ Eric: I think that if I wanna stay in 3ds Max i'll have to make some decimation alright!

Marco Splash

i always had enough memory to handle undecimated models in xNormal, but it's a different story in 3dsmax for sure.

i was also told by Grant Hill that with the proper knowledge of C++ I could write a plug-in to make the conversion but since I lack the skills of doing so, I was hoping someone on the forums would have that kind of tool to share!

perna, do you know if I would run into the same TS incompatibility if I was to bake in xNormal and render using 3pointstudio in Max?

Bal

One downside to decimation is if you did lots of detailed polypainting with textures etc, and are using those extra triangles (that are useless for the shapes) to show colour variations.
Guess you can always throw some UVs on and convert it to a map, but you'll most likely lose detail there too.

leechdemon

Are you inverting any of your color channels? I've found that, with default settings in xNormals and Max, you have to invert the Green color channel to get an xNormal bake to appear correctly in Max. The Tangency itself isn't the problem, it's just a simple inversion of that channel.

For me, anyways...

Tyrone70

Bal wrote: »

One downside to decimation is if you did lots of detailed polypainting with textures etc, and are using those extra triangles (that are useless for the shapes) to show colour variations.
Guess you can always throw some UVs on and convert it to a map, but you'll most likely lose detail there too.

I would still use Xnormal to bake a difffuse and AO map from a raw hi poly. The good thing about Xnormal is that you can easily use the same projection cage from Max. so essentially, you'd have 2 or three different versions of your hi poly.
1. Decimated just enough to keep valuable normal detail while still able to run in max (500 k is usually a good number for me, but of course it will very depending on the computer and the amount of detail needed.)
2. Raw hi poly you can use for the diffuse and AO in Xnormal
3. (optional) A very decimated version of the hi poly so Max doesn't chug when you try to build a good cage around it.

also, Leechdemon, if you find your self constantly having to flip the green channel in Photoshop, just go to Xnormal's normal map options and change Y+ to Y-

leechdemon

Tyrone70 wrote: »

also, Leechdemon, if you find your self constantly having to flip the green channel in Photoshop, just go to Xnormal's normal map options and change Y+ to Y-

I would, but the basis we're using in our render engine matches xNormals' basis, so I just left it. I've inverted it on Max's RTT and the Photoshop nVidia plugin so that they all match though. ^_^

Bal

Tyrone70, yeah sure, but that makes the workflow more complicated again, more files, more apps, etc.

Marco Splash

the swizzling was already taken care of, which means the Y axis is put to negative for it to be ok in Max.

considering i'm teaching a course on character modelling to new students, I'd say the less software implicated and the shorter workflow, the best it is.

but in the end, IMO, you gotta do what you gotta do if you want to have good results even if it implies using a ton of softwares.

Tyrone70

totally understandable.
i get how the workflow and amount of files increases, but I'm not sure how the software list increases. I always figured people used Maz/Maya, Zbrush/Mudbox, Xnormal and Photoshop in their workflow.