Maya - XNormal - Handplane - Marmoset Toolbag 2 Workflow

lordsme

Hello,

I'm trying to setup a workflow for hard surface assets based on these steps:
- hipoly & lowpoly modeling/UV in Maya 2012
- triangulate and export lowpoly and hipoly FBXs
- normal map (object space) bake in XNormal
- triangulate and export H3D mesh for Handplane
- object space to tangent space map conversion for Maya viewport 2.0 using Handplane
- object space to tangent space map conversion for Marmoset Toolbag 2 using Hanplane (which settings?! I tried Maya again)

Everything works fine, except the last step: I can't find the right settings to get correct shading in Toolbag. I tried almost everything in Toolbag: loading FBXs, OBJs, changing tangent space option... I also tried to load the object space map (just for a check) and the shading seems broken too...

Shading in Maya Viewport 2.0 is perfect.

Can anyone help me setup a right workflow?

AdvisableRobin

Can you post pictures of your highpoly, lowpoly and normal map? Maybe some images of the specific shading errors you are getting.

Also try just baking a tangent space normal map.

lordsme

Here are 2 images:
- hi poly and low poly shading in Maya, no problems
- hi poly and low poly shading in Marmoset, low poly shading seems to suffer of bad difference between tangents and normal map...

Normal map is in tangent space.

lordsme

And here's part of the baked normal map in tangent space

lordsme

And here's the shading of the lowpoly, so you can see its normals (only UV shell borders are hard edges)

jfitch

Have you attempted just doing a tangent space in xnormal? You may also have to look at the settings for your normal map in toolbag, make sure they're default and srgb isn't turned on.

AdvisableRobin

When you import a mesh into Marmoset you can change the tangent basis of that mesh in the options. There is one for Xnormal. Second make sure that on the material under normal map Scale & Bias is checked.

As far as the shading of the lowpoly, the extreme gradients you are seeing are because you have 90 degree angles without either hard edge and UV splitting them or beveling them down to less extreme angles.

lordsme

I forgot to mention that I use Maya 2012.

jfitch wrote: »

Have you attempted just doing a tangent space in xnormal?

Doing a simple tangent space bake from XNormal results in the same shading artifacts even in Maya 2012.

Already tried to change settings in Marmoset, can't fix it.

I set the lowpoly shading like that because I tried to follow the workflow described on other forum topics, with only UV shell border edges set as hard edges.
And if I set all edges hard, baking artifacts appear on all edges.

At this point I'm totally confused... is there any example around on a workflow at least between Maya 2012 and Marmoset 2.0?

AdvisableRobin

So, its important to understand that you don't necessarily need to harden edges at your UV splits. But you absolutely must split your UVs along your hard edges. Giving your sharp angles hard edges will reduce the shading gradients and yield a better result on the normal map because the gradients won't be so extreme. Once you make a pass through on your hard edges then you need to split your UVs along those edges.

An alternative to using a hard edge would be to bevel the geometry, thus removing the sharp 90 degree angle, which means you won't necessarily have to split your UVs along that edge.

If you haven't yet, I would recommend reading THIS

lordsme

Thanks Robin,
I already read it, and if I understood, I chose to use an “Averaged projection mesh” workflow. Instead of setting all edges to soft, I set as hard edges the UV borders only. Yes, this creates some extreme gradients in the lowpoly shade and in the baked normal. But it looks perfect in Maya 2012, using the workflow I mentioned in the first post of this thread.

Can't understand why this is not working in Marmoset.

Then yes, I could use more hard edges, then I should split the UVs in more shells...

lordsme

I found that there's something related to lock/unlock normals. It seems that FBX export does not properly export tangents and binormals, even if I check the "Export tangent and binormals" option. So using the Handplane H3D exporter works fine, but FBX exported for Marmoset are not fine at all. The same FBX, re-imported in Maya, is broken. But it's fixed if I check the import option "Unlock normals", that if I understand what that means, makes Maya re-calculate normals...

lordsme

I finally found a solution, quite painfully.
Here's the workflow that I found, not 100% perfect but that's definitely a starting point.

- lowpoly and hipoly modeling in Maya
- set a minimum amount of hard edges on the lowpoly to avoid too sharp angles (>=90°), and unwrap in a way that every hard edge is also a UV border
- model a cage for baking
- export hipoly, lowpoly and cage as OBJs
- bake in xNormal an object space normal map
- export lowpoly as H3D (Handplane file format)
- use Handplane to convert from object space to normal space
- in Maya, load the tangent space normal map as "tangent space / bump", use Viewport 2.0 or High Quality Rendering
- in Marmoset, load the lowpoly OBJ, set its tangent space to "Maya", load the tangent space normal map in the material Surface slot, enable "Scale & Bias", uncheck "sRGB"

The result is: perfect normals in Maya, almost good normals in Marmoset.

Then, I found some facts that were creating the big problems:
- Marmoset Toolbag can't load properly 16bit TIF normal maps
- Maya 2012 tangents & Viewport 2.0 are very buggy: everytime I unlock or do anything to normals, the tangents change and so the shading. Undoing does not restores them. You need to re-open the scene to get them back in place and have a correct shading. Everything get worse if you try to export a lowpoly FBX, with or without tangents, because you're not sure anymore what kind of tangents went into it, because you're not sure if your viewport is right or not...!!!

AdvisableRobin

Can you post some images of your results? Are they better or did you completely solve your problem?

lordsme

Here's the lowpoly model, normal mapped, wire and base shading in Maya viewport.
Next time I'll surely add more bevels, to reduce gradients in the normal map, anyway in Maya it's working fine also like this.

AdvisableRobin

Don't trust the Maya viewport, its wonky and doesn't work the same way a game engine would work. Get your stuff working in your engine of choice or Marmoset.

lordsme

And here's a screen from Marmoset, where you can see the hipoly in front of the lowpoly, with the shading artifacts on the lowpoly highlighted in red. They only show up on angled faces.

Normal map is also tweaked in Quixel NDO to add some details as you can see, but that does not matter, shading artifact are always there in Marmoset...

Andrew_Sovetkin

How did you solve the problem?