Normals/Smoothing Group problem

Zelenkov

Hey guys. I'm making a lowpoly slot machine as a prop. So I've made my hipoly and started casting. And this is the part of the model I'm having most problems with. I got a bit of a problem with normals and smoothing groups. They seem to bake and look fine in Maya but look bad in Marmoset. I've tried averaging normals, softening edges, rebuilding parts of geo but after 3 days and no luck. I would very much appreciate help with this problem. Thanks.

Left: Maya; Right: Marmoset.

Ben Apuna

When you say "looks good in Maya", do you mean in the viewport? or in a actual render?

The normals need to look good in the viewport and even then the tangent basis will be different for Maya and Marmoset so you will never get the exact same result.

It looks like you're having some issues with your normal map where it's going around corners. There are basically two ways you can handle 90° angles with normal maps on your low poly geometry.

1a. Split the normals (harden edges/separate smoothing groups) along the corners.

1b. Split the UVs along the corners as well and make sure that there is some padding between the resulting shells to avoid the "black edge issue".

1c. Bake your normal map using a averaged cage.

or

2a. Bevel the corners of the low poly so that they aren't 90° anymore.

2b. Make sure all the edges are smooth (soften edges/single smoothing group).

2c. Stitch the UVs of those sides together so that they make one continuous shell.

In either case be sure that you also triangulate your low poly before you bake and use the exact same triangulation on your in-game model.

There's actually a 3rd option which is to use a normal map bake and shader that is synced up with the viewport display such as the 3Point Shader for Max, but that requires Max and has isn't useful for Marmoset.

I hope that helps.