Marmoset Toolbag 3 Normal Maps Artifacts Help

NURBS

So I’m trying to learn Marmoset Toolbag and something that I really like so far is the baking feature, but I’m have a problem when baking normal maps. I’m modeling a Ghost from Destiny in Maya LT 2018 to practice but when I bake the normal map in marmoset I get all these weird artifacts that only appear on the normal map. Other maps I bake, like AO maps, look just fine. I'm not sure what the problem is. Any help is appreciated.

The AO map appears fine.

This is what the high-poly model looks like.

EarthQuake

Try changing the Tangent Space to Mikk / xNormal

If that doesn't resolve it, I would suggest adding hard edges along your UV borders. You've got some very sharp angles in the low poly mesh it looks like, and that can the normal map's job harder than it needs to be. Basically, if those edges are soft/averaged, the normal map has to store rather extreme data to try and account for it.

NURBS

Changing the tangent helped a little bit but the artifacts are still there. About those soft edges; which ones should be hard? Or should the entire thing be hardened?

EarthQuake

Generally speaking, it helps to make the edges at the UV borders hard. Firstly, because any hard edges that do not coincide with UV borders will create artifacts in the baked map. Secondly, because UV boarders already have doubled vertex counts, so using hard edges here does not increase the vertex count when it comes to rendering the mesh in game, so it's a fairly sensible practice.

A couple rules of thumb:

1. If you have hard edges, you must have matching UV seams along those edges
2. Along UV seams, you do not need to have hard edges, but generally speaking there are few disadvantages and many advantages to doing so

Now, you may need to experiment with this. Depending on how your UVs are laid out, you may find you need extra hard edges, at which point you'll have to alter your UV layout to separate the elements at the hard edge lines.

NURBS

So... I should harden edges that have a UV seam?

EarthQuake

NURBS said:

So... I should harden edges that have a UV seam?

Yes, try that.

NURBS

Holy crap it worked! The bake looks sooo much better! I never would have guessed that edge hardness/smoothness could change it that much.

EarthQuake

NURBS said:

Holy crap it worked! The bake looks sooo much better! I never would have guessed that edge hardness/smoothness could change it that much.

Great! Yeah, it can make a very big difference. If you look at the mesh before and after hardening the normals (with no normal map applied) you can sort of see why. Notice how the shading gets very extreme as it runs over faces that have sharp angles. The normal map has to try to account for this, and it's more difficult the more extreme the vertex normals are. So what you're doing here is basically making the normal map's job easier.

NURBS

I see. Well, thank you for all the help! I can already tell that my ability to bake hags increased drastically from this information!