Potentially obvious issue with object space normals.

The Conundrical Preposter

Despite having more and more intuitive workflows every time I sit down to bake normal maps out for a project, there always seems to be that one thing that just plain doesn't work.

At the moment, I'm modeling and assigning vertex normals in Modo, exporting and baking out normal maps from marginally higher poly models in Handplane, and finally viewing it Marmoset Toolbag 2. The eventual models will be for Unity and I plan on tossing some of them through Quixel suite.

Anyway, right now I've hit one of those annoyingly basic holdups and can't get around it. I'm trying to use object space maps because the objects are dynamic and will need it and because those bakes are, for whatever reason, coming out cleanest. The problem is that Marmoset Toolbag is previewing object space maps--seemingly by default--with one side receiving blackened reflections.

This problem is mostly visible because I've turned up the reflectivity to max. Even without any map applied, the object space checkbox applies this half darkened effect and no matter what vertex normals I bake out and display with, I can't seem to make any combination that will get rid of it. For now, the tangent space maps will do but will inevitably have to be converted later on.

Pictures for reference using a really simple door:

1. Basic doors (one rotated 180 degrees) showing the max intensity's default. Both sides match.


2. Flipping the object space on presumably has an effect because the default grey is being read as two separate directions, satisfying one side but not the other. That's fine.


3. Applying the actual object space map clears up the edges of the model again but retains the blackened side. The file type is a PNG 8-bit.


4. With the intensity of reflectivity turned back down again and the albedo set to preview the normals, there's not really any visible problem.

So is this an issue with the normal maps, the vertex normals (which are the same both in the bake object and the display), the file format/gamma/colorspace, or a product of Marmoset's reflectivity or OS displaying?

I'm pretty sure I've missed something obvious seeing as I can't even fix a cube (which is pretty sad). I look forward to the inevitable embarrassment when it's pointed out so long as I can move forward with the rest of the project.

EarthQuake

Object space normal maps have some serious limitations. For instance, object space maps store normal data in absolute space, so you can't bake an OS model and then rotate it it or instance it in a way that will change the rotation. You can see from the last image that the normals are facing the opposite direction on the area where you're having problems.

You should use tangent space maps instead, which store normal data in relative space and are much more flexible.

For more information on baking and normal maps in general, check out this tutorial: https://www.marmoset.co/posts/toolbag-baking-tutorial/

The Conundrical Preposter

Ah. I see what happened. When my brain clicked back online it assumed "object" space was relative to the object without realizing that it referred specifically to the object's center. In that sense it acts just like world space. I'm just as bad at getting the "source" and "target" mesh terms right and prefer using "low" and "high" so that I never mix them up.

Shouldn't object space still be able to capture the inverse side of an object? That black splotch exists on the back of the cube even when I rotate the camera rather than the object. The UVs are not flipped or overlapping. The only reason I entertained using OS normals was because they occasionally baked out better results for whatever reason. Though I'm not using them, I'm curious as to why they wouldn't work at all.

EarthQuake

It's because you've rotated the mesh after baking. It's not a specific side of the mesh that is the problem, it's that you've baked absolute object (aka world) space normal directions into that map, and then rotated the mesh. What you're telling the shader is that left is right and right is left, which results in lighting errors.

When using object space normal maps, transformations need to happen in a way so that the app can keep track of them and update the lighting accordingly. If you rotate the model in your 3D app and export it, you lose this transform data. If you rotate the model in Toolbag, or a game engine, the transforms are retained and applied correctly to the lighting.

Saying all this, object space maps are not typically used in games, aside from as a content source to generate material information.