Toolbag's baked normal map looks different depending on format they're applied to, FBX or OBJ.

CatPump

I've been baking all of my maps with Toolbag 3, and the results are really outstanding.  BUT! When I do my baking with a pair of OBJs (high & low poly), and then apply the resulting normal map to an FBX, I get weirdness around the seams... always the goddamn seams. 

OBJ

FBX


Any solutions? 

EarthQuake

It could be that the mesh normals and/or tangents are saved differently from your 3D app depending on whether you're using FBX or OBJ.

If you're using FBX, try turning off bi-normals/tangents in the export setting, as this will cause your 3D app to save custom tangents, which TB3 will use by default. You can also go to your meshes and manually select your desired tangent space. If tangent space shows up blank, this means the tangents were imported from the FBX.

CatPump

Thanks. The first thing I did was mess around with the FBX export settings and toggled the tangents option but to no avail, the problem persisted without effect. 

I cycled through the tangent space options just for kicks, but since Tooldbag itself is responsible for the normal map, naturally no other option fit any better. 

I ended up baking the hi-poly OBJ to a lo-poly FBX after manually tweaking the scale difference, which ended up solving the problem, but only as a work around.

Normal maps should, in theory, work the same regardless of the format they're applied to, which is why I'm surprised to be the first to signal this.  Maybe I'm missing something, but I feel it's pretty common to bake in OBJ, then transfer to FBX for animation. 

EarthQuake

After you've changed the tangent space, you'll need to rebake the map, as Toolbag's baker uses whatever tangent space your mesh is set to.

Also just to clarify, what renderer are these screenshots from? Is this Toolbag or something else?

In theory the format should not matter, assuming everything stays the same like triangulation, uvs, normals, and tangents, however it's pretty easy for one of these to vary in some way which is probably what is causing the issue.

thomasp

when this happens to me it's always because of vertex normals. usually a case of forgetting to triangulate before export or triangulation changing between bake export and animation mesh.

e.g. if you originally bake to triangulated OBJ but hand off a quad mesh in whatever format, it's going to trigger a reevaluation of vertex normals upon import.

EarthQuake

Yeah, and this is complicated by which/how many 3D apps you're transferring to/from as well. For instance, the exact normals you'll get out of Max or Maya may vary. Generally if you need to export an OBJ/FBX to send off to be animated, the best mesh to bake would be from your animation app, temporarily triangulated and with normals locked so you can make sure nothing changes. 

CatPump

Thanks guys, yeah I most likely forgot to triangulate.