Topology problem, maybe baking problem?? In need of help!

Trevor

I have been struggling with this one piece of geometry for four days. No matter how much or how little topology I give this object it just will not shade correctly inside marmoset two or bake adequately. Modeling done in 3ds Max.

List of things tried:

-Baked in both 3ds max and Xnormal
-Countless retopo's each with different geometry flow
-Image resolution bake increase
-Smoothing seams everywhere possible
-control loops on the low poly
-Bake object space normals and converted to tangent space with Handplane half a dozen times
-Probably other things but Im frustrated and cant remember them off hand.

Link to download .max file: https://www.dropbox.com/sh/wr7nesxd8bjnelz/AADS-Bcdv-BKViilkTxCFK-7a?dl=0

Problem Normals


Problem Shading


Safe to say im stumped. I could really use some of your wisdom and help Polycount!

SuperFranky

Looks like a non-issue to me, honestly. It's barely noticeable.


If it really bothers you, then I'll explain: because of the way your mesh is built and your UV's are separated, you get a strong gradient on this "peak" with or without the added cuts to make it more rounded. You get this strong artifact simply because the strong gradient just gets tightened even further, creating the artifact you see. You can get rid of it by making smarter UV cuts and smoothing groups to relax gradients, or simply getting rid of the cuts on the peak. I suggest splitting the front faces in half, applying separate smoothing groups and see if that helps.

Trevor

Okay, got the bake to come out fine. I used the object space to tangent space method. Baked normals in Xnormal then used Handplane to convert to t-space. This method is the best to lessen the gradients so far. Now another problem introduces it self, faceting on some faces but not all. The faceted areas are within there own smoothing groups and there are no inverted faces or overlapping verts. So how to fix this?

huffer

The most important thing is getting a correct sync between baker and 3D viewer. If it's the same tangent space between them you can get away with any gradients. If you bake in 3dsmax use the 3dsmax tangent space in Marmoset's mesh settings, if you export use .fbx with tangents and binormals to carry along the tangent space calculatipons. The normal map should look like that, gradients in the normal map compensate vertex shading gradients in the mesh to recreate the high poly's perfect shading. I'm not sure how accurate is using the handplane method, plus no support for xnormal/mikk.

acassani

If you use a mirror modifier (like it seems you did here) make sure there is not faces inside geometry just where the mirrored geometry and the original geometry meets.

Trevor

Exporting as FBX fixed the faceting.

I tried the synced workflow but got... worse results. Now im really confused.
Here are the two results both viewed with the same FBX in marmoset two and tangent space set to 3ds Max.

A. Xnormal to Handplane. Baked with OBJ in XN as 16 bit tiff World Space (see link for why 16 bit tiff). Map and low poly OBJ put into Handplane and output 16 bit tangent space conversion then down sampled to 8bit and saved as tga.

B. Xnormal. Baked with FBX low poly and cage exported out of max with tangents and binormals. Used same 16 to 8 bit conversion method described above then saved to tga. Intresting note, exporting the cage out of the projection modifier in 3ds max did not work in Xnormal. I had to duplicate the low poly and apply a push modifier then export that as a cage.



Am I doing the FBX baking method wrong? Because those results on B are highly undesirable or am I truly that picky about the outcome. A is what I was looking for in the first place.

SuperFranky

If you bake in Xnormal, why would you set tangent space in Marmoset to 3ds Max?

Trevor

Because I bake Y- since I primarily use UE4 and Max . Isn't 3ds Max correct normal orientation Y-?

SuperFranky

Trevor wrote: »

Because I bake Y- since I primarily use UE4 and Max . Isn't 3ds Max correct normal orientation Y-?

normal orientation doesn't have anything to do with tangent space.

Trevor

So I went back and tried different tangent space settings and normal map A shaded correctly on all tangent space settings. I Just had to flip Y in some cases. normal map B still has those artifacts regardless of tangent setting or Y orientation. It feels like I am missing something here with the fundamentals of normal mapping.

huffer

Simply put, tangents and binormals are the tangent space calculations, a gizmo (or swizzle) of three vectors per each vertex that encodes and decodes (at baking time and rendering time) the normal map gradients taking into account the vertex shading, vertex normals, and UV coordinates. Each program calculates this differently (the vectors have different values). Not only 3dsmax calculates it differently, but the Y vector is in the opposite direction (compared to xnormal). So sometimes flipping the Y doesn't mean that much, you need first of all to use the correct tangent space.

Here are the two safest methods to get a perfectly synced bake with UE4. UE4 uses the Mikk tangent space (or something alomst identical), 3Dsmax has its own tangent space. I only found out this stuff and understood it a couple of months ago You can:

- bake in 3dsmax, then export .fbx with tangents and binormals, import in Toolbag 2 / UE4 (remember to import tangents and binormals), and set tangent space to 3dsmax inside Toolbag 2 (this means there's a different swizzle, with the Y vector flipped on-mesh). There is an issue with 3dsmax, after you bake and apply the normal map you'll get some artifacts, you need to add an Edit Normals modifier (and don't change anything), then collapse, then export.

This means that: your bake is synced to 3dsmax tspace, not UE4 / mikk, but it will look the same in all programs as long as you carry along tangents and binormals (you're just exporting the tangent space along with the low poly). You can also bake in xNormal with this .fbx, but your tangent space will still be the 3dstudio max one (as long as xNormal detects tangents and binormals in a mesh, it will use those, not recalculate them).

- export obj from 3dsmax (or .fbx with tangents and binormals OFF) to bake in xNormal, and enable "Compute binormals in pixel shader" in xnormals plugin settings for tangent basis calculator. Now your bake is synced with mikk, but it will look good only and only in UE4 / Toolbag 2 (version 2.05 and later, I'm not 100% sure but in earlier versions mikk tangent space was implemented differently). Now you have to set the tangent space to xNormal / Mikk inside Toolbag 2, and don't forget do discard tangents and binormals on further exports.

- I would not use handplane, since it doesn't have a setting for UE4 / mikk. You could try to bake object space in xnormal, then convert in handplane to tangent space using the 3d studio max preset, then see how the normal map looks like in max, if it's good, then export .fbx with tangents and binormals further along to UE4 and Toolbag. But you can bake directly in 3dsmax. You could also use handplane if you wanted a bake synced with 3dsmax for example, but only had acces to Maya.

Also note, that it's important to have a proper sync in case you have gradients on mesh, if you model hard surface, and have lots of supporting loops and hard edges you could get away without syncing tangent spaces, but in this case you have to get it synced.