Is my tangent basis un-synced?

ryebot

I'm having a heck of a time trying to eliminate seams in my normal map bake. This was baked in substance painter. The seams are pretty harsh, the colors are totally mismatched.



This seems to suggest to me that that there's a setting I didn't set right. I'm reading a lot about having a consistent tangent basis between my map and my viewport, and that would seem to account for a big error like this.

As far as I understand, the mesh has one smoothing group. (I clicked on the whole object in maya and selected Mesh Display > soften edge).

This is driving me crazy, I'd be really grateful for any insight into the problem!

JedTheKrampus

Color mismatches at UV seams are supposed to happen. What does it look like when you view the normal map when it's used as a normal map?

ryebot

Here it is painted in Substance. The seams are pretty visible, and the shading is affected.



I could re-UV the model to try and better hide the seams, but there's no way I could hide them all.

spitty

it looks like your other textures (diff, roughness) might be causing the seams.  throw a quick fill layer on it in substance painter and see if you still see seams.

ryebot

here is it with just a red fill. They're a bit harder to see, but still there, specially on the back-side. The shading runs a tad darker along the seam and then switches to a lighter color.

spitty

yeah i guess there is still a bit of a seam.  i still try to hide uv borders, but i've been lucky with baking in substance painter, haven't seen any seams on my uv borders.

if it matters, i export my low poly from 3dsmax as an fbx with smoothing, convert deforming dummies to bones, triangulate, and preserve edge orientation checked.

in substance painter under edit, project configuration, my normal map format is direct x and compute tangent space per fragment is unchecked

i think my settings in bake textures are mostly default. i think i bumped up the dilation width.

ryebot

Thanks, @spitty! I'll double-check my settings. I think I've done most of that stuff (coming from Maya). It's driving me crazy. I'll probably juggle UVs around as much as I can, I will be pretty sad if this is the best result I can get.

Joopson

The color of a particular part of a normal map depends entirely on how the UV shells are oriented in relation to one another, so it's normal for their colors not to match up, when viewed like that, and isn't indicative of any problem.

As for the seam, it's really pretty minor. How close will you be seeing that part of the mesh? It might not even be necessary to do anything about it, depending on your needs.

Otherwise, bake at a higher resolution and see if that gets rid of, or lessens, the seams. I think that may be a big part of the problem.

ryebot

@Joopson Ah, since I'd given painter a high-poly to look at, I thought that was just as good as baking normals from the high-poly. I didn't think about baking the map on the high-poly itself. I did a quick test from a normal map generated from Zbrush (though it traced too shallow so I couldn't just use that map) and the seams were pretty much gone.

I'm having trouble baking a normal map from my high-poly mesh in Painter. Baking a map from a high-poly obj just gives me a blank map. I guess that's why Zbrush asks me to go to my lowest sub-div before I generate a normal map.

Guess I have some more research to do.

ryebot

Wait, but to calculate normals, there needs to be a difference between two surfaces.

So wouldn't baking at a higher resolution result in a map with fewer differences, and thus lower quality? I'm a really confused newbie here.

pior

Check my posts/explanations in this thread: http://polycount.com/discussion/174725/smoothing-groups-and-substance-painter-normal-map-baking#latest

Good luck !

demilich

Your issue with your seams can be a number of things. I think it might be your tangents and bi-normals not matching up correctly. Make sure the lowpoly model you used to bake with is the same model you are applying the normal map to in substance painter. As a test, you should re-export your lowpoly from Maya or whatever 3d package you use and as fbx with tangents and bi-normals checked and I believe Smoothing groups checked. Also make sure the fbx export is older than 2016. I think there are issues with fbx 2016 where Substance Designer/Painter will not import the model correctly. Use that export mesh to rebake and apply your normal map on that. Also in Substance Painter make sure that the project configuration is set to the correct normal map format that you baked it to. OpenGL or Direct X. It is found in the Edit tab.

ryebot

Thanks, @pior! That was a good read. I went ahead and downloaded Xnormal (it's a little bit scarier to me than SP) and was able to reproduce the errors with its baker. At least I'm getting consistency!

So basically, if I setup everything correctly, I shouldn't be able to notice any seams in my normal maps?

ryebot

demilich said:

 Also make sure the fbx export is older than 2016. I think there are issues with fbx 2016 where Substance Designer/Painter will not import the model correctly.

Hmm! I'm actually using Maya 2016, so maybe that's the issue. Though I was able to reproduce the same problems through baking with XNormal (though the model was the same fbx I used in substance). Maya 2016 is all I've got though, so I might have to just use OBJs or something if that's the case.

demilich

Its not Maya 2016, just the 2016 fbx exporter. You could select an older year. Also make sure the fbx exporter does not have Split per-vertex Normals checked as that will cause hard seams on your lowpoly.

musashidan

Try these exact steps and you should output a perfect bake from SP:

The usual basics of good uvs, triangulated mesh, high-low matching closely.

Apply averaged smoothing to your model in Maya(in Max this would be a single smoothing group )

Export .fbx with smoothing checked and tangents/binormals UNCHECKED

Import to SP and make sure to check 'compute tangent space per fragment'

Set ray-distance to between .01 and .03

AA to 4x

Bake

The whole idea behind a 'synced' workflow is that you allow the baker/destination engine to compute the tangents/bi-normals as they both use the same tangent basis(mikkt in most modern cases)

So you don't want to export them from the .fbx file from Max or Maya as they don't use the same tangent basis. That is why you leave it unchecked. Xnormal also uses mikkt now by default but it is also OpenGL by default, whereas SP is DirectX by default. Not a huge deal, just a swizzle flip.

EarthQuake

What is the context of this area? Is it something the user will look at this close?

These sort of seams are typically visible on nearly any model if you zoom in close enough, but generally not visible in real world scenarios.

ryebot

Thank you, @musashidan! as well as everyone for all of their help up to this point.

When we talk about smoothing the model in Maya, are we talking about setting soft edges on the whole model, or averaging normals and averaging vertices also? (I experimented with these options, but averaging normals made my mesh look like a scrambled egg that wouldn't bake and averaging vertices changed the topology).

Okay, I followed those steps (I was originally setting my ray distance much lower because of a problem area on the mesh, .005, so this introduces new problems which I'll have to fix later) with a flat color on the model, the area looks better than it did before... problem solved, I think? I went ahead and rebaked the rest of my textures.

But, when I put my paint back on the model, the seams are still very evident, as well as differences in height/cavity right along the seams. I'm using triplanar projection every place it is possible to in my masks and effects.





It's still visible if you just look at flat color, but in the end, my object isn't just going to be a flat color, so what's the point of it only looking good there? Or is this the best I can expect?



So let's back up, if I'm following the baking procedure correctly, then the problem has to be with my low-poly/highpoly differences and my UV map, right? I actually did retopo on this mesh halfway though my process and used it as the base for all of my sculpting in Zbrush.

We haven't talked much about UVS, here's what mine look like for the model. Would love to know if they could be improved!



Now, I'm probably gonna go ahead and rearrange my UVs again, but I'm worried I'm just going to encounter the same problems in other areas. I'm also happy to share my model with anyone willing to take a look at it.

I hope I'm not coming off as ungrateful or demanding. I'm just feeling really frustrated. I've been stuck on this step longer than I'd care to admit, and how would I ever hope to become a 3D artist if something so simple is such an obstacle for me? In the end, like @Joopson said, maybe I should just ignore and move on, but even then it's going to bug me like hell. If better results are possible, I want to achieve them. Thanks again everyone for your kindness in helping a stranger on the internet.

ryebot

EarthQuake said:

What is the context of this area? Is it something the user will look at this close?

These sort of seams are typically visible on nearly any model if you zoom in close enough, but generally not visible in real world scenarios.

Okay, that's really good to know! Maybe my expectations are just wildly out of sync with what is actually possible.

Most of the seams aren't a big deal, but the seam on the lateral fins are a bit visible. Instead of cutting the mesh from front to back, maybe I'll cut it along the sides so this geometry can flow together on the UV.



By the way, thank you for all your other threads on this topic @EarthQuake, they've been a big help to my learning.

musashidan

Definitely, more work is needed on the UVs.

can you upload the low/high poly? I'd like to take a look.

ryebot

@musashidan Sure thing. I PMed you download links. I have changed the UVs to minimize the issue above, but I'm sure it can still be a lot better. My UV process is basically using 3D coat to make quick cuts > unwrap in 3d coat > evenly scale tiles > fine tune in maya.

BrutalKiwi

Hello! I'm newbie with substance and faced similar trouble with seams on my normal map.
It displayed with terrible lightning artifacts either in viewport and render


For me solution was just changing normal map format from DirectX to OpenGL . (seems simple now )

Also , if you place your normal map in diffuse channel - its normal that you see seams. 
Normal map was created with Zbrush , model exported from Maya.

However , I didn't found smart way to export model with right smoothing from Maya (in maya I use CatClark smoothing method so it looks fine in render with low poly grid) and best solution I found its smoothing mesh and softening edges in maya and then exporting (fbx 2018 , Smoothing group - checked ). If some one know how to do this with low poly  mesh I'd be highly appreciated since my computer not such powerful to work with high poly models in substance =(

Hope that's help !)

BrantAugust

Hi,


I just found out that changing the normal maps mode from "auto" to "sRGB" in Substance painter fixed the seams on my Spiderman Model.  

mahenz24

Try this method, It worked for me. 
1.Apply soft edges to your low obj model in maya
2. In substance > Edit>Project Configuration> bring your new obj make sure you use OpenGl Tick and Tangent tick. 
3. Bake again your Normal map with High res. 
4. Make sure you view you 3d space Texture sets in full res. Like 2k or 4 k 

let me now if it worked for you. Cheers  

poopipe

BrantAugust said:

Hi,


I just found out that changing the normal maps mode from "auto" to "sRGB" in Substance painter fixed the seams on my Spiderman Model.  

That suggests you've done something truly awful to your normal map. 

What happened to it before it reached painter?  (£20 on photoshop being involved)