Visible normal map seams
polycounter lvl 6
Hello, I'm baking normal maps in Painter for quite some time and noticed it creates a seam on the UV island border, it's invisible in Painter's viewport but quite noticeable in a game. To fix this I am using a painted normal layer but maybe there is a better way to do it?
Replies
Hm, could be an issue how the texture is setup in engine (handedness, sRgb)? Which engine do you use?
These seams (in terms of color on the baked map) are absolutely to be expected on a tangent-space normalmap. To understand this better, make a few test bakes with UV islands rotated in the UVs in a few different ways and you'll notice that these colors will change each time. That's because the colors relate to the X and Y axis (or rather, U and V) of the coordinate system being used to encode the relative surface angle orientation for each given pixel. Whereas on a displacement map or a regular texture map (color, roughness ...), nothing would change when rotating UVs, as these type of textures do not depend on the U and V orientation since they respectively only encode Z or a local surface color or value for a given pixel.
The resulting shading is smooth in Painter, because it is reading (and applying) the shading from this texture information consistently ...
...and the resulting shading is not smooth in the game you are making content for, because it expects something slightly different. If you are lucky all you'll have to do is inverting the green channel of your map, as this is the number one cause of inconsistency between games and bakers. But the cause could also be a (very) vast number of other things. Start by overriding all other textures in the game with some default grays in order to have a much clearer view on the shading. If the nipple looks concave rather than of convex then you're in luck, as you'll just have to invert a channel. But imho that's unlikely here, as you would have probably already noticed such an issue by now.
Unfortunately, the solution you are thinking of consisting of intuitively painting over these seams will likely not do what you are expecting. However, if you are really pressed for time, the one hacky thing you can do would be to generously airbrush a neutral normalmap blue over the area, which will basically let the renderer to show the bare lowpoly as opposed to any normalmapped detail. As dirty of a hack as this may seem, it can be quite effective on organic models since the details of the diffuse and roughness will likely conceal the smoothness of the raw lowpoly shading. That's only useful for organics with shallow detail though, and will not work with more complex surfaces.
Of course the best course of action is to first and foremost what kind of norm the game is expecting. If this precise information is not available, then you might have to settle with a compromise.
- - - - -
And it goes without saying : you might want to start by mentioning which game you are attempting to mod.
Thanks for the detailed explanation, after that I started testing with a dummy normal map and found that Painter is innocent here. In-game normal maps don't differ but look like the exporter slightly changes the direction of normals on border edges. And probably because of this the "seam" become visible.
As pior said. The normal map is most likely being baked using a different space to the one the engine is using.
You need to bake in the same space or you will get these artefacts.
If it something as common as unity or unreal then you should be using mikktspace and you need to ensure that tangent and binormal information in the mesh is being passed through fully at every export/import step.
If it's a different engine you need to find some documentation on how to bake correct normal maps for it
@HenryVIII : Adding to the above and based on your screenshot with the purple representation of the normals there seems to be an actual shading seam along this UV edge which has zero reason to be there. So whatever process/exporter caused that didn't just slightly change the normals - it very much broke/split them.
That said, you mention an "exporter". But this screenshot of yours with the split normals shown in purple is obviously taken in an app like Max/Maya, meaning that you've re-imported the model there. And this alone could create such a split, as some barebones model importers can sometimes break surface normals at UV seams. Or perhaps you mean the model "extractor" that allowed you to mod it.
I would highly recommend to be much, much more clear and precise about the steps you are taking, otherwise you may end be only partially adressing your issue or perhaps making it worse.
And, the seam will not become "more visible" because of the split in shading ; it will become visible, period. Because once more, the fact that the RGB colors of the normalmap change across the UV islands is not the issue. You need to completely disregard what you may think from just looking at the normalmap.
- - - -
So again, please try this : take the UV island representing the back of your character, rotate it by a big amount between 90 to 180 degress, and rebake. You'll notice that despite the *even stronger* shift in RGB colors across the UV edge on the tangent space normalmap, the overall look ingame will not change *at all* (and the faint seam will not be any stronger). That's because the shading information is encoded in relation to the U and V axis of the texture space. In other words : if a detail in the surface is encoded by "some amount of green', if you rotate that bit in the UVs by 90degrees and rebake, it will be encoded by "some amount of red". But the end result will be 100% the same shading-wise.
Only once you'll understand this you'll be more willing to completely ignore what you may guess by just looking at the map and you'll be more likely to find the actual root of your issue.
... which is either the model receiving an unwanted hard edge somehow, or a tangent space basis mismatch as mentionned by @poopipe . Perhaps both !
- - - -
Also, another remark : because of the complexity of these processes it also possible that the assets you are working with may be wonky to begin with. For instance, a few years ago there was still a myth going around that convinced people that somehow needed to put hard edges in place where they 100% shouldn't be. Something like that can very much happen in production and cause wonky assets, themselves requiring workarounds to compensate. This is even more likely considering that the lighting in your screenshot seems to indicate a game released around 2010 or so ... So again there might be a need for a compromise there.
Assuming the split is present in the engine version of the model a matching tangent basis would (should) account for the UV split/hard edge (mostly)
if we're looking at around 2010 it's not unlikely that the engine in question would be using a tangent basis matching their primary modelling or baking package
As such it's probably worth running the bake in Maya using native tangent space or Xnormal since they're the most likely candidates to see if you can find a match.
What game is it though? There's a reasonable chance someone on here worked on it or at least on the same engine and would be able to tell you exactly how they baked stuff.
Yeah, this is pre-pbr MTframework with red channel-to-alpha compression trick, I will try to bake it in Xnormal