I sculpted this model in ZBrush, baked the normals in Substance Painter (where they look completely fine on the model). I've done this workflow with nearly every object I've ever created and have never had this issue.
I tried switching from DirectX to OpenGL in Substance. I tried inverting the green channel of the normal map in GIMP, but it just seems to make different parts of the map inside out. I have also tried re-baking. When I researched, I could pretty much only find people recommending to invert the green channel. As far as I can tell, the normals of the high & low poly are facing the correct way. I'm not sure what's going on but would love to find out it's super simple to fix
Thanks!
The normal map before inverting green channel.
After inverting the green channel.
Material preview with inverted green channel normal map looks super weird.
It looks fine to me after baking in Substance Painter? But the map itself is still messed up.
Replies
But
If it looks fine in painter where exactly are you having the problem?
You're not describing your problem clearly, as "messed up" doesn't mean anything especially when talking about a texture which is just a mathematical representation of surface information. At the very least you could show the high that the low+bake is supposed to represent.
So, even though I have no idea what the problem is ... my guess is that your are probably plugging this map without setting it to "non-color" information, somewhere.
(Inverting a green channel shouldn't be done as some sort of guesswork - you either know what your renderer/game expects or you dont. You can also ditch the nomenclature "opengl" vs "directx" as this is meaningless even though some applications insist on using these terms. Better just refer to the green channel as "up" or "down" in regards to white being used for normals pointing up or not IMHO)
Also, the pixelation showing up on the edges of your asset seems to indicate that you are making these edges hard on the model, without splitting them off in your UVs. This guarantees that you will get such glitches, as this pretty much means that you are asking your texture to have infinite resolution over these infinitely harsh edge transitions. You need to either set these edges to smooth and do whatever you want with the UVs, or set them as hard and split your UVs accordingly so that the pixels can naturally bleed over the edges of the UV islands. Hard edges + continuous UVs will cause glitches because the "other side" of the texture will always show up on the "wrong side" of the hard edge.
Open GL normals = + Y up