Normal baking and edge distortion
interpolator
I'm trying to bake out my maps in Substance Painter, and cannot solve an issue I'm having with the edges in certain areas being warped/bent.
Some areas, such as the buckle, are perfectly fine. Other areas, such as the edges of the belt and corners of the top and bottom, are warped.
I used to run into this problem when the vertex normal smoothing was off on the lowpoly, but that shouldn't be the case here. I've smoothed everything, and split many of the hard angled areas into separate meshes entirely for the sake of using "Bake by Mesh Name".
I experimented quite a bit with setting various edges as soft or hard, and I should have plenty of padding between UV islands. Nothing I do will correct this, though.
I'm using the basic bake settings; I haven't changed anything except to apply "Bake by Mesh Name". Tweaking these at random improved nothing anyway.
Interestingly, when I initially did the bake inside Modo, the result looked fine, at least inside Modo. The reason I'm now baking the basic map in Painter instead is that, after importing it into Painter, it looked fairly botched up, with the lighting pointing in the wrong directions and harsh UV seams (inverting the green channel didn't fix it).
This is the normal imported from Modo. As you can see the edge warping is not there, but the lighting is off with harsh UV seams.
The Modo-baked map seems to look basically fine in Marmoset Toolbag as well, but my intention was to add additional detail to it in Painter, so obviously it needs to work properly in Painter.
One assumption I had was that, perhaps, Modo was not exporting the smoothing properly. So I imported the lowpoly into Maya, redid the smoothing, then exported it again from there. But the result was the same.
I've attached the lowpoly and highpoly files I'm working with.
-Edit- I've tried a newer bake from Modo inside Marmoset, and I do actually seem to be having a similar problem. There might be something fundamentally wrong with the lowpoly mesh, but I can't figure out what.
Some areas, such as the buckle, are perfectly fine. Other areas, such as the edges of the belt and corners of the top and bottom, are warped.
I used to run into this problem when the vertex normal smoothing was off on the lowpoly, but that shouldn't be the case here. I've smoothed everything, and split many of the hard angled areas into separate meshes entirely for the sake of using "Bake by Mesh Name".
I experimented quite a bit with setting various edges as soft or hard, and I should have plenty of padding between UV islands. Nothing I do will correct this, though.
I'm using the basic bake settings; I haven't changed anything except to apply "Bake by Mesh Name". Tweaking these at random improved nothing anyway.
Interestingly, when I initially did the bake inside Modo, the result looked fine, at least inside Modo. The reason I'm now baking the basic map in Painter instead is that, after importing it into Painter, it looked fairly botched up, with the lighting pointing in the wrong directions and harsh UV seams (inverting the green channel didn't fix it).
This is the normal imported from Modo. As you can see the edge warping is not there, but the lighting is off with harsh UV seams.
The Modo-baked map seems to look basically fine in Marmoset Toolbag as well, but my intention was to add additional detail to it in Painter, so obviously it needs to work properly in Painter.
One assumption I had was that, perhaps, Modo was not exporting the smoothing properly. So I imported the lowpoly into Maya, redid the smoothing, then exported it again from there. But the result was the same.
I've attached the lowpoly and highpoly files I'm working with.
-Edit- I've tried a newer bake from Modo inside Marmoset, and I do actually seem to be having a similar problem. There might be something fundamentally wrong with the lowpoly mesh, but I can't figure out what.
Replies
So I have no idea how to actually invert the green channel on the baked normal.
As far as the Painter bake, I think the problem is a combination of the way it averages normals, combined with the fact that cages are practically unsupported. The option exists, but it seems to be largely ignored by the bake, and doesn't work with Bake By Name.
That combined means that it probably requires additional geometry to help support the bake.
In both cases, the padding was also tighter than I initially thought.
-Edit- Narrowed the Modo bake problem down further. It uses OpenGL, but I need it to use DirectX, and haven't figured out how to change that.
If I set Painter to use OpenGL instead of DirectX, the normal renders fine. Problem here is, other software like UE4 also uses DirectX.
-Edit- I don't know if there is a way to properly change the Modo bake from OpenGL to DirectX, but I've found that Painter will automatically convert it to DirectX when choosing the UE4 export option anyway, and I can get Painter to properly utilize it as OpenGL, so there's no problem there anymore.
Now that I've solved that particular issue, I can work on figuring out the rest of it.
I have noticed as well that utilizing SubD to create the highpoly results in a lot of shape being lost. Many areas pull in and become too small, regardless of edge creasing.
I think what's going on is that the low poly count not only makes it difficult for the lowpoly to match the rounded curves of the highpoly, but also creates too many sharp angles that cannot reasonably be split within the UV.
I'm going to try upping the base polycount and see how well that helps.
As an interesting aside, everything I've found while searching claims that UV border edges should be hardened, but that invariably creates a distorted result for me. The only way I'm able to get a clean bake is by smoothing everything.
I even understand the theory behind why the hard edges should create a better result. But for some reason, no matter how much padding I use, this results in hard lines within the final normal bake along the hardened edges.
-Edit- On closer examination, it actually seems to have simply made the issues.... much less noticeable. They are still there. Enough to ignore, though I feel like I'm still doing something wrong here.
No amount of getting the high and low to line up, adjusting padding, straightening UVs, hardening/softening normals, will eradicate the issue.
Here you can see the bowing along most of the UV borders. Especially the belts.
I've included the newest iteration of the meshes as well.