Home Adobe Substance

Normal baking and edge distortion

interpolator
Offline / Send Message
Zack Maxwell interpolator
I'm trying to bake out my maps in Substance Painter, and cannot solve an issue I'm having with the edges in certain areas being warped/bent.

Some areas, such as the buckle, are perfectly fine. Other areas, such as the edges of the belt and corners of the top and bottom, are warped.
I used to run into this problem when the vertex normal smoothing was off on the lowpoly, but that shouldn't be the case here. I've smoothed everything, and split many of the hard angled areas into separate meshes entirely for the sake of using "Bake by Mesh Name".
I experimented quite a bit with setting various edges as soft or hard, and I should have plenty of padding between UV islands. Nothing I do will correct this, though.
I'm using the basic bake settings; I haven't changed anything except to apply "Bake by Mesh Name". Tweaking these at random improved nothing anyway.

Interestingly, when I initially did the bake inside Modo, the result looked fine, at least inside Modo. The reason I'm now baking the basic map in Painter instead is that, after importing it into Painter, it looked fairly botched up, with the lighting pointing in the wrong directions and harsh UV seams (inverting the green channel didn't fix it).

This is the normal imported from Modo. As you can see the edge warping is not there, but the lighting is off with harsh UV seams.
The Modo-baked map seems to look basically fine in Marmoset Toolbag as well, but my intention was to add additional detail to it in Painter, so obviously it needs to work properly in Painter.

One assumption I had was that, perhaps, Modo was not exporting the smoothing properly. So I imported the lowpoly into Maya, redid the smoothing, then exported it again from there. But the result was the same.

I've attached the lowpoly and highpoly files I'm working with.

-Edit- I've tried a newer bake from Modo inside Marmoset, and I do actually seem to be having a similar problem. There might be something fundamentally wrong with the lowpoly mesh, but I can't figure out what.

Replies

  • FancyMug
    Offline / Send Message
    FancyMug polycounter lvl 7
    This might help you with the warped edges. 
    https://www.youtube.com/watch?v=MnuK6xyi-qY
  • Zack Maxwell
    Offline / Send Message
    Zack Maxwell interpolator
    Thanks for that. I'm getting closer to a solution. In particular, the largest problem I seem to be having with bakes made in Modo is that the green channel is in fact inverted, and the option to change that doesn't actually change the baked normal; just the way Modo itself renders the normal in the material it generates.
    So I have no idea how to actually invert the green channel on the baked normal.
    As far as the Painter bake, I think the problem is a combination of the way it averages normals, combined with the fact that cages are practically unsupported. The option exists, but it seems to be largely ignored by the bake, and doesn't work with Bake By Name.
    That combined means that it probably requires additional geometry to help support the bake.
    In both cases, the padding was also tighter than I initially thought.

    -Edit- Narrowed the Modo bake problem down further. It uses OpenGL, but I need it to use DirectX, and haven't figured out how to change that.
    If I set Painter to use OpenGL instead of DirectX, the normal renders fine. Problem here is, other software like UE4 also uses DirectX.
    -Edit- I don't know if there is a way to properly change the Modo bake from OpenGL to DirectX, but I've found that Painter will automatically convert it to DirectX when choosing the UE4 export option anyway, and I can get Painter to properly utilize it as OpenGL, so there's no problem there anymore.
    Now that I've solved that particular issue, I can work on figuring out the rest of it.
  • Larry
    Offline / Send Message
    Larry interpolator
    Sometimes it is a bug in the viewport, but you can notice it if you zoom in or out. Your low poly cylinder's corners certainly come outside the high poly , so it doesn't bake them. Try to inflate your high poly a little bit, or push inwards the vertices in your low poly. In 3ds max there is a modifier called "push", but i dont know if something equivalent exists in other programs.Also, straighten all the rectangular uv islands you have!! Sometimes some distortion comes from the pixelization of your baking, if your lines are straight, expecially on rectangular uv islands, your bakes will be straight as well. Except from the uv islands i circled, ALL THE REST can/should be straight lines. Ex
  • Zack Maxwell
    Offline / Send Message
    Zack Maxwell interpolator
    I've been looking around, and I think I'm onto something with it needing more geometry. I thought I was doing well on the polycount, but it turns out it matches what you might have seen 10 years ago. Based on what I've found, for a modern game I could easily have 5x the polycount that I do now.
    I have noticed as well that utilizing SubD to create the highpoly results in a lot of shape being lost. Many areas pull in and become too small, regardless of edge creasing.
    I think what's going on is that the low poly count not only makes it difficult for the lowpoly to match the rounded curves of the highpoly, but also creates too many sharp angles that cannot reasonably be split within the UV.
    I'm going to try upping the base polycount and see how well that helps.
  • Zack Maxwell
    Offline / Send Message
    Zack Maxwell interpolator
    Looks like I was correct. Adding additional geometry seems to have fixed everything.
    As an interesting aside, everything I've found while searching claims that UV border edges should be hardened, but that invariably creates a distorted result for me. The only way I'm able to get a clean bake is by smoothing everything.
    I even understand the theory behind why the hard edges should create a better result. But for some reason, no matter how much padding I use, this results in hard lines within the final normal bake along the hardened edges.
    -Edit- On closer examination, it actually seems to have simply made the issues.... much less noticeable. They are still there. Enough to ignore, though I feel like I'm still doing something wrong here.
    No amount of getting the high and low to line up, adjusting padding, straightening UVs, hardening/softening normals, will eradicate the issue.


    Here you can see the bowing along most of the UV borders. Especially the belts.
    I've included the newest iteration of the meshes as well.
  • goekbenjamin
    Offline / Send Message
    goekbenjamin polycounter lvl 5
    This is something i struggled today, do you have any clue now how to solve this issue?
Sign In or Register to comment.