Home Technical Talk

Baking Artifacts (90 degrees on curved edges) - Help me understand proper high-low modelling

This is an issue that has plagued me ever since I first started modelling, and after seeing it rear its ugly head again, I've decided to finally tackle it seriously and hopefully put this issue to rest once and for all. A long time ago I thought Substance Painter was just bad at baking high/low normals and would manually "fix" the normal maps by painting over the artifacts (while it does get rid of them, it also means I'm erasing the details brought in by the high poly mesh, rendering it all pointless). After baking the same meshes in Blender, I've confirmed that I'm doing something wrong on a fundamental level. Either my methods are flawed, or my small monkey brain has the wrong understanding of something important.

Images of the problem:

[I've given them all a reflective metal materials to better highlight the artifacts]
MESH 1 - The most common problem is this ugly warping/dents/artifacts along rounded edges. Straight lines and edges are passable when baked

MESH 2 - This one makes it apparent that its copying the shape of the high poly from the direction of the face, but anything that goes past that is just black void. The bevelled cylinder on the bottom doesn't have these artifacts, but I think I also used a sharper, more defined bevel for it (or could it be because it has an angle instead of sitting at 90 degrees like the problem cylinder above it?)

What I know about the issue:

- If it appears, it always appears on edges, and usually round/cylindrical ones.
- I can hide it well enough as long as I'm not using a reflective material like metal. It becomes very apparent with reflective metal materials and sticks out like a sore thumb
- Bumping up the image/map size from 2k to 4k only reduces the jaggedness of the artifacts, but does not get rid of them, same deal with anti-aliasing
- Playing around with dilation width and max/min frontal ray tracing seems to have no effect on these specific artifacts. Cage doesn't fix the issue either (that or I did something wrong with the cage)

Theory - The high and low poly meshes don't match closely enough

Inspecting a lot of the cylindrical/round shapes makes me believe this might be it, but I thought the ray tracing is supposed to handle this difference?
This is MESH 1's high and low poly, seen from side view. The outer mesh is the low poly, and the high poly is outlined in orange. I believe it shrunk because I used a subdivision surface modifier on it and didn't add enough reinforcing edge loops (The more I added, the sharper it got, and I wanted a smoother edge for the high poly).




I've read the popular thread "Understanding averaged normals and ray projection/Who put waviness in my normal map?"...
...what I'm getting from the post is that my low and high poly cylinders don't match closely enough.
  1. Is adding more definition/edges/bevels to a low poly cylinder the only way to fix this? I thought the ray tracing part high-low poly baking would let me get away with less defined cylinders, but am I just shooting myself in the foot for trying to cheap out on extra polys?
  2. If so, is it even possible to keep a low poly "low poly" if I need to add extra geometry for cylindrical/round objects just to avoid this? Cylinders seem to add a lot of polys when you want them round.
  3. On sharp edges and bevels, I've seen a lot of models where the low poly has a sharp edge with no bevel, but the texture shows that a wider smoother bevel was baked into it, rather than a smaller more defined one. Is there a trick to this or am I already doing it properly?
If you know what problem I'm running into, I could use some help understanding what I'm doing wrong, or point me towards some links/helpful reading that explains this or what I'm doing wrong. I can provide the .fbx or .blend files if needed.

Apologies if this post is formatted poorly or if I'm missing a very obvious answer, I've spent nearly all day trying to stamp out the problem and my brain is feeling fried right now.

Replies

  • JackofSpades261
    Looking back again at the 2nd image made me think that maybe 90 degree angles are causing this somehow (at least for curved/rounded edges, anyway. Straight lines seem to have no issue with 90 degree angles). Going off of this idea I tried a wide bevel around the outside corner of the low poly (and build a corresponding high poly to fit the modified mesh)
    It's definitely better and the artifacts aren't as rampant, so I tried a smaller bevel this time:

    ...and now the artifacts are arguably worse here than the original 90 degree mesh. The wide bevelling works, but throwing a wide bevel on every sharp edge I have isn't feasible. Back to the drawing board. The more I look into this, the more it sounds like I have to make my low poly meshes less "low poly". Maybe I'll look at some Sketchfab models for clues.
  • gnoop
    Offline / Send Message
    gnoop polycounter
    Just bake  bevel ( rounding corners) shader  on your low poly without hi poly.   Cylindrical shapes would always give you those artifacts and  you have a choice of having them that way . it sometimes could be preferable  even with such "ray missing target" artifacts.    Or having what bevel shader does.  A low poly circle,  angled but with bevels .      And those dents you showed in your first screen  are usually a result of having not enough resolution and padding.

    You can do  low to hi bake then  bevel shader bake and  mix them together  depending n what you prefer for any exact place
  • JackofSpades261
    Thanks for the advice gnoop!
    Went and looked into Blender's bevel shader, baked it into a normal map, then imported said map to Substance Painter:
    It's not perfect, but it's a hell of a lot cleaner than the others. I could hide whats left of the artifacts with dirt or grunge.
    Although the artifacts become more apparent with smaller bevels, here's the normal map with half the bevel size of the above image.
    However, I did take a look at the padding as well, and tried baking the original mesh in Substance again, but with quite a bit of extra padding in the UV map, and it may be the cleanest one yet:

    2k map on the left, and 4k map on the right.
    [Anti-aliasing at 4x4 and both front and back ray trace distance set to 0.001]
    While it didn't fix it completely, padding did make a big difference here. I'll just have to go with this for now, but I don't intend to stop searching

    Unless a perfect solution exists out there, I'll just have to keep these in mind when I'm modelling in the future:
    • Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.
    • Ensure all UV's have enough padding (will have to look into how much is really needed)
    • Either reduce amount of resolution on high poly cylinders, or don't skimp on resolution of low poly cylinders (unless there's a trick to this)
    In this specific case it seems to be a padding + resolution problem for me, but I'll welcome any additional advice/opinions on this topic though if anyone wants to chime in. I could have sworn I've seen models where people get away with sharp 90 degree angles with nice bevelled normal maps and I would like to know how to do so as well (or my memory is bad and there's nothing like that, went looking at models on Sketchfab to look at examples. A lot of "hard edges" had a bevel, the few 90 degree edges I could find were on pieces that didn't matter or weren't metallic/reflective, so any normal artifacts that would have shown up could be easily hidden with roughness, grunge, or disguised as intentional dents)
  • Kanni3d
    Offline / Send Message
    Kanni3d ngon master

    • Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.

    Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

    Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.

  • poopipe
    Offline / Send Message
    poopipe veteran polycounter
    simple solution (will work on 90% of cylinder-ish meshes)

    bake the top of the cylinder to a flat surface
    don't bake to the sides of the lowpoly

    basically, dont bake onto a cylinder

    thank me later
  • kio
    Offline / Send Message
    kio polycounter lvl 14
    @FrankPolygon now this is a very nice in depth resource on this topic! Thanks for writing this down - I have seen all of these issues& solutions over the years so it's cool to see it all nicely broken down :)
  • JackofSpades261
    @FrankPolygon A sincere thank you for the detailed and comprehensive explanation! I knew something was wrong, but didn't realize it was multiple problems all happening at once. I believe you've saved me a lifetime's worth of trouble on this topic since it's kind of difficult to google if you don't know what the exact problem is.

    I'll likely be referring back to this post from time to time until I've committed this information to memory.
  • Neox
    Online / Send Message
    Neox insane polycounter
    Kanni3d said:

    • Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.

    Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

    Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.


    heh sounds like a dream to me, many productions avoid them like the plague. but not for the reasons you would assume.
    it usually comes down to, how much content needs to be pushed and how many people with how many dccs work on a projects. if the answer is many on all of these, usually hard edges get ditched because all soft works completely software agnostic. while hard edges/smoothinggroups/locked normals are very software specific.

    and yes, it is painful and yes it eats just as much, if not more verts to compensate
  • Kanni3d
    Offline / Send Message
    Kanni3d ngon master
    Neox said:
    Kanni3d said:

    • Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.

    Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

    Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.


    heh sounds like a dream to me, many productions avoid them like the plague. but not for the reasons you would assume.
    it usually comes down to, how much content needs to be pushed and how many people with how many dccs work on a projects. if the answer is many on all of these, usually hard edges get ditched because all soft works completely software agnostic. while hard edges/smoothinggroups/locked normals are very software specific.

    and yes, it is painful and yes it eats just as much, if not more verts to compensate

    sounds like a royal pain haha, but yeah i understand that's the pipeline/workflow you must take when handling tons of staff (freelancers?) using various dcc's.
Sign In or Register to comment.