This is an issue that has plagued me ever since I first started modelling, and after seeing it rear its ugly head again, I've decided to finally tackle it seriously and hopefully put this issue to rest once and for all. A long time ago I thought Substance Painter was just bad at baking high/low normals and would manually "fix" the normal maps by painting over the artifacts (while it does get rid of them, it also means I'm erasing the details brought in by the high poly mesh, rendering it all pointless). After baking the same meshes in Blender, I've confirmed that I'm doing something wrong on a fundamental level. Either my methods are flawed, or my small monkey brain has the wrong understanding of something important.
Images of the problem:
[I've given them all a reflective metal materials to better highlight the artifacts]
MESH 1 - The most common problem is this ugly warping/dents/artifacts along rounded edges. Straight lines and edges are passable when baked
MESH 2 - This one makes it apparent that its copying the shape of the high poly from the direction of the face, but anything that goes past that is just black void. The bevelled cylinder on the bottom doesn't have these artifacts, but I think I also used a sharper, more defined bevel for it (or could it be because it has an angle instead of sitting at 90 degrees like the problem cylinder above it?)
What I know about the issue:
- If it appears, it always appears on edges, and usually round/cylindrical ones.
- I can hide it well enough as long as I'm not using a reflective material like metal. It becomes very apparent with reflective metal materials and sticks out like a sore thumb
- Bumping up the image/map size from 2k to 4k only reduces the jaggedness of the artifacts, but does not get rid of them, same deal with anti-aliasing
- Playing around with dilation width and max/min frontal ray tracing seems to have no effect on these specific artifacts. Cage doesn't fix the issue either (that or I did something wrong with the cage)
Theory - The high and low poly meshes don't match closely enough
Inspecting a lot of the cylindrical/round shapes makes me believe this might be it, but I thought the ray tracing is supposed to handle this difference?
This is MESH 1's high and low poly, seen from side view. The outer mesh is the low poly, and the high poly is outlined in orange. I believe it shrunk because I used a subdivision surface modifier on it and didn't add enough reinforcing edge loops (The more I added, the sharper it got, and I wanted a smoother edge for the high poly).
I've read the popular thread "Understanding averaged normals and ray projection/Who put waviness in my normal map?"...
...what I'm getting from the post is that my low and high poly
cylinders don't match closely enough.
- Is adding more definition/edges/bevels to a low poly cylinder the only way to fix this? I thought the ray tracing part high-low poly baking would let me get away with less defined cylinders, but am I just shooting myself in the foot for trying to cheap out on extra polys?
- If so, is it even possible to keep a low poly "low poly" if I need to add extra geometry for cylindrical/round objects just to avoid this? Cylinders seem to add a lot of polys when you want them round.
- On sharp edges and bevels, I've seen a lot of models where the low poly has a sharp edge with no bevel, but the texture shows that a wider smoother bevel was baked into it, rather than a smaller more defined one. Is there a trick to this or am I already doing it properly?
If you know what problem I'm running into, I could use some help understanding what I'm doing wrong, or point me towards some links/helpful reading that explains this or what I'm doing wrong. I can provide the .fbx or .blend files if needed.
Apologies if this post is formatted poorly or if I'm missing a very obvious answer, I've spent nearly all day trying to stamp out the problem and my brain is feeling fried right now.