Baking Artifacts (90 degrees on curved edges) - Help me understand proper high-low modelling

JackofSpades261

This is an issue that has plagued me ever since I first started modelling, and after seeing it rear its ugly head again, I've decided to finally tackle it seriously and hopefully put this issue to rest once and for all. A long time ago I thought Substance Painter was just bad at baking high/low normals and would manually "fix" the normal maps by painting over the artifacts (while it does get rid of them, it also means I'm erasing the details brought in by the high poly mesh, rendering it all pointless). After baking the same meshes in Blender, I've confirmed that I'm doing something wrong on a fundamental level. Either my methods are flawed, or my small monkey brain has the wrong understanding of something important.

Images of the problem:

[I've given them all a reflective metal materials to better highlight the artifacts]

MESH 1 - The most common problem is this ugly warping/dents/artifacts along rounded edges. Straight lines and edges are passable when baked

MESH 2 - This one makes it apparent that its copying the shape of the high poly from the direction of the face, but anything that goes past that is just black void. The bevelled cylinder on the bottom doesn't have these artifacts, but I think I also used a sharper, more defined bevel for it (or could it be because it has an angle instead of sitting at 90 degrees like the problem cylinder above it?)

What I know about the issue:

- If it appears, it always appears on edges, and usually round/cylindrical ones.

- I can hide it well enough as long as I'm not using a reflective material like metal. It becomes very apparent with reflective metal materials and sticks out like a sore thumb

- Bumping up the image/map size from 2k to 4k only reduces the jaggedness of the artifacts, but does not get rid of them, same deal with anti-aliasing

- Playing around with dilation width and max/min frontal ray tracing seems to have no effect on these specific artifacts. Cage doesn't fix the issue either (that or I did something wrong with the cage)

Theory - The high and low poly meshes don't match closely enough

Inspecting a lot of the cylindrical/round shapes makes me believe this might be it, but I thought the ray tracing is supposed to handle this difference?

This is MESH 1's high and low poly, seen from side view. The outer mesh is the low poly, and the high poly is outlined in orange. I believe it shrunk because I used a subdivision surface modifier on it and didn't add enough reinforcing edge loops (The more I added, the sharper it got, and I wanted a smoother edge for the high poly).

I've read the popular thread "Understanding averaged normals and ray projection/Who put waviness in my normal map?"...

https://polycount.com/discussion/81154/understanding-averaged-normals-and-ray-projection-who-put-waviness-in-my-normal-map/p1

...what I'm getting from the post is that my low and high poly cylinders don't match closely enough.

1. Is adding more definition/edges/bevels to a low poly cylinder the only way to fix this? I thought the ray tracing part high-low poly baking would let me get away with less defined cylinders, but am I just shooting myself in the foot for trying to cheap out on extra polys?
   
2. If so, is it even possible to keep a low poly "low poly" if I need to add extra geometry for cylindrical/round objects just to avoid this? Cylinders seem to add a lot of polys when you want them round.
3. On sharp edges and bevels, I've seen a lot of models where the low poly has a sharp edge with no bevel, but the texture shows that a wider smoother bevel was baked into it, rather than a smaller more defined one. Is there a trick to this or am I already doing it properly?
   

If you know what problem I'm running into, I could use some help understanding what I'm doing wrong, or point me towards some links/helpful reading that explains this or what I'm doing wrong. I can provide the .fbx or .blend files if needed.

Apologies if this post is formatted poorly or if I'm missing a very obvious answer, I've spent nearly all day trying to stamp out the problem and my brain is feeling fried right now.

JackofSpades261

Looking back again at the 2nd image made me think that maybe 90 degree angles are causing this somehow (at least for curved/rounded edges, anyway. Straight lines seem to have no issue with 90 degree angles). Going off of this idea I tried a wide bevel around the outside corner of the low poly (and build a corresponding high poly to fit the modified mesh)

It's definitely better and the artifacts aren't as rampant, so I tried a smaller bevel this time:

...and now the artifacts are arguably worse here than the original 90 degree mesh. The wide bevelling works, but throwing a wide bevel on every sharp edge I have isn't feasible. Back to the drawing board. The more I look into this, the more it sounds like I have to make my low poly meshes less "low poly". Maybe I'll look at some Sketchfab models for clues.

gnoop

Just bake  bevel ( rounding corners) shader  on your low poly without hi poly.   Cylindrical shapes would always give you those artifacts and  you have a choice of having them that way . it sometimes could be preferable  even with such "ray missing target" artifacts.    Or having what bevel shader does.  A low poly circle,  angled but with bevels .      And those dents you showed in your first screen  are usually a result of having not enough resolution and padding.

You can do  low to hi bake then  bevel shader bake and  mix them together  depending n what you prefer for any exact place

JackofSpades261

Thanks for the advice gnoop!

Went and looked into Blender's bevel shader, baked it into a normal map, then imported said map to Substance Painter:

It's not perfect, but it's a hell of a lot cleaner than the others. I could hide whats left of the artifacts with dirt or grunge.

Although the artifacts become more apparent with smaller bevels, here's the normal map with half the bevel size of the above image.

However, I did take a look at the padding as well, and tried baking the original mesh in Substance again, but with quite a bit of extra padding in the UV map, and it may be the cleanest one yet:

2k map on the left, and 4k map on the right.

[Anti-aliasing at 4x4 and both front and back ray trace distance set to 0.001]

While it didn't fix it completely, padding did make a big difference here. I'll just have to go with this for now, but I don't intend to stop searching

Unless a perfect solution exists out there, I'll just have to keep these in mind when I'm modelling in the future:

• Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.
  
• Ensure all UV's have enough padding (will have to look into how much is really needed)
• Either reduce amount of resolution on high poly cylinders, or don't skimp on resolution of low poly cylinders (unless there's a trick to this)

In this specific case it seems to be a padding + resolution problem for me, but I'll welcome any additional advice/opinions on this topic though if anyone wants to chime in. I could have sworn I've seen models where people get away with sharp 90 degree angles with nice bevelled normal maps and I would like to know how to do so as well (or my memory is bad and there's nothing like that, went looking at models on Sketchfab to look at examples. A lot of "hard edges" had a bevel, the few 90 degree edges I could find were on pieces that didn't matter or weren't metallic/reflective, so any normal artifacts that would have shown up could be easily hidden with roughness, grunge, or disguised as intentional dents)

Kanni3d

• Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.
  

Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.

poopipe

simple solution (will work on 90% of cylinder-ish meshes)

bake the top of the cylinder to a flat surface
don't bake to the sides of the lowpoly

basically, dont bake onto a cylinder

thank me later

FrankPolygon

There's at least three separate types of baking artifacts present in each of the examples above:

• Pixelation / jagged aliasing in the normal textures.
  
• Bright highlights around UV seams and hard edges.
• Dark shadows around the inner and outer edges of shapes.
• Wavy edges around around the outsides of the cylinders.

When trying to resolve multiple issues it's important to isolate the variables first then troubleshoot each one independently. Breaking down a complex problem like this will make it a lot easier to see exactly what effect each change has on the results. Which also makes it a lot easier to understand exactly what's causing each problem and what the options are for resolving each type of baking artifact.

Pixelation artifacts are generally going to be textel density issues and are often caused by low resolution textures or improper scaling of the UV islands in the low poly mesh. Insufficient textel density tends to produce normal textures that have a jagged appearance and can also cause sampling errors where the details in the high poly model are smaller than a single pixel in the normal texture's UV layout.

Textel density can be adjusted by increasing the size of the texture sheet, decreasing the size of the object or prioritizing [increasing / decreasing] the scale of specific UV islands. Every project is different so the appropriate textel density will depend on a variety of different factors like art style, object size, player view distance, platform specific resource constraints, etc.

https://polycount.com/discussion/227582/how-3d-model-fidelity-is-usually-measured

https://polycount.com/discussion/217178/texel-density-texture-map-resolutions-reference-sheet

Below is an example that shows how increasing the textel density helps resolve pixelation issues and increases the accuracy of the normal information stored in the pixels around the UV seams. One of the simplest ways to increase the textel density is to increase the size of the normal texture.

Doubling the texture size from 128px² to 256px² resolves the worst of the pixelation issues. The higher pixel count in the textures provides more space to store baked normal information. Which provides an increased level of accuracy that helps reduce the visibility of the split in the shading that's caused by the hard edges. Doubling the texture size again to 512px² resolves practically all of the pixelation and is accurate enough to hide any noticeable seams at this view distance.

It is worth noting that, while increasing the texture size may work on smaller assets, a 4x increase in textel density would be much more impractical to use on larger assets or when starting at higher texture resolutions. After a model reaches a certain size or level of complexity it may be necessary to move away from unique UV unwraps on single texture sheets and instead rely on alternate texturing methods like tiling textures and trim sheets.

Bright highlights around hard edges [including split mesh components] tend to be ray misses that generate extreme normal values on either side of a given edge. These types of ray misses are generally caused by a break in the surface of the target mesh and are exacerbated by insufficient UV padding on either side of the split edges. It's also possible to create noticeable seams by switching the baking settings from an averaged (smoothed) cage to an explicit (unsmoothed) cage.

Two common root causes for this type of artifact are having hard edges that aren't backed up with UV splits and packing the UVs or baking the normals without sufficient padding.

Hard edges that don't correspond to any UV splits essentially create a zero padding situation between the [new] UV islands and the normal data from both sides of the split edge bleeds over into the adjacent UV island. This can be resolved by either removing the hard edge or backing it up with a UV split and re-packing the UV islands so there's sufficent room for padding.

https://polycount.com/discussion/comment/2727831/#Comment_2727831

https://polycount.com/discussion/comment/2748332/#Comment_2748332

If all of the low poly mesh splits have their corresponding UV splits and assuming there's sufficient textel density for a given view distance than any remaining artifacts of this type are likely [but not always] caused by a lack of padding or the way the baking algorithm handles calculating normal data around split edges.

It's also worth noting that there's a limit to the accuracy of the normal information baked around a split edge. So if the textures MIP down or if the object is extremely large or viewed at an extreme closeup it is possible for the seams in the normal map to re-appear. Using larger textures to increase the accuracy of the baked normals has a fairly steep drop off in efficiency after a certain size. So it often makes sense to use some additional geometry in place of normal textures on extremely large objects or objects that will be viewed in extreme close ups.

Below is an example of how certain bake settings can produce visible or partially visible seam lines around hard edges.

In the first column the bake settings are set to explicit normals with no padding. With explicit normals the baking cage is split and this means some of the rays are lost when they project through the gaps in the mesh. This tends to produce splits in the baked normals that looks very similar to a zero padding situation where a hard edge doesn't have a corresponding UV split.

Without any padding the normal data is only deposited [mostly] inside of the UV islands. If the UV alignment isn't pixel perfect then this can create minor artifacts around the edges since there's no usable normal information in the adjacent pixels.

In the second column the bake settings are set to averaged normals with no padding. With averaged normals the baking cage is a continuous surface and this means most of the rays are captured since there's no gaps in the mesh. This produces a fairly clean result but since there's no padding and the UV islands aren't perfectly aligned on the pixels in the normal texture there's a few spotty areas around the seam where there's missing normal data.

In the third column the bake settings are set to averaged normals with maximum padding. This produces the cleanest result since most of the rays are captured by the continuous mesh and the outer edges of the UV islands are padded with extrapolated normal data that helps reduce the visibility of the seams.

There does tend to be some application specific differences in the quality of padding but if the low poly mesh and UV layout is setup correctly then most applications should produce usable results with reasonable texture sizes.

The dark shadows around the edges of the shapes are horizon occlusion that's suppressing potential light leaks from whatever normal vectors are assumed to be backwards facing or obscured when rendered from the camera's current position. This is generally caused by having areas where there's a significant difference between the shape of the high poly and low poly models. It can also be caused by ray misses where the high poly falls outside of the low poly since those normal vectors will be recorded as always being backwards facing.

https://polycount.com/discussion/comment/2724629#Comment_2724629

Below is an example that illustrates how normal data that falls within an area that wouldn't normally be visible on the high poly is rendered as a back outline. Adding some bevels to the low poly can be a quick fix but there are certain situations where it makes more sense to further simplify the low poly and abstract some of the shapes in the high poly to produce better baking results for assets that have stricter resource constraints.

Waviness in the normal texture also tends to be caused by shape differences between the high poly and low poly models. This is more of a geometry [ shape] issue through: the larger segments in the low poly circles tend to distort whatever normal vectors they catch. So increasing the segment counts of the curved surfaces is generally going to be more effective than beveling when it comes to resolving waviness.

It's also worth noting that there's certain situations where UV distortion and specific triangulation patterns will tend to produce waviness or skewing in the normal textures. Adding geometry also solve these types of distortion but sometimes it's worth re-triangulating the area with a different edge pattern or straightening / relaxing the UVs to correct the issues without having to use additional geometry.

https://polycount.com/discussion/comment/2724383/#Comment_2724383

Here's an example that shows how the correlation between the shape overlap of the high poly and low poly can be used to explain both the over occluded areas and the wavy details in the normal textures. Beveling the edges of the low poly helps it match the rounded edges of the high poly and increasing the number of segments in the low poly cylinders will help it follow the curvature of the high poly.

It's generally going to be acceptable to use application specific workarounds, like the rounded edge shader, when baking normal textures. However there are certain considerations that need to be addressed when working as part of a team or when shipping the asset off for another artist to work with.

When using these types of workarounds it's important to make sure all the artists who might need to re-bake the normals have access to the same software and that all of the settings used to generate the initial bake are well documented and included in the project files during the hand off. That way the next artist in the production chain who needs to re-bake something won't have to figure out what's missing and what was done before the files were sent over.

Painting out bake errors might be fine when working alone on something that will only ever be used once but there's few things worse than being sent an asset package that's missing the files and documentation needed to get clean bakes on the first go. Often the artist who did the previous work won't remember everything that was done or won't be there to explain it. So this kind of thing can end up creating a lot of unnecessary work in the long run.

Even when working alone it's foolish to count on being able to remember exactly what was done to the bakes six months or a year ago when the asset suddenly needs to be reworked. Massive waste of time and massively frustrating. More often then not it's better to put in the effort to get things working correctly.

Resolving the artifacts and optimizing the normal bakes really comes down to looking at the model's intended use and the projects overall constraints and designing things around those factors. The textel density in the original examples seems to be very low. Which kind of suggests that either these objects are very large or the texture size is a limiting factor.

If the texture size is the limiting factor then it may make more sense to simplify the low poly model by removing the through hole geometry and prioritize the textel density of the cylinder's face by scaling up the UV island to use up all the texture space freed up by deleting the hole geometry.

If the object is very large then it might make sense to look at either increasing the number of texture sheets or looking at an alternate texturing workflow like tiling textures and trim sheets.

If the texture size isn't a limiting factor and the small holes are an important detail that will be viewed up close then it probably makes sense to increase the textel density and more closely match the low poly geometry to the high poly geometry.

Here's an example that compares simplifying the low poly model while using the same texture size to increasing both the geometry density and texture size. Left to right: simplified geometry, current geometry, denser geometry. Texture previews are shown in 1:1 proportion to each other. Prioritizing the face of the simplified cylinder over it's sides does resolve most of the pixelation artifacts but sacrifices the visible depth of the holes in the original model. Increasing the geometry density and texture size will improve the quality but also significantly increase the resource costs when compared to the original model.

Optimization is also about making tradeoffs. Which approach makes the most sense for any particular model really depends on each project's specific goals and technical constraints. This why it's important to take the time to break down complex problems, make samples and evaluate the different results. All of the examples below use the same high poly model, 512px² texture size and averaged normals.

Here's a basic low poly model with 90° corners and hard edges that are backed up with UV splits. The only noticeable artifacts are the occluded areas and the waviness around the outer edges of the shapes.

Here's the same basic low poly model with 45° chamfers on the corners and hard edges that aren't backed up with UV splits. The artifacts in the occluded areas are mostly resolved and some of the waviness is less noticeable but there's a strong highlight around the edge splits. Using bevels with hard edges does increase the amount of geometry when compared to the first sample.

Here's the same basic low poly model with 45° chamfers on the corners and hard edges that are backed up with UV splits. The artifacts in the occluded ares are mostly resolved and some of the waviness is less noticeable. Adding the UV seams and padding to hard edges has resolved most of the strong highlights around the edge splits but part of the seams are still visible when viewed up close.

Once an edge has been split by a hard edge or UV seam then it's split. So adding UV seams to the hard edges doesn't increase the amount of geometry beyond what's in the previous example. Adding UV seams and splitting up UV islands does increase the amount of texture space used [wasted] by padding between the individual UV islands.

Here's the same basic low poly model with 45° chamfers on the corners but only the top face of the cylinder has hard edges and UV splits. The artifacts in the occluded ares are mostly resolved and some of the waviness is less noticeable. Using fewer hard edges and UV splits means this model uses slightly less geometry than the previous example.

Removing some of the hard edges and the corresponding UV seams reduced the number of visible smoothing splits but increased the intensity of the normal gradation in these areas. Strong normal gradation can cause visible artifacts under certain conditions but since it doesn't in this example it's passable.

Here's the same basic low poly model with 45° chamfers on the corners but only the bottom edge of the chamfers have hard edges and UV splits. The artifacts in the occluded ares are mostly resolved and some of the waviness is less noticeable. Using fewer hard edges and UV splits means this model uses slightly less geometry than the example with hard edges on both sides of the chamfers.

Moving the hard edges and UV splits to the bottom of the chamfers has created shading issues on the low poly model that require strong gradients in the normals to compensate. Overall textel density is too low and the textures aren't able to hold enough normal data to accurately compensate for the smoothing issues without creating strong linear artifacts around the individual polygons.

Increasing the texture size allows it to capture more information but even at 2048px² some of the linear surface artifacts are still visible around the smaller triangles. Texture size left to right: 512px² 1024px² 2048px²

Using face weighted normals to control the shading issues on the previous example allows for a fairly clean bake with a 512px² texture.

Here's a denser version of the first low poly model with 90° corners and hard edges that are backed up with UV splits. The only noticeable artifacts are the occluded areas and some very minor waviness around the outer edges of the shapes. At this size and view distance there's relatively minor improvements over the previous examples so the increased geometry density would have to be justified by being something that would be viewed up close.

Here's a version of the previous model with 45° chamfers on the corners, hard edges backed up with UV splits on the lower edges and face weighted normals. Overall baking results will be quite clean with the hard edge placed at the bottom or top of the chamfer.

The same observation about the use of bevels apply to the cylinders with more segments so there wasn't any real reason to duplicate all of those examples. Same for low poly meshes with no hard edges [single smoothing group]. There's a lot of different combinations that could be tried. So at this point it really comes down to deciding where the seams and hard edges need to be to optimize the textel density and shading behavior.

Like Eric mentioned: hard edges are essential for controlling shading behavior and normal gradation. There's a lot of different variables and while part of it may be application specific there's a lot of things in the low poly setup that can effect how things look when they're baked. Using hard edges doesn't have to be an all or nothing approach either. Some mild normal gradation is fine. It just comes down to what the resource constraints are and what the baking results look like.

Addendum: Smooth shading the entire mesh [using a single smoothing group] does work consistently across multiple platforms but it also tends to generate strong gradients in the baked normal textures.

This strong normal gradient can cause visible artifacts at lower resolutions. One easy solution is to increase the number of samples and the size of the textures but on models with lower textel densities there's an upper limit to the texture size.

Adding additional edge loops around sharp corners and shape transitions can help even out the smooth shading behavior. Which helps minimize the severity of the normal gradients and the baking artifacts they tend to generate. One downside to this approach is that, depending on how many additional edge loops are added, the total vertex count can be greater than a comparable mesh using hard edges.

The example below shows the results of smooth shading the entire mesh without any additional loops, with loops on one side of the shape transitions and loops on both sides of the shape transitions. With a texture size of 512px² there are normal artifacts present on all three samples but increasing the texture size does resolve most of the issues.

The first two examples in comparison below use smooth shading only and the third example uses hard edges. Even with the benefit of addition edges and face weighted normals the smooth shaded mesh struggles with normal artifacts but the mesh with the hard edges produces clean bakes with minimal normal gradation.

Viewing things from the player's perspective can help inform what parts of the mesh can be optimized and that can help make a decision when selecting specific strategies for reducing poly counts, placing hard edges / UV seams and prioritizing specific UV islands to hit the desired textel density.

Most things are going to be tradeoffs because after a certain point there's going to be diminishing returns. Past that point it no longer makes sense to try and improve things that players wont notice or that don't really affect game play or storytelling. The example below shows how three different strategies start to look pretty similar as the view distance increases. As was also mentioned in the thread, things can be simplified even further.

Which one is right for any specific project really comes down to what the model has to do. There's just certain limitations to be aware of and it takes a bit of practice to really understand how to bring all the different elements together to produce consistent results.

Additional resources:

https://marmoset.co/posts/toolbag-baking-tutorial/

https://substance3d.adobe.com/documentation/bake/substance-bakers-172818436.html

kio

@FrankPolygon now this is a very nice in depth resource on this topic! Thanks for writing this down - I have seen all of these issues& solutions over the years so it's cool to see it all nicely broken down

JackofSpades261

@FrankPolygon A sincere thank you for the detailed and comprehensive explanation! I knew something was wrong, but didn't realize it was multiple problems all happening at once. I believe you've saved me a lifetime's worth of trouble on this topic since it's kind of difficult to google if you don't know what the exact problem is.

I'll likely be referring back to this post from time to time until I've committed this information to memory.

Neox

Kanni3d said:

• Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.
  

Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.

heh sounds like a dream to me, many productions avoid them like the plague. but not for the reasons you would assume.

it usually comes down to, how much content needs to be pushed and how many people with how many dccs work on a projects. if the answer is many on all of these, usually hard edges get ditched because all soft works completely software agnostic. while hard edges/smoothinggroups/locked normals are very software specific.

and yes, it is painful and yes it eats just as much, if not more verts to compensate

Kanni3d

Neox said:

Kanni3d said:

• Avoid 90 degree hard edges if possible, at the very least, on curved/cylindrical objects. Have surfaces at an angle or bevel to break it up.
  

Hard edges should exist, even with the above bevel shader trick. You shouldn't really "avoid" hard edges, especially 90 degree hard edges since that's the absolute upper limit. If you're avoiding 90 degree hard edges, your entire model with be soft/smoothed, and you'll have extremely bad shading - which will then translate into extremely bad gradients in your bakes/normal maps.

Usually 35-45 degrees will receive a hard edge (which also gets a UV split). Simple as that. To remove/alleviate most of your artifacting comes down to your UVs. How well you unwrap them, straighten them, cut them, pack them etc. which we've never seen so far.

heh sounds like a dream to me, many productions avoid them like the plague. but not for the reasons you would assume.

it usually comes down to, how much content needs to be pushed and how many people with how many dccs work on a projects. if the answer is many on all of these, usually hard edges get ditched because all soft works completely software agnostic. while hard edges/smoothinggroups/locked normals are very software specific.

and yes, it is painful and yes it eats just as much, if not more verts to compensate

sounds like a royal pain haha, but yeah i understand that's the pipeline/workflow you must take when handling tons of staff (freelancers?) using various dcc's.