Of interest to most who will read this thread, I wrote an extensive
tutorial that covers this topic and many other common baking issues.
It's geared towards baking in Toolbag, but most of the Basics and Best
Results sections apply universally. Check it out on the Marmoset site: https://www.marmoset.co/posts/toolbag-baking-tutorial/
Toolbag 3's baker has real-time skew map painting, which can alleviate many of the issues described in this thread, so I would recommend using that in most cases. However, this is still a good read if you're looking to understand why skewing occurs, or if you're using a baker that can't account for skewing.
This is really a continuation of the waviness thread
, but I figured i would start a new topic here as I feel this is essential information and a lot of people still struggle with the concept. If you don’t understand what a cage/averaged projection mesh is, be sure to read the waviness thread first.
First off, a normal map baked from plane.
As this is a simple plane, the vertex normals point in the same direction as the face normal, so the detail is projected without any skewing. Its important to talk about vertex normals and face normals separately here, because vertex normals are averaged between the adjacent faces, so even though your polygon is flat, your vertex normals almost certainly are not. Additionally, when I refer to vertex normals, I’m talking about the normals of the projection mesh/cage, which should be averaged and ignore the mesh normals of your lowpoly entirely (if not, you’ll get gaps, read the other thread
if you’re confused at this point).
With a simple cube we can see the skewing problem in all its glory. The vertex normals shoot out at 45 degree angles from the face normal, which means that the ray projection direction is coming from that skewed angle as well. This is the root cause of skewing.
By adding more geometry, we alter the vertex normals. Additional geometry allows us to push the skewed normals closer to the edges of the planar shape, which means the majority of the flat area will have vertex normals pointing in a similar to the face normal. This is the primary cure for dealing with skewed details, making sure the vertex normals match up as well as possible to the face normals.
There are a variety of methods in which we can do this.
- Original mesh, extreme skewing visible
- Tessellation, or “skewmesh”, skewing is removed entirely. Peter Kojesta explains this in more detail here: https://polycount.com/discussion/144715/skewmesh-tutorial
- Beveled edge, minor skewing visible
- Beveled edge with 2 support loops, there is a tiny bit of skewing here as well but it is essentially gone.
- My preferred method, cutting in the minimal amount of geometry to “spot fix” any issues. Skewing is removed entirely here as well.
# 3 and 4 are great if if you have the polygon budget to leave them in the final low poly.
# 2 is handy trick but only helps with certain types of geometry (more on that later) so I don’t consider it a general purpose method. It also requires making a copy of the mesh, tessellating, baking to an object space map, and then finally converting back to tangent space using the original lowpoly, via Handplane or Xnormal.
# 5 is the method I’ve been using for years. My basic workflow is to get my base lowpoly created (while saving room in my poly budget), doing a test bake to check for problems, and then cutting in some extra verts where needed to deal with skewing. Yes, this method adds to the triangle count, but usually not so much that you would blow your budget. Even if you have a super strict budget, you can use the same duplicate, bake to object space and convert to tangent space trick that Peter explains in his thread.
Most people assume skewing is solely an artifact of the bake settings, the low poly, or some combination of the two. This could not be further from the truth. The exact shape of your highpoly element plays a significant role as well.
This is a super common mistake that I see people make, they will model detail meant to be baked to a normal map with huge extrusions or indentations. The further the floating element is from the surface of the lowpoly mesh, the more extreme the skewing problems will be. The important thing to remember here is that we don’t care about depth at all*. What we want is the normal information, that is to say the direction each polygon on the highpoly element faces. A normal map baker can only see direction, it doesn’t care about depth.
Try to image yourself as a camera pointing along the vertex normal, what you see is what the baker will pick up. The more extreme the depth information, the larger the divergence will be between what you want to bake and what the baker actually records.
*Depth can be important for other map types, like AO and displacement maps, but that's a different topic.
Here is the same sets of meshes baked with the modified high poly. While # 3 and # 4 show small improvements, the worst case mesh (# 1) shows drastic change. What was completely unusable before is now borderline okay, simply from reducing the depth of the bolt. # 2 and # 5 remain the same as skewing was removed entirely with the added geometry.
Here is a more realistic use case, we’ve got some skewing problems, and another type of issue that is much less frequently talked about. When you have mis-matches in your highpoly and lowpoly geometry, (in this case clear indentations that are modeled into the high but not into the low) you can get some pretty nasty artifacts as well. Here, from the front or side view, the cylindrical detail set into the indentation doesn’t really come through at all.
Its important to consider how your highpoly model with transfer down to your lowpoly. Ideally, you should either model supporting geometry into the lowpoly for this shape, or model the high in a way that is more favorable to the low.
Here, I modified the shape of the inset cylinder to better match the exterior shape of the lowpoly. This is very important, as any shape mismatches between the high and low will compound skewing artifacts. I also cut in some additional verts in key places to further reduce skewing.
So, why would I cut in geometry manually when I can simply tessellate the mesh as Peter shows in his video? Its simple really, this method is more reliable and easier to control. When you start dealing with complex meshes, many areas will not benefit from tessellating the cage mesh, in fact, you may introduce errors by indiscriminately tessellating. Here is an image I put together a while back to test that method out.
When it comes to shapes that should be round, tessellation does not help, and can actually hurt the end result. I would be very hesitant to use this method with complex assets that contain a variety of different geometry types, for instance hard edged flat surfaces along with cylinders, organic curves, etc.