Normal Map Baking - Hard Edge Problem
polycounter lvl 3
I'm trying to bake a high to low normal map using Substance Painter (I've tried Substance Designer too). Usually I'll use all soft edges on my low poly, but that gives me some severe gradients, especially on this model. So I've been reading up on using hard edges to get a better bake. I'm using Maya, so I mark certain edges as hard, make sure those edges are split UVs, and bake in Substance Painter, but I get the following artifacts.
Does anyone have any insight on this? From what I've read I should be able to use hard edges in a bake without any artifacts. Here's the normal map I get from the bake, let me know if you need to see anything else.
Thanks!
Does anyone have any insight on this? From what I've read I should be able to use hard edges in a bake without any artifacts. Here's the normal map I get from the bake, let me know if you need to see anything else.
Thanks!
Replies
To the OP, try increasing the max frontal and rear search distances for your bake. Also increase the dilation slider to give your map some more padding. If you could post a screenshot of your bake settings it would be helpful.
The weirdest things about this to me are 1: there's no visible artifact on the normal map, and 2: substance designer's renderer doesn't render this artifact, but painter, Unreal and Unity all do.
I fixed the issue before by conforming the low poly better, but I re-created the problem by moving a couple of vertices just a little bit so now it bakes like this again:
Really appreciate the help!
Just as I thought, it is not the artifact in a general sense — the baking works just fine. Actually, what you're getting is a correct result from a technical perspective of baking with such lowpoly/highpoly combination. The question is — why exactly it bakes like this? I've put together a few images to explain it. We will talk about that particular area at the bottom, but area at the right has exactly the same cause, so it will apply to both cases.
Here is the lowpoly with normal map applied as a color map (diffuse). As we can see, in fact this "artifact" is present on the normal map. This kinda greenish area marked with red outline will look on the model as a "flat" normals that we're perceiving as an artifact. But in fact it is not. It's just that normal information containted in that particular area is wrong for us.
But how that wrong normal information actually got there? Let me demonstrate that with a picture. There is a highpoly shown with a red wireframe on it, the lowpoly is a semi-transparent with the same normal map applied in a color channel, and a few cylinders that penetrate the lowpoly and touch the surface of the highpoly.
Now, the cylinders shown actually very roughly represent how raycasting works in this particular area during the bake with averaged projection mesh. The reason of why we're getting this greenish "wrong" normals is just because averaged baking rays that were casted in this area are hitting the inner part of the highpoly mesh (see where ends of the cylinders touch it), and such hits will record corresponding normal information into texture. Around the top cylinder begins an area where bake rays actually start to hit "correct" parts of the highpoly, so we're starting to see this smooth bevel.
In other words: wrong normals are recorded into normal map in some areas because of how bake rays are casted from an averaged projection mesh. They're hitting "wrong" parts of the highpoly because of the angle they were casted, resulting in a visual "artifact" on a surface (which is not, what you're actully seeing there is some other part of the highpoly). In fact, this has very much in common with waviness that are often to be seen on baked cylindrical objects. The underlying reason is the same, explained in great details in this thread: http://polycount.com/discussion/81154/understanding-averaged-normals-and-ray-projection-who-put-waviness-in-my-normal-map
The averaged projection mesh ray distribution is not the same (at least in Substance Painter) when you use all-soft lowpoly and lowpoly with hard edges. I get slightly different bake results in terms of ray direction when using both types of meshes, even with Average Normals option turned ON. However, when using an all-soft mesh, there is no difference in bake between Average Normals = ON and Average Normals = OFF. Which is to be expected, because when there is no split vertex normals, SP has nothing to average and for baking it will use vertex normals imported from the mesh itself. So, It seems that when using lowpoly with hard edges the normals in SP are averaged slightly differently when using the same mesh with no hard edges, and this affects the direction in which baking rays are cast.
But the reason why there is a shading error when using lowpoly with hard edges and no such error when using all-soft lowpoly is not because of ray distribution difference. Well, it MAY get some contribution to the overall bake outcome, but it is hard to account exactly how large it is. The actual reason can be found in the blue channel of the normal map baked to the lowpoly with hard edges.
In the bottom left part of this picture the blue channel of the normal map is presented. We can clearly see that part of the map where the shading issue is — it is the small gray portion of the map inside the UV shell with red arrow pointing from it. If we were to sample this area with an eyedropper in Photoshop, we will find out that its value is RGB: 117,117,117.
To understand why this will cause us trouble we should remember what the blue channel of a normal map actually means. In tangent-space normal map blue channel is used to store information about how polygon is aligned relative to a camera — i.e. completely towards the camera (perpendicular to it) with a value of RGB : 255 (white), or almost parallel to the camera with a value of 128 (middle gray). Generally speaking, all correct geometry configurations should fit inside this range — meaning that there should be no data inside the blue channel below the value of 128.
When value below 128 is recorded, basically it will mean that the polygon is facing away from the camera and we shouldn't see it at all. I guess that when Substance Painter detects this value inside the blue channel of the normal map, it just can't process it correctly (remember, we actually shouldn't see this polygons) and ignores that part of the normal map. Maybe SP and other engines were specifically configured to behave like this — that I don't know for sure.
This is why we're getting this shading error that looks like normals aren't presented in this area at all — a flat, standing out part of the surface. The mystery is — why we're getting such values inside the blue channel of the NM from a mesh with hard edges and not getting them in the same area of NM for a mesh with all-soft edges?
To answer that, we should closely look at the actual surface normals of the meshes themselves and remember one more trait of normal maps. Tangent-space normal maps actually store the difference between the surface normals of a lowpoly mesh and the surface normals of a highpoly mesh. In a practical sense this means that tangent-space normal maps are computed against the surface normals of a lowpoly mesh.
If we take a look at the surface normals of the hard-edged lowpoly mesh at the top left part of the picture, we can see that [because of the hard edges] they are completely flat in the area where our shading issue is. If we look at the blue channel of the normal map that is just below, we can see that the values in this area is also completely flat with an actual value of 255 (white). However, when we're approaching the area with shading issue, we can see that the map values are quickly changing to already mentioned value of RGB: 117.
The important thing is — we're getting there a very low value of 117 because of a drastical difference between flat lowpoly surface normals and the normals that are caught from a highpoly with baking rays. When normals are computed against those flat lowpoly surface normals, the baker "sees" that it should record pretty low values into the blue channel of NM to correctly represent the data that was acquired by the rays.
But why in the first place we're getting such "low" values? Here's the quick scheme that I've put together to illustrate this. Also you can get back to the image with cylinders in my previous post to get another perspective on this.
Basically it is because of how baking rays are projected AND how highpoly was built. As you can see on the scheme, some rays that are projected in the area where our shading issue is will hit highpoly faces that are facing "away" from a "flat" lowpoly surface normals. This will produce some values in the blue channel of the NM that will be below RGB: 128, and result in shading errors that we see.
As for why such errors aren't present on a mesh with all-soft edges — I think it is partly because of how baking rays are projected (slightly differently) and mostly because of averaged surface normals of the lowpoly itself. With all-soft edges mesh, the surface normals around our problematic area already aren't flat - they're averaged because of soft edges. This means that normals will be computed against a less extreme differences between the surface normals of lowpoly mesh and surface normals of highpoly mesh, increasing the probability that values in the blue channel of NM will stay in "correct" 128-255 range.
If we're to take a look at my first picture in this post, in right half of it we would see the situation I've just described. The surface normals of the lowpoly are soft and averaged, and below there is a blue channel from a NM that is less contrast and has less gray values because it was computed against already "soft" surface normals.
So, in the end I think it comes to the highpoly mesh: you should be able to get correct looking bakes both from an all-soft lowpoly and a lowpoly with hard edges if you build your highpoly slightly differently. In this particular example, inside that cheese hole there shouldn't be any faces which normals point away from that "flat" area on the side of the cheese piece. In other words, you shouldn't cut out more that a half of a sphere in places like that. See second picture in this post for an example.
One more thing to keep in mind: highpoly vertex normals also affect the bake results. In Maya, when I've tried to modify your highpoly to get rid of that faces that look in wrong direction, at first I was pretty confused, as the bake was still producing this shading error. It turned out that vertex normals of the highpoly were "locked", so no matter how I changed highpoly shape, its vertex normals still were looking into wrong direction. I had to do Mesh Display -> Unlock Normals to get them aligned with actual geometry and get correct bake result. This means that in general you should be aware of not only your lowpoly vertex normals, but highpoly vertex normals as well.
Well, that's it. Thank you for your question, I think now I understand normal maps better myself. Just hoping I wasn't speaking nonsense with this.
I'm using Max but for some reason the face normals of the low-poly .fbx were locked so even though I applied a single smoothing group
all corners were remaining non-averaged. In Max you can use an Edit Normals modifier manually change the normals so I averaged all the normals. My usual workflow is to always use a single SG and allow SP to compute the tangents/bi-normals(mikkT) , cageless bake with front/rear in SP set to .03 As you can see - apart from the waviness which is obviously due to the low-poly curves having too few edges - the edges are perfect.
I also re-UVed your mesh as I like to work with as few shells as possible: in this case a single shell for test purposes(for a real asset I would maximise the UV space but it doesn't really matter with MikkT.
@musashidan, I was able to achieve a pretty solid bake with an all-soft low poly as well. I guess I just wanted add hard edges/smoothing groups to my baking toolbelt (like this thread) without running into problems in the future. Thanks for taking the time to help me out!