Getting Confused with Normal Mapping [Solved]
polycounter lvl 5
Hi all!
The topic of normal maps seems (pun intended) to come up a lot here, so please excuse me for posting on it again.
I recently became a 3D game art student, and I'm mostly self taught at this point. I gave myself the challenge of once and for all figuring out the normal map and AO baking workflow. I'd like to think I've come a long way from my first try ever, but while working on my most recent asset, I ran into a problem...
Here's the asset with baked normal and AO maps (1k):
And here is the issue:
I've been having this "overly visible seams" issue for about 6 months now, and have tried very hard to find a clear answer as to what causes it. I make seams on all my hard edges, and I make sure there are separate smoothing groups assigned to each UV island, but still the problem persists. I have yet to try a solution that gives me a literally seamless result like I saw here this month: http://polycount.com/discussion/comment/2565971#Comment_2565971
So, I'm hoping that this amazing community can give me some final answers so that I can start making progress towards that level of work.
My questions:
> I do not use a cage, I bake by mesh name in Substance Painter (2.6.1). I have read that this is a bad way to bake maps, but I have also read the opposite. Which is true?
> I have heard that seams always show up in SP's viewport because of the shader used, and is not reliable to use as a baseline. Is this true?
> Is it because I simply haven't added enough geometry to my low-poly?
> WTF is an "averaged" cage?? I've read about them everywhere, but can't seem to find the proper workflow to make one, or what the difference between an "averaged" cage and a standard cage is.
The most recent place I found "averaged" cages mentioned on Polycount:
http://polycount.com/discussion/73593/normal-maps-can-somone-explain-this-black-edge-issue
And my final question, should normals on a mesh look like this before baking textures:
Or like this:
And what is the difference between the two?
I have a feeling that separate smoothing groups for each UV island has something to do with it, but I'm not sure.
UV's for reference, with smoothing groups set for baking:
I REALLY appreciate anyone willing to help me or point me to somewhere these questions have already been answered.
Thanks!
Z
The topic of normal maps seems (pun intended) to come up a lot here, so please excuse me for posting on it again.
I recently became a 3D game art student, and I'm mostly self taught at this point. I gave myself the challenge of once and for all figuring out the normal map and AO baking workflow. I'd like to think I've come a long way from my first try ever, but while working on my most recent asset, I ran into a problem...
Here's the asset with baked normal and AO maps (1k):
And here is the issue:
I've been having this "overly visible seams" issue for about 6 months now, and have tried very hard to find a clear answer as to what causes it. I make seams on all my hard edges, and I make sure there are separate smoothing groups assigned to each UV island, but still the problem persists. I have yet to try a solution that gives me a literally seamless result like I saw here this month: http://polycount.com/discussion/comment/2565971#Comment_2565971
So, I'm hoping that this amazing community can give me some final answers so that I can start making progress towards that level of work.
My questions:
> I do not use a cage, I bake by mesh name in Substance Painter (2.6.1). I have read that this is a bad way to bake maps, but I have also read the opposite. Which is true?
> I have heard that seams always show up in SP's viewport because of the shader used, and is not reliable to use as a baseline. Is this true?
> Is it because I simply haven't added enough geometry to my low-poly?
> WTF is an "averaged" cage?? I've read about them everywhere, but can't seem to find the proper workflow to make one, or what the difference between an "averaged" cage and a standard cage is.
The most recent place I found "averaged" cages mentioned on Polycount:
http://polycount.com/discussion/73593/normal-maps-can-somone-explain-this-black-edge-issue
And my final question, should normals on a mesh look like this before baking textures:
Or like this:
And what is the difference between the two?
I have a feeling that separate smoothing groups for each UV island has something to do with it, but I'm not sure.
UV's for reference, with smoothing groups set for baking:
I REALLY appreciate anyone willing to help me or point me to somewhere these questions have already been answered.
Thanks!
Z
Replies
1) You don't have to use a cage when using match by name (it is basically making one for you based off those ray distance values), but I have experienced cage bakes looking slightly better.
2) That would be very surprising if true but I can't speak to that.
3) No, but more geometry means you can get the low more accurate to the high, and so the normal map has to do less work.
4) The only rule is that you need to split up your UV islands exactly how you split your smoothing groups (I'll emphasize that it is not the other way around). This is really simple to get started off with Max by using the make uvs from smoothing groups button in the UV editor.
5) Averaged cage is redundant, but descriptive. They are one and the same. Averaged describes the fact that a cage needs to have all averaged normals, aka 1 smoothing group.
Now for a bit of talk on your issue.
Any minor seams you see after baking correctly is perfectly natural and a consequence of the limited resolution and bit depth of your texture. This may be a boatload of info for someone getting started, but it's a twofold problem that I'll break down the best I can.
First: 8 bits per channel textures can't store perfect normals.
Let's say at a given pixel the red value in the normal map should be 0.12345. Well assuming this is 8 bit textures, that works out to a value of 31.47975. The issue is that 8 bit per channel color is only integers, so the closest int would be 31, a value that works out to 0.12156yadayadayada. At the split edges of a model this can cause the face on one side of the edge to shade slightly different than the other face.
Second: The density of these values is limited by the pixel grid.
Think of a detail going diagonally across your texture sheet. It will actually have less resolution than if it were going straight horizontal or vertical, as the pixels do not align to that detail. Extending from that, any point on the surface that does not fall exactly in the center of a pixel is interpolated from the neighboring pixels and can be totally wrong. If you don't have enough pixels to begin with, you've got more surface area for error.
The solution is to keep your technical limitations in mind at every step of the process and make the best of what you have. You probably won't have the luxury of 16 bit normals nor will you ever have enough pixels to make certain things look impeccable. Just get things as clean and accurate as you can, make sure your high poly edges are soft enough to be captured by the normal map, and call it a day!
So essentially, seams exist because of truncated information? Interesting...
Now for new questions
I have heard about the bit depth problem with normal mapping before, and have read that something called 'dithering' is the solution. What exactly is that if you don't mind me asking, and how does one do it?
Does anit-aliasing act as a buffer for this 8-bit issue?
Thanks again!
Edit: I decided to bake at different resolutions to 'test' what you were saying about detail going diagonally/pixel interpolation, and I discovered that the seam gets thinner the higher resolution the bake is. I'm assuming this is because there are more pixels, and therefore more information to cover the seam up.
1k:
2k:
http://polycount.com/discussion/148303/of-bit-depths-banding-and-normal-maps
Basically it adds a bit of noise as you convert a texture from 16 bits to 8 bits. This can be beneficial because when that rounding I mentioned happens, you can end up with pixels that are the same value when in reality they should be slightly different. The addition of noise makes it to where the values are just different enough that no banding will occur. It's very much like normal dithering of gradients, like this:
Imagine this as a very subtle, large gradient in a 16 bit normal map, which if you were to simply round down to 8 bits it might look like the left image below:
So if there aren't enough values in 8 bits to get the smooth gradient, you'd get banding lines wherever the next value starts. By dithering you use the same amount of values but stagger them to approximate the proper transition. Check out the thread I linked above for visuals with actual normal maps that use this.
You can do it manually with photoshop by simply converting a 16 bit texture to 8 bits, but some bakers support it internally.
Antialiasing helps with cleaning up jagged areas, or areas where there is tons of detail in your high poly, because it is taking more samples for each pixel:
Think of the image like this: Consider the line as a mathematically perfect line (a vector image) that needs to get turned into pixels (rasterized). The left line is created by filling in the pixels with black if the area of the pixel is covered enough by the line. If not, it's white. The right line is more accurate to the original vector line, because it has a sense of how much area of the pixel the line is intersecting, and fills it with shades of grey respectively.
Similarly, antialiasing a normal map (or with any baked texture) will be more accurate to the original high poly normals, because it better captures the combined normals of the area of the surface the pixel intersects rather than capturing a single normal from a single point the pixel sits over. If you imagine a detail in the high poly that's smaller than a pixel in your texture, it should be immediately obvious how getting a single sample doesn't necessarily tell the whole story. It needs to be an average of the normals within the area of the pixel to be accurate.
So when using this, it won't be an end all solution but it will certainly be much higher quality than with no AA.
The purest result you can possibly get out of 8 bits is to antialias, dither, and make sure you have a big enough texture. Do note however that for an extremely glossy surface the slight noise of dithering gives the surface a bit of a frosted look, but in practice this rarely matters. You can clearly see this in the thread I linked.
BTW, your results are pretty good already, you're definitely on the right path. I think you just have to read more about the process by looking around the long-running threads here on PC. Just avoid youtube videos and non-reputable random tutorials
I read that thread @somedoggy and I tried it for myself. It doesn't make a huge difference, but the difference is quite apparent if you're looking for it. I can't wait to use it to remove some jagged edges on a higher resolution bake.
Here is my final dithered 1k map:
I'm very happy with this result now that I know this is considered a pretty good bake. The seams become almost invisible if I bump it up to 2k:
Sorry about that @pior , I'm still a bit new to this whole forum thing. I really appreciate your advice, and I will certainly be perusing the threads on Polycount more!
I think I understand. Like it says here?
http://polycount.com/discussion/107196/youre-making-me-hard-making-sense-of-hard-edges-uvs-normal-maps-and-vertex-counts/p1
It says that an "averaged projection" will create a seamless bake, because it ignores the mesh normals and just uses the cage. This would result in no gaps, which are also known as seams. Is that what you mean when you say it's pointless to worry about smoothing groups?
Again I really appreciate it!
I never said that it is "pointless to worry about smoothing groups", quite the contrary. What I am saying is that it is pointless to put hard edges on all UV islands. It is a widely spread misconception that pops up about once a week here. There's probably a popular tutorial out there claiming that (or being very unclear about it), which only causes people to not understand the process.
As for hard edges causing a cage to be split : Some baking apps do that, some don't. The best is to have a solution where you can place hard edges where needed on your model, and still be able to bake through a continuous cage (which is what people probably mean when they say "averaged"). Usually, a baking tool allowing you to load a specific model as cage is a good way to work around that. TB3 does things differently if I am not mistaken - even with a low with hard edges, the projection will be continuous which is great. I do not know what SP does in that regard because I don't see the point of using this app for baking in the first place. The SP documentation probably covers it though.
BTW about quoting your sources : this is precisely to avoid what almost just happened here, ie you assuming that I said "not to worry about smoothing groups"
"I think i understand [...]"
To be fair ... you either understand or you don't
Good luck !
Are you telling your baker to pad out your textures? Your latest image looks like a bit of the black background in the render is leaking through.
Other than that, listen to Pior, dude knows what he's talking about.
@Zalek4 you won't see much mileage out of dithering on such a simple asset, but it is more apparent on glossy, reflective surfaces.
Check out the Basics and Best Results sections.
"I never said that it is "pointless to worry about smoothing groups", quite the contrary. What I am saying is that it is pointless to put hard edges on all UV islands. It is a widely spread misconception that pops up about once a week here. There's probably a popular tutorial out there claiming that (or being very unclear about it), which only causes people to not understand the process."
@pior this is probably in reference to something I've written in the past, I would guess: http://polycount.com/discussion/107196/youre-making-me-hard-making-sense-of-hard-edges-uvs-normal-maps-and-vertex-counts/p1
While using hard edges at uv seams is certainly not a requirement, there are benefits to doing so (the normal map has to do less heavy lifting so it's more forgiving if you don't have a synced tangent pipeline, it compresses better, it mips better, it LODs better in some cases etc), and very few disadvantages unless you're baking with an unsmoothed (unaveraged, using the explicit mesh normals, whatever you want to call it) cage (which is generally not recommended). A hard edge on a model is not implicitly bad. If you've got a UV seam, the normal map already has to account for the difference in tangents (UV islands oriented differently mean different data on each edge of the seam) so a hard edge along a UV seam generally causes no harm. Also, using a hard edge where you have a UV seam comes at no performance cost, because to the renderer, the verts are duplicated either way.
What sort of art style you're using is completely irrelevant, normals and bitangents don't care whether you're going for a realistic style or Pixar style or something in-between. That said, the type of asset you're working on may determine how effective or beneficial the use of hard edges is. If you're creating a 100,000 triangle organic mesh with no extreme angles in sight, the benefits of using hard edges at UV seams may be moot. For hard surface work where you're typically dealing with many extreme angles even with high geometry counts, using hard edges along UVs is typically more fruitful.
There are cases like LODing where it may be beneficial to not use hard edges in certain locations on an asset as well. For instance, if you put a hard edge on every uv seams that means putting hard edges on cylinder seams, which can look a little worse when applying a normal map baked for LOD0 onto an LOD1 or lower detail mesh. Saying that, applying a normal map baked for one set of geometry is always going to result in shading errors when applying it an LOD mesh with altered geometry/mesh normals. Avoiding hard edges won't save the day here, and typically, using hard edges logically will make for more reliable LODing rather than the opposite. For instance, if you have a 90 degree angle at the bottom of a shoe, you're probably going to have a UV seam there, and you'll get the best results for LODs by having a hard edge there too.
TL;DR: "What I am saying is that it is pointless to put hard edges on all UV islands." is just as misinformed and bad advice as saying you must use hard edges along UV seams. The reality is somewhere in the middle and tends to be situational. Trying to set concrete "rules" for this stuff is generally a bad practice in any case.
Now, if you do have hard edges, you'll need to split your UVs along those edges. That's something where it's not really a matter of opinion or situation but always a requirement - otherwise you'll get bad seams along the edges. I think people get confused and flip these two concepts around/think they are the same thing, which is why the original sentiment is so pervasive.
The demonification of the hard edge dates back to the dark ages of poor baking software and worse baking workflows. Basically, before we collectively figured out how this whole baking thing works. In the old days, baking software didn't have an option to bake with an averaged or smoothed cage (I won't explain what this is, check the first section of my TB baking tutorial linked above), and there were misguided tutorials that recommend not baking with an averaged cage - so any area where you had hard edges you would get a seam or gap from the break in projection. Heck, out of the box xNormal bakes with an unaveraged cage, and this is a tool that a lot of people still use.
This lead to a decade long mantra of "You can't use hard edges for good bakes!" - a totally inaccurate, and still pervasive sentiment to this day. If I had a nickle for every time someone said "you gotta use 1 smoothing group for good bakes" I would have a lot of nickles.
Of course, there are indeed circumstances where one might want to "harden all UV edges" - it depends on the kind of asset. And of course not every asset can be made with everything soft - it depends.
My point is that there is a popular misconception floating around that tends to make artists believe that they *should* harden all UV islands. This is a problem I've ran into time and time again, it usually goes like this :
> I receive a baked model for review
> I notice unneeded hard edges in places where they absolutely should not be (along the vertical symmetry line of a piece of armor ; along the UV seams inside the legs and arms ; around the neck and shoulders ; and so on)
> I then ask the artist : "Why did you harden these edges in particular? The ones on the shoe make sense, but why did you put one around the neck, and one right in the middle of this smooth piece of armor ?"
> The answer I usually get is : "Huh, because UV islands ?"
This is then followed by a long explanation/deconstruction of the whole process until the artists understands what's going on.
So that's basically what I am getting at : a little workflow tip that definitely has some use in specific cases, then gets taken as an absolute and is applied to cases where it is actually detrimental - causing the confusion expressed by the OP.
As said, there are indeed types of assets where this is useful (this is what I mean by "art style" - a poor choice of words). But I come from a background where applying this brute force approach would have been detrimental to every single model - hence my recommendations above, which basically consists of making sure that artists understand the necessary and sufficient conditions of a workflow as opposed to applying brute force solutions that works great in some cases but can actually be detrimental in others.
You're saying there are certain situations that are exceptions to the separate smoothing groups for UV islands "rule". I now know to put "rule" in quotes thanks to you guys.
I went and baked the same object I originally posted in MT3 rather than use SP for it, and I got similar results, but the seams are less apparent at the same resolution as the original bake. Now I understand what you meant:And some probably do it to different degrees than others. I'll have to try them out. Thanks for the pointers.
I ran some of my other assets through MT3 just to see if the same thing would happen, but the bakes came out stunning (at least by my current standards).
That makes me think there may be some tweaking needed for the original model I showed here.
I've been reading your thread @EarthQuake , and I must say it is very interesting. I will be tinkering with things in the near future to see what results I can achieve. It helped get me on the right track. @somedoggy , yeah, but now I know what dithering is and how to do it. Thanks for pointing me in the right direction.
Thanks for the long responses guys. They really helped! I've got a long way to go, but I'm getting there!