Of interest to most who will read this thread, I wrote an extensive
tutorial that covers this topic and many other common baking issues.
It's geared towards baking in Toolbag, but most of the Basics and Best
Results sections apply universally. Check it out on the Marmoset site: https://www.marmoset.co/posts/toolbag-baking-tutorial/
This is an issue that pops up more and more frequently these days. Now that many engines/renderers are finally using synced normal map workflows, the weaknesses in our pipeline become easier to spot.
First off, what is bit depth and why is it important? Bit depth refers to how many values an image can store. A 24 bit image can store 8 bits, or 256 values of information per channel. From this point forward, if I say 8, 16 or 32 bit, assume I mean per channel. 256 values are usually enough for content like diffuse, specular or gloss maps because these maps tend to use up a broader range of the value spectrum.
Tangent space normal maps, on the other hand, tend to use a much narrower value range. Worse yet, the more similar the low poly surface is to the highpoly, the more precious the bit depth precision gets, as the difference that is recorded the between the high and low poly normals (which is essential for accurate shading) is very small, often to a degree lower than 1/256th, which is the most precise value that can be stored in an 8 bit texture.
So here we have the perfect storm, a lowpoly object with smooth curves that very closely matches the surface of the highpoly, with a fully reflective glossy material.
If we bake an 8 bit map with a baker that does not dither the result (eg, xNormal and Maya) we get stair-stepping artifacts where the texture can’t record precise enough values.
If we increase our bit depth to 16 in xNormal by baking a TIFF file, we get a much smoother, cleaner result.
So, if 16 is good, 32 bit must be better, right? In theory yes, however, in practice its not generally the case. Here we have a 32 bit map, and we can see there is essentially zero difference between the 16 and 32 bit bakes.
Storage space is trivial these days, so why wouldn’t you always bake with 32 bit? Well, first off, the difference typically is not noticeable. Secondly, its difficult to work with 32 bit files from bakes to final texturing (many features are disabled in PhotoShop in 32 bit mode), you can, if so inclined, with 16 bit files. Thirdly, converting from 32 bit to 16 or 8 bit space is more prone to user error (by default, when you convert from 32 bit image to 16 or 8 bit, you will be presented with tone mapping options in photoshop, which is something you never want to do to a normal map).
What about in game? 16 and 32 bit file formats are simply not supported (or at least not commonly used due to texture memory constraints) by most game engines. So, at the end of the day you most likely need to output a 8 bit file. Starting from 16 or 32 bit source allows the content to be dithered while down-sampling. Dithering adds noise which breaks up the stair stepping pattern. You might not like the noisy look, but it usually looks way better than banding/stair stepping. Additionally, most assets won’t have perfectly smooth mirror-like surfaces, so it’s unlikely the noise will be noticed in the final product. I’ll mention Max here, as Max dithers when baking out 8 bit files, so if you bake in Max you really don’t need to worry about any of this (unless you want to generate cavity, ao, etc maps from your normal map, then having a higher bit depth file is handy).
Now, lets look at a real world asset. Here is a sword I modeled recently.
Yet again, baking to a higher bit depth helps to remove banding, and down-converting to 8-bit dithers the result to more effectively cram the values into the 8 bit space.
Now, to maintain a bit of perspective, take particular note of the final textured shot with 8-bit (undithered) normal map. What were previously very obvious and annoying artifacts have gone away almost entirely. You can still see some issues here and there if you know where to look, but in game/motion and viewed from a reasonable viewing distance you would be very hard pressed to tell. This doesn’t even take into account the horrors that texture compression will play on your normal map.
One final image to help visualize why this happens. Here I've increased the contrast of the normal map significantly to expose the problem. Even pushed to this extreme level, which you would never do in production, the 16 bit map holds up relatively well.