After several days, or I would say weeks, of digging into baking magic, I'm at the point where I have no idea how to interpret results and I would appreciate any advice of more experienced people.
The problem is - these distortions at the surfaces:
My steps for this model are: 1. I've created UV splits for all the polys which have smoothing split and hardened the edge boundaries of the current selected UV map via Modo's Vertex Map palette
"My steps for this model are: 1. I've created UV splits for all the polys which have smoothing split"
That part is good.
"and hardened the edge boundaries of the current selected UV map via Modo's Vertex Map palette"
That part is absolutely not necessary.
You only need to UVsplit the edges that you decided to set as hard - you do *not* need to harden all the edges that also happen to be at UV islands, as this will introduce hard edges where they are not needed (and before anyone jumps in saying that they read a tutorial telling them to do so, or that this or that studio does it that way : the fact that someone does something does not mean that it is necessary).
At this point and as @Obscura mentioned you want to learn more about bit depth. But you also probably want to look into implementing face-weighted vertex normals into your workflow, as it will bring the shading of your lowpoly even closer to that of the high you will be baking from - and thus less "counter shading" in the baked map.
Of course this is all assuming that you are using a well unified tangent basis, but it looks like you are fine on that front.
Not sure but maybe you didn't do something correctly. Did you got the noise pattern with the 16 bits method? Because thats what you should get after converting it back to 8 bits. It should dither out. It would look like its grainy.
1. I've created UV splits for all the polys which have smoothing split and hardened the edge boundaries of the current selected UV map via Modo's Vertex Map palette
I think the order here is backwards here. You unwrap first, then set hard edges to match up with UV edges - this varies based on your unwrap. For something this simple in geometry, you could also chamfer all edges and keep it as one smoothing group - but the vert normals need custom weights.
Maps of normals produce beautiful images but they almost double the texture volume. Maps of normals which do not cause artifacts must be stored in textures of 8 bits per channels at least (RGB8), or better of 16 bits (!G16R16), like in all ATI's demos. In order to make the texture volume smaller the company carried out some experiments with DXTn formats (lossy compression called S3TC in OpenGL) and found out that the DXT5 was the most optimal: in this format each of two coordinates (x, y) is stored in its channel: color and alpha. In this format the memory size required is 4 times lower (32 bits against 8 bits)! As the further extension, ATI will support a new format in its new chips. It's based on the DXTC and previously named ATI2N. It differs from DXT5 in compression of only a color component (out of three) using the method identical for the alpha channel compression. The new format will tremendously increase the precision of compressed texture storage compared to the DXT5, with the size being the same.
"I think the order here is backwards here. You unwrap first, then set hard edges to match up with UV edges - this varies based on your unwrap."
@Butthair : No. What you describe here is a bruteforce approach, which produces a lot of unnecessary hard edges - and offers no 100% guarantee that a specific shading issue is fixed. In fact, it will more than likely create problems since some hard edges will undoubtedly end up over symmetry lines - a place where you really, really don't want them to be.
The order is : the user decides which edges will need to be set to hard. Then, the model is unwrapped so that for every edge that is hard, there is also a UV split.
Sure, what you describe may work in many cases - but this is not the Necessary And Sufficient approach, and shows a misunderstanding of the reason why UV splits and hard edges are used in the first place (no offense here, this is a very widespread misunderstanding).
I do not have a lot of experience in my back, but this is what I actually do to fix this kind of problems with bits.
1º Export normal map in .Tiff 16 bit, you shouldn't have any weird seam problem like you had. 2º Open your 16 bit .TIFF in photoshop and lower ir to 8bit inside photoshop. 3º Export your new 8 bit map with , Png or targa. 4º Now your normals will be dithered and you wont see any seam.
I don't know if you are asking for this but, maybe I can help someone, thanks.
I tend to make as little UV seams as possible, which I treat like hard edges, so I suppose my order is wrong, although I have the same hard edge intention in mind when unwrapping. Visually, it is easier to set your hard edges first, then set edges.
As I think on it, I still prefer setting my UV edges first. This way I have less UV edges overall, and set hard edges accordingly for bake. If I were to resuse anything for kitbashing or modular, I would have more splits, but if it's a one off or self contained model, I see no drawbacks. But with less UV splits it also means less smoothing splits and ultimately less vert data in the model - optimized.
@Butthair : Anyone has their own workflows and reasons for them - I have zero problems with that, to each their own ... as long as I don't have to deal with the outcome What I am getting at is that when it comes to technical stuff like this, it is important to understand the very reasons why things are done a certain way (as said : the necessary and sufficient conditions) - because if not, things get misinterpreted really fast.
Hard edges are helpful to control shading. Wherever a hard edge is placed, there is also a need for a UV split, since without one the RGB information of the normalmap will abruptly change over the edge and bleed over.
It certainly is less catchy than "harden all UV edges" ... but it is the correct way of thinking about it. I've had to deal with countless models that needed to be reworked/reUVed/rebaked because they had a pointless hard edge going down their symmetry line, because the modeler applied the misguided dogma of "hardening all UV edges". I suppose the last time I've had to deal with that was one time too many, and I am now making a point of correcting this misunderstanding every time I come across it online
In my opinion there's nothing hard or easy about these things - once the topic is well understood, placing hard edges where needed is second nature and unwrapping is always a trivial task anyways.
Not sure but maybe you didn't do something correctly. Did you got the noise pattern with the 16 bits method? Because thats what you should get after converting it back to 8 bits. It should dither out. It would look like its grainy.
I do not have a lot of experience in my back, but this is what I actually do to fix this kind of problems with bits.
1º Export normal map in .Tiff 16 bit, you shouldn't have any weird seam problem like you had. 2º Open your 16 bit .TIFF in photoshop and lower ir to 8bit inside photoshop. 3º Export your new 8 bit map with , Png or targa. 4º Now your normals will be dithered and you wont see any seam.
I don't know if you are asking for this but, maybe I can help someone, thanks.
Hmm.... I didn't convert 16bit into 8bit. Just sent 16bit Png and Tga straight into Marmoset. So, I did as you suggest: lowed 16-bit Png to 8bit inside photoshop, and it solved the problem:
But I'm not sure, why 16bit doesn't work without conversion... In case I
would like to use it without any noise, since I've created 16bit
anyway...
Anyway, thank you very much for your advice, guys.
Replies
1. I've created UV splits for all the polys which have smoothing split"
That part is good.
"and hardened the edge boundaries of the current selected UV map via Modo's Vertex Map palette"
That part is absolutely not necessary.
You only need to UVsplit the edges that you decided to set as hard - you do *not* need to harden all the edges that also happen to be at UV islands, as this will introduce hard edges where they are not needed (and before anyone jumps in saying that they read a tutorial telling them to do so, or that this or that studio does it that way : the fact that someone does something does not mean that it is necessary).
At this point and as @Obscura mentioned you want to learn more about bit depth. But you also probably want to look into implementing face-weighted vertex normals into your workflow, as it will bring the shading of your lowpoly even closer to that of the high you will be baking from - and thus less "counter shading" in the baked map.
Of course this is all assuming that you are using a well unified tangent basis, but it looks like you are fine on that front.
"3. Tried both 16-bit PNG and TGA to avoid bit of depth problems http://polycount.com/discussion/148303/of-bit-depths-banding-and-normal-maps/p1 "
model
I think the order here is backwards here. You unwrap first, then set hard edges to match up with UV edges - this varies based on your unwrap.
For something this simple in geometry, you could also chamfer all edges and keep it as one smoothing group - but the vert normals need custom weights.
For the normal map, yeah I still some banding even in sketchfab. Best format for normals is 3DC - https://en.wikipedia.org/wiki/3Dc
From polycount wiki:
@Butthair : No. What you describe here is a bruteforce approach, which produces a lot of unnecessary hard edges - and offers no 100% guarantee that a specific shading issue is fixed. In fact, it will more than likely create problems since some hard edges will undoubtedly end up over symmetry lines - a place where you really, really don't want them to be.
The order is : the user decides which edges will need to be set to hard. Then, the model is unwrapped so that for every edge that is hard, there is also a UV split.
Sure, what you describe may work in many cases - but this is not the Necessary And Sufficient approach, and shows a misunderstanding of the reason why UV splits and hard edges are used in the first place (no offense here, this is a very widespread misunderstanding).
1º Export normal map in .Tiff 16 bit, you shouldn't have any weird seam problem like you had.
2º Open your 16 bit .TIFF in photoshop and lower ir to 8bit inside photoshop.
3º Export your new 8 bit map with , Png or targa.
4º Now your normals will be dithered and you wont see any seam.
I don't know if you are asking for this but, maybe I can help someone, thanks.
I tend to make as little UV seams as possible, which I treat like hard edges, so I suppose my order is wrong, although I have the same hard edge intention in mind when unwrapping. Visually, it is easier to set your hard edges first, then set edges.
As I think on it, I still prefer setting my UV edges first. This way I have less UV edges overall, and set hard edges accordingly for bake. If I were to resuse anything for kitbashing or modular, I would have more splits, but if it's a one off or self contained model, I see no drawbacks.
But with less UV splits it also means less smoothing splits and ultimately less vert data in the model - optimized.
Hard edges are helpful to control shading. Wherever a hard edge is placed, there is also a need for a UV split, since without one the RGB information of the normalmap will abruptly change over the edge and bleed over.
It certainly is less catchy than "harden all UV edges" ... but it is the correct way of thinking about it. I've had to deal with countless models that needed to be reworked/reUVed/rebaked because they had a pointless hard edge going down their symmetry line, because the modeler applied the misguided dogma of "hardening all UV edges". I suppose the last time I've had to deal with that was one time too many, and I am now making a point of correcting this misunderstanding every time I come across it online
In my opinion there's nothing hard or easy about these things - once the topic is well understood, placing hard edges where needed is second nature and unwrapping is always a trivial task anyways.
Hmm.... I didn't convert 16bit into 8bit. Just sent 16bit Png and Tga straight into Marmoset.
So, I did as you suggest: lowed 16-bit Png to 8bit inside photoshop, and it solved the problem:
But I'm not sure, why 16bit doesn't work without conversion... In case I would like to use it without any noise, since I've created 16bit anyway...
Anyway, thank you very much for your advice, guys.
With that in mind, OP did the right steps for baking the normals.