Baking, Cage, Triangulated, Hard Edge, Normal Group, and Seams

PyrZern

What are the conclusion on this ??

Usually I been working on organic stuff, so there really isn't any hard edges around. But I been doing some more stuff with sharper edges as well.

Anyway; what I generally do is.
* Keep the Quad LowPoly for final render.
* Triangulate the LowPoly I want to bake. Save as Tris for baking.
* Inflate it a bit to fully cover the HighRes, save as Tri Cage for baking.
* Slap baked textures onto the Quad LowPoly to animate and render.

What I have heard.
* If there's seam on the UV, then Hard Edge (for Maya) or separate the Normal Group (for Max) along the seam as well.  This is quite a hit or miss for me.


I mean, the hard edge is still there right ?? It shows on the Render. Especially with lighting.

Tits

Anyway; what I generally do is.
* Keep the Quad LowPoly for final render. 
Sure, does look a bit cleaner than triangulated mesh for renders
* Triangulate the LowPoly I want to bake. Save as Tris for baking.
Triangulating is not always required but is definitly a good thing to do. On organic meshe the triangulation isn't always as important, but on complexe and hard surface mesh the triangulation can make a big difference. Triangulating your mesh before baking ensure that the triangulation use for your bake is the same as the triangulation use for the final mesh. 
* Inflate it a bit to fully cover the HighRes, save as Tri Cage for baking.
I don't know in which program you bake so I won't comment on that, but you should make sure you cage goes over all the high poly mesh to avoid missing area.
* Slap baked textures onto the Quad LowPoly to animate and render.
To ensure that the triangulation doesn't change when importing your character in engine for rendering It's never a bad idea to import your triangulated mesh
if you render inside of the same software you baked from it should be fine without triangulating

What I have heard.
* If there's seam on the UV, then Hard Edge (for Maya) or separate the Normal Group (for Max) along the seam as well.  This is quite a hit or miss for me.
No, if you use a hard edge, or use a different smoothing group THEN you should split your UVs, the opposite isn't true You DO NOT need (I mean you can if you want to...but why?) change the smoothing group for every UV shell. 
In the case of your hand using one smoothing group is just logical, there is no sharp angles and it's a pretty organic shape. 
Splitting smoothing group is require in area where some sharp angles would be used (the common cube exemple with 90 degree angle). In that case you would split smoothing group for each face and then split the UVS.

PyrZern

@Tits Thank you for quick rescue !

Oh, so it's not "hard edge if you split UV", but instead "split UV if you hard edge" instead ??

Other questions
* Must the LowPoly to be baked (triangulated or not) and the custom-made Cage have the exact same UV and hard edge ? (usually I inflate Tri LP in zbrush to cover my sculpt, and export as tri cage to be in xNormal or Substance.)

Tits

My pleasure!

Oh, so it's not "hard edge if you split UV", but instead "split UV if you hard edge" instead ??
Exaclty! No need to put hard edge everywhere,  use an hard edge only for harsh angle creating a lot of shading on your low poly, then split the uvs!

For the other question, I don't export cage often but the cage should only be a pushed version of the low poly mesh, and so should have the same numbers of verticles, same UV etc (xnormal is very picky with that too). not sure if the smoothing groups change anything in that case you could ask @EarthQuake

I've been using knald instead of xnormal for a little while now. it allows you to create a cage with a slider straight from knald viewport, it's very easy and trouble-free.
I know xnormal also offers something similar but it was kinda buggy, laggy and the controlled were a lil bit of a pain in the ass.
I recommend you checking knald, I know it's not free ( ) but it definitly made the baking process a lot less of a pain in the ass
I hear substance is great too, haven't really tried it too for baking
  

jaker3278

Are you using enough edge padding? It would be great to be able to find a solution to those seam problems your having. 

PyrZern

@Tits
@dustinbrown


I believe the normal group/hardedge of the Cage doesn't matter.   (I inflate Tris into Cage in ZBrush which automatically hardedge everything upon Export.) It still bakes ok. So I believe hard edges of the Cage itself doesn't matter.   But I have a feeling that the UV coordinate of the Cage does matter. 

@jaker3278
I'm just now switching to Substance Designer to do my baking needs. Not sure about the Edge Padding in here. (I was using xNormal before, and I didn't have problem with seams though)...   There's "Dilation Width" which I set to 4, for 4k map baking.

I'm gonna try bumping that up to 8 and see how things go.

Tits

I believe the dilatation is the ''padding'' setting.
4 pixel is very tiny tho.
They usually recommend 16px padding for a 2k map

jaker3278

Your latest render looks better than the last one. Like @Tits said try 16px. Substance has an Average Normal check box, leave this unchecked if your using floaters of any sort. 

PyrZern

First off, I believe the UV mapping of the cage itself does matter. (in case the UV of your LP and your Cage doesn't match) I had to copy/paste UV of the new TriLP to the old inflated Cage for it to work. (had to move LP's UV around a tad).

Anyway.


@jaker3278
Average Normals check box is grey out since I use custom Cage. No floats used here.

Here's with 32 Width Dilation. Seam's still there (and it's eating into other UV space. Crap)

jaker3278

On most of the normal maps iv seen they usually have some seam at the UV borders, i could be totally wrong but i dont think you are not going to get a perfect transition. I stand to be corrected though. 

PyrZern

Ok, quick question. How can everyone get a nice bake without using custom Cage in tight area ??  Like, around fingers, thigh gaps, behind the ears, or eye corner (eyelid thingy). Are my lowpoly too low poly that the angular cage overlaps something else before it fully engulf the sculpt ??


I notice that the textures I bake from Substance Designer isn't so clean.  The white background of AO maps isn't pure white. ANd the purple background of Normal map is slightly off. So when I overlay them on top of each other, or multiply for AO, the hue slightly shift the more layers there are. And seam becomes more visible. I notice if the hand layer is at the very top layer with normal blending, the seam is more harder to see. But when I change it to Overlay to allow other layers to show, the seam is easier to see.



My settings. 
 

@jaker3278

I mean, yes, in the grand scale of things, no one gonna see it.

PyrZern

@dustinbrown
Would higher tri count makes it easier to bake and all that too ?

Tits

Could be higher but you definitly shouldn't have a seam there, especially since you are baking 4k maps.

musashidan

The cage mesh is averaged so has a single smoothing group. The cage doesn't require any sort of UVs. The dilation is obviously going to be affected by the edge-padding from the unwrap. So if you unwrapped with 16px padding then this is as high as the dilation can go, after that it blends into opposing pixels automatically and essentially 'cuts off'. Personally, I bake in Substance Painter. I don't use a cage, ever. I always bake everything - organic or hard-surface - as a single smoothing group(averaged), don't export tangents/bi-normals in my .fbx, and allow baker(SP) and destination(UE4) to compute the T/BN on import. This is a synced workflow and requires no UV/SG splits. And yes, the higher vert-count your game mesh the better the baking result due to the fact that the mesh will be a much closer representation of the silhouette of the high-poly.