Edit: Changed title since I think It gave the wrong impression what this is about.
I was noticing that the default cage in Marmoset doesn't do much with Max Offset once it encompasses the object enough. Usually in other programs if I overpushed a cage it would skew the map more and more. (Example, pushing an indented beveled corner to it overlapped would give errors in other programs, but not here)
I was getting a bit of ye old waviness in cylinders, and was trying to do the workaround of pushing a custom cage as close as possible to the ends as this usually reduces waviness to near 0. However, upon import it pretty much shows the same results as the default Marmo cage even though the Marmo cage is further out from the sides/ends.
Is there a way to turn off this effect? I realize in most cases its a benefit to have on, but I'm not dealing with super high polycounts nor mesh resolutions. I was thinking it was the smooth cage checkbox, but that's the default for other bakers as well.
Replies
In this case, you can alter the mesh to improve waviness. Your low poly has small bevels on either side that contribute little to the silhouette, and not enough segments to round out the part. If you drop the bevels, you can double the segments without increasing the triangle count.
^ 16 vs 32 sides
"Should 32 sides be a minimum for a cylindrical bake in Marmoset?"
It really depends on the viewing distance for the assets. With 16 sides on your mesh, you can straighten the skewing with a custom cage, but it's always going to be 16 sides so it will look bad from the front. Doing this sort of ray direction editing is really just trading good results at one angle for bad results at another. Using more segments will tend to help from all angles. If your viewing distance is far enough that the player won't notice that the element is 16 sided, they usually won't notice the waviness either, and the waviness will tend to look better from most angles too.
If this was something like a FPV weapon asset, you would want to prioritize for the front view silhouette, which generally means giving more sides to the cylinders so that they look round. If it's something you can zoom in on with an iron sights view, it's not out of the ordinary to have 64+ sides for a prominent or large element.
"I admit a bit confusion as well. I thought edges ~90 degrees either needed a bevel or a hard edge/smoothing group as per the direction would be too great and leave a line seam on the displayed normal map? Or is because a cylinder is a curved line versus straight between vertices that wont come into play? (If so I have been needlessly beveling all these years)"
Generally speaking, if you have a ~90 degree edge a bevel or a hard edge is recommended, for cylinders or otherwise. With the sort of asset you have, there are natural seams for a hard edge though, and hard edges are not bad - assuming the baker doesn't split the cage at hard edges (XN does this by default). Of course if you're using hard edges, you need to make sure you have a UV split along the edge.
There are some additional concerns here as well: Long, thin strips of triangles tend to be inefficient to render, so if you're beveling every 90 degree angle as a matter of course, that may not be a wise choice.
On the other hand, hard edges and UV seams add additional vertexes, which is really more important than triangle count when it comes to rendering. But generally you would use a hard edge where you break your UVs anyway, so the hard edge verts are usually free. Saying that, if you have a case where you have 90 degree edge don't have a UV split for some reason, you may be able to add a bevel there without increasing the true vertex count. All of this tends to be rather academic on modern GPUs though, and isn't really worth optimizing for to this level of detail unless you have a very low spec target.
I have strong limits for lowpoly models and this limits not take account of hard edges, just tris, like in editor (maya). I can't add edge loop without increasing polycount - and game engine not allow me to load this model... ( Dota 2 Workshop Tool not count hard edges but count every tringle, for example)
We need an adequate cage (which will give us control over a direction of projection per vertex) for bakes. Maya's envelopes = garbage. It maybe easier for understanding for some junior artists, but it halves your abilities for complex bakes. And it just one example of many. I mean cages are not obsolete and probably will never be.
Thanks.