Understanding cages when baking a normal map
greentooth
So I would like to ask a question that I'm asking myself since a very long time now.
I'm a Maya user and now experienced to deal with vertex normals. I have also read some things here and there about smoothing groups and I think I understand them too, even if I don't use them. However, I really don't understand what is the benefit of a cage is over the use of vertex normals.
I traditionally bake with xNormal without using a cage (using the ray distance feature). From what I have discussed with 3DS Max users, a cage is just the same mesh that your low-poly pushed away in the direction of the normal of the faces. You can however break the projection by making hard-edges with the help of different smoothing groups.
Let me explain what I don't understand with an example :
While I see the benefit of a cage to use hard-edges on UV seams and get a really clean bake, I really don't understand how a cage can justify the use of more geometry to counteract the distortion made by the cage projection (as shown above in my example, I'm not talking about the gradients but about the wrong bake of the cylinders on the top of the cube).
You could say that I just have to tweak my cage, or to add more geometry, but I see this as problem, not a solution (polycount still matters and tweaking takes time).
I also see an other point, this one directly related to my own workflow : I'm somebody that bake often, very often, even when my asset is not finished at all. I prefer to bake often than baking only in the end. Just to adjust my work on the way and avoid to go back on my asset to fix errors. So with cage I will have to redo it every-time, while tweaking vertex normal take a seconds and I only need to update the part I have changed.
So while I see the benefit of a clean bake (the difference is very subtle is some cases), I don't believe it's worth the time compared on how easy and fast vertex normals are to use.
This is my current point of view, but seeing some much people working with cages make me feels that I'm totally wrong and I missed something about cages. That's why I'm asking for a true enlightenment.
I'm a Maya user and now experienced to deal with vertex normals. I have also read some things here and there about smoothing groups and I think I understand them too, even if I don't use them. However, I really don't understand what is the benefit of a cage is over the use of vertex normals.
I traditionally bake with xNormal without using a cage (using the ray distance feature). From what I have discussed with 3DS Max users, a cage is just the same mesh that your low-poly pushed away in the direction of the normal of the faces. You can however break the projection by making hard-edges with the help of different smoothing groups.
Let me explain what I don't understand with an example :
While I see the benefit of a cage to use hard-edges on UV seams and get a really clean bake, I really don't understand how a cage can justify the use of more geometry to counteract the distortion made by the cage projection (as shown above in my example, I'm not talking about the gradients but about the wrong bake of the cylinders on the top of the cube).
You could say that I just have to tweak my cage, or to add more geometry, but I see this as problem, not a solution (polycount still matters and tweaking takes time).
I also see an other point, this one directly related to my own workflow : I'm somebody that bake often, very often, even when my asset is not finished at all. I prefer to bake often than baking only in the end. Just to adjust my work on the way and avoid to go back on my asset to fix errors. So with cage I will have to redo it every-time, while tweaking vertex normal take a seconds and I only need to update the part I have changed.
So while I see the benefit of a clean bake (the difference is very subtle is some cases), I don't believe it's worth the time compared on how easy and fast vertex normals are to use.
This is my current point of view, but seeing some much people working with cages make me feels that I'm totally wrong and I missed something about cages. That's why I'm asking for a true enlightenment.
Replies
Your cage example also looks more Lego-like than your non-cage example, which leads me to wonder if your cage didn't go out far enough to capture the details of the four rounded cylinders on top.
As I'm saying above, tweaking the cage/adding more geo to compensate this problem is possible, I simply don't understand why someone would choose this process which feels for me far more complicated and time eater than tweaking vertex normals.
So maybe my question is more about workflows than technical problems...
while it is true that baking without a cage gives better surface results (for any bolts/protruding geo), it typically only works well on flat surfaces. when your surface starts to get more complex, coupled with many uv splits/hard edges, a non cage bake starts to look worse and worse.
Personally I think different meshes call for different methods. Some can easily get away without a cage, which can make making one seem like excessive work. If you dont need one, you don't need one. Other meshes however can become a pain without them (without a bevel like that, I would think a vertex split along a hard edge could start causing the rays to miss? There are examples on the wiki of this). I also suck at getting a good ray distance going even when using the calculator; sometimes I find one value might be too low to capture all the geometry of the highpoly, but if I find a value that is high enough to reach those spots then I might see errors in other parts of the mesh where things are tighter.
Granted I'm far more used to baking more organic shapes, so I'm sure hard-surface guys will really know what's up.
My work flow leaves all the UV mapping, smoothing groups, and texture baking till the very end. Thanks to many of the people on this site (especially EarthQuake), I've been able to read a ton on baking normals and all the crazy stuff that's going on under the hood. The only rule I really follow is "UV Seam = hard edge". Combined with a proper cage, my bakes come out perfect nearly every time, the first time.
I made a quick update on my cage bake to avoid any confusion :
So in this case, you guys would simply tweak the vertex of the cage to fix the projection of the cylinders ?
That's what I'm not sure to understand, if you keep an global cage or if you tweak it a lot...
Either that or squish the HP parts down. It's baking at pretty much a 90 degree angle so there will only be so much detail that gets baked.
If I'm using vertex normals I'm sure I get exactly 90 degress since I make sure that my normal are perfectly perpendicular to the face I'm baking on (in this example of course).
This thread covers the issue in a lot of depth: http://www.polycount.com/forum/showthread.php?t=81154 Read the whole thing and the come back if there is something you're still unsure about.
If you're saying you simply edit your vertex normals to give a better projection, this is certainly something you can do but it doesn't really mean you're not using a "cage" in the typical sense of the word. This is also something that may be difficult and time consuming to do with more complex meshes, but yeah, its certainly an option.
In Maya, "using a cage" is really what you have "match using" set to in your transfer maps section. One of the settings breaks the projection at the edges, which will give you a more straight projection in most cases at the cost of annoying seams along your hard edges, while the other setting keeps those gaps welded at the expense of skewed projection(if you don't account for it). I'm on my laptop here so I can't remember which setting does which, but it should be mentioned in that thread.
I can understand wanting neat vertex normals and such, but when it comes to capturing a robust normal map, I don't quite understand why you aren't following the HP topology more closely.
I have already read it a lot of time, but I have hard to time to apply the logic of some parts with the usage of a cage.
This is exactly what I'm saying !
When I say I don't use a cage, of course I'm talking about using an external geometry shape to drive the projection. Otherwise I'm aware the processur is similar. That's why I created this topic, to understand better why the cage seems so beneficial while for me tweaking my vertex normals seem to give at least the same quality bake with less editing time.
From my way or working, and because I use a lot of scripts to help me, I believe it take less time to setup some specific faces rather than setup a cage. Again that's my way or working, my inexperience with cage are probably the source of this believing.
Thanks, I will take a look (even if I mostly work with xNormal).
I'm not saying you need to tweak all your vertex normals, not at all. In the end, most of my mesh use averaged normals, which result to something similar of an averaged cage projection.
What I'm saying is that tweaking some vertex normals here and there seems to me faster and foremost better to bake (especially in the case of floating geo like on weapons).
But again, that's my way of working and that's why I wanted your point of view guys !
1. Use averaged normals on your entire mesh, no hard edges or custom edited normals. Using a cage or not has no bearing on your baked result(unless you manually edit your cage, which works differently per app, hand editing the cage/envelope in Maya for instance doesn't effect ray direction, only distance)
2. Use hard edges, using or not using a cage makes a big difference here, if you don't use cage you'll get gaps/seams along your hard edges, so its very important to use a cage
3. Use custom edited normals instead of hard edges
#3 Is really the least common, however, if you can fit it in your workflow, it can be a really smart way to work.
On the other hand, custom normals may not transfer well between apps, or into your target engine at all, and are more complex and time consuming to set up especially with a complex mesh. If you do your modeling in a separate app like Modo, and then set up your vert normals in Maya, you'll have to redo all that work if you say, need to re-pack your uvs in Modo and re-import. So its not really the most flexible option if you're dealing with more than 1 app.
Also, if you want to do a mix of #2 and #3, ie: some hand edited vertex normals, but also some hard edges(I've needed to do this before on projects to make sure normal maps work well with various LOD meshes) you're going to have to use a cage to avoid the hard edge gap/seams issue. So you're back to a cage anyway.
In practice, generally what I do is #2 + simply cut in a little more geometry to account for floaters, in your image example, you would just need to cut in 1 vert right in the middle of that top face. So one extra vert, 2 extra tris, not really a big deal. Even with a complex asset with a lot of floaters of 5000 tris or so, you may only end up adding 2-300 or so tris, and you can make sure that your projection works well regardless of the app you're baking in, with no rework when you need to make changes. When I model my lowpoly meshes, I make sure to save a certain chunk of my triangle budget for fixing ray projection errors. I like doing this because its easy and predictable to do in every app, a project may call for me to do bakes in Maya, Max or Xnormal, and this works the same in each.
Also, in terms of high poly modeling, if you only intend to capture the detail on the tops of those cylinders, there is no need for them to extend that far from the base highpoly mesh. Make sure your floaters and details like this are as close to the surface of your lowpoly mesh as possible. Remember, normal maps can only capture angles, not depth. You'll get more issues with skewing etc if your floaters are too tall/deep or too far from the base surface.
Thanks for this explanation, I think I understand it now. Time to update my workflow then !
[EDIT] As discussing this with a friend, he asked me a good question : in the case of a sword, how would you process to get a good bake with a cage while getting nice hard-edge for the thin part of the blade ?
Is that possible ?
Very hard edge in highpoly
hard edge in lowpoly
cage bake
Should work fine?