Hello everyone! I already made a few examples about this to a few of you, but I thought it would be more useful to make a tutorial from it, so I won't have to write it down every time when it comes into the topic. Also this technique is becoming more and more popular as the next gen limitations allows us to use more geo, and less normalmap hackery. You can see this in action, and it was heavily used in Star Citize (as I see, their entire workflow built around this technique),Alien Isolation, and in many racing game, even from the previous generation. I would bet the cars in GTA4/5 made similarly! Probably there are many other real examples too, but I know only these.
So...What are vertex normals?... They are used to light/shade the geometry, every vertex has one. How they work?... Its pretty simple, there is a dot product calculation between the light vector, and the vertex normal. If its exactly facing the light direction, the you will get completely lit vertex/pixels. The more direction difference between them will cause more shadowing.
So basically they are determining how the model should shade under light.
If we take an example of a box, which has smoothing groups, and light to it from the top, we will get the top side being lit, and the other sides will be dark, or black, if we have black ambient color, or no any image based lighting, light environment, or such.
It works like this. But the problems are coming, when you don't want the "hard edge" look. People are solving this with baking a normalmap from a round edged object, and do all the hack that needed then. This worked, works, and it will work in the future too, but there are other, probably easier and less time consuming solutions too. The problem is that if you set the smoothing to use only one smoothing group, you it will make the shading drastically worse, if you don't have enough geometry. Here comes what the tutorial will explain, custom vertex normals. An another solution is to simply add some support edgeloops. Maybe I'll explain those too in an another tutorial later.
So lets get started.
Here is a cube, with a single chamfer on the edges. You would think it should shade almost good. It won't, because of the average normal direction between the faces.This is the downside of the default smoothing. It won't represent the flat sides well, it still has very visible gradients.
But from here, the only thing you need to do, is to fix that average direction to be a proper direction, which is exactly perpendicular to the box's sides in this case.
As you can see, after making them being perpendicular to the sides the gradients are completely gone. The model could be used even without a normalmap, it has smooth edges, and there is no any ugly shading.
I won't go into details about how to edit them in this and that app, maybe also in the future, if there will be a request.
Here are a few, bit more complex examples of the proper usage of this. In some of the examples, I'm showing support looping too.
This whole thing is a big game changer in my opinion, I don't always bake now, rather I decide if its needed at all, or if a chamfer with edited normals is enough. I would highly recommend at least trying it out to everyone, who do a lot of normalmapped object, because you can save many hours while still getting good quality at the end. With doing this, you can skip the "making a whole highpoly model" part of a creation process, and bake only the details that are not some simple rounded corners. Its also very useful to reduce gradients when you are baking!It can completely remove them if you are doing it good, so you can use less smoothing/uv splits, and texture easier.
I hope you found this thread useful! Similar explanation about support edgeloops , and advanced usage of similar tricks will probably come soon.
PS: sorry for the poor english
Also, maybe its a bit chaotic atm, I just sat down and wrote it down in one run. I'll try to make it more organized soon.