# A short explanation about custom vertex normals (tutorial)

grand marshal polycounter
Offline / Send Message
grand marshal polycounter
Hello everyone! I already made a few examples about this to a few of you, but I thought it would be more useful to make a tutorial from it, so I won't have to write it down every time when it comes into the topic. Also this technique is becoming more and more popular as the next gen limitations allows us to use more geo, and less normalmap hackery. You can see this in action, and it was heavily used in Star Citize (as I see, their entire workflow built around this technique),Alien Isolation, and in many racing game, even from the previous generation. I would bet the cars in GTA4/5 made similarly! Probably there are many other real examples too, but I know only these.

So...What are vertex normals?... They are used to light/shade the geometry, every vertex has one. How they work?... Its pretty simple, there is a dot product calculation between the light vector, and the vertex normal. If its exactly facing the light direction, the you will get completely lit vertex/pixels. The more direction difference between them will cause more shadowing.
So basically they are determining how the model should shade under light.
If we take an example of a box, which has smoothing groups, and light to it from the top, we will get the top side being lit, and the other sides will be dark, or black, if we have black ambient color, or no any image based lighting, light environment, or such.
It works like this. But the problems are coming, when you don't want the "hard edge" look. People are solving this with baking a normalmap from a round edged object, and do all the hack that needed then. This worked, works, and it will work in the future too, but there are other, probably easier and less time consuming solutions too. The problem is that if you set the smoothing to use only one smoothing group, you it will make the shading drastically worse, if you don't have enough geometry. Here comes what the tutorial will explain, custom vertex normals. An another solution is to simply add some support edgeloops. Maybe I'll explain those too in an another tutorial later.

So lets get started.

Here is a cube, with a single chamfer on the edges. You would think it should shade almost good. It won't, because of the average normal direction between the faces.This is the downside of the default smoothing. It won't represent the flat sides well, it still has very visible gradients.

But from here, the only thing you need to do, is to fix that average direction to be a proper direction, which is exactly perpendicular to the box's sides in this case.

As you can see, after making them being perpendicular to the sides the gradients are completely gone. The model could be used even without a normalmap, it has smooth edges, and there is no any ugly shading.

I won't go into details about how to edit them in this and that app, maybe also in the future, if there will be a request.

Here are a few, bit more complex examples of the proper usage of this. In some of the examples, I'm showing support looping too.

This whole thing is a big game changer in my opinion, I don't always bake now, rather I decide if its needed at all, or if a chamfer with edited normals is enough. I would highly recommend at least trying it out to everyone, who do a lot of normalmapped object, because you can save many hours while still getting good quality at the end. With doing this, you can skip the "making a whole highpoly model" part of a creation process, and bake only the details that are not some simple rounded corners. Its also very useful to reduce gradients when you are baking!It can completely remove them if you are doing it good, so you can use less smoothing/uv splits, and texture easier.

I hope you found this thread useful! Similar explanation about support edgeloops , and advanced usage of similar tricks will probably come soon.

PS: sorry for the poor english
Also, maybe its a bit chaotic atm, I just sat down and wrote it down in one run. I'll try to make it more organized soon.

## Replies

• Offline / Send Message
polycounter lvl 8
Nice explaination I´m using this technique often to create my lowpolys because it saves me a lot of trouble with nasty gradients.

I did my set for the september 2014 polycount challenge just with custom normals. No highpolys were hurt during the creation of this set.

http://www.polycount.com/forum/showpost.php?p=2146316&postcount=143
• Offline / Send Message
polycounter lvl 8
Awesome write-up, I was interested in this after hearing Star Citizen using it for their ship work flows. How complex can the geometry be before you need to start actually modelling a high poly, any examples?

Heres a gif I just made! Awesome. I only know the basics, but are any of those other parameters useful? It'd be nice if some of the pros on polycount posted their tips and tricks but I guess they don't want to give away their secrets to competitors in this industry
• Offline / Send Message
Very useful method for when you can't use normal maps, I.e. due to massive scale like spaceships. I tried to push for it on Elite but unfortunately we're still using support loops to manage the shading instead. It was also used heavily on Alien Isolation.
The biggest downside is that vertex normal editing is very finicky, in Max at least.

some scripts for Max:

http://www.polycount.com/forum/showpost.php?p=1199836&postcount=41 This one is fully manual. Slower results but much cleaner.
http://www.polycount.com/forum/showpost.php?p=2230470&postcount=28
https://github.com/ByteHazard/wnormals

ALso more here http://wiki.polycount.com/wiki/Vertex_normal

The quad chamfer plugin also supports this and it's very powerful if you're using this kind of workflow.

[ame]
• Offline / Send Message
ngon master
I can see big areas being done like this but what do they do with the high frequency detail? Do they paint normal maps in apps like ndo for example?
• Offline / Send Message
Tiling materials and smart masks. And lots of geo to support it.
• Offline / Send Message
ngon master
Thanks Joost.
• Offline / Send Message
interpolator
Yeah, I am using this method for years. Let you mostly forget about baking being synced or not except within those bevels where long triangles and radial normals still create shading /reflecting errors.

Took me a decade to plead Blender and Modo devs in every possible forum for editable vertex normals.

One weird thing is that it have always been so underestimated by 3d soft developers. I remember one Light Wave guy even asked me why I need edited normals at all.
• Offline / Send Message
grand marshal polycounter
Hey guys! I try to answer to every of you, and to share my opinion now.

Wiki - I'm also using this for a long while now, when I don't have exact limits. I use this, and many support loops with my personal projects, and to be honest, now I'm not trying to get it good with smoothing splits... Your results are showing the potentials very will. This month's challenge seem to fit this method, so I'll do it as a bigger experiment. Hopefully I'll find enough free time to finish it. But because of the timesaving with this method, I'll probably be able to finish it in time. Especially if I'll use simple support loops too.

tynew - No, actually I'm testing and using this for a long while now, that thread just made me sure, its an actually working thing in production work, and it has real potentials in the game industry. Well, It can be any complex if you are using enough geo. Just take a look at that Star Citizen Ship. Of course, he is using cheats like decals a lot, but well, it actually works, you can't notice if you are just a player, and not someone who meet such things every day. Here is one "complex" model's example, that I made using supportloops. Not chamfering, but actually these are almost the same, at least in result. The support looping gives more polygons, but it also gives even better shading in the corners. So here is a picture. It was a wip to my client and unfortunately I'm not allowed to show more at the moment. No any highpoly modeling, just decent amount of geo overall, and supported edges. I think this prop would fit nicely to the vr requirements.

As for the parameters and such, I also use this average selected, as a starting point. This won't always work, especially with complex surfaces and shapes, but still it can be a good start. Then I simply move/rotate the normals manually to make them perfect. If the model allows, I do quick selections from ortho view, and manipulate multiple normals at once. Sorry, I won't go into more details this time, as the first post says, maybe later ^^ But what I wrote can completely work alone, however its a bit slow...

Joost - Actually I was interested about your opinion, because you are one guy who can utilize the potentials very well. So Elite use supportloops instead. Nice! As I wrote above, I like supportloops more, because they gives even better shading in the corners. And to be honest, I don't give a f about the extra geo, because there are actual examples for doing this in the game industry, and it looks like its completely ok, it it makes it looking better. And also geo is clearly not a problem nowadays. I think more people should know about these techniques, and the industry should go into this direction. Well, hopefully it will. We see more and more examples nowadays ^^ Also yeah, I've heard that Alien isolation used one of these, But I wasn't clearly sure. I add it into the first post. And thanks for the links!

kanga - Actually I have nothing to say, Joost told you everything that you need to know

gnoop - I completely agree, you can simply delete this sync and smoothing / uv split thing from your head, once you start using edited normals or support loops. Long triangles are just bad boys, but you know, you already use a lot of extra geo when using such a smoothing solution, and it won't make a difference, if you add a connect! Just for making triangles shorter. Supportloops gives even better shading, but I mentioned this at least 2 times in this post already

Well, you would be surprised, but its a thing that people ask why you need edited normals at all, but... Where I live, I got "what the hell is this synced workflow that you are talking about" question so much times from "professional" programmers here. I think I don't have to say more As to answer better, yeah, it is really underestimated. Maybe such games as Elite, Star Citizen, Alien Isolation and I could continue, will show them the truth

Edit - Oh I found a few more , better examples in my sketchbook. Some of them use supportloops instead, but its almost the same thing.

-chair, cola machine, and the terminal used supportloops. Only "med-poly" geo, and use of Ndo, no any baking. Again its a very similar thing in functionality.
Unfortunately I couldn't ever finish this scene because of a reason...

This one used chamfers and supportloops too.

And here is my favorite No any baking.

and, as the most complex, finished, and working example, I added some chamfering and supportloops to the lowpoly to this, and it looks like the normalmap of this works in every engine, because of the mostly reduced gradients. I used only one smoothing group. I tried it in UE, Unity,Toolbag (I know its easy there because of the choose-able tangent space ) , and even in blender. This one was baked from a highpoly, and then the purpose to use chamfers/supporting was to make the cleanest bake thats possible. The mesh sits around 25k tris btw.
• Offline / Send Message
polycounter lvl 12
Whoa, this method looks great.

So is the only downside that you are going to have to use a ton more geo on everything with all the extra bevels and chamfers?
• Offline / Send Message
polycounter lvl 9
Kris is always striving to be innovative! Awesome write-up!
• Offline / Send Message
polycounter lvl 13
We use this technique a lot on mobile. This combined with vertex colors... you can get AWESOME lighting cheap.
• Offline / Send Message
Whoa, this method looks great.

So is the only downside that you are going to have to use a ton more geo on everything with all the extra bevels and chamfers?

It actually has the same amount of verts as just using a UV split and/or smoothing split. Or potentially less if you account for tri-stripping? I think the main issue performance wise is the long and tiny triangles, though I suppose it's easy enough to LOD those out fairly quickly. And again there's definitely some workflow issues.

I like supportloops more, because they gives even better shading in the corners.

I personally don't think support loops are more aesthetically pleasing, you get a lot of black shading which isn't there with FWVN. And there's the silhouette. And I don't think the extra geo is negligible. On the other hand it can be significantly easier to edit the loops compared to chamfers. That's why I prefer using quad chamfer as in the video above with variable edge width by using edge creasing. I did make a little presentation a while back on this subject which I might share if anyone's interested? Another benefit of face weighted vertex normals is that apparently it looks better in VR as well, so it might gain popularity.

Also FWVN alone and FWVN combined with normal maps are 2 separate things. The obvious benefit for FWVN alone is that you don't need a normal map and for example when you have massive structures you can't really get away with just using a 2k normal map. As with anything it does have trade-offs though.

When you use FWVN with normal maps you get bigger UV islands(unless you're using 1 smoothing group), better silhouette and no gradients on big surfaces. Which shouldn't technically be an issue if you're using a synced workflow( a lot of people still don't!) but you do get better results with mipping if you don't have gradients and I think you get cleaner results with programs like DDO(not 100% sure on this one) The downside is that chamfers are a pain in the ass to model and unwrap.

edit: Another image. There's other factors to consider, like UV splits, having a lot of thin and small triangles at a sub pixel level and tri stripping ( which I don't fully understand tbh) but it might help clear some things up.

• Offline / Send Message
ngon master
Thanks guys.
So what I understand is you are modeling an asset and editing the normals. To help with the normal editing you add support loops or breaks between surfaces to allow you to edit the normals freely, like in the examples you show above. For the high frequency details I guess you could just use ndo, or ddo or substance designer/painter to do the rust on the bolts etc. I suppose those texture apps support smoothing groups. So the advantage is not having to make a high poly and a low poly, just a mid poly and work the groups.

Above there is a pic of a little helicopter. The smooth cylinders on that model blow my mind. I would like to see a wire frame of that. You are also saying that the extra geometry is not a problem for modern game engines? With something as complex as a sci-fi character or say a mech, would this technique be advantageous?

This is a very interesting thread.
Back to the wiki

Edit: I see I posted the same time as Jooost who provides more info.
• Offline / Send Message
polycounter lvl 4
Wait, wait - so all those examples only have one smoothing group per object...? Sorry for being dense, it's the first time I ever heard about this and it sounds like a revolution.
• Offline / Send Message
grand marshal polycounter
Yes, I used one smoothing group on all off of my examples.

Joost - Yeah silhouette is an anther question. It also a lot faster to do than FWVN. I think it looks better beceuase you get the "fat" looking rounding, as your example also show it. I would say its becuse of the more geo, and you get better light interpolation (?..I'm not sure in this). Probably a face weighted double chamfer would give the same. I do use one sg when use either a FWVN or supportloops.

Here is a picture that is showing the vertex normals in the different cases. Basically you get "fwvn" by default with supportloops. I know its actually not that, but it looks like.
• Offline / Send Message
You don't HAVE to use 1 smoothing group per object but I guess you could. I wouldn't recommend having really tiny bevels that you wouldn't ever notice.
• Offline / Send Message
grand marshal polycounter
That can be true. But. There is a recommendation for using a bit of exaggeration when using rounded edges with normal maps. The same applies for details in general in my opinion. And I apply this on every 3d art, when its possible, whether its modeled or normalmapped. I try to make the bevels visible from further too, so tipically they have 1,5 size or more compared to how would be actually. I do this less, when we are talking about something that is always close to the camera.

Anyways, I really don't like the hard edge look. But I keep them wen needed. Luckily its rarely needed. Also, people shouldn't go crazy and bevel every single corner, sometimes it better to simply use an individual element. But this thing is connected to an another topic.
• Offline / Send Message
Yeah I know it looks better, and I'd probably do it myself. But it's worth considering that if you do it to every edge in every asset you could run into some performance issues in a production environment.
• Offline / Send Message
polycounter lvl 4
I'll definitely be trying this for this month challenge! What are you refering to with FWVN? I'm guessing the last 3 are weighted vertex normal but I can't figure the F. Edit: FACE!!!
• Offline / Send Message
grand marshal polycounter
Well, you can always use LODs, and then it shouldn't cause any issue,but without that, you are right ^^ And you have nothing to do with these methods when its an organic shaped asset. Its more useful on hard surface stuff.

Mant1k0re - Its face weighted vertex normals.

And about the challenge, I will do the exact same thing if I will have enough free time
• Offline / Send Message
polycounter lvl 8
@Joost

I just tried doing FWVN and it's pretty awesome. I like how with this method you have a lot of control over which edges have chunkier edges + editing the vertex normals by hand. This seems like a step up from just hand chamfering edges and then doing vertex normals since that kind of seems destructive in comparison.
The downside is that chamfers are a pain in the ass to model and unwrap.
Besides this one thing has got me wondering about the workflow. Say for example there are certain objects in your UVs that have wonky UV shells(all that hard surface organic zbrush stuff) because of being relaxed, that would be impossible to hand paint normals onto the surface. So I suppose hard surface objects that have organic shapes should have a proper high poly instead?
• Offline / Send Message
grand marshal polycounter
tynew - You simply need a proper low or call it "med poly" topology and edgeflow. You don't need any highpoly if you can make proper "med poly" meshes. Chamfers are pain to uv because you can't break them in the corners, because you don't have an edge there. Also, its very hadrd to LOD properly and efficiently, it will need many adjustment after removing the chamfers. You can build a relatively fast working method around this, by pre thinking on the lods, and build the 1st lod to be good for simple edge removing. This is almost impossible with chamfers. But...Optimizing the support loop method is way easier. you can have a hard edged simple low poly version without the loops as lod1 for example. From here, its pretty much the same as usually.

"hand paint normals" - wut?
• Offline / Send Message
polycounter lvl 8
Obscura wrote: »
"hand paint normals" - wut?
I was meaning painting normals in ndo2, as that seems what is necessary with something like this workflow.

Might be a bit hard to explain what I mean. I'm thinking that using this workflow would be very hard to do with objects like this:

The UV's would be warped so any kind of micro details like paneling and indents look like they must have an actual high poly model instead of painting the normals in ndo2 because they would be warped UVS.
• Offline / Send Message
grand marshal polycounter
Oh I see what you mean... You can still model everything out on the med (low) poly mesh, and use chamfering or supporting, but it will take like 100k tris at the end for the full character, but!... You can take advantage of the tiling, highres materials, and masks, and there are so much examples in this generation of games for such high tris count on main characters. Also for the smaller details like screwing, you can use the same, as what was used on the Star Citizen ship. Decals! You can have a map for this kind of things, a collection. And you can simply put a plane onto your mesh, and uv it to the proper place on the map. You can also bake just parts. I would bake the "cushion" for example to a med poly mesh.

Alternatively, you can make an "unified" med poly mesh with chamfers or supports on the corners thats visible on the silhouette, or important for shading, and do the "painting". That can also perfectly work. I don't know which would be faster, but the second would result less geometry.

Or... You can do the usual workflow, but add chamfering or supporting at the corners, so you don't have to be synced, don't have to add hard edges, etc, it will bake perfectly.

Also, cheat with the uvs to work better for you! you can make it work with "painting".
• Offline / Send Message
polycounter lvl 10
Wow this is pretty eye opening, thanks for the tips. Are custom vertex normals as easily achieved in other programs like Maya?
• Offline / Send Message
polycounter lvl 11
Used in conjunction with a decal technique its awesome for ships and environments!

I wouldn't use this method for characters the vert count gets pretty high, and this technique doesn't LOD well.
• Offline / Send Message
polycounter lvl 5
For Blender users, the Blend4Web plugin allows you to manually edit a mesh's normals.

Barely touched the stuff yet, just if anyone's interested
• Offline / Send Message
grand marshal polycounter
Just a remark and heads up, edited normals doesn't work well with mirroring and symmetry, at least in max. Reseting the transformations doesn't help. So if you are working with a model that has mirrored or symmetry-ed parts, then its better to collapse this, and reset the transformations, and editing the normals just then. Detach also seems like messing up the normals.
• Offline / Send Message
Detach also seems like messing up the normals.
I think this is a Max 2015 bug. 2013 never did that...
• Offline / Send Message
grand marshal polycounter
Its happening with me in 2013.
• Offline / Send Message
Thanks for posting this! I'm definitely going to incorporate this into my work. This has certainly made me feel more confident with vertex normals, demystifying them a bit.
• Offline / Send Message
polycounter lvl 11
How would you go about doing this on a curved surface, like the sides of a cylinder? These demonstrations are all on cubes.
• Offline / Send Message
grand marshal polycounter
Something like these?

• Offline / Send Message
polycounter lvl 8
Just wondering does anyone know a fast way to assign normals by hand in max? Without using scripts it looks like I have to select individual faces one by one to choose what I want to average which seems tedious... I can't seem to use face to face/edge loop tools for quick selection.
• Offline / Send Message
interpolator
There is Normal theft script somewhere on script spot. You just need to make densely tessellated copy of your object and borrow average normal directions from it

pc. Actually in many cases you even don't need to tessellate, just keep bevels with hard edges on reference object.

This method actually is not a complete solution. You just transfer those problem making normalmap gradients from being evenly distributed to those small bevels only. Inside those bevels it may still be nasty.

So in some places a hard edge could give you better result since with enough UV padding and UV islands/borders in same direction you could get more perfectly beveled looking result without any actual geometry bevels.
• Offline / Send Message
grand marshal polycounter
@gnoop - I was thinking on using that with the custom normals method, but unfortunately its ruining the best side of the technique. That, you need only one model. However, sometimes it can help, and you can make part meshes to steal from. But I don't think that would give perfect result.

@tynew - Sorry, it works like that. You can do quick selections in ortho view, and manipulate multiple normals with hand then. This works mostly when an edge is parallel to an axis. Unfortunately averaging rarely works, probably because of how it does its job "AVERAGING". Lets say you have a corner with a chamfer on it. There will be one face, a chamfer, and one face again. If you would like to average the normals on one face , it won't make the normal exactly perpendicular to the face, because of the neighbour face's direction. So I rarely use it, if I do, I do it only for making them a bit closer to what I want, so I will have to do a bit less moving with hand.
• Offline / Send Message
tynew wrote: »
Just wondering does anyone know a fast way to assign normals by hand in max? Without using scripts it looks like I have to select individual faces one by one to choose what I want to average which seems tedious... I can't seem to use face to face/edge loop tools for quick selection.

I'd recommend using the first link I posted on the previous page.
A possible workflow would be to assign a different material ID to the main faces to run the script on. Could also use selection sets instead but they seem to break often. Alternatively the other 2 scripts do an automatic averaging for all faces based on face area. Which does a decent enough job in a lot of cases.

And then there's the quad chamfer method which works really well IMO.

There's no clear cut workflow though. You'll have to construct your own workflow if you want to use FWVN. And again I'd like to stress that there's definitely a lot of drawbacks to FWVN but if done right it can produce some very nice results.

http://www.polycount.com/forum/showpost.php?p=1199836&postcount=41 This one is fully manual. Slower results but much cleaner.
http://www.polycount.com/forum/showpost.php?p=2230470&postcount=28
https://github.com/ByteHazard/wnormals
• Offline / Send Message
polycounter lvl 8
@Gnoop
I tried slide normal thief with a low poly cage and it worked pretty well giving me the same top result as my below image. Although now I can't replicate that anymore for some reason.

@Obscura
What method do you use to adjust your normals, just moving the handles by hand?

@Joost
I have max 2012 I tried running the first link script in maxscript editor/dragging into viewport but nothing happens. Could you explain your process for using the quad chamfer method? I tried doing the same thing as your video but It still gives me the flat shading problems?

I don't really know what I'm doing but here is what i tried anyway:
• Offline / Send Message
Under the shading tab change it from manual to sharp boundary and set round shade to 1.0

The script works on face selections. So you make a face selection and then press a hotkey to trigger the script. After dragging the script into max it will appear in the Normal Tools category.
• Offline / Send Message
grand marshal polycounter
Yeah, I mostly move them with hand. I was thinking on a workflow where you add chamfers, and supportloops next to the chamfers. Then making the normals stay in direction somehow, removing the chamfer, and you get a new chamfer from the supportloops, with proper normals. Unfortunately I couldn't make them stay in the proper direction. After removing the original chamfer, they are all messed up. Any advice on how to make them "freezed"?
• Offline / Send Message
polycounter lvl 8
Thanks Joost, that's exactly what I needed

One final question is what should the geometry/edge flow be like using this? The overall shading is great (dont mind the really dark parts down the bottom that's the cubemap horizon/grass) but I seem to get some artifacts where edge loops are.

• Offline / Send Message
interpolator
There is a convenient approach for this in Blender. I have pretty old Max so probably it would work in modern Max too.

You make all the edge bevels in modifier . Blender allows to make every beveled edge having its own chamfer width by "weight" adjusting in real time. Then make an object copy and turn bevels/chamfers into multi-segment curved ones (one click). Rarely a few extra sudivision/edge loops might be also necessary. Then just transfer normals or export both in Max and use normal theft script there.

I had been always doing the transfer in Max before but right now Blender seems like doing correct fbx export of normals and its data transfer modifier works no worse than theft script. I would say even better since it transfers in real time like Maya.
• Offline / Send Message
tynew wrote: »
Thanks Joost, that's exactly what I needed

One final question is what should the geometry/edge flow be like using this? The overall shading is great (dont mind the really dark parts down the bottom that's the cubemap horizon/grass) but I seem to get some artifacts where edge loops are.

Are you referring to the sharp vertical lines?
That's just what happens when you have 2 edges close to each other. Not a bug, just a feature?
• Offline / Send Message
Is it possible to use this in conjunction with a normal map, for smaller surface detail?

If yes, are there any workflow adjustments and/or import/export settings that need to be done in a specific fashion?
• Offline / Send Message
grand marshal polycounter
Yes, its possible! And actually this method works the best in the way you imagined. Then it gives the exact same look as if you would have bake. In some cases you need Ndo or such.

After making the edit on the normals, the only thing you need is to export the normals too with the model, and on the import, enable import normals. Nothing more needed.
• Offline / Send Message
Okay so let's me get this straight.

1- model lowpoly
2- bevel edges
3- manually align vertex normals as desired
4- make UVs
5- triangulate and flip edges as needed for nice shading

6- export phase! FBX settings:
-Smoothing Groups -> YES
-Split per-vertex Normals -> ??
-Tangents and Binormals -> ??

7- Open NDO or similar software, create new flat normal map (right?), add detail where needed

8- Import model in your engine. Import settings?
• Offline / Send Message
grand marshal polycounter
Exactly. For the fbx exporter, I only enable smoothing groups. For imporing, you only need to enable import normals.
• Offline / Send Message