Normal maps and smoothing group woes
polycounter lvl 8
...one more time!
Asked about this before, but I'm afraid I still don't get this...sorry! Where exactly should one place smoothing groups to make normal maps look nice?
Here's a mini-example. On one screenshot you can see my lowpoly corner with the checker map on it. On the other one is the same piece in Unity3D with normal map on it. The normal maps come from ZBrush where I slightly rounded the hard corners, applied noise, etc.
It's not totally horrible over all, but I do get those nasty dark areas close to the corners.
Is this even a smoothing group problem? I have tried just about anything - no groups, one group for everything, separate groups for about 90 degree angles... The one in the screen shot is using one group for all and it gives the best looks in Unity, but I wonder if I could make it look even better?
Asked about this before, but I'm afraid I still don't get this...sorry! Where exactly should one place smoothing groups to make normal maps look nice?
Here's a mini-example. On one screenshot you can see my lowpoly corner with the checker map on it. On the other one is the same piece in Unity3D with normal map on it. The normal maps come from ZBrush where I slightly rounded the hard corners, applied noise, etc.
It's not totally horrible over all, but I do get those nasty dark areas close to the corners.
Is this even a smoothing group problem? I have tried just about anything - no groups, one group for everything, separate groups for about 90 degree angles... The one in the screen shot is using one group for all and it gives the best looks in Unity, but I wonder if I could make it look even better?
Replies
Thanks for the tips. I went ahead and made up another, more clear, example.
I am basically aiming for smooth, soft corners look in this example. It works at some places and fails miserably in others - and I still don't understand the logic behind it :poly122:
I modeled this in Max, took it to ZBrush and slapped some noise on it to see the effect. Then I took it back to Max. I cleared all the smoothing groups and then used auto smooth with the default 45 as a starting point. Then I assigned common groups for those corner areas (5 in the screenshot) and that's pretty much the result. The window holes are not done, but tips for doing that are also most welcome.
http://www.polycount.com/forum/showthread.php?t=77539
This thread explains it pretty well, so does the normal map section on the polycount wiki.
I redid the groups, they didn't match the UV shells completely, but now they do. Yet, I don't think the result looks any better... I guess there's something else that needs to be fixed as well?
Like for instance, are my UV seams badly placed? Does it matter that much?
I'm not totally sure what I should do with my normals, but I added an edit normals modifier and broke the normals in the corner and averaged them (shown in green).
Unfortunately it wasn't quite that easy - you can see the unmodified object (same corner) there in the screenshot and if I deselect them both, I can't really tell a difference on the seam.
Am I totally on the wrong track here?
I have tried pretty much everything suggested here, but I still haven't got this right.
After frustrating the heck out of me with smoothing groups and edit normals, I tried one more time to put everything on a single smoothing group - this is what it looks like.
What surprises me, is the fact that one of the sides looks perfect - just that smooth, round transition from wall to roof what I was looking for. And the other one doesn't. Yet they are both separate UV islands as well as the roof itself. This just drives me up the wall!
If you brake the normals, and later average them, it's the same as not braking at all.
And if you use auto smooth groups at 45, your smoothing groups won't match the uv borders.
You should select the same polygons of your uv groups and arbitrarily assign a smoothing group.
Once you make this, you have to rebake your normals. You will not be able to use normals baked for a previous version of your mesh, with different smoothing groups.
Ah, that is definitely mistake #1 - I actually though it doesn't need re-baking!
@Vig,
I have been baking the maps in ZBrush and using GoZ to bring them back to max. It creates a DirectX -shader, which I disregarded at first, just taking the model and the normal map to the engine. But then I noticed that changing smoothing groups or tweaking the normals gives instant results on the viewport, so I kind of bypassed the engine part. Anyways, good point, maybe it's not the same at all, I will need to check all changes in the engine (Unity3D) as well.
This reminds me of one thing - is there currently a solid way to get the smoothing groups into ZBrush in the first place?
I learned from ZBrush forums that the old Smoothing groups plugin for ZBrush 3 still works in Z4, and I installed it... - it does prevent smoothing as you would expect, but otherwise I'm not 100% sure if it works with the baking as well (since I just learned from Pedro that it matters too).
Back to debugging....
I sometimes get an error about "turned edges" when I export models from Max.
"The plug-in does not retain the orientation of Diagonals on Editable Poly objects. This could have an impact on modified UV coordinates. Refer to the Online Help for more information."
What exactly does this mean?
So, if you have already baked the normals of your object, and for some reason the diagonals are modified after that, you may get unwanted results.
As for the ZBrush question, I don't believe there's a way to tell ZBrush to take the smoothing groups in consideration when baking normals. ZBrush does not use any smoothing really, that's why the lower subD levels are always segmented, faceted. What it does when you import an object with smoothing groups is to make sure the borders will remain sharp once you start subdividing. But the shading is not affected and I don't think this information is stored at all. I may be wrong though, I never bake normals in ZBrush, I prefer baking them in Max.
So do you use ZBrush at all? I'm just getting started with that and I used to bake in Max, too. For the little tests that I've done so far, I just got the idea that it would be better to bake in ZBrush, because the hi-poly meshes are so huge. I tried to bring one into Max for baking, but it just froze.
I could of course use decimation master, but in this case with surface noise I think it really needs all the polys it can get.
I have also tried xNormal several times today, but I'm doing something wrong obviously, because it won't calculate the ray distances (everything stays at zero and I get blank maps). Yet I have successfully used it with meshes exported from Max only.
----
I just did one last test for tonight. I put everything in one single smoothing group and added edge loops to hold the geometry together in ZBrush.
It looks pretty good in Unity, but are there some drawbacks for using this instead of smoothing groups? At least the new loops show slight stretching in ZBrush - and the end result, but I might have messed up the UVs as well...
The drawback of having these extra edge loops is to have a higher polycount at the end. But if optimization is not an issue on this project, it may be a good solution.
That part-by-part approach is worth a try too, but perhaps not in this case, as the model is really a very organic looking building - almost like a soft edged sand castle - and is all one part.
I used the edge loops only in the sculpting phase - not in the engine! Their purpose was just to hold the model together when subdivided in ZBrush - the "sculpting mesh" also included a bunch of loops to prevent elongated polygons. I delete all of these when making the in-game mesh.
Of course, one must be careful when altering the model, so that UVs would stay intact and the baked maps would still work. This is actually another thing I've been trying to get my head around, with partial success so far. I was asking about this in another thread during the week.
If you are baking your normals in ZBrush using a mesh with more loops than the final low poly mesh, even though the uv's may be the same, the map will not work correctly.
See, normal maps are not like any other map that are just visual information applied to the surface.
Normal maps work based on the meshes' normals. They can not be successfully shared among meshes with different topology, because the normals of these meshes will probably be different.
So, if you have more loops holding the shape on the mesh you are using in ZBrush, and you are baking your normals there, with this mesh, and then applying this map in Max, or in the Engine, on a more optimized mesh, the normals map will not have been made for this mesh, and will not look correct.
And as for what I was saying about baking it on separate passes, exporting parts of the object from ZBrush, I have done that for organic models too. Creatures, characters, You just need to hide parts of the object, in the case of the characters, I would export the head and shoulders, torso, legs, and I would make sure each part has a bit of the border of the other parts, so the head and shoulders has a bit of the upper part of the torso, the torso has a bit of the upper part of the legs, so they overlap. and then put everything together in Photoshop later. But only if the decimated mesh did not hold enough information.
However, I'm not quite sure if it's that sensitive - I might be totally wrong here, so please correct me if that's the case, but I made a little test.
Here's a concrete block. A very basic boxy shape that I made up in Max and then took it to ZBrush for "sculpting". I tried to do everything the same way as with the previous tests. The model that went to Zbrush had a bunch of edges giving it fairly even geometry. When the maps were done I imported that into Unity and applied the maps - it's the one in the background.
Then I took the away most of the extra edges, took that to Unity and slapped on the same normal map - it's not identical, but it's not a complete mess either, or what you think?
Or is this some special case where the problem is just not that apparent?
In case of a more complex mesh you will probably have issues.
But, if that was not the case, if the mesh is a little more complex, you will probably find the need to use the same mesh for baking normals and displaying them. Because the normal map needs to work in sync with the mesh.
I was so concentrated on thinking about flat walls, roofs and floors for this current project, that I didn't really think in the context of more complex surfaces. Anyways, it's probably best to start baking in Max anyway - it seems way simpler approach for this.
I just went and got myself a tutorial from Eat3D on this very subject...
Which training video did you get from Eat3D on the Normal's subject you talked about? I am interested in this as well.
Thanks
But read the description before you buy! It clarified the baking process for me, actually showing 2 ways to do it, but it's not about smoothing groups.
Just reworking an old model to see if I can minimize the seams and - surprise - I bumped into this topic again!
So once again I googled about smoothing groups and I found two articles that really got me confused, here and here.
Both seem to claim that you should only have one smoothing group - is this complete rubbish or what am I not getting here?
quotes:
"It is generally better not to use smoothing groups or hard edges to add definition to the low-poly game mesh."
"This means you can use bevels to give you better visual fidelity while delivering the same performance cost as smoothing groups... the triangle count is higher but the vertex count is actually what matters in the game, not the triangle count."
"Up until now, smoothing groups (or hard edges) have been a good way to accentuate features of a low poly model and make the details more clear and readable. It used to be important to use smoothing groups carefully to create a good model. Throw all of that out the window. Smoothing groups are an enemy to normal maps. Your low poly model should have one smoothing group (no hard edges)"
This is the most accurate quote. It's not a case of "don't use smoothing groups". You just have to keep in mind that they add extra vertexes which takes up more memory.
It's all about the vertex/polygon ratio. If your model has more vertexes than polygons I'd look into how many smoothing groups you've got and see about reducing them.
Keep in mind that max doesn't display smoothing groups like a game engine would. You need to test your work in a game engine to get the full picture.
As a hypothetical example a lot of the time I'll come across a mesh someone has made that's say 4000 verts and 2000 polygons. So I'll open it up and make a bunch of pieces one smoothing group and re-bake the normal map. This might make it about 2000 verts and 2000 polygons. It looks almost the same but is now half the memory footprint.
It is true that if you can use one smoothing group do so, smoothing groups is not something you add because it's fun it's because you need them.
And bevels make things look prettier most of the times, instead of a hard edge.
I use smoothing groups frequently more than bevels, cause it makes it much easier to texture atleast on hard surface objects.
I hope you understand what i mean, sorry for breaking things up
A. Vert count is important, but triangle count is also important, vert count may be more important but traingales are certainly not free. Adding bevels instead of smoothing groups will result in worse performance, this is a fact. It may not be that much of a difference in the end result, but it certainly isnt free.
B. Long-thing triangles(created by small bevels) are actually a bottleneck in rendering, so they should be avoided whenever possible.
Now, contrary, taking a model with 1 smoothing group, and a model with hard edges along the UV seams, there IS NO PERFORMANCE DIFFERENCE HERE. Because the verts are split at hte uvs regardless, and not split a 3 time when using a hard edge/smoothing group.
Personally I do a bit of both, however as long as your bakes are set up correctly, you can get seamless results using smoothing groups, you will have cleaner textures that compress better, mip better, and are easier to generate detail maps(crazybump) from.
So to recap, using smoothing groups properly has absolutely no negative side effects when done properly, and offers better performance. The main issue is that its a bit of a pain in the ass to set up, however if you have a simple script to assign hard edges along uv borders, it will save a huge amount of time.
I've attempted to script something and been thoroughly lost in how they handle smoothing groups and its nearly impossible to find UV boarders, its as if they don't actually exists in maxscript... there was some ways to get pelt seams but not everyone uses pelt seams. It has been suggested to autodesk a few times on a few forums but no official word yet on something being included or exposed to maxscript so we can script it. Fingers crossed...
for 3dsmax, www.scriptspot.com
for Maya, www.creativecrash.com