Smoothing groups and Substance Painter Normal Map Baking...

KnobbyNobbes

OK, so, I am really doing my best to figure out all this stuff on my own/by reading discussions on this, but I think I need someone to dumb this down a bit further for me....

So, my problem is that currently, I am trying to model and texture a low poly sword in Substance Painter (because Quixel's inability to handle texture seams is driving me away from it currently). So, I follow the fundamentals tutorials for substance painter and I get to the baking textures section. I take my model, bake the textures, and see that the normal map has errors along the corners. Specifically, it looks like someone took a needle and thread and did some stitching along the edges. So, I google what the problem might be, and discover the mysterious term 'smoothing groups'. Determined to learn how to solve my problem, down the rabbit hole I go. Here I am two days later and I am just as confused as I was at the start. I came across the "Your making me Hard, Understanding hard surfaces w/e" polycount discussion page and tried to incorporate what I said, but that does not seem to fix everything, thought it does get things the closest.

So, here I am to give an accounting of what I have tried and what I think is going on, and hopefully someone here will tell me what I am not understanding, or what I should do differently.

Here are all of the different methods of smoothing groups I tried to apply to my lowpoly mesh in order to fix things, and the resulting bakes. I also tried at one point to change the smoothing group of the high poly mesh, but that just crashed Maya.


Here is the low poly model

Here is the high poly model

And here is where the UV seams are...


At this point, I really don't know where I should start trying other things. As far as I can tell, I've tried most of whatever suggestions I've seen others suggest online, other than just rebuilding the whole thing. Any help would be greatly appreciated.


Also, in a separate but different issue, what is up with the curvature maps that substance painter bakes? Its like it was drawn by a kid with a crayon compared to what comes out of xnormal. Is it supposed to be that way? Would Xnormal's type of curvature map not be enough or something?

pior

Let's do things in order. Here are some steps :

- On your model, make sure that IF you put a hard edge somewhere, it is on an edge that is ALSO a UV island border. Do not place a single hard edge "at random" without making sure that the edge matches this condition (and yes, that might mean that your UVs might need to be redone accordingly )
- Make a cage model for ray casting (Xnormal will need it so that rays are not split where hard edges are)
- Bake the tangent space normal map in Xnormal
- Check the results in the built-in Xnormal viewer
- Post your a screenshot of your result here.

Once all that is correct you will be on solid grounds to move to Substance Painter (And yes it might be possible to do the bake directly in SP but let's not worry about that for now since it might introduce some new variables like different triangulation interpretations for instance).

Regarding the Curvature map : you are better of using the one generated directly from within Substance Painter, since the dynamic material effects will expect it that way in order to catch highlights and cracks properly.

Good luck !

Jerc

If our curvature feels too strong, simply decrease the "details" parameter in the curvature baking options.

KnobbyNobbes

Ok, so I did what you said pior but things seem to have just turned out worse. There no hard edges anywhere but at the edges of the UV map, and I redid the UV map to accommodate the change. Here is the render from Xnormal Viewer.

And, because it looked so surprisingly bad, I checked how it looked in SP as well, but it still looks worse than what I was trying earlier.

Really not certain what I am doing wrong.

Edit: Oops, forgot to attach the resulting normal map.

pior

Hi - let's dig deeper then Do you have the hi and low files available for download somewhere ?

KnobbyNobbes

Yea, here is a google drive link.

https://drive.google.com/open?id=0Bwy8O94ajvT3S3NHV0hVZlFoSnc

Also, I've been trying out exploding the sword and creating cages of the individual parts, both with bevels on some edges and without, AND both with the hard edge UV trick and without, just to cover my bases for testing. Ideally I don't want to have to explode and cage every mesh I do that has a high poly base, but might as well check. Haven't seen the results of those tests yet, but will see them soon.

pior

Thanks, the lowpoly file explains it all.

Your lowpoly model is made of way too many intersecting parts (as a matter of fact, pretty much every part is its own standalone "floating" primitive at the moment). Start by redoing the lowpoly model as one continuous and watertight "shell", like this:



Also make sure that your highpoly model has nicely rounded edges as opposed to the razor sharp ones you have on some parts, like the curve on the right here. If left that way these sharp parts will not bake well and the resulting model + normalmap will not catch any highlights.

KnobbyNobbes

Ok, I guess I'll try and remake everything tomorrow then. However, I'll just show you the results of trying out various combinations of using beveling/not using beveling and using the hard edge UV trick, and the bakes I got out of Substance Painter as a result.

First, I tried No Bevels and No Hard Edge trick

Then, I tried With Bevels and No Hard Edge trick

Then, No Bevels and With Hard Edge Trick

Then, With Bevels and With Hard Edge trick

Finally, thinking I was about to strike gold, I tried having the Beveled edges (even though they were UV seams) be soft edges and using the Hard Edge trick. Unfortunately, the beveled edges still came out wrong, and I have no idea why.

Out of all my tests, the No Bevels and With Hard Edge trick gave a near perfect result, but I am still a bit hazy as to WHY it came out better than the others. For now, I'll just try and remodel everything and see what I can get from there. Though, I must say, I think I have read somewhere that having everything not be one connected piece was an acceptable way to model now, so long as the result looks good. If that is true, why is that, and if it isn't true, why is it better to have everything a solid piece?

pior

You won't need either bevel or hard edges for this test. The first thing you need to do is to get the lowpoly model done as suggested. Contrary to popular belief there are really no "tricks" when it comes to normalmap baking, especially now that we have access to the unified Mikkt basis.

There are of course many different ways to model (separated parts or not) BUT in your specific case you do need to make it continuous for the sake of surface transitions. Merely intersecting primitive-like parts will look very cheap, and will not behave nicely in substance painter since it is precisely based on interpreting the surface information of models (as well as various input maps : curvature, position, and so on ... ) in order to create convincing material weathering effect.

(And as pointed out earlier you also need to get into the habit of beefing up your highpoly model to avoid razor sharp edges and un-bake-able details like the ridges you have on the handle. So on top of working on the lowpoly model, I would also recommend you to address that too. This is another factor contributing to weak bakes, and taking the time to do it right will save you a lot of frustration in the long run. If anything, for the sake of testing you could simply dynamesh everything in zbrush and relax the result a bit.)

Also stay in Xnormal (baker/viewer) for now, no need to bake in SP. There is no point in introducing an extra variable at this point of your learning.

Good luck !

KnobbyNobbes

Ok, so, I tried doing it your way. I remade my mesh (or at least most of it for testing purposes) as a single piece and no bevels, though I was not able to create a HP directly from that mesh because of smoothing errors distorting the mesh that I was not able to figure out how to fix, no matter where I put supporting edge loops. So, I used my previous High Poly mesh instead, since it still matched pretty exactly, and the results were.... less than pleasing. Its safe to say that they were the worst of my bake results so far.

So, in order to test how you do things, I decided I would just go with a simple box. I decided to put bevels on this box, however, because I already have established getting seamless bakes with no bevels on my previous attempts, and I had heard on a forum somewhere (maybe this forum) that putting bevels on LP objects helps alot with the edges being able to catch light well, which makes sense to me. Anyway, so I made just a simple box, no UV distortion at all. Smoothed it to make the HP. Stuck it into Xnormal..... There is a reason I have been avoiding Xnormal, and this illustrates that reason perfectly. Xnormal, for me, gives at best noisy normal maps, all on default settings, and at worst this horrific result.

So, you can see the huge difference between what Xnormal puts out and what Substance Painter puts out, at least for me.

Then, of course, I came up with the idea of just using the low poly instead of the high poly for baking things out. So, I try it out with this result.

Better results than the previous tests. But again, the seams are visible, but only when painted over. Obviously, I am not aware of some hidden ruleset of normal maps that obviously exist if even a simple box cannot achieve perfect results with my current methods. If it were a problem with the mesh, using a simple box would have solved it and resulted in perfect normal maps. For the heck of it, I tried the new low poly instead of high poly trick on my sword mesh to see what would happen.

Again, fewer errors than the previous attempts, but still errors. I know that you think that my meshes being separate are my issue, but the errors I am finding are at the edges of the solid meshes. Any insight you could provide as to the cause of these edge errors would be much appreciated.

musashidan

Pior meant to retopo your highpoly as a single lowpoly mesh, not rebuild your high entirely! Trying to model your high as a single contiguous mesh is not good practice, especially with sub-d modeling. Also, personally, I would build the low in multiple parts and bake by mesh name, which only casts rays on matching high/low models based on a naming convention. This has the same effect as an exploded mesh bake.

KnobbyNobbes

HA! After TWO WHOLE WEEKS, I have finally come to the answer of what was going wrong! I needed to have my High poly meshes BIGGER than my low poly meshes, or at least to the point where there were no edges from my low poly sticking out of my high poly. ALSO, you need to have all your UV edges set to HARD, so at least I was halfway to the answer for about a week now. Found the answer on this page:  http://docs.cryengine.com/display/SDKDOC2/Common+XNormal+Baking+Errors+and+How+to+Solve+Them

I mean yea, of COURSE it makes sense when you think about it, but it was not anywhere on my radar at the time.

NOW LOOK AT THESE BEAUTIFUL EDGES! LOOK AT THEM!

Though I do have one final little error that is kind of stumping me. Some weird shenanigans going on with the blade that, of course, has not happened until this point. Might just try to remake the cage to see if that fixes it.

pior

Ha ! You probably fixed it thanks to another factor, because ... you definitely do not need the highpoly to be bigger than the low. As a matter of fact you usually want the opposite - that is to say your low being slightly beefier than what your retopology step gets you, in order for the lowpoly model to not look too weak visually.

Do not underestimate the seemingly small issue with the blade. If it's there, that means there is still an error somewhere as it definitely should be flawless. Keep at it !

(Same thing with the issues you are getting with Xnormal - they do not come from the program, but from an error in your setup. The app is the industry standard, literally : it was the first to implement Mikkt tangent calculation, which is the very reason why we can now get flawless bakes. Toolbag, game engines and Substance then followed later.)

Anyways - 2 weeks is nothing compared to the years during which it was technically impossible to get such results because the process was not standardized

[Edit]
Here's an example with the extreme cube scenario - 12 tris, no added bevels, all smooth, no weighted vertex normals (Maya does that by default, and it can be helpful in some cases but that's not really relevant here). Baked perfectly in Xnormal and imported perfectly in SP (SPs calls ydown/yup "DirectX" and "OpenGL", which is definitely confusing for new users. This needs to be set according to the way your bake was made).


Good luck !

KnobbyNobbes

Hhhhmmmmm, So, I've tried remaking the cage. No change. I've tried making the LP blade bigger than the HP blade. No change. Tried making the LP blade SMALLER than the HP blade. No Change. Tried running the original through Xnormal. While xnormal still gives me fuzzy/unplesant results, there was NO ERROR in on the blade. So.... makes me think that there is some issue within Substance Painter that is causing the error? Its weird that the error doesn't even follow any geometry/edges....

Substance Painter:


Xnormal:

pior

While I have never baked normals in SP (never needed it for that to this day) I would assume that its baking tool is accurate since it has long been tested by now. So from there, a difference in result could be caused by the model not being triangulated before baking, with SP and Xn potentially triangulating things differently (for instance I can confirm that there is difference in triangulation algorithm between Toolbag and Xnormal).

So, double check that your lowpoly model is properly triangulated. It could be that, or some error on your model that went unnoticed... And if all fails, you could always upload your two models again for people to have a look at what you currently have.

KnobbyNobbes

That was it! You glorious glorious man. I dont know how, but somehow the HP didn't triangulate completely, even though it looked like it did. So, on your suggestion, I tried pressing the triangulate button on it again and it gave a result! Now my mesh is free and clear and wonderfully bug free. Thank you! Now I get to move past this and bury my head into trying to learn substance painter and designer!

pior

I suppose you meant LowPoly, not HighPoly

KnobbyNobbes

No, it was actually the high poly not being fully triangulated. Low poly was triangulated just fine. Guess Substance Painter is just particular that way.

pior

Interesting ! There was probably something a little odd about the model in that area, but by all means it's good to hear that it fixed it for you

kanga

Just ran a bake test in SP2 on a simple door. Everything in one smoothing group, set to smooth. All one mesh but with floating and intersecting geometry. This model was not exploded! The test bake in xnormal didnt work as expected due to the intersections I guess. The fbx version of the model didnt bake in substance but the .obj version did, and pretty damn well I might add. I was just curious if it would work. I triangulated the mesh with a modifier and exported with the modifier applied in Blender so I could more easily edit the mesh without triangulation later if I needed. The applied shadows in substance look crappy but the shadowing in UE4 is perfectly fine.





In Unreal from a little distance the hard edges on the metal parts are perfectly round, close up the metal bits look hard but baked properly with no seams. The wooden slats are all seperate boxes and have beveled edges, the 2 horizontal  pieces totally intersect with the vertical ones! I let SP2 calculate the tangent space per fragment as well. There is no intersection between hi and low cages per piece. Even as one model you can mask via uv island so adding detail without affecting other components is a breeze. Not modeling the lowpoly as a single sock saved loads of time and made uving very quick. The bake was not done using the splitting naming convention in SP2. The only lowpoly optimization was reducing the  bolts and extending the apron into the wood and metal parts, plus removing the back polys on the metal bits. The 2 horizontal wood pieces have full surround polys because you see them through the back of the door. They should not have baked, but they did. Not sure this would work with curved intersections, but we will see. Hail SP2.
Cheerio