Smoothing groups, Tangents and Binormals, Triangulate
polycounter lvl 6
To be clear, I know what to do if I want to get my low/high models from Maya to Substance/Unity.
My question is about why and what:
- So Maya stores "hard edge information" instead of smoothing groups, does it mean Substance don't support them? or does it mean most game engines only use smoothing groups anyway?
- As far as I know Unity use the same tangent space as Substance and Maya nowadays, so is there any benefit to check this option? Even re-calculation should result in the same Tangents and Binormals right?
- Apparently Maya 2018 no longer likes both "Triangulate" and "Smoothing Groups" checked, saying it's "unsupported". Was it always a hack to "triangulate on export" instead of "triangulate before export"?
My question is about why and what:
- So Maya stores "hard edge information" instead of smoothing groups, does it mean Substance don't support them? or does it mean most game engines only use smoothing groups anyway?
- As far as I know Unity use the same tangent space as Substance and Maya nowadays, so is there any benefit to check this option? Even re-calculation should result in the same Tangents and Binormals right?
- Apparently Maya 2018 no longer likes both "Triangulate" and "Smoothing Groups" checked, saying it's "unsupported". Was it always a hack to "triangulate on export" instead of "triangulate before export"?
Replies
There's also a bug in Maya 2018 that causes crashes if you have tangents and binormals checked.
IIRC Unity uses mikktspace (Mikkelsen works there), but Maya doesn't(?). There may be differences in the ways that the tangent space is calculated. I haven't tested it thoroughly but now I'm curious. Either way, mikktspace allows you to override it with your own ts and bs, so if you bake with them and import them it should be fine.
Personally, I would bake in a mikktspace software, such as xNormal or marmoset toolbag and recalculate the tangents and binormals on import into Unity. That way you'll be sure your spaces are the same.
Edit: I'm also unsure about the smoothing groups problem. I don't even know what software supports them besides Max. Hopefully someone else can chime in. The documentation page is fairly vague.
In the end, they're just vertex normals. The way to set them is just handled differently in Maya/Max.
As for the tangent space, what really matters is that it is the same between your baker and your engine.
Both unity and substance can override the tangent space with mikktspace which is the industry standard.
No idea what space maya is using, but it doesnt matter if you override it,
You don't have to export tangents/binormals . Just check calculate tangent space is on for your mesh in unity, which is the default behavior i think.
If Maya uses mikktspace, you can export them and use import tangents in unity, but it should be exactly the same, so there's no point to do it.
If it uses something else, you have to be sure to keep mayas tangents/binormals in both baker and engine.
I tried exporting sample models with/without Smoothing Group option checked, but honestly can't tell if it makes any difference in the baking result (SP still identify the hard edges properly).
SG just stores groups of faces that should have averaged normals between them, not the normal vectors values per say. It's specific to Max and I get how it's confusing.
But the vertex normals data are always stored in the fbx, SG checked or not, and that's what substance is reading, hopefuly.
This option is probably only there to make a bridge between maya and max, if you want SG back in Max from the fbx.
- Hard edges for highpoly to lowpoly baking are not just a personal preference, they are an essential tool to create clean bakes for some assets.
The best approach is not to use them "just because", but rather, to clearly understand why they are useful. But of course that is only possible once personally coming across an asset that would require them in the first place (not all assets require hard edges).
Now all that said, I personally don't bake in SP (I don't really see the point of it at all, when Toolbag3 is an all-around far superior solution) so maybe there are some quirks introduced by SP itself.
My question is specifically this: https://forum.allegorithmic.com/index.php/topic,19167.0.html (I am posting on Allegorithmic forum because ultimately the question is about SP).
BTW, to all, we have 2 possible regressions in Maya 2018 (Autodesk is working on a fix):
(1) "Smoothing Group" + "Triangulate" on export causes warning.
(2) "Tangents and Binormals" + "Triangulate" cause crash.
They change Triangulate method and ATF plugin in this release so things are a bit messed up.
Personally I feel like AD still modifying their FBX import/export plugins after all these years is a huge red flag. It's especially annoying when a modeler uses the latest FBX exporter version "just because", resulting in files incompatible with some third-party apps in a pipeline.
FBX is a fantastic concept, but executed poorly. Tread carefully, and stick to plugin versions that work well for your needs. IIRC older versions of the I/O plugin (2012 ?) have all the features needed for baking, and there's also no need to use the latest and buggiest AD software in the first place. Find something stable (ideally one or two revisions back) and stick to it for as long as possible. No need for the latest Maya version either, it will always be buggy so why bother ?
You could run something like Maya 2015 or whichever version has a good track record, along with a solid FBX plugin itself from a year or two earlier, and you'd be pretty much guaranteed that it would work everywhere.
I think a shift in that regard is bound to happen sooner or later. FBX may be a widespread standard at the moment, but who knows what could happen in the near future...
I guess what I am trying to get at is that while they are a thing on the Max authoring side of things and as you just confirmed are also present as a face property in the FBX data itself, there's is no guarantee that a third part app interprets them as such - for instance, a non-Autodesk program could have a tickbox saying "import smoothing groups", but effectively just reading the vertex normals information, and skipping/ignoring the SG flags altogether ...
The part that trips me out is ... why does the FBX Maya exporter even have an option saying "export smoothing groups" when Maya doesn't have a way to create/store them to begin with ? That means that this option either infers them from the hard edges, assigning face SGs at random (in which case it should be called "convert to SGs" instead) ; or, maybe it leaves them empty and simply writes vertex data.
Either way the current naming of the option doesn't really makes sense to me, and from there, it being buggy is not really surprising
//There are two kinds of smoothing info:
//1. Smoothing groups info which is saved by polygon. It usually come from 3ds Max, because 3ds Max can set smoothing groups for polygon.
//2. Hard/soft edges info which is saved by edge. It usually come from Maya, because Maya can set hard/soft edges.
I've still never used max, is it possible for smoothing groups to overlap "just right" so only certain edges are hard? In maya you can do something like 3 edges in a loop as hard, never been sure if you can do that in max.
anyway, one can save a fbx to ascii format to see whats in there with the different options.
My own cube test shows no difference whatsoever (intentionally trying to bake smooth bevels down to hard edges OR soft edges).
I'm very sorry for the delayed response on our forums. I posted there as well.
Substance Painter will read the soften/harden data set in Maya when you export as FBX. This data is used to shade the mesh in the Painter viewport similar to maya based on vertex normal smoothing. In maya you are smoothing/hardening the vertex normals. Smoothing Groups are used in FBX file format and it's useful to have it enabled for other software that supports it such as 3ds Max. However, this option is not directly used by Painter if you export from Maya. It doesn't cause a problem to have it enabled if you are using the same fbx in Painter and then again in other applications.
I am also getting the issue where Maya 2018 reports triangulation and Smoothing Groups is not supported. I'm not sure what's going on there as it works in 2017. Also, with Painter we currently support FBX version up to 2017.
Please let me now if that doesn't make sense and I'm very sorry for the confusion in the video. I should have left the smoothing group part out when discussing the FBX export options since I was using Maya : )
Cheers,
Wes
(Also a good piece of information to know SP supports up to FBX 2017 so far)
(BTW, if you read a few replies back, that Maya 2018 export warning is likely a bug, at least they are investigating internally.)