Smoothing groups, Tangents and Binormals, Triangulate

bitinn

To be clear, I know what to do if I want to get my low/high models from Maya to Substance/Unity.

My question is about why and what:

- So Maya stores "hard edge information" instead of smoothing groups, does it mean Substance don't support them? or does it mean most game engines only use smoothing groups anyway?

- As far as I know Unity use the same tangent space as Substance and Maya nowadays, so is there any benefit to check this option? Even re-calculation should result in the same Tangents and Binormals right?

- Apparently Maya 2018 no longer likes both "Triangulate" and "Smoothing Groups" checked, saying it's "unsupported". Was it always a hack to "triangulate on export" instead of "triangulate before export"?

Axi5

Triangulating via the FBX plugin has caused issues for me in the past. I'd recommend triangulating before export.

There's also a bug in Maya 2018 that causes crashes if you have tangents and binormals checked.

IIRC Unity uses mikktspace (Mikkelsen works there), but Maya doesn't(?). There may be differences in the ways that the tangent space is calculated. I haven't tested it thoroughly but now I'm curious. Either way, mikktspace allows you to override it with your own ts and bs, so if you bake with them and import them it should be fine.

Personally, I would bake in a mikktspace software, such as xNormal or marmoset toolbag and recalculate the tangents and binormals on import into Unity. That way you'll be sure your spaces are the same.

Edit: I'm also unsure about the smoothing groups problem. I don't even know what software supports them besides Max. Hopefully someone else can chime in. The documentation page is fairly vague.

Noors

Smoothing groups and hard edges are the same thing.
In the end, they're just vertex normals. The way to set them is just handled differently in Maya/Max.

As for the tangent space, what really matters is that it is the same between your baker and your engine.
Both unity and substance can override the tangent space with mikktspace which is the industry standard.
No idea what space maya is using, but it doesnt matter if you override it,
You don't have to export tangents/binormals . Just check calculate tangent space is on for your mesh in unity, which is the default behavior i think.
If Maya uses mikktspace, you can export them and use import tangents in unity, but it should be exactly the same, so there's no point to do it.
If it uses something else, you have to be sure to keep mayas tangents/binormals in both baker and engine.

bitinn

Thx @Axi5 and @Noors.

Yeah I discovered that Maya 2018 bug as well.

I am baking in Substance Painter, so usually I only select Smoothing Group or Triangulate on Maya export (given same tangent space between Unity and SP). But the question is I don't quite know why Smoothing Group helps: I always assumed it helps because SP somehow doesn't understand Maya hard edges, so converting them to Smoothing Group helps the baking process (raycasting from low to high).

I tried exporting sample models with/without Smoothing Group option checked, but honestly can't tell if it makes any difference in the baking result (SP still identify the hard edges properly).

Noors

I doubt substance interprets smoothing groups. It just reads normals per vertex.
SG just stores groups of faces that should have averaged normals between them, not the normal vectors values per say. It's specific to Max and I get how it's confusing.
But the vertex normals data are always stored in the fbx, SG checked or not, and that's what substance is reading, hopefuly.
This option is probably only there to make a bridge between maya and max, if you want SG back in Max from the fbx.

bitinn

Noors said:

I doubt substance interprets smoothing groups. It just reads normals per vertex.
SG just stores groups of faces, not the normal data per say.
This option is probably only there to make a bridge between maya and max, or if you want tour SG back in Max from the fbx.

I don't mean to argue, but official allegorithmic tutorials I came across usually export with Smoothing Group enabled. Again, I am not sure this option is aimed at SP, or game engine, or simply personal habit. They never fully explained it.

pior

- Every time you read "smoothing groups", replace it in your mind with "support for split vertex normals". Smoothing groups are not a thing in FBX and OBJ, it's just a terminology/approach to create hard edges in Max. There is probably a historical reason for that (probably related to the .3ds format structure?) but that's irrelevant.

- Hard edges for highpoly to lowpoly baking are not just a personal preference, they are an essential tool to create clean bakes for some assets.

The best approach is not to use them "just because", but rather, to clearly understand why they are useful. But of course that is only possible once personally coming across an asset that would require them in the first place (not all assets require hard edges).

Now all that said, I personally don't bake in SP (I don't really see the point of it at all, when Toolbag3 is an all-around far superior solution) so maybe there are some quirks introduced by SP itself.

bitinn

@pior Thx for the tips, I do understand the basics (split vertex normal, baking raycast, when to triangulate low... etc.)

My question is specifically this: https://forum.allegorithmic.com/index.php/topic,19167.0.html (I am posting on Allegorithmic forum because ultimately the question is about SP).

BTW, to all, we have 2 possible regressions in Maya 2018 (Autodesk is working on a fix):

(1) "Smoothing Group" + "Triangulate" on export causes warning.
(2) "Tangents and Binormals" + "Triangulate" cause crash.

They change Triangulate method and ATF plugin in this release so things are a bit messed up.

pior

Well, with FBX being Autodesk's attempt at bridging their own apps it's very possible that they write redundant data in there (after all, SGs in the Max sense are indeed per-face properties so I wouldn't be surprised if FBX was holding both vertex normals *and* a Max-specific SG index). And then of course that doesn't mean that FBX importers in third party apps interpret the data the way AD intends it.

Personally I feel like AD still modifying their FBX import/export plugins after all these years is a huge red flag. It's especially annoying when a modeler uses the latest FBX exporter version "just because", resulting in files incompatible with some third-party apps in a pipeline.

FBX is a fantastic concept, but executed poorly. Tread carefully, and stick to plugin versions that work well for your needs. IIRC older versions of the I/O plugin (2012 ?) have all the features needed for baking, and there's also no need to use the latest and buggiest AD software in the first place. Find something stable (ideally one or two revisions back) and stick to it for as long as possible. No need for the latest Maya version either, it will always be buggy so why bother ?

You could run something like Maya 2015 or whichever version has a good track record, along with a solid FBX plugin itself from a year or two earlier, and you'd be pretty much guaranteed that it would work everywhere.

I think a shift in that regard is bound to happen sooner or later. FBX may be a widespread standard at the moment, but who knows what could happen in the near future... 

Spoon

pior said:I personally don't bake in SP (I don't really see the point of it at all, when Toolbag3 is an all-around far superior solution) 

Hi @pior ! Would you mind elaborating a bit on this? What makes it far superior? The bakes themselves, or cage\skew fixing etc?

Farfarer

pior said:

- Every time you read "smoothing groups", replace it in your mind with "support for split vertex normals". Smoothing groups are not a thing in FBX and OBJ, it's just a terminology/approach to create hard edges in Max. There is probably a historical reason for that (probably related to the .3ds format structure?) but that's irrelevant.

Smoothing groups are a thing in FBX. Although they are not an official part of the OBJ specification.

pior

Right, I should have corrected this (went back over this topic statement in the folllow-up post).

I guess what I am trying to get at is that while they are a thing on the Max authoring side of things and as you just confirmed are also present as a face property in the FBX data itself, there's is no guarantee that a third part app interprets them as such - for instance, a non-Autodesk program could have a tickbox saying "import smoothing groups", but effectively just reading the vertex normals information, and skipping/ignoring the SG flags altogether ...

The part that trips me out is ... why does the FBX Maya exporter even have an option saying "export smoothing groups" when Maya doesn't have a way to create/store them to begin with ? That means that this option either infers them from the hard edges, assigning face SGs at random (in which case it should be called "convert to SGs" instead) ; or, maybe it leaves them empty and simply writes vertex data.

Either way the current naming of the option doesn't really makes sense to me, and from there, it being buggy is not really surprising

throttlekitty

I'd always been a little curious about that, here's what the FBX SDK says (face-based or edge-based). So I guess it's just an option for if someone needs to export to max or something that expects maxlike smoothing groups.

//There are two kinds of smoothing info:
//1. Smoothing groups info which is saved by polygon. It usually come from 3ds Max, because 3ds Max can set smoothing groups for polygon.
//2. Hard/soft edges info which is saved by edge. It usually come from Maya, because Maya can set hard/soft edges.

 I've still never used max, is it possible for smoothing groups to overlap "just right" so only certain edges are hard? In maya you can do something like 3 edges in a loop as hard, never been sure if you can do that in max.

pior

Yeah, with Max SGs you'd do that by assigning multiple groups to a given face, creating blends. This scenario is pretty much the main weakness of the SG paradigm.

Noors

Theres a built in tool now to prevent the headache, but yeah it basically assigns faces to several groups, in a dirty way. I think polytools3d also made a tool, wich does a cleaner job as usual.

anyway, one can save a fbx to ascii format to see whats in there with the different options.

Axi5

I don't think this is just a matter of seeing what the FBX file produces, it's clearly going to store smoothing group data if you check the smoothing groups option. The heart of Bitinn's question is wondering what Substance Painter does with this data, which should be very simple to do. Just do two FBX's, see if the vertex normals are preserved or not. I can't find any documentation on Substance Painter/Designer that specifies the FBX has to have smoothing groups enabled, it's only in a training video that they mention it.

bitinn

I waited for 2 days on Allegorithmic forum and see no reply on the smoothing group question. I think it isn't a requirement to export with smoothing group (anymore).

My own cube test shows no difference whatsoever (intentionally trying to bake smooth bevels down to hard edges OR soft edges).

wesm

bitinn said:

I waited for 2 days on Allegorithmic forum and see no reply on the smoothing group question. I think it isn't a requirement to export with smoothing group (anymore).

My own cube test shows no difference whatsoever (intentionally trying to bake smooth bevels down to hard edges OR soft edges).

Hi, 
I'm very sorry for the delayed response on our forums. I posted there as well.

Substance Painter will read the soften/harden data set in Maya when you export as FBX. This data is used to shade the mesh in the Painter viewport similar to maya based on vertex normal smoothing. In maya you are smoothing/hardening the vertex normals. Smoothing Groups are used in FBX file format and it's useful to have it enabled for other software that supports it such as 3ds Max. However, this option is not directly used by Painter if you export from Maya. It doesn't cause a problem to have it enabled if you are using the same fbx in Painter and then again in other applications. 

I am also getting the issue where Maya 2018 reports triangulation and Smoothing Groups is not supported. I'm not sure what's going on there as it works in 2017. Also, with Painter we currently support FBX version up to 2017. 

In your example above, you have a hard edge low poly cube and bake a high res mesh of the cube that is smoothed. You can get a similar result as if you baked the low poly cube as fully smoothed. This is because in the Substance Painter baker we have an option to use Average Normals for the baking. This produces a similar result if you had Averaged Normals disabled and then baked the low res mesh with smoothing. However, with hard edges if you don't have a UV split that corresponds to the smoothing split, then you will get seams as you can see in the hard edge cube example.

Please let me now if that doesn't make sense and I'm very sorry for the confusion in the video. I should have left the smoothing group part out when discussing the FBX export options since I was using Maya  : )  

Cheers, 
Wes

 

bitinn

@wesm glad to hear your explanation and confirmation on this! It wasn't a big deal to begin with, I just want to make sure I understood it correctly.

(Also a good piece of information to know SP supports up to FBX 2017 so far)

(BTW, if you read a few replies back, that Maya 2018 export warning is likely a bug, at least they are investigating internally.)