Modo. Baking errors on imported .FBX

musashidan

So I decided to test out Modo's baking rounded edge shader to NM feature, hoping to incorporate it into my workflow. I've been troubleshooting this for the last few nights and have run out of ideas. I'm a long-term Max user who has only dabbled sparingly in Modo. The issues occur on imported .FBX(from Max) Vs native Modo geo.

I started with a real-world piece I'm currently working on, but decided to troubleshoot back to basics with a classic simple 'boxy' box.

I've tested this Vs an identical mesh created within Modo, using the exact same process and the bake was flawless. This led me to believe it is an .FBX importer issue. Here's what I've tried so far:

Used latest .fbx exporter from Max and legacy rollbacks. Modo doesn't seem to have the option to choose import rollbacks(only export)

Gradually stripped out the .fbx from Max to only recognise geo(disabling smoothing/normals/etc)

Cleared all .fbx normals(smoothing) and reworked them withing Modo's Vertex toolkit.

Applied brand new normals in Modo on a clean, bare geo-only import.

Tried both .fbx imported normals and Modo vertex normals in Vertex maps>Other maps>Vertex normal.

Ensured imported geo scale was suitable to Modo's 1M cube.

Set the system padding to 32.

Here are the results of the Modo native geo bake



And the shit imported .fbx bake





The strange thing about this is that some of the edges have baked OK. Weird because it should be 'all or nothing'. I followed an exact process for both meshes.

And here are the baked maps:

Modo native Mesh:


Imported .fbx bake.....with artifacts...


Any help greatly appreciated.

Bek

Share the fbx?

First thoughts are try mesh cleanup, check no imported materials are fucking with things. Another thought is check if UV's are flipped; not sure if that makes a diff with round edge shader as I don't use it.

musashidan

I ran Mesh cleanup during my troubleshooting and no errors were spat. I didn't save materials in the .fbx. I just over-wrote the file as I'm still testing. I'm off to bed now but I'll upload a similar .fbx file tomorrow.

Cheers mate.

Bek

Just checked and flipped uv's are fine. What version of max, and what exact version of .fbx are you using? And what version of modo? Try fbx 2006.

Another thing to test is import the fbx into modo, export it as obj, reimport and try again. Or try obj from max.

musashidan

@bek I'm using Max 2018, Modo 11.2v2. and tried several .fbx versions: 2018(latest) 2014/2015 , 2016/2017.

I've imported an .Obj and it has baked fine.


I just set up the vertex normals in Modo. So at least we know it's definitely an .fbx issue. I'll keep testing. Unfortunately .Obj import is lacking any kind of options in Modo. You can't even set the scale as a multiplier. Also, the fact that you can't set the .fbx version on import in Modo is a bit shit. Thanks for the help.

musashidan

@bek here's an .fbx(2016/17) https://www.dropbox.com/sh/l654cv3czqsmv42/AAAXQ2aahYcewwvgCVBtNFuza?dl=0

I've almost given up at this stage. The fact that some edges are baking correct and others are completely broken on a simple 6-sided cube makes little sense. As I mentioned above, it should be all edges or none. I could still use .Obj but was really hoping not to have to. .Fbx should be standard now considering Modo is really trying hard to incorporate UE4 into its appeal.

Be great if you could test bake your end. I'm just baking a duplicate with a rounded edges shader. Cheers.

musashidan

I decided to go a different route and ended up finding a custom Chamfer Node for SD. Extremely quick and simple to set up, and the results are not too bad. Here's an SD screen of the result(this was the main test piece I was using in Modo before reverting to a box)

The wip mesh still has shit UVs and terrible auto-triangulation but I'm pleased with the results. Unfortunately I think I'll have to shelf Modo as I can just see it being too much of a headache trying to jump Max>Modo>S Painter>UE4. This SD node I'll definitely introduce to my workflow.

Thanks for the help.

leplatin

@musashidan dropbox link is probably broken
I have similar issues with exported from 3Ds fbx 2016/2017 with cage baking ( looks like messed up UVs) but without cage everything looks fine. It's weird... 

musashidan

@leplatin thanks for the heads up. Link fixed.

Yes, it is weird. I never use cage baking. Shouldn't really be any need anymore now that we have mikkT as standard. So your non-cage bakes are coming out good?

leplatin

@musashidan I always use cage baking with round edge shader in Modo...Just habit.
I have recently learned about synced workflow because of you 

Your mesh without cage

With cage

I even created new mesh in same layer (new UVs/vertex normals/cage)

After all I've created new project (new mesh)

UPD: Well...after resetting/freezing  everything bakes without any issues

UPD2: Another mesh from 3Ds MAX/ fbx 2014/2015. Freezing/resetting. Super weird. I've give up on Modo   

UPD3: I had to freeze transformations first and THEN apply vertex normals/hard edges. I did the opposite thats why normal maps baked wrong.

musashidan

@leplatin Thanks for taking the time. I did mess about with freeze/reset at some poin,t but as I'd completely reset Xform in Max I didn't worry about it too much.

Have you tried without a cage? The thing I don't get with the Modo bake is that it's Mikkt and synced(averaged) but it still seems to require UV/Smoothing splits. This shouldn't be essential in this workflow.

Also, have you had a successful bake if you don't redo the vertex normals in Modo, but use the imported .fbx normals in 'other maps'?

leplatin

Yes, with and without cage, after freezing everything baked without errors.
Since you can create Mikkt tangent basis for exporting to Substance/UE4 it's not originally Mikkt I guess. And  UV splits yeah..

Funny thing that problems started after redoing vertex normals, baking was succesful with imported fbx normals 

SenSayNyu

If you look at Item channels list there is PreRotation transform applied with -90° on X axis. I'm not sure why it even imports (maybe different xyz axis orientation on MAX?) but it breaks baking. Setting it to 0 will fix everything.

musashidan

Thanks for the help. I haven't had time to run any more tests. I have a ton of UV unwrapping that I have to get done. I hope to get some time tomorrow and I'll post up some results.

Cheers.