Maya to UE4 - morph targets' shading breaks, help please!

unit187

Hey, I've spent almost a week trying to solve it, but I am running out of options, and I have to ask the community for help.

I first encountered the issue while exporting my character with animated corrective blend shapes. The animation works fine, but as you can see on the screenshot the shading is completely broken.

I've tried to solve the issue, but it didn't get me anywhere, so I've decided to go with a simplier setup - setting up the blend shapes on a basic cylinder. The setup is simple: 3 bones, 2 of which influence the mesh, classic linear skinning, corrective blendshape made over mesh deformed by animation.

I did my testing on 3 slightly different setups :

a) a cylinder with default Maya smoothing;

b) a cylinder with soft/hard edges manually assigned;

c) a cylinder with soft/hard edges made using the script https://polycount.com/discussion/comment/1767197/#Comment_1767197 (doesn't really make a lot of sense in this case, but my character is made in Max, so might be useful in that case. An).

As you can see on the screenshot, the shading breaks too. It is not that obvious on static image, but when you move around it looks totally wrong.

I have tried a lot of stuff, and none of it works, am I missing something? I feel like it should be much simplier and I am missing something obvious, but can't put my finger on it.

What is correct pipeline that works for you guys?

Or maybe I am misunderstanding blend shapes in animation, and they should be used for very minor corrections? They seem to work fine when I do extremely simple modifications, like ribcage expanding on breathing in, but in cases like fixing the elbow when the skinning can no longer hold the shape it just breaks.

What I have tried so far:

1) Countless combinations of checkboxes (like export / import smoothing groups, import normals, import normals and tangents, etc.) both on export and import.

2) Different software versions I have available at work and at home: Maya 2018 and 2019, UE 4.22.3 and 4.23.0, FBX ver. 2011, 2018, 2019.

3) In Maya I've tried different combinations of locked and unlocked vertex normals, soft and hard edges, even used a script I found to convert locked vertex normals to soft and hard edges: https://polycount.com/discussion/comment/1767197/#Comment_1767197

4) Tried making corrective blend shapes using sculpting, and tried with only moving vertices by manipulator (thought maybe sculpting does something to vertex normals). Used shape editor, and also tried manually making blend shapes. I even went as far as made whole pose as a blend shape, but it didn't help too...

5) Changed evaluation order so the blend shape is computed before the skin cluster. Though Unreal doesn't care and evaluates morph target after the skin no matter how I set it up in Maya. But it doesn't help too as I see shading issues anyway.

6) I've lost track of what else I did, so I am out of ideas on how to fix it.

unit187

I'd very appreciate any advice or suggestion, thanks

Obscura

If you upload the fbx of the cylinder test, I can take a look at it and see what breaks. Most likely its just some import setting inside Unreal.

unit187

Obscura said:

If you upload the fbx of the cylinder test, I can take a look at it and see what breaks. Most likely its just some import setting inside Unreal.

Sure, here they are: https://drive.google.com/open?id=1f6foZBdl6nfb6xmw_UbgyGIoCfMPzy--

I was told I probably overdo the blendshapes, meaning the vertices move way too far, and I should rely more on joints to better hold the deformations.

Obscura

I'm not getting the error you are seeing.

These were my import settings:

maybe its the normal import methos set to a wrong setting on your end?

unit187

Thanks for looking into it.

I think you need to tap the checkbox for "import animation" and play the scene (simulate is enough to run the animation), the issue is visible only when the morph target activates during the animation You also have "compute normals" checked, which is not ideal, especially for more complex meshes with manually adjusted smoothing groups.

Obscura

I was manually setting the morph target. But I'll try and see it it appears with your animation.

Obscura

Ok, I don't see the error with the anim imported either. Though I still have compute normals set.

Obscura

To be more clear, I work a lot with morph targets in UE where at my job. But those are some higher poly characters so one smoothing group is fine for us, so I never bother importing normals for them.

Obscura

Even with the normal import method set to import normals, the error doesn't appear.

Obscura

Do you maybe have compute skin cache enabled in the project settings? Or recompute tangents? For correct normals with Alembic? From our experience, those can mess up the normals of skeletal meshes.

unit187

I'll look into these settings once I get home, thanks for the suggestion!
I will also experiment with compute normals, maybe there is something in this too.

unit187

I didn't have those settings enabled, though someone suggested I should try to use those exact checkboxes:

Support Compute Skincache

Force all skinned meshes to recompute tangents

It actually somewhat solves my issue, which is a relief (got a backup plan now!), however it has a disadvantage - the UV seams are now sometimes noticable in places where they were previously invisible. I did a google search on this and found other people reporting the same issue, which persists from older engine versions.

Not ideal, but huge improvement from what I had. I will keep this solution in mind, but I still would like to understand a proper way to set up morph targets.

I am going to try the whole process again with new test mesh and clean project, and will save my result and import settings. Maybe will share the project itself. Want to double-check everything again and have clear steps to reproduce the result.

unit187

Okay, I did some more careful tests. I usually get desired results when working with simple meshes like cylinders, but once I get back to complex shapes (characters, clothing), I still get artifacts here and there.

I've decided to stick with forced recalculation of tangents in Project Settings for now. While it has some issues with UV seams, it has accaptable quality, so it is good enough for me. 

Thanks for the help and suggestions!