Automatic lipsync

Elie

Hey all, how would you automate lipsync ? I'm using Maya, Unity, and I'm willing to learn motionBuilder. I have done the rig of a character in Maya including facial with the advanced skeleton. I'm pretty happy with my keyframe animations, but I'm looking for a faster result to gain more time and animate the body. Any help would be great for research. I'm sure some fellow animators out there are already doing this ?

Mark Dygert

It's pretty much a sliding scale, what saves you time, costs you in quality and what gives you quality costs you in time. There really aren't any quality automated lip sync tools that do a good job, they're all pretty terrible but some of them might be within your tolerance of acceptable, it really depends on how low you can let your standards go.

It also depends heavily on the type of game and the pipeline you are thinking of using.

So for example...
1) You want to do a high end PC title with realistic cinematic scenes that have close ups of the face and head. You want some fairly expressive faces with accurate lip sync, then you will probably want to go with some kind of facial mocap system. It will probably use blendshapes/morph targets or at the very least a crap-ton of joints in the face. Depending on which route you go, blends or joints, there are different systems available and some can be twisted to work for the other with a bit of heavy lifting.

• Faceware is fairly good results but also fairly expensive with non-trival setup
• dynamixyz looks to be pretty decent, but I haven't used it.
• f-clone is decent-ish, the results could work if your quality bar is low enough
  
• Faceshift WAS really good and at a decent price, I used it in production it was great, it had a few hitches in it's pipeline but it gave great results and was super easy to use. Sadly Apple bought them out and shut it all down... fuck-you-apple.
  
• iClone Face is probably one of the better solutions out there, but it aims to create a base to work off of, you'll spend time cleaning it up, just like any other system. They do have some nice tools to help that process. For the price point, and the amount of clean up to get it up to quality, I'm faster doing it from scratch.

or...

2) You want to a mobile game that features a cartoon cardboard cutout character that doesn't resemble a human at all, it has single jaw bone that flaps when sound files play. For that there are a handful of solutions, floating around, in scripts and in unity plugins.

• Voice O-Matic is fairly decent, I've used it in a production setting and it can work, if you are willing to sacrifice quality for speed.
• Unity has some automated lip sync tools available in its store, like LipSync Pro and Simple Unity LipSync. Just be warned, the results are pretty terrible, hopefully you can live with it.

There might be some other options floating around out there, but these are the ones I've come into contact with or spent some time researching.

Elie

Thank you Mark, for this complete answer, I want to try the Voice O-Matic. I've seen on their page that it exists from Maya 7 to 2015. Do you think I can use it for Maya 2018?

Elie

Well... I tested quickly LipSync Pro in Unity, but sadly couldn't get good results yet. I went through quite a lot of problems with a single blendshape not working as it should! It's such a pain...

Mark Dygert

The last time I used Voice-O-Matic was with Maya 2014, I would contact them because there have been some pretty big changes to Maya and a lot of the old plug-ins don't work. If they aren't keeping it up to date, that is a shame, it was a pretty decent tool. VOM was the one tool I used where the examples on their site where pretty trashy but the results where better. It works with blendshapes but had a workflow to convert the animation to a bone based setup, if your game didn't deal with blendshapes that well.

Elie

Well my biggest problem with the blendshapes I create in Maya: once in Unity the hole object translates up in the Y axis while playing the deformation which is very annoying. Did anyone already experienced that?

Elie

Just so you know, I rigged my character using the advanced skeleton for Maya, and the simplified-face button to import it in Unity.

Elie

Hey, I could get a pretty descent result with the add target feature for blendshape in Maya, but still can't merge facial lipsync pro animation with a keyframed body into Unity...

Elie

Okay I managed to make this work. I didn't understand everything but the only thing was missing was an animator controller. Still I have to make those beautiful blendshapes now. Any trick to transfer some cool looking poses I already have (by driven keying bones controls) into the targets of my main blendshape in the shape editor?