Automatic lipsync
polycounter lvl 7
Hey all, how would you automate lipsync ? I'm using Maya, Unity, and I'm willing to learn motionBuilder. I have done the rig of a character in Maya including facial with the advanced skeleton. I'm pretty happy with my keyframe animations, but I'm looking for a faster result to gain more time and animate the body. Any help would be great for research. I'm sure some fellow animators out there are already doing this ?
Replies
It also depends heavily on the type of game and the pipeline you are thinking of using.
So for example...
1) You want to do a high end PC title with realistic cinematic scenes that have close ups of the face and head. You want some fairly expressive faces with accurate lip sync, then you will probably want to go with some kind of facial mocap system. It will probably use blendshapes/morph targets or at the very least a crap-ton of joints in the face. Depending on which route you go, blends or joints, there are different systems available and some can be twisted to work for the other with a bit of heavy lifting.
- Faceware is fairly good results but also fairly expensive with non-trival setup
- dynamixyz looks to be pretty decent, but I haven't used it.
- f-clone is decent-ish, the results could work if your quality bar is low enough
- Faceshift WAS really good and at a decent price, I used it in production it was great, it had a few hitches in it's pipeline but it gave great results and was super easy to use. Sadly Apple bought them out and shut it all down... fuck-you-apple.
- iClone Face is probably one of the better solutions out there, but it aims to create a base to work off of, you'll spend time cleaning it up, just like any other system. They do have some nice tools to help that process. For the price point, and the amount of clean up to get it up to quality, I'm faster doing it from scratch.
or...2) You want to a mobile game that features a cartoon cardboard cutout character that doesn't resemble a human at all, it has single jaw bone that flaps when sound files play. For that there are a handful of solutions, floating around, in scripts and in unity plugins.
- Voice O-Matic is fairly decent, I've used it in a production setting and it can work, if you are willing to sacrifice quality for speed.
- Unity has some automated lip sync tools available in its store, like LipSync Pro and Simple Unity LipSync. Just be warned, the results are pretty terrible, hopefully you can live with it.
There might be some other options floating around out there, but these are the ones I've come into contact with or spent some time researching.