Blender: tutorial: Lip Syncing

www.blender.nl
this document is online at http://www.blender.nl/showitem.php?id=46

Lip Syncing				2000 10 17
	Chris Want			id46

Introduction

Can *you* use Blender to make groovy character animations? The answer is YES! I was able to create my first lip sync by using a nifty little shareware program called Magpie (to decompose the audio track) and through the magic of Blender's Relative Vertex Key system. Here is the final animation faceV12.mpg (3.2MB); the Blender project may be downloaded below. This article is an overview of the steps I went through to create this lip sync.

Download:

faceV12_blend.zip

A frame from the animation

Relative Vertex Keys are needed for speech and facial expressions

First Things First

The first job (which is beyond the scope of this tutorial) is to create a model. The mesh of the model should have several relative vertex keys built in that represent the basic phonemes of speech. The model I use has 11 such keys, but it is possible to do decent animations with less. An excellent introduction to using phonemes in computer animation can be found on this page..

You should also create several other keys which model other facial movements and expressions -- my model has 16 additional RVK's that control his eyes, eye brows, moustache and hair. It would also be useful to have an IKA skeleton for your model -- certainly a neck and a skull would greatly increase the possibilities for your character. My model has an IKA skeleton on layer 3 with control empties on layer 2. At the risk of sounding like a T.V. cooking show chef I will say "Here's one I prepared earlier":

Download:

Lipsync_start_blend.zip

Next we need a soundclip. The dulcet tones of NaN's "Fearless Leader" Ton Roosendaal should to the trick! We also have to download a copy of the Magpie shareware program. It can be obtained from the Magpie web site: http://thirdwish.simplenet.com/

Download:

TON.wav

Have you been pronouncing Blender correctly? Find out today -- straight from the horse's mouth!

Isolating a Word

Locating the word "Blender"

O.K., lets start plotting out where our phonemes will go! Start Magpie. Select the menu "File->Open..." and select the file ton.wav. We will be mapping our favorite word "Blender" (the second occurance) The first trick is to find it! This is done by highlighting parts of the wav and pressing the "Play Selection" button (the "Play" button which is between two red vertical bars) After locating and highlighting the word (it should lie roughly between 00:00:04:17 and 00:00:05:03, which is indicated in the bottom right corner of the window) press the magnifying glass button to zoom into this part of the waveform.

At this point we should choose a mouth set to look at from the menu "Mouth->Select Mouth Set". The one I like the best is the default one as we don't have to look at a pair of dead eyes while the mouth is moving ("Billy" gives me nightmares!).

We want to see the mouth only!

Isolating Phonemes

B! Oh B! Where could you be?

O.K., lets find the first phoneme (this should "B" fun!) The important tool in this part of the exercise is the "Play From Selection" button (the "Play" button which is preceded by a red verticle bar). Pick frame 142 and press this button. This plays frame 142 and all frames after it in our zoomed sound clip. What is Ton saying? I hear "lender". This tells me that the "B" phoneme is earlier than this frame. Now try frame 141 and press the button. I hear "Blender"! This tells me that I will give this frame the "B" phoneme. Double click the "B" on the side bar to assign a B phoneme to frame 141.

Continue this exercise with the rest of the phonemes in this word. Pay attention to the animated cartoon mouth -- if something doesn't look quite right try moving a phoneme around by a frame or two. The frames that I get for the phonemes are as follows: "B" on frame 141; "L" on frame 143; "E" on frame 144; "N" on frame 146; "D" on frame 147; "U" on frame 148 (the "E" sounds like a "U"); and "R" on frame 150. If you don't get these values, don't fret: there is no one true method for mapping out the phonemes!

Map out the Other Words

If you have time, map out the rest of the words in the clip. It is highly recommended that you also go through the "Getting Started" section of the help file that comes with Magpie. Here is a copy of my Magpie file for this sound clip:

Download:

TON.mps

Getting the Exposure Sheet into Blender

From Magpie to Notepad to Blender

Having mapped out our sound clip, our next task is to use this info to construct some mesh IPO's for our relative vertex keys. We have a couple of choices. We could save it in the magpie format (an MPS file) and then alternate between the magpie window and the blender window as we construct our IPO's. No thanks! Blender has a built in text editor so wouldn't it be nice to have the magpie data as text right inside our file? (the answer is yes!)

We do this using the following steps: A) Press the copy to clipboard button; B) Launch notepad and select the menu "Edit->Paste"; C) Save the file as Magpie.txt; D) Open your model in blender and bring up a text window (shift-F11) and select "Open New" from the Menu and locate the file Magpie.txt from the file browser.

I actually like to do one variation to this technique: I like to start my animation on frame 1001, so when I paste the text output in notepad I add one thousand to all of the frame numbers.

The reason why I do this is because I like to reserve frames 1-999 for viewing and modelling my RVK's -- if you look at my file you'll find that on frame 1 you'll see the base mesh, frame 11 you'll see Key 1, frame 21 you'll see key 2, etc. I will adopt this convention for the rest of the tutorial, so for example the "B" phoneme will now land on frame 1141 instead of frame 141.

I like to view my keys on frames 11-271

Modelling the IPO's

Now to model the mesh IPO's. We have seven phonemes which luckily are represented by only four relative vertex keys: Key 8 gives us the "B"; Key 6 gives us the "L", "N" and "D" mouth shapes; Key 4 gives us the "E" and "U" sounds; and Key 5 gives us the "R".

Lets shape the IPO for the first phoneme (this should "B" cool -- yeah, yeah I know: it wasn't funny the first time!). Select the Key 8 IPO and enter edit mode. We will allow the "B" phoneme two frames to form so shift-left-click the value 0.0 on frame 1139. We want the phoneme to be fully present on frame 1141 so shift-left-click the value 1.0 on frame 1141. We want the "B" to be absent when the "L" phoneme appears on frame 1143 so shift-left-click 0.0 on frame 1143. Finally (this is optional), I like the handles on my IPO's to be of vector type so I select all vertices and press V_KEY.

Modelling more IPO's

Blender!

Groovy, lets do the "L". Select the IPO for key 6 and enter edit mode. The "B" phoneme appears on frame 1141 so we set the "L" IPO to zero on frame 1141. The "L" phoneme is fully present on frame 1143 so we shift-left-click value 1.0 on that frame. Finally, the "E" phoneme starts on frame 1144 so we want the "L" phoneme to be "Blended away" by that point: shift-left-click 0.0 on frame 1144. Continue this method until you have mapped out all the phonemes on your exposure sheet.

Putting it All Together

The last task which must be done is to sync up the audio with the video. The software I used to do mine was Ulead Video Studio, a commercial product that came free with my video card. There is a free product called DDClip free that will also do this for you. I like the price and the interface for DDClip better than ULead, but DDClip is very limited in terms of output options, and in particular it will not compress your output file. DDClip is suitable for making animations for your own viewing pleasure, but you should find something else if you want to distribute your work over the net (this might involve dishing out some coin!). DDClip free can be downloaded from here: http://www.softlab-nsk.com/ddclipro/free.html.

It often looks more natural if you delay the start of your audio by a few frames. When I sync up the audio and the video perfectly on my animation it looks like the dubbing on a Godzilla movie! This is a rule in character animation: the sound of the phonemes should be anticipated by the motion. For example, look at the "B" phoneme. The sound of the "B" doesn't occur when the mouth is closed -- it happens when the mouth pops open!

Syncing up the video and audio in Ulead Video Studio

Jeepers! Something's Missing!

If you play the movie as it is right now, you'll say to yourself "Jeepers! Somethings missing!" It is reminiscent of that creepy "Billy" mouthset that comes with Magpie. The eyes seem dead. The mouth is moving but there's no feeling (however, he's still a better actor than Jean-Claude Van Damme!)

Some Tips

I have adopted a few loose "rules" when animating this character. He is a cartoon so his expressions should be highly exaggerated so these rules may not be appropriate for other characters you might want to animate. I use them as a guideline, as a means to provide consistency of character -- but I won't lose any sleep when I choose to ignore them. Here they are:

"T & A" Blender style! (Hehe)

Blink on "percussive" consanants such as T, B, P or D. Unless you are trying to make your character look sleepy, the blink should be very quick (I use 1 or 2 frames to close the eyes, and two frames for the eyes to open). Relative Vertex Keys #12 and #13 control my models eyelids.

Raise and tilt the eyebrows on some vowels. On my model, this is done using Relative Vertex Keys #21-#24.

Raise the moustache on "E" vowels or other phonemes that give the top lip a flat shape. Lower the moustache on "O"'s and other phonemes that give the mouth a round shape. Relative vertex keys #26 and #27 take care of this.

Why not say it *and* spray it?!?

Occasionally introduce some kind of asymmetry to the animation. Perhaps only make one eyebrow raise breifly (this can make your character look thoughtful). Perhaps make one eyelid drop slightly. Occasionally rock the head to and fro as he is talking, in beat with the syllables.

Give your character some kind of quirk: mine spits every time he pronounces a "P" or "B" phoneme (these phonemes involve popping air between closed lips). The spit is parented to the head bone and jumps to layer 1 from invisible layer 20 and back again. These spittings are also anticipated: before he spits his neck leans backwards then snaps forward quickly as the spit comes out. His hair also quickly stands on end sometimes when he says the accented syllable of an important work.

Conclusion/final words/....

I hope this overview of how I made my lip sync proves to be helpful to someone, somewhere. The time to do this lipsync (not including the time taken to create the model, set the IKA skeleton, and hack together the Relative Vertex Keys) was about 16 hours. Only about 2 hours were spent using the Magpie program and about 4 hours shaping the IPO's for the phonemes. The rest of the time was spent tweeking the facial expressions, modelling spit, and plotting out the key frames for the head and neck.

The time needed to create the model and lay out the IKA's was the real time consumer and was an ongoing process that spanned several months. Luckily, this model has a fairly low polynomial count so the modelling of the Relative Vertex Keys was fairly painless.

Creating the RVK's should always be the last part of the model construction process because you must be 100% satisfied with your base mesh before you can start working on them -- I had to throw out my RVK's a couple of times because I realized I needed to add some vertices to my mesh! That said, although creating a usable model can be time consuming and frustrating, it is also highly rewarding. I hope more people decide to use Blender to make cool character animations! Happy Blendering!

Download:

faceV12.mpg

A frame from the animation