Blendshapes Face Controls

So we added something cool in NOMU. We came up with a brilliant idea: the protagonist should mirror the player's face in real time, and the trick behind it is blendshapes.

A blendshape (also called a morph target or shape key) is a saved deformation of a mesh: sculpt the ghost's face into a smile, store it, and now you can dial from neutral to that smile with a single 0-to-1 slider. MediaPipe hands over exactly those sliders, a whole set of face blendshape scores like jaw-open and mouth-pucker, updating live from the webcam.

My original ambition was fancier: map MediaPipe's face landmarks directly onto the ghost's face topology, so the mesh would deform vertex for vertex with the player. But we shifted approaches and quietly filed that idea under someday due to an upcoming deadline. Instead, our artist made clean animations for each expression, and I drove them with the incoming data.

Rather than just hit play, I used the blendshape scores and landmark offsets to scrub each animation's playhead directly: open your mouth halfway and the ghost's mouth animation sits at its halfway frame, not fully open, not closed. The expression tracks your face continuously instead of firing a fixed clip.

Although it is a shortcut, it turned out pretty good. It runs fast, and nobody watching the ghost blink back at them ever asked how it was implemented. I'm glad we made that call to switch approaches.