NOMU
Pilot a ghost with your face. A research game steered entirely by facial expressions.
NOMU is a research-driven game built by a team of six graduate students at Carnegie Mellon University's Entertainment Technology Center (ETC). In collaboration with the CMU Physical Intelligence Lab (again), our goal was a web-based experience that uses facial gestures to steer a ghost through space, designed to study motor-skill acquisition in a fun, engaging way. As the programmer on the team, I integrated the machine-learning models, built the research data-collection pipeline, and made sure players' facial movements mapped cleanly to the game's responses.
From ukuleles to faces
NOMU didn't start as a face game at all. It began as a two-handed tracking project in the spirit of Gesture Quest, this time teaching players the hand motions of strumming a ukulele. I got reliable two-hand detection working, and the team piled up ukulele gameplay ideas we were genuinely excited about.
Then the direction changed. The client felt hand-tracking data wasn't precise enough for the motor-skill research they needed, and asked us to pivot to facial control instead. Setting the ukulele ideas aside stung, but it was the right call for their goals, and it taught me to build for what a client actually needs rather than the demo I'd fallen for. So we rebuilt the entire game around face control in five weeks, which in game-development time is roughly a long weekend, and came out the other side with something we were proud of.
Game system: a balance of speed and structure
The core architecture bridges real-time machine-learning inference with responsive game mechanics. I built a FastAPI backend to host custom facial and hand-recognition models, enabling low-latency communication with the Unity client. This backend processed live webcam input, ran real-time inference, and streamed detection data (head rotations, facial expressions) back to the game over WebSockets.
On the Unity side, I built a modular, event-driven gameplay system that consumed this data to drive navigation and shooting. The system responded to nods, shakes, and mouth movements with precise timing for a tight gameplay loop. To support rapid iteration, I leaned on ScriptableObjects for gesture-action mappings and Unity's event system to decouple input processing from game logic. Every interaction was traceable, timestamped, and configurable, so researchers could analyze performance trends and gesture accuracy in depth.
Face detection: real-time tracking and gesture recognition
NOMU's control scheme is driven entirely by facial gestures. Facial detection was
powered by Google MediaPipe, with head-rotation angles (yaw, pitch, and roll)
calculated using OpenCV's solvePnP, reconstructing 3D head-pose estimation in real
time for precise spatial control of the ghost.
Beyond head orientation, the system read facial expressions through MediaPipe's blendshape outputs: mouth opening, eye blinking, mouth rolling, and lip puckering, each with a real-time confidence score. By setting configurable thresholds on those scores, we mapped expressions directly to mechanics: opening the mouth to eat, puckering to shoot. All gesture data, positional and expression-based, was logged for analysis.
Data collection: motor control and expression data
To serve the research goals, a data-collection pipeline was built directly into the game. Every gesture, from head rotations to blendshape activations, was recorded in real time alongside confidence scores and timestamps, so researchers could track not just which gestures were performed, but how accurately and consistently they were detected.
The system captured both raw detection data from the FastAPI server and contextual gameplay events from Unity. When a player opened their mouth to fire or tilted their head to steer, those actions were logged with gameplay state: position, level progress, and success or failure. The pipeline was configurable, so researchers could adjust the sampling rate or selectively log event types per study without touching the underlying code. That turned NOMU into more than a game; it became a functional research tool.
Game development: bringing gestures to life in Unity
Head-rotation data, calculated using OpenCV's pose estimation, mapped directly to the ghost's movement, letting players steer through a space maze by turning their heads. On top of navigation I layered expression-triggered mechanics: opening the mouth eats planet bullets, puckering fires a projectile to destroy meteors, blinking dashes through narrow gaps, and mouth-rolling shrinks the ghost to slip through pipes.
The feature playtesters loved most was the ghost's animated face. Using MediaPipe's blendshapes, the ghost mirrors the player's expressions in real time: blink your left eye and the ghost blinks its left eye; open your mouth and it follows. There is nothing quite like watching a room full of grad students blink and pucker at a webcam in the name of science, and yet that expressive mirroring built a surprising emotional connection between player and avatar.
Deployment and accessibility
NOMU was originally envisioned as a fully web-based game, playable in any browser with a webcam. Midway through, the project pivoted to prioritize research outcomes and controlled testing over public access. We delivered a standalone desktop build tailored to the CMU Physical Intelligence Lab, ensuring stable performance and compatibility with the backend server. I documented the deployment process (setting up the FastAPI inference server and Unity client) so the lab team could install, run, and update the game independently for ongoing experiments.
Reflection
NOMU pushed me to think beyond traditional gameplay and consider how games can serve as tools for data collection and analysis. The positive feedback from playtesters, especially those unfamiliar with gesture-based interaction, was hugely rewarding, and the project deepened my interest in the intersection of games and research.