Lip Sync for AI Assistants: Making Characters Talk in Real Time
How real-time lip sync works for AI assistants and characters: visemes, mapping speech to mouth shapes, timing with text-to-speech, and listening and thinking states.
Core Takeaway
Real-time lip sync maps speech to a small set of mouth shapes called visemes and switches between them in time with the audio. Many text-to-speech services can return viseme events with timestamps, which a character animation uses as inputs. Add listening, thinking and idle states, blend between mouth shapes rather than snapping, and keep the character small enough to run smoothly on phones.
Visemes: the building block
A viseme is the visual shape of the mouth for a group of sounds. Many sounds look the same on the lips, so a character needs far fewer mouth shapes than there are phonemes, often somewhere between about ten and twenty. Microsoft's Azure Speech documentation, for example, describes returning viseme IDs with timing as speech is synthesised.
Getting timing from speech
There are two main sources of timing:
- Text-to-speech with viseme or word events: the most accurate, since the service knows what it is saying
- Audio analysis: estimating mouth openness from the volume of the audio, simpler but less precise
- A hybrid: audio amplitude for openness, plus viseme events for shape
Build the character as states
Talking is only one state. An assistant also needs idle, listening, thinking and speaking states, plus reactions such as happy or confused. Drive the mouth with a viseme input, and blend between shapes over a few milliseconds so it does not flicker.
Latency and performance
Lip sync looks wrong when the mouth leads or lags the audio by more than a small amount. Schedule viseme changes against the audio clock, not timers. Keep the character lightweight, pause it when the app is in the background, and provide a reduced-motion version without constant mouth movement.
Building this in ExodeUI
In ExodeUI, which we make, an assistant character is a state machine with idle, listening, thinking and speaking states plus a mouth input your code sets from viseme events. Our free AI orb gallery shows the simpler, abstract version of the same states, ready to drop into a chat UI.
Key Principles to Remember
- Visemes are mouth shapes for groups of sounds
- Get timing from text-to-speech viseme events where possible
- Design idle, listening, thinking and speaking states
- Sync to the audio clock and blend between shapes
Frequently Asked Questions
Sources
- Viseme — Wikipedia
- Get facial position with viseme — Microsoft Learn (Azure AI Speech)