# Lip Sync for AI Assistants: Making Characters Talk in Real Time

> Real-time lip sync maps speech to a small set of mouth shapes called visemes and switches between them in time with the audio. Many text-to-speech services can return viseme events with timestamps, which a character animation uses as inputs. Add listening, thinking and idle states, blend between mouth shapes rather than snapping, and keep the character small enough to run smoothly on phones.

Updated November 2026. Source: https://www.exodeui.com/learn/lip-sync-animation-ai-assistant/

## Key takeaways

- Visemes are mouth shapes for groups of sounds
- Get timing from text-to-speech viseme events where possible
- Design idle, listening, thinking and speaking states
- Sync to the audio clock and blend between shapes

## Visemes: the building block

A viseme is the visual shape of the mouth for a group of sounds. Many sounds look the same on the lips, so a character needs far fewer mouth shapes than there are phonemes, often somewhere between about ten and twenty. Microsoft's Azure Speech documentation, for example, describes returning viseme IDs with timing as speech is synthesised.

## Getting timing from speech

There are two main sources of timing:

- Text-to-speech with viseme or word events: the most accurate, since the service knows what it is saying
- Audio analysis: estimating mouth openness from the volume of the audio, simpler but less precise
- A hybrid: audio amplitude for openness, plus viseme events for shape

## Build the character as states

Talking is only one state. An assistant also needs idle, listening, thinking and speaking states, plus reactions such as happy or confused. Drive the mouth with a viseme input, and blend between shapes over a few milliseconds so it does not flicker.

## Latency and performance

Lip sync looks wrong when the mouth leads or lags the audio by more than a small amount. Schedule viseme changes against the audio clock, not timers. Keep the character lightweight, pause it when the app is in the background, and provide a reduced-motion version without constant mouth movement.

## Building this in ExodeUI

In ExodeUI, which we make, an assistant character is a state machine with idle, listening, thinking and speaking states plus a mouth input your code sets from viseme events. Our free AI orb gallery shows the simpler, abstract version of the same states, ready to drop into a chat UI.

## Frequently asked questions

### What is a viseme?

The visible mouth shape for a group of speech sounds. Characters use a small set of visemes to lip sync.

### How do I lip sync to AI-generated speech?

Use a text-to-speech service that returns viseme or word timings, and set the character's mouth input as each event arrives in sync with the audio.

### Can I lip sync from audio alone?

Roughly, by mapping volume to mouth openness. It looks less accurate than using viseme data.

### Does a voice assistant need a character?

No. An abstract orb with clear states is often enough and easier to keep consistent.

## Related

- [Agentic UI states](https://www.exodeui.com/learn/agentic-ui-states/)
- [How to design an AI chatbot UI](https://www.exodeui.com/questions/how-to-design-an-ai-chatbot-ui/)
- [Interactive mascots in UI](https://www.exodeui.com/learn/interactive-mascot-ui/)
- [Animation blending](https://www.exodeui.com/learn/animation-blending/)
