Skip to main content
ExodeUI

EXODEUI DESKTOP

A little space for good ideas.

Choose your platform to get the official ExodeUI installer.

Looking for Intel Mac or other builds? Explore the web studio or check all download options.

Prefer the browser? Try for free in your browser →
AI & UX1 min read·Updated November 2026·By the ExodeUI team

Lip Sync for AI Assistants: Making Characters Talk in Real Time

How real-time lip sync works for AI assistants and characters: visemes, mapping speech to mouth shapes, timing with text-to-speech, and listening and thinking states.

Direct Summary & Definition

Core Takeaway

Real-time lip sync maps speech to a small set of mouth shapes called visemes and switches between them in time with the audio. Many text-to-speech services can return viseme events with timestamps, which a character animation uses as inputs. Add listening, thinking and idle states, blend between mouth shapes rather than snapping, and keep the character small enough to run smoothly on phones.

Visemes: the building block

A viseme is the visual shape of the mouth for a group of sounds. Many sounds look the same on the lips, so a character needs far fewer mouth shapes than there are phonemes, often somewhere between about ten and twenty. Microsoft's Azure Speech documentation, for example, describes returning viseme IDs with timing as speech is synthesised.

Getting timing from speech

There are two main sources of timing:

  • Text-to-speech with viseme or word events: the most accurate, since the service knows what it is saying
  • Audio analysis: estimating mouth openness from the volume of the audio, simpler but less precise
  • A hybrid: audio amplitude for openness, plus viseme events for shape

Build the character as states

Talking is only one state. An assistant also needs idle, listening, thinking and speaking states, plus reactions such as happy or confused. Drive the mouth with a viseme input, and blend between shapes over a few milliseconds so it does not flicker.

Latency and performance

Lip sync looks wrong when the mouth leads or lags the audio by more than a small amount. Schedule viseme changes against the audio clock, not timers. Keep the character lightweight, pause it when the app is in the background, and provide a reduced-motion version without constant mouth movement.

Building this in ExodeUI

In ExodeUI, which we make, an assistant character is a state machine with idle, listening, thinking and speaking states plus a mouth input your code sets from viseme events. Our free AI orb gallery shows the simpler, abstract version of the same states, ready to drop into a chat UI.

Key Principles to Remember

  • Visemes are mouth shapes for groups of sounds
  • Get timing from text-to-speech viseme events where possible
  • Design idle, listening, thinking and speaking states
  • Sync to the audio clock and blend between shapes

Frequently Asked Questions

Sources

  1. Viseme — Wikipedia
  2. Get facial position with viseme — Microsoft Learn (Azure AI Speech)

Related Guides & Examples

States for your AI assistant

Free animated orbs with idle, listening, thinking and speaking states your code switches.

See the AI orb gallery
Try for free