Realtime Voice AI: Latency Budgets & Turn Taking

Realtime Voice Latency and Turn Taking

In human conversation, the gap between one person finishing and the other starting is typically a few hundred milliseconds. Voice agents that take two or three seconds to respond feel broken, and agents that jump in while the user is pausing to think feel rude. Making a voice agent conversational is mostly a problem of latency — shrinking every stage between the user's last word and the agent's first sound — and turn taking — deciding correctly when the user has finished, and stopping gracefully when the user interrupts.

These problems cut across the whole stack: audio capture and transport, voice activity detection, streaming speech recognition, language model time-to-first-token, streaming speech synthesis, and playback buffers. Each stage contributes delay and failure modes, so teams need an explicit latency budget, per-stage measurements, and turn-taking logic tuned on real conversations.

TL;DR

Measure voice-to-voice latency: from the end of user speech to the first audible agent audio, at p50 and p95.
Budget each stage: endpointing, final transcript, LLM first token, TTS first audio, network and buffers.
Endpointing waits for silence; tune it, or add semantic turn detection, so mid-sentence pauses don't trigger replies.
Support barge-in: detect user speech during playback, stop audio fast, cancel generation, and keep only what the user heard in context.
Use echo cancellation and minimum-speech rules so the agent doesn't interrupt itself or stop for "uh-huh."
Stream everything — partial transcripts, tokens, sentence-chunked TTS — and choose transport (WebRTC, WebSockets, telephony) deliberately.

Quick Example

Silence-based endpointing on 20 ms voice activity frames, and a voice-to-voice latency budget:

The 300 ms "um" does not end the turn, but the price is 500 ms of waiting after every utterance — the single largest item in the budget. Lowering the silence threshold makes the agent snappier and more likely to interrupt; semantic turn detection is the usual way out of that trade-off.

Core Concepts

The Latency Budget

Track p95, not just averages — a slow tool call or a cold TTS connection makes occasional turns feel broken. See LLM Inference & Serving and Tail Latency.

Architectures

Cascaded pipelines (STT → LLM → TTS) are flexible: you choose each model, inspect transcripts, and use text tools. Latency is the sum of stages, reduced by streaming between them.
Speech-to-speech models take and produce audio directly, cutting stages and preserving tone and prosody, with less visibility and control over intermediate text.
Hybrid designs use speech-to-speech for conversation and text pipelines for tools, logging, and compliance. See Voice AI.

Turn Detection

Users reading out account numbers, addresses, or thinking aloud pause mid-turn; test with these cases.

Barge-In and Interruptions

When the user speaks during agent playback: stop audio within a few hundred milliseconds, cancel in-flight LLM and TTS work, and truncate the agent's turn in the conversation context to what was actually played, so the model does not believe the user heard text that was never spoken. Distinguish interruptions from backchannels ("mm-hmm," "right") and from the agent's own audio leaking into the microphone, using acoustic echo cancellation, minimum speech duration, or minimum word counts before treating speech as an interruption.

Audio Quality and Transport

Use Opus over WebRTC for browsers and apps (built-in jitter buffering, echo cancellation, and congestion control), WebSockets for simpler server-to-server streaming, and telephony gateways for phone calls (8 kHz narrowband audio, extra network latency, DTMF). Apply noise suppression and gain control carefully, since aggressive processing can hurt recognition. See WebRTC and WebSockets.

Transcripts and Observability

Log user and agent turns with timestamps, partial and final transcripts, interruption points, and per-stage timings in one trace per turn. Redact sensitive data. For agents that take actions, record which words triggered which tool calls. See AI Agents.

Best Practices

Instrument Every Stage

Emit timestamps for end of speech, final transcript, first token, first audio byte, and playback start for every turn.

Stream End to End

Feed partial transcripts forward, stream tokens into TTS by sentence or clause, and start playback on the first audio chunk.

Tune Turn Taking on Real Calls

Use recorded conversations with pauses, numbers, accents, and noise to set thresholds and evaluate turn detectors.

Make Interruptions Cheap

Design generation, TTS, and tool calls to be cancellable, and keep context aligned with what the user heard.

Common Mistakes

Waiting for Each Stage to Finish

Agent Hearing Itself

Context Includes Unheard Text

FAQ

What latency should a voice agent target?

Many teams aim for voice-to-voice latency under about one second at p50, with p95 close behind; human conversational gaps are typically a few hundred milliseconds, so lower is better.

What is endpointing?

Deciding when the user has finished speaking, usually by detecting a period of silence after speech, optionally combined with models that predict whether the utterance is complete.

What is barge-in?

The ability for a user to interrupt the agent while it is speaking, which requires detecting user speech during playback, stopping audio, and cancelling in-progress generation.

Should I use a speech-to-speech model or a cascaded pipeline?

Speech-to-speech models offer lower latency and natural prosody; cascaded pipelines offer more control, model choice, and text visibility. Many products combine them.

Why use WebRTC for voice agents?

WebRTC provides low-latency audio transport with Opus, jitter buffers, echo cancellation, and congestion control, which are hard to replicate over plain WebSockets.