Why voice agents pause, and how to shorten the silence

info@dravin.ai (Dravin AI) — Mon, 05 Oct 2026 18:30:00 GMT

TL;DR

A turn waits on end-of-turn detection, the final transcript, the model’s first token, the first audio and the network.
Before the turn is confirmed, start only work you can discard: drafts, retrieval and read-only lookups.
Measure each stage per turn, with median and 95th percentile response time and how often the agent interrupts callers.

When a caller stops talking, a voice agent’s reply waits on a chain of steps: the decision that the caller has finished, the final transcript, the model’s first token, the first audio and the trip back over the network. Below we take one turn apart, then cover which work can start before the caller finishes, what to do when they cut in, and how to measure each stage.

Where a turn’s time goes

People leave little silence between turns. A 2009 study of recorded informal conversation in 10 languages put the gap after yes-or-no questions at 208 ms on average (median 100 ms), with language averages from 7 ms in Japanese to 469 ms in Danish [1]. Producing even one new word takes around 600 ms or more in experiments, which implies that listeners predict where a turn will end and prepare their reply while the other person is still talking [2].

The traditional cascaded voice agent (separate speech recognition, language model and speech synthesis) waits for silence and only then runs each stage in order. In a 2025 study where 39 adults talked to a robot, a baseline built this way waited for the recogniser’s end-of-speech signal (about 1 s of silence, by the authors’ estimate), then spent about 0.5 s in a hosted language model and about 1 s in a streaming speech synthesis service. Its replies most often came 2.6 s after the person stopped (median 2.7 s) [3].

One turn of a cascaded voice agent

End of turn: is the caller done?
Final transcript: partials settle
First token: prompt is read
First audio: voice starts
Network: back to the caller

The first two stages overlap; then each waits for the one before. In the 2025 robot study’s baseline, about 1 s of silence, 0.5 s of language model time and 1 s of synthesis sum to 2.5 s, near the most common reply, 2.6 s [3].

Two releases in the past week address parts of this chain. On 1 October Microsoft released MAI-Transcribe-2-Streaming, which returns partial transcripts while people speak; Microsoft says that in most cases words appear “as early as 320 milliseconds after they’re spoken” [4]. On 5 October Amazon made Nova 2.5 Sonic generally available, a speech-to-speech model that keeps controllable turn-taking and asynchronous tool calling; Amazon says its latency is lower but gives no figure [5].

Two of the three designs in OpenAI’s guide are one model that interprets audio, decides what to do and responds in speech, and a chained pipeline where you can inspect or transform the text between stages and replace each component [6]. A speech-to-speech turn still waits for the end of the turn, the first audio and the network. We build both kinds, and our voice agents section lists what that work covers.

Deciding the caller has finished

The simplest end-of-turn detector waits for silence. OpenAI’s server VAD “chunks the audio based on periods of silence”, and a shorter silence setting ends turns sooner [7]. Silence is a weak signal because people pause mid-sentence. LiveKit’s example is a caller who says “I need to think about that for a moment” and goes quiet, and a detector that only hears silence interrupts them [8]. The robot study describes the trade-off: depending on the threshold, “either long response delays or frequent interruptions” [3].

AWS turns that trade-off into a setting. It documents three endpointing sensitivities for Nova 2 Sonic, which wait for a pause of 1.5 s, 1.75 s or about 2 s, and says the choice “affects both the latency of responses and the likelihood of interrupting users who are still speaking”; it suggests the slowest for “elderly or speech-impaired users” [9]. The guide still names only Nova 2 Sonic, so confirm the values on 2.5.

A model can judge whether the words and the voice sound finished. OpenAI’s semantic VAD scores how likely it is that the user is done and waits for a timeout only when that probability is low [7]. LiveKit’s turn detector listens to what is said and how it is said; with it, the framework waits 0.3 to 2.5 s after speech, against 0.5 to 3.0 s without it [8]. Deepgram’s Flux recogniser emits turn events with an end-of-turn confidence [10]. In the robot study, a system that used two turn-taking models, both to judge the end of the turn and to start drafting early, cut the median response time from 2.7 s to 1.5 s against the silence baseline, and the interruption rate from 16.6% to 6.9% [3].

We set the detector per audience and task: a caller reading out an account number needs more patience than one confirming a booking with a yes.

From final transcript to first audio

A streaming recogniser sends partial transcripts while the caller talks and revises them as more audio arrives; Microsoft describes its new model committing “a stable transcript once the utterance ends” [4]. In a 2019 Google design, a streaming first pass produced the live predictions and a second decoder finalised them at the end of the utterance, cutting word errors by 17 to 22% relative to the first pass alone on a voice search task, adding under 200 ms to finalisation [11]. Either way the final text arrives after the caller stops and can differ from the last partial. LiveKit reports that wait per turn as transcription_delay [12].

The language model reads the whole prompt before its first token: instructions, tool definitions, the conversation so far and the new turn. Prefix caching skips the part that matches an earlier request, and it “only reduces the time of processing the queries (the prefilling phase)” [13]. Append each new turn and leave everything before it unchanged, as OpenAI also advises, so the cache keeps matching [14]. Our serving playbook covers time to first token on your own GPUs.

If your speech synthesis accepts streamed text, start it on the reply’s first words: Deepgram’s docs tell developers not to hold text back for sentence boundaries, because “the server streams a turn’s audio as tokens arrive” [15].

Every turn crosses the network twice: the caller’s last words travel to the agent and the reply travels back. ITU-T G.114 says that if one-way, mouth-to-ear delay stays below 150 ms, most applications “will experience essentially transparent interactivity” [16].

Starting work before the caller finishes

The later stages can overlap the caller’s speech if the agent is willing to throw work away. Deepgram calls this eager end of turn: on a medium-confidence EagerEndOfTurn event, send the transcript to the language model and start a reply; on TurnResumed, cancel it; on EndOfTurn, deliver it [17]. In Deepgram’s own example, an eager event fires on “Hi I need to cancel my subscription”, then the caller adds “please” and a TurnResumed event follows [10]. Deepgram puts the saving at “hundreds of milliseconds” and, for complex, slow language model setups, at 100 to 200 ms for 50 to 70% more calls [17]. Deepgram still recommends starting with EndOfTurn alone [17].

LiveKit Agents does something similar by default. It starts generation as soon as a final transcript arrives, before the turn is confirmed, and holds speech synthesis back unless you opt in. It skips utterances longer than 10 s, which are more likely to change, and its docs warn that the feature “doesn’t always reduce latency” [18]. The robot study’s system drafted tentative replies from the growing transcript, cancelling the previous draft each time, and sometimes had a reply ready at its 0.5 s minimum [3].

Before the turn is confirmed, do only work you can discard: drafting a reply, retrieving documents, or a read-only lookup such as checking availability while the caller is still describing what they want [14]. Bookings, payments and messages wait. OpenAI’s guidance on transcript fragments warns that a fragment “may be incomplete, and later speech can change the request”, and says to discard outdated results, avoid duplicate actions and keep the usual permission and confirmation checks before consequential actions [14].

Talking while tools run

A tool call puts silence back into the middle of a turn. Nova 2 Sonic handles part of this inside the model: its asynchronous tool calling keeps “accepting user input while tools are running” and answers new questions without waiting for pending results [19]. It does not cancel a pending call automatically when the caller changes their mind; every result reaches the model, which decides what to do with a stale one [19]. Since nothing is cancelled, a write started for an abandoned request would also complete, so any tool that writes should wait for explicit confirmation.

In a cascaded agent you fill the gap yourself. LiveKit’s advice is to start speaking when a tool may take more than a second, with a short, pre-written filler line that is pre-rendered if it repeats. The same guide says to put a timeout on every call to an external system, because a backend that hangs blocks the session, and to block interruptions inside tools that write data, since a cut-in “can leave the operation half-done” [20]. OpenAI’s guide says to measure acknowledgements such as “I’m checking” separately from the answer the caller needs [6].

When the caller cuts in

Barge-in has two jobs: stop the audio at once, and make the model’s history match what the caller heard [21]. Audio is generated faster than it plays, so some is already queued on the client when the caller interrupts; AWS’s guide for Nova 2 Sonic tells the client to stop playback and clear that queue on the interruption signal [22]. Deepgram also says to stop playback locally without waiting for the server and to drop audio frames still arriving [21].

If the history keeps the whole reply, the model assumes the caller heard sentences that were never played, and a follow-up such as “what was that last thing?” goes wrong [23]. OpenAI’s Realtime API removes the unplayed part itself on WebRTC and SIP connections, where the server tracks playback; over a WebSocket the client sends conversation.item.truncate with how much was played [23]. LiveKit Agents truncates its history to the part the user heard [24].

Not every sound is an interruption. Callers say “uh-huh” or “okay” as acknowledgements [18], and echo or background noise can open a false turn [17]. LiveKit’s adaptive mode uses an audio model to tell real interruptions from acknowledgements [18], and by default the agent resumes where it stopped when an interruption produces no words [24].

Measuring each stage

Measure each turn from the end of the caller’s speech to the first audio they hear, and keep each stage’s share. LiveKit Agents records, per turn, the time from end of speech to the end-of-turn decision, the transcription delay, the language model’s first token, the synthesiser’s first audio after its first text and the end-to-end latency [12]. Its e2e_latency stops when the agent begins responding [12]. With a realtime speech-to-speech model the language model and synthesis fields stay empty [12], so there you compare configurations end to end.

Report the median and the 95th percentile across similar calls, as OpenAI’s voice agent guide recommends, and change one factor at a time on the same scenarios [6]. The guide says to measure how long callers wait for a useful spoken answer, so stop the clock at client playback [6]. Its test plan moves from synthetic speech to replayed human recordings, then to a simulated caller for multi-turn conversations [6].

Track how often the agent interrupts callers next to response time: with a silence threshold, shortening the wait tends to raise interruptions [3]. For eager or preemptive generation, log how often a draft is cancelled against how often it is used [17]. More broadly, OpenAI’s guide says to check whether faster responses change task success, tool correctness or interruptions [6].

Checklist

Write the target down: end of speech to first audio at the median and 95th percentile, plus a ceiling on how often the agent interrupts callers.
Log each stage’s timing per turn, on replayed real calls.
Tune end-of-turn detection per audience, and prefer a detector that hears words and intonation to a fixed silence timer.
Before the turn is confirmed, start only work you can discard, and cancel it when the caller keeps talking.
Keep the start of the prompt stable for prefix caching; if your speech synthesis accepts streamed text, send tokens as they arrive.
For tools that may take over a second, play a short pre-rendered line; add timeouts; confirm before any write.
On barge-in, stop playback locally, clear queued audio and truncate the history to what the caller heard.
Filter acknowledgements and echo before treating a sound as an interruption.
Change one setting at a time and rerun the same scenarios.

Sources

This post first appeared on Dravin AI.

Dravin AI · Blog · Voice agents