Dravin AI · Blog · Inference optimisation

Why voice agents pause, and how to shorten the silence

info@dravin.ai (Dravin AI) — Mon, 05 Oct 2026 18:30:00 GMT

TL;DR

A turn waits on end-of-turn detection, the final transcript, the model’s first token, the first audio and the network.
Before the turn is confirmed, start only work you can discard: drafts, retrieval and read-only lookups.
Measure each stage per turn, with median and 95th percentile response time and how often the agent interrupts callers.

When a caller stops talking, a voice agent’s reply waits on a chain of steps: the decision that the caller has finished, the final transcript, the model’s first token, the first audio and the trip back over the network. Below we take one turn apart, then cover which work can start before the caller finishes, what to do when they cut in, and how to measure each stage.

Where a turn’s time goes

People leave little silence between turns. A 2009 study of recorded informal conversation in 10 languages put the gap after yes-or-no questions at 208 ms on average (median 100 ms), with language averages from 7 ms in Japanese to 469 ms in Danish [1]. Producing even one new word takes around 600 ms or more in experiments, which implies that listeners predict where a turn will end and prepare their reply while the other person is still talking [2].

The traditional cascaded voice agent (separate speech recognition, language model and speech synthesis) waits for silence and only then runs each stage in order. In a 2025 study where 39 adults talked to a robot, a baseline built this way waited for the recogniser’s end-of-speech signal (about 1 s of silence, by the authors’ estimate), then spent about 0.5 s in a hosted language model and about 1 s in a streaming speech synthesis service. Its replies most often came 2.6 s after the person stopped (median 2.7 s) [3].

One turn of a cascaded voice agent

End of turn: is the caller done?
Final transcript: partials settle
First token: prompt is read
First audio: voice starts
Network: back to the caller

The first two stages overlap; then each waits for the one before. In the 2025 robot study’s baseline, about 1 s of silence, 0.5 s of language model time and 1 s of synthesis sum to 2.5 s, near the most common reply, 2.6 s [3].

Two releases in the past week address parts of this chain. On 1 October Microsoft released MAI-Transcribe-2-Streaming, which returns partial transcripts while people speak; Microsoft says that in most cases words appear “as early as 320 milliseconds after they’re spoken” [4]. On 5 October Amazon made Nova 2.5 Sonic generally available, a speech-to-speech model that keeps controllable turn-taking and asynchronous tool calling; Amazon says its latency is lower but gives no figure [5].

Two of the three designs in OpenAI’s guide are one model that interprets audio, decides what to do and responds in speech, and a chained pipeline where you can inspect or transform the text between stages and replace each component [6]. A speech-to-speech turn still waits for the end of the turn, the first audio and the network. We build both kinds, and our voice agents section lists what that work covers.

Deciding the caller has finished

The simplest end-of-turn detector waits for silence. OpenAI’s server VAD “chunks the audio based on periods of silence”, and a shorter silence setting ends turns sooner [7]. Silence is a weak signal because people pause mid-sentence. LiveKit’s example is a caller who says “I need to think about that for a moment” and goes quiet, and a detector that only hears silence interrupts them [8]. The robot study describes the trade-off: depending on the threshold, “either long response delays or frequent interruptions” [3].

AWS turns that trade-off into a setting. It documents three endpointing sensitivities for Nova 2 Sonic, which wait for a pause of 1.5 s, 1.75 s or about 2 s, and says the choice “affects both the latency of responses and the likelihood of interrupting users who are still speaking”; it suggests the slowest for “elderly or speech-impaired users” [9]. The guide still names only Nova 2 Sonic, so confirm the values on 2.5.

A model can judge whether the words and the voice sound finished. OpenAI’s semantic VAD scores how likely it is that the user is done and waits for a timeout only when that probability is low [7]. LiveKit’s turn detector listens to what is said and how it is said; with it, the framework waits 0.3 to 2.5 s after speech, against 0.5 to 3.0 s without it [8]. Deepgram’s Flux recogniser emits turn events with an end-of-turn confidence [10]. In the robot study, a system that used two turn-taking models, both to judge the end of the turn and to start drafting early, cut the median response time from 2.7 s to 1.5 s against the silence baseline, and the interruption rate from 16.6% to 6.9% [3].

We set the detector per audience and task: a caller reading out an account number needs more patience than one confirming a booking with a yes.

From final transcript to first audio

A streaming recogniser sends partial transcripts while the caller talks and revises them as more audio arrives; Microsoft describes its new model committing “a stable transcript once the utterance ends” [4]. In a 2019 Google design, a streaming first pass produced the live predictions and a second decoder finalised them at the end of the utterance, cutting word errors by 17 to 22% relative to the first pass alone on a voice search task, adding under 200 ms to finalisation [11]. Either way the final text arrives after the caller stops and can differ from the last partial. LiveKit reports that wait per turn as transcription_delay [12].

The language model reads the whole prompt before its first token: instructions, tool definitions, the conversation so far and the new turn. Prefix caching skips the part that matches an earlier request, and it “only reduces the time of processing the queries (the prefilling phase)” [13]. Append each new turn and leave everything before it unchanged, as OpenAI also advises, so the cache keeps matching [14]. Our serving playbook covers time to first token on your own GPUs.

If your speech synthesis accepts streamed text, start it on the reply’s first words: Deepgram’s docs tell developers not to hold text back for sentence boundaries, because “the server streams a turn’s audio as tokens arrive” [15].

Every turn crosses the network twice: the caller’s last words travel to the agent and the reply travels back. ITU-T G.114 says that if one-way, mouth-to-ear delay stays below 150 ms, most applications “will experience essentially transparent interactivity” [16].

Starting work before the caller finishes

The later stages can overlap the caller’s speech if the agent is willing to throw work away. Deepgram calls this eager end of turn: on a medium-confidence EagerEndOfTurn event, send the transcript to the language model and start a reply; on TurnResumed, cancel it; on EndOfTurn, deliver it [17]. In Deepgram’s own example, an eager event fires on “Hi I need to cancel my subscription”, then the caller adds “please” and a TurnResumed event follows [10]. Deepgram puts the saving at “hundreds of milliseconds” and, for complex, slow language model setups, at 100 to 200 ms for 50 to 70% more calls [17]. Deepgram still recommends starting with EndOfTurn alone [17].

LiveKit Agents does something similar by default. It starts generation as soon as a final transcript arrives, before the turn is confirmed, and holds speech synthesis back unless you opt in. It skips utterances longer than 10 s, which are more likely to change, and its docs warn that the feature “doesn’t always reduce latency” [18]. The robot study’s system drafted tentative replies from the growing transcript, cancelling the previous draft each time, and sometimes had a reply ready at its 0.5 s minimum [3].

Before the turn is confirmed, do only work you can discard: drafting a reply, retrieving documents, or a read-only lookup such as checking availability while the caller is still describing what they want [14]. Bookings, payments and messages wait. OpenAI’s guidance on transcript fragments warns that a fragment “may be incomplete, and later speech can change the request”, and says to discard outdated results, avoid duplicate actions and keep the usual permission and confirmation checks before consequential actions [14].

Talking while tools run

A tool call puts silence back into the middle of a turn. Nova 2 Sonic handles part of this inside the model: its asynchronous tool calling keeps “accepting user input while tools are running” and answers new questions without waiting for pending results [19]. It does not cancel a pending call automatically when the caller changes their mind; every result reaches the model, which decides what to do with a stale one [19]. Since nothing is cancelled, a write started for an abandoned request would also complete, so any tool that writes should wait for explicit confirmation.

In a cascaded agent you fill the gap yourself. LiveKit’s advice is to start speaking when a tool may take more than a second, with a short, pre-written filler line that is pre-rendered if it repeats. The same guide says to put a timeout on every call to an external system, because a backend that hangs blocks the session, and to block interruptions inside tools that write data, since a cut-in “can leave the operation half-done” [20]. OpenAI’s guide says to measure acknowledgements such as “I’m checking” separately from the answer the caller needs [6].

When the caller cuts in

Barge-in has two jobs: stop the audio at once, and make the model’s history match what the caller heard [21]. Audio is generated faster than it plays, so some is already queued on the client when the caller interrupts; AWS’s guide for Nova 2 Sonic tells the client to stop playback and clear that queue on the interruption signal [22]. Deepgram also says to stop playback locally without waiting for the server and to drop audio frames still arriving [21].

If the history keeps the whole reply, the model assumes the caller heard sentences that were never played, and a follow-up such as “what was that last thing?” goes wrong [23]. OpenAI’s Realtime API removes the unplayed part itself on WebRTC and SIP connections, where the server tracks playback; over a WebSocket the client sends conversation.item.truncate with how much was played [23]. LiveKit Agents truncates its history to the part the user heard [24].

Not every sound is an interruption. Callers say “uh-huh” or “okay” as acknowledgements [18], and echo or background noise can open a false turn [17]. LiveKit’s adaptive mode uses an audio model to tell real interruptions from acknowledgements [18], and by default the agent resumes where it stopped when an interruption produces no words [24].

Measuring each stage

Measure each turn from the end of the caller’s speech to the first audio they hear, and keep each stage’s share. LiveKit Agents records, per turn, the time from end of speech to the end-of-turn decision, the transcription delay, the language model’s first token, the synthesiser’s first audio after its first text and the end-to-end latency [12]. Its e2e_latency stops when the agent begins responding [12]. With a realtime speech-to-speech model the language model and synthesis fields stay empty [12], so there you compare configurations end to end.

Report the median and the 95th percentile across similar calls, as OpenAI’s voice agent guide recommends, and change one factor at a time on the same scenarios [6]. The guide says to measure how long callers wait for a useful spoken answer, so stop the clock at client playback [6]. Its test plan moves from synthetic speech to replayed human recordings, then to a simulated caller for multi-turn conversations [6].

Track how often the agent interrupts callers next to response time: with a silence threshold, shortening the wait tends to raise interruptions [3]. For eager or preemptive generation, log how often a draft is cancelled against how often it is used [17]. More broadly, OpenAI’s guide says to check whether faster responses change task success, tool correctness or interruptions [6].

Checklist

Write the target down: end of speech to first audio at the median and 95th percentile, plus a ceiling on how often the agent interrupts callers.
Log each stage’s timing per turn, on replayed real calls.
Tune end-of-turn detection per audience, and prefer a detector that hears words and intonation to a fixed silence timer.
Before the turn is confirmed, start only work you can discard, and cancel it when the caller keeps talking.
Keep the start of the prompt stable for prefix caching; if your speech synthesis accepts streamed text, send tokens as they arrive.
For tools that may take over a second, play a short pre-rendered line; add timeouts; confirm before any write.
On barge-in, stop playback locally, clear queued audio and truncate the history to what the caller heard.
Filter acknowledgements and echo before treating a sound as an interruption.
Change one setting at a time and rerun the same scenarios.

Sources

This post first appeared on Dravin AI.

More users per GPU: a playbook for self-hosted LLM serving

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Prefill keeps the GPU’s arithmetic busy while decode waits on memory, so most of the work is giving decode a bigger batch without letting prefill stall it.
Measure goodput first: the request rate each GPU serves while a target share of requests meet their latency limits.
Pull the levers in order of cost and risk: engine settings and prompt layout, then quantisation and speculative decoding, then parallelism and disaggregation.

A model that answers well in a notebook can be too slow and too expensive once hundreds of people use it at once. Before adding GPUs, we work through engine settings, prompt layout and a few serving techniques in a fixed order, and check each change against numbers we trust.

One fact sits under everything here. Prefill, where the model reads the prompt, keeps the GPU’s arithmetic busy [1], [2]. Decode, where it writes one token at a time per request, uses little of that arithmetic and mostly waits on memory bandwidth [3]. Most throughput work comes down to giving decode a bigger batch while stopping prefill from stalling it.

We serve open-weight models privately, quantised, on vLLM. Versions are current as of 23 September 2026: vLLM 0.30.0, SGLang 0.5.20 and TensorRT-LLM 1.2.1, each the latest stable release [4]. Flags and metric names below are vLLM’s unless we name another engine.

The order follows cost and risk. Batching, KV cache sizing, prefix caching and chunk size are configuration on a current engine, and vLLM 0.30.0 turns most of them on by default, so the work is sizing and checking them. Quantisation and speculative decoding change the arithmetic the model runs, so they need your own evals and your own load. Parallelism and disaggregation change the topology, so they come last.

Lever	What it buys	What it costs
Batch and KV cache sizing	More requests in flight	Per-user latency rises with the batch
Prefix caching and routing	Skipped prefill for shared prompts	Cache memory, and uneven load across replicas
Chunk size	Steady streams while long prompts load	A later first token
Quantisation	Fewer bytes per token, room for more requests	Accuracy risk; formats tied to GPU generation
Speculative decoding	Faster streams at low load	Spare compute; the gain fades as the batch fills
Admission control	Latency targets held under load	Some requests deferred or rejected
Replicas and expert parallelism	Capacity beyond one GPU group	More GPUs and fast interconnect
Disaggregation	Separate control of TTFT and ITL	Two pools to size, RDMA, experimental support

Measure before you tune

Four numbers describe a serving system.

Metric	What it measures	Why it matters
TTFT	Time from sending a request to receiving the first token [5]	How long a user stares at a blank screen
ITL and TPOT	The gap between streamed outputs (ITL), or a request’s decode time spread over its output tokens (TPOT) [6]	How fast the answer streams
Output tokens per second	Output tokens per second, summed over all concurrent requests [5]	What the hardware is worth to you
Goodput	The highest request rate per GPU served while a target share of requests meet their latency limits [7]	The number to optimise

Tools disagree on whether ITL counts the first token, so NVIDIA’s guide says to “compare results only when definitions align” [5]. Pick one tool and keep it.

DistServe defines goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned” [7], where the SLO (service-level objective) is the latency limit you commit to. It leads the dashboard because raw tokens per second can keep rising while users wait longer. System throughput and per-user speed pull against each other, and most levers below trade one for the other.

vllm bench serve measures all of this: set --request-rate and --burstiness to your arrival pattern, cap load with --max-concurrency, and pass your TTFT and TPOT limits in milliseconds to --goodput [6]. Report p50 and p99 for each; averages hide the slow tail. Use your own prompt lengths, output lengths and shared prefixes. Published benchmarks show which levers are worth trying; your own traffic shows how much each one gives.

In production, watch vllm:num_requests_waiting for queueing, vllm:kv_cache_usage_perc and vllm:num_preemptions for memory pressure, and vllm:prefix_cache_hits against vllm:prefix_cache_queries for cache reuse [8].

Get the batch and the KV cache right

The first lever is the engine. Early servers ran a batch until its longest request finished, so new requests queued behind it. Orca introduced “iteration-level scheduling”: the scheduler decides after every decoding step, so finished requests leave and new ones join at once. On a 175B model the paper reports 36.9 times the throughput of NVIDIA FasterTransformer at the same latency [9]. Every current engine does this, under the names continuous batching or in-flight batching [10].

The next constraint is the KV cache, the stored keys and values for every token of every running request. For a 13B model the PagedAttention paper works it out at 800 KB per token in FP16, so a single 2,048-token request can hold 1.6 GB [11]. The general form is 2 × layers × KV heads × head dimension × bytes per element, per token. It grows linearly with context length and concurrency, so the KV cache sets the batch ceiling. Grouped-query attention, where several query heads share one key-value head, shrinks the KV heads term [12]; a model’s config lists its KV head count, so check it before you choose the model.

GPU memory as 48 fixed-size KV cache blocks. Static batch with one slab per request: four requests each reserve a 12-block slab for their maximum length and fill part of it; B has finished but its slab stays held until the batch ends, so queued requests wait and 17 of 48 blocks hold tokens. Continuous batching with a paged cache: requests take small scattered blocks step by step; B's blocks free after step 3, new request E takes all of them at step 4, and by step 6 41 of 48 blocks hold tokens. See the animated diagram in the post.

A slab reserves a request’s maximum length up front and holds it until the request, or the batch, ends. A paged cache adds blocks as tokens arrive and returns them the moment a request finishes.

Before PagedAttention, only 20.4 to 38.2% of KV cache memory held actual token states. Storing the cache in small pages, as an operating system pages virtual memory, brought waste close to zero and gave 2 to 4 times the throughput of FasterTransformer and Orca at the same latency [11].

With a current engine, the work is sizing. vLLM 0.30.0 sets gpu_memory_utilization to 0.92 and picks max_num_seqs and max_num_batched_tokens by GPU [13]. When the cache runs out, requests are preempted and redo their prefill; the documented fixes include a higher gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or more tensor or pipeline parallelism to free memory for the cache [14]. A rising vllm:num_preemptions counter is the first thing we look at. On a new deployment we benchmark at the defaults first, then change one setting per run.

Leave CUDA graphs on. A decode step is a series of tiny GPU jobs, and graph replay skips the CPU cost of launching each one [15]; vLLM captures graphs by default, at some cost in memory and startup time [16]. Keep --enforce-eager for debugging [14].

The cost: a bigger batch raises tokens per second but also each user’s latency, the trade-off Sarathi-Serve is named for [1].

Stop recomputing the same prompt

Many workloads resend the same long prefix: a system prompt, tool definitions, a document, earlier turns of a chat. Prefix caching keeps that prefix’s KV cache, and a new request that shares it skips that part of prefill [17]. SGLang’s RadixAttention keeps prefixes in a prefix tree (a radix tree) with least-recently-used eviction [18]. Both engines turn it on by default in their current releases [13], [19].

It “only reduces the time of processing the queries (the prefilling phase)” [17], so it pays most where prompts are long and answers short: RAG, agents and document chat.

A cache only hits on an identical token prefix. Put stable content first (system prompt, tools, documents), variable content last, and keep timestamps and request IDs out of the top. In SGLang, --schedule-policy lpm reorders the queue by longest prefix match to raise the hit rate [20].

Routing that ignores the cache scatters requests that share a prefix across replicas, and one published benchmark shows the cost [21]. It ran 8 replicas of a 32B model on 16 H100s, with shared-prefix traffic whose cache demand was six times what one replica could hold. A router that tracked each replica’s actual cache contents produced 8,730 output tokens per second against 4,429 for random or load-only routing, and cut the 90th-percentile TTFT from over 90 seconds to about half a second; a router that estimated cache contents from its own routing history reached 6,944 tokens per second and 31 seconds [21]. SGLang’s router keeps such an estimate, an approximate copy of each worker’s prefix tree [22], and NVIDIA Dynamo 1.5.0, an orchestration layer that runs on top of engines such as vLLM, SGLang and TensorRT-LLM, routes “based on worker load and KV cache overlap” [23].

The cost: cache affinity fights load balance, so good routers blend a cache score with a load score, and the cache takes memory that could hold more running requests.

Keep long prompts from stalling everyone

One 50-page document arriving mid-stream can freeze every other user’s output while its prefill runs. Sarathi-Serve’s fix is chunked prefill: cut the prompt into pieces and mix each piece into ordinary decode steps. Against the vLLM of early 2024, the paper reports 2.6 times the serving capacity for a 7B model on one A100 and up to 3.7 times for a 34B model on two A100s [1].

vLLM has since made chunked prefill the default and schedules pending decodes before any prefill [14]. What remains is the dial. In vLLM’s docs, smaller max_num_batched_tokens values such as 2,048 “achieve better ITL”, larger ones “achieve better time to first token”, and values above 8,192 suit throughput, “especially for smaller models on large GPUs” [14]. vLLM 0.30.0’s server already defaults to 8,192 on an H100 and 2,048 on an A100 [13]. SGLang calls the same dial --chunked-prefill-size [20]. Set it against your written TTFT and ITL targets, measured at p99.

Timeline of GPU steps on one replica with five users streaming tokens while a 30,000-token prompt arrives. Without chunking, the prompt runs as one long prefill step and all five token streams stop until it ends, so inter-token latency spikes; the new user then gets a first token. With chunked prefill, the prompt is cut into eight equal slices, each riding along with a normal decode step; the five streams keep ticking at a slightly slower rate and the new user’s first token arrives a little later. A chunk-size scale shows the trade: small chunks such as 2,048 tokens keep streams smooth and delay the first token, large chunks such as 8,192 give an earlier first token and bumpier streams. See the animated diagram in the post.

Chunked prefill cuts a long prompt into slices and runs each one alongside ordinary decode steps, so other users keep receiving tokens while it loads. The two chunk sizes marked are the examples in vLLM’s docs [14].

Spend fewer bytes per token

Decode is limited by the bytes the GPU reads per token, so storing weights and cache in fewer bits gives more tokens per second.

FP8 weights and activations halve weight memory and need Ada (L40S), Hopper (H100, H200) or Blackwell (B200, GB200) GPUs [24]. One study ran more than 500,000 accuracy evaluations on an open model family at 8B, 70B and 405B, and separately benchmarked inference speed in vLLM on A6000, A100 and H100 GPUs. It found FP8 “effectively lossless across all model scales” and well-tuned INT8 within 1 to 3% [25].

INT4 weight-only (AWQ [26], GPTQ [27]) cuts weight memory about four times against FP16. The same study found it the most cost-efficient when requests run one at a time; under continuous batching, 8-bit formats reached the highest peak throughput, though 4-bit stayed more efficient at low latency [25]. Weight-only 4-bit speeds up memory-bound decode, and as the batch grows the GPU turns compute-bound: on Ampere GPUs, the authors of one 4-bit kernel report close to the full 4 times speed-up per layer up to batch 16 to 32, still significant but shrinking at 64 to 128, and up to 2.8 times end to end in vLLM [28]. For a busy server, that points to an 8-bit format first, unless tight latency limits keep the batch small.

NVFP4 stores 4-bit values in blocks of 16 that share an FP8 scale. NVIDIA reports about 3.5 times less memory than FP16 and 1.8 times less than FP8, and 1% or less accuracy loss moving a 671B mixture-of-experts (MoE) reasoning model from FP8 to NVFP4 [29]. FP4 arithmetic needs Blackwell, though older GPUs can still load NVFP4 weights: Nemotron 3.5 Lightning ships an NVFP4 checkpoint that runs on NVIDIA’s NVFP4 kernels across Blackwell, Hopper and Ampere [30]. On the older GPUs that saves memory and bandwidth, without the FP4 arithmetic.

The KV cache can be quantised separately, which raises the number of requests that fit. vLLM supports an FP8 KV cache and recommends calibrated scales [31]. With Qwen3.8-27B on an RTX PRO 6000 Blackwell, the SGLang, Qwen and NVIDIA teams fitted 70 concurrent requests at 32K context with an NVFP4 KV cache, against 44 with FP8. At matched concurrency, peak-batch decode throughput rose 26 to 30%. SGLang still marks the support experimental [32].

The cost is accuracy and hardware lock-in. Formats are tied to GPU generations, uncalibrated KV scales are a risk, and upgrades can break checkpoints: vLLM 0.30.0 dropped GPTQ activation ordering (the “act-order” option), so check any GPTQ checkpoint before you upgrade [13]. We quantise before we add GPUs, and judge each format on the evaluation set of the product it serves.

Draft tokens while there is spare compute

Speculative decoding lets a cheap drafter propose several tokens and the target model verify them in one pass, keeping the target model’s output distribution [3]. Leviathan and colleagues report 2 to 3 times on an 11B encoder-decoder model [3]. Each target pass yields (1 − α^γ+1) / (1 − α) tokens on average, where α is the acceptance rate and γ the number of drafted tokens: at α = 0.8 and γ = 4, about 3.4. The price is extra arithmetic: each step runs γ+1 positions through the target model in parallel, so “the number of concurrent arithmetic operations grows by a factor of γ+1”, and the method helps only when compute is spare [3].

So the gain shrinks as the batch fills, and vLLM aims speculation at “medium-to-low QPS, memory-bound workloads” [33]. EAGLE-3, a small draft head that reads the target model’s own hidden states, reports up to 6.5 times per-request speed-up on its best task and 1.38 times throughput at batch 64 in SGLang [34]. Plan a busy server around the second number.

Per-user speed-up from EAGLE 3.1, by concurrency

Concurrency 1: 2.03
Concurrency 4: 1.71
Concurrency 16: 1.66

Speed-up in each user’s output speed against no speculation, on a trillion-parameter-class MoE in NVFP4, tensor parallel across four GB200 GPUs, coding dataset [35].

The chart shows the speed each user sees, the number speculation targets; even so, the gain narrows from 2.03 to 1.66 times between concurrency 1 and 16 [35]. P-EAGLE, a newer drafter, shows the same pattern against EAGLE-3: on a 20B open-weight MoE on one B200, its gain fell from 1.55 times at concurrency 1 to 1.05 times at 64 on one chat benchmark, and from 1.69 to 1.25 times on a coding benchmark [36]. The EAGLE 3.1 authors note that speculative decoding “often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts”, the fragility 3.1 was built to reduce [35], so track vllm:spec_decode_num_accepted_tokens against vllm:spec_decode_num_draft_tokens on your own prompts [33]. vLLM can also shrink the draft as load rises: dynamic speculative decoding sets the number of drafted tokens per batch-size range, and an opt-in adaptive verification mode (DSpark drafters only) sizes verification per request from the drafter’s confidence [33].

Multi-token prediction (MTP) trains the model to draft its own next tokens [37]. Nemotron 3.5 Lightning has it built in [30], so there is no separate draft model to train or host; Gemma 4 ships ready-made MTP drafters that attach to the base model and share its embeddings, so there is nothing to train [38]. NVIDIA says “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases” [30].

Protect the latency target

Past a certain queue depth, one more request makes every request late. Cap concurrency where goodput peaks in your benchmark, reject or defer early with a retry signal (an HTTP 429 with a Retry-After header), and give interactive traffic priority: vLLM has --scheduling-policy priority [39] and SGLang has priority scheduling [19]. Nightly jobs belong in their own lane, where “achieving a large batch size is the most important thing” [20].

The cheapest request is one the big model never sees. We route classification and extraction to the smallest model that works, which takes that traffic off the serving fleet entirely; our post on typed decision models covers that lever.

Scale out across GPUs

Tensor parallelism splits every layer across GPUs and needs fast links between them; pipeline parallelism gives each GPU a run of layers; data parallelism runs whole copies of the model (replicas). vLLM advises tensor parallelism within a node, and pipeline parallelism across nodes or on GPUs without NVLink, NVIDIA’s fast GPU-to-GPU link [40]. SGLang says to “always favor data parallelism” for throughput when memory allows [20]. In practice: the smallest tensor-parallel group that holds the weights and a healthy KV cache, then more replicas.

MoE models such as DeepSeek V4, Kimi K3, GLM-5.3 and Mistral Large 3 activate a small share of their weights per token. Expert parallelism places experts on separate GPUs and works best with data parallelism; since tokens spread unevenly, vLLM’s --enable-eplb rebalances experts from live load statistics [41]. The SGLang team combined expert parallelism with prefill-decode disaggregation for a 671B MoE on H100s; in a decode-only test on 9 nodes of 8 GPUs, with prefill capacity assumed unlimited, it reached about 22,300 output tokens per second per node for 2,000-token inputs, about 5 times a 16-GPU tensor-parallel baseline [42]. That needs both techniques, several nodes and fast interconnect; on one node, tensor parallelism and replicas are simpler.

Split prefill and decode last

The final lever runs prefill and decode on separate GPU pools. DistServe found that “adding a single prefill job to a batch of decoding requests significantly slows down both processes”, and reports 7.4 times more requests or 12.6 times tighter SLOs than colocated systems, with over 90% of requests within their latency limits [7]. Splitwise reports 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput at the same cost and power [2].

Yet vLLM’s documentation says plainly: “Disaggregated prefill DOES NOT improve throughput” [43]. Both claims hold. The papers measure goodput under latency targets and cost-normalised capacity; vLLM’s statement is best read as raw tokens per second. Disaggregation leaves the GPUs as fast as they were. What it changes is interference. Chunked prefill stops the worst stalls, but every mixed step still carries a slice of someone’s prompt, which adds to ITL, and colocated phases “share their resource and parallelism settings” [7]. Separate pools remove both: decode steps carry no prompt work, and each pool gets its own parallelism and batch size, so more requests meet their targets and TTFT and ITL can be tuned separately [43]. The pools are sized separately too: longer prompts call for more prefill GPUs, longer answers for more decode GPUs.

Prefill and decode on separate GPU pools. Prompts of varying length arrive at a router, which scores replicas by prefix overlap and load, and go to a prefill pool where each GPU runs a wide, compute-heavy prefill. Each finished prompt’s KV cache crosses an RDMA link as a stream of packets to a decode pool, where many requests stream tokens in one large, steady batch. A TTFT gauge and an ITL gauge stay inside their target bands. Longer prompts call for more prefill GPUs; longer answers call for more decode GPUs. See the animated diagram in the post.

The router scores replicas by prefix overlap and load. Prefill hands each prompt’s KV cache to the decode pool over RDMA, so a long prompt does not stall the running decode streams. Shaded bands mark the latency targets.

Conditions matter. On GB200 with a 671B reasoning MoE, TensorRT-LLM’s rate-matched curves, which assume ideal pool ratios and free KV transfer, show 1.4 to 1.8 times the output throughput per GPU at the same per-user speed, with MTP off, at 4,400 input and 1,200 output tokens, mostly at lower concurrency; on 8,192-input, 256-output traffic its real end-to-end runs fell 0 to 25% short of the rate-matched curve [44]. SGLang moves the KV cache between pools with RDMA (remote direct memory access, which copies memory between machines without going through the CPU) transfer engines, on InfiniBand or RoCE devices or AWS EFA [45]; vLLM labels the feature experimental [43] and TensorRT-LLM beta [10]. For a single model on a single GPU, Dynamo’s README says the engine alone “is probably sufficient” [23]. Reach for disaggregation with long prompts, several nodes and fast interconnect. We treat it as the last step, after chunk size, routing and replicas have been tuned against goodput.

Checklist

Write the SLO down: TTFT and ITL at p99, and the share of requests that must meet them.
Benchmark on your own prompt lengths, output lengths, shared prefixes and arrival pattern, and report goodput.
Dashboard queue depth, KV cache usage, preemptions and prefix-cache hit rate.
Run a current engine with CUDA graphs on; size memory and batch settings until preemptions stop.
Put stable prompt content first, confirm prefix caching is on, and route by prefix across replicas.
Set the chunk size against your TTFT and ITL targets.
Try FP8 on Hopper, NVFP4 on Blackwell, INT4 weight-only at low concurrency, and a quantised KV cache; check accuracy on your own evals.
Add speculative decoding where concurrency is low to medium, try a model’s built-in MTP heads at higher concurrency with a shorter draft, and watch the acceptance rate.
Cap concurrency where goodput peaks, prioritise interactive traffic and give batch jobs their own lane.
Use the smallest tensor-parallel group that fits, then replicas; expert parallelism with load balancing for MoE.
Split prefill and decode only with long prompts, several nodes and RDMA.

Sources

This post first appeared on Dravin AI.