Dravin AI · Blog · Self-hosted LLMs

Typed decisions on your own GPUs

info@dravin.ai (Dravin AI) — Fri, 02 Oct 2026 18:30:00 GMT

TL;DR

SGLang 0.5.21’s /v1/decisions reads each label’s next-token probability from a chat model and renormalises over the labels, generating no text.
SGLang says those probabilities are not calibrated, and that they can shift with batching and prefix caching.
Fit a temperature or Platt scaling on your labelled data, test replay variance, and compare against a fine-tuned classifier.

On 29 September 2026 OpenAI announced a Decisions API in limited preview [1]. SGLang 0.5.21 now ships /v1/decisions, which turns a chat model you already serve into a classifier with a probability for every option [3], [5]. This follow-up to our post on typed decision models shows how the endpoint reads those probabilities out of a decoder model, why they are not calibrated, how to calibrate and test them, and when to run decisions on your own GPUs.

What OpenAI and SGLang shipped

OpenAI’s DevDay recap says the Decisions API focuses its model on “a specific set of user-defined questions with finite pre-defined answers”, given text or images, to classify content, route requests or choose an agent’s next action [1]. Its developer account says the API runs on GPT-6 Luna [2]. The recap also notes “a broad release planned in the coming days” [1]. Unlike the other developer announcements in the recap, the Decisions API entry links to no documentation, and the recap says nothing about how it works, so nothing below describes OpenAI’s design [1].

SGLang, the open-source serving engine, released 0.5.21 on PyPI on 1 October and on GitHub on 2 October [4], [3]; the code had merged on 25 September, before DevDay [7]. The route needs no special checkpoint: any generation model served with a Jinja chat template can answer if per-request checks pass [5]. Two models are validated, Qwen3.8-27B and Qwen3.5-35B-A3B, each on one H200 in BF16 [5].

A request carries one input, as text or a JSON object or array, and a list of questions: a choice among 2 to 26 options, a score on 2 to 10 levels, or a yes or no. Release 0.5.21 has no image field [5]. A second route, /v1/systemone, accepts the request shape of TypeSafe’s System One API, which serves Jev, so TypeSafe’s official SDKs can point their base URL at your server, with documented exceptions [5].

How a chat model returns a probability per option

The server writes each question as one user message: the input, a blank line, the question, one line per option with a letter label such as “A: billing”, and an instruction to answer with one label only [5], [6]. Levels are labelled 0 to 9, and yes or no questions use the words yes and no. The message goes through the model’s chat template with thinking turned off, and the server checks that each label adds exactly one distinct token after the prompt, returning a 400 when one does not [5], [6]. Letters keep every label to one token, whatever an option is called.

For a ticket and three teams, the user message reads:

The integration keeps failing.

Question: Which team should handle this ticket?
A: billing
B: technical
C: sales
Answer with the letter of one option only.

With thinking off, Qwen’s chat template then renders a closed reasoning block, so the next token is the answer [5], [6].

The model runs one prefill pass per question, the phase where it reads the prompt, and generates nothing: usage reports zero completion tokens [5]. At the answer position it holds a probability for every token in its vocabulary. The server keeps only the label tokens’ log-probabilities, divides them by the request’s temperature T (default 1) and takes a softmax over them. Every log-probability carries the same vocabulary-wide normalising term, so it cancels [5], [6]. A choice answer is the most probable option, and a score is the probability-weighted mean level [5].

Renormalising hides how much probability sat elsewhere, so the server also reports label_mass, the raw probability of all the labels together; a low value means most of it sits outside the offered answers [5].

From a typed question to a probability

Render each question: thinking off, labels one token each
Prefill per question: no tokens generated
Softmax over labels: divided by T, plus label_mass
Calibrate, then threshold: on your labelled data

Steps 1 to 3 run inside SGLang 0.5.21. Step 4 is yours: the returned probabilities are not calibrated.

Questions are answered independently and batched together; each is its own prompt that repeats the input, so prompt tokens grow with the number of questions [5]. Laya, from our Jev post, answers every question in one forward pass.

Constrained decoding reaches the same numbers. In SGLang, a one-token regex such as (A|B|C) at temperature 0, with log-probabilities returned, gives the same probabilities as the /v1/score path this route uses, for one-character labels in the same cache state [5]. A JSON schema with enum fields gives no clean probability per option, because an enum value can span several tokens [5].

What the endpoint refuses, and what that rules out

Most of the rules protect the answer position, the one place a number is read. Qwen3.8-27B and Qwen3.5-35B-A3B think by default; the route turns thinking off, refuses requests that turn it back on, and refuses prompts that leave a reasoning block open, so the answer is never read inside the reasoning [5]. That means the model answers without working anything out first. Compute arithmetic, dates and multi-step logic in code and pass the result in, as TypeSafe advises for Jev.

The server also refuses built-in conversation templates, servers started with --enable-mis or --dllm-algorithm, and a model field that names a LoRA adapter, so decisions cannot run on one [5]. It does not check where a new model’s template puts the answer, so before relying on a model beyond the validated two, the docs say to inspect the prompt token ids the route can return [5].

A decision takes a single input, not a chat history; a conversation goes in as text or a JSON array [5]. A choice stops at 26 options. /v1/systemone accepts up to 255 with two-letter labels where the tokeniser and chat template allow it, as the Qwen models’ do; SGLang warns that these labels include common words with unequal priors, so answers above 26 options can depend on option order [5].

Why the probabilities are not calibrated

SGLang’s documentation says “None of these values is a calibrated probability that the decision is correct” [5]. Four effects pull a chat model’s label probabilities away from calibration.

Post-training comes first. In a 2022 study, Anthropic found its large base models well calibrated when options were shown with letters and scored by the letter, as this route does, in prompts with a few worked examples; with no examples, which is how this route asks, calibration was weaker. Its RLHF-tuned versions looked badly miscalibrated until one temperature, T = 2.5, largely fixed them on several evaluations, though the authors warn that heavier RL training might distort calibration beyond that repair [9]. Both validated Qwen models are chat models [5], so test the same repair on yours.

Letter priors add a second skew. Across 20 LLMs, a 2023 study traced a preference for particular option letters to token bias: whatever the content, the model puts more probability on some letter tokens than on others, so the letter an option lands on can move its probability [10].

Wording is the third. In a 2021 study of few-shot prompts, the format, the examples and even their order swung accuracy from near chance to near the state of the art. On one sentiment prompt, the default 0.5 threshold gave frequent false positives where a threshold of 0.68 would have been 94% accurate on its validation set [11]. SGLang owns the wording around your question and ships each change as a new prompt_format_version [5]; the question and option text are yours, so test rewordings on labelled data.

Finally, probability can sit outside the labels. The probabilities always sum to 1 over the offered labels, so a choice picks an option even when none fits. label_mass shows the leak, though on yes or no questions it reads low even on clear cases, because it counts only the lowercase yes and no tokens while the model also puts probability on “Yes” and “No” [5]. Anthropic’s models were also reluctant to use a “none of the above” option, while their true or false judgements stayed well calibrated [9]. Of the Jev post’s two fixes, that favours a separate yes or no question before the choice.

Calibrating on your own server

Fit calibration per question, on labelled examples from your own traffic kept apart from the test set.

Temperature scaling fits a single number T by minimising log loss on held-out data, and never changes which option wins [12]. Here it maps onto the request’s temperature, which divides the label logits before the softmax [5]. Fit T offline and pass it in that field: the chosen option stays put, the probabilities and so the score’s weighted mean move, and label_mass does not change [5], [12]. One temperature covers every question in a request [6], so send questions with different temperatures separately, or apply T on your side: raise each probability returned at the default T of 1 to the power 1/T, renormalise, and recompute any score. SDK users have to do the latter, since /v1/systemone refuses a temperature field [5].

A temperature cannot correct a lean towards one label. For yes or no, Platt scaling fits a slope and an intercept on the log-odds, and the intercept absorbs a model that says yes too often; for choices, vector scaling fits a weight and a bias per option [12]. Isotonic regression can correct any monotone distortion but overfits small sets: with fewer than about 200 to 1,000 labelled cases, Platt scaling did better for all nine learning methods in a 2005 study [13].

Two corrections go after label priors directly. Contextual calibration scores a content-free input such as “N/A” and rescales the outputs so that input comes out uniform [11]. PriDe estimates the letter prior by permuting options on a small sample, for example 5% of the test set, and removes it from the remaining predictions [10]. Both remove bias, and the 2021 paper separates that goal from statistical calibration [11], so measure calibration again afterwards.

Refit when you change the model or its quantisation, upgrade SGLang, or receive a new prompt_format_version.

Testing replay and drift

The general evaluation is in the Jev post; a self-hosted endpoint adds its own tests.

Start with replay variance. On the two validated models, which mix full and linear attention, SGLang saw probabilities move by up to 0.07, and label_mass by up to about 0.14, between cold and prefix-cached requests and across batch compositions; the chosen option did not change in its checks [5]. One general cause: batch size changes how GPU kernels split their sums, and floating-point addition is not associative, so identical inputs can score differently [8]. Send your test set several times, cold and warm, alone and mixed with chat traffic, and route a band around each threshold, at least as wide as the movement you measure, to a person. On Qwen3.5-35B-A3B, --disable-radix-cache gave identical values across sequential repeats, at the cost of prefix reuse [5].

Keep every decision replayable. With return_prompt_token_ids, an answer carries its exact prompt and label token ids, and /v1/score re-scores them to the same values when temperature and cache state match [5]. Store them with the decision, and send prompt_format_version so that an upgrade which changes the wording fails loudly instead of shifting answers [5].

Shuffle option order on a sample and compare, since letter priors move answers [10]. Set a label_mass floor per question type from labelled data, remembering that yes or no reads low. Leave any temperature in --preferred-sampling-params unset, since it can scale the probabilities and label_mass when a decision shares a batch with generation or runs under speculative decoding [5].

Hosted preview or your own GPUs

Self-hosting makes most sense when you already serve an open model: decisions run on the same server, with no extra flag, alongside chat traffic [5]. Inputs stay on your hardware, any answer can be replayed from stored token ids, and you choose the model, its version and the calibration. The work is all prefill, so the prefix caching and chunk size levers in our serving playbook apply, though prefix caching is also one source of the drift SGLang measured. TypeSafe’s SDKs time out after 10 seconds and retry, so raise that timeout for many questions over a long input [5].

Code written for Jev can switch to /v1/systemone by changing its base URL. Its thresholds should not come along unchecked: the confidence that route returns is uncalibrated, and SGLang warns that thresholds tuned on another provider may not transfer [5].

A hosted API suits low volume, or a team that will not run GPUs; once OpenAI’s preview opens up, test it on the same labelled set.

Neither replaces a trained classifier where one fits. With labelled data and a stable label set, a small fine-tuned model is the stronger default: a 2024 study found smaller fine-tuned models beat larger zero-shot prompted models on every text classification task it tested [14]. Our default is unchanged: a small classifier fine-tuned on your labels, calibrated and thresholded, with the uncertain tail sent to an LLM or a person. A decision endpoint fits where that default cannot yet: when labels change per request, when no labelled data exists, or to draft labels, checked by people, for the classifier you train next.

Checklist

Pin the SGLang version, model and quantisation, send prompt_format_version with every request, and refit when any of them changes.
On a new model, check the returned prompt token ids for the answer position.
Leave the temperature in --preferred-sampling-params unset on servers that take decisions.
Fit a temperature per question on held-out labelled traffic, apply it per request or on your side, and add Platt or vector scaling if one label is over-predicted.
Shuffle option order on a sample, and correct letter priors if answers move.
Measure replay variance cold, warm and under load, and send a band at least that wide around each threshold to a person.
Set a label_mass floor per question type, and gate choices with a yes or no question where no option may fit.
Store each decision’s prompt and label token ids for replay.
Compare against a classifier fine-tuned on the same labels before you commit.

Sources

This post first appeared on Dravin AI.

More users per GPU: a playbook for self-hosted LLM serving

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Prefill keeps the GPU’s arithmetic busy while decode waits on memory, so most of the work is giving decode a bigger batch without letting prefill stall it.
Measure goodput first: the request rate each GPU serves while a target share of requests meet their latency limits.
Pull the levers in order of cost and risk: engine settings and prompt layout, then quantisation and speculative decoding, then parallelism and disaggregation.

A model that answers well in a notebook can be too slow and too expensive once hundreds of people use it at once. Before adding GPUs, we work through engine settings, prompt layout and a few serving techniques in a fixed order, and check each change against numbers we trust.

One fact sits under everything here. Prefill, where the model reads the prompt, keeps the GPU’s arithmetic busy [1], [2]. Decode, where it writes one token at a time per request, uses little of that arithmetic and mostly waits on memory bandwidth [3]. Most throughput work comes down to giving decode a bigger batch while stopping prefill from stalling it.

We serve open-weight models privately, quantised, on vLLM. Versions are current as of 23 September 2026: vLLM 0.30.0, SGLang 0.5.20 and TensorRT-LLM 1.2.1, each the latest stable release [4]. Flags and metric names below are vLLM’s unless we name another engine.

The order follows cost and risk. Batching, KV cache sizing, prefix caching and chunk size are configuration on a current engine, and vLLM 0.30.0 turns most of them on by default, so the work is sizing and checking them. Quantisation and speculative decoding change the arithmetic the model runs, so they need your own evals and your own load. Parallelism and disaggregation change the topology, so they come last.

Lever	What it buys	What it costs
Batch and KV cache sizing	More requests in flight	Per-user latency rises with the batch
Prefix caching and routing	Skipped prefill for shared prompts	Cache memory, and uneven load across replicas
Chunk size	Steady streams while long prompts load	A later first token
Quantisation	Fewer bytes per token, room for more requests	Accuracy risk; formats tied to GPU generation
Speculative decoding	Faster streams at low load	Spare compute; the gain fades as the batch fills
Admission control	Latency targets held under load	Some requests deferred or rejected
Replicas and expert parallelism	Capacity beyond one GPU group	More GPUs and fast interconnect
Disaggregation	Separate control of TTFT and ITL	Two pools to size, RDMA, experimental support

Measure before you tune

Four numbers describe a serving system.

Metric	What it measures	Why it matters
TTFT	Time from sending a request to receiving the first token [5]	How long a user stares at a blank screen
ITL and TPOT	The gap between streamed outputs (ITL), or a request’s decode time spread over its output tokens (TPOT) [6]	How fast the answer streams
Output tokens per second	Output tokens per second, summed over all concurrent requests [5]	What the hardware is worth to you
Goodput	The highest request rate per GPU served while a target share of requests meet their latency limits [7]	The number to optimise

Tools disagree on whether ITL counts the first token, so NVIDIA’s guide says to “compare results only when definitions align” [5]. Pick one tool and keep it.

DistServe defines goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned” [7], where the SLO (service-level objective) is the latency limit you commit to. It leads the dashboard because raw tokens per second can keep rising while users wait longer. System throughput and per-user speed pull against each other, and most levers below trade one for the other.

vllm bench serve measures all of this: set --request-rate and --burstiness to your arrival pattern, cap load with --max-concurrency, and pass your TTFT and TPOT limits in milliseconds to --goodput [6]. Report p50 and p99 for each; averages hide the slow tail. Use your own prompt lengths, output lengths and shared prefixes. Published benchmarks show which levers are worth trying; your own traffic shows how much each one gives.

In production, watch vllm:num_requests_waiting for queueing, vllm:kv_cache_usage_perc and vllm:num_preemptions for memory pressure, and vllm:prefix_cache_hits against vllm:prefix_cache_queries for cache reuse [8].

Get the batch and the KV cache right

The first lever is the engine. Early servers ran a batch until its longest request finished, so new requests queued behind it. Orca introduced “iteration-level scheduling”: the scheduler decides after every decoding step, so finished requests leave and new ones join at once. On a 175B model the paper reports 36.9 times the throughput of NVIDIA FasterTransformer at the same latency [9]. Every current engine does this, under the names continuous batching or in-flight batching [10].

The next constraint is the KV cache, the stored keys and values for every token of every running request. For a 13B model the PagedAttention paper works it out at 800 KB per token in FP16, so a single 2,048-token request can hold 1.6 GB [11]. The general form is 2 × layers × KV heads × head dimension × bytes per element, per token. It grows linearly with context length and concurrency, so the KV cache sets the batch ceiling. Grouped-query attention, where several query heads share one key-value head, shrinks the KV heads term [12]; a model’s config lists its KV head count, so check it before you choose the model.

GPU memory as 48 fixed-size KV cache blocks. Static batch with one slab per request: four requests each reserve a 12-block slab for their maximum length and fill part of it; B has finished but its slab stays held until the batch ends, so queued requests wait and 17 of 48 blocks hold tokens. Continuous batching with a paged cache: requests take small scattered blocks step by step; B's blocks free after step 3, new request E takes all of them at step 4, and by step 6 41 of 48 blocks hold tokens. See the animated diagram in the post.

A slab reserves a request’s maximum length up front and holds it until the request, or the batch, ends. A paged cache adds blocks as tokens arrive and returns them the moment a request finishes.

Before PagedAttention, only 20.4 to 38.2% of KV cache memory held actual token states. Storing the cache in small pages, as an operating system pages virtual memory, brought waste close to zero and gave 2 to 4 times the throughput of FasterTransformer and Orca at the same latency [11].

With a current engine, the work is sizing. vLLM 0.30.0 sets gpu_memory_utilization to 0.92 and picks max_num_seqs and max_num_batched_tokens by GPU [13]. When the cache runs out, requests are preempted and redo their prefill; the documented fixes include a higher gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or more tensor or pipeline parallelism to free memory for the cache [14]. A rising vllm:num_preemptions counter is the first thing we look at. On a new deployment we benchmark at the defaults first, then change one setting per run.

Leave CUDA graphs on. A decode step is a series of tiny GPU jobs, and graph replay skips the CPU cost of launching each one [15]; vLLM captures graphs by default, at some cost in memory and startup time [16]. Keep --enforce-eager for debugging [14].

The cost: a bigger batch raises tokens per second but also each user’s latency, the trade-off Sarathi-Serve is named for [1].

Stop recomputing the same prompt

Many workloads resend the same long prefix: a system prompt, tool definitions, a document, earlier turns of a chat. Prefix caching keeps that prefix’s KV cache, and a new request that shares it skips that part of prefill [17]. SGLang’s RadixAttention keeps prefixes in a prefix tree (a radix tree) with least-recently-used eviction [18]. Both engines turn it on by default in their current releases [13], [19].

It “only reduces the time of processing the queries (the prefilling phase)” [17], so it pays most where prompts are long and answers short: RAG, agents and document chat.

A cache only hits on an identical token prefix. Put stable content first (system prompt, tools, documents), variable content last, and keep timestamps and request IDs out of the top. In SGLang, --schedule-policy lpm reorders the queue by longest prefix match to raise the hit rate [20].

Routing that ignores the cache scatters requests that share a prefix across replicas, and one published benchmark shows the cost [21]. It ran 8 replicas of a 32B model on 16 H100s, with shared-prefix traffic whose cache demand was six times what one replica could hold. A router that tracked each replica’s actual cache contents produced 8,730 output tokens per second against 4,429 for random or load-only routing, and cut the 90th-percentile TTFT from over 90 seconds to about half a second; a router that estimated cache contents from its own routing history reached 6,944 tokens per second and 31 seconds [21]. SGLang’s router keeps such an estimate, an approximate copy of each worker’s prefix tree [22], and NVIDIA Dynamo 1.5.0, an orchestration layer that runs on top of engines such as vLLM, SGLang and TensorRT-LLM, routes “based on worker load and KV cache overlap” [23].

The cost: cache affinity fights load balance, so good routers blend a cache score with a load score, and the cache takes memory that could hold more running requests.

Keep long prompts from stalling everyone

One 50-page document arriving mid-stream can freeze every other user’s output while its prefill runs. Sarathi-Serve’s fix is chunked prefill: cut the prompt into pieces and mix each piece into ordinary decode steps. Against the vLLM of early 2024, the paper reports 2.6 times the serving capacity for a 7B model on one A100 and up to 3.7 times for a 34B model on two A100s [1].

vLLM has since made chunked prefill the default and schedules pending decodes before any prefill [14]. What remains is the dial. In vLLM’s docs, smaller max_num_batched_tokens values such as 2,048 “achieve better ITL”, larger ones “achieve better time to first token”, and values above 8,192 suit throughput, “especially for smaller models on large GPUs” [14]. vLLM 0.30.0’s server already defaults to 8,192 on an H100 and 2,048 on an A100 [13]. SGLang calls the same dial --chunked-prefill-size [20]. Set it against your written TTFT and ITL targets, measured at p99.

Timeline of GPU steps on one replica with five users streaming tokens while a 30,000-token prompt arrives. Without chunking, the prompt runs as one long prefill step and all five token streams stop until it ends, so inter-token latency spikes; the new user then gets a first token. With chunked prefill, the prompt is cut into eight equal slices, each riding along with a normal decode step; the five streams keep ticking at a slightly slower rate and the new user’s first token arrives a little later. A chunk-size scale shows the trade: small chunks such as 2,048 tokens keep streams smooth and delay the first token, large chunks such as 8,192 give an earlier first token and bumpier streams. See the animated diagram in the post.

Chunked prefill cuts a long prompt into slices and runs each one alongside ordinary decode steps, so other users keep receiving tokens while it loads. The two chunk sizes marked are the examples in vLLM’s docs [14].

Spend fewer bytes per token

Decode is limited by the bytes the GPU reads per token, so storing weights and cache in fewer bits gives more tokens per second.

FP8 weights and activations halve weight memory and need Ada (L40S), Hopper (H100, H200) or Blackwell (B200, GB200) GPUs [24]. One study ran more than 500,000 accuracy evaluations on an open model family at 8B, 70B and 405B, and separately benchmarked inference speed in vLLM on A6000, A100 and H100 GPUs. It found FP8 “effectively lossless across all model scales” and well-tuned INT8 within 1 to 3% [25].

INT4 weight-only (AWQ [26], GPTQ [27]) cuts weight memory about four times against FP16. The same study found it the most cost-efficient when requests run one at a time; under continuous batching, 8-bit formats reached the highest peak throughput, though 4-bit stayed more efficient at low latency [25]. Weight-only 4-bit speeds up memory-bound decode, and as the batch grows the GPU turns compute-bound: on Ampere GPUs, the authors of one 4-bit kernel report close to the full 4 times speed-up per layer up to batch 16 to 32, still significant but shrinking at 64 to 128, and up to 2.8 times end to end in vLLM [28]. For a busy server, that points to an 8-bit format first, unless tight latency limits keep the batch small.

NVFP4 stores 4-bit values in blocks of 16 that share an FP8 scale. NVIDIA reports about 3.5 times less memory than FP16 and 1.8 times less than FP8, and 1% or less accuracy loss moving a 671B mixture-of-experts (MoE) reasoning model from FP8 to NVFP4 [29]. FP4 arithmetic needs Blackwell, though older GPUs can still load NVFP4 weights: Nemotron 3.5 Lightning ships an NVFP4 checkpoint that runs on NVIDIA’s NVFP4 kernels across Blackwell, Hopper and Ampere [30]. On the older GPUs that saves memory and bandwidth, without the FP4 arithmetic.

The KV cache can be quantised separately, which raises the number of requests that fit. vLLM supports an FP8 KV cache and recommends calibrated scales [31]. With Qwen3.8-27B on an RTX PRO 6000 Blackwell, the SGLang, Qwen and NVIDIA teams fitted 70 concurrent requests at 32K context with an NVFP4 KV cache, against 44 with FP8. At matched concurrency, peak-batch decode throughput rose 26 to 30%. SGLang still marks the support experimental [32].

The cost is accuracy and hardware lock-in. Formats are tied to GPU generations, uncalibrated KV scales are a risk, and upgrades can break checkpoints: vLLM 0.30.0 dropped GPTQ activation ordering (the “act-order” option), so check any GPTQ checkpoint before you upgrade [13]. We quantise before we add GPUs, and judge each format on the evaluation set of the product it serves.

Draft tokens while there is spare compute

Speculative decoding lets a cheap drafter propose several tokens and the target model verify them in one pass, keeping the target model’s output distribution [3]. Leviathan and colleagues report 2 to 3 times on an 11B encoder-decoder model [3]. Each target pass yields (1 − α^γ+1) / (1 − α) tokens on average, where α is the acceptance rate and γ the number of drafted tokens: at α = 0.8 and γ = 4, about 3.4. The price is extra arithmetic: each step runs γ+1 positions through the target model in parallel, so “the number of concurrent arithmetic operations grows by a factor of γ+1”, and the method helps only when compute is spare [3].

So the gain shrinks as the batch fills, and vLLM aims speculation at “medium-to-low QPS, memory-bound workloads” [33]. EAGLE-3, a small draft head that reads the target model’s own hidden states, reports up to 6.5 times per-request speed-up on its best task and 1.38 times throughput at batch 64 in SGLang [34]. Plan a busy server around the second number.

Per-user speed-up from EAGLE 3.1, by concurrency

Concurrency 1: 2.03
Concurrency 4: 1.71
Concurrency 16: 1.66

Speed-up in each user’s output speed against no speculation, on a trillion-parameter-class MoE in NVFP4, tensor parallel across four GB200 GPUs, coding dataset [35].

The chart shows the speed each user sees, the number speculation targets; even so, the gain narrows from 2.03 to 1.66 times between concurrency 1 and 16 [35]. P-EAGLE, a newer drafter, shows the same pattern against EAGLE-3: on a 20B open-weight MoE on one B200, its gain fell from 1.55 times at concurrency 1 to 1.05 times at 64 on one chat benchmark, and from 1.69 to 1.25 times on a coding benchmark [36]. The EAGLE 3.1 authors note that speculative decoding “often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts”, the fragility 3.1 was built to reduce [35], so track vllm:spec_decode_num_accepted_tokens against vllm:spec_decode_num_draft_tokens on your own prompts [33]. vLLM can also shrink the draft as load rises: dynamic speculative decoding sets the number of drafted tokens per batch-size range, and an opt-in adaptive verification mode (DSpark drafters only) sizes verification per request from the drafter’s confidence [33].

Multi-token prediction (MTP) trains the model to draft its own next tokens [37]. Nemotron 3.5 Lightning has it built in [30], so there is no separate draft model to train or host; Gemma 4 ships ready-made MTP drafters that attach to the base model and share its embeddings, so there is nothing to train [38]. NVIDIA says “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases” [30].

Protect the latency target

Past a certain queue depth, one more request makes every request late. Cap concurrency where goodput peaks in your benchmark, reject or defer early with a retry signal (an HTTP 429 with a Retry-After header), and give interactive traffic priority: vLLM has --scheduling-policy priority [39] and SGLang has priority scheduling [19]. Nightly jobs belong in their own lane, where “achieving a large batch size is the most important thing” [20].

The cheapest request is one the big model never sees. We route classification and extraction to the smallest model that works, which takes that traffic off the serving fleet entirely; our post on typed decision models covers that lever.

Scale out across GPUs

Tensor parallelism splits every layer across GPUs and needs fast links between them; pipeline parallelism gives each GPU a run of layers; data parallelism runs whole copies of the model (replicas). vLLM advises tensor parallelism within a node, and pipeline parallelism across nodes or on GPUs without NVLink, NVIDIA’s fast GPU-to-GPU link [40]. SGLang says to “always favor data parallelism” for throughput when memory allows [20]. In practice: the smallest tensor-parallel group that holds the weights and a healthy KV cache, then more replicas.

MoE models such as DeepSeek V4, Kimi K3, GLM-5.3 and Mistral Large 3 activate a small share of their weights per token. Expert parallelism places experts on separate GPUs and works best with data parallelism; since tokens spread unevenly, vLLM’s --enable-eplb rebalances experts from live load statistics [41]. The SGLang team combined expert parallelism with prefill-decode disaggregation for a 671B MoE on H100s; in a decode-only test on 9 nodes of 8 GPUs, with prefill capacity assumed unlimited, it reached about 22,300 output tokens per second per node for 2,000-token inputs, about 5 times a 16-GPU tensor-parallel baseline [42]. That needs both techniques, several nodes and fast interconnect; on one node, tensor parallelism and replicas are simpler.

Split prefill and decode last

The final lever runs prefill and decode on separate GPU pools. DistServe found that “adding a single prefill job to a batch of decoding requests significantly slows down both processes”, and reports 7.4 times more requests or 12.6 times tighter SLOs than colocated systems, with over 90% of requests within their latency limits [7]. Splitwise reports 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput at the same cost and power [2].

Yet vLLM’s documentation says plainly: “Disaggregated prefill DOES NOT improve throughput” [43]. Both claims hold. The papers measure goodput under latency targets and cost-normalised capacity; vLLM’s statement is best read as raw tokens per second. Disaggregation leaves the GPUs as fast as they were. What it changes is interference. Chunked prefill stops the worst stalls, but every mixed step still carries a slice of someone’s prompt, which adds to ITL, and colocated phases “share their resource and parallelism settings” [7]. Separate pools remove both: decode steps carry no prompt work, and each pool gets its own parallelism and batch size, so more requests meet their targets and TTFT and ITL can be tuned separately [43]. The pools are sized separately too: longer prompts call for more prefill GPUs, longer answers for more decode GPUs.

Prefill and decode on separate GPU pools. Prompts of varying length arrive at a router, which scores replicas by prefix overlap and load, and go to a prefill pool where each GPU runs a wide, compute-heavy prefill. Each finished prompt’s KV cache crosses an RDMA link as a stream of packets to a decode pool, where many requests stream tokens in one large, steady batch. A TTFT gauge and an ITL gauge stay inside their target bands. Longer prompts call for more prefill GPUs; longer answers call for more decode GPUs. See the animated diagram in the post.

The router scores replicas by prefix overlap and load. Prefill hands each prompt’s KV cache to the decode pool over RDMA, so a long prompt does not stall the running decode streams. Shaded bands mark the latency targets.

Conditions matter. On GB200 with a 671B reasoning MoE, TensorRT-LLM’s rate-matched curves, which assume ideal pool ratios and free KV transfer, show 1.4 to 1.8 times the output throughput per GPU at the same per-user speed, with MTP off, at 4,400 input and 1,200 output tokens, mostly at lower concurrency; on 8,192-input, 256-output traffic its real end-to-end runs fell 0 to 25% short of the rate-matched curve [44]. SGLang moves the KV cache between pools with RDMA (remote direct memory access, which copies memory between machines without going through the CPU) transfer engines, on InfiniBand or RoCE devices or AWS EFA [45]; vLLM labels the feature experimental [43] and TensorRT-LLM beta [10]. For a single model on a single GPU, Dynamo’s README says the engine alone “is probably sufficient” [23]. Reach for disaggregation with long prompts, several nodes and fast interconnect. We treat it as the last step, after chunk size, routing and replicas have been tuned against goodput.

Checklist

Write the SLO down: TTFT and ITL at p99, and the share of requests that must meet them.
Benchmark on your own prompt lengths, output lengths, shared prefixes and arrival pattern, and report goodput.
Dashboard queue depth, KV cache usage, preemptions and prefix-cache hit rate.
Run a current engine with CUDA graphs on; size memory and batch settings until preemptions stop.
Put stable prompt content first, confirm prefix caching is on, and route by prefix across replicas.
Set the chunk size against your TTFT and ITL targets.
Try FP8 on Hopper, NVFP4 on Blackwell, INT4 weight-only at low concurrency, and a quantised KV cache; check accuracy on your own evals.
Add speculative decoding where concurrency is low to medium, try a model’s built-in MTP heads at higher concurrency with a shorter draft, and watch the acceptance rate.
Cap concurrency where goodput peaks, prioritise interactive traffic and give batch jobs their own lane.
Use the smallest tensor-parallel group that fits, then replicas; expert parallelism with load balancing for MoE.
Split prefill and decode only with long prompts, several nodes and RDMA.

Sources

This post first appeared on Dravin AI.