More users per GPU: a playbook for self-hosted LLM serving

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Prefill keeps the GPU’s arithmetic busy while decode waits on memory, so most of the work is giving decode a bigger batch without letting prefill stall it.
Measure goodput first: the request rate each GPU serves while a target share of requests meet their latency limits.
Pull the levers in order of cost and risk: engine settings and prompt layout, then quantisation and speculative decoding, then parallelism and disaggregation.

A model that answers well in a notebook can be too slow and too expensive once hundreds of people use it at once. Before adding GPUs, we work through engine settings, prompt layout and a few serving techniques in a fixed order, and check each change against numbers we trust.

One fact sits under everything here. Prefill, where the model reads the prompt, keeps the GPU’s arithmetic busy [1], [2]. Decode, where it writes one token at a time per request, uses little of that arithmetic and mostly waits on memory bandwidth [3]. Most throughput work comes down to giving decode a bigger batch while stopping prefill from stalling it.

We serve open-weight models privately, quantised, on vLLM. Versions are current as of 23 September 2026: vLLM 0.30.0, SGLang 0.5.20 and TensorRT-LLM 1.2.1, each the latest stable release [4]. Flags and metric names below are vLLM’s unless we name another engine.

The order follows cost and risk. Batching, KV cache sizing, prefix caching and chunk size are configuration on a current engine, and vLLM 0.30.0 turns most of them on by default, so the work is sizing and checking them. Quantisation and speculative decoding change the arithmetic the model runs, so they need your own evals and your own load. Parallelism and disaggregation change the topology, so they come last.

Lever	What it buys	What it costs
Batch and KV cache sizing	More requests in flight	Per-user latency rises with the batch
Prefix caching and routing	Skipped prefill for shared prompts	Cache memory, and uneven load across replicas
Chunk size	Steady streams while long prompts load	A later first token
Quantisation	Fewer bytes per token, room for more requests	Accuracy risk; formats tied to GPU generation
Speculative decoding	Faster streams at low load	Spare compute; the gain fades as the batch fills
Admission control	Latency targets held under load	Some requests deferred or rejected
Replicas and expert parallelism	Capacity beyond one GPU group	More GPUs and fast interconnect
Disaggregation	Separate control of TTFT and ITL	Two pools to size, RDMA, experimental support

Measure before you tune

Four numbers describe a serving system.

Metric	What it measures	Why it matters
TTFT	Time from sending a request to receiving the first token [5]	How long a user stares at a blank screen
ITL and TPOT	The gap between streamed outputs (ITL), or a request’s decode time spread over its output tokens (TPOT) [6]	How fast the answer streams
Output tokens per second	Output tokens per second, summed over all concurrent requests [5]	What the hardware is worth to you
Goodput	The highest request rate per GPU served while a target share of requests meet their latency limits [7]	The number to optimise

Tools disagree on whether ITL counts the first token, so NVIDIA’s guide says to “compare results only when definitions align” [5]. Pick one tool and keep it.

DistServe defines goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned” [7], where the SLO (service-level objective) is the latency limit you commit to. It leads the dashboard because raw tokens per second can keep rising while users wait longer. System throughput and per-user speed pull against each other, and most levers below trade one for the other.

vllm bench serve measures all of this: set --request-rate and --burstiness to your arrival pattern, cap load with --max-concurrency, and pass your TTFT and TPOT limits in milliseconds to --goodput [6]. Report p50 and p99 for each; averages hide the slow tail. Use your own prompt lengths, output lengths and shared prefixes. Published benchmarks show which levers are worth trying; your own traffic shows how much each one gives.

In production, watch vllm:num_requests_waiting for queueing, vllm:kv_cache_usage_perc and vllm:num_preemptions for memory pressure, and vllm:prefix_cache_hits against vllm:prefix_cache_queries for cache reuse [8].

Get the batch and the KV cache right

The first lever is the engine. Early servers ran a batch until its longest request finished, so new requests queued behind it. Orca introduced “iteration-level scheduling”: the scheduler decides after every decoding step, so finished requests leave and new ones join at once. On a 175B model the paper reports 36.9 times the throughput of NVIDIA FasterTransformer at the same latency [9]. Every current engine does this, under the names continuous batching or in-flight batching [10].

The next constraint is the KV cache, the stored keys and values for every token of every running request. For a 13B model the PagedAttention paper works it out at 800 KB per token in FP16, so a single 2,048-token request can hold 1.6 GB [11]. The general form is 2 × layers × KV heads × head dimension × bytes per element, per token. It grows linearly with context length and concurrency, so the KV cache sets the batch ceiling. Grouped-query attention, where several query heads share one key-value head, shrinks the KV heads term [12]; a model’s config lists its KV head count, so check it before you choose the model.

GPU memory as 48 fixed-size KV cache blocks. Static batch with one slab per request: four requests each reserve a 12-block slab for their maximum length and fill part of it; B has finished but its slab stays held until the batch ends, so queued requests wait and 17 of 48 blocks hold tokens. Continuous batching with a paged cache: requests take small scattered blocks step by step; B's blocks free after step 3, new request E takes all of them at step 4, and by step 6 41 of 48 blocks hold tokens. See the animated diagram in the post.

A slab reserves a request’s maximum length up front and holds it until the request, or the batch, ends. A paged cache adds blocks as tokens arrive and returns them the moment a request finishes.

Before PagedAttention, only 20.4 to 38.2% of KV cache memory held actual token states. Storing the cache in small pages, as an operating system pages virtual memory, brought waste close to zero and gave 2 to 4 times the throughput of FasterTransformer and Orca at the same latency [11].

With a current engine, the work is sizing. vLLM 0.30.0 sets gpu_memory_utilization to 0.92 and picks max_num_seqs and max_num_batched_tokens by GPU [13]. When the cache runs out, requests are preempted and redo their prefill; the documented fixes include a higher gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or more tensor or pipeline parallelism to free memory for the cache [14]. A rising vllm:num_preemptions counter is the first thing we look at. On a new deployment we benchmark at the defaults first, then change one setting per run.

Leave CUDA graphs on. A decode step is a series of tiny GPU jobs, and graph replay skips the CPU cost of launching each one [15]; vLLM captures graphs by default, at some cost in memory and startup time [16]. Keep --enforce-eager for debugging [14].

The cost: a bigger batch raises tokens per second but also each user’s latency, the trade-off Sarathi-Serve is named for [1].

Stop recomputing the same prompt

Many workloads resend the same long prefix: a system prompt, tool definitions, a document, earlier turns of a chat. Prefix caching keeps that prefix’s KV cache, and a new request that shares it skips that part of prefill [17]. SGLang’s RadixAttention keeps prefixes in a prefix tree (a radix tree) with least-recently-used eviction [18]. Both engines turn it on by default in their current releases [13], [19].

It “only reduces the time of processing the queries (the prefilling phase)” [17], so it pays most where prompts are long and answers short: RAG, agents and document chat.

A cache only hits on an identical token prefix. Put stable content first (system prompt, tools, documents), variable content last, and keep timestamps and request IDs out of the top. In SGLang, --schedule-policy lpm reorders the queue by longest prefix match to raise the hit rate [20].

Routing that ignores the cache scatters requests that share a prefix across replicas, and one published benchmark shows the cost [21]. It ran 8 replicas of a 32B model on 16 H100s, with shared-prefix traffic whose cache demand was six times what one replica could hold. A router that tracked each replica’s actual cache contents produced 8,730 output tokens per second against 4,429 for random or load-only routing, and cut the 90th-percentile TTFT from over 90 seconds to about half a second; a router that estimated cache contents from its own routing history reached 6,944 tokens per second and 31 seconds [21]. SGLang’s router keeps such an estimate, an approximate copy of each worker’s prefix tree [22], and NVIDIA Dynamo 1.5.0, an orchestration layer that runs on top of engines such as vLLM, SGLang and TensorRT-LLM, routes “based on worker load and KV cache overlap” [23].

The cost: cache affinity fights load balance, so good routers blend a cache score with a load score, and the cache takes memory that could hold more running requests.

Keep long prompts from stalling everyone

One 50-page document arriving mid-stream can freeze every other user’s output while its prefill runs. Sarathi-Serve’s fix is chunked prefill: cut the prompt into pieces and mix each piece into ordinary decode steps. Against the vLLM of early 2024, the paper reports 2.6 times the serving capacity for a 7B model on one A100 and up to 3.7 times for a 34B model on two A100s [1].

vLLM has since made chunked prefill the default and schedules pending decodes before any prefill [14]. What remains is the dial. In vLLM’s docs, smaller max_num_batched_tokens values such as 2,048 “achieve better ITL”, larger ones “achieve better time to first token”, and values above 8,192 suit throughput, “especially for smaller models on large GPUs” [14]. vLLM 0.30.0’s server already defaults to 8,192 on an H100 and 2,048 on an A100 [13]. SGLang calls the same dial --chunked-prefill-size [20]. Set it against your written TTFT and ITL targets, measured at p99.

Timeline of GPU steps on one replica with five users streaming tokens while a 30,000-token prompt arrives. Without chunking, the prompt runs as one long prefill step and all five token streams stop until it ends, so inter-token latency spikes; the new user then gets a first token. With chunked prefill, the prompt is cut into eight equal slices, each riding along with a normal decode step; the five streams keep ticking at a slightly slower rate and the new user’s first token arrives a little later. A chunk-size scale shows the trade: small chunks such as 2,048 tokens keep streams smooth and delay the first token, large chunks such as 8,192 give an earlier first token and bumpier streams. See the animated diagram in the post.

Chunked prefill cuts a long prompt into slices and runs each one alongside ordinary decode steps, so other users keep receiving tokens while it loads. The two chunk sizes marked are the examples in vLLM’s docs [14].

Spend fewer bytes per token

Decode is limited by the bytes the GPU reads per token, so storing weights and cache in fewer bits gives more tokens per second.

FP8 weights and activations halve weight memory and need Ada (L40S), Hopper (H100, H200) or Blackwell (B200, GB200) GPUs [24]. One accuracy study ran more than 500,000 evaluations on an open model family at 8B, 70B and 405B, in vLLM on A6000, A100 and H100 GPUs. It found FP8 “effectively lossless across all model scales” and well-tuned INT8 within 1 to 3% [25].

INT4 weight-only (AWQ [26], GPTQ [27]) cuts weight memory about four times against FP16. The same study found it the most cost-efficient when requests run one at a time, and 8-bit formats fastest under continuous batching [25]. Weight-only 4-bit speeds up memory-bound decode, and as the batch grows the GPU turns compute-bound: the authors of one 4-bit kernel report close to the full 4 times speed-up per layer up to batch 16 to 32, still significant but shrinking at 64 to 128, and up to 2.8 times end to end in vLLM [28]. For a busy server, that points to an 8-bit format first.

NVFP4 stores 4-bit values in blocks of 16 that share an FP8 scale. NVIDIA reports about 3.5 times less memory than FP16 and 1.8 times less than FP8, and 1% or less accuracy loss moving a 671B mixture-of-experts (MoE) reasoning model from FP8 to NVFP4 [29]. FP4 arithmetic needs Blackwell, though older GPUs can still load NVFP4 weights: Nemotron 3.5 Lightning ships an NVFP4 checkpoint that runs on NVIDIA’s NVFP4 kernels across Blackwell, Hopper and Ampere [30]. On the older GPUs that saves memory and bandwidth, without the FP4 arithmetic.

The KV cache can be quantised separately, which raises the number of requests that fit. vLLM supports an FP8 KV cache and recommends calibrated scales [31]. With Qwen3.8-27B on an RTX PRO 6000 Blackwell, the SGLang, Qwen and NVIDIA teams fitted 70 concurrent requests at 32K context with an NVFP4 KV cache, against 44 with FP8. At matched concurrency, peak-batch decode throughput rose 26 to 30%. SGLang still marks the support experimental [32].

The cost is accuracy and hardware lock-in. Formats are tied to GPU generations, uncalibrated KV scales are a risk, and upgrades can break checkpoints: vLLM 0.30.0 dropped GPTQ activation ordering (the “act-order” option), so check any GPTQ checkpoint before you upgrade [13]. We quantise before we add GPUs, and judge each format on the evaluation set of the product it serves.

Draft tokens while there is spare compute

Speculative decoding lets a cheap drafter propose several tokens and the target model verify them in one pass, keeping the target model’s output distribution [3]. Leviathan and colleagues report 2 to 3 times on an 11B encoder-decoder model [3]. Each target pass yields (1 − α^γ+1) / (1 − α) tokens on average, where α is the acceptance rate and γ the number of drafted tokens: at α = 0.8 and γ = 4, about 3.4. The price is extra arithmetic: each step runs γ+1 positions through the target model in parallel, so “the number of concurrent arithmetic operations grows by a factor of γ+1”, and the method helps only when compute is spare [3].

So the gain shrinks as the batch fills, and vLLM aims speculation at “medium-to-low QPS, memory-bound workloads” [33]. EAGLE-3, a small draft head that reads the target model’s own hidden states, reports up to 6.5 times per-request speed-up on its best task and 1.38 times throughput at batch 64 in SGLang [34]. Plan a busy server around the second number.

Per-user speed-up from EAGLE 3.1, by concurrency

Concurrency 1: 2.03
Concurrency 4: 1.71
Concurrency 16: 1.66

Speed-up in each user’s output speed against no speculation, on a trillion-parameter-class MoE in NVFP4, tensor parallel across four GB200 GPUs, coding dataset [35].

The chart shows the speed each user sees, the number speculation targets; even so, the gain narrows from 2.03 to 1.66 times between concurrency 1 and 16 [35]. P-EAGLE, a newer drafter, shows the same pattern against EAGLE-3: on a 20B open-weight MoE on one B200, its gain fell from 1.55 times at concurrency 1 to 1.05 times at 64 on one chat benchmark, and from 1.69 to 1.25 times on a coding benchmark [36]. The EAGLE 3.1 authors note that speculative decoding “often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts”, the fragility 3.1 was built to reduce [35], so track vllm:spec_decode_num_accepted_tokens against vllm:spec_decode_num_draft_tokens on your own prompts [33]. vLLM can also shrink the draft as load rises: dynamic speculative decoding sets the number of drafted tokens per batch-size range, and an opt-in adaptive verification mode (DSpark drafters only) sizes verification per request from the drafter’s confidence [33].

Multi-token prediction (MTP) trains the model to draft its own next tokens [37]. Nemotron 3.5 Lightning has it built in [30], so there is no separate draft model to train or host; Gemma 4 ships ready-made MTP drafters that attach to the base model and share its embeddings, so there is nothing to train [38]. NVIDIA says “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases” [30].

Protect the latency target

Past a certain queue depth, one more request makes every request late. Cap concurrency where goodput peaks in your benchmark, reject or defer early with a retry signal (an HTTP 429 with a Retry-After header), and give interactive traffic priority: vLLM has --scheduling-policy priority [39] and SGLang has priority scheduling [19]. Nightly jobs belong in their own lane, where “achieving a large batch size is the most important thing” [20].

The cheapest request is one the big model never sees. We route classification and extraction to the smallest model that works, which takes that traffic off the serving fleet entirely; our post on typed decision models covers that lever.

Scale out across GPUs

Tensor parallelism splits every layer across GPUs and needs fast links between them; pipeline parallelism gives each GPU a run of layers; data parallelism runs whole copies of the model (replicas). vLLM advises tensor parallelism within a node, and pipeline parallelism across nodes or on GPUs without NVLink, NVIDIA’s fast GPU-to-GPU link [40]. SGLang says to “always favor data parallelism” for throughput when memory allows [20]. In practice: the smallest tensor-parallel group that holds the weights and a healthy KV cache, then more replicas.

MoE models such as DeepSeek V4, Kimi K3, GLM-5.3 and Mistral Large 3 activate a small share of their weights per token. Expert parallelism places experts on separate GPUs and works best with data parallelism; since tokens spread unevenly, vLLM’s --enable-eplb rebalances experts from live load statistics [41]. The SGLang team combined expert parallelism with prefill-decode disaggregation for a 671B MoE on H100s; in a decode-only test on 9 nodes of 8 GPUs, with prefill capacity assumed unlimited, it reached about 22,300 output tokens per second per node for 2,000-token inputs, about 5 times a 16-GPU tensor-parallel baseline [42]. That needs both techniques, several nodes and fast interconnect; on one node, tensor parallelism and replicas are simpler.

Split prefill and decode last

The final lever runs prefill and decode on separate GPU pools. DistServe found that “adding a single prefill job to a batch of decoding requests significantly slows down both processes”, and reports 7.4 times more requests or 12.6 times tighter SLOs than colocated systems, with over 90% of requests within their latency limits [7]. Splitwise reports 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput at the same cost and power [2].

Yet vLLM’s documentation says plainly: “Disaggregated prefill DOES NOT improve throughput” [43]. Both claims hold. The papers measure goodput under latency targets and cost-normalised capacity; vLLM’s statement is best read as raw tokens per second. Disaggregation leaves the GPUs as fast as they were. What it changes is interference. Chunked prefill stops the worst stalls, but every mixed step still carries a slice of someone’s prompt, which adds to ITL, and colocated phases “share their resource and parallelism settings” [7]. Separate pools remove both: decode steps carry no prompt work, and each pool gets its own parallelism and batch size, so more requests meet their targets and TTFT and ITL can be tuned separately [43]. The pools are sized separately too: longer prompts call for more prefill GPUs, longer answers for more decode GPUs.

Prefill and decode on separate GPU pools. Prompts of varying length arrive at a router, which scores replicas by prefix overlap and load, and go to a prefill pool where each GPU runs a wide, compute-heavy prefill. Each finished prompt’s KV cache crosses an RDMA link as a stream of packets to a decode pool, where many requests stream tokens in one large, steady batch. A TTFT gauge and an ITL gauge stay inside their target bands. Longer prompts call for more prefill GPUs; longer answers call for more decode GPUs. See the animated diagram in the post.

The router scores replicas by prefix overlap and load. Prefill hands each prompt’s KV cache to the decode pool over RDMA, so a long prompt does not stall the running decode streams. Shaded bands mark the latency targets.

Conditions matter. On GB200 with a 671B reasoning MoE, TensorRT-LLM’s rate-matched curves, which assume ideal pool ratios and free KV transfer, show 1.4 to 1.8 times the output throughput per GPU at the same per-user speed, at 4,400 input and 1,200 output tokens, mostly at lower concurrency; on 8,192-input, 256-output traffic its real end-to-end runs fell 0 to 25% short of the rate-matched curve [44]. SGLang moves the KV cache between pools with RDMA (remote direct memory access, which copies memory between machines without going through the CPU) transfer engines, on InfiniBand or RoCE devices or AWS EFA [45]; vLLM labels the feature experimental [43] and TensorRT-LLM beta [10]. For a single model on a single GPU, Dynamo’s README says the engine alone “is probably sufficient” [23]. Reach for disaggregation with long prompts, several nodes and fast interconnect. We treat it as the last step, after chunk size, routing and replicas have been tuned against goodput.

Checklist

Write the SLO down: TTFT and ITL at p99, and the share of requests that must meet them.
Benchmark on your own prompt lengths, output lengths, shared prefixes and arrival pattern, and report goodput.
Dashboard queue depth, KV cache usage, preemptions and prefix-cache hit rate.
Run a current engine with CUDA graphs on; size memory and batch settings until preemptions stop.
Put stable prompt content first, confirm prefix caching is on, and route by prefix across replicas.
Set the chunk size against your TTFT and ITL targets.
Try FP8 on Hopper, NVFP4 on Blackwell, INT4 weight-only at low concurrency, and a quantised KV cache; check accuracy on your own evals.
Add speculative decoding where concurrency is low to medium, try a model’s built-in MTP heads at higher concurrency with a shorter draft, and watch the acceptance rate.
Cap concurrency where goodput peaks, prioritise interactive traffic and give batch jobs their own lane.
Use the smallest tensor-parallel group that fits, then replicas; expert parallelism with load balancing for MoE.
Split prefill and decode only with long prompts, several nodes and RDMA.

Sources

This post first appeared on Dravin AI.

Dravin AI · Blog · LLM serving