Dravin AI · Blog

More users per GPU: a playbook for self-hosted LLM serving

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Prefill keeps the GPU’s arithmetic busy while decode waits on memory, so most of the work is giving decode a bigger batch without letting prefill stall it.
Measure goodput first: the request rate each GPU serves while a target share of requests meet their latency limits.
Pull the levers in order of cost and risk: engine settings and prompt layout, then quantisation and speculative decoding, then parallelism and disaggregation.

A model that answers well in a notebook can be too slow and too expensive once hundreds of people use it at once. Before adding GPUs, we work through engine settings, prompt layout and a few serving techniques in a fixed order, and check each change against numbers we trust.

One fact sits under everything here. Prefill, where the model reads the prompt, keeps the GPU’s arithmetic busy [1], [2]. Decode, where it writes one token at a time per request, uses little of that arithmetic and mostly waits on memory bandwidth [3]. Most throughput work comes down to giving decode a bigger batch while stopping prefill from stalling it.

We serve open-weight models privately, quantised, on vLLM. Versions are current as of 23 September 2026: vLLM 0.30.0, SGLang 0.5.20 and TensorRT-LLM 1.2.1, each the latest stable release [4]. Flags and metric names below are vLLM’s unless we name another engine.

The order follows cost and risk. Batching, KV cache sizing, prefix caching and chunk size are configuration on a current engine, and vLLM 0.30.0 turns most of them on by default, so the work is sizing and checking them. Quantisation and speculative decoding change the arithmetic the model runs, so they need your own evals and your own load. Parallelism and disaggregation change the topology, so they come last.

Lever	What it buys	What it costs
Batch and KV cache sizing	More requests in flight	Per-user latency rises with the batch
Prefix caching and routing	Skipped prefill for shared prompts	Cache memory, and uneven load across replicas
Chunk size	Steady streams while long prompts load	A later first token
Quantisation	Fewer bytes per token, room for more requests	Accuracy risk; formats tied to GPU generation
Speculative decoding	Faster streams at low load	Spare compute; the gain fades as the batch fills
Admission control	Latency targets held under load	Some requests deferred or rejected
Replicas and expert parallelism	Capacity beyond one GPU group	More GPUs and fast interconnect
Disaggregation	Separate control of TTFT and ITL	Two pools to size, RDMA, experimental support

Measure before you tune

Four numbers describe a serving system.

Metric	What it measures	Why it matters
TTFT	Time from sending a request to receiving the first token [5]	How long a user stares at a blank screen
ITL and TPOT	The gap between streamed outputs (ITL), or a request’s decode time spread over its output tokens (TPOT) [6]	How fast the answer streams
Output tokens per second	Output tokens per second, summed over all concurrent requests [5]	What the hardware is worth to you
Goodput	The highest request rate per GPU served while a target share of requests meet their latency limits [7]	The number to optimise

Tools disagree on whether ITL counts the first token, so NVIDIA’s guide says to “compare results only when definitions align” [5]. Pick one tool and keep it.

DistServe defines goodput as “the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned” [7], where the SLO (service-level objective) is the latency limit you commit to. It leads the dashboard because raw tokens per second can keep rising while users wait longer. System throughput and per-user speed pull against each other, and most levers below trade one for the other.

vllm bench serve measures all of this: set --request-rate and --burstiness to your arrival pattern, cap load with --max-concurrency, and pass your TTFT and TPOT limits in milliseconds to --goodput [6]. Report p50 and p99 for each; averages hide the slow tail. Use your own prompt lengths, output lengths and shared prefixes. Published benchmarks show which levers are worth trying; your own traffic shows how much each one gives.

In production, watch vllm:num_requests_waiting for queueing, vllm:kv_cache_usage_perc and vllm:num_preemptions for memory pressure, and vllm:prefix_cache_hits against vllm:prefix_cache_queries for cache reuse [8].

Get the batch and the KV cache right

The first lever is the engine. Early servers ran a batch until its longest request finished, so new requests queued behind it. Orca introduced “iteration-level scheduling”: the scheduler decides after every decoding step, so finished requests leave and new ones join at once. On a 175B model the paper reports 36.9 times the throughput of NVIDIA FasterTransformer at the same latency [9]. Every current engine does this, under the names continuous batching or in-flight batching [10].

The next constraint is the KV cache, the stored keys and values for every token of every running request. For a 13B model the PagedAttention paper works it out at 800 KB per token in FP16, so a single 2,048-token request can hold 1.6 GB [11]. The general form is 2 × layers × KV heads × head dimension × bytes per element, per token. It grows linearly with context length and concurrency, so the KV cache sets the batch ceiling. Grouped-query attention, where several query heads share one key-value head, shrinks the KV heads term [12]; a model’s config lists its KV head count, so check it before you choose the model.

GPU memory as 48 fixed-size KV cache blocks. Static batch with one slab per request: four requests each reserve a 12-block slab for their maximum length and fill part of it; B has finished but its slab stays held until the batch ends, so queued requests wait and 17 of 48 blocks hold tokens. Continuous batching with a paged cache: requests take small scattered blocks step by step; B's blocks free after step 3, new request E takes all of them at step 4, and by step 6 41 of 48 blocks hold tokens. See the animated diagram in the post.

A slab reserves a request’s maximum length up front and holds it until the request, or the batch, ends. A paged cache adds blocks as tokens arrive and returns them the moment a request finishes.

Before PagedAttention, only 20.4 to 38.2% of KV cache memory held actual token states. Storing the cache in small pages, as an operating system pages virtual memory, brought waste close to zero and gave 2 to 4 times the throughput of FasterTransformer and Orca at the same latency [11].

With a current engine, the work is sizing. vLLM 0.30.0 sets gpu_memory_utilization to 0.92 and picks max_num_seqs and max_num_batched_tokens by GPU [13]. When the cache runs out, requests are preempted and redo their prefill; the documented fixes include a higher gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or more tensor or pipeline parallelism to free memory for the cache [14]. A rising vllm:num_preemptions counter is the first thing we look at. On a new deployment we benchmark at the defaults first, then change one setting per run.

Leave CUDA graphs on. A decode step is a series of tiny GPU jobs, and graph replay skips the CPU cost of launching each one [15]; vLLM captures graphs by default, at some cost in memory and startup time [16]. Keep --enforce-eager for debugging [14].

The cost: a bigger batch raises tokens per second but also each user’s latency, the trade-off Sarathi-Serve is named for [1].

Stop recomputing the same prompt

Many workloads resend the same long prefix: a system prompt, tool definitions, a document, earlier turns of a chat. Prefix caching keeps that prefix’s KV cache, and a new request that shares it skips that part of prefill [17]. SGLang’s RadixAttention keeps prefixes in a prefix tree (a radix tree) with least-recently-used eviction [18]. Both engines turn it on by default in their current releases [13], [19].

It “only reduces the time of processing the queries (the prefilling phase)” [17], so it pays most where prompts are long and answers short: RAG, agents and document chat.

A cache only hits on an identical token prefix. Put stable content first (system prompt, tools, documents), variable content last, and keep timestamps and request IDs out of the top. In SGLang, --schedule-policy lpm reorders the queue by longest prefix match to raise the hit rate [20].

Routing that ignores the cache scatters requests that share a prefix across replicas, and one published benchmark shows the cost [21]. It ran 8 replicas of a 32B model on 16 H100s, with shared-prefix traffic whose cache demand was six times what one replica could hold. A router that tracked each replica’s actual cache contents produced 8,730 output tokens per second against 4,429 for random or load-only routing, and cut the 90th-percentile TTFT from over 90 seconds to about half a second; a router that estimated cache contents from its own routing history reached 6,944 tokens per second and 31 seconds [21]. SGLang’s router keeps such an estimate, an approximate copy of each worker’s prefix tree [22], and NVIDIA Dynamo 1.5.0, an orchestration layer that runs on top of engines such as vLLM, SGLang and TensorRT-LLM, routes “based on worker load and KV cache overlap” [23].

The cost: cache affinity fights load balance, so good routers blend a cache score with a load score, and the cache takes memory that could hold more running requests.

Keep long prompts from stalling everyone

One 50-page document arriving mid-stream can freeze every other user’s output while its prefill runs. Sarathi-Serve’s fix is chunked prefill: cut the prompt into pieces and mix each piece into ordinary decode steps. Against the vLLM of early 2024, the paper reports 2.6 times the serving capacity for a 7B model on one A100 and up to 3.7 times for a 34B model on two A100s [1].

vLLM has since made chunked prefill the default and schedules pending decodes before any prefill [14]. What remains is the dial. In vLLM’s docs, smaller max_num_batched_tokens values such as 2,048 “achieve better ITL”, larger ones “achieve better time to first token”, and values above 8,192 suit throughput, “especially for smaller models on large GPUs” [14]. vLLM 0.30.0’s server already defaults to 8,192 on an H100 and 2,048 on an A100 [13]. SGLang calls the same dial --chunked-prefill-size [20]. Set it against your written TTFT and ITL targets, measured at p99.

Timeline of GPU steps on one replica with five users streaming tokens while a 30,000-token prompt arrives. Without chunking, the prompt runs as one long prefill step and all five token streams stop until it ends, so inter-token latency spikes; the new user then gets a first token. With chunked prefill, the prompt is cut into eight equal slices, each riding along with a normal decode step; the five streams keep ticking at a slightly slower rate and the new user’s first token arrives a little later. A chunk-size scale shows the trade: small chunks such as 2,048 tokens keep streams smooth and delay the first token, large chunks such as 8,192 give an earlier first token and bumpier streams. See the animated diagram in the post.

Chunked prefill cuts a long prompt into slices and runs each one alongside ordinary decode steps, so other users keep receiving tokens while it loads. The two chunk sizes marked are the examples in vLLM’s docs [14].

Spend fewer bytes per token

Decode is limited by the bytes the GPU reads per token, so storing weights and cache in fewer bits gives more tokens per second.

FP8 weights and activations halve weight memory and need Ada (L40S), Hopper (H100, H200) or Blackwell (B200, GB200) GPUs [24]. One accuracy study ran more than 500,000 evaluations on an open model family at 8B, 70B and 405B, in vLLM on A6000, A100 and H100 GPUs. It found FP8 “effectively lossless across all model scales” and well-tuned INT8 within 1 to 3% [25].

INT4 weight-only (AWQ [26], GPTQ [27]) cuts weight memory about four times against FP16. The same study found it the most cost-efficient when requests run one at a time, and 8-bit formats fastest under continuous batching [25]. Weight-only 4-bit speeds up memory-bound decode, and as the batch grows the GPU turns compute-bound: the authors of one 4-bit kernel report close to the full 4 times speed-up per layer up to batch 16 to 32, still significant but shrinking at 64 to 128, and up to 2.8 times end to end in vLLM [28]. For a busy server, that points to an 8-bit format first.

NVFP4 stores 4-bit values in blocks of 16 that share an FP8 scale. NVIDIA reports about 3.5 times less memory than FP16 and 1.8 times less than FP8, and 1% or less accuracy loss moving a 671B mixture-of-experts (MoE) reasoning model from FP8 to NVFP4 [29]. FP4 arithmetic needs Blackwell, though older GPUs can still load NVFP4 weights: Nemotron 3.5 Lightning ships an NVFP4 checkpoint that runs on NVIDIA’s NVFP4 kernels across Blackwell, Hopper and Ampere [30]. On the older GPUs that saves memory and bandwidth, without the FP4 arithmetic.

The KV cache can be quantised separately, which raises the number of requests that fit. vLLM supports an FP8 KV cache and recommends calibrated scales [31]. With Qwen3.8-27B on an RTX PRO 6000 Blackwell, the SGLang, Qwen and NVIDIA teams fitted 70 concurrent requests at 32K context with an NVFP4 KV cache, against 44 with FP8. At matched concurrency, peak-batch decode throughput rose 26 to 30%. SGLang still marks the support experimental [32].

The cost is accuracy and hardware lock-in. Formats are tied to GPU generations, uncalibrated KV scales are a risk, and upgrades can break checkpoints: vLLM 0.30.0 dropped GPTQ activation ordering (the “act-order” option), so check any GPTQ checkpoint before you upgrade [13]. We quantise before we add GPUs, and judge each format on the evaluation set of the product it serves.

Draft tokens while there is spare compute

Speculative decoding lets a cheap drafter propose several tokens and the target model verify them in one pass, keeping the target model’s output distribution [3]. Leviathan and colleagues report 2 to 3 times on an 11B encoder-decoder model [3]. Each target pass yields (1 − α^γ+1) / (1 − α) tokens on average, where α is the acceptance rate and γ the number of drafted tokens: at α = 0.8 and γ = 4, about 3.4. The price is extra arithmetic: each step runs γ+1 positions through the target model in parallel, so “the number of concurrent arithmetic operations grows by a factor of γ+1”, and the method helps only when compute is spare [3].

So the gain shrinks as the batch fills, and vLLM aims speculation at “medium-to-low QPS, memory-bound workloads” [33]. EAGLE-3, a small draft head that reads the target model’s own hidden states, reports up to 6.5 times per-request speed-up on its best task and 1.38 times throughput at batch 64 in SGLang [34]. Plan a busy server around the second number.

Per-user speed-up from EAGLE 3.1, by concurrency

Concurrency 1: 2.03
Concurrency 4: 1.71
Concurrency 16: 1.66

Speed-up in each user’s output speed against no speculation, on a trillion-parameter-class MoE in NVFP4, tensor parallel across four GB200 GPUs, coding dataset [35].

The chart shows the speed each user sees, the number speculation targets; even so, the gain narrows from 2.03 to 1.66 times between concurrency 1 and 16 [35]. P-EAGLE, a newer drafter, shows the same pattern against EAGLE-3: on a 20B open-weight MoE on one B200, its gain fell from 1.55 times at concurrency 1 to 1.05 times at 64 on one chat benchmark, and from 1.69 to 1.25 times on a coding benchmark [36]. The EAGLE 3.1 authors note that speculative decoding “often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts”, the fragility 3.1 was built to reduce [35], so track vllm:spec_decode_num_accepted_tokens against vllm:spec_decode_num_draft_tokens on your own prompts [33]. vLLM can also shrink the draft as load rises: dynamic speculative decoding sets the number of drafted tokens per batch-size range, and an opt-in adaptive verification mode (DSpark drafters only) sizes verification per request from the drafter’s confidence [33].

Multi-token prediction (MTP) trains the model to draft its own next tokens [37]. Nemotron 3.5 Lightning has it built in [30], so there is no separate draft model to train or host; Gemma 4 ships ready-made MTP drafters that attach to the base model and share its embeddings, so there is nothing to train [38]. NVIDIA says “MTP is best suited for medium to high concurrency, with the optimal draft length decreasing as concurrency increases” [30].

Protect the latency target

Past a certain queue depth, one more request makes every request late. Cap concurrency where goodput peaks in your benchmark, reject or defer early with a retry signal (an HTTP 429 with a Retry-After header), and give interactive traffic priority: vLLM has --scheduling-policy priority [39] and SGLang has priority scheduling [19]. Nightly jobs belong in their own lane, where “achieving a large batch size is the most important thing” [20].

The cheapest request is one the big model never sees. We route classification and extraction to the smallest model that works, which takes that traffic off the serving fleet entirely; our post on typed decision models covers that lever.

Scale out across GPUs

Tensor parallelism splits every layer across GPUs and needs fast links between them; pipeline parallelism gives each GPU a run of layers; data parallelism runs whole copies of the model (replicas). vLLM advises tensor parallelism within a node, and pipeline parallelism across nodes or on GPUs without NVLink, NVIDIA’s fast GPU-to-GPU link [40]. SGLang says to “always favor data parallelism” for throughput when memory allows [20]. In practice: the smallest tensor-parallel group that holds the weights and a healthy KV cache, then more replicas.

MoE models such as DeepSeek V4, Kimi K3, GLM-5.3 and Mistral Large 3 activate a small share of their weights per token. Expert parallelism places experts on separate GPUs and works best with data parallelism; since tokens spread unevenly, vLLM’s --enable-eplb rebalances experts from live load statistics [41]. The SGLang team combined expert parallelism with prefill-decode disaggregation for a 671B MoE on H100s; in a decode-only test on 9 nodes of 8 GPUs, with prefill capacity assumed unlimited, it reached about 22,300 output tokens per second per node for 2,000-token inputs, about 5 times a 16-GPU tensor-parallel baseline [42]. That needs both techniques, several nodes and fast interconnect; on one node, tensor parallelism and replicas are simpler.

Split prefill and decode last

The final lever runs prefill and decode on separate GPU pools. DistServe found that “adding a single prefill job to a batch of decoding requests significantly slows down both processes”, and reports 7.4 times more requests or 12.6 times tighter SLOs than colocated systems, with over 90% of requests within their latency limits [7]. Splitwise reports 1.4 times the throughput at 20% lower cost, or 2.35 times the throughput at the same cost and power [2].

Yet vLLM’s documentation says plainly: “Disaggregated prefill DOES NOT improve throughput” [43]. Both claims hold. The papers measure goodput under latency targets and cost-normalised capacity; vLLM’s statement is best read as raw tokens per second. Disaggregation leaves the GPUs as fast as they were. What it changes is interference. Chunked prefill stops the worst stalls, but every mixed step still carries a slice of someone’s prompt, which adds to ITL, and colocated phases “share their resource and parallelism settings” [7]. Separate pools remove both: decode steps carry no prompt work, and each pool gets its own parallelism and batch size, so more requests meet their targets and TTFT and ITL can be tuned separately [43]. The pools are sized separately too: longer prompts call for more prefill GPUs, longer answers for more decode GPUs.

Prefill and decode on separate GPU pools. Prompts of varying length arrive at a router, which scores replicas by prefix overlap and load, and go to a prefill pool where each GPU runs a wide, compute-heavy prefill. Each finished prompt’s KV cache crosses an RDMA link as a stream of packets to a decode pool, where many requests stream tokens in one large, steady batch. A TTFT gauge and an ITL gauge stay inside their target bands. Longer prompts call for more prefill GPUs; longer answers call for more decode GPUs. See the animated diagram in the post.

The router scores replicas by prefix overlap and load. Prefill hands each prompt’s KV cache to the decode pool over RDMA, so a long prompt does not stall the running decode streams. Shaded bands mark the latency targets.

Conditions matter. On GB200 with a 671B reasoning MoE, TensorRT-LLM’s rate-matched curves, which assume ideal pool ratios and free KV transfer, show 1.4 to 1.8 times the output throughput per GPU at the same per-user speed, at 4,400 input and 1,200 output tokens, mostly at lower concurrency; on 8,192-input, 256-output traffic its real end-to-end runs fell 0 to 25% short of the rate-matched curve [44]. SGLang moves the KV cache between pools with RDMA (remote direct memory access, which copies memory between machines without going through the CPU) transfer engines, on InfiniBand or RoCE devices or AWS EFA [45]; vLLM labels the feature experimental [43] and TensorRT-LLM beta [10]. For a single model on a single GPU, Dynamo’s README says the engine alone “is probably sufficient” [23]. Reach for disaggregation with long prompts, several nodes and fast interconnect. We treat it as the last step, after chunk size, routing and replicas have been tuned against goodput.

Checklist

Write the SLO down: TTFT and ITL at p99, and the share of requests that must meet them.
Benchmark on your own prompt lengths, output lengths, shared prefixes and arrival pattern, and report goodput.
Dashboard queue depth, KV cache usage, preemptions and prefix-cache hit rate.
Run a current engine with CUDA graphs on; size memory and batch settings until preemptions stop.
Put stable prompt content first, confirm prefix caching is on, and route by prefix across replicas.
Set the chunk size against your TTFT and ITL targets.
Try FP8 on Hopper, NVFP4 on Blackwell, INT4 weight-only at low concurrency, and a quantised KV cache; check accuracy on your own evals.
Add speculative decoding where concurrency is low to medium, try a model’s built-in MTP heads at higher concurrency with a shorter draft, and watch the acceptance rate.
Cap concurrency where goodput peaks, prioritise interactive traffic and give batch jobs their own lane.
Use the smallest tensor-parallel group that fits, then replicas; expert parallelism with load balancing for MoE.
Split prefill and decode only with long prompts, several nodes and RDMA.

Sources

This post first appeared on Dravin AI.

Typed decision models: what early tests show

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Jev and Laya answer typed questions (a choice, a score or a yes/no) with a probability attached, instead of writing text.
Early independent tests: Jev was the fastest model measured, its calibration is mixed, and Laya is near chance until it is fine-tuned.
Before adopting one, test it on a held-out set from your own traffic, against a classifier fine-tuned on your own data.

On 15 September 2026 TypeSafe AI released Jev in early access. It calls Jev its first “System One Model”, a name taken from Daniel Kahneman’s fast, intuitive System 1 thinking [1]. Jev does not write text. You give it a piece of state (a ticket, an email, a JSON object) and a set of typed questions, and it returns a choice, a score or a yes/no drawn from options you defined, with a probability attached. Three days later Convai Innovations published Laya, an open-weight model built on the same idea [11] [13]. Since version 0.3.7 it also ships a server that accepts the same request shape as Jev [11].

Both are days old. The idea still deserves attention, because it describes what many production AI systems do on every request: make a small decision, such as routing a ticket, moderating a message or checking an agent’s tool call, and know when to leave it alone.

The short version: Jev is fast, though the gap is large only against LLMs left in their slowest default modes [15]. On calibration, which is the point of the product, the independent tests are mixed [14] [15]. Laya is near chance until it is fine-tuned [11]. The comparison that should decide adoption is against a classifier fine-tuned on your own data, and the only one we have found is Laya’s own, on a benchmark its checkpoint was fine-tuned on [11].

What a typed decision model is

TypeSafe describes Jev as “a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out” [1]. A request has two parts. The state is the input: a string, a JSON object or an array of text. Images, audio and video are not supported yet [2]. The questions use three primitives [2]:

Choice picks one option from a list you supply, such as which team should handle a ticket.
Score places the state on a rubric, for example customer frustration on a scale from 0 to 2.
Noul is TypeSafe’s name for a yes/no question: it asks whether a statement is true and returns a probability between 0 and 1.

The Register gave an example of what a three-department routing question might return [17]:

{"billing": 0.08, "technical": 0.85, "sales": 0.07}

It paired that answer with a confidence of 0.82. The probabilities are per option. The confidence is a separate summary that TypeSafe derives from the shape of the whole distribution, and it is the number you set thresholds on; a Noul returns only its probability [5].

Jev’s internals are unpublished. TypeSafe says it built “a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)” [1]. TechCrunch reports that the company has said little about that architecture, that outside observers suspect it sits on top of an open-weight LLM, and that the former OpenAI researcher who started the company says Jev is trained only on synthetic data [16]. So Jev can be judged only by what it returns, which is what the rest of this post does.

Laya is the clearest public account of how such a model can work, because its model card spells it out [11]. The English checkpoint is ModernBERT-large, a bidirectional encoder (it reads the whole input at once), fully fine-tuned, with a small decision head on top: 421M parameters in total. A multilingual checkpoint built on mmBERT has 322M. Each option in a question is scored at its own [MASK] token, the placeholder slot a BERT-style model is trained to fill, and the scores are normalised across that question’s options. The options arrive with the request, so a new schema needs no retraining, and every question in a call is answered in a single forward pass. Training rewards the model with strictly proper scoring rules, which pay the most only when the reported probabilities are honest.

Laya, then, is a BERT-style classifier whose label set arrives with each request, trained with an objective that rewards honest probabilities. That places the category between two tools most teams already know. A fine-tuned classifier is fast and accurate, but it needs labelled data and knows only its fixed labels. A zero-shot classifier takes its labels at request time, and one served as the baseline in an independent test below [14]. What typed decision models add is calibration as a training objective, several typed questions answered in one call, and an interface built for software to branch on [1] [11].

How this differs from asking an LLM to classify

Any LLM will produce a label from a prompt and a JSON schema. Three things are different, and the first is narrower than it looks.

The format is guaranteed by construction. “By providing typed, structured values, Jev can avoid the parsing and validating that must be done to process text responses from LLMs,” as The Register put it [17]. An LLM gets most of that guarantee from constrained decoding, which limits its output to a JSON schema or a forced tool call. The independent benchmark below ran every LLM that way: seven of the eight returned malformed answers on at most 0.7% of decisions in any suite, one small model reached 6.2% on one suite, and Jev returned none [15]. The format advantage is real but small, so the differences that matter are the next two.

The work happens in one step. An LLM writes its answer token by token; a typed decision model scores every option of every question at once. TypeSafe credits this parallel design for the speed [1] [3]. The design pays most when one call asks several questions: Laya answers all of them in a single forward pass [11], and TypeSafe says adding questions “barely changes the response time” [9].

Two lanes answer the same ticket routing question. On the left an LLM writes a JSON answer one token at a time, route technical, and a parser then checks it. On the right a typed decision model reads the ticket once and fills a probability for every option: billing 0.08, technical 0.85, sales 0.07, then a confidence of 0.82. The right lane returns every probability in one step; the left lane writes its answer in order and then checks it. See the animated diagram in the post.

The same routing question, asked two ways. The LLM writes its answer one token at a time, then a parser checks it. The typed decision model scores every option in one pass and adds a confidence value. Values from The Register’s example [17].

The probability is the product. A typed decision model is trained so that its per-option probabilities match outcomes, which is the aim of both TypeSafe’s RLCD and Laya’s proper scoring rules [1] [11]. LLMs asked to state their confidence tend to be overconfident, which Xiong and colleagues found across several models [20]. The research is mixed: Tian and colleagues found verbalised confidence from RLHF-tuned models can be better calibrated than their token probabilities [21]. Where an API exposes token log-probabilities, an LLM’s probabilities over the labels can also be read directly, and Kadavath and colleagues found large models well calibrated on multiple-choice and true/false questions posed in the right format [22].

Type safety guarantees the shape of the answer and says nothing about whether the answer is right. TypeSafe says Jev “can’t hallucinate” [1]; The Register noted that structured output with probabilities “does not preclude the possibility of being incorrect” [17]. And because the caller fixes the options, a Choice always picks one of them, even when none fits. TypeSafe’s own documentation says “the Choice is relative, settling which option, while each Noul is absolute and can be low for all of them” [4]. The practical fix is to add a “none of these” option, or to put a separate Noul in front of the Choice as a gate.

Why calibrated probabilities matter in production

A model is calibrated when its probabilities match reality across many predictions. TypeSafe’s primer puts it plainly: “Outcomes assigned a probability of 0.2 should occur about 20% of the time. Outcomes assigned a probability of 0.8 should occur about 80% of the time” [7].

Each small decision a production system makes carries two questions: what is the answer, and is it safe to act on. The CTO of Earendil, quoted by TechCrunch, described the trade: “The user has to say, okay, if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it. But if it’s 95%, sure, then I can do something with it” [16].

TypeSafe’s documentation recommends three bands: act automatically when confidence is high; proceed with caution in the middle, for example by asking a person to confirm; and below that, route to a person, ask for clarification or fall back to another system. It adds that “a confidence threshold is not one number”, since actions with worse consequences deserve a stricter bar [5].

The operating number that follows is coverage at an error budget, an idea from selective classification, where a model is allowed to abstain [23]. Choose the error rate you can accept, say 5%. Set the threshold on labelled data so that errors among the items acted on stay within that rate, then measure what share of traffic the model handles alone. Everything else goes to a person or a larger model. One limit applies throughout: calibration is a property of groups of predictions, and in TypeSafe’s words it “does not guarantee that an individual answer is correct” [2].

Tickets flow into a tagging gate, each with a confidence score. At 0.75 or above the item is acted on, from 0.50 it goes to confirm, and below 0.50 it goes to a person or an LLM. Four of six tickets clear the act threshold and are handled automatically. A stricter payment gate acts only at 0.97 or above, so a decision at 0.80, acted on at tagging, is held for confirmation before any money moves. Set your own thresholds on labelled data, one per action. See the animated diagram in the post.

A confidence gate turns a confidence value into an action. The bar rises with the cost of a mistake, so a decision at 0.80 is acted on at tagging and held before any money moves. Set your own thresholds on labelled data, one per action.

What the first tests show

TypeSafe’s blog gives end-to-end response times of 70 to 500 ms. Its home page claims Jev is “193.6x faster”, a figure the blog says comes from its workflow evals, in which each LLM ran at its provider’s default reasoning setting, and TypeSafe expects these “are on the higher end of real world gains” [1] [10]. The same post says the evals were made by its own capability team, “so some bias could exist”, and that the reference answers are the average of GPT-6 Astra and Claude Fable 5.1 [1] [10]. Accuracy in those evals therefore means agreement with two frontier LLMs, where most buyers would assume human labels. TypeSafe argues that this reference, if anything, favours the OpenAI and Anthropic models in the comparison [1].

Two independent benchmarks with published protocols followed within days. decision-model-benchmark ran Jev against eight LLMs from five providers, each constrained to a JSON schema or a forced tool call, with thinking turned off where the provider allowed it. It declares no vendor sponsorship and publishes its raw logs [15]. Calibration error below is expected calibration error (ECE), the average gap between stated confidence and observed accuracy; lower is better and 0 is perfect.

Finding	Jev	The eight LLMs
Accuracy, 77-way banking intent	76.3%	70.9% to 81.3%
Accuracy, SMS spam	93.0%	66.1% to 94.9%
Median latency per request	264 to 276 ms	303 ms to 5.6 s
Most options accepted in one question	255	512 (the most tested)
Low confidence (0.5 or below) when no option is correct	49.7%	97.3% to 100% for seven; 64.7% for one
Answers changed by shuffling option order	13%	15% to 37%
Calibration error, banking intent	0.083	0.043 to 0.226
Calibration error, SMS spam	0.249	0.042 to 0.197
Calibration error, suite built to force uncertainty	0.246	0.039 to 0.122

The authors’ summary is that “no class wins on quality”. Jev was the fastest model measured, about 1.2 times faster than the fastest LLM set-up (an open-weight model on specialised inference hardware) and 10 to 16 times faster than models in thinking mode, and they conclude that “the vendor’s speedup claim holds only against LLMs left in their slowest default mode” [15]. Calibration is mixed: Jev sits inside the LLMs’ range on banking intent and above it on spam and on the forced-uncertainty suite. The 49.7% is the relative Choice described above at work.

jev-benchmarks, a pre-registered pilot on 300 held-out examples against an open zero-shot classifier, found Jev more accurate on two of three tasks, where it could handle 83% and 86% of items alone at a 5% error budget [14]. On the third, emotion labelling, Jev put zero probability on the true label for 16% of examples and could automate none of them at that budget; the open classifier managed 2%. The authors fitted those thresholds on the same small slice, so they describe this sample only [14].

Laya documents its own limits unusually well [11] [12]. Its English base checkpoint scores 0.362 on a benchmark of four typed-decision workflows without fine-tuning: above the 0.318 random baseline, and below the 0.461 you would get by always guessing the most common answer. The headline 0.766 belongs to a checkpoint fine-tuned on that benchmark’s own training split. As shipped, the probabilities are overconfident. Fitting one temperature (a single number that sharpens or softens every probability) per question type and option count moves mean ECE from 0.466 to 0.081, per its model card, measured before a temperature change in a later release [11].

Held-out results are weaker than in-training ones. On held-out toxic chat, moderation accuracy is 0.530, barely above chance on a balanced split, and the authors put it bluntly: “hand-picked examples work, real traffic does not” [12]. The English checkpoint scored 0.000 accuracy on Khmer, and its authors warn that it stays confident while wrong, which is why their router picks a checkpoint by script before the model runs [11] [12]. The confidence figures from that language sweep were measured before the same temperature change; the maker has since re-run it, and accuracy held while the confidence figures moved [12]. A confidence gate protects you only on inputs like the ones the model was trained and calibrated on, so check the language and shape of inputs before the gate.

Laya’s card also has a table that puts it ahead of Jev on most rows, while stating that the Jev figures are “third-party published, never measured here” [11]. Its Jev calibration figure is the 0.246 from the forced-uncertainty suite above, and its own figure is measured after the temperature refit. Each side’s table suits its maker, and neither is a like-for-like comparison.

Both products are young. Jev is eight days old at the time of writing. At launch, access was through a waitlist [1], and TechCrunch reports that the company “briefly lost the ability to serve users from its API because demand was so high” [16]. Its rate limits “can change without notice”, and the same weights serve every account, so it cannot be fine-tuned on your data. TypeSafe says “Jev is not trained on customer requests or responses”, and that English is where its accuracy is best [3].

Laya’s first release reached PyPI on 18 September 2026 [13]. Its weights and code are open under Apache 2.0, so you can inspect and run it yourself. It needs fine-tuning and a temperature fit on your task first. Its escalation head, action.act_probability, “carries no usable signal yet”, and the card advises gating on confidence instead. Its bundled server listens on every network interface (0.0.0.0) with no authentication unless LAYA_API_KEY is set [11], so set the key before the port is reachable.

The evidence so far is vendor evals, one 300-example independent pilot and one independent benchmark of five small suites.

Choosing between an LLM, a typed model and a fine-tuned classifier

TypeSafe’s list of failure modes for jev-1.13 covers literal reading, arithmetic, date comparisons, questions whose answer depends on first answering another question, and counting: “jev-1.13 does not count reliably” [4]. Its advice is to keep arithmetic, dates and counting in code [4], and to have each question ask one well-scoped thing, composing the answers in code [8].

With that in mind, each of the three options has a place.

Option	When it fits
An LLM	The output is language, code or an explanation someone will read, or the decision needs several steps of reasoning. Jev “does not generate text, write code, or hold a conversation” [6]. An LLM can also draft labels for training a classifier, with people checking a sample, and give a second opinion on a low-confidence case before it reaches a person.
A typed decision model	The options are fixed per request, one call asks several questions, and you have no labelled data yet. Used zero-shot, Jev beat an open zero-shot classifier on two of three tasks in one pilot [14]. Laya fits only after fine-tuning, since its base checkpoints are near chance [11], and its authors advise keeping Choice questions under about 20 options [12].
A fine-tuned classifier	You have labelled data and the label set is stable. A 2024 study by Bucher and Martini found small fine-tuned models consistently outperformed zero-shot prompted LLMs on text classification [24].

How we evaluate a decision model before trusting it

Build a held-out set from your own traffic, labelled by people and never used for prompt tuning or threshold fitting. Laya’s results show why: spam, which was in its training mix, scores 0.993, against the held-out moderation result above [12].
Measure discrimination and probability quality separately. Accuracy and macro-F1 (which counts rare labels as much as common ones) for the first. For the second, Brier score and log loss, which penalise confident wrong answers, and calibration error with a reliability diagram like the one below.
Fit a temperature on a separate calibration split and report calibration before and after. A hosted model’s weights cannot be tuned, but its output probabilities can still be recalibrated on your split. Temperature scaling has a single parameter and is often enough [18]. Laya found overconfidence on some tasks and underconfidence on others [12], so check each question type.
Report coverage at your error budget for each action separately, since each action has its own threshold [5].
Probe the edges. Shuffle option order, include items with no correct option, ask a question and its negation, and plant instructions inside the state. TypeSafe’s own docs show a question and its negation on the same ticket returning 0.72 and 0.47, which sum to 1.19 [4]. They also note that injected content “can move the answer” [4], which matters most when the model is a guardrail.
Measure latency on the real path, at the median and the 95th percentile. Laya’s 33 to 40 ms is one question on a T4 GPU, depending on the checkpoint; on CPU its card gives 193 to 464 ms [11]. Jev’s figures are hosted round trips.
Plan for drift. Pin the model version, since TypeSafe warns that an alias moves when a new release ships [3]. Log every probability, and recheck calibration on fresh labelled samples on a schedule. Calibration fitted on past data degrades when the inputs shift [19].

A reliability diagram plots predicted probability against observed frequency, with the diagonal as perfect calibration. The bins start below the diagonal, meaning the model is overconfident. A single temperature dial turns and the bins settle close to the diagonal: for Laya's English checkpoint the model card reports mean expected calibration error falling from 0.466 to 0.081. The bars sketch the pattern; only the error figures are measured. In general, when the inputs shift, a fit made on past data drifts off the diagonal again, so calibration has to be rechecked on fresh labels. See the animated diagram in the post.

One temperature, fitted on a calibration split, pulls the bins close to the diagonal. A shift in the inputs pulls them off again, so the fit needs rechecking on fresh labels. The bars sketch the pattern; the error figures are from Laya’s model card [11].

Where this fits in how we build

We send classification to small trained classifiers and keep LLMs for work that needs language. Typed decision models share that view: a decision your software branches on should come from a model trained to return a calibrated label. Our view is that the most dependable path is still a small model fine-tuned on your own labelled data, for the reasons in the table above, and the gap between Laya’s zero-shot and fine-tuned results points the same way. We have not found a comparison of Jev with a classifier fine-tuned on a buyer’s own data, so that is the test worth running. Since Jev’s weights cannot be tuned per account [3], the tuning moves into how you word the questions and criteria, and into the thresholds in your code.

In practice we fine-tune encoder classifiers on a client’s data, calibrate and threshold their outputs, gate each action by confidence, and send the uncertain tail to an LLM or a person. Jev and Laya are worth testing against that baseline, on a held-out set, before either replaces it. If you have a classification step that is slow, expensive or hard to trust, tell us about it.

Sources

TypeSafe AI, “Introducing System One Models & Jev”, 15 September 2026. typesafe.ai
TypeSafe docs, “System One”. docs.typesafe.ai/concepts/system-one
TypeSafe docs, “Models” (jev-1.13.0, rate limits, fine-tuning, data handling, version pinning). docs.typesafe.ai/models
TypeSafe docs, “Jev 1.13 jaggedness”, last reviewed 17 September 2026. docs.typesafe.ai/model-jaggedness/jev-1.13
TypeSafe docs, “Confidence”. docs.typesafe.ai/confidence
TypeSafe docs, “Jev with coding agents”. docs.typesafe.ai/introduction/coding-agents
TypeSafe docs, “AI primer”. docs.typesafe.ai/introduction/machine-learning-primer
TypeSafe docs, “Introduction”. docs.typesafe.ai/introduction
TypeSafe docs, “Primitives”. docs.typesafe.ai/primitives
TypeSafe, workflow evals. evals.typesafe.ai
Convai Innovations, Laya model card (v0.3.7). huggingface.co/convaiinnovations/laya
Laya, BENCHMARKS.md. github.com/NandhaKishorM/laya
Laya on PyPI, first release 18 September 2026. pypi.org/project/laya
jev-benchmarks, pre-registered pilot, September 2026. github.com/AbdelStark/jev-benchmarks
decision-model-benchmark, README and v2 report of record, September 2026. github.com/nibzard/decision-model-benchmark
Tim Fernholz, “A new kind of AI model from a ChatGPT inventor is thrilling developers”, TechCrunch, 18 September 2026. techcrunch.com
Thomas Claburn, “TypeSafe AI debuts model for machines that plays Doom”, The Register, 16 September 2026. theregister.com
Guo, Pleiss, Sun and Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017. arXiv:1706.04599
Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift”, NeurIPS 2019. arXiv:1906.02530
Xiong et al., “Can LLMs Express Their Uncertainty?”, ICLR 2024. arXiv:2306.13063
Tian et al., “Just Ask for Calibration”, 2023. arXiv:2305.14975
Kadavath et al., “Language Models (Mostly) Know What They Know”, 2022. arXiv:2207.05221
Geifman and El-Yaniv, “Selective Classification for Deep Neural Networks”, NeurIPS 2017. arXiv:1705.08500
Bucher and Martini, “Fine-Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification”, 2024. arXiv:2406.08660

This post first appeared on Dravin AI.

The 90 firms that fix AI-generated code

info@dravin.ai (Dravin AI) — Mon, 21 Sep 2026 18:30:00 GMT

TL;DR

We found 90 firms that fix apps built with AI tools. Over half name security as the problem; only two name duplicated logic.
Not one of the 90 names runaway AI API spend as something they fix.
The market has settled on one sequence: read the code, stabilise what is dangerous, refactor or rebuild what will not hold, then watch it.

In September 2026 we went looking for companies that sell cleanup of AI-generated code: apps built with Lovable, Bolt, Cursor, Replit, v0, Claude Code and similar tools, by people who then got stuck. We found 90. This post is what those 90 say they fix, how they package the work, and what we think a buyer should demand from any of them.

Why this category exists

In our reading of the 90 service descriptions, the typical caller in 2026 is a business owner, a product manager or a COO who built something real with an AI tool and then met real users, a payment that fails, or a security email from a stranger.

The demand is visible in public. On r/ExperiencedDevs, the thread “Getting more calls to fix ai generated codebases than actual new builds lately” reached 405 upvotes and 102 comments; one comment: “Cleanup contracts are gonna be a whole market segment.” On r/SaaS, an MVP dev shop said it lost half its pipeline to Claude Code in 2025; of the prospects who tried it instead, about a third shipped, a third broke in production, and three came back for cleanups. On r/vibecoding, a post pointing to a vibe-code fix service, asking builders about getting their “mess” fixed, drew 1,086 upvotes and 125 comments.

The failure rate is measurable. Veracode’s 2025 report found that AI-generated code introduced security flaws in 45% of its tests. How Bad Is It?, a public URL scanner for Lovable, Bolt and v0 apps, publishes a running counter: 4,271 apps reviewed, median score 38 out of 100.

45% of Veracode tests where AI-generated code introduced a security flaw. Source: Veracode, 2025
38 median score out of 100 across 4,271 Lovable, Bolt and v0 apps scanned. Source: How Bad Is It?

Who the 90 are

Type of firm	Count
Established dev agencies that added a cleanup service line	48
Human audit specialists	13
Agencies formed for rescue work	10
Automated scanners	10
Solo fixers and freelance studios	6
Marketplaces and expert networks	3

Twenty are in the USA, 18 in Europe outside the UK, 7 in India, 6 in the UK, 4 in Canada, 11 elsewhere, and 24 give no location. Size is mostly undisclosed (53 of 90); of the rest, 6 have 250 or more people, 10 are mid-sized and 21 are small, micro or one-person.

It is a young and thin market. We could open 75 of the 90 websites; 15 are known only from directory listings, and three domains (Vibe App Rescue, VibeCheckAudit, VibeFixLab) no longer resolve. Fifty-five name an AI builder in their service copy; Lovable leads (50), then Cursor (46), Bolt (44) and Replit (41).

What they say breaks

We counted which failure classes each firm’s service description names.

Failure class named in the service description	Entries (of 90, scanners included)
Security in general (vulnerabilities, OWASP, hardening)	48
Missing or broken tests	25
Performance and scalability	25
No deploy pipeline, CI/CD or hosting setup	23
Authentication and access control (including Supabase RLS)	18
Data model or database problems	15
Secrets and API keys in code	10
Monitoring and observability	9
Documentation	9
Payments and Stripe	8
Duplicated logic and coupling	2
Hallucinated APIs	1
Runaway AI API spend	0

Security is the sales pitch. Over half the market names it, and seven of the ten automated scanners are security scanners: Supabase row-level-security probes, leaked-key detection, public buckets. Beesoul, a US-listed audit shop (figures from its directory listing), reports that 10.3% of the Lovable apps it audited had critical RLS vulnerabilities, and that a typical MVP has 8 to 14 findings. Sidekick Interactive’s case study is typical of the genre: exposed API keys, database holes and broken payments in one app.

Structure is under-sold. Only two of the 90 entries, one of them a scanner, mention duplicated logic or unsafe coupling by name, though that is what makes an AI-generated codebase slow and risky to change. A security patch is usually local; untangling several copies of the same auth check, each slightly different, is where the work goes.

Nobody sells a spending cap. Not one of the 90 names uncontrolled AI API usage as something they fix; the two that mention “cost” at all mean something else. Yet on r/cursor, a thread about a Cursor bill that spiked within one hour, after a PM asked the agent to tag 87 tasks, reached 228 upvotes. An app that calls a model on every request with no budget, rate limit or per-user cap is a bill waiting to happen.

How they package it

The 80 human-service firms (the 90 minus the 10 scanners) use four patterns, often more than one at once.

Packaging pattern	Firms (of 80)
Audit, assessment or review named as the first step	61
Audit or assessment promised within 48 hours	10
Work described in sprints	9
Rebuild or re-architecture offered as one option	9
Explicitly against rebuilding	5
Retainer, maintenance or monitoring mentioned	16

Audit first. Valletta Software (Malta) is the cleanest version: a senior engineer reviews the repository across eight areas, delivers the audit in 48 hours with a debrief call, then scopes the cleanup, typically six to eight weeks. MetaCTO runs a 48-hour code audit, then promises production-ready in 30 days, with weekly demos.

Fixed-scope rescue. AxonBuild sells a 10-working-day production-engineering sprint covering auth, secrets, payments, CI/CD and monitoring, ending in a verified handover with two weeks of defect cover and 30 days of async support. Relux Works (Armenia) runs a one-week audit, then a two-to-three-week stabilisation sprint, then what it calls “de-vibe-coding”: moving the app to a production architecture.

Rebuild. Usually offered for the broken parts only, and five firms say outright that they avoid it. Bitnoise (Poznań) shows the whole sequence: audit in 3 to 7 business days, stabilisation sprint 2 to 4 weeks, re-architecture 6 to 12 weeks.

Retainer. Vibe-Audit (Barcelona, a one-person shop) runs a pre-launch security sweep delivered in 48 hours, done-for-you fixes, and then ongoing security monitoring.

Side by side, the market has settled on one sequence: read the code, stabilise what is dangerous, refactor or rebuild what will not hold, then watch it.

How cleanup is sold

Read the code: audit or review first
Stabilise: what is dangerous now
Refactor or rebuild: what will not hold
Watch it: retainer or monitoring

The sequence the 80 human-service firms describe, often combining more than one pattern.

What a good cleanup engagement includes

Use this on anyone you are considering, including us.

A written review before any commitment, with findings ranked by severity and a verdict: ship, patch, refactor or rewrite. If the verdict is “not worth fixing”, they should say so.
An experienced engineer who has actually read the code. A scanner finds leaked keys; it does not find the auth check that is bypassed on one route.
Security tested against the running app, not only the repository: authentication, per-row authorisation, secrets rotated rather than just deleted from git, public storage buckets closed.
Characterisation tests written before refactoring, so current behaviour is pinned and the cleanup cannot silently change it.
Duplicated logic collapsed and the data model reviewed, because that decides whether the next feature takes a day or a month.
A deploy pipeline and monitoring at the end: staging, CI, error tracking, backups. Working on one person’s laptop does not count as done.
AI API spend bounded: budgets, rate limits, per-user caps and an alert. Nobody in the 90 sells this; ask for it anyway.
Handover: documentation, and you holding the repository, the cloud accounts and the keys. You own the code from day one.
A fixed scope with a definition of done and a defect-cover window afterwards.

How we do it

Dravin AI is an AI software company. We build AI systems and software, and we clean up AI-built apps. Against the list above, this is what we commit to:

A written report first. Read-only access to one repository, and within 48 hours a written report with findings ranked by severity: the file, the risk and the fix, and what to leave alone. If nothing needs fixing, it says so. There is no obligation to go further.
Secrets. A secret we find in your repository is reported the same day, ahead of the report, and rotated within 24 hours.
Auth and data access. Every route’s auth check, database access policies included, is traced in the code. Testing the running app is agreed on the call, because it needs access beyond the repository.
Tests before refactoring. Tests on the paths that make money, run on every pull request, and structure changed in small, tested steps.
Deploys. From the repository, with a rollback that has been tried.
AI API spend. Timeouts, budgets and retry caps on every model call.
Ownership. The code is yours from the first commit, we work on branches, never main, and our access is removed within seven days of handover, confirmed in writing.
Defect cover. Anything that misses the statement of work, reported in writing within 30 days of handover, is corrected under it.

An engagement starts with a call about what you built, what it does in production and what worries you. If that is an app that got you to traction, book one or request a code review.

Method

Every count comes from the cleanup list in our research workbook: 90 firms, each with a segment, a region, a size band, a description of what it does, its delivery model and its stated turnaround. We read the list on 22 September 2026. The workbook itself is not published; these are the counting rules.

Type, region, size: counts of each firm’s segment, region and size band. “Small, micro or one-person” is Small (10 to 49) plus Micro (2 to 9) plus Solo.
Reachability: whether we could fetch the firm’s site or know it only from a directory listing, and whether its domain still resolves.
Tool mentions: a case-insensitive search of each firm’s description for each builder’s name.
Failure classes: a case-insensitive keyword search of the description only. Security: secur, vuln, OWASP, pentest. Tests: test, QA; Sherlock Forensics matches only through “pentest” and is excluded, so 25 rather than the mechanical 26. Performance: perf, scalab, bottleneck. Deploy: CI/CD, CI, deploy, pipeline, hosting, DevOps. Auth: auth, access control, RLS. Data model: data model, database, DB, schema, persistence, RLS. Secrets: secret, key(s), credential. Monitoring: monitor, observab, Sentry. Documentation: doc, docs, document or documentation as whole words. Payments: Stripe, payment. Duplication: duplic, coupling, dead code, untangle. Hallucination: hallucinat. AI spend: cost, spend, token, bill (the two hits, *instinctools’ “cost-benefit” and Vibe Code Rescue’s “perf & cost”, are not API spend). Scanner types: the descriptions of the 10 automated scanners.
Packaging: over the 80 firms that are not automated scanners, searching the description, the delivery model and the stated turnaround unless a rule says otherwise. Audit-first: audit, assess, review, scan, diagnos or scorecard. 48 hours: the turnaround promises an audit, assessment, scan or check result in 48 hours or less; fix times and reply times excluded. The ten: MetaCTO, Pfaff Digital, Sonder, VibeAudits, Mitrix Technology, Valletta Software, ShipClarity, Vibe-Audit, VibeCodeBlue and one solo fixer. Sprints: “sprint” anywhere in those three fields. Rebuild offered: rebuild, re-architect, rewrite or full build (11 firms), minus the two of them that also say “without rebuild”, “no rewrites”, “not rebuild”, “diagnosis-over-rebuild” or “without starting from scratch”. Against rebuilding: one of those five phrases in those three fields, the firm’s hero line or the workbook’s notes on how each firm positions itself (5 firms). Retainer: retainer, maintenance, ongoing, monitoring, subscription, defect cover or post-rescue support (16 firms).
Named offers, turnarounds and claims are quoted from what each firm publishes, unaudited.
Reddit and studies: r/ExperiencedDevs (405 upvotes, 102 comments), r/SaaS (65 upvotes, 66 comments) and r/cursor (228 upvotes) from our AI-agency landscape research (48 searches, 2,837 posts, 17 September 2026); r/vibecoding (1,086 upvotes, 125 comments, 24 December 2025) from a separate Reddit sweep for the workbook; Veracode’s 45% from the same landscape research.

This post first appeared on Dravin AI.

What 843 AI agencies actually sell

info@dravin.ai (Dravin AI) — Mon, 21 Sep 2026 18:30:00 GMT

TL;DR

Automation and custom builds account for 457 of the 843 firms we mapped. Only 31 sell audit, security or testing.
The thinnest part of the market is what a system needs once it is live: evaluation, monitoring and someone who answers when it breaks.
Size is unknown for 569 of the 843, so ask who will do the work, what happens after the build and how they will know it works.

Between 17 and 22 September 2026 we built a workbook of every AI agency, consultancy and AI-native delivery firm we could find: 843 firms across India, the US, Europe and a long tail of other regions. We wanted to see the market the way a buyer does. Every count below was computed from that workbook; the method is at the end.

The short version: most of the market sells the same two things, automation and custom builds, which together account for 457 of the 843 firms. Only 31 sell audit, security or testing. And the work a team needs once something is in production, evaluation, monitoring and someone on the hook when it breaks, is the thinnest part of the map.

What we counted and how

Eight research passes (Americas and ANZ, India, Europe and the Middle East, design-led studios, consulting and advisory, Asian, vertical and voice agencies, AI-code cleanup, freelancers), directory sweeps (Clutch, GoodFirms, DesignRush, Sortlist, and the n8n, Make and Zapier directories), and 211 firms carried over from our earlier landscape note. Duplicates were merged on domain. Of the 843 sites, 819 loaded and 619 homepages were read on the research date.

Each firm has a region, one of 35 segments, a size band and a description of what it does. Kept separately, outside the 843: 90 firms that fix AI-generated code, 103 independent consultants, and a 40-row catalogue of every offer type we saw, each rated by how many credible sellers it has.

One caveat: size is known for only 274 of the 843. Most firms do not say how big they are, which matters later.

The regional shape

Firms by region · 843 firms

India: 209
USA: 162
Europe (as tagged): 146
UK: 55
Australia and NZ: 41
Asia (other): 39
Middle East: 35
Latin America: 32
Canada: 28
Africa: 14
Global, multi-country, other or unknown: 82

All 843 firms by the region each is tagged with. Europe is the tag as found and holds some UK-based firms; the Method breaks down the last row.

The regions sell different things. India’s 209 are build shops: 46 dev studios, 27 boutiques, 21 mid-sized IT firms, 29 automation agencies. The US mix is flatter: 21 vertical agencies, 21 automation agencies, 19 dev studios, 11 forward-deployed engineering firms and all 9 big-consultancy AI arms. Europe holds the incumbents: 31 IT majors and consultancies, 15 nearshore firms, 14 data engineering firms. The UK leans to advice: 6 of the 14 governance firms and 5 of the 11 training firms are British.

What firms sell

The 35 segments group into nine offer families. The grouping is ours; every firm lands in exactly one.

Firms by offer family · 843 firms

Automation and agents as a service: 246
Custom build: 211
Embedded and enterprise delivery: 123
Advice and strategy: 117
Data engineering: 62
Audit, security and QA: 31
Platforms and tooling: 26
Training and community: 18
Other: 9

All 843 firms by offer family. Audit, security and QA, the rust bar, is 31 firms; automation and custom build together are 457. The table below lists the segments inside each family.

Offer family	Firms	Segments inside
Automation and agents as a service	246	142 SMB automation agencies, 70 vertical agencies, 26 voice-AI agencies, 4 productised services, 4 automation-and-coaching hybrids
Custom build	211	123 dev studios, 27 Indian boutiques, 26 design-led studios, 15 nearshore firms, 11 engineering firms with an AI practice, 9 AI-native delivery firms
Embedded and enterprise delivery	123	35 FDE firms, 31 IT majors based in Europe, 21 Indian mid-sized IT firms, 9 big-consultancy arms, 8 established Indian engineering firms, 7 platform-plus-services firms, 6 IT-major practices, 6 lab and hyperscaler arms
Advice and strategy	117	51 AI consultancies, 19 research and advisory, 17 strategy consultancies, 12 industry consultancies, 12 MSPs, 6 fractional-leadership firms
Data engineering	62	50 data and AI engineering firms, 12 analytics consultancies
Audit, security and QA	31	14 governance and compliance, 10 security and red-teaming, 7 QA and testing
Platforms and tooling	26	15 AI-native startups, 11 developer-productivity and AI-impact platforms
Training and community	18	11 training firms, 7 coaching communities
Other	9	Uncategorised

Automation-as-a-service and custom build account for 457 of the 843; audit, security and QA are 31. That ratio is the most useful fact in the workbook. Almost everyone will build the thing. Very few will tell you whether the thing is safe, correct or still working a month later.

What is crowded and what is thin

The catalogue rates each of the 40 offers by how many credible sellers we found: 22 crowded, 15 moderate, 3 thin.

Crowded:

Custom builds (agents, RAG systems, voice agents, internal tools, integrations) are the default offer of nearly every automation agency. The catalogue’s note is that capability is much the same everywhere; firms differ by vertical depth or proof.
Discovery and readiness audits take the least capital to start, so almost every new operator leads with one.
Staff augmentation is the default Indian services business, with hundreds of Indian shops selling the same thing.
AI support agents that resolve tickets are sold by nine platforms, bundled into the inbox the buyer already owns.
Data plumbing for AI (pipelines, warehouses, vector databases) is offered by every data consultancy and is the easiest thing on the list to buy.

Thin:

Evaluation harnesses, benchmarking and observability. Dozens of funded tool vendors, very few independents doing the implementation, because it takes real ML judgement.
Data-quality and AI-data-readiness audits. Crowded as software, thin as a service a person delivers.
Managed agents: monitoring, evals and a support SLA for AI systems already in production. Half a dozen vendors sell the dashboard; the human side, someone who answers when the agent breaks at 2am, is thin.

Codebase audit sits next to them, rated moderate: the automated side is crowded and well funded, and the human review is where the catalogue sees the gap. The cleanup list shows that gap starting to fill: 90 firms now sell cleanup of AI-generated code, 48 of them as a service line inside an existing dev agency and 13 as human audit specialists.

The firm counts say the same thing. 457 firms build. 31 audit, secure or test. 11 are platforms in the developer-productivity and AI-impact segment, measuring what AI is doing to engineering teams. Meanwhile the landscape’s Reddit sweep found an r/ExperiencedDevs thread titled “Getting more calls to fix ai generated codebases than actual new builds lately” at 405 upvotes.

What to check before hiring anyone

Ask who will be on the work. Size is unknown for 569 of the 843, and a website rarely tells you whether you are talking to a two-person studio or a firm of 7,000. Ask how experienced the engineers are, who checks their code before it merges, and whether any of it is subcontracted.
Ask what happens after the build. The market is crowded with builds and thin on evaluation, monitoring and support for systems already in production. Get a written answer on who maintains the thing, how a failure is detected and who answers when it breaks, before you sign for the thing.
Ask how they will know it works. An agent or a RAG system without an evaluation harness is a demo. Few firms sell evaluation as a service, so ask to see the test set, the metrics and the regression checks for the system you are buying, and ask what happens when a model version changes underneath it.
If you already have AI-generated code, get it read before you scale it. 457 firms will happily build on top of it; 31 sell audit, security or testing, plus the 90 cleanup firms outside the main list. A careful read finds the auth check missing on one route, the key committed to the repository and the same logic copied into several places. Each is easier to fix before the next feature lands on top of it.

What we do

Dravin AI is an AI software company. We build AI systems and software. Coding agents draft in parallel, each in its own isolated environment, automated checks run on every pull request, and an engineer approves each change before it merges. We also audit code and clean up AI-built apps.

What we build runs from agents, voice AI, fine-tuned models, classical ML and forecasting systems to web, mobile and desktop apps, and we deploy and run them in the cloud.

We also work in the thin part of this map: evaluation harnesses, and AI-generated codebases made safe enough for a real team to own. An engagement starts with a call. If any of the above describes where you are, book one and bring the repository.

Method

Every count comes from our research workbook, built between 17 and 22 September 2026 and counted on 22 September. The workbook itself is not published; these are the counting rules.

843: every firm on the main list; 211 of them came from our earlier landscape note.
Regions: the region each firm is tagged with, exact match. The 82 is Global 33, Other 24, Unknown 23, Multi-country or remote 2. Europe is the tag as found; it holds some UK-based firms.
Segments and families: each firm’s segment, exact match, filtered by region where one is named. The nine families are our grouping of the 35 segment values; the table is the mapping and it sums to 843. The 11 platforms are the segment “Dev-productivity / AI-impact platform”.
Size: each firm’s size band: Solo 17, Micro 60, Small 97, Mid 76, Large 24, Unknown 569; 274 known.
819 and 619: sites that passed a link check, and homepages read on the research date.
90, 48 and 13: the firms on the cleanup list; those whose segment is a dev agency with a cleanup service line; those whose segment is a human audit of AI-generated code.
103 consultants: the firms on the list of independent consultants and freelancers.
40 offers; 22, 15, 3: the offer catalogue’s 40 entries, each rated crowded, moderate or thin by how many credible sellers we found. The crowded and thin notes paraphrase the catalogue’s notes on custom builds, discovery and readiness audits, staff augmentation, support agents, data plumbing, evaluation, data-quality audits, managed agents and codebase audit.
The r/ExperiencedDevs thread (405 upvotes): from the Reddit sweep in our AI-agency landscape note.

This post first appeared on Dravin AI.