Dravin AI · Blog · Calibration

Typed decisions on your own GPUs

info@dravin.ai (Dravin AI) — Fri, 02 Oct 2026 18:30:00 GMT

TL;DR

SGLang 0.5.21’s /v1/decisions reads each label’s next-token probability from a chat model and renormalises over the labels, generating no text.
SGLang says those probabilities are not calibrated, and that they can shift with batching and prefix caching.
Fit a temperature or Platt scaling on your labelled data, test replay variance, and compare against a fine-tuned classifier.

On 29 September 2026 OpenAI announced a Decisions API in limited preview [1]. SGLang 0.5.21 now ships /v1/decisions, which turns a chat model you already serve into a classifier with a probability for every option [3], [5]. This follow-up to our post on typed decision models shows how the endpoint reads those probabilities out of a decoder model, why they are not calibrated, how to calibrate and test them, and when to run decisions on your own GPUs.

What OpenAI and SGLang shipped

OpenAI’s DevDay recap says the Decisions API focuses its model on “a specific set of user-defined questions with finite pre-defined answers”, given text or images, to classify content, route requests or choose an agent’s next action [1]. Its developer account says the API runs on GPT-6 Luna [2]. The recap also notes “a broad release planned in the coming days” [1]. Unlike the other developer announcements in the recap, the Decisions API entry links to no documentation, and the recap says nothing about how it works, so nothing below describes OpenAI’s design [1].

SGLang, the open-source serving engine, released 0.5.21 on PyPI on 1 October and on GitHub on 2 October [4], [3]; the code had merged on 25 September, before DevDay [7]. The route needs no special checkpoint: any generation model served with a Jinja chat template can answer if per-request checks pass [5]. Two models are validated, Qwen3.8-27B and Qwen3.5-35B-A3B, each on one H200 in BF16 [5].

A request carries one input, as text or a JSON object or array, and a list of questions: a choice among 2 to 26 options, a score on 2 to 10 levels, or a yes or no. Release 0.5.21 has no image field [5]. A second route, /v1/systemone, accepts the request shape of TypeSafe’s System One API, which serves Jev, so TypeSafe’s official SDKs can point their base URL at your server, with documented exceptions [5].

How a chat model returns a probability per option

The server writes each question as one user message: the input, a blank line, the question, one line per option with a letter label such as “A: billing”, and an instruction to answer with one label only [5], [6]. Levels are labelled 0 to 9, and yes or no questions use the words yes and no. The message goes through the model’s chat template with thinking turned off, and the server checks that each label adds exactly one distinct token after the prompt, returning a 400 when one does not [5], [6]. Letters keep every label to one token, whatever an option is called.

For a ticket and three teams, the user message reads:

The integration keeps failing.

Question: Which team should handle this ticket?
A: billing
B: technical
C: sales
Answer with the letter of one option only.

With thinking off, Qwen’s chat template then renders a closed reasoning block, so the next token is the answer [5], [6].

The model runs one prefill pass per question, the phase where it reads the prompt, and generates nothing: usage reports zero completion tokens [5]. At the answer position it holds a probability for every token in its vocabulary. The server keeps only the label tokens’ log-probabilities, divides them by the request’s temperature T (default 1) and takes a softmax over them. Every log-probability carries the same vocabulary-wide normalising term, so it cancels [5], [6]. A choice answer is the most probable option, and a score is the probability-weighted mean level [5].

Renormalising hides how much probability sat elsewhere, so the server also reports label_mass, the raw probability of all the labels together; a low value means most of it sits outside the offered answers [5].

From a typed question to a probability

Render each question: thinking off, labels one token each
Prefill per question: no tokens generated
Softmax over labels: divided by T, plus label_mass
Calibrate, then threshold: on your labelled data

Steps 1 to 3 run inside SGLang 0.5.21. Step 4 is yours: the returned probabilities are not calibrated.

Questions are answered independently and batched together; each is its own prompt that repeats the input, so prompt tokens grow with the number of questions [5]. Laya, from our Jev post, answers every question in one forward pass.

Constrained decoding reaches the same numbers. In SGLang, a one-token regex such as (A|B|C) at temperature 0, with log-probabilities returned, gives the same probabilities as the /v1/score path this route uses, for one-character labels in the same cache state [5]. A JSON schema with enum fields gives no clean probability per option, because an enum value can span several tokens [5].

What the endpoint refuses, and what that rules out

Most of the rules protect the answer position, the one place a number is read. Qwen3.8-27B and Qwen3.5-35B-A3B think by default; the route turns thinking off, refuses requests that turn it back on, and refuses prompts that leave a reasoning block open, so the answer is never read inside the reasoning [5]. That means the model answers without working anything out first. Compute arithmetic, dates and multi-step logic in code and pass the result in, as TypeSafe advises for Jev.

The server also refuses built-in conversation templates, servers started with --enable-mis or --dllm-algorithm, and a model field that names a LoRA adapter, so decisions cannot run on one [5]. It does not check where a new model’s template puts the answer, so before relying on a model beyond the validated two, the docs say to inspect the prompt token ids the route can return [5].

A decision takes a single input, not a chat history; a conversation goes in as text or a JSON array [5]. A choice stops at 26 options. /v1/systemone accepts up to 255 with two-letter labels where the tokeniser and chat template allow it, as the Qwen models’ do; SGLang warns that these labels include common words with unequal priors, so answers above 26 options can depend on option order [5].

Why the probabilities are not calibrated

SGLang’s documentation says “None of these values is a calibrated probability that the decision is correct” [5]. Four effects pull a chat model’s label probabilities away from calibration.

Post-training comes first. In a 2022 study, Anthropic found its large base models well calibrated when options were shown with letters and scored by the letter, as this route does, in prompts with a few worked examples; with no examples, which is how this route asks, calibration was weaker. Its RLHF-tuned versions looked badly miscalibrated until one temperature, T = 2.5, largely fixed them on several evaluations, though the authors warn that heavier RL training might distort calibration beyond that repair [9]. Both validated Qwen models are chat models [5], so test the same repair on yours.

Letter priors add a second skew. Across 20 LLMs, a 2023 study traced a preference for particular option letters to token bias: whatever the content, the model puts more probability on some letter tokens than on others, so the letter an option lands on can move its probability [10].

Wording is the third. In a 2021 study of few-shot prompts, the format, the examples and even their order swung accuracy from near chance to near the state of the art. On one sentiment prompt, the default 0.5 threshold gave frequent false positives where a threshold of 0.68 would have been 94% accurate on its validation set [11]. SGLang owns the wording around your question and ships each change as a new prompt_format_version [5]; the question and option text are yours, so test rewordings on labelled data.

Finally, probability can sit outside the labels. The probabilities always sum to 1 over the offered labels, so a choice picks an option even when none fits. label_mass shows the leak, though on yes or no questions it reads low even on clear cases, because it counts only the lowercase yes and no tokens while the model also puts probability on “Yes” and “No” [5]. Anthropic’s models were also reluctant to use a “none of the above” option, while their true or false judgements stayed well calibrated [9]. Of the Jev post’s two fixes, that favours a separate yes or no question before the choice.

Calibrating on your own server

Fit calibration per question, on labelled examples from your own traffic kept apart from the test set.

Temperature scaling fits a single number T by minimising log loss on held-out data, and never changes which option wins [12]. Here it maps onto the request’s temperature, which divides the label logits before the softmax [5]. Fit T offline and pass it in that field: the chosen option stays put, the probabilities and so the score’s weighted mean move, and label_mass does not change [5], [12]. One temperature covers every question in a request [6], so send questions with different temperatures separately, or apply T on your side: raise each probability returned at the default T of 1 to the power 1/T, renormalise, and recompute any score. SDK users have to do the latter, since /v1/systemone refuses a temperature field [5].

A temperature cannot correct a lean towards one label. For yes or no, Platt scaling fits a slope and an intercept on the log-odds, and the intercept absorbs a model that says yes too often; for choices, vector scaling fits a weight and a bias per option [12]. Isotonic regression can correct any monotone distortion but overfits small sets: with fewer than about 200 to 1,000 labelled cases, Platt scaling did better for all nine learning methods in a 2005 study [13].

Two corrections go after label priors directly. Contextual calibration scores a content-free input such as “N/A” and rescales the outputs so that input comes out uniform [11]. PriDe estimates the letter prior by permuting options on a small sample, for example 5% of the test set, and removes it from the remaining predictions [10]. Both remove bias, and the 2021 paper separates that goal from statistical calibration [11], so measure calibration again afterwards.

Refit when you change the model or its quantisation, upgrade SGLang, or receive a new prompt_format_version.

Testing replay and drift

The general evaluation is in the Jev post; a self-hosted endpoint adds its own tests.

Start with replay variance. On the two validated models, which mix full and linear attention, SGLang saw probabilities move by up to 0.07, and label_mass by up to about 0.14, between cold and prefix-cached requests and across batch compositions; the chosen option did not change in its checks [5]. One general cause: batch size changes how GPU kernels split their sums, and floating-point addition is not associative, so identical inputs can score differently [8]. Send your test set several times, cold and warm, alone and mixed with chat traffic, and route a band around each threshold, at least as wide as the movement you measure, to a person. On Qwen3.5-35B-A3B, --disable-radix-cache gave identical values across sequential repeats, at the cost of prefix reuse [5].

Keep every decision replayable. With return_prompt_token_ids, an answer carries its exact prompt and label token ids, and /v1/score re-scores them to the same values when temperature and cache state match [5]. Store them with the decision, and send prompt_format_version so that an upgrade which changes the wording fails loudly instead of shifting answers [5].

Shuffle option order on a sample and compare, since letter priors move answers [10]. Set a label_mass floor per question type from labelled data, remembering that yes or no reads low. Leave any temperature in --preferred-sampling-params unset, since it can scale the probabilities and label_mass when a decision shares a batch with generation or runs under speculative decoding [5].

Hosted preview or your own GPUs

Self-hosting makes most sense when you already serve an open model: decisions run on the same server, with no extra flag, alongside chat traffic [5]. Inputs stay on your hardware, any answer can be replayed from stored token ids, and you choose the model, its version and the calibration. The work is all prefill, so the prefix caching and chunk size levers in our serving playbook apply, though prefix caching is also one source of the drift SGLang measured. TypeSafe’s SDKs time out after 10 seconds and retry, so raise that timeout for many questions over a long input [5].

Code written for Jev can switch to /v1/systemone by changing its base URL. Its thresholds should not come along unchecked: the confidence that route returns is uncalibrated, and SGLang warns that thresholds tuned on another provider may not transfer [5].

A hosted API suits low volume, or a team that will not run GPUs; once OpenAI’s preview opens up, test it on the same labelled set.

Neither replaces a trained classifier where one fits. With labelled data and a stable label set, a small fine-tuned model is the stronger default: a 2024 study found smaller fine-tuned models beat larger zero-shot prompted models on every text classification task it tested [14]. Our default is unchanged: a small classifier fine-tuned on your labels, calibrated and thresholded, with the uncertain tail sent to an LLM or a person. A decision endpoint fits where that default cannot yet: when labels change per request, when no labelled data exists, or to draft labels, checked by people, for the classifier you train next.

Checklist

Pin the SGLang version, model and quantisation, send prompt_format_version with every request, and refit when any of them changes.
On a new model, check the returned prompt token ids for the answer position.
Leave the temperature in --preferred-sampling-params unset on servers that take decisions.
Fit a temperature per question on held-out labelled traffic, apply it per request or on your side, and add Platt or vector scaling if one label is over-predicted.
Shuffle option order on a sample, and correct letter priors if answers move.
Measure replay variance cold, warm and under load, and send a band at least that wide around each threshold to a person.
Set a label_mass floor per question type, and gate choices with a yes or no question where no option may fit.
Store each decision’s prompt and label token ids for replay.
Compare against a classifier fine-tuned on the same labels before you commit.

Sources

This post first appeared on Dravin AI.

Typed decision models: what early tests show

info@dravin.ai (Dravin AI) — Tue, 22 Sep 2026 18:30:00 GMT

TL;DR

Jev and Laya answer typed questions (a choice, a score or a yes/no) with a probability attached, instead of writing text.
In one independent benchmark Jev was the fastest model measured; Laya is near chance until it is fine-tuned.
Before adopting one, test it on a held-out set from your own traffic, against a classifier fine-tuned on your own data.

On 15 September 2026 TypeSafe AI released Jev in early access. It calls Jev its first “System One Model”, a name taken from Daniel Kahneman’s fast, intuitive System 1 thinking [1]. Jev does not write text. You give it a piece of state (a ticket, an email, a JSON object) and a set of typed questions, and it returns a choice, a score or a yes/no drawn from options you defined, with a probability attached. Three days later Convai Innovations published Laya, an open-weight model built on the same idea [11] [13]. Since version 0.3.7 it also ships a server that accepts the same request shape as Jev [11].

Both launched in mid-September 2026, so there is little independent evidence yet. The idea still deserves attention, because it describes what many production AI systems do on every request: make a small decision, such as routing a ticket, moderating a message or checking an agent’s tool call, and know when to leave it alone.

The short version: Jev is fast, though the gap is large only against LLMs left in their slowest default modes [15]. On calibration, which is the point of the product, the independent tests are mixed, and one benchmark’s authors have since said its calibration figures for Jev are not directly comparable with the LLMs’ [14] [15]. Laya is near chance until it is fine-tuned [11]. The comparison that should decide adoption is against a classifier fine-tuned on your own data, and the only one we have found is Laya’s own, on a benchmark its checkpoint was fine-tuned on [11].

What a typed decision model is

TypeSafe describes Jev as “a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out” [1]. A request has two parts. The state is the input: a string, a JSON object or an array of text. Images, audio and video are not supported yet [2]. The questions use three primitives [2]:

Choice picks one option from a list you supply, such as which team should handle a ticket.
Score places the state on a rubric, for example customer frustration on a scale from 0 to 2.
Noul is TypeSafe’s name for a yes/no question: it asks whether a statement is true and returns a probability between 0 and 1.

The Register gave an example of what a three-department routing question might return [17]:

{"billing": 0.08, "technical": 0.85, "sales": 0.07}

It paired that answer with a confidence of 0.82. The probabilities are per option. The confidence is a separate summary that TypeSafe derives from the shape of the whole distribution, and it is the number you set thresholds on; a Noul returns only its probability [5].

Jev’s internals are unpublished. TypeSafe says it built “a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)” [1]. TechCrunch reports that the company has said little about that architecture, that outside observers suspect it sits on top of an open-weight LLM, and that the former OpenAI researcher who started the company says Jev is trained only on synthetic data [16]. So Jev can be judged only by what it returns, which is what the rest of this post does.

Laya is the clearest public account of how such a model can work, because its model card spells it out [11]. The English checkpoint is ModernBERT-large, a bidirectional encoder (it reads the whole input at once), fully fine-tuned, with a small decision head on top: 421M parameters in total. A multilingual checkpoint built on mmBERT has 322M. Each option in a question is scored at its own [MASK] token, the placeholder slot a BERT-style model is trained to fill, and the scores are normalised across that question’s options. The options arrive with the request, so a new schema needs no retraining, and every question in a call is answered in a single forward pass. Training rewards the model with strictly proper scoring rules, which pay the most only when the reported probabilities are honest.

Laya, then, is a BERT-style classifier whose label set arrives with each request, trained with an objective that rewards honest probabilities. That places the category between two tools most teams already know. A fine-tuned classifier is fast and accurate, but it needs labelled data and knows only its fixed labels. A zero-shot classifier takes its labels at request time, and one served as the baseline in an independent test below [14]. What typed decision models add is calibration as a training objective, several typed questions answered in one call, and an interface built for software to branch on [1] [11].

How this differs from asking an LLM to classify

Any LLM will produce a label from a prompt and a JSON schema. Three things are different, and the first is narrower than it looks.

The format is guaranteed by construction. “By providing typed, structured values, Jev can avoid the parsing and validating that must be done to process text responses from LLMs,” as The Register put it [17]. An LLM gets most of that guarantee from constrained decoding, which limits its output to a JSON schema or a forced tool call. The independent benchmark below asked every LLM for structured output, through a JSON schema, a forced tool call or plain JSON mode: seven of the eight returned malformed answers on at most 0.7% of decisions in any suite, one small model reached 6.2% on one suite, and Jev returned none [15]. The format advantage is real but small, so the differences that matter are the next two.

The work happens in one step. An LLM writes its answer token by token; a typed decision model scores every option of every question at once. TypeSafe credits this parallel design for the speed [1] [3]. The design pays most when one call asks several questions: Laya answers all of them in a single forward pass [11], and TypeSafe says adding questions “barely changes the response time” [9].

Two lanes answer the same ticket routing question. On the left an LLM writes a JSON answer one token at a time, route technical, and a parser then checks it. On the right a typed decision model reads the ticket once and fills a probability for every option: billing 0.08, technical 0.85, sales 0.07, then a confidence of 0.82. The right lane returns every probability in one step; the left lane writes its answer in order and then checks it. See the animated diagram in the post.

The same routing question, asked two ways. The LLM writes its answer one token at a time, then a parser checks it. The typed decision model scores every option in one pass and adds a confidence value. Values from The Register’s example [17].

The probability is the product. A typed decision model is trained so that its per-option probabilities match outcomes, which is the aim of both TypeSafe’s RLCD and Laya’s proper scoring rules [1] [11]. LLMs asked to state their confidence tend to be overconfident, which Xiong and colleagues found across several models [20]. The research is mixed: Tian and colleagues found verbalised confidence from RLHF-tuned models can be better calibrated than their token probabilities [21]. Where an API exposes token log-probabilities, an LLM’s probabilities over the labels can also be read directly, and Kadavath and colleagues found large models well calibrated on multiple-choice and true/false questions posed in the right format [22].

Type safety guarantees the shape of the answer and says nothing about whether the answer is right. TypeSafe says Jev “can’t hallucinate” [1]; The Register noted that structured output with probabilities “does not preclude the possibility of being incorrect” [17]. And because the caller fixes the options, a Choice always picks one of them, even when none fits. TypeSafe’s own documentation says “the Choice is relative, settling which option, while each Noul is absolute and can be low for all of them” [4]. The practical fix is to add a “none of these” option, or to put a separate Noul in front of the Choice as a gate.

Why calibrated probabilities matter in production

A model is calibrated when its probabilities match reality across many predictions. TypeSafe’s primer puts it plainly: “Outcomes assigned a probability of 0.2 should occur about 20% of the time. Outcomes assigned a probability of 0.8 should occur about 80% of the time” [7].

Each small decision a production system makes carries two questions: what is the answer, and is it safe to act on. The CTO of Earendil, quoted by TechCrunch, described the trade: “The user has to say, okay, if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it. But if it’s 95%, sure, then I can do something with it” [16].

TypeSafe’s documentation recommends three bands: act automatically when confidence is high; proceed with caution in the middle, for example by asking a person to confirm; and below that, route to a person, ask for clarification or fall back to another system. It adds that “a confidence threshold is not one number”, since actions with worse consequences deserve a stricter bar [5].

The operating number that follows is coverage at an error budget, an idea from selective classification, where a model is allowed to abstain [23]. Choose the error rate you can accept, say 5%. Set the threshold on labelled data so that errors among the items acted on stay within that rate, then measure what share of traffic the model handles alone. Everything else goes to a person or a larger model. One limit applies throughout: calibration is a property of groups of predictions, and in TypeSafe’s words it “does not guarantee that an individual answer is correct” [2].

Tickets flow into a tagging gate, each with a confidence score. At 0.75 or above the item is acted on, from 0.50 it goes to confirm, and below 0.50 it goes to a person or an LLM. Four of six tickets clear the act threshold and are handled automatically. A stricter payment gate acts only at 0.97 or above, so a decision at 0.80, acted on at tagging, is held for confirmation before any money moves. Set your own thresholds on labelled data, one per action. See the animated diagram in the post.

A confidence gate turns a confidence value into an action. The bar rises with the cost of a mistake, so a decision at 0.80 is acted on at tagging and held before any money moves. Set your own thresholds on labelled data, one per action.

What the first tests show

TypeSafe’s blog gives end-to-end response times of 70 to 500 ms. Its home page claims Jev is “193.6x faster”, a figure the blog says comes from its workflow evals, in which each LLM ran at its provider’s default reasoning setting, and TypeSafe expects these “are on the higher end of real world gains” [1] [10]. The same post says the evals were made by its own capability team, “so some bias could exist”, and that the reference answers are the average of GPT-6 Astra and Claude Fable 5.1 [1] [10]. Accuracy in those evals therefore means agreement with two frontier LLMs, where most buyers would assume human labels. TypeSafe argues that this reference, if anything, favours the OpenAI and Anthropic models in the comparison [1].

Two independent benchmarks with published protocols followed within days. decision-model-benchmark ran Jev against eight LLMs from five providers, each asked for its answer through a JSON schema, a forced tool call or plain JSON mode, with thinking off on the Anthropic models and at the lowest level on GLM. It declares no vendor sponsorship and publishes its raw logs [15]. Calibration error below is expected calibration error (ECE), the average gap between stated confidence and observed accuracy; lower is better and 0 is perfect.

Finding	Jev	The eight LLMs
Accuracy, 77-way banking intent	76.3%	70.9% to 81.3%
Accuracy, SMS spam	93.0%	66.1% to 94.9%
Median latency per request	264 to 276 ms	303 ms to 5.6 s
Most options accepted in one question	255	512 (the most tested)
Low confidence (0.5 or below) when no option is correct	49.7%	97.3% to 100% for seven; 64.7% for one
Answers changed across shuffled option orders and repeat runs	13%	15% to 37%
Calibration error, banking intent	0.083	0.043 to 0.226
Calibration error, SMS spam	0.249	0.042 to 0.197
Calibration error, suite built to force uncertainty (items with a hidden reference answer only)	0.246	0.039 to 0.122

The authors’ summary at the time was that “no class wins on quality”. Jev was the fastest model measured, about 1.2 times faster than the fastest LLM set-up (an open-weight model on specialised inference hardware) and 10 to 16 times faster than models in thinking mode, and they concluded that “the vendor’s speedup claim holds only against LLMs left in their slowest default mode” [15]. On calibration error, Jev sits inside the LLMs’ range on banking intent and above it on spam and on the forced-uncertainty suite, though the correction below limits what that comparison shows.

On 26 September 2026 the authors corrected the report [15]. Jev’s confidence score is provider-defined, while each LLM was prompted for the probability that its answer was right, so the three calibration rows are not like-for-like. The LLMs, unlike Jev, had been told to report low confidence when unsure, so the 49.7% row is not like-for-like either. And the shuffle row also counts answers that changed between identical repeats; across option orders, with repeats matched, the disagreement rate is 11.7% for Jev against 12.0% to 30.0% for the LLMs.

jev-benchmarks, a pre-registered pilot on 300 held-out examples against an open zero-shot classifier, found Jev more accurate on two of three tasks, where it could handle 83% and 86% of items alone at a 5% error budget [14]. On the third, emotion labelling, Jev put zero probability on the true label for 16% of examples and could automate none of them at that budget; the open classifier managed 2%. The authors fitted those thresholds on the same small slice, so they describe this sample only [14].

Laya documents its own limits unusually well [11] [12]. Its English base checkpoint scores 0.362 on a benchmark of four typed-decision workflows without fine-tuning: above the 0.318 random baseline, and below the 0.461 you would get by always guessing the most common answer. The headline 0.766 belongs to a checkpoint fine-tuned on that benchmark’s own training split. As shipped, the probabilities are overconfident. Fitting one temperature (a single number that sharpens or softens every probability) per question type and option count moves mean ECE from 0.466 to 0.081, per its model card [11].

Held-out results are weaker than in-training ones. On held-out toxic chat, moderation accuracy is 0.530, barely above chance on a balanced split, and the authors put it bluntly: “hand-picked examples work, real traffic does not” [12]. The English checkpoint scored 0.000 accuracy on Khmer, and its authors warn that it stays confident while wrong, which is why their router picks a checkpoint by script before the model runs [11] [12]. The confidence figures from that language sweep predate a later change that clamped the model’s temperatures; the maker has since re-run the sweep, and the English checkpoint’s accuracy reproduced exactly while its confidence figures moved [12]. A confidence gate protects you only on inputs like the ones the model was trained and calibrated on, so check the language and shape of inputs before the gate.

Laya’s card also has a table that puts it ahead of Jev on most rows, while stating that the Jev figures are “third-party published, never measured here” [11]. Its Jev calibration figure is the 0.246 from the forced-uncertainty suite above, and its own figure is measured after the temperature refit. Each side’s table suits its maker, and neither is a like-for-like comparison.

Both products are young. At Jev’s launch, access was through a waitlist [1], and TechCrunch reports that TypeSafe “briefly lost the ability to serve users from its API because demand was so high” [16]. Jev’s rate limits “can change without notice”, and the same weights serve every account, so it cannot be fine-tuned on your data. TypeSafe says “Jev is not trained on customer requests or responses”, and that English is where its accuracy is best [3].

Laya’s first release reached PyPI on 18 September 2026 [13]. Its weights and code are open under Apache 2.0, so you can inspect and run it yourself. It needs fine-tuning and a temperature fit on your task first. Its escalation head, action.act_probability, “carries no usable signal yet”, and the card advises gating on confidence instead. Its bundled server listens on every network interface (0.0.0.0) with no authentication unless LAYA_API_KEY is set [11], so set the key before the port is reachable.

The evidence so far is vendor evals, one 300-example independent pilot and one independent benchmark of five small suites.

Choosing between an LLM, a typed model and a fine-tuned classifier

TypeSafe’s list of failure modes for jev-1.13 covers literal reading, arithmetic, date comparisons, questions whose answer depends on first answering another question, and counting: “jev-1.13 does not count reliably” [4]. Its advice is to keep arithmetic, dates and counting in code [4], and to have each question ask one well-scoped thing, composing the answers in code [8].

Option	When it fits
An LLM	The output is language, code or an explanation someone will read, or the decision needs several steps of reasoning. Jev “does not generate text, write code, or hold a conversation” [6]. An LLM can also draft labels for training a classifier, with people checking a sample, and give a second opinion on a low-confidence case before it reaches a person.
A typed decision model	The options are fixed per request, one call asks several questions, and you have no labelled data yet. Used zero-shot, Jev beat an open zero-shot classifier on two of three tasks in one pilot [14]. Laya fits only after fine-tuning, since its base checkpoints are near chance [11], and its authors advise keeping Choice questions under about 20 options [12].
A fine-tuned classifier	You have labelled data and the label set is stable. A 2024 study by Bucher and Martini found small fine-tuned models consistently outperformed zero-shot prompted LLMs on text classification [24].

How we evaluate a decision model before trusting it

Build a held-out set from your own traffic, labelled by people and never used for prompt tuning or threshold fitting. Laya’s results show why: spam, which was in its training mix, scores 0.993, against the held-out moderation result above [12].
Measure discrimination and probability quality separately. Accuracy and macro-F1 (which counts rare labels as much as common ones) for the first. For the second, Brier score and log loss, which penalise confident wrong answers, and calibration error with a reliability diagram like the one below.
Fit a temperature on a separate calibration split and report calibration before and after. A hosted model’s weights cannot be tuned, but its output probabilities can still be recalibrated on your split. Temperature scaling has a single parameter and is often enough [18]. Laya found overconfidence on some tasks and underconfidence on others [12], so check each question type.
Report coverage at your error budget for each action separately, since each action has its own threshold [5].
Probe the edges. Shuffle option order, include items with no correct option, ask a question and its negation, and plant instructions inside the state. TypeSafe’s own docs show a question and its negation on the same ticket returning 0.72 and 0.47, which sum to 1.19 [4]. They also note that injected content “can move the answer” [4], which matters most when the model is a guardrail.
Measure latency on the real path, at the median and the 95th percentile. Laya’s 33 to 40 ms is one question on a T4 GPU, depending on the checkpoint; on CPU its card gives 193 to 464 ms [11]. Jev’s figures are hosted round trips.
Plan for drift. Pin the model version, since TypeSafe warns that an alias moves when a new release ships [3]. Log every probability, and recheck calibration on fresh labelled samples on a schedule. Calibration fitted on past data degrades when the inputs shift [19].

A reliability diagram plots predicted probability against observed frequency, with the diagonal as perfect calibration. The bins start below the diagonal, meaning the model is overconfident. A single temperature dial turns and the bins settle close to the diagonal: for Laya's English checkpoint the model card reports mean expected calibration error falling from 0.466 to 0.081. The bars sketch the pattern; only the error figures are measured. In general, when the inputs shift, a fit made on past data drifts off the diagonal again, so calibration has to be rechecked on fresh labels. See the animated diagram in the post.

One temperature, fitted on a calibration split, pulls the bins close to the diagonal. A shift in the inputs pulls them off again, so the fit needs rechecking on fresh labels. The bars sketch the pattern; the error figures are from Laya’s model card [11].

Where this fits in how we build

We send classification to small trained classifiers and keep LLMs for work that needs language. Typed decision models share that view: a decision your software branches on should come from a model trained to return a calibrated label. Our view is that the most dependable path is still a small model fine-tuned on your own labelled data, for the reasons in the table above, and the gap between Laya’s zero-shot and fine-tuned results points the same way. We have not found a comparison of Jev with a classifier fine-tuned on a buyer’s own data, so that is the test worth running. Since Jev’s weights cannot be tuned per account [3], the tuning moves into how you word the questions and criteria, and into the thresholds in your code.

In practice we fine-tune encoder classifiers on a client’s data, calibrate and threshold their outputs, gate each action by confidence, and send the uncertain tail to an LLM or a person. Jev and Laya are worth testing against that baseline, on a held-out set, before either replaces it. If you have a classification step that is slow, expensive or hard to trust, tell us about it.

Sources

TypeSafe AI, “Introducing System One Models & Jev”, 15 September 2026. typesafe.ai
TypeSafe docs, “System One”. docs.typesafe.ai/concepts/system-one
TypeSafe docs, “Models” (jev-1.13.0, rate limits, fine-tuning, data handling, version pinning). docs.typesafe.ai/models
TypeSafe docs, “Jev 1.13 jaggedness”, last reviewed 17 September 2026, as archived on 22 September 2026. web.archive.org
TypeSafe docs, “Confidence”. docs.typesafe.ai/confidence
TypeSafe docs, “Jev with coding agents”. docs.typesafe.ai/introduction/coding-agents
TypeSafe docs, “AI primer”. docs.typesafe.ai/introduction/machine-learning-primer
TypeSafe docs, “Introduction”. docs.typesafe.ai/introduction
TypeSafe docs, “Primitives”. docs.typesafe.ai/primitives
TypeSafe, workflow evals. evals.typesafe.ai
Convai Innovations, Laya model card (v0.3.7), revision of 23 September 2026. huggingface.co/convaiinnovations/laya
Laya, BENCHMARKS.md, as of 4 October 2026. github.com/NandhaKishorM/laya
Laya on PyPI, first release 18 September 2026. pypi.org/project/laya
jev-benchmarks, pre-registered pilot, September 2026. github.com/AbdelStark/jev-benchmarks
decision-model-benchmark: README and v2 report of record, 18 September 2026 and v3 correction notice and report, 26 September 2026
Tim Fernholz, “A new kind of AI model from a ChatGPT inventor is thrilling developers”, TechCrunch, 18 September 2026. techcrunch.com
Thomas Claburn, “TypeSafe AI debuts model for machines that plays Doom”, The Register, 16 September 2026. theregister.com
Guo, Pleiss, Sun and Weinberger, “On Calibration of Modern Neural Networks”, ICML 2017. arXiv:1706.04599
Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift”, NeurIPS 2019. arXiv:1906.02530
Xiong et al., “Can LLMs Express Their Uncertainty?”, ICLR 2024. arXiv:2306.13063
Tian et al., “Just Ask for Calibration”, 2023. arXiv:2305.14975
Kadavath et al., “Language Models (Mostly) Know What They Know”, 2022. arXiv:2207.05221
Geifman and El-Yaniv, “Selective Classification for Deep Neural Networks”, NeurIPS 2017. arXiv:1705.08500
Bucher and Martini, “Fine-Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification”, 2024. arXiv:2406.08660

This post first appeared on Dravin AI.