Typed decisions on your own GPUs

On 29 September 2026 OpenAI announced a Decisions API in limited preview [1]. SGLang 0.5.21 now ships /v1/decisions, which turns a chat model you already serve into a classifier with a probability for every option [3], [5]. This follow-up to our post on typed decision models shows how the endpoint reads those probabilities out of a decoder model, why they are not calibrated, how to calibrate and test them, and when to run decisions on your own GPUs.

What OpenAI and SGLang shipped

OpenAI’s DevDay recap says the Decisions API focuses its model on “a specific set of user-defined questions with finite pre-defined answers”, given text or images, to classify content, route requests or choose an agent’s next action [1]. Its developer account says the API runs on GPT-6 Luna [2]. The recap also notes “a broad release planned in the coming days” [1]. Unlike the other developer announcements in the recap, the Decisions API entry links to no documentation, and the recap says nothing about how it works, so nothing below describes OpenAI’s design [1].

SGLang, the open-source serving engine, released 0.5.21 on PyPI on 1 October and on GitHub on 2 October [4], [3]; the code had merged on 25 September, before DevDay [7]. The route needs no special checkpoint: any generation model served with a Jinja chat template can answer if per-request checks pass [5]. Two models are validated, Qwen3.8-27B and Qwen3.5-35B-A3B, each on one H200 in BF16 [5].

A request carries one input, as text or a JSON object or array, and a list of questions: a choice among 2 to 26 options, a score on 2 to 10 levels, or a yes or no. Release 0.5.21 has no image field [5]. A second route, /v1/systemone, accepts the request shape of TypeSafe’s System One API, which serves Jev, so TypeSafe’s official SDKs can point their base URL at your server, with documented exceptions [5].

How a chat model returns a probability per option

The server writes each question as one user message: the input, a blank line, the question, one line per option with a letter label such as “A: billing”, and an instruction to answer with one label only [5], [6]. Levels are labelled 0 to 9, and yes or no questions use the words yes and no. The message goes through the model’s chat template with thinking turned off, and the server checks that each label adds exactly one distinct token after the prompt, returning a 400 when one does not [5], [6]. Letters keep every label to one token, whatever an option is called.

For a ticket and three teams, the user message reads:

The integration keeps failing.

Question: Which team should handle this ticket?
A: billing
B: technical
C: sales
Answer with the letter of one option only.

With thinking off, Qwen’s chat template then renders a closed reasoning block, so the next token is the answer [5], [6].

The model runs one prefill pass per question, the phase where it reads the prompt, and generates nothing: usage reports zero completion tokens [5]. At the answer position it holds a probability for every token in its vocabulary. The server keeps only the label tokens’ log-probabilities, divides them by the request’s temperature T (default 1) and takes a softmax over them. Every log-probability carries the same vocabulary-wide normalising term, so it cancels [5], [6]. A choice answer is the most probable option, and a score is the probability-weighted mean level [5].

Renormalising hides how much probability sat elsewhere, so the server also reports label_mass, the raw probability of all the labels together; a low value means most of it sits outside the offered answers [5].

From a typed question to a probability

1. 01Render each question: thinking off, labels one token each
2. 02Prefill per question: no tokens generated
3. 03Softmax over labels: divided by T, plus label_mass
4. 04Calibrate, then threshold: on your labelled data

Questions are answered independently and batched together; each is its own prompt that repeats the input, so prompt tokens grow with the number of questions [5]. Laya, from our Jev post, answers every question in one forward pass.

Constrained decoding reaches the same numbers. In SGLang, a one-token regex such as (A|B|C) at temperature 0, with log-probabilities returned, gives the same probabilities as the /v1/score path this route uses, for one-character labels in the same cache state [5]. A JSON schema with enum fields gives no clean probability per option, because an enum value can span several tokens [5].

Enjoying this? New posts by email

Leave this field empty

Email

SubscribeSubscribing

What the endpoint refuses, and what that rules out

Most of the rules protect the answer position, the one place a number is read. Qwen3.8-27B and Qwen3.5-35B-A3B think by default; the route turns thinking off, refuses requests that turn it back on, and refuses prompts that leave a reasoning block open, so the answer is never read inside the reasoning [5]. That means the model answers without working anything out first. Compute arithmetic, dates and multi-step logic in code and pass the result in, as TypeSafe advises for Jev.

The server also refuses built-in conversation templates, servers started with --enable-mis or --dllm-algorithm, and a model field that names a LoRA adapter, so decisions cannot run on one [5]. It does not check where a new model’s template puts the answer, so before relying on a model beyond the validated two, the docs say to inspect the prompt token ids the route can return [5].

A decision takes a single input, not a chat history; a conversation goes in as text or a JSON array [5]. A choice stops at 26 options. /v1/systemone accepts up to 255 with two-letter labels where the tokeniser and chat template allow it, as the Qwen models’ do; SGLang warns that these labels include common words with unequal priors, so answers above 26 options can depend on option order [5].

Why the probabilities are not calibrated

SGLang’s documentation says “None of these values is a calibrated probability that the decision is correct” [5]. Four effects pull a chat model’s label probabilities away from calibration.

Post-training comes first. In a 2022 study, Anthropic found its large base models well calibrated when options were shown with letters and scored by the letter, as this route does, in prompts with a few worked examples; with no examples, which is how this route asks, calibration was weaker. Its RLHF-tuned versions looked badly miscalibrated until one temperature, T = 2.5, largely fixed them on several evaluations, though the authors warn that heavier RL training might distort calibration beyond that repair [9]. Both validated Qwen models are chat models [5], so test the same repair on yours.

Letter priors add a second skew. Across 20 LLMs, a 2023 study traced a preference for particular option letters to token bias: whatever the content, the model puts more probability on some letter tokens than on others, so the letter an option lands on can move its probability [10].

Wording is the third. In a 2021 study of few-shot prompts, the format, the examples and even their order swung accuracy from near chance to near the state of the art. On one sentiment prompt, the default 0.5 threshold gave frequent false positives where a threshold of 0.68 would have been 94% accurate on its validation set [11]. SGLang owns the wording around your question and ships each change as a new prompt_format_version [5]; the question and option text are yours, so test rewordings on labelled data.

Finally, probability can sit outside the labels. The probabilities always sum to 1 over the offered labels, so a choice picks an option even when none fits. label_mass shows the leak, though on yes or no questions it reads low even on clear cases, because it counts only the lowercase yes and no tokens while the model also puts probability on “Yes” and “No” [5]. Anthropic’s models were also reluctant to use a “none of the above” option, while their true or false judgements stayed well calibrated [9]. Of the Jev post’s two fixes, that favours a separate yes or no question before the choice.

Calibrating on your own server

Fit calibration per question, on labelled examples from your own traffic kept apart from the test set.

Temperature scaling fits a single number T by minimising log loss on held-out data, and never changes which option wins [12]. Here it maps onto the request’s temperature, which divides the label logits before the softmax [5]. Fit T offline and pass it in that field: the chosen option stays put, the probabilities and so the score’s weighted mean move, and label_mass does not change [5], [12]. One temperature covers every question in a request [6], so send questions with different temperatures separately, or apply T on your side: raise each probability returned at the default T of 1 to the power 1/T, renormalise, and recompute any score. SDK users have to do the latter, since /v1/systemone refuses a temperature field [5].

A temperature cannot correct a lean towards one label. For yes or no, Platt scaling fits a slope and an intercept on the log-odds, and the intercept absorbs a model that says yes too often; for choices, vector scaling fits a weight and a bias per option [12]. Isotonic regression can correct any monotone distortion but overfits small sets: with fewer than about 200 to 1,000 labelled cases, Platt scaling did better for all nine learning methods in a 2005 study [13].

Two corrections go after label priors directly. Contextual calibration scores a content-free input such as “N/A” and rescales the outputs so that input comes out uniform [11]. PriDe estimates the letter prior by permuting options on a small sample, for example 5% of the test set, and removes it from the remaining predictions [10]. Both remove bias, and the 2021 paper separates that goal from statistical calibration [11], so measure calibration again afterwards.

Refit when you change the model or its quantisation, upgrade SGLang, or receive a new prompt_format_version.

Testing replay and drift

The general evaluation is in the Jev post; a self-hosted endpoint adds its own tests.

Start with replay variance. On the two validated models, which mix full and linear attention, SGLang saw probabilities move by up to 0.07, and label_mass by up to about 0.14, between cold and prefix-cached requests and across batch compositions; the chosen option did not change in its checks [5]. One general cause: batch size changes how GPU kernels split their sums, and floating-point addition is not associative, so identical inputs can score differently [8]. Send your test set several times, cold and warm, alone and mixed with chat traffic, and route a band around each threshold, at least as wide as the movement you measure, to a person. On Qwen3.5-35B-A3B, --disable-radix-cache gave identical values across sequential repeats, at the cost of prefix reuse [5].

Keep every decision replayable. With return_prompt_token_ids, an answer carries its exact prompt and label token ids, and /v1/score re-scores them to the same values when temperature and cache state match [5]. Store them with the decision, and send prompt_format_version so that an upgrade which changes the wording fails loudly instead of shifting answers [5].

Shuffle option order on a sample and compare, since letter priors move answers [10]. Set a label_mass floor per question type from labelled data, remembering that yes or no reads low. Leave any temperature in --preferred-sampling-params unset, since it can scale the probabilities and label_mass when a decision shares a batch with generation or runs under speculative decoding [5].

Hosted preview or your own GPUs

Self-hosting makes most sense when you already serve an open model: decisions run on the same server, with no extra flag, alongside chat traffic [5]. Inputs stay on your hardware, any answer can be replayed from stored token ids, and you choose the model, its version and the calibration. The work is all prefill, so the prefix caching and chunk size levers in our serving playbook apply, though prefix caching is also one source of the drift SGLang measured. TypeSafe’s SDKs time out after 10 seconds and retry, so raise that timeout for many questions over a long input [5].

Code written for Jev can switch to /v1/systemone by changing its base URL. Its thresholds should not come along unchecked: the confidence that route returns is uncalibrated, and SGLang warns that thresholds tuned on another provider may not transfer [5].

A hosted API suits low volume, or a team that will not run GPUs; once OpenAI’s preview opens up, test it on the same labelled set.

Neither replaces a trained classifier where one fits. With labelled data and a stable label set, a small fine-tuned model is the stronger default: a 2024 study found smaller fine-tuned models beat larger zero-shot prompted models on every text classification task it tested [14]. Our default is unchanged: a small classifier fine-tuned on your labels, calibrated and thresholded, with the uncertain tail sent to an LLM or a person. A decision endpoint fits where that default cannot yet: when labels change per request, when no labelled data exists, or to draft labels, checked by people, for the classifier you train next.

Checklist

1. Pin the SGLang version, model and quantisation, send prompt_format_version with every request, and refit when any of them changes.
2. On a new model, check the returned prompt token ids for the answer position.
3. Leave the temperature in --preferred-sampling-params unset on servers that take decisions.
4. Fit a temperature per question on held-out labelled traffic, apply it per request or on your side, and add Platt or vector scaling if one label is over-predicted.
5. Shuffle option order on a sample, and correct letter priors if answers move.
6. Measure replay variance cold, warm and under load, and send a band at least that wide around each threshold to a person.
7. Set a label_mass floor per question type, and gate choices with a yes or no question where no option may fit.
8. Store each decision’s prompt and label token ids for replay.
9. Compare against a classifier fine-tuned on the same labels before you commit.

Sources

1. 1OpenAI, “DevDay 2026 Recap” (29 Sep 2026)
2. 2OpenAI Developers on X, Decisions API announcement (29 Sep 2026)
3. 3SGLang, “v0.5.21” release notes (2 Oct 2026)
4. 4PyPI, “sglang 0.5.21” (1 Oct 2026)
5. 5SGLang docs, “Decision models”, at the v0.5.21 tag (2 Oct 2026)
6. 6SGLang source, “serving_decisions.py”, at the v0.5.21 tag (2 Oct 2026)
7. 7SGLang, pull request 41208, “Add a System One compatible /v1/systemone route” (merged 25 Sep 2026)
8. 8SGLang docs, “Deterministic Inference”, at the v0.5.21 tag (2 Oct 2026)
9. 9Anthropic, “Language Models (Mostly) Know What They Know” (Jul 2022)
10. 10Tsinghua University and WeChat AI, “Large Language Models Are Not Robust Multiple Choice Selectors” (ICLR 2024)
11. 11UC Berkeley, University of Maryland and UC Irvine, “Calibrate Before Use: Improving Few-Shot Performance of Language Models” (ICML 2021)
12. 12Cornell University, “On Calibration of Modern Neural Networks” (ICML 2017)
13. 13Cornell University, “Predicting Good Probabilities With Supervised Learning” (ICML 2005)
14. 14Stanford University and University of Zurich, “Fine-Tuned ‘Small’ LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification” (Jun 2024)