Agentic systems make decisions all the time: which tool to call, which category a request falls into, where to route a message. Each of these is a choice from a fixed list that is known in advance. LLMs, however, are text generators. They produce output one token at a time and every token needs a full pass through the model before the next one can start. Even a structured output such as {"tool": "search_docs"} takes around ten tokens, so a simple choice ends up paying for ten sequential passes.
A system-one decision model avoids that cost. The name borrows from Kahneman’s two systems of thinking: system one answers at a glance, while system two reasons step by step in words. A Jev-style decision model works the first way. It receives the prompt and the list of allowed answers, reads the prompt once and returns a score for each answer. No generation, only decision. The single pass that reads the prompt is called the prefill and it is the only computation that runs.
This article shows how to turn an ordinary LLM into such a decision model on TensorRT Edge-LLM, without any training, using a small trick built from logit bias and logprobs. It then measures the trick against structured output and closes with the runtime’s built-in option that does the same job more cleanly. The trick only makes sense once you see how an LLM turns its scores into probabilities, so that is where the article starts.
An LLM takes in the input prompt and works out its meaning. To predict the next token, it assigns a score to every token in its vocabulary. These raw scores are called logits, z, and they can be positive or negative. The larger a token’s score, the more weight it has to be selected as the next token.
However, an LLM does not simply pick the highest score. It samples the next token from a probability distribution over its vocabulary. So the raw scores have to be converted into probabilities: the likelihood of each vocabulary token being the next token in the generation. To do that, each score is exponentiated, which makes it positive and then divided by the sum of all the exponentiated scores, so that the results add up to one, just like a probability distribution. This operation is called the softmax. For a vocabulary of V tokens, the probability of token i is:
This works, but vocabularies are huge, around 150,000 tokens and the less likely tokens end up with probabilities on the order of 10^-44. Numbers that small are too small for computers to handle reliably. So these probabilities are converted to logprobs by taking their logarithm, which turns 10^-44 into about -101, an ordinary number. Logprobs have a second advantage. The joint probability of several tokens is a product of their probabilities and in log space that product becomes a sum, which is an easier and more stable operation.
Taking the log of the softmax undoes the exponent:
Here is an example with a toy vocabulary of just four tokens, “The”, “A”, “B” and “C”, whose scores are 3, 2, 1 and 0:
The sum of the exponentiated scores is 31.19 and its log is 3.440. Every logprob in the last column is simply its score minus 3.440. The rest of the article builds on this: a logprob is the score shifted down by one number that is the same for every token, so the gap between any two tokens is identical in scores and in logprobs.
A decision model does not care about the whole vocabulary, only about the allowed answers, called labels. Say a support ticket must be routed to one of three teams and the prompt asks the model to reply with A, B or C. In the toy vocabulary, A, B and C are the labels and “The” stands in for everything else the model could say.
The goal is the model’s preference among those three labels alone. So the softmax is taken over the labels only: exponentiate each label’s score and divide by the sum over the labels, instead of the sum over the whole vocabulary. Writing L for the set of labels:
With the toy scores, the label values are e², e¹ and e⁰, which are 7.39, 2.72 and 1.00. They sum to 11.11, so q(A) = 0.665, q(B) = 0.245 and q(C) = 0.090. The same numbers come out of the probability column in the earlier table: 0.237 / (0.237 + 0.087 + 0.032) = 0.665. That second route shows what q means. It is the model’s probability for each label, given that the answer must be one of the labels, which is what statisticians call a conditional probability.
Keep in mind what q does not say. It is not the chance that a label is correct and it ignores how much probability the model gave to the labels in the first place. In the toy, the three labels together got only 0.356, because the model really wanted to write “The” (0.644). The value q(A) = 0.665 hides that completely. This turns out to be the most common way the method goes wrong and the section on failures comes back to it.
Computing q needs the scores or the logprobs, of the labels. An inference server does not return all 150,000 of them, because that would be wasteful at every step. It returns logprobs only for the N most likely tokens and there is no way to ask for one specific token. Anything outside that top-N window is invisible.
In the toy, a window of N = 3 returns The, A and B. C is missing, so q cannot be computed. A real model is worse. The labels might rank 21st, 11th and 23rd, with ordinary words such as “The”, “I” or “Sure” ranked above them and a window of 3 shows none of them. A bigger window helps only if you know how deep the labels sit and the runtime caps it at 50.
So the problem is precise: bring the labels into the top N without changing how the model compares them with each other.
The fix is to add the same large number, call it c, to the score of every label and leave every other token alone. Most servers expose this as a logit bias. With c = 100, the labels jump above every other token, so a window of N equal to the number of labels returns exactly the labels.
This does not distort the decision, because the softmax only cares about gaps between scores. A's lead over B is 2 - 1 = 1 before the bias and 102 - 101 = 1 after it. In the formula, adding c multiplies every label's e^z by the same factor e^c and that factor cancels between the top and the bottom of the fraction:
Think of a race where every runner gets the same 100-metre head start over the crowd. They all end up ahead of the crowd and the order among the runners does not change.
The head start must be big enough for the weakest label to beat the strongest token that is not a label:
In the toy, the best non-label scores 3 and the weakest label scores 0, so any c above 3 works. In a real model, labels deep in the ranking can sit 20 or more points below the top token. The runtime allows a bias of at most 100, so the trick works as long as that distance stays under 100.
Here is the toy with c = 10, which is easier to read than 100 but behaves the same way:
The new sum of exponentiated scores is e¹² + e¹¹ + e¹⁰ + e³ = 244,675 and its log is 12.408. The top three tokens are now A, B and C and the last column already reads 0.665, 0.245 and 0.090, exactly the label-only softmax from before.
The server returns each label’s biased score minus 12.408. To get q, the code exponentiates those logprobs and divides by their sum. Both the bias and the 12.408 are shared by every label, so they cancel and q comes out exactly.
Why are the returned values not already summing to 1? Because a sliver of probability, epsilon, still sits on the non-labels:
With c = 10 it is 20.09 / 244,675 = 0.00008, so the labels sum to 0.99992. With c = 100 it drops to about 7 × 10^-44, which a float32 cannot even register, so the label sum is exactly 1.0 and the returned logprobs already equal q.
Renormalizing still earns its place for three reasons. Servers print logprobs with a limited number of digits, so the sum may read 0.999998. If the bias is ever clipped or weaker than expected, renormalizing still gives the right q, as long as every label made it into the window. And subtracting the largest logprob before exponentiating protects against underflow when all the label logprobs are very negative. Overflow cannot happen at this step, because logprobs are never above 0.
TensorRT Edge-LLM has no built-in scoring endpoint, but it has the two things the trick needs: a per-token logit bias and top-K logprobs. The label token ids come from the Hugging Face tokenizer of the exported model. Versions matter here, because the runtime changed between releases, so the article notes wherever a behaviour depends on the release. Each request parameter has one job:
top_k and top_logprobs look redundant but do different jobs. top_k decides which token is sampled and emitted, while top_logprobs decides how many candidates are reported and the scores come from the second.
The trick works because of the order in which the runtime does things. When the model finishes reading the prompt, its scores land in a buffer. The runtime adds the bias to those scores, picks the next token and only then computes the logprobs from the same, now biased, scores. That ordering is what puts the labels into the window.
A few more details of how the runtime handles this are worth knowing. Before exponentiating, the log-softmax subtracts the largest score, so a biased score near 100 never turns into e¹⁰⁰, a number far beyond what a 32-bit float can hold. Logprobs are always computed at temperature 1, whatever sampling temperature you set. The scores are kept in 32-bit floats, which matters for the precision discussion later. And the limits are firm: the bias must stay between -100 and 100 and at most 50 logprobs come back per step.
A quick sanity check is still worth running on your own setup: compare the renormalized output with a softmax computed directly from the Hugging Face model’s scores on the same prompts.
Go back to the labels ranked 21st, 11th and 23rd. The bias lifts all three by the same amount, so their order and their ratios stay exactly as the model had them. B was preferred to A before the push, by the same factor and the trick invents no preference of its own. What it throws away is how much probability the model put on the labels at all. After the push, a label that was ranked 1st and one that was ranked 500th look identical.
Here is what that can hide. Suppose “The” scores 8 and the labels sit far below it: A = -2, B = -0.5, C = -3. The label-only softmax gives q = 0.171, 0.766 and 0.063, which reads as a confident “B at 77%”. But the total probability on the three labels, (e^-2 + e^-0.5 + e^-3) divided by the full sum, is about 0.00027. The model was 99.97% sure the next token was not a label at all and q is just its ranking of three tokens it did not want.
So nothing guarantees that B is correct. Labels ranked 11th to 23rd mean the model was trying to say something else, which usually points to a prompt or chat-template problem. The ordering that deep in the ranking does carry some signal, but training rarely tunes it directly, so it is noisier than the ordering at the top. The bias trick also makes the problem harder to see: it replaces the full-vocabulary sum with a new one, so the label mass can no longer be recovered from the biased call.
The first line of defence is the prompt. Instead of only asking the question, shape the prompt so that the most natural next token is a label:
Route the support ticket to the right team. Decide which team should handle it.Choices:A: Billing and paymentsB: Technical supportC: Account accessD: None of the aboveTicket:I was charged twice for the same subscription.Answer with only the letter of the right choice.Label:
Every part of this prompt pushes in the same direction. Each choice is tied to a single letter, the instruction asks for only that letter and the prompt stops right at Label:. At that point, the continuation most consistent with everything above is one letter, so A, B, C and D climb to the top of the ranking on their own, ideally into the top few. Compare a prompt that ends with the ticket text: there the model's natural next move is to keep writing the ticket or start a sentence such as "The ticket should..." and the labels sink deep. A well-shaped prompt leaves the bias only a small job and the label mass stays high. The D option helps too, since it gives the model a legitimate answer when a ticket fits none of the teams, instead of pushing that probability onto words outside the labels.
One tokenization detail comes with ending at Label:. If the prompt is sent as raw text, the next token in most tokenizers is the letter with a leading space, " A" rather than "A", so the label ids must be taken from that form. If it goes through a chat template, the answer starts a fresh assistant turn and the letter has no leading space.
The prompt lowers the risk but does not remove it. Small models, ambiguous tickets or a template mismatch can still push the labels down, so a check is still needed. Run the same prompts without the bias on a labelled evaluation set and measure two things. The hit rate is how often the top token is one of the labels. The label mass is the total probability on the labels, read from the top-K logprobs, which gives a lower bound since labels outside the window are missed. If the hit rate is not close to 100%, fix the prompt before trusting any q. In production, a low label mass is a solid reason to send the decision to a fallback.
On paper, adding 100 changes nothing. Inside a computer it can, because of how numbers are stored. Floating-point formats keep a fixed number of significant digits, not a fixed number of decimal places. Picture a ruler whose tick marks spread further apart the further along you read: small numbers are measured finely, large numbers coarsely and every value is rounded to the nearest tick.
The number of significant binary digits a format keeps is its mantissa bits, m. Between 2^e and 2^(e+1), the ticks are 2^e × 2^-m apart and the spacing doubles every time the value crosses a power of two. A raw logit around 20 sits between 16 and 32. After +100 it sits around 120, between 64 and 128, where the ticks are four times farther apart (eight times once the raw logit is above 28).
That rounding shifts each biased label score slightly, so the gap between two labels can be off by up to one full tick. For two labels that are truly tied at 50/50, an error of one tick moves the split by roughly a quarter of the tick size. Here is how that plays out across formats:
A concrete bf16 case makes the table real. Take two labels with true scores 14.20 and 14.13, a gap of 0.07. After the bias, fp16 stores them as 114.1875 and 114.125, a gap of 0.0625, which is close enough. bf16 rounds both to 114.0. The gap vanishes and a true 51.7 / 48.3 split comes back as 50 / 50.
The fp8 and nvfp4 rows show how fast this gets worse as mantissa bits disappear. An fp8 E4M3 value near 120 can only land on multiples of 8, so a true gap of 1 becomes either 0 or 8. An nvfp4 element (E2M1, largest value 6) can hold only 0, 0.5, 1, 1.5, 2, 3, 4 and 6, times a scale shared by a block of 16 values. If the scale is set so that 100 maps to 6, the nearest value below it is about 67, so neighbouring values are 33 apart.
In practice, nobody stores logits in fp8 or nvfp4. Those formats are for weights, activations and the KV cache. So the question for any runtime is which format holds the scores at the moment the bias is added. TensorRT Edge-LLM keeps them in fp32 and adds the bias there. Near 120 the fp32 ticks are 0.0000076 apart, so on this runtime +100 costs no measurable accuracy. The raw scores themselves are only as precise as the layer that computes them, which runs in fp16 (see the next subsection), but that error exists with or without the bias.
The table still matters if you move the trick to a runtime that keeps logits in bf16 or fp16. There, use the smallest bias that still satisfies the head-start condition, for example the largest gap measured offline plus a margin or convert the logits to fp32 before adding the bias.
Quantizing the model’s weights with fp8, nvfp4 or int4_awq (the quantize tool’s name for INT4 AWQ) acts earlier than the rounding above. Lower-precision weights make the model compute slightly different scores in the first place, z + δ, where δ is a small error on each token. Between two labels, q shifts by a factor of e^(δ_i - δ_j). The bias is added after this error already exists, so +100 neither makes it bigger nor hides it. The error should grow as the bit-width shrinks, with int4_awq the noisiest, but how much is something to measure: compare q from the quantized model with q from the unquantized fp16 build on the same prompts.
One detail of the quantize command matters here. Passing only --quantization (fp8, nvfp4 or int4_awq) leaves the LM head, the final layer that produces the scores, unquantized. It is quantized only if you also set--lm_head_quantization, which is off by default. An unquantized head runs in plain fp16, even for a bf16 checkpoint and its output goes into the runtime's fp32 scores. So the weight-quantization error in the logits comes from the rest of the model alone, unless you opt the head in as well.
Small models often favor an option because of where it sits in the list rather than what it says. Measure this on the evaluation set and where it matters, run the decision several times with the labels shuffled and average the scores.
Calibration is a separate concern. A softmax over three labels always adds up to 1, even when the model has no idea, so q is a relative score and not the probability of being right. To get trustworthy thresholds, fit a calibration map such as temperature scaling or isotonic regression on labelled data and set routing thresholds from that instead of from raw q.
If a label spans several tokens, its probability is a product: the probability of the first token, times the probability of the second given the first and so on:
Each extra token needs one more pass through the model, which is exactly the cost a decision model exists to avoid and the bias trick cannot score it anyway. Longer labels also get smaller products simply because they are longer. The practical answer is single-token labels, with their meaning spelled out in the prompt.
The code below is a reconstruction with the prompt text and port filled in, so check the names (LLM, SamplingParams, num_logprobs, disable_spec_decode) against your build. First, start the server. It builds the engine from a Hugging Face model id or loads an engine you already built:
pip install -e ".[server]"python -m experimental.server --model Qwen/Qwen2.5-0.5B-Instruct --port 8000# or: --model /path/to/llm_engine
The decision function follows the article step by step. It turns each label into a single token id, builds a prompt that ends where the answer goes, sends the biased request, checks that every label made it into the window and renormalizes:
import mathfrom transformers import AutoTokenizerfrom experimental.server import LLM, SamplingParamsHF_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"tok = AutoTokenizer.from_pretrained(HF_MODEL) # same tokenizer the export usesllm = LLM(model=HF_MODEL) # or LLM(model="/path/to/llm_engine")def decide(ticket, choices, question="Which category matches the ticket?"): labels = [chr(ord("A") + i) for i in range(len(choices))] label_ids = [] for label in labels: ids = tok.encode(label, add_special_tokens=False) if len(ids) != 1: raise ValueError(f"{label!r} is not a single token: {ids}") label_ids.append(ids[0]) lines = "\n".join(f"{l} = {c}" for l, c in zip(labels, choices)) prompt = (f"Ticket:\n{ticket}\n\nQuestion:\n{question}\n\n" f"Allowed labels:\n{lines}\n\nReturn only the label.\nLabel:") params = SamplingParams( max_tokens=1, top_k=1, num_logprobs=len(label_ids), # <= 50 logit_bias={i: 100.0 for i in label_ids}, # max allowed bias is 100 disable_spec_decode=True, # required on v0.10.0 only; optional from v0.10.1 ) out = llm.generate([prompt], params)[0] step0 = {e.token_id: e.logprob for e in out.logprobs[0]} missing = [i for i in label_ids if i not in step0] if missing: raise RuntimeError(f"labels not in top-{len(label_ids)}: {missing}") m = max(step0[i] for i in label_ids) exps = [math.exp(step0[i] - m) for i in label_ids] z = sum(exps) probs = {c: e / z for c, e in zip(choices, exps)} return max(probs, key=probs.get), probsprint(decide("I was charged twice for the same subscription.", ["billing and payments", "technical support", "account access"]))
The same request works over HTTP. The ids 32, 33 and 34 are the Qwen token ids for A, B and C. Read choices[0].logprobs.content[0].top_logprobs from the response and renormalize it exactly as the Python version does. As in the Python version, disable_spec_decode is needed only on v0.10.0:
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{"role": "user", "content": "Ticket:\n...\nLabel:"}], "max_tokens": 1, "top_k": 1, "logprobs": true, "top_logprobs": 3, "logit_bias": {"32": 100, "33": 100, "34": 100}, "disable_spec_decode": true}'
The opening claimed that generating a structured answer costs far more than reading one token. Here is a direct test on TensorRT Edge-LLM. The same engine routed six support tickets, each run several times, with two methods. The first is the logit-bias trick from this article. The second is the usual approach, structured output: the runtime’s guided decoding forces the model to write JSON that matches a schema, here an object with a single label field restricted to A, B or C.
def run_guided_json(llm, tok, ticket, choices, max_tokens): prompt, labels = build_prompt(ticket, choices) schema = { "type": "object", "properties": {"label": {"type": "string", "enum": labels}}, "required": ["label"], "additionalProperties": False, } params = SamplingParams(max_tokens=max_tokens, top_k=1, temperature=0.0, guided_decoding=("json_schema", json.dumps(schema, sort_keys=True))) out = llm.generate([prompt], params)[0] try: label = json.loads(out.text)["label"] pred = choices[labels.index(label)] except (ValueError, KeyError, TypeError): pred = None return pred, out.text, len(out.token_ids), None
For the ticket “My account got locked after too many attempts.”, the structured path returns { "label": "C" }, while the bias path returns the single token A, B or C along with its scores. The results:
At the median, structured output is 4.54 times slower. The token counts explain almost all of it. The bias call reads the prompt and produces one token in about 17 ms. The JSON call produces 9.2 tokens on average and its extra 8.2 tokens add about 61 ms, roughly 7.5 ms per token. Every one of those tokens is braces, quotes and the word “label”, text that carries no information because the schema already fixed it. The decision itself is the single letter in the middle.
Both methods parsed and answered all six tickets correctly, so this test says nothing about accuracy. Six easy tickets cannot separate two methods that read the same model’s preferences and a fair accuracy comparison needs a larger and harder labelled set. What the test does show is that the latency gap is real and that it comes from decode steps, exactly as the opening argued.
The logit-bias trick is a proof of concept. It shows that an ordinary LLM can act as a one-pass decision model, using nothing but two generic request parameters and it explains why that works. On TensorRT Edge-LLM there is a cleaner way to get the decision itself: guided decoding with the choice constraint. Guided decoding first appears in v0.11.0. It does not exist in v0.10.0 or v0.10.1, so on those releases the bias trick is the only option. [1]
def run_guided_choice(llm, tok, ticket, choices, max_tokens): prompt, labels = build_prompt(ticket, choices) params = SamplingParams(max_tokens=max_tokens, top_k=1, temperature=0.0, guided_decoding=("choice", json.dumps(labels))) out = llm.generate([prompt], params)[0] return parse_label(out.text, labels, choices), out.text, len(out.token_ids), None
Guided decoding uses a library called XGrammar. The choice list is shorthand for a tiny grammar that says the whole output must be exactly one of the listed strings, for example root ::= "A" | "B" | "C". At every generation step, the runtime does the following:
This is the bias trick turned inside out. Instead of lifting the labels by +100, it pushes every other token down to an effectively infinite negative score. Both leave the gaps between the labels untouched, so with single-letter labels and top_k = 1, guided choice picks the same label as the argmax of q. Because XGrammar tracks valid prefixes, it also handles choices that span several tokens, which the bias trick cannot do.
On v0.11.0, the runtime applies the guided mask right after the logit bias and before it computes the logprobs, all on the same scores. So the logprobs are taken from the masked scores. Every masked token contributes e^(-1e30 — max), which is zero and the log-softmax runs over the allowed tokens alone. For single-token choices, asking for top-N logprobs with N equal to the number of choices therefore returns exactly the label-only softmax q from earlier in this article. You get the decision and its scores, with no bias and no window problem.
Guided choice constrains the format of the answer, not its correctness. It also hides label mass exactly the way the bias trick does: the logprobs it reports are already normalized over the allowed tokens, so they always look confident, even when the model had almost no probability on any label. The prompt shaping and the hit-rate check from the failure section therefore still apply. For multi-token choices, selection happens one token at a time, so the result is not guaranteed to be the choice with the highest total probability.
Two runtime limits are also worth knowing. Guided decoding is not supported on block-diffusion engines. And for thinking models, the constraint only starts after the reasoning block ends, so keep thinking disabled for decisions.
A decision does not need generation. An agent picking a tool, a category or a route needs one forward pass and a score for each allowed answer and an ordinary LLM already computes those scores in its logits. The label-only softmax turns them into a decision and the +100 head start gets the labels past the top-N window without changing a single gap between them. On TensorRT Edge-LLM, the runtime adds the bias before it computes the logprobs and keeps the scores in fp32, so the bias costs no accuracy.
The measurements make the case in numbers: one generated token in about 17 ms against 9.2 tokens of JSON in about 79 ms, with the difference spent almost entirely on characters the schema had already fixed. The real risk is not the arithmetic but the model’s intent. If the model never wanted to write a label, every method here will still return a confident-looking answer, so shape the prompt to end where the answer goes and measure the hit rate before trusting any score.
Which method to use comes down to the release you run:
Build a Jev-Style Decision Model from an LLM was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.