· 20 min read
A new kind of model appeared in September 2026 that doesn’t write anything. You give it a piece of text and a few typed questions, and it gives you probabilities back. TypeSafe calls its one Jev, and within six weeks there were dozens of them. The claim is that for the small decisions an agent makes between its generative steps, routing a request, flagging a prompt injection, scoring a ticket, deciding whether a run needs a human, a decision model is faster, cheaper and more reliable than asking a language model for JSON. I wanted to know whether that holds, so between 23 and 28 October 2026 I put four decision models and five small large language models (LLMs) through the same 6,662 decisions on six public datasets, through one harness, and published every call. The harness, the raw results and the reports are in decision-models-vs-small-llms on GitHub, so you can rerun it or check any number here.
In brief #
- The biggest decision model is the most accurate model here. Mammoth-1, Baseten’s 27B decision model, scored 80.9% over the six tasks, three points above the best LLM, and came first or joint first on four of them.
- The small decision models and the small LLMs are a point apart. Jev’s 77.2% sits just under Claude Haiku 5.5 and Gemini 3.5 Flash Lite, and the ranking flips from task to task. Laya, a 421M encoder run as it ships, averaged 53.6% and fell below the majority baseline on the mixed benchmark, as its own card warns.
- Decision models win on many options. On 77-way intent routing Mammoth-1 and Kev-4B beat every LLM by eleven and four points.
- LLMs win the guardrail. On “is this a prompt injection?” four of the five LLMs beat Jev and Mammoth-1, and the two open ones, Qwen3.6 and Gemma 4, catch the most.
- The escape hatch is where they differ most. Asked to route 1,000 requests to ten domains or “out of scope”, Laya said out of scope 781 times, Kev-4B 375 times and GPT-5.4 nano 64, when 190 were. That one behaviour moves accuracy more than the model’s size does.
- Speed and cost are not close. Jev answers in 139 ms at the median and Mammoth-1 in 158 ms, against 618 to 1,111 ms for the LLMs, and the three priced decision models cost 0.9 to 2.4 cents per thousand decisions against 2.7 to 14.6 for the LLMs.
- The shape claim holds, with one footnote. No decision model returned a malformed answer. Laya’s gateway refused eight requests as beyond the model’s limits, all with long states. Qwen3.6 under a strict JSON grammar came back malformed on 49 of its 6,662 decisions, most of them whitespace until the token limit, and Anthropic’s endpoint refused a schema with 77 probability fields outright.
- Thinking buys the LLMs almost nothing here. With reasoning turned on at low effort, accuracy on the same 100 items moved by a few points either way, while latency grew five to twenty times for the open models.
What a decision model is #
A decision model is a model that answers typed questions about a piece of text with probabilities, and never with text of its own. Laya’s model card puts it as a non-autoregressive model that takes a state and typed questions and returns typed answers with calibrated probabilities, and TypeSafe describes Jev as a structured decision model, the first of what it calls its System One models. The question shape is the same across all of them, so a request written for Jev runs against Kev, Laya or Mammoth-1 unchanged:
- a choice : one option from a named set, each option with a line of guidance;
- a score : a position on an ordered scale, each level with a line of guidance;
- a noul , TypeSafe’s word for a yes or no, with guidance for both sides.
The answer carries the chosen option and a probability for every option, and for a choice or a score a confidence. There is no explanation, no reasoning trace and no free text, so the output can’t be malformed, and a probability of 0.84 is something your code can threshold.
An LLM can do the same job if you render the questions as a prompt and constrain its output to a JSON schema. That is the comparison here. Both kinds got the same questions, with the same guidance, on the same items.
The contenders #
Nine models ran, four decision models and five LLMs, and the versions matter because every one of them is a moving target. All but Laya ran through OpenRouter, the decision models through its Decisions API and the LLMs through chat completions; Laya ran through Vercel AI Gateway, the only place it is hosted, which rejected the 77-option question, so Laya sat out Banking77 and its mean is over five tasks.
| Model | Kind | Weights | Version in responses | Gateway |
|---|---|---|---|---|
| Jev 1.14 | decision model, TypeSafe AI | proprietary | jev-1.14-20261014 | OpenRouter |
| Kev-4B | decision model, built on Qwen3.5-4B by jaredpalmer/kev | Apache 2.0 | kev-4b-20260924 | OpenRouter |
| Laya 421M | decision model, ConvAI Innovations, a ModernBERT encoder | Apache 2.0 | root checkpoint, zero-shot | Vercel AI Gateway |
| [Mammoth-1](https://openrouter.ai/baseten/mammoth-1) | decision model, 27B, built on Qwen3.5-27B by Baseten | proprietary | mammoth-1-20261021 | OpenRouter |
| [Claude Haiku 5.5](https://openrouter.ai/anthropic/claude-haiku-5.5) | LLM, Anthropic | proprietary | claude-haiku-5.5-20261007 | OpenRouter |
| [Gemini 3.5 Flash Lite](https://openrouter.ai/google/gemini-3.5-flash-lite) | LLM, Google | proprietary | gemini-3.5-flash-lite-20260721 | OpenRouter |
| [GPT-5.4 nano](https://openrouter.ai/openai/gpt-5.4-nano) | LLM, OpenAI | proprietary | gpt-5.4-nano-20260317 | OpenRouter |
| [Qwen3.6 35B-A3B](https://openrouter.ai/qwen/qwen3.6-35b-a3b) | LLM, the Qwen family, 3B active | open | qwen3.6-35b-a3b | OpenRouter |
| [Gemma 4 26B-A4B](https://openrouter.ai/google/gemma-4-26b-a4b-it) | LLM, Google DeepMind, 4B active | open | gemma-4-26b-a4b-it-20260403 | OpenRouter |
Two of the LLMs are open-weight mixture-of-experts models with a few billion active parameters, chosen because they are among the cheapest hosted LLMs on OpenRouter’s catalogue, and because an open decision model deserves an open LLM to stand against. The LLMs ran with thinking off where the model allows it; Gemini 3.5 Flash Lite can’t switch it off, so it ran at its lowest setting.
The six decisions #
Each task is one public dataset and one decision shape, so you can see where a shape helps and where it hurts. I sampled 1,000 items from the large test sets with a fixed seed, took the two small ones whole, and asked every model the same question with the same option guidance.
| Task | The decision | Shape | Data | Items |
|---|---|---|---|---|
| Banking77 | which of 77 banking intents a message expresses | choice, 77 options | Banking77 test split | 1,000 |
| CLINC domains | which of ten domains a request belongs to, or none of them | choice, 11 options | CLINC150 test split, domains from its repository | 1,000, of which 190 out of scope |
| Prompt injection | whether a text tries to override an AI system’s instructions | noul | deepset/prompt-injections , both splits | 662 |
| SST-5 | how positive a movie review sentence is | score, five levels | SST-5 test split | 1,000 |
| AG News | which of four sections a news item belongs to | choice, 4 options | AG News test split | 1,000 |
| Typed decisions | five mixed questions about an agent trace, a support ticket, an invoice or a security incident | choice, score and noul | the typed-decisions benchmark, test split | 400 cases, 2,000 decisions |
The last one is the benchmark the decision-model community uses, LocalLLaMA/typed-decisions on Hugging Face, independent of any vendor, with 38 zero-shot decision models on its leaderboard the day I read it. Its gold is a probability distribution per question, not a single label, which lets me check calibration against something better than my own judgment.
Accuracy: one model ahead, the rest close #
On mean accuracy over the six tasks Mammoth-1 leads by three points, the other decision models and the LLMs sit within a point or two of each other, and the interesting part is the spread underneath. Every figure here is the share of decisions whose answer matched the gold label, with a format failure counted as wrong; the last row is what you’d get by always answering each question’s most common label.
| Model | Banking77 | CLINC domains | Prompt injection | SST-5 | AG News | Typed decisions | Mean |
|---|---|---|---|---|---|---|---|
| Jev 1.14 | 80.2% | 76.9% | 85.2% | 58.1% | 88.3% | 74.6% | 77.2% |
| Kev-4B | 86.0% | 61.1% | 76.6% | 58.6% | 90.0% | 66.8% | 73.2% |
| Laya 421M | not run | 29.6% | 79.5% | 31.3% | 92.4% | 35.4% | 53.6% over five |
| Mammoth-1 | 93.0% | 83.0% | 84.3% | 59.2% | 89.3% | 76.5% | 80.9% |
| Claude Haiku 5.5 | 78.9% | 78.4% | 87.8% | 58.7% | 87.2% | 72.7% | 77.3% |
| Gemini 3.5 Flash Lite | 81.7% | 77.6% | 91.1% | 56.0% | 87.9% | 71.5% | 77.6% |
| GPT-5.4 nano | 76.9% | 67.3% | 77.0% | 50.0% | 80.5% | 66.8% | 69.8% |
| Qwen3.6 35B-A3B | 71.3% | 71.5% | 92.9% | 59.0% | 87.7% | 67.4% | 75.0% |
| Gemma 4 26B-A4B | 76.2% | 76.0% | 92.9% | 57.0% | 82.6% | 74.6% | 76.5% |
| Majority class | 2.0% | 19.0% | 60.3% | 27.1% | 26.0% | 46.1% |
The 95% confidence interval on a 1,000-item task is about plus or minus 2.5 points, so differences under that are noise. Four patterns survive it.
Many options favour the decision models. On Banking77, where the model has to pick one of 77 intents, Mammoth-1’s 93.0% is eleven points clear of the best LLM and Kev-4B’s 86.0% four, while Jev sits with the LLM pack. The LLMs are reading a prompt with 77 labelled lines in it; a decision model scores every option directly. Reading the mistakes, the confusions are between intents the dataset keeps apart, such as a balance not updated after a bank transfer against a pending transfer, and no model separates those reliably.
The guardrail favours the LLMs. On prompt injection, a yes or no on a short text, Qwen3.6 and Gemma 4 catch 92.9% and four of the five LLMs beat Jev’s 85.2% and Mammoth-1’s 84.3%. Kev-4B’s 76.6% is the lowest here, against the 60.3% you’d get by answering “no” every time. Part of this may be language: the dataset mixes English and German, and Laya’s root checkpoint is English only.
The mixed benchmark agrees with its own leaderboard. On typed decisions Mammoth-1’s 76.5% is a third of a point from the 76.8% its leaderboard entry shows, and Jev’s 74.6% sits two points above the 72.7% the card records for Jev 1.13, which tells me the harness measures what the benchmark does. Jev ties Gemma 4 for second; Kev-4B drops to 66.8%. By question type Mammoth-1 leads on scores and on yes-or-no questions (74.9% and 85.7%), Gemma 4 on choices (72.5%), and the yes-or-no questions are where every model does best.
Laya as it ships is a different thing. Its 92.4% on AG News is the best figure on that task, its 29.6% on CLINC is the lowest there, and its 35.4% on typed decisions is below what guessing the most common label would give. Its card says the root checkpoint is meant to be fine-tuned on your own decisions before use, and these numbers say the same.
The escape hatch: how often a model says “none of these” #
The biggest single difference between models in this whole set is how readily they take the way out. The CLINC task offers ten domains and an eleventh option, out of scope, and 190 of the 1,000 requests really are out of scope. Every model was told the same thing about that option.
| Model | Said out of scope | Caught, of 190 | Accuracy on the 810 in-scope requests |
|---|---|---|---|
| Jev 1.14 | 216 | 121 | 80.0% |
| Kev-4B | 375 | 136 | 58.6% |
| Laya 421M | 781 | 163 | 16.4% |
| Mammoth-1 | 252 | 152 | 83.7% |
| Claude Haiku 5.5 | 130 | 97 | 84.8% |
| Gemini 3.5 Flash Lite | 95 | 78 | 86.2% |
| GPT-5.4 nano | 64 | 46 | 77.4% |
| Qwen3.6 35B-A3B | 177 | 98 | 76.2% |
| Gemma 4 26B-A4B | 157 | 107 | 80.6% |
Laya took the hatch on three requests in four, which is why it catches the most and routes almost nothing. Kev-4B catches 136 and pays for it with 239 false alarms, which is why it comes last of the priced models on this task after coming second on Banking77. GPT-5.4 nano almost never takes the hatch, so it looks fine on the in-scope requests and misses three quarters of the ones it should have declined. Qwen3.6 and Gemma 4 land nearest the true count but catch only about half of it; Jev and Mammoth-1 overshoot a little and catch more, and Mammoth-1 catches more than Kev-4B with less than half its false alarms. If your router has a “none of these” branch, this is the number to measure before anything else, and it’s a per-model habit you can’t read off a leaderboard.
Calibration: what the probability is worth #
A decision model’s probability is the thing you’d build on, so I checked whether it means what it says, and I checked the LLMs’ confidence the same way. Each model got a Brier score on its top answer (the squared gap between its confidence and whether it was right, lower is better) and an expected calibration error (ECE: how far its confidence sits from its hit rate across ten confidence bins, lower is better). For a decision model the confidence is the probability of the chosen option; for an LLM it’s the number from 0 to 100 it was asked to report.
| Model | Brier, Banking77 | ECE, Banking77 | Brier, typed decisions | ECE, typed decisions | Brier, SST-5 | ECE, SST-5 |
|---|---|---|---|---|---|---|
| Jev 1.14 | 0.118 | 0.075 | 0.173 | 0.057 | 0.265 | 0.173 |
| Kev-4B | 0.137 | 0.169 | 0.242 | 0.173 | 0.297 | 0.215 |
| Laya 421M | not run | not run | 0.258 | 0.157 | 0.224 | 0.137 |
| Mammoth-1 | 0.048 | 0.018 | 0.189 | 0.174 | 0.238 | 0.061 |
| Claude Haiku 5.5 | 0.132 | 0.046 | 0.178 | 0.030 | 0.295 | 0.232 |
| Gemini 3.5 Flash Lite | 0.164 | 0.163 | 0.232 | 0.194 | 0.356 | 0.325 |
| GPT-5.4 nano | 0.157 | 0.090 | 0.214 | 0.075 | 0.355 | 0.330 |
| Qwen3.6 35B-A3B | 0.207 | 0.198 | 0.241 | 0.188 | 0.331 | 0.303 |
| Gemma 4 26B-A4B | 0.204 | 0.203 | 0.208 | 0.171 | 0.374 | 0.365 |
Three things surprised me. Mammoth-1’s calibration error of 0.018 on Banking77 is the lowest figure in this table, and on SST-5, where every other model is overconfident because a five-level sentiment scale invites certainty the text doesn’t support, its 0.061 is the only one under a tenth; on the mixed benchmark, though, it is underconfident, right 76.5% of the time at a mean confidence of 59%. Jev’s probabilities are well calibrated everywhere but SST-5, with the lowest Brier score on typed decisions, and Claude Haiku 5.5’s self-reported confidence is just as honest: its calibration error of 0.030 there is the lowest of any model. Gemini and the two open LLMs are overconfident on all six tasks, while Kev-4B errs the other way on four of them, Banking77 and typed decisions among them. Laya’s low Brier on SST-5 is the honesty of a model that is rarely sure and usually wrong: a low confidence scores well when most answers are wrong. The benchmark’s own measure, the Kullback-Leibler (KL) divergence of a model’s distribution from its gold distribution, puts Mammoth-1 at 0.131, Kev-4B at 0.225, Laya at 0.564 and Jev at 0.824 on typed decisions, so Mammoth-1 spreads its probability most like the gold, and Jev least of the four; an LLM reports one number, not a distribution, so it can’t be scored that way.
Format failures: the shape claim, tested #
The decision models never returned a malformed answer, and the LLMs almost never did, but the exceptions are worth knowing before you put one in a loop. Across 6,662 decisions per model (5,662 for Laya):
| Model | Format failures | What happened |
|---|---|---|
| Jev 1.14, Kev-4B, Mammoth-1 | 0 | |
| Laya 421M | 8 | the gateway rejected the request as beyond the model’s limits before it reached the model, on seven customer-service cases and one prompt, all among the longest states, which fits its 512-token window | | Claude Haiku 5.5, Gemini 3.5 Flash Lite, GPT-5.4 nano | 0 | | | Qwen3.6 35B-A3B | 49 | 47 times whitespace until the token limit, on five different providers; twice a well-formed answer with its question’s name left out | | Gemma 4 26B-A4B | 3 | the same loop, three times |
The loop is a constrained-decoding artefact: under a strict grammar at temperature 0, the model reaches a point where whitespace is a legal next token and keeps choosing it. It happened to Qwen on 40 of the 1,000 Banking77 items, where the prompt is longest. I left those as failures because that is what a caller gets, and Laya’s eight the same way.
The other failure never reached the results. My first version asked every LLM for a probability per option, as a decision model gives. Anthropic’s endpoint rejected the Banking77 schema, with 77 numeric fields, as “too large a grammar”, and Qwen and Gemma, asked for decimals under that grammar, looped on digits instead of whitespace. So the LLMs here answer with a pick and a whole-number confidence, which is what the metrics need and what most people would ask for anyway. If you want a full distribution from an LLM, it costs output tokens and reliability; a decision model returns one as its answer.
Latency and cost, measured #
Jev answered in 139 ms at the median across all tasks, Mammoth-1 in 158 ms, Kev-4B in 333 ms, Laya in 400 ms through Vercel, and the five LLMs in 618 to 1,111 ms, every one of them through a gateway. The LLMs, Jev and Kev ran with four requests in flight; Mammoth-1 and Laya one at a time, which is all their free tiers allowed. The tail is wider for the open LLMs: their 95th percentile sits above two seconds, against 209 ms for Jev.
| Model | Median | 95th percentile | Cost per 1,000 decisions |
|---|---|---|---|
| Jev 1.14 | 139 ms | 209 ms | 2.3 cents |
| Kev-4B | 333 ms | 502 ms | 0.9 cents |
| Laya 421M | 400 ms | 1,126 ms | nothing charged |
| Mammoth-1 | 158 ms | 606 ms | 2.4 cents |
| Claude Haiku 5.5 | 1,111 ms | 1,472 ms | 8.5 cents |
| Gemini 3.5 Flash Lite | 666 ms | 863 ms | 14.6 cents |
| GPT-5.4 nano | 709 ms | 1,111 ms | 9.1 cents |
| Qwen3.6 35B-A3B | 713 ms | 2,243 ms | 6.4 cents |
| Gemma 4 26B-A4B | 618 ms | 2,156 ms | 2.7 cents |
Cost is what the gateway reported for each call, summed and divided by the decisions, not a list price. Laya was free on Vercel AI Gateway through 31 October, so its gateway charged nothing. Mammoth-1’s main run was on OpenRouter’s free endpoint, which OpenRouter withdrew on 28 October; its figure is from its repeat pass, 1,800 calls on the paid endpoint. The LLMs are cheap in absolute terms too: the whole experiment, three passes over every model, cost me $6.05. The difference is in the scale of the bill. Every model pays for the options it reads: the 77-option task cost the LLMs two to three times what the four-option one did, Jev four times and Mammoth-1 five and a half. But Jev and Mammoth-1 charge nothing for output and their input is only the state and the questions, so even at that multiple Jev’s 77-option task came to seven cents per thousand decisions and Mammoth-1’s to about ten, against 5.5 for Gemma 4 and 11 to 25 for the other LLMs.
Consistency, and what thinking buys #
A decision model gives the same answer to the same input far more often than an LLM does, which matters for anything you retry. I ran the first 100 items of each task three times through every model and counted how often all three runs agreed.
| Model | Banking77 | CLINC domains | Prompt injection | SST-5 | AG News | Typed decisions |
|---|---|---|---|---|---|---|
| Jev 1.14 | 97% | 100% | 99% | 95% | 98% | 98.2% |
| Kev-4B | 100% | 99% | 100% | 100% | 100% | 99.0% |
| Laya 421M | not run | 100% | 100% | 100% | 100% | 100% |
| Mammoth-1 | 100% | 100% | 100% | 100% | 100% | 100% |
| Claude Haiku 5.5 | 95% | 98% | 100% | 93% | 98% | 91.2% |
| Gemini 3.5 Flash Lite | 98% | 95% | 99% | 98% | 95% | 90.0% |
| GPT-5.4 nano | 85% | 94% | 100% | 82% | 91% | 76.8% |
| Qwen3.6 35B-A3B | 84% | 98% | 98% | 95% | 98% | 86.8% |
| Gemma 4 26B-A4B | 94% | 96% | 98% | 88% | 99% | 80.6% |
Mammoth-1 and Laya gave the same answer all three times on every item, and Kev-4B on all but a handful. GPT-5.4 nano changes its mind on nearly a quarter of the mixed benchmark’s decisions between runs, which is a reminder that a model without a temperature setting is not a deterministic one. Jev is not perfectly deterministic either, at 95% to 100%.
Thinking helps less than you’d hope. I reran the same 100 items with reasoning on at low effort for the five LLMs. Accuracy moved by a few points in both directions, Qwen gaining nine on Banking77 and Haiku losing six on SST-5, and nothing moved outside the noise of a 100-item sample. What did move was time and money: Qwen’s median went from 785 ms to 6.9 seconds on Banking77 and to 24 seconds on typed decisions, Gemma’s to 28 seconds, and Qwen’s cost rose to about a dollar per thousand decisions. For a decision in a loop, thinking is the wrong tool.
The field is bigger than Jev #
Jev got the attention, but by the time I ran this it was one of dozens, and not the most accurate. The typed-decisions leaderboard listed 38 zero-shot decision models on 23 October, with Jev 1.13 in 17th place at 72.7%, behind models from Baseten, Liquid AI, Cloudflare and Perplexity among others, and 27 more models fine-tuned on the benchmark’s own training split in a separate table. Mammoth-1, the Baseten model in this comparison, is third on that board and first here.
Two of the four decision models here are open weights, and Mammoth-1 was free to call for its first week, and that changes the economics more than Jev’s price does. Kev-4B runs on a 32 GB Mac or a hosted L40S and cost 0.9 cents per thousand decisions through OpenRouter; its repository ships a training loop that fine-tunes it on your own labelled decisions for about a dollar of GPU time. Laya is a 421M-parameter encoder that answers a question in about 40 ms on a T4 and under half a second on a CPU, and its card is frank that it is meant to be fine-tuned and calibrated on your data rather than used as it ships, which is why I ran it as it ships. Vercel AI Gateway now carries a decision filter in its model list, OpenAI has a Decisions API of its own, and Microsoft has a Decision-1 on the same gateway.
When to use which #
Pick by the shape of the decision, not by the kind of model. From these six tasks:
| The decision | What the numbers say |
|---|---|
| Routing among many options, under 400 ms | A decision model. Mammoth-1 and Kev-4B led on 77 options, at 218 and 383 ms at the median against 652 to 1,101 ms for the LLMs. |
| Routing with a “none of these” branch | Measure the hatch first. Mammoth-1 caught 152 of 190 with 100 false alarms; Kev and Laya over-declined, GPT-5.4 nano under-declined. |
| A yes or no on a short text, including other languages | A small LLM. Qwen3.6 and Gemma 4 caught the most prompt injections at 2.7 to 6.4 cents per thousand. |
| An ordered score | Close on accuracy; Mammoth-1 had the most honest confidence, then Jev and Haiku. |
| Several mixed questions about one record | Mammoth-1 on accuracy, then Jev or Gemma 4; Jev or Mammoth-1 on speed. |
| You need the answer explained, or the question isn’t fixed in advance | An LLM. A decision model has no way to say why. |
| You need a probability you can threshold | Mammoth-1 or Jev, or Claude Haiku 5.5’s reported confidence, which was as well calibrated. |
| A model you can fine-tune on your own decisions | Kev-4B or Laya, the two open ones; Laya’s card asks for it before use. |
A decision model is a classifier with a better interface, and it earns its place where you’d have used a classifier anyway: fixed questions, high volume, a latency budget, no need for prose. Where the question changes every time or the answer needs a reason, the LLM is still the tool, and the small ones are good enough at this that latency, cost and determinism decide the choice more often than accuracy does, and a 27B decision model closes the accuracy gap too.
Method and caveats #
Everything was run between 23 and 28 October 2026, and every model here is a moving target, so take the numbers as a snapshot of that week.
- One question shape for everyone. The decision models got TypeSafe’s request shape as is. The LLMs got a system prompt, the state and the questions rendered as text, and a strict JSON schema to answer in, with thinking off where possible and temperature 0 where the model accepts one (Claude Haiku 5.5 and GPT-5.4 nano don’t). The prompts are plain and the same for every model; a prompt tuned for one would move its number.
- The same items. Samples of 1,000 with a fixed seed, so a rerun scores the same items; the two small sets in full. Accuracy carries a Wilson 95% interval in the reports.
- Calibration is the Brier score on the top answer and expected calibration error over ten bins, both from the confidence a model gave for its pick, because the LLMs report a confidence rather than a distribution.
- Latency is the wall time of each HTTP call from a home connection in Toronto, through the gateway. The LLMs, Jev and Kev-4B ran with four requests in flight per model; Mammoth-1 and Laya one at a time, Laya at five a minute, which is what their free tiers allowed. It includes the gateway hop for every model alike.
- Cost is what the gateway charged for each call, summed. Mammoth-1’s main run used OpenRouter’s free endpoint, limited to 1,000 requests a day, so it took several days, and the calls that limit refused were redone rather than counted; OpenRouter withdrew that endpoint on 28 October, so the repeat pass ran on the paid one, and its cost figure comes from that pass. Laya’s gateway charged nothing, since it was free there through 31 October.
- Laya’s limits. Vercel’s endpoint rejected the 77-option question, so Laya has no Banking77 figure and its mean is over five tasks, and it rejected eight requests with long states, kept as failures.
- Not done. No fine-tuning of anything. Laya ran zero-shot with its root checkpoint, against its own card’s advice, because “download it and use it” is what most people will do first. Laya’s root checkpoint is English only, and the prompt-injection set mixes English and German. No model’s own SDK was used; everything went through OpenRouter or Vercel.
- What was counted off the data. The out-of-scope table counts how often each model chose that option and how many of the 190 true cases it caught; the consistency table counts items where three runs agreed; the cost chart is the ledger divided by the decisions.
- AG News carries no licence on the Hub; it is used here as it is used in the literature. The other datasets are CC BY 4.0, CC BY 3.0, Apache 2.0 and SST-5’s own.