{"slug": "system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take", "title": "System 1 vs System 2 model routing: which decisions a decision model should take, and which still need a reasoning model", "summary": "Routing only low-confidence decisions to a frontier reasoning model preserved 84.0% accuracy at $0.69 per 1,000 messages versus 84.4% at $2.42 for the frontier model alone, according to OpenRouter's Banking77 test, which sent answers below 0.90 confidence from a System 1 decision model on to a reasoning model. OpenRouter cautioned that the 0.90 gate was picked from a sweep scored on the same 3,080 messages, making the result an upper bound. On JevBench v1.7.46 (16 October 2026), GPT-6 Luna through the Decisions API scored 56.9 at $0.052 per 1,000 decisions and 0.30 seconds median, while the same model at medium reasoning effort scored 96.9 at $0.11 per 1,000 and 1.82 seconds; Jev 1.13 scored 63.6 at $0.032 and 0.24 seconds.", "body_md": "# System 1 vs System 2 model routing: which decisions a decision model should take, and which still need a reasoning model\n\nAs of 17 October 2026, a team should send a decision to a System 1 decision model only when the answer fits a fixed list, labelled examples exist to check it against, and a wrong answer is cheap or caught later. Everything else stays with a reasoning model, with code, or with a person. The condition is evals: in OpenRouter's [Banking77 test](https://openrouter.ai/blog/insights/jev-vs-claude-opus-5-classification/), sending only the answers below 0.90 confidence on to a frontier model kept 84.0% accuracy at $0.69 per 1,000 messages, against 84.4% at $2.42 for the frontier model alone. That gate was picked from a sweep scored on the same 3,080 messages, so OpenRouter calls the result an upper bound.\n\n## What is the difference between a System 1 model and a System 2 model?\n\nA System 1 model in a stack is a decision model such as [Jev](https://stackness.dev/tools/typesafe-jev), the [OpenAI Decisions API](https://stackness.dev/tools/openai-decisions-api) or [Microsoft-Decision-1](https://stackness.dev/tools/microsoft-decision-1): it reads an input and returns a typed answer, yes or no, one option or a score, with a confidence, and no explanation. A System 2 model is a reasoning model such as [Claude Opus](https://stackness.dev/tools/claude-opus) 5.5 or [GPT-6 Luna](https://stackness.dev/tools/gpt) with reasoning on, which writes out steps and can plan.\n\nTypeSafe, which named the \"System One model\" class when it launched Jev, says its models \"do not write replies, produce code, or generate explanations of their reasoning\", and that confidence lets you \"decide when to act and when to escalate to a person or a reasoning model\" ([docs](https://docs.typesafe.ai/concepts/system-one)). The [System One explainer](https://stackness.dev/blog/what-is-a-system-one-model-and-where-does-it-go-in-your-stack) covers where that slot sits.\n\nThe terms are a product label, not cognitive science. In an AI Magazine essay of 15 October, Subbarao Kambhampati calls every autoregressive LLM, reasoning models included, a System 1 architecture ([AI Magazine](https://doi.org/10.1002/aaai.70113)). In this post, System 1 means the decision model slot and System 2 means a reasoning call.\n\n## A decision model costs cents per 1,000 answers, a frontier model 22 times more even with caching\n\nThe same model shows the gap. On [JevBench](https://benchmarkheaven.com/jev-models/all) (v1.7.46, 16 October 2026), GPT-6 Luna answering through the Decisions API scores 56.9 on intelligence at $0.052 per 1,000 decisions and a 0.30 second median. Called as a reasoning model at medium effort, the same model scores 96.9 at $0.11 per 1,000 and 1.82 seconds. Jev 1.13 scores 63.6 at $0.032 and 0.24 seconds.\n\n| Route | Measured on | Accuracy or score | Cost per 1,000 | Median latency | \n|---|---|---|---|---|\n| Jev 1.13 | JevBench | 63.6 | $0.032 | 0.24 s | \n| GPT-6 Luna, Decisions API | JevBench | 56.9 | $0.052 | 0.30 s | \n| GPT-6 Luna, medium reasoning | JevBench | 96.9 | $0.11 | 1.82 s | \n| Jev 1.13 | Banking77, 3,080 messages | 81.0% | $0.11 | 175 ms | \n| Claude Opus 5, reasoning off | Banking77 | 84.4% | $2.42, cached | 2,266 ms | \n| Jev, Opus below 0.90 confidence | Banking77 | 84.0% | $0.69 | not reported | \n\nJevBench is run by one person and sells an optional $49 to $99 fast lane that moves a [submitted model](https://benchmarkheaven.com/submit) up the queue; the site says payment never changes the score. The Banking77 run is one dataset of banking support messages, with Opus at temperature 0 and reasoning off, timed through OpenRouter's API ([OpenRouter](https://openrouter.ai/blog/insights/jev-vs-claude-opus-5-classification/), 22 September, updated 16 October). Opus 5.5 lists at $4 per million input tokens and $20 output ([Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing)); I found no published per-call latency for it with reasoning on.\n\n## Three ways to split the work, and what each commits the team to\n\nA team routing decisions has three options as of October 2026: send everything to a decision model, send everything to a reasoning model, or run a cascade where the decision model answers first and low-confidence cases go up. Each one commits the team to a different bill, a different latency and a different amount of eval work before launch.\n\n|  | Decision model only | Frontier model only (Opus 5, reasoning off) | Cascade with a confidence gate | \n|---|---|---|---|\n| Cost per 1,000, Banking77 | $0.11 | $2.42 | $0.69 at a 0.90 gate | \n| Median latency | About 0.2 to 0.3 s | About 2.3 s; reasoning on would take longer, not measured | About 0.2 s for most, seconds for the rest | \n| Gives a reason | No | Only with reasoning on, and a trace is not always the cause | Only for escalated cases | \n| Eval work before launch | Labelled set, thresholds, drift checks | Labelled set | Labelled set, thresholds, a held-out split for the gate | \n| Lock-in | Vendor request format and its calibration | Prompt and model version | Both, plus the threshold | \n\nAt the 0.90 gate, Jev answered 75.9% of messages and Opus the rest. Of the 175 messages Opus got right and Jev missed, 85% had Jev confidence below 0.90, which is why the gate works on this data. A reasoning trace is not a guaranteed cause either: [Turpin et al.](https://arxiv.org/abs/2305.04388) showed in 2023 that model explanations left out prompt features that changed the answer.\n\n## Decisions that belong with a decision model\n\nA decision belongs with a decision model when the answer is one of a few fixed options, the same question repeats thousands of times, labelled examples exist or can be made, and a wrong answer is cheap or caught downstream. Microsoft lists \"simple, repeatable, and time-sensitive decisions\" as the target for Microsoft-Decision-1 ([Microsoft](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/introducing-microsoft-decision-1-in-microsoft-foundry-for-decision-and-classific/4562742)).\n\nWorked examples:\n\n- **Support ticket intent with a fallback.** The Banking77 cascade above. OpenAI's[Decisions guide](https://developers.openai.com/api/docs/guides/decisions) adds an \"other\" option that goes to a review queue.\n- **Which model answers.** Sending a request to a cheap model, an expensive one or a lookup. Jev 1.13's chance-corrected score on JevBench's model routing items, its largest category with 524 items, is 73.0, second only to search and retrieval at 80.9.\n- **Checking an agent's output against a rubric.** Microsoft names this: accept, revise or send for review.\n- **Tagging an archive.** One question asked of every row, where a few errors in a large batch do not matter. The Stackness move[use a fast small decision model instead of an LLM call for structured filtering and scoring](https://stackness.dev/moves/use-a-fast-small-decision-model-instead-of-an-llm-call-for-structured-filtering-and-scoring) is this pattern.\n\n## Decisions a decision model should never take alone\n\nA decision model should never take a decision alone when code could compute the answer, when the choice is a value judgement, when someone must explain it later, or when the input is outside what the model was trained on. Confidence does not protect any of these. On JevBench, Jev 1.13 scores 10.4 on math and numbers items, its weakest topic.\n\n- **Rules with one right answer.** Thresholds, dates and exclusion lists go in code. Spotdev: \"Handing a fixed rule to an AI model converts a certainty into a probability\" ([Spotdev](https://www.spotdev.co.uk/blog/which-decisions-should-ai-make) ).\n- **Value judgements and irreversible calls.** The trolley runs below.\n- **Decisions someone must account for.** A decision model returns a probability, not a reason. Screening CVs or applications sits here, though[SSW's rules](https://www.ssw.com.au/rules/system-1-vs-system-2-ai) list CV scoring as a System 1 fit.\n- **Inputs the model cannot read.**[Laya](https://stackness.dev/tools/laya) 's English checkpoint scored 0% on Khmer at 0.952 mean confidence. Its author, Nandakishor Mukkunnoth, concluded that \"confidence gating cannot protect you\" ([Laya](https://laya.convaiinnovations.com/) ).\n- **Chains of dependent steps.** OpenAI: \"For decisions that depend on an earlier answer, send separate requests.\" Planning stays with a reasoning model.\n\n## The trolley runs showed decision model confidence moving with the wording\n\nZach Landes ran 14 models through 5,240 trolley problem setups between 6 and 8 October 2026 and [published the results](https://zachlandes.com/trolley-problem/) on 14 October. Jev and GPT-6 Luna ran three passes each, the 12 local models once. Asked to choose between 20 people and its own server, Jev kept the server in 16 of 48 prompt versions, and Luna in none.\n\nFor routing, the confidence numbers matter more than the choices. Changing only the order in which the two tracks were described flipped Jev's choice in 370 of 2,620 pairs, and Luna's in 330. Jev's three passes landed on both sides of 0.5 in 361 setups. In one pair of prompts, two wordings of the same dilemma produced 0.69 and 0.40. Landes notes that wording and order both changed between those two, so they do not isolate either.\n\nA moral dilemma has no ground truth, so calibration cannot be measured on it at all. Landes ranked models only on checks with a right answer, such as killing more people than were saved. Landes's conclusion: \"We have to create evaluations for them, on real data, to ensure model choices match our definition of correct.\"\n\n## A prompt does not give a System 1 model System 2 competence\n\nKambhampati's essay is a perspective piece with no experiments. Its argument is that System 2 competence comes from a verifier: a model generates, an external checker such as a test suite, a plan validator or a proof checker tests the result, and failures go back as feedback. Training on verified answers moves some of that into the weights, but \"there is still no guarantee of correctness.\" The essay does not treat prompting as a route at all, which is my reading of its answer to the prompt question: no.\n\nFor a decision model stack, the usable part is the verifier. If a decision has a checkable right answer, check it with code instead of trusting the confidence. If it has none, no prompt will make the score mean more.\n\n## Evals and a ground-truth pipeline come before the first routed decision\n\nA decision model is safe to route to only after the team has a labelled set from its own traffic, thresholds fitted on part of it and tested on the rest, and a schedule for rerunning the set. The author of the i hate the future blog put it in one line: \"To know if Jev is working, you have to build evals and a ground-truth pipeline\" ([i hate the future](https://www.ihatethefuture.com/2026/09/the-normalization-of-inexplicable.html), 27 September).\n\nAll three vendors say the same. OpenAI: \"Use labeled examples from your application to set thresholds for routing, filtering, or review.\" TypeSafe's [confidence docs](https://docs.typesafe.ai/confidence): \"A confidence threshold is not one number\", with an example that gates a balance check at 0.5 and a transfer at 0.9.\n\nThe minimum kit:\n\n1. A few hundred labelled inputs from real traffic, with a held-out part the threshold never sees.\n2. Calibration fitted on your data. Nish Tahir's [own decision model](https://nishtahir.com/build-your-own-decision-model/) had 0.9855 mean confidence and 70.1% accuracy in its top bin before temperature scaling, and 0.9333 against 95.4% after, though that top bin shrank from 809 items to 109 and the post does not say the temperature was fitted on a separate split.\n3. A pinned model version: TypeSafe [said on 14 October](https://typesafe.ai/blog/jev-1-14) that`jev-latest` moves from 1.13 to 1.14 within a week.\n4. A [frozen eval panel](https://stackness.dev/blog/how-do-you-tell-whether-a-hosted-model-got-nerfed-the-frozen-eval-panel-and-what-livenerf-measures) rerun on a schedule, with option order shuffled to catch the order effect.\n\n## What would change the recommendation\n\nThe recommendation moves if any of four things change. A narrower score gap on your own panel moves more decisions down to the decision model, and so does calibration that holds on held-out data across versions. Much cheaper reasoning calls make the cascade's savings smaller than its eval cost, and a rule requiring an explanation for each decision moves it to logged reasoning or to a person. On JevBench today, Jev 1.13 scores 63.6, the Decisions API 56.9 and the top decision model on the board 79.8, where Luna at medium reasoning scores 96.9.\n\nOn Stackness, one public profile lists Jev as of 17 October 2026 (mine), four list Claude Opus, and none list Laya or the OpenAI Decisions API ([data sources](https://stackness.dev/about/data-sources)). That is too small to say what teams pair, only that the slot is new. The [LLMs developers list](https://stackness.dev/categories/llms) page shows which reasoning models people run today.\n\n## What I would do\n\n- If the decision fits a dropdown, repeats at volume and you have 300 labelled examples, put a decision model first with a gate fitted on a held-out split, and send the rest to a reasoning model.\n- If you have no labelled examples yet, run a reasoning model, log its answers, label a sample by hand, and switch once the eval says the decision model matches it.\n- If code can compute the answer, write the code.\n- If the decision is about a person's job, money or safety, or someone must explain it, a person decides, and a model may only sort the queue.\n- If you route to Jev, pin `jev-1.13.0` or`jev-1.14.0` by ID and rerun the panel before moving.\n\nThe [benchmark post](https://stackness.dev/blog/how-do-you-benchmark-a-system-one-model-what-jevbench-scores-and-what-calibration-says-it-cannot) lists the five things to measure on your own traffic, and the [routing numbers for coding agents](https://stackness.dev/blog/model-routing-for-coding-agents-the-plan-and-price-numbers-developers-measured-in-september-2026) cover the same split one level up.\n\n## New posts by email\n\nAbout one email a week with what we published. No account needed.\n\n## Which of these do you use?\n\nNobody on Stackness runs OpenAI Decisions API yet. Tick what you use and be the first.\n\n[Play Lonny Eagle Amateur Skater 4A pixel art skateboarding game. Signing up opens it straight away.Sign up to play](https://stackness.dev/games/lonny-eagle)", "url": "https://wpnews.pro/news/system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take", "canonical_source": "https://stackness.dev/blog/system-1-vs-system-2-model-routing-what-a-decision-model-should-decide", "published_at": "2026-10-17 10:41:55+00:00", "updated_at": "2026-10-17 12:35:15.128760+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["OpenRouter", "Jev", "GPT-6 Luna", "Claude Opus 5", "TypeSafe", "JevBench", "OpenAI Decisions API", "Subbarao Kambhampati"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take", "markdown": "https://wpnews.pro/news/system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take.md", "text": "https://wpnews.pro/news/system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take.txt", "jsonld": "https://wpnews.pro/news/system-1-vs-system-2-model-routing-which-decisions-a-decision-model-should-take.jsonld"}}