{"slug": "your-next-ai-call-shouldn-t-write-a-word", "title": "Your Next AI Call Shouldn't Write a Word", "summary": "A new lab shipped a \"System One\" model that returns calibrated probabilities for typed questions in a single parallel forward pass instead of generating text, cutting end-to-end latency to 70-500 milliseconds versus 3-329 seconds for frontier models and pricing input at $0.042 per million tokens with no output-token billing. An independent benchmark puts its classification accuracy near the smallest LLMs at 83-87%, and open-weight clones speaking the same wire format appeared within a week. The first library built on the primitive replaces LLM judges with typed decisions, running eight evals in one request for $0.00006 and 0.33 seconds versus $0.46 and 4 seconds for a chat-model equivalent.", "body_md": "For three years we have been asking models built to talk for judgments our code needs as values, parsing whatever comes back, and calling the retry logic a pipeline.\n\n## 1. A Model Gave Up Writing Text. It Got Two Orders of Magnitude Faster.\n\nA new lab [shipped what it calls a System One model](https://typesafe.ai/blog/introducing-system-one-models-and-jev) this week, and the interesting part is what it deliberately cannot do: generate a string. You send it unstructured program state plus a set of typed questions — yes/no, pick one of these, score on this rubric — and it returns a calibrated probability for each one. All of them in a single parallel forward pass, not one token at a time. The pitch is a frontier-intelligence function call: state in, typed decisions out.\n\nThe numbers are the argument. End-to-end response time of 70 to 500 milliseconds against 3 to 329 seconds for frontier models on the same shaped query. Input at $0.042 per million tokens, with output tokens not billed at all, because there are no output tokens. Schema violations are not rare, they are structurally impossible — which matters less for a chatbot than for a decision buried four layers deep in a dependency chain with a latency budget. And every answer ships with a calibrated confidence score. The lab's sharpest line: a model that can do a task 95% of the time but cannot tell you which 5% it missed has not automated anything.\n\nBe a skeptic about the intelligence, though. An independent benchmark puts accuracy near the *smallest* LLMs on classification tasks — 83 to 87% on the standard intent-classification sets — with calibration that varies by task. This is not a smarter model; it is the same judgment at a thousandth of the cost, which is a different and more useful thing. The ecosystem read that immediately: open-weight clones speaking the same wire format appeared within a week, one [doing 45 decisions per second offline on an M4 Mac](https://gist.github.com/fordnox/e592d0f68b543fd044be8e6d040863a0), and someone has already bolted the primitive into [Python as a smart if statement](https://github.com/sumanmichael/jevlang).\n\n**Why it matters:**\n\n- **For ICs:** notice which of your LLM calls exist only to produce a label, a route, or a score. Those were never writing tasks, and they are the cheapest wins in your codebase.\n- **For leaders:** your AI bill is mostly output tokens spent on prose nobody reads. Chat models for humans, decision models for code is a budget line, not a research curiosity.\n- **For founders:** \"frontier intelligence\" was never one market. Intelligence-per-second at four cents per million tokens is a category a general chat lab is structurally bad at serving.\n\n## 2. Evals Went First, Because the Judge Was Always the Bottleneck\n\nThe first library built on this primitive [replaces LLM judges with typed decisions](https://github.com/openlayer-ai/jevals), and the economics explain why evals were the beachhead. Walk the source of a standard eval suite and the cost shows up fast: faithfulness is two LLM calls, answer relevancy is three plus embeddings, context precision is one call per retrieved chunk. Six to eleven round trips and several seconds to score one sample. So teams sample 1% of traffic, run it nightly, and the results never get near the request path.\n\nThe measured comparison is eight evals — tool choice, grounding, scope, relevancy, completeness, indirect injection, PHI — in one request: $0.00006 and 0.33 seconds, at a p50 of 244ms. The same suite with a chat model emulating the interface came in at $0.46 and 4 seconds. More damning than the cost is the variance: on identical traces, GPT and Claude judges showed 92x to 913x the run-to-run score variance of the decision model. A judge that changes its mind between runs is a mood ring, not a test suite.\n\nCheap and stable enough changes where the check lives. An eval costing six hundredths of a cent can run on every trace, and the same definition becomes a gate *inside* the loop — risk-scoring a refund or a DELETE statement before it executes, catching instructions hidden in a tool result before the model reads them. One definition offline and in production means monitoring and enforcement cannot drift apart, which is the failure mode every team with a nightly eval job eventually finds. The pattern is leaking outward already: an [OpenTelemetry connector](https://github.com/ishantanu/jevmetrics) now asks a model whether an unfamiliar metric is worth retaining, then applies ordinary deterministic policy to the probability.\n\n**Why it matters:**\n\n- **For ICs:** if your agent tests use an LLM judge, measure its variance across repeated runs before trusting a single regression it reports.\n- **For leaders:** \"we eval 1% nightly\" was a cost decision disguised as a methodology. That constraint just moved, and per-trace coverage is now affordable.\n- **For founders:** guardrails priced per frontier-model call could never sit in the request path. At sub-cent latency they can — a product category reopening.\n\n## 3. Prompts Aren't the Thing. Determinism Is.\n\nWhich brings us to the uncomfortable claim from an engineer who has spent a year making consumer agents behave: [prompts aren't real](https://evaluation.club). Not unimportant — not *real*. His structured-output field kept exploding past an 80-character limit until he renamed it from title to heading, a fix he calls fully deranged and expects to break again. Any new instruction drops the agent into a different contextual universe than the one you tested. So his team stopped curating prompts, built pass-power-k suites with holdout sets, and handed the prompt to an optimizer that rewrites it automatically. What ends up in there? Nobody reads it. The measurement is the artifact; domain experts should be producing labeled datasets, not paragraphs.\n\nThe same discipline arrives from the other direction in a quieter post reviving Gary Bernhardt's functional core and imperative shell, [weakened from purely functional to merely deterministic](https://outdata.net/blog/260803). Unseeded RNGs, network calls, the system clock — push them into the shell, collect the repeatable logic into a core, and practice what the author calls the defragmentation of determinism. A language model is simply the most non-deterministic dependency any of us has ever taken. Typed decisions are how you make it behave like every other hostile input: hand a value across the boundary and let a deterministic core decide what to do with it.\n\n**Why it matters:**\n\n- **For ICs:** the leverage is not in prompt craft. It is in the test harness and in shrinking the surface where the model gets to improvise.\n- **For leaders:** stop staffing a prompt-owning team. Staff the people who can produce golden datasets and the engineers who can wire an optimizer to them.\n- **For founders:** reliability is becoming an architecture problem, not a model problem — which means it is available to you without a training run.\n- The broader pattern: every mature engineering field eventually pushes its uncertainty to the edges and keeps a testable core. AI is just late to it.\n\n## The Verdict: Real or Hype?\n\n**Typed decision models → Real but early.** Genuinely fast, genuinely unhallucinating where the answer is a label, but accuracy sits near a small classifier — calibrate against your own labels before routing money through it. **LLM judges as your test suite → Hype.** Nothing with 900x run-to-run variance belongs in an assertion. **Determinism as AI architecture discipline → Real.** The cheapest reliability work available, and it needs no new vendor.", "url": "https://wpnews.pro/news/your-next-ai-call-shouldn-t-write-a-word", "canonical_source": "https://fromtheterminal.substack.com/p/your-next-ai-call-shouldnt-write", "published_at": "2026-09-21 22:40:16+00:00", "updated_at": "2026-09-21 23:22:50.673309+00:00", "lang": "en", "topics": ["ai-products", "ai-tools", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["typesafe.ai", "System One", "openlayer-ai", "jevlang", "M4 Mac", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/your-next-ai-call-shouldn-t-write-a-word", "markdown": "https://wpnews.pro/news/your-next-ai-call-shouldn-t-write-a-word.md", "text": "https://wpnews.pro/news/your-next-ai-call-shouldn-t-write-a-word.txt", "jsonld": "https://wpnews.pro/news/your-next-ai-call-shouldn-t-write-a-word.jsonld"}}