{"slug": "stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and", "title": "Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and…", "summary": "Splitting an AI feature's decision step from its generation step into separate inference endpoints makes the decision roughly 2.6× faster and calibrated, according to measurements published by the author on a local CPU server running the shipped server v1.6.0. Reading a category off one prefill of a 4B model took 355 ms versus about 920 ms to generate the same category, and an exact repeat over the same choice set hit the prefix cache in 84 ms. A 2B calibrated decision model served at POST /v1/systemone reports an expected calibration error of 0.054, which the author says lets a probability gate (act at p ≥ 0.90, confirm at 0.50 ≤ p < 0.90, escalate below 0.50) skip the expensive generation entirely when the answer says so.", "body_md": "*Most “AI features” are secretly two steps — a decision and a generation. Splitting them across a decision endpoint and a chat endpoint makes the decision ~2.6× faster, makes it calibrated, and lets you skip the expensive generation entirely when the answer says so. All numbers below are measured on a local CPU server.*\n\nWatch what your LLM actually does in a typical product feature:\n\nA support ticket arrives. The model reads it, figures out it’s a billing issue, decides it’s urgent — and *then* writes the reply. A search query comes in. The model works out whether it’s even a real query, what language it’s in, what the user meant — and *then* rewrites it. A RAG answer is drafted. Something decides whether it’s grounded enough to show — and *then* it ships.\n\nEvery one of these is **a decision followed by a generation**. And in most stacks, one chat model does both jobs inside a single generation call: “think about the category, then write the answer.”\n\nThat’s the slowest and least trustworthy way to do it, for two separate reasons:\n\n1. **The decision pays the full decode tax.** Producing the word “billing” costs the same sequential, token-by-token decode loop as producing a paragraph. You wait for prose to learn one bit of information.\n\n2. **The decision is uncalibrated.** When a chat model writes “I’m fairly confident this is billing,” that confidence is *text*, not a probability. You can’t threshold it, chart it, or defend it in a design review.\n\nThe fix is an old systems idea applied to inference: **pipeline the stages, and make each stage use the cheapest mechanism that does the job.**\n\nA decision has a fixed answer space — you know the possible answers before you ask. So don’t make the model *write* the answer. Run the prompt through the model once (the same “reading” pass that precedes any generation), then read the probability distribution **over just your allowed answers** directly off the logits. One prefill. Zero generated tokens.\n\nOn a local inference server this is a dedicated endpoint. The friendly form takes plain-text choices:\n\n```\nPOST /v1/score { \"model\": \"q35–4b\", \"query\": \"My card was charged twice this month.\", \"choices\": [\"billing\", \"shipping\", \"technical support\", \"none of the above\"] }  → { \"decision\": \"billing\", \"probabilities\": { \"billing\": 0.94, \"shipping\": 0.02, \"technical support\": 0.03, \"none of the above\": 0.01 } }\n```\n\nThe speedup is not subtle. Measured on the same 4B model, same M4-class CPU box:\n\nAsking the model to *generate* the category: ~920 ms. Reading it from one prefill: **355 ms**. And because the prompt is structured stable-part-first, repeated decisions over the same choice set hit the prefix cache — an exact repeat lands in **84 ms**. The decision stage of your pipeline just became an order of magnitude cheaper than the generation stage. That’s the point.\n\nFor decisions you’ll wire into business logic, there’s a stronger tool: a model *trained* for deciding. POST /v1/systemone (a 2B calibrated decision model) takes a state plus **typed questions** — yes/no probabilities, choices with criteria, ordered scores — and answers several of them in one request, because the state is prefilled once and forked per question. Its probabilities are calibrated (ECE 0.054): when it says 0.9, it’s right about 90% of the time. That one property is what makes the next stage possible.\n\nBetween the decision and the generation sits the most boring and most valuable component in the pipeline: an if statement.\n\n```\np ≥ 0.90 → act: run the generation on the routed path 0.50 ≤ p < 0.90 → confirm: proceed, but flag for review p < 0.50 → escalate: stop - a human sees it, with the probability attached\n```\n\nThis only works because the probabilities are calibrated — thresholding an uncalibrated score is theater. Here’s the gate with three **real, unedited measurements** from the shipped server (v1.6.0) plotted on it:\n\n· A support screenshot showing a payment-error dialog was routed to the **billing team at 0.9449** — the decision model read the words “card declined by issuer” *inside the image*. Above the act line: auto-route, no human needed.\n\n· A scanned claim form with a blank signature line scored **0.5598** on “are all required fields filled?” — squarely in the confirm band. Not rejected (three of four fields are plainly filled), not auto-accepted either. Exactly right.\n\n· Keyboard-mash gibberish queries measure **0.78–0.88** on a gibberish gate — which is *why* the reject threshold sits at 0.90, not at a round number someone liked.\n\nNotice what the middle case means operationally: the pipeline’s “uncertain” outcomes are not failures. They’re the system correctly routing ambiguity to a human instead of hiding it inside fluent prose.\n\nNow the expensive step — the actual writing — runs **only on the inputs that earned it**, with a prompt specialized by the decision:\n\n```\nPOST /v1/chat/completions { \"model\": \"q35–2b\", \"messages\": [ { \"role\": \"system\", \"content\": \"You are the BILLING support responder. …\" }, { \"role\": \"user\", \"content\": \"<the ticket>\" } ] }\n```\n\nThis is where the pipeline’s real economics show up. The per-call speedup on the decision (2.6×) is nice; the structural speedup is bigger: **every input the gate stops is a generation you never run.** If 30% of a queue gets escalated, deduplicated, or rejected at decision time, that’s 30% of your most expensive compute deleted — not optimized, deleted. The decision stage that enables it costs 355 ms a call.\n\nThere are three more structural wins hiding in the same shape:\n\n· **Right-size the generator.** Once routing is explicit, easy paths go to a 2B model and only the hard path pays for a big one. The decision model is effectively a traffic controller for model selection.\n\n· **Batch the gates.** Need five checks on one input? Ask all five questions in *one* decision request — the state is prefilled once, each extra question is nearly free. (Measured: a three-question request over a screenshot took the same ~12 s as a two-question one; on text, extra questions cost milliseconds.)\n\n· **Cache the repeats.** Decision prompts are highly repetitive by construction (same instruction, same choices, varying input). That’s the prefix cache’s best case — and it’s why the 84 ms number exists.\n\nHere’s a production-shaped example — the classic search pipeline (spell-correct → classify → expand), rebuilt as decide-then-generate:\n\n**Stage 1 (one decision request, three questions):** is this query gibberish? what language? what intent? — the gibberish gate rejects junk at p ≥ 0.90 before anything else spends time on it.\n\n**Stage 2 (generation, small model):** correct the spelling.\n\n**Stage 3 (generation, small model):** expand synonyms, assemble the “understood query” object.\n\nThe full chain measures **~6 seconds on CPU** — which tells you exactly what it’s for: offline and async enrichment (query-log mining, zero-result analysis, building suggestion dictionaries), not the live search path. That honesty matters: pipelines don’t make everything fast; they make the *structure* of your spend visible, so each stage can be placed where its latency belongs.\n\nA pipeline that gates inputs can also gate **outputs**. The newest piece of this stack (POST /v1/evaluate) grades a generated answer before it ships: deterministic validators first (valid JSON? truncated? do the citations match the evidence you passed?) in under a millisecond, then — for factual claims — the same calibrated decision model acts as a judge, answering bounded rubric questions against *your* evidence: supported? contradicted? complete? sufficient?\n\nIn measured fixtures, a supported answer scored p(support) 0.809, an unsupported one 0.222 — and an answer carrying a prompt-injection attempt hidden in the evidence stayed at 0.248, because the judge treats evidence as untrusted data. A deterministic failure short-circuits the judge (no model time wasted on an answer that’s already dead), and missing evidence caps the verdict at “uncertain” — it can never silently pass.\n\nWhich closes the loop: **decide → gate → generate → verify**, every stage on the same local box, every threshold a number you can defend.\n\n1. **Split the jobs.** Decisions are read off one prefill (355 ms, deterministic); generation is the only part that should pay the decode loop.\n\n2. **Calibration is the load-bearing property.** Gates built on calibrated probabilities (ECE 0.054) route ambiguity to humans instead of hiding it; gates built on prose confidence are decoration.\n\n3. **The biggest speedup is subtraction.** Every gated-out input is a generation you never run; every routed input is a big-model call downgraded to a small-model call.\n\n4. **Bundle and cache.** Many questions, one state, one request; stable prompt prefixes make repeats nearly free.\n\n5. **Gate both directions.** Evaluate outputs with the same machinery — deterministic checks for structure, a calibrated judge for grounding.\n\nEverything above runs on one local CPU server with an OpenAI-compatible API — the decision endpoints, the chat models, the evaluator, and the prefix cache are one binary, and none of the tickets, screenshots, or forms in these measurements left the machine.\n\n*All latency and probability figures are measured outputs from the shipped* *v1.6.0 server* *(Apple-M4-class CPU), captured while writing this article; the methodology notes live on the project’s performance page.*\n\n[Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and…](https://blog.devgenius.io/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-decisions-and-8e21e886ba73) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and", "canonical_source": "https://blog.devgenius.io/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-decisions-and-8e21e886ba73?source=rss----4e2c1156667e---4", "published_at": "2026-10-31 06:02:42+00:00", "updated_at": "2026-10-31 06:17:16.181585+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "large-language-models", "ai-tools"], "entities": ["/v1/score", "/v1/systemone", "v1.6.0"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and", "markdown": "https://wpnews.pro/news/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and.md", "text": "https://wpnews.pro/news/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and.txt", "jsonld": "https://wpnews.pro/news/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-and.jsonld"}}