# Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and… > Source: > Published: 2026-10-31 06:02:42+00:00 *Most “AI features” are secretly two steps — a decision and a generation. Splitting them across a decision endpoint and a chat endpoint makes the decision ~2.6× faster, makes it calibrated, and lets you skip the expensive generation entirely when the answer says so. All numbers below are measured on a local CPU server.* Watch what your LLM actually does in a typical product feature: A support ticket arrives. The model reads it, figures out it’s a billing issue, decides it’s urgent — and *then* writes the reply. A search query comes in. The model works out whether it’s even a real query, what language it’s in, what the user meant — and *then* rewrites it. A RAG answer is drafted. Something decides whether it’s grounded enough to show — and *then* it ships. Every one of these is **a decision followed by a generation**. And in most stacks, one chat model does both jobs inside a single generation call: “think about the category, then write the answer.” That’s the slowest and least trustworthy way to do it, for two separate reasons: 1. **The decision pays the full decode tax.** Producing the word “billing” costs the same sequential, token-by-token decode loop as producing a paragraph. You wait for prose to learn one bit of information. 2. **The decision is uncalibrated.** When a chat model writes “I’m fairly confident this is billing,” that confidence is *text*, not a probability. You can’t threshold it, chart it, or defend it in a design review. The fix is an old systems idea applied to inference: **pipeline the stages, and make each stage use the cheapest mechanism that does the job.** A decision has a fixed answer space — you know the possible answers before you ask. So don’t make the model *write* the answer. Run the prompt through the model once (the same “reading” pass that precedes any generation), then read the probability distribution **over just your allowed answers** directly off the logits. One prefill. Zero generated tokens. On a local inference server this is a dedicated endpoint. The friendly form takes plain-text choices: ``` POST /v1/score { "model": "q35–4b", "query": "My card was charged twice this month.", "choices": ["billing", "shipping", "technical support", "none of the above"] } → { "decision": "billing", "probabilities": { "billing": 0.94, "shipping": 0.02, "technical support": 0.03, "none of the above": 0.01 } } ``` The speedup is not subtle. Measured on the same 4B model, same M4-class CPU box: Asking the model to *generate* the category: ~920 ms. Reading it from one prefill: **355 ms**. And because the prompt is structured stable-part-first, repeated decisions over the same choice set hit the prefix cache — an exact repeat lands in **84 ms**. The decision stage of your pipeline just became an order of magnitude cheaper than the generation stage. That’s the point. For decisions you’ll wire into business logic, there’s a stronger tool: a model *trained* for deciding. POST /v1/systemone (a 2B calibrated decision model) takes a state plus **typed questions** — yes/no probabilities, choices with criteria, ordered scores — and answers several of them in one request, because the state is prefilled once and forked per question. Its probabilities are calibrated (ECE 0.054): when it says 0.9, it’s right about 90% of the time. That one property is what makes the next stage possible. Between the decision and the generation sits the most boring and most valuable component in the pipeline: an if statement. ``` p ≥ 0.90 → act: run the generation on the routed path 0.50 ≤ p < 0.90 → confirm: proceed, but flag for review p < 0.50 → escalate: stop - a human sees it, with the probability attached ``` This only works because the probabilities are calibrated — thresholding an uncalibrated score is theater. Here’s the gate with three **real, unedited measurements** from the shipped server (v1.6.0) plotted on it: · A support screenshot showing a payment-error dialog was routed to the **billing team at 0.9449** — the decision model read the words “card declined by issuer” *inside the image*. Above the act line: auto-route, no human needed. · A scanned claim form with a blank signature line scored **0.5598** on “are all required fields filled?” — squarely in the confirm band. Not rejected (three of four fields are plainly filled), not auto-accepted either. Exactly right. · Keyboard-mash gibberish queries measure **0.78–0.88** on a gibberish gate — which is *why* the reject threshold sits at 0.90, not at a round number someone liked. Notice what the middle case means operationally: the pipeline’s “uncertain” outcomes are not failures. They’re the system correctly routing ambiguity to a human instead of hiding it inside fluent prose. Now the expensive step — the actual writing — runs **only on the inputs that earned it**, with a prompt specialized by the decision: ``` POST /v1/chat/completions { "model": "q35–2b", "messages": [ { "role": "system", "content": "You are the BILLING support responder. …" }, { "role": "user", "content": "" } ] } ``` This is where the pipeline’s real economics show up. The per-call speedup on the decision (2.6×) is nice; the structural speedup is bigger: **every input the gate stops is a generation you never run.** If 30% of a queue gets escalated, deduplicated, or rejected at decision time, that’s 30% of your most expensive compute deleted — not optimized, deleted. The decision stage that enables it costs 355 ms a call. There are three more structural wins hiding in the same shape: · **Right-size the generator.** Once routing is explicit, easy paths go to a 2B model and only the hard path pays for a big one. The decision model is effectively a traffic controller for model selection. · **Batch the gates.** Need five checks on one input? Ask all five questions in *one* decision request — the state is prefilled once, each extra question is nearly free. (Measured: a three-question request over a screenshot took the same ~12 s as a two-question one; on text, extra questions cost milliseconds.) · **Cache the repeats.** Decision prompts are highly repetitive by construction (same instruction, same choices, varying input). That’s the prefix cache’s best case — and it’s why the 84 ms number exists. Here’s a production-shaped example — the classic search pipeline (spell-correct → classify → expand), rebuilt as decide-then-generate: **Stage 1 (one decision request, three questions):** is this query gibberish? what language? what intent? — the gibberish gate rejects junk at p ≥ 0.90 before anything else spends time on it. **Stage 2 (generation, small model):** correct the spelling. **Stage 3 (generation, small model):** expand synonyms, assemble the “understood query” object. The full chain measures **~6 seconds on CPU** — which tells you exactly what it’s for: offline and async enrichment (query-log mining, zero-result analysis, building suggestion dictionaries), not the live search path. That honesty matters: pipelines don’t make everything fast; they make the *structure* of your spend visible, so each stage can be placed where its latency belongs. A pipeline that gates inputs can also gate **outputs**. The newest piece of this stack (POST /v1/evaluate) grades a generated answer before it ships: deterministic validators first (valid JSON? truncated? do the citations match the evidence you passed?) in under a millisecond, then — for factual claims — the same calibrated decision model acts as a judge, answering bounded rubric questions against *your* evidence: supported? contradicted? complete? sufficient? In measured fixtures, a supported answer scored p(support) 0.809, an unsupported one 0.222 — and an answer carrying a prompt-injection attempt hidden in the evidence stayed at 0.248, because the judge treats evidence as untrusted data. A deterministic failure short-circuits the judge (no model time wasted on an answer that’s already dead), and missing evidence caps the verdict at “uncertain” — it can never silently pass. Which closes the loop: **decide → gate → generate → verify**, every stage on the same local box, every threshold a number you can defend. 1. **Split the jobs.** Decisions are read off one prefill (355 ms, deterministic); generation is the only part that should pay the decode loop. 2. **Calibration is the load-bearing property.** Gates built on calibrated probabilities (ECE 0.054) route ambiguity to humans instead of hiding it; gates built on prose confidence are decoration. 3. **The biggest speedup is subtraction.** Every gated-out input is a generation you never run; every routed input is a big-model call downgraded to a small-model call. 4. **Bundle and cache.** Many questions, one state, one request; stable prompt prefixes make repeats nearly free. 5. **Gate both directions.** Evaluate outputs with the same machinery — deterministic checks for structure, a calibrated judge for grounding. Everything above runs on one local CPU server with an OpenAI-compatible API — the decision endpoints, the chat models, the evaluator, and the prefix cache are one binary, and none of the tickets, screenshots, or forms in these measurements left the machine. *All latency and probability figures are measured outputs from the shipped* *v1.6.0 server* *(Apple-M4-class CPU), captured while writing this article; the methodology notes live on the project’s performance page.* [Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and…](https://blog.devgenius.io/stop-making-one-model-do-two-jobs-pipelines-of-inference-that-speed-up-both-decisions-and-8e21e886ba73) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.