Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and…

Most “AI features” are secretly two steps — a decision and a generation. Splitting them across a decision endpoint and a chat endpoint makes the decision ~2.6× faster, makes it calibrated, and lets you skip the expensive generation entirely when the answer says so. All numbers below are measured on a local CPU server.

Watch what your LLM actually does in a typical product feature:

A support ticket arrives. The model reads it, figures out it’s a billing issue, decides it’s urgent — and then writes the reply. A search query comes in. The model works out whether it’s even a real query, what language it’s in, what the user meant — and then rewrites it. A RAG answer is drafted. Something decides whether it’s grounded enough to show — and then it ships.

Every one of these is a decision followed by a generation. And in most stacks, one chat model does both jobs inside a single generation call: “think about the category, then write the answer.”

That’s the slowest and least trustworthy way to do it, for two separate reasons:

1. The decision pays the full decode tax. Producing the word “billing” costs the same sequential, token-by-token decode loop as producing a paragraph. You wait for prose to learn one bit of information.

2. The decision is uncalibrated. When a chat model writes “I’m fairly confident this is billing,” that confidence is text, not a probability. You can’t threshold it, chart it, or defend it in a design review.

The fix is an old systems idea applied to inference: pipeline the stages, and make each stage use the cheapest mechanism that does the job.

A decision has a fixed answer space — you know the possible answers before you ask. So don’t make the model write the answer. Run the prompt through the model once (the same “reading” pass that precedes any generation), then read the probability distribution over just your allowed answers directly off the logits. One prefill. Zero generated tokens.

On a local inference server this is a dedicated endpoint. The friendly form takes plain-text choices:

POST /v1/score { "model": "q35–4b", "query": "My card was charged twice this month.", "choices": ["billing", "shipping", "technical support", "none of the above"] }  → { "decision": "billing", "probabilities": { "billing": 0.94, "shipping": 0.02, "technical support": 0.03, "none of the above": 0.01 } }

The speedup is not subtle. Measured on the same 4B model, same M4-class CPU box:

Asking the model to generate the category: ~920 ms. Reading it from one prefill: 355 ms. And because the prompt is structured stable-part-first, repeated decisions over the same choice set hit the prefix cache — an exact repeat lands in 84 ms. The decision stage of your pipeline just became an order of magnitude cheaper than the generation stage. That’s the point.

For decisions you’ll wire into business logic, there’s a stronger tool: a model trained for deciding. POST /v1/systemone (a 2B calibrated decision model) takes a state plus typed questions — yes/no probabilities, choices with criteria, ordered scores — and answers several of them in one request, because the state is prefilled once and forked per question. Its probabilities are calibrated (ECE 0.054): when it says 0.9, it’s right about 90% of the time. That one property is what makes the next stage possible.

Between the decision and the generation sits the most boring and most valuable component in the pipeline: an if statement.

p ≥ 0.90 → act: run the generation on the routed path 0.50 ≤ p < 0.90 → confirm: proceed, but flag for review p < 0.50 → escalate: stop - a human sees it, with the probability attached

This only works because the probabilities are calibrated — thresholding an uncalibrated score is theater. Here’s the gate with three real, unedited measurements from the shipped server (v1.6.0) plotted on it:

· A support screenshot showing a payment-error dialog was routed to the billing team at 0.9449 — the decision model read the words “card declined by issuer” inside the image. Above the act line: auto-route, no human needed.

· A scanned claim form with a blank signature line scored 0.5598 on “are all required fields filled?” — squarely in the confirm band. Not rejected (three of four fields are plainly filled), not auto-accepted either. Exactly right.

· Keyboard-mash gibberish queries measure 0.78–0.88 on a gibberish gate — which is why the reject threshold sits at 0.90, not at a round number someone liked.

Notice what the middle case means operationally: the pipeline’s “uncertain” outcomes are not failures. They’re the system correctly routing ambiguity to a human instead of hiding it inside fluent prose.

Now the expensive step — the actual writing — runs only on the inputs that earned it, with a prompt specialized by the decision:

POST /v1/chat/completions { "model": "q35–2b", "messages": [ { "role": "system", "content": "You are the BILLING support responder. …" }, { "role": "user", "content": "<the ticket>" } ] }

This is where the pipeline’s real economics show up. The per-call speedup on the decision (2.6×) is nice; the structural speedup is bigger: every input the gate stops is a generation you never run. If 30% of a queue gets escalated, deduplicated, or rejected at decision time, that’s 30% of your most expensive compute deleted — not optimized, deleted. The decision stage that enables it costs 355 ms a call.

There are three more structural wins hiding in the same shape:

· Right-size the generator. Once routing is explicit, easy paths go to a 2B model and only the hard path pays for a big one. The decision model is effectively a traffic controller for model selection.

· Batch the gates. Need five checks on one input? Ask all five questions in one decision request — the state is prefilled once, each extra question is nearly free. (Measured: a three-question request over a screenshot took the same ~12 s as a two-question one; on text, extra questions cost milliseconds.)

· Cache the repeats. Decision prompts are highly repetitive by construction (same instruction, same choices, varying input). That’s the prefix cache’s best case — and it’s why the 84 ms number exists.

Here’s a production-shaped example — the classic search pipeline (spell-correct → classify → expand), rebuilt as decide-then-generate:

Stage 1 (one decision request, three questions): is this query gibberish? what language? what intent? — the gibberish gate rejects junk at p ≥ 0.90 before anything else spends time on it.

Stage 2 (generation, small model): correct the spelling.

Stage 3 (generation, small model): expand synonyms, assemble the “understood query” object.

The full chain measures ~6 seconds on CPU — which tells you exactly what it’s for: offline and async enrichment (query-log mining, zero-result analysis, building suggestion dictionaries), not the live search path. That honesty matters: pipelines don’t make everything fast; they make the structure of your spend visible, so each stage can be placed where its latency belongs.

A pipeline that gates inputs can also gate outputs. The newest piece of this stack (POST /v1/evaluate) grades a generated answer before it ships: deterministic validators first (valid JSON? truncated? do the citations match the evidence you passed?) in under a millisecond, then — for factual claims — the same calibrated decision model acts as a judge, answering bounded rubric questions against your evidence: supported? contradicted? complete? sufficient?

In measured fixtures, a supported answer scored p(support) 0.809, an unsupported one 0.222 — and an answer carrying a prompt-injection attempt hidden in the evidence stayed at 0.248, because the judge treats evidence as untrusted data. A deterministic failure short-circuits the judge (no model time wasted on an answer that’s already dead), and missing evidence caps the verdict at “uncertain” — it can never silently pass.

Which closes the loop: decide → gate → generate → verify, every stage on the same local box, every threshold a number you can defend.

1. Split the jobs. Decisions are read off one prefill (355 ms, deterministic); generation is the only part that should pay the decode loop.

2. Calibration is the load-bearing property. Gates built on calibrated probabilities (ECE 0.054) route ambiguity to humans instead of hiding it; gates built on prose confidence are decoration.

3. The biggest speedup is subtraction. Every gated-out input is a generation you never run; every routed input is a big-model call downgraded to a small-model call.

4. Bundle and cache. Many questions, one state, one request; stable prompt prefixes make repeats nearly free.

5. Gate both directions. Evaluate outputs with the same machinery — deterministic checks for structure, a calibrated judge for grounding.

Everything above runs on one local CPU server with an OpenAI-compatible API — the decision endpoints, the chat models, the evaluator, and the prefix cache are one binary, and none of the tickets, screenshots, or forms in these measurements left the machine.

All latency and probability figures are measured outputs from the shipped v1.6.0 server (Apple-M4-class CPU), captured while writing this article; the methodology notes live on the project’s performance page.

Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and… was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.