Stop Making One Model Do Two Jobs: Pipelines of Inference That Speed Up Both Decisions and… Splitting an AI feature's decision step from its generation step into separate inference endpoints makes the decision roughly 2.6× faster and calibrated, according to measurements published by the author on a local CPU server running the shipped server v1.6.0. Reading a category off one prefill of a 4B model took 355 ms versus about 920 ms to generate the same category, and an exact repeat over the same choice set hit the prefix cache in 84 ms. A 2B calibrated decision model served at POST /v1/systemone reports an expected calibration error of 0.054, which the author says lets a probability gate (act at p ≥ 0.90, confirm at 0.50 ≤ p < 0.90, escalate below 0.50) skip the expensive generation entirely when the answer says so. Most “AI features” are secretly two steps — a decision and a generation. Splitting them across a decision endpoint and a chat endpoint makes the decision ~2.6× faster, makes it calibrated, and lets you skip the expensive generation entirely when the answer says so. All numbers below are measured on a local CPU server. Watch what your LLM actually does in a typical product feature: A support ticket arrives. The model reads it, figures out it’s a billing issue, decides it’s urgent — and then writes the reply. A search query comes in. The model works out whether it’s even a real query, what language it’s in, what the user meant — and then rewrites it. A RAG answer is drafted. Something decides whether it’s grounded enough to show — and then it ships. Every one of these is a decision followed by a generation . And in most stacks, one chat model does both jobs inside a single generation call: “think about the category, then write the answer.” That’s the slowest and least trustworthy way to do it, for two separate reasons: 1. The decision pays the full decode tax. Producing the word “billing” costs the same sequential, token-by-token decode loop as producing a paragraph. You wait for prose to learn one bit of information. 2. The decision is uncalibrated. When a chat model writes “I’m fairly confident this is billing,” that confidence is text , not a probability. You can’t threshold it, chart it, or defend it in a design review. The fix is an old systems idea applied to inference: pipeline the stages, and make each stage use the cheapest mechanism that does the job. A decision has a fixed answer space — you know the possible answers before you ask. So don’t make the model write the answer. Run the prompt through the model once the same “reading” pass that precedes any generation , then read the probability distribution over just your allowed answers directly off the logits. One prefill. Zero generated tokens. On a local inference server this is a dedicated endpoint. The friendly form takes plain-text choices: POST /v1/score { "model": "q35–4b", "query": "My card was charged twice this month.", "choices": "billing", "shipping", "technical support", "none of the above" } → { "decision": "billing", "probabilities": { "billing": 0.94, "shipping": 0.02, "technical support": 0.03, "none of the above": 0.01 } } The speedup is not subtle. Measured on the same 4B model, same M4-class CPU box: Asking the model to generate the category: ~920 ms. Reading it from one prefill: 355 ms . And because the prompt is structured stable-part-first, repeated decisions over the same choice set hit the prefix cache — an exact repeat lands in 84 ms . The decision stage of your pipeline just became an order of magnitude cheaper than the generation stage. That’s the point. For decisions you’ll wire into business logic, there’s a stronger tool: a model trained for deciding. POST /v1/systemone a 2B calibrated decision model takes a state plus typed questions — yes/no probabilities, choices with criteria, ordered scores — and answers several of them in one request, because the state is prefilled once and forked per question. Its probabilities are calibrated ECE 0.054 : when it says 0.9, it’s right about 90% of the time. That one property is what makes the next stage possible. Between the decision and the generation sits the most boring and most valuable component in the pipeline: an if statement. p ≥ 0.90 → act: run the generation on the routed path 0.50 ≤ p < 0.90 → confirm: proceed, but flag for review p < 0.50 → escalate: stop - a human sees it, with the probability attached This only works because the probabilities are calibrated — thresholding an uncalibrated score is theater. Here’s the gate with three real, unedited measurements from the shipped server v1.6.0 plotted on it: · A support screenshot showing a payment-error dialog was routed to the billing team at 0.9449 — the decision model read the words “card declined by issuer” inside the image . Above the act line: auto-route, no human needed. · A scanned claim form with a blank signature line scored 0.5598 on “are all required fields filled?” — squarely in the confirm band. Not rejected three of four fields are plainly filled , not auto-accepted either. Exactly right. · Keyboard-mash gibberish queries measure 0.78–0.88 on a gibberish gate — which is why the reject threshold sits at 0.90, not at a round number someone liked. Notice what the middle case means operationally: the pipeline’s “uncertain” outcomes are not failures. They’re the system correctly routing ambiguity to a human instead of hiding it inside fluent prose. Now the expensive step — the actual writing — runs only on the inputs that earned it , with a prompt specialized by the decision: POST /v1/chat/completions { "model": "q35–2b", "messages": { "role": "system", "content": "You are the BILLING support responder. …" }, { "role": "user", "content": "