Three Tiers, One of Them Tested

A developer built a three-tier model routing system for email extraction, using input length to select between nano, mini, and full models, with a quality gate and spend ledger. The nano tier is validated on 20 samples, while mini and full tiers are marked UNTESTED, and the system escalates to untested tiers when needed.

An email shows up with enough in it to create work for a recruiter. Names, a company, sometimes a role, sometimes a phone number. Often a forwarded chain with three dead signatures stuck to the bottom. A person reads that and makes a judgment. A system has to turn it into fields without pretending every line is equally clear. The expensive mistake is quiet. Route everything to the full reasoning model and the form looks fine, and you learn about the spend on the invoice. Route everything to the cheapest model and the weak guesses surface later, after a lookup came back empty or the operator stopped trusting the form. So I put a router in front of extraction. Size the request, pick the cheapest model that will do, check what comes back, write the decision down. First version called the full model for everything. Short notes. Long transcripts. Forwarded threads nobody had trimmed. The output was acceptable, which is the problem, because acceptable output is not evidence that anything is being watched. The extraction path already has work queued behind the model call. app/services/extraction intelligence.py takes the extracted fields and goes outward: customer database, shared mailbox search, referrer checks. That ordering is the reason the router sits where it does. A wrong email field turns a working lookup into an empty one, and by then the money is already spent. Spending it before anything downstream has had a chance to show the input was trivial is backwards. Three tiers, picked by input length. TextLengthThresholds in well shared/config/production config.py : nano takes 0 to 3,000 characters, mini takes 3,000 to 7,000, full takes anything over 7,000. The annotations beside those bounds are the part worth reading. Nano is marked validated. Mini and full are both marked UNTESTED, in caps, in the source. The file says why. The validation cohort was messages under 3,000 characters, twenty samples, dated 2025-10-31. The price table repeats it. Nano at \$0.05 / \$0.15 per million tokens is annotated validated on that same date. Mini at \$0.25 / \$0.75 and full at \$1.25 / \$3.75 both carry UNTESTED. Further down the same file, ProductionValidation records the result those annotations point at. Twenty samples, 100 percent success rate, gpt-5-nano cohort only. Twenty for twenty is a good sign and a small number. Run the bound and a perfect 20 of 20 is still consistent with a true success rate around 86 percent. That is not an argument against shipping nano. It is an argument against reading "100%" as a fact about the model rather than a fact about twenty emails. So escalation, which the whole design leans on, moves a request onto a tier nobody has measured on this workload. The router will happily do it. It just cannot tell you what happens next. The transitions matter more than the tier names. Drop a state from that path and it stops being a router. It becomes a lookup table with opinions. php flowchart TD request Request arrives -- complexity Complexity estimate complexity -- budget Budget check budget -- |under hard stop| choose Cheapest adequate model budget -- |over hard stop| block Emit overrun event choose -- quality Quality check quality -- |adequate| accept Accept result quality -- |weak result| escalate Escalate model tier escalate -- quality accept -- ledger Spend ledger update block -- ledger The cheap path is only safe because a quality gate sits behind it. QualityTargets , same file, sets a floor per use case. Email intake 0.85. Digest 0.90. Research enrichment 0.90. Batch processing 0.80, with a comment in the source saying speed was given priority over precision. Character count picks a starting tier. It has no idea whether a thin digest summary is good enough to put in front of a client, so the gate has to know what the output is for. The dollar ceilings sit beside the quality floors, one per context type. Email intake gets \$2.00, with a note that a typical extraction runs one to three API calls. Weekly digest generation gets \$5.00 for ten to twenty candidates. Bulk operations get \$10.00 for a hundred records and up. Enrichment through Apollo and Firecrawl gets \$3.00 for three to five calls. The header names the consumers, talentwell curator.py and vault conversation service.py , which is the sort of thing I want in a config file, because it tells the next person which blast radius they are editing. Those figures put the guardrails on a scale you can hold. The warning on a single email intake fires at \$1.60. 3. The trace is supposed to be the receipt Routing with no trace turns into argument. Somebody sees a bill or a bad extraction, asks why the system went that way, and the answer has to sit next to the workflow. Not in a spreadsheet a person maintains by hand. app/services/extraction workflow tracer.py holds an ExtractionStep with a plain lifecycle. start stamps a time. complete stamps another and computes duration ms . fail does the same and keeps the error message. What matters is what it hands back. python def to dict self - Dict str, Any : """Convert step to dictionary for JSON serialization.""" return { "step name": self.step name, "step type": self.step type, "status": self.status, "duration ms": self.duration ms, "fields extracted": self.fields extracted, "data sources": self.data sources, "confidence scores": self.confidence scores, "input summary": {k: f"<{type v . name } " for k, v in self.input data.items }, "output summary": {k: f"<{type v . name } " for k, v in self.output data.items }, "error message": self.error message, "timestamp": datetime.fromtimestamp self.start time .isoformat if self.start time else None } Two things in there are right. input summary and output summary keep the type name and drop the value, so nobody debugging a workflow reads a candidate's email. Deliberate, and it has held up. Status, duration, confidence and sources also come back together. That separates a slow full-model run that passed from a slow nano run that failed and escalated. Same wall clock. Different event entirely. Now read that dictionary for what is missing. No tier. No price. No quality score. No budget state, anywhere in the step or in the tracer that owns it. The routing decision leaves nothing behind. The budget events are real. They live somewhere else. well shared/cache/voit.py emits voit.budget warning at 80 percent of the ceiling and voit.budget overrun past it. There is a voit.budget overrun amount metric too. Here is the annoying part. Those events already carry model tier and input char count . The tier is right there, tagged onto the spend event, in a different subsystem with a different sink. What neither event carries is a request id, so there is no way to say that this overrun and that extraction are the same email. Which means the question I built this router to answer, which tier ran and what did it cost, cannot be answered from the trace. Duration and confidence come out of one system. Spend comes out of another. There is no key tying them to the same email. What is instrumented is the extraction. The routing decision, the thing that costs money, leaves no record I can query. The failure that ordering was supposed to catch turns up one step later. The customer database lookup takes a token from an OAuth proxy, then searches contacts by email, accounts by website domain, contacts by first and last name. No content from upstream and it returns an empty list. An error from upstream and it logs, then returns an empty list. Right call for the operator. The form keeps moving when a lookup misses. It also makes an empty result ambiguous. Maybe the contact genuinely is not there. Maybe the model pulled the wrong address off a forwarded chain and the lookup did exactly what it was asked with a value that was already garbage. The response is identical either way. The surrounding constants make the earlier decision visible. Service timeout is 10 seconds. Shared mailbox search looks back 365 days by default. Neither is unusual. Both mean a wrong field gets a genuine, patient, well-behaved search that was never going to find anything. That is the argument for scoring the extraction before it feeds enrichment, rather than bolting a fourth fallback search on behind it. Warning at 80 percent of the ceiling. Hard stop at 100. Carrying on past the second one would mean the system ignores its own runtime contract, so it stops. People feel that. A hard stop withholds an answer somebody wanted. Escalation makes a slow request slower. A ceiling set badly turns the warning into noise the operator learns to click past. All real. I would still rather have those happen where someone can see them than have spend drift quietly until the invoice lands. One temptation I keep turning down. A tidy wrapper that accepts text and returns fields would make every call site read beautifully while discarding input size, selected tier, applied threshold, quality score, budget state, escalation path, ledger write. Those are dependencies, not noise. Two changes. Neither is clever. The step dictionary gets the fields the router already has in hand at the moment it decides. Tier, threshold applied, measured price, quality score. The router is holding all four and drops them on the floor, so this is plumbing, not design. The VoIT budget events get a request id attached so a warning or an overrun can be joined to the workflow that caused it. Right now the two systems are describing the same email and have no way to know it. The tier validation is harder and it is not code. Mini and full say UNTESTED because nobody has run the cohort. Until somebody does, every escalation is a guess with a price attached. The 100 percent in that validation record describes twenty emails, all of which went to nano. Spend belongs in the request path because it changes what the system does, the same way validation does. I believed I had built that. I had built the half that decides.