{"slug": "three-tiers-one-of-them-tested", "title": "Three Tiers, One of Them Tested", "summary": "A developer built a three-tier model routing system for email extraction, using input length to select between nano, mini, and full models, with a quality gate and spend ledger. The nano tier is validated on 20 samples, while mini and full tiers are marked UNTESTED, and the system escalates to untested tiers when needed.", "body_md": "An email shows up with enough in it to create work for a recruiter. Names, a company, sometimes a role, sometimes a phone number. Often a forwarded chain with three dead signatures stuck to the bottom. A person reads that and makes a judgment. A system has to turn it into fields without pretending every line is equally clear.\n\nThe expensive mistake is quiet. Route everything to the full reasoning model and the form looks fine, and you learn about the spend on the invoice. Route everything to the cheapest model and the weak guesses surface later, after a lookup came back empty or the operator stopped trusting the form. So I put a router in front of extraction. Size the request, pick the cheapest model that will do, check what comes back, write the decision down.\n\nFirst version called the full model for everything. Short notes. Long transcripts. Forwarded threads nobody had trimmed. The output was acceptable, which is the problem, because acceptable output is not evidence that anything is being watched.\n\nThe extraction path already has work queued behind the model call. `app/services/extraction_intelligence.py`\n\ntakes the extracted fields and goes outward: customer database, shared mailbox search, referrer checks. That ordering is the reason the router sits where it does. A wrong email field turns a working lookup into an empty one, and by then the money is already spent. Spending it before anything downstream has had a chance to show the input was trivial is backwards.\n\nThree tiers, picked by input length. `TextLengthThresholds`\n\nin `well_shared/config/production_config.py`\n\n: nano takes 0 to 3,000 characters, mini takes 3,000 to 7,000, full takes anything over 7,000.\n\nThe annotations beside those bounds are the part worth reading. Nano is marked validated. Mini and full are both marked UNTESTED, in caps, in the source. The file says why. The validation cohort was messages under 3,000 characters, twenty samples, dated 2025-10-31. The price table repeats it. Nano at \\$0.05 / \\$0.15 per million tokens is annotated validated on that same date. Mini at \\$0.25 / \\$0.75 and full at \\$1.25 / \\$3.75 both carry UNTESTED.\n\nFurther down the same file, `ProductionValidation`\n\nrecords the result those annotations point at. Twenty samples, 100 percent success rate, gpt-5-nano cohort only.\n\nTwenty for twenty is a good sign and a small number. Run the bound and a perfect 20 of 20 is still consistent with a true success rate around 86 percent. That is not an argument against shipping nano. It is an argument against reading \"100%\" as a fact about the model rather than a fact about twenty emails.\n\nSo escalation, which the whole design leans on, moves a request onto a tier nobody has measured on this workload. The router will happily do it. It just cannot tell you what happens next.\n\nThe transitions matter more than the tier names. Drop a state from that path and it stops being a router. It becomes a lookup table with opinions.\n\n``` php\nflowchart TD\nrequest[Request arrives] --> complexity[Complexity estimate]\ncomplexity --> budget[Budget check]\nbudget -->|under hard stop| choose[Cheapest adequate model]\nbudget -->|over hard stop| block[Emit overrun event]\nchoose --> quality[Quality check]\nquality -->|adequate| accept[Accept result]\nquality -->|weak result| escalate[Escalate model tier]\nescalate --> quality\naccept --> ledger[Spend ledger update]\nblock --> ledger```\n\nThe cheap path is only safe because a quality gate sits behind it. `QualityTargets`, same file, sets a floor per use case. Email intake 0.85. Digest 0.90. Research enrichment 0.90. Batch processing 0.80, with a comment in the source saying speed was given priority over precision.\n\nCharacter count picks a starting tier. It has no idea whether a thin digest summary is good enough to put in front of a client, so the gate has to know what the output is for.\n\nThe dollar ceilings sit beside the quality floors, one per context type. Email intake gets \\$2.00, with a note that a typical extraction runs one to three API calls. Weekly digest generation gets \\$5.00 for ten to twenty candidates. Bulk operations get \\$10.00 for a hundred records and up. Enrichment through Apollo and Firecrawl gets \\$3.00 for three to five calls. The header names the consumers, `talentwell_curator.py` and `vault_conversation_service.py`, which is the sort of thing I want in a config file, because it tells the next person which blast radius they are editing.\n\nThose figures put the guardrails on a scale you can hold. The warning on a single email intake fires at \\$1.60.\n\n## 3. The trace is supposed to be the receipt\n\nRouting with no trace turns into argument. Somebody sees a bill or a bad extraction, asks why the system went that way, and the answer has to sit next to the workflow. Not in a spreadsheet a person maintains by hand.\n\n`app/services/extraction_workflow_tracer.py` holds an `ExtractionStep` with a plain lifecycle. `start()` stamps a time. `complete()` stamps another and computes `duration_ms`. `fail()` does the same and keeps the error message. What matters is what it hands back.\n\n``` python\ndef to_dict(self) -> Dict[str, Any]:\n    \"\"\"Convert step to dictionary for JSON serialization.\"\"\"\n    return {\n        \"step_name\": self.step_name,\n        \"step_type\": self.step_type,\n        \"status\": self.status,\n        \"duration_ms\": self.duration_ms,\n        \"fields_extracted\": self.fields_extracted,\n        \"data_sources\": self.data_sources,\n        \"confidence_scores\": self.confidence_scores,\n        \"input_summary\": {k: f\"<{type(v).__name__}>\" for k, v in self.input_data.items()},\n        \"output_summary\": {k: f\"<{type(v).__name__}>\" for k, v in self.output_data.items()},\n        \"error_message\": self.error_message,\n        \"timestamp\": datetime.fromtimestamp(self.start_time).isoformat() if self.start_time else None\n    }\n```\n\nTwo things in there are right. `input_summary`\n\nand `output_summary`\n\nkeep the type name and drop the value, so nobody debugging a workflow reads a candidate's email. Deliberate, and it has held up. Status, duration, confidence and sources also come back together. That separates a slow full-model run that passed from a slow nano run that failed and escalated. Same wall clock. Different event entirely.\n\nNow read that dictionary for what is missing.\n\nNo tier. No price. No quality score. No budget state, anywhere in the step or in the tracer that owns it. The routing decision leaves nothing behind.\n\nThe budget events are real. They live somewhere else. `well_shared/cache/voit.py`\n\nemits `voit.budget_warning`\n\nat 80 percent of the ceiling and `voit.budget_overrun`\n\npast it. There is a `voit.budget_overrun_amount`\n\nmetric too.\n\nHere is the annoying part. Those events already carry `model_tier`\n\nand `input_char_count`\n\n. The tier is right there, tagged onto the spend event, in a different subsystem with a different sink. What neither event carries is a request id, so there is no way to say that this overrun and that extraction are the same email.\n\nWhich means the question I built this router to answer, which tier ran and what did it cost, cannot be answered from the trace. Duration and confidence come out of one system. Spend comes out of another. There is no key tying them to the same email.\n\nWhat is instrumented is the extraction. The routing decision, the thing that costs money, leaves no record I can query.\n\nThe failure that ordering was supposed to catch turns up one step later. The customer database lookup takes a token from an OAuth proxy, then searches contacts by email, accounts by website domain, contacts by first and last name. No content from upstream and it returns an empty list. An error from upstream and it logs, then returns an empty list.\n\nRight call for the operator. The form keeps moving when a lookup misses.\n\nIt also makes an empty result ambiguous. Maybe the contact genuinely is not there. Maybe the model pulled the wrong address off a forwarded chain and the lookup did exactly what it was asked with a value that was already garbage. The response is identical either way.\n\nThe surrounding constants make the earlier decision visible. Service timeout is 10 seconds. Shared mailbox search looks back 365 days by default. Neither is unusual. Both mean a wrong field gets a genuine, patient, well-behaved search that was never going to find anything. That is the argument for scoring the extraction before it feeds enrichment, rather than bolting a fourth fallback search on behind it.\n\nWarning at 80 percent of the ceiling. Hard stop at 100. Carrying on past the second one would mean the system ignores its own runtime contract, so it stops.\n\nPeople feel that. A hard stop withholds an answer somebody wanted. Escalation makes a slow request slower. A ceiling set badly turns the warning into noise the operator learns to click past. All real. I would still rather have those happen where someone can see them than have spend drift quietly until the invoice lands.\n\nOne temptation I keep turning down. A tidy wrapper that accepts text and returns fields would make every call site read beautifully while discarding input size, selected tier, applied threshold, quality score, budget state, escalation path, ledger write. Those are dependencies, not noise.\n\nTwo changes. Neither is clever.\n\nThe step dictionary gets the fields the router already has in hand at the moment it decides. Tier, threshold applied, measured price, quality score. The router is holding all four and drops them on the floor, so this is plumbing, not design.\n\nThe VoIT budget events get a request id attached so a warning or an overrun can be joined to the workflow that caused it. Right now the two systems are describing the same email and have no way to know it.\n\nThe tier validation is harder and it is not code. Mini and full say UNTESTED because nobody has run the cohort. Until somebody does, every escalation is a guess with a price attached. The 100 percent in that validation record describes twenty emails, all of which went to nano.\n\nSpend belongs in the request path because it changes what the system does, the same way validation does. I believed I had built that. I had built the half that decides.", "url": "https://wpnews.pro/news/three-tiers-one-of-them-tested", "canonical_source": "https://dev.to/romiteld/the-router-picks-a-tier-and-the-trace-never-says-which-2c3h", "published_at": "2026-08-23 11:16:11+00:00", "updated_at": "2026-08-23 11:43:46.848586+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "mlops", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/three-tiers-one-of-them-tested", "markdown": "https://wpnews.pro/news/three-tiers-one-of-them-tested.md", "text": "https://wpnews.pro/news/three-tiers-one-of-them-tested.txt", "jsonld": "https://wpnews.pro/news/three-tiers-one-of-them-tested.jsonld"}}