I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.

A developer benchmarked eight LLMs for a niche BaZi birth-chart app and found that the flagship model cost 5.8x more yet lost on accuracy. The developer built a routing layer with ordered fallback chains and error classification, revealing that generic benchmarks are useless for niche domains and that 429 errors can be two different failures.

My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found. The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less. Generic benchmarks were useless to me. My acceptance criteria were: Running my own corpus through the candidates produced findings no leaderboard would have surfaced: 400 InvalidParameter: The value of the enable thinking parameter is restricted to True , so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's previous generation was equally accurate at lower cost, which is exactly what you want in a fallback. The eval's outputs — which models are allowed, in what order, at what price — live in one file. A Route is a provider endpoint + key plus a model plus that model's list price: js const PRICE: Record<string, number, number = { 'small-fast': 0.1, 0.4 , // USD per 1M tokens, in/out 'mid-plus': 0.4, 1.6 , 'mid-plus-prev': 0.5, 3.0 , 'flagship': 2.5, 7.5 , } const DEFAULT CHAINS: Record<Tier, string = { free: 'small-fast', 'legacy-plus' , paid: 'mid-plus', 'mid-plus-prev', 'flagship' , } Each tier gets an ordered fallback chain : the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with the primary model on the backup account — because when your account balance dies, every model on it dies together, and only a different key helps. The subtle part of fallback chains isn't trying the next model — it's knowing when the next model helps at all. Every failure gets classified into one of three moves: function classify e: unknown : 'retry' | 'next' | 'fatal' { if e instanceof LLMHttpError { const { status, body } = e if status === 401 || status === 403 return 'fatal' // new model won't fix your key if status = 500 return 'retry' // transient, same route if status === 429 return /RateQuota|rate limit/i.test body ? 'retry' : 'next' if status === 400 || status === 404 return /model|not.?found|InvalidParameter/i.test body ? 'next' : 'fatal' return 'next' } return 'retry' // network-layer: ECONNRESET, DNS, timeout } The one that bites people: 429 is two different errors wearing one status code. Rate-limit throttling is transient — back off and retry the same model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing. When the whole chain is exhausted, the app returns placeholder text with an ok: false flag — and the flag exists because of a real trap: never persist a fallback stub. A paid user whose reading gets cached as " placeholder " sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. ok gates the database write; failures stay ephemeral and self-heal on the next request. Every business action one reading = up to 7 parallel calls emits a usage event, and each call's cost is computed against the model that actually served it , not the one you intended: js const intended = primaryModel tier const fellBack = served.some m = m == intended capture 'llm usage', { kind, tier, model: servedModels, primary model: intended, fell back: fellBack, input tokens, output tokens, cost usd, } fell back: true is the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call ~3.7k in / 0.4k out : $0.0021 on the paid-tier model, $0.0005 on the free-tier one — so a two-call free reading lands near $0.001 . Those aren't estimates; they're what the meter read. enable thinking or your provider's equivalent is the biggest single cost lever fell back flag. Silent fallback is silent margin change.The app all this serves is auspiceoracle.com https://auspiceoracle.com/en — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article next in the series .