{"slug": "i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and", "title": "I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.", "summary": "A developer benchmarked eight LLMs for a niche BaZi birth-chart app and found that the flagship model cost 5.8x more yet lost on accuracy. The developer built a routing layer with ordered fallback chains and error classification, revealing that generic benchmarks are useless for niche domains and that 429 errors can be two different failures.", "body_md": "My app generates personalized readings for BaZi — Chinese \"Four Pillars\" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.\n\nThe results generalize to any \"LLM in a niche domain\" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.\n\nGeneric benchmarks were useless to me. My acceptance criteria were:\n\nRunning my own corpus through the candidates produced findings no leaderboard would have surfaced:\n\n`400 InvalidParameter: The value of the enable_thinking parameter is restricted to True`\n\n), so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's *previous* generation was equally accurate at lower cost, which is exactly what you want in a fallback.\n\nThe eval's outputs — which models are allowed, in what order, at what price — live in one file. A `Route`\n\nis a provider (endpoint + key) plus a model plus that model's list price:\n\n``` js\nconst PRICE: Record<string, [number, number]> = {\n  'small-fast':   [0.1, 0.4],   // USD per 1M tokens, in/out\n  'mid-plus':     [0.4, 1.6],\n  'mid-plus-prev':[0.5, 3.0],\n  'flagship':     [2.5, 7.5],\n}\n\nconst DEFAULT_CHAINS: Record<Tier, string[]> = {\n  free: ['small-fast', 'legacy-plus'],\n  paid: ['mid-plus', 'mid-plus-prev', 'flagship'],\n}\n```\n\nEach tier gets an **ordered fallback chain**: the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with *the primary model on the backup account* — because when your account balance dies, every model on it dies together, and only a different key helps.\n\nThe subtle part of fallback chains isn't trying the next model — it's knowing *when* the next model helps at all. Every failure gets classified into one of three moves:\n\n```\nfunction classify(e: unknown): 'retry' | 'next' | 'fatal' {\n  if (e instanceof LLMHttpError) {\n    const { status, body } = e\n    if (status === 401 || status === 403) return 'fatal'   // new model won't fix your key\n    if (status >= 500) return 'retry'                       // transient, same route\n    if (status === 429)\n      return /RateQuota|rate limit/i.test(body) ? 'retry' : 'next'\n    if (status === 400 || status === 404)\n      return /model|not.?found|InvalidParameter/i.test(body) ? 'next' : 'fatal'\n    return 'next'\n  }\n  return 'retry'  // network-layer: ECONNRESET, DNS, timeout\n}\n```\n\nThe one that bites people: **429 is two different errors wearing one status code.** Rate-limit throttling is transient — back off and retry the *same* model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing.\n\nWhen the whole chain is exhausted, the app returns placeholder text with an `ok: false`\n\nflag — and the flag exists because of a real trap: **never persist a fallback stub.** A paid user whose reading gets cached as \"(placeholder)\" sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. `ok`\n\ngates the database write; failures stay ephemeral and self-heal on the next request.\n\nEvery business action (one reading = up to 7 parallel calls) emits a usage event, and each call's cost is computed against **the model that actually served it**, not the one you intended:\n\n``` js\nconst intended = primaryModel(tier)\nconst fellBack = served.some((m) => m !== intended)\ncapture('llm_usage', {\n  kind, tier, model: servedModels, primary_model: intended,\n  fell_back: fellBack, input_tokens, output_tokens, cost_usd,\n})\n```\n\n`fell_back: true`\n\nis the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call (~3.7k in / 0.4k out): **$0.0021** on the paid-tier model, **$0.0005** on the free-tier one — so a two-call free reading lands near **$0.001**. Those aren't estimates; they're what the meter read.\n\n`enable_thinking`\n\n(or your provider's equivalent) is the biggest single cost lever`fell_back`\n\nflag. Silent fallback is silent margin change.The app all this serves is [auspiceoracle.com](https://auspiceoracle.com/en) — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article (next in the series).", "url": "https://wpnews.pro/news/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and", "canonical_source": "https://dev.to/shanni/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-was-16x-the-cost-for-nothing-246e", "published_at": "2026-08-29 01:48:02+00:00", "updated_at": "2026-08-29 02:18:37.148116+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and", "markdown": "https://wpnews.pro/news/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and.md", "text": "https://wpnews.pro/news/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-cost-5-8x-more-and.jsonld"}}