# I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.

> Source: <https://dev.to/shanni/i-benchmarked-8-llms-for-a-niche-production-app-the-flagship-was-16x-the-cost-for-nothing-246e>
> Published: 2026-08-29 01:48:02+00:00

My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.

The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.

Generic benchmarks were useless to me. My acceptance criteria were:

Running my own corpus through the candidates produced findings no leaderboard would have surfaced:

`400 InvalidParameter: The value of the enable_thinking parameter is restricted to True`

), so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's *previous* generation was equally accurate at lower cost, which is exactly what you want in a fallback.

The eval's outputs — which models are allowed, in what order, at what price — live in one file. A `Route`

is a provider (endpoint + key) plus a model plus that model's list price:

``` js
const PRICE: Record<string, [number, number]> = {
  'small-fast':   [0.1, 0.4],   // USD per 1M tokens, in/out
  'mid-plus':     [0.4, 1.6],
  'mid-plus-prev':[0.5, 3.0],
  'flagship':     [2.5, 7.5],
}

const DEFAULT_CHAINS: Record<Tier, string[]> = {
  free: ['small-fast', 'legacy-plus'],
  paid: ['mid-plus', 'mid-plus-prev', 'flagship'],
}
```

Each tier gets an **ordered fallback chain**: the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with *the primary model on the backup account* — because when your account balance dies, every model on it dies together, and only a different key helps.

The subtle part of fallback chains isn't trying the next model — it's knowing *when* the next model helps at all. Every failure gets classified into one of three moves:

```
function classify(e: unknown): 'retry' | 'next' | 'fatal' {
  if (e instanceof LLMHttpError) {
    const { status, body } = e
    if (status === 401 || status === 403) return 'fatal'   // new model won't fix your key
    if (status >= 500) return 'retry'                       // transient, same route
    if (status === 429)
      return /RateQuota|rate limit/i.test(body) ? 'retry' : 'next'
    if (status === 400 || status === 404)
      return /model|not.?found|InvalidParameter/i.test(body) ? 'next' : 'fatal'
    return 'next'
  }
  return 'retry'  // network-layer: ECONNRESET, DNS, timeout
}
```

The one that bites people: **429 is two different errors wearing one status code.** Rate-limit throttling is transient — back off and retry the *same* model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing.

When the whole chain is exhausted, the app returns placeholder text with an `ok: false`

flag — and the flag exists because of a real trap: **never persist a fallback stub.** A paid user whose reading gets cached as "(placeholder)" sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. `ok`

gates the database write; failures stay ephemeral and self-heal on the next request.

Every business action (one reading = up to 7 parallel calls) emits a usage event, and each call's cost is computed against **the model that actually served it**, not the one you intended:

``` js
const intended = primaryModel(tier)
const fellBack = served.some((m) => m !== intended)
capture('llm_usage', {
  kind, tier, model: servedModels, primary_model: intended,
  fell_back: fellBack, input_tokens, output_tokens, cost_usd,
})
```

`fell_back: true`

is the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call (~3.7k in / 0.4k out): **$0.0021** on the paid-tier model, **$0.0005** on the free-tier one — so a two-call free reading lands near **$0.001**. Those aren't estimates; they're what the meter read.

`enable_thinking`

(or your provider's equivalent) is the biggest single cost lever`fell_back`

flag. Silent fallback is silent margin change.The app all this serves is [auspiceoracle.com](https://auspiceoracle.com/en) — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article (next in the series).