# Benchmarking Gemini 2.5 Flash vs 3.1 Flash-Lite vs Gemma 4 with LLM judge (Claude Fable 5)

> Source: <https://dev.to/io_56a2cd4aff31d/benchmarking-gemini-25-flash-vs-31-flash-lite-vs-gemma-4-with-llm-judge-claude-fable-5-356>
> Published: 2026-07-18 05:17:12+00:00

*Cross-posted from the IO reader blog, where the full version includes all 36 unedited transcripts side by side.*

With `gemini-2.5-flash`

scheduled to retire on October 16, we needed to audition replacements for a production workload that generic leaderboards say nothing about: twelve distinct "reading lens" prompts — literary close reading, Socratic questioning, EN→KO translation under a hard register constraint, vocabulary coaching — that our iOS reading app serves through Firebase AI Logic.

So we ran the audition the unfashionable way: one adversarial stimulus, every production prompt verbatim, three models, and a human reading + LLM judge (Claude Fable 5).

**Stimulus.** The closing paragraph of Melville's "Whiteness of the Whale" chapter — chosen to break things: archaic vocabulary (*subtile*, *palsied*, *charnel-house*), one enormous periodic sentence, nineteenth-century natural philosophy, and a live metaphysical argument. If a model holds this paragraph steady through twelve different lenses, it holds almost anything a reader will meet.

**Models.**

`gemini-2.5-flash`

— the incumbent engine`gemini-3.1-flash-lite`

— its smaller, newer successor candidate`gemma-4-26b-a4b-it`

— open-weights reasoning contender**Protocol.** Production prompts verbatim as a single user turn; temperature 0.4; `thinkingBudget: 0`

on the Gemini models; reasoning left on for Gemma (its serving default). One generation per cell — 12 × 3 = 36 calls — with latency and token counts taken from the API's own `usageMetadata`

.

| Model | Runs | Median | P90 | $/1M in·out | Measured billed ¢/reply |
|---|---|---|---|---|---|
| gemini-2.5-flash | 12/12 | 1.7 s | 2.8 s | 0.30 · 2.50 | 0.057¢ |
| gemini-3.1-flash-lite | 12/12 | 2.2 s | 3.0 s | 0.25 · 1.50 | 0.038¢ |
| gemma-4-26b-a4b-it | 12/12 | 49.6 s* | 126.8 s* | 0.15 · 0.60 | 0.150¢ |

**Shared research endpoint with reasoning enabled — compare Gemma's quality here, not its speed. "Billed ¢/reply" is computed from each model's measured median prompt and output tokens at list price, including hidden reasoning tokens, which are billed as output.*

Word caps, fixed output structures, question-only constraints, and the Korean formal-register rule (해라체, never conversational endings) all held across every generation. A prompt contract, it turns out, is a property of the contract — not of the model behind it. That's what makes engines swappable without the product changing shape.

Gemini 3.1 Flash-Lite — 40% cheaper on output than the incumbent — kept producing the sharpest specific observations. It was the only model to name Melville's "colourless, all-colour" paradox outright, and it described the prose as "a rhetorical trap" whose cumulative clauses leave the reader "as intellectually blinded as the wretched infidel."

The cleanest contrast: one of our lenses must end every response with a single unsettling question. Same paragraph, same instructions, three minds —

"Does the terror of whiteness truly stem from its objective qualities, or from humanity's ingrained need to project meaning and warmth onto a fundamentally indifferent cosmos?" — gemini-2.5-flash

"If color is a lie, is truth merely the blindness of the void?" — gemini-3.1-flash-lite

"If the truth is a shroud, is the only way to remain sane to remain deceived?" — gemma-4-26b

The incumbent is thorough and slightly padded. The understudy is compressed and exact. The open-weights model writes like it's been saving that sentence all week.

Gemma 4 26B has the **lowest list price** on the bench and the **highest measured cost per reply** — roughly 4× Flash-Lite. The gap is hidden reasoning: with its default thinking mode on, it deliberated for 5,000–10,000 unseen tokens per answer, and reasoning tokens bill as output.

If you're comparing models on list price, check `usageMetadata.thoughtsTokenCount`

first. For self-hosted batch pipelines Gemma's quality is real and the economics change entirely; for interactive serving through a metered API, list-price comparisons are fiction.

One generation per cell — a structured qualitative reading, not a statistical claim. Judging wasn't blind. Gemma's latency reflects a shared research endpoint, not managed serving. And one passage is one genre; the next run adds an analytical text, a modern novel, and poetry.

All 36 outputs — unedited, side by side, including both Korean translations and the answers we found weakest — are in the full research note:

**→ ioreader.app/blog/the-whiteness-test**

*Disclosure: IO is our app; the benchmark used its production prompts. Curious how others are accounting for thinking-token billing when comparing reasoning models — comments welcome.*