{"slug": "benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude", "title": "Benchmarking Gemini 2.5 Flash vs 3.1 Flash-Lite vs Gemma 4 with LLM judge (Claude Fable 5)", "summary": "A developer benchmarked Gemini 2.5 Flash, Gemini 3.1 Flash-Lite, and Gemma 4 26B on 12 production prompts for an iOS reading app, using Melville's 'Whiteness of the Whale' as a stress test. Gemini 3.1 Flash-Lite was the cheapest and produced the sharpest observations, while Gemma 4 had the highest cost per reply due to hidden reasoning tokens.", "body_md": "*Cross-posted from the IO reader blog, where the full version includes all 36 unedited transcripts side by side.*\n\nWith `gemini-2.5-flash`\n\nscheduled to retire on October 16, we needed to audition replacements for a production workload that generic leaderboards say nothing about: twelve distinct \"reading lens\" prompts — literary close reading, Socratic questioning, EN→KO translation under a hard register constraint, vocabulary coaching — that our iOS reading app serves through Firebase AI Logic.\n\nSo we ran the audition the unfashionable way: one adversarial stimulus, every production prompt verbatim, three models, and a human reading + LLM judge (Claude Fable 5).\n\n**Stimulus.** The closing paragraph of Melville's \"Whiteness of the Whale\" chapter — chosen to break things: archaic vocabulary (*subtile*, *palsied*, *charnel-house*), one enormous periodic sentence, nineteenth-century natural philosophy, and a live metaphysical argument. If a model holds this paragraph steady through twelve different lenses, it holds almost anything a reader will meet.\n\n**Models.**\n\n`gemini-2.5-flash`\n\n— the incumbent engine`gemini-3.1-flash-lite`\n\n— its smaller, newer successor candidate`gemma-4-26b-a4b-it`\n\n— open-weights reasoning contender**Protocol.** Production prompts verbatim as a single user turn; temperature 0.4; `thinkingBudget: 0`\n\non the Gemini models; reasoning left on for Gemma (its serving default). One generation per cell — 12 × 3 = 36 calls — with latency and token counts taken from the API's own `usageMetadata`\n\n.\n\n| Model | Runs | Median | P90 | $/1M in·out | Measured billed ¢/reply |\n|---|---|---|---|---|---|\n| gemini-2.5-flash | 12/12 | 1.7 s | 2.8 s | 0.30 · 2.50 | 0.057¢ |\n| gemini-3.1-flash-lite | 12/12 | 2.2 s | 3.0 s | 0.25 · 1.50 | 0.038¢ |\n| gemma-4-26b-a4b-it | 12/12 | 49.6 s* | 126.8 s* | 0.15 · 0.60 | 0.150¢ |\n\n**Shared research endpoint with reasoning enabled — compare Gemma's quality here, not its speed. \"Billed ¢/reply\" is computed from each model's measured median prompt and output tokens at list price, including hidden reasoning tokens, which are billed as output.*\n\nWord caps, fixed output structures, question-only constraints, and the Korean formal-register rule (해라체, never conversational endings) all held across every generation. A prompt contract, it turns out, is a property of the contract — not of the model behind it. That's what makes engines swappable without the product changing shape.\n\nGemini 3.1 Flash-Lite — 40% cheaper on output than the incumbent — kept producing the sharpest specific observations. It was the only model to name Melville's \"colourless, all-colour\" paradox outright, and it described the prose as \"a rhetorical trap\" whose cumulative clauses leave the reader \"as intellectually blinded as the wretched infidel.\"\n\nThe cleanest contrast: one of our lenses must end every response with a single unsettling question. Same paragraph, same instructions, three minds —\n\n\"Does the terror of whiteness truly stem from its objective qualities, or from humanity's ingrained need to project meaning and warmth onto a fundamentally indifferent cosmos?\" — gemini-2.5-flash\n\n\"If color is a lie, is truth merely the blindness of the void?\" — gemini-3.1-flash-lite\n\n\"If the truth is a shroud, is the only way to remain sane to remain deceived?\" — gemma-4-26b\n\nThe incumbent is thorough and slightly padded. The understudy is compressed and exact. The open-weights model writes like it's been saving that sentence all week.\n\nGemma 4 26B has the **lowest list price** on the bench and the **highest measured cost per reply** — roughly 4× Flash-Lite. The gap is hidden reasoning: with its default thinking mode on, it deliberated for 5,000–10,000 unseen tokens per answer, and reasoning tokens bill as output.\n\nIf you're comparing models on list price, check `usageMetadata.thoughtsTokenCount`\n\nfirst. For self-hosted batch pipelines Gemma's quality is real and the economics change entirely; for interactive serving through a metered API, list-price comparisons are fiction.\n\nOne generation per cell — a structured qualitative reading, not a statistical claim. Judging wasn't blind. Gemma's latency reflects a shared research endpoint, not managed serving. And one passage is one genre; the next run adds an analytical text, a modern novel, and poetry.\n\nAll 36 outputs — unedited, side by side, including both Korean translations and the answers we found weakest — are in the full research note:\n\n**→ ioreader.app/blog/the-whiteness-test**\n\n*Disclosure: IO is our app; the benchmark used its production prompts. Curious how others are accounting for thinking-token billing when comparing reasoning models — comments welcome.*", "url": "https://wpnews.pro/news/benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude", "canonical_source": "https://dev.to/io_56a2cd4aff31d/benchmarking-gemini-25-flash-vs-31-flash-lite-vs-gemma-4-with-llm-judge-claude-fable-5-356", "published_at": "2026-07-18 05:17:12+00:00", "updated_at": "2026-07-18 05:32:01.399150+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "developer-tools"], "entities": ["Gemini 2.5 Flash", "Gemini 3.1 Flash-Lite", "Gemma 4 26B", "Claude Fable 5", "Firebase AI Logic", "Melville"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude", "markdown": "https://wpnews.pro/news/benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude.md", "text": "https://wpnews.pro/news/benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude.txt", "jsonld": "https://wpnews.pro/news/benchmarking-gemini-2-5-flash-vs-3-1-flash-lite-vs-gemma-4-with-llm-judge-claude.jsonld"}}