Every week, a new LLM drops with a headline score that makes previous models look obsolete. MMLU, HumanEval, GPQA — these benchmarks test PhD-level knowledge and competitive programming. They’re impressive. They’re also irrelevant to most of us.
The average user doesn’t need a model that can ace a quantum mechanics exam. They need one that follows instructions reliably, outputs valid JSON when asked, calls the right tool at the right time, and doesn’t hallucinate basic facts. Most benchmarks don’t measure any of that.
So I built a tool that does.
I wrote a Go CLI that hits OpenRouter’s API across four suites:
┌─────────────────────────────────────────────────────┐│ Suite │ What It Measures │├────────────────┼────────────────────────────────────┤│ Accuracy │ Knowledge + math (15 tasks) ││ Instruction │ Deterministic follow-through (12) ││ Tools │ Function calling (BFCL-style, 12) ││ JSON │ Schema compliance (5 tasks) │└────────────────┴────────────────────────────────────┘
Each task runs 3 times for reliability scoring. The tool pulls live pricing from OpenRouter, calculates cost per correct answer, and outputs JSON, CSV, and Markdown reports.
Why OpenRouter? Two reasons. First, it aggregates dozens of models from different providers behind a single API — you can compare a Chinese model against an American model side by side without managing two SDKs. Second, BYOK (Bring Your Own Key) means I can use my own key, my own quota, and my own billing. No vendor lock-in.
And if OpenRouter is hiring — they should definitely hit me up. I’d love to work on the tooling that makes this kind of cross-provider benchmarking trivial.
I compared two free-tier models:
The size difference is staggering — 550 billion parameters versus a “flash” model. Intuition says the bigger model wins. The data says otherwise.
These 15 tasks test what a model actually knows — capitals, arithmetic, science, history — and whether it can produce the right answer. Unlike academic benchmarks that test PhD-level expertise, these are the kinds of questions a real user might ask on any given day. The results reveal a clear pattern: the smaller model handles factual recall with surprising reliability, while the larger model’s struggles with basic math and simple facts suggest that raw parameter count doesn’t always translate to everyday usefulness.
Ling gets 11 of 15 right on at least one try. Nemotron gets 9. But the pattern is telling — Nemotron fails on basic math and factual recall consistently, while Ling’s misses are scattered.
Ling nails every single instruction-following task. Nemotron struggles with constraints — no commas, no digits, exact word counts. This is where the “general usability” gap is most visible. A model that can’t follow “don’t use commas” is a model that will frustrate real users.
Ling calls the right tools 10 of 11 times. Nemotron gets 8. The biggest gap: parallel tool calls (multiple tools in one request). Nemotron fails to call two tools simultaneously three times out of three.
Both models score 100% on structured output. This is the one area where they’re evenly matched.
The tool is a Go CLI that accepts model names, API tokens, and suite configurations via command-line flags, then sends each task to the OpenRouter API. It executes every task across multiple runs, scoring pass/fail for each trial while tracking latency, token usage, and cost — then aggregates the results into per-suite and overall scores. Finally, it writes three output formats (JSON, CSV, and Markdown) so you can inspect the raw data, import it into a spreadsheet, or share a readable summary.
┌─────────────┐ ┌──────────────────────┐ ┌─────────────────┐│ CLI Flags │───▶ OpenRouter API ────▶ Live Pricing ││ -token │ │ /v1/chat/completions│ │ /v1/models ││ -models │ │ BYOK auth │ │ Cache ││ -suites │ └──────────────────────┘ └─────────────────┘│ -runs │ │└─────────────┘ ▼ ┌─────────────────┐ │ Task Execution │ │ 4 suites × N │ │ tasks × R runs │ └────────┬────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ JSON │ │ CSV │ │ Markdown│ │ Report │ │ Trials │ │ Summary │ └──────────┘ └──────────┘ └──────────┘┌────────────────────────────────────────┐│ Legend │├────────────────────────────────────────┤│ N = Number of tasks per test suite ││ R = Number of execution runs per task │└────────────────────────────────────────┘
The entire tool is written in Go using only the standard library — net/http, encoding/json, flag, sort, math, crypto/sha256. No external dependencies.
The results tell a clear story: smaller, focused models can outperform massive ones on real-world tasks. Ling 3.0 Flash is a “flash” model — designed for speed and instruction following. Nemotron 3 Ultra is a 550B behemoth that struggles with basic math and consistently ignores formatting constraints.
This is exactly the kind of evaluation that matters for choosing a model for production work. Not MMLU scores. Not HumanEval rankings. But: does it follow instructions? Does it call the right tools? Does it output valid JSON? And how much does each correct answer cost?
The benchmark tool is open source and written in Go. You can run it yourself:
$ go install github.com/cheikh2shift/go-snippets/llm-bench@latest$ llm-bench \ -token "your-openrouter-key" \ -models "inclusionai/ling-3.0-flash:free,nvidia/nemotron-3-ultra-550b-a55b:free" \ -temp 0 --max-tokens 70000 --retry-delay 60
Source code: https://github.com/cheikh2shift/go-snippets/tree/main/llm-bench
The Ling model is far more capable than its size suggests — and it shows in Lite Agent, where it handles real tasks reliably and quickly. The Nemotron model, despite its 550B parameters, hallucinates basic facts and ignores formatting constraints that matter in production.
If you’re choosing a model for actual work — not just benchmarking — the answer isn’t always the biggest one. Sometimes it’s the fastest, cheapest, and most obedient one in the room.
The benchmark tool is at github.com/cheikh2shift/go-snippets/tree/main/llm-bench. Run it. Compare your own models.
Thank you for reading!