{"slug": "i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x", "title": "I Tested 4 LLM APIs for Coding Work — All 20 Runs Passed, But Speed Varies 10x", "summary": "A developer tested four LLM APIs for coding tasks, finding all 20 runs passed but speed varied by up to 23x. The free tier landscape has shifted, with providers like NVIDIA NIM, Groq, OpenRouter, and the paid Cline API all succeeding on real coding tasks, though OpenRouter's MiniMax M3 was fastest at 2.9 seconds mean wall time.", "body_md": "I tested 3 free LLM API providers and one paid Claude plan on the same 5 real coding tasks. All 20 runs passed on first attempt.\n\nThe story isn't whether the free tiers work — they do. The story is that the free tier landscape in 2026 has shifted hard. What worked 12 months ago (Cerebras Llama 3.3, free OpenRouter Llama 3.3 70B) now requires a credit card or has been deprecated. What works today is different — and faster.\n\nI started this on a Saturday because the conversation about Cerebras' new card requirement was getting more heat than the data. Felt like the right time to actually measure what works in 2026, not just argue about it. Here's the data, the methodology, and the things I didn't expect.\n\nI wanted to know three things:\n\nTo find out, I tried to test 7 free-tier providers. Of those 7, two required a credit card on file (Cerebras and the Z.ai free tier), two had model IDs that returned 404s on the day I tested, and one (ClinePass) turned out to be a paid aggregator, not a free tier. That left 3 genuinely free providers plus 1 paid control:\n\n| Provider | Model | Cost | Card required? |\n|---|---|---|---|\n| NVIDIA NIM | `nvidia/nemotron-3-super-120b-a12b` |\nFree | No |\n| Groq | `qwen/qwen3.6-27b` |\nFree | Yes (free tier) |\n| OpenRouter | `minimax/minimax-m3:free` |\nFree | No |\n| Cline API | `anthropic/claude-fable-5.1` |\nPaid | Yes |\n\nThe Cline API is a single-billed aggregator that routes to Anthropic, OpenAI, and Google behind one key. It's not free, but it's cheaper than calling Anthropic direct. I used it as the \"what does paid Claude look like\" baseline.\n\nThe task suite covers the four parallelism buckets from real coding-agent work:\n\n`--verbose`\n\nflag to a CLI commandEach task runs in a fresh git worktree branched from `main`\n\n. I read the current file contents into the prompt, told the model \"output each modified file in a `file:path`\n\ncode block\", parsed the response, wrote the new files, and ran the task's verifier (existing test suite plus a content check). Pass means the verifier exits 0 on first attempt — no human edit, no retry.\n\n20 (provider, task) pairs. 1 trial each. ~5 minutes wall time per provider on the network.\n\nThat's the result. No provider failed any task. The difference shows up in speed, not correctness.\n\n| Provider | Pass | Mean wall | Mean tok/s | Mean TTFT | Cost |\n|---|---|---|---|---|---|\n| OpenRouter MiniMax M3 | 5/5 | 2.9 s |\n178 | 1.4 s | Free |\n| Cline API Claude | 5/5 | 7.4 s | 279 | 4.1 s | Paid |\n| Groq Qwen 3.6 27B | 5/5 | 8.8 s | 511 |\n5.8 s | Free |\n| NVIDIA NIM Nemotron 120B | 5/5 | 27.0 s | 409 | 23.0 s | Free |\n\nRead the table right-to-left on speed and you get the story:\n\n| Task | NVIDIA NIM | Groq | OpenRouter | Cline API |\n|---|---|---|---|---|\n| csv-header-infer | 27.1 | 1.6 | 2.3 | 5.2 |\n| function-docstring | 14.5 | 14.6 | 2.1 | 7.2 |\n| function-unit-test | 16.9 | 4.0 | 4.8 | 10.4 |\n| refactor-api-call | 14.6 | 2.7 | 2.5 | 6.5 |\n| cli-flag | 61.7 | 21.1 | 2.7 | 7.9 |\nMean |\n27.0 |\n8.8 |\n2.9 |\n7.4 |\n\nThe cli-flag task is the most striking. NVIDIA took 61.7 seconds (a long thinking block on a multi-file change); OpenRouter finished in 2.7. Same prompt, same task, 23x speed difference.\n\n**The free tier landscape has changed hard in 12 months.** I went in expecting to test 7 free providers. The 4 left standing are different from what every \"free LLMs\" listicle I read recommended 12 months ago. Llama 3.3 70B on Cerebras used to be the default recommendation; it now requires a card. The free OpenRouter `:free`\n\npool rotates through different models and rate-limits unpredictably. If you copy-paste a 2024 \"best free LLMs\" list, half the entries no longer work the same way.\n\n**All 4 free providers passed all 5 tasks.** I expected at least one failure. I did not get one. At this task complexity (real engineering work, not research-grade), the model capability gap has narrowed to style, not correctness.\n\n**The Cline API is the dark-horse recommendation.** I had not heard of it before this test. I went in expecting to compare free tiers to Anthropic direct — I had not realized there was a single-billed aggregator routing to Anthropic, OpenAI, and Google behind one key, with prices below direct Anthropic. For a small team that wants a paid baseline without managing per-provider API keys, this is the cleanest answer. I'll be using it for the team benchmarks going forward.\n\n**The 1.8x speed difference matters more than the price difference.** At a personal scale, the $0 vs ~$0.012/task price gap is rounding error. The 2.5x to 9x speed difference is the real signal. For an interactive coding agent, 3 seconds feels responsive and 27 seconds feels broken.\n\n| Situation | First pick | Fallback | Why |\n|---|---|---|---|\n| Personal coding, lowest latency | OpenRouter | Groq | 2.5s P50, surprisingly good |\n| Personal coding, best raw quality | Cline API | Groq | Claude is the gold standard; Groq is the closest free |\n| No credit card on file | NVIDIA NIM | OpenRouter | Both work without card; NVIDIA is slow but high quality |\n| 5+ engineers, team coverage | Cline API | + Groq fallback | Free tiers throttle under team load |\n| Building on top of Claude Code | Cline API | — | Same Anthropic models, single key, lower price |\n\nFor everyone else: the free tiers work today. Don't assume \"free\" means \"broken\" — the providers in this test all produced working code on the first try. The honest ceiling right now is \"responsive enough for personal work, not for team load.\"\n\nThe benchmark is in the public repo:\n\n```\ngit clone github.com:Pitambarmahato/hardnumbers-experiments\ncd hardnumbers-experiments/free-tier-llm-coders\npython -m venv .venv\n.venv/bin/pip install -r requirements.txt\necho \"NVIDIA_NIM_API_KEY=nvapi-...\" >> .env\necho \"GROQ_API_KEY=gsk_...\" >> .env\necho \"OPENROUTER_API_KEY=sk-or-...\" >> .env\necho \"CLINEPASS_API_KEY=sk_...\" >> .env\nchmod 600 .env\n.venv/bin/python src/benchmark.py --providers all\n```\n\nThe full 20-run benchmark takes about 5 minutes. Results land in `results/free_tier_<timestamp>.json`\n\nwith the wall time, tokens, rate-limit hits, and pass/fail for each pair.\n\nA few things I didn't measure and would change the recommendation:\n\nFor personal work, OpenRouter is the surprise winner and I'll keep using it. For team work, the Cline API is the cleanest paid answer and I'll be testing it across more workflows.\n\nThe next benchmark in this series is going to be **multi-agent throughput** — 1 vs 2 vs 4 vs 8 parallel agents on the same task suite. The story there is whether parallelization gives you 4x speedup or whether tightly coupled work punishes you for adding agents. That article drops next week.\n\nIf you have API keys for free LLM providers I missed, open a PR on the [experiments repo](https://github.com/Pitambarmahato/hardnumbers-experiments) with the new provider — the benchmark is designed to be extended.\n\nThe full data, methodology, and what-didn't-test section is at [hardnumbers.dev/articles/4-free-llm-apis-vs-claude-5-coding-tasks-real-data](https://hardnumbers.dev/articles/4-free-llm-apis-vs-claude-5-coding-tasks-real-data) — that's the canonical version with the full breakdown, including rate-limit hit analysis and the Cline API vs direct Anthropic cost comparison.", "url": "https://wpnews.pro/news/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x", "canonical_source": "https://dev.to/pitambarmahato/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x-4bpl", "published_at": "2026-09-03 04:11:11+00:00", "updated_at": "2026-09-03 04:24:19.445252+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["NVIDIA NIM", "Groq", "OpenRouter", "Cline API", "MiniMax M3", "Qwen 3.6 27B", "Nemotron 120B", "Claude"], "alternates": {"html": "https://wpnews.pro/news/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x", "markdown": "https://wpnews.pro/news/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x.md", "text": "https://wpnews.pro/news/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x.txt", "jsonld": "https://wpnews.pro/news/i-tested-4-llm-apis-for-coding-work-all-20-runs-passed-but-speed-varies-10x.jsonld"}}