I Tested 4 LLM APIs for Coding Work — All 20 Runs Passed, But Speed Varies 10x

I tested 3 free LLM API providers and one paid Claude plan on the same 5 real coding tasks. All 20 runs passed on first attempt.

The story isn't whether the free tiers work — they do. The story is that the free tier landscape in 2026 has shifted hard. What worked 12 months ago (Cerebras Llama 3.3, free OpenRouter Llama 3.3 70B) now requires a credit card or has been deprecated. What works today is different — and faster.

I started this on a Saturday because the conversation about Cerebras' new card requirement was getting more heat than the data. Felt like the right time to actually measure what works in 2026, not just argue about it. Here's the data, the methodology, and the things I didn't expect.

I wanted to know three things:

To find out, I tried to test 7 free-tier providers. Of those 7, two required a credit card on file (Cerebras and the Z.ai free tier), two had model IDs that returned 404s on the day I tested, and one (ClinePass) turned out to be a paid aggregator, not a free tier. That left 3 genuinely free providers plus 1 paid control:

The Cline API is a single-billed aggregator that routes to Anthropic, OpenAI, and Google behind one key. It's not free, but it's cheaper than calling Anthropic direct. I used it as the "what does paid Claude look like" baseline.

The task suite covers the four parallelism buckets from real coding-agent work:

--verbose

flag to a CLI commandEach task runs in a fresh git worktree branched from main

. I read the current file contents into the prompt, told the model "output each modified file in a file:path

code block", parsed the response, wrote the new files, and ran the task's verifier (existing test suite plus a content check). Pass means the verifier exits 0 on first attempt — no human edit, no retry.

20 (provider, task) pairs. 1 trial each. ~5 minutes wall time per provider on the network.

That's the result. No provider failed any task. The difference shows up in speed, not correctness.

Read the table right-to-left on speed and you get the story:

The cli-flag task is the most striking. NVIDIA took 61.7 seconds (a long thinking block on a multi-file change); OpenRouter finished in 2.7. Same prompt, same task, 23x speed difference.

The free tier landscape has changed hard in 12 months. I went in expecting to test 7 free providers. The 4 left standing are different from what every "free LLMs" listicle I read recommended 12 months ago. Llama 3.3 70B on Cerebras used to be the default recommendation; it now requires a card. The free OpenRouter :free

pool rotates through different models and rate-limits unpredictably. If you copy-paste a 2024 "best free LLMs" list, half the entries no longer work the same way.

All 4 free providers passed all 5 tasks. I expected at least one failure. I did not get one. At this task complexity (real engineering work, not research-grade), the model capability gap has narrowed to style, not correctness.

The Cline API is the dark-horse recommendation. I had not heard of it before this test. I went in expecting to compare free tiers to Anthropic direct — I had not realized there was a single-billed aggregator routing to Anthropic, OpenAI, and Google behind one key, with prices below direct Anthropic. For a small team that wants a paid baseline without managing per-provider API keys, this is the cleanest answer. I'll be using it for the team benchmarks going forward.

The 1.8x speed difference matters more than the price difference. At a personal scale, the $0 vs ~$0.012/task price gap is rounding error. The 2.5x to 9x speed difference is the real signal. For an interactive coding agent, 3 seconds feels responsive and 27 seconds feels broken.

For everyone else: the free tiers work today. Don't assume "free" means "broken" — the providers in this test all produced working code on the first try. The honest ceiling right now is "responsive enough for personal work, not for team load."

The benchmark is in the public repo:

git clone github.com:Pitambarmahato/hardnumbers-experiments
cd hardnumbers-experiments/free-tier-llm-coders
python -m venv .venv
.venv/bin/pip install -r requirements.txt
echo "NVIDIA_NIM_API_KEY=nvapi-..." >> .env
echo "GROQ_API_KEY=gsk_..." >> .env
echo "OPENROUTER_API_KEY=sk-or-..." >> .env
echo "CLINEPASS_API_KEY=sk_..." >> .env
chmod 600 .env
.venv/bin/python src/benchmark.py --providers all

The full 20-run benchmark takes about 5 minutes. Results land in results/free_tier_<timestamp>.json

with the wall time, tokens, rate-limit hits, and pass/fail for each pair.

A few things I didn't measure and would change the recommendation:

For personal work, OpenRouter is the surprise winner and I'll keep using it. For team work, the Cline API is the cleanest paid answer and I'll be testing it across more workflows.

The next benchmark in this series is going to be multi-agent throughput — 1 vs 2 vs 4 vs 8 parallel agents on the same task suite. The story there is whether parallelization gives you 4x speedup or whether tightly coupled work punishes you for adding agents. That article drops next week.

If you have API keys for free LLM providers I missed, open a PR on the experiments repo with the new provider — the benchmark is designed to be extended.

The full data, methodology, and what-didn't-test section is at hardnumbers.dev/articles/4-free-llm-apis-vs-claude-5-coding-tasks-real-data — that's the canonical version with the full breakdown, including rate-limit hit analysis and the Cline API vs direct Anthropic cost comparison.