{"slug": "which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out", "title": "Which LLM should I actually code with? I built a small benchmark to find out", "summary": "A developer built a small benchmark to compare LLMs for coding tasks across Python, C#, and Bash. All three models achieved 100% pass@3 on problems they were allowed to answer, but one model was content-filtered on four problems, reducing its effective coverage. The developer concluded that accuracy is no longer a differentiator, so the choice should be based on speed and cost per language.", "body_md": "I kept going back and forth on which model to reach for in my actual day job. Every \"which LLM is best at code\" thread turns into vibes and screenshots, and none of it answered the question I had, which is which one to open when I have real work in the languages I use. So I stopped guessing and built a small benchmark to settle it for myself.\n\nIt is deliberately small. 14 problems across Python, C#, and Bash, three models, three attempts each at temperature 0.7 with a 10 second timeout. Every attempt runs in a sandboxed Docker container and gets scored on pass@k, cost, and latency. It is not an authoritative ranking and I am not pretending it splits hairs. It is enough to show the shape.\n\nAccuracy is not the differentiator anymore. All three models solved every problem they were allowed to answer, 100% [pass@3](mailto:pass@3). If I only looked at pass rates I would have learned nothing, because they all pass.\n\nThe catch hides in \"allowed to answer.\" One model got content filtered out of four problems, so its perfect score covers 10 of the 14, not the whole set. A perfect score on the problems you answered and a perfect score on the whole bench are not the same result, which is why the leaderboard sorts on coverage first.\n\nIf they all pass, the decision comes down to what you pay and how long you wait. That is where the spread lives.\n\nSo the honest summary is boring in the best way. Pick on speed and price, because accuracy already agrees.\n\nOne result is not like the others. One model was quick and cheap on Python and C#, a second or two per problem, then fell off a cliff on Bash at 90 to 168 seconds per problem. Four slow Bash problems dragged its average latency up to something that makes it look slow overall when it is really fine everywhere except Bash. If your day is mostly shell, that matters a lot. If it is Python, you would never notice.\n\nThat is why a single \"which model is fastest\" number is a lie. Fastest at what, in which language, is the only version of the question worth asking.\n\nFor my work it came down to speed and cost per language. The benchmark did the one job I wanted. It turned a running argument in my head into a few numbers I can re-run whenever the models change.", "url": "https://wpnews.pro/news/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out", "canonical_source": "https://dev.to/peculiarengineer/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out-3nbg", "published_at": "2026-07-12 03:41:18+00:00", "updated_at": "2026-07-12 04:15:12.474164+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out", "markdown": "https://wpnews.pro/news/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out.md", "text": "https://wpnews.pro/news/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out.txt", "jsonld": "https://wpnews.pro/news/which-llm-should-i-actually-code-with-i-built-a-small-benchmark-to-find-out.jsonld"}}