The 200x Cost Spread Hiding in Your Model Picker

Netlify's Agent Runners benchmark of 11 AI models on the same website-building brief found a roughly 200x cost spread, with Claude Opus 5 averaging 519 credits per run (spiking to 1,055) versus DeepSeek V4 Flash at 2.4 credits, while quality did not track cost linearly. The company published the generated sites and costs, highlighting that one Opus run can consume Netlify's entire 300-credit free tier, and that public leaderboards hide such cost variance.

AI https://sourcefeed.dev/c/ai Article The 200x Cost Spread Hiding in Your Model Picker Netlify's 11-model bake-off says less about which model won than about how you should be choosing. Priya Nair https://sourcefeed.dev/u/priya nair The model picker used to be a settings-page afterthought. Now it's load-bearing. GitHub Copilot https://github.com/features/copilot ships a dropdown with models from three vendors, agent platforms route per-task, and Netlify https://www.netlify.com/ 's Agent Runners expose a whole menu of models through OpenRouter https://openrouter.ai/ . Which raises a question nobody's leaderboard answers: when the dropdown has eleven options, which one do you actually click? Netlify just published the most honest attempt at an answer I've seen in a while. The team fed the same brief — a one-page site for a neighbourhood coffee shop, with hours, address, a short menu, a photo, and nothing dynamic — to 11 models, three runs each, and published the generated sites alongside what each run cost in platform credits. The lineup spans Claude Opus 5 and Sonnet 5, two GPT 5.6 variants, two Geminis, two Kimi models, GLM 5.2, and two DeepSeek https://www.deepseek.com/ V4 variants. The number that matters is 200x Forget who "won." The finding with teeth is the cost spread. Claude Opus 5 averaged 519 credits across its three runs, with one run spiking to 1,055. DeepSeek V4 Flash averaged 2.4. That's a roughly 200x difference for the same brief on the same platform — and for scale, Netlify's free tier includes 300 credits total. One Opus run at the wrong moment eats your entire monthly allowance building a static coffee-shop page, and overage on Pro runs $10 per 1,500 credits. Quality did not track cost anywhere near linearly. Opus produced the richest output — custom vector art, dark mode, detail nobody asked for — but with wildly unpredictable spend. GPT 5.6 in low-effort mode showed strong design instincts for about a quarter of the cost. Gemini 3.1 Pro, a flagship, did the bare minimum across all three runs. And DeepSeek V4 Flash's best run was genuinely usable at around two credits. The frontier tier bought polish and unpredictability; the budget tier bought adequacy and variance. Neither bought reliability. Yes, the methodology is thin — that's not the point The Hacker News thread landed the obvious punches: three runs is not a sample, one prompt is not a benchmark, and a one-shot build says nothing about the multi-turn, feedback-heavy way people actually work with these tools. One commenter called single-sample evals "essentially worthless." Netlify itself concedes that what matters in production is whether a model understands your platform, validates its own work, and takes correction — none of which a coffee-shop one-shot measures. All true, and all slightly beside the point. The realistic alternative for most developers isn't a rigorous eval — it's vibes plus a leaderboard aggregating tasks that aren't yours, with costs hidden entirely. What Netlify demonstrated, maybe more usefully than it intended, is the method: pick a brief that resembles your real work, run it a few times per model, and look at outputs next to costs. That last part is the rare bit. Public rankings will tell you a model's Elo; they will not tell you it occasionally quadruples its own spend on an identical prompt. The thread surfaced something else worth filing away: convergence. Several commenters noted the outputs looked eerily alike — one person recognized the same design Claude had produced for their own site redesign, which is a rough way to learn your bespoke landing page is a default. Model "personalities" are real, but so is the shared training-data gravity pulling everything toward the same tasteful purple-tinged layout. If differentiation matters, it comes from your follow-up prompts, not your model choice. Steal the method, not the verdict A quieter Dev.to piece by Edward Li makes the argument Netlify's data supports: pick your first model from evidence, not memory. His version of the discipline is almost boring — one representative task, one project key, cheapest plausible model first, and read the request logs before touching your application code. The fields he says to actually inspect: requested versus served model, tokens in both directions, latency, real cost, retry indicators, and whether the output survived the downstream step it feeds. Sticker price lies constantly. A "cheap" model that needs longer context and two retries out-spends an "expensive" one that one-shots clean. The daily-driver experience matches. Another practitioner writeup, from months of running Copilot's multi-model setup with spec files for repo context, settled into exactly this escalation ladder: Claude for convention-heavy, multi-service work — at a reported burn of around $200 a month in tokens — Gemini as the budget workhorse that needs iteration, and OpenAI's models relegated to standalone scripts after repeatedly failing to update impacted classes. Whatever you think of the specific rankings, the shape is the point: nobody who measures ends up loyal to one model. They end up with a routing policy. So here's the policy the evidence supports. Treat model selection like dependency selection: pin a default, measure it on your own brief, and upgrade deliberately when it fails — not because a leaderboard moved. Default to the cheapest model that clears your bar; the 200x spread means frontier-by-default is simply the wrong default for scaffolding, boilerplate, and static builds. Reserve the expensive model for the tasks where the cheap one demonstrably falls over, and re-run your three-sample check when versions change, because these results will be stale in a quarter. Netlify's experiment isn't science and doesn't pretend to be. It's a costed demonstration that the spread — in price, in quality, in consistency — is real and large, and that the only benchmark with predictive power for your work is your own prompt, run more than once, with the bill attached. That test costs you an afternoon and a few hundred credits. Shipping the wrong default costs you every month. Sources & further reading - Choosing an AI model: one prompt, 11 models, different results https://www.netlify.com/blog/one-prompt-11-models-very-different-results/ — netlify.com - Choosing an AI model: one prompt, 11 models, different results - discussion https://news.ycombinator.com/item?id=49285327 — news.ycombinator.com - Pick your first AI model from evidence, not memory https://dev.to/edward li 71f26791eac62b8/pick-your-first-ai-model-from-evidence-not-memory-1c33 — dev.to - Beyond the Hype: Choosing the Right Model for Your Daily Workflow https://dev.to/vishesh/beyond-the-hype-choosing-the-right-model-for-your-daily-workflow-1aka — dev.to Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.