Benchmarks show scores. Dashboards show usage. What ships?

OpenCode 2.0 introduces Pragmatikos, a new scoring system that ranks AI model pairings by real-world shipping outcomes rather than benchmark scores. Based on 1,210 developer sessions from 2 contributors, the system evaluates planner-builder pairs across six weighted axes including ship rate, cost per ship, and precision, with scores expected to change as more data is contributed.

For OpenCode 2.0 Benchmarks test single model. Developers often pair them. A benchmark is an exam: a clean task, a hidden answer key, one model, nobody steering. Real work is the job: a messy repo, your tools, your steering, and increasingly one model planning while another builds. Pragmatikos scores the job, not the exam : every planner → builder pairing, on real sessions, by what actually shipped. Rankings below are pooled from real developer sessions. claude-fable → grok · 62 / 100 the rankings Models and pairings, ranked by what ships Did the work land in a commit, how many nudges did it take, how clean was it, what did it cost. Scored on real sessions, not lab tasks. Scores are expected to change as more data is contributed. 1,210 Cycles<sup </sup · 2 contributors join in This is a first answer, not the final one. It's built from the sessions developers have shared so far. The more histories in the pool, the harder the ranking gets to argue with. why it exists Three questions. Three instruments. The first two are useful and stay useful. Developers choosing a model and a workflow are missing the third answer. question 01 · the lab Benchmarks ask: how capable is this model? SWE-bench, Terminal-bench, Aider, LMArena. Controlled tasks, hidden tests, preference votes — the best way to compare models on equal footing and know each model’s ceiling. - signal - pass rate · votes - blind spot - One model, a clean task, nobody steering. question 02 · the crowd Usage rankings ask: what is the market using? OpenCode Data, OpenRouter Ranking. Tokens, users, retention, dollars per session — the best way to see adoption and where the mix is shifting. - signal - tokens · users · $/session - blind spot - Popular is not productive — and still one model at a time. question 03 · the recorder Pragmatikos asks: what ships in real work? Turns, phases, edits and errors from real sessions, correlated with what shipped. The one instrument that scores the setups developers actually run. - signal - session record × shipped outcomes - blind spot - Anything it never saw. Observational, not a benchmark. Capable. Popular. Effective. Only one of them has a row for the setup you actually run. the blind spot both miss Same builder. Different planner. Opposite results. The build model is identical. One planner got it to a commit in a few turns; the other burned dozens and shipped nothing. A model ranking can't see this. A pairing ranking can. claude-fable-5.1 → grok-4.6 0 turns grok-4.6 → grok-4.6 0 turns Real sessions. Unshipped work counts against the model. how the score works Six tiers, ten axes, one weighted mean Outcome weight 30 Ship rate and one-shot rate prompt plus approval . Cost of a ship weight 20 Turns, hours and dollars per shipped cycle. Precision weight 15 Tool errors and human interventions. Discipline weight 15 Share of cycles verified after the last edit. Efficiency weight 10 Edits per human turn. Latency weight 10 Median seconds per assistant step. Pool-relative Every axis is a log distance to the pooled average — log odds for rates — on a fixed ×4 span, so ×2 and ÷2 sit the same distance from the line. Evidence-weighted Small groups shrink toward the pool with k = 10 pseudo-cycles. A two-cycle fluke nearly vanishes. Failure lowers the rate Unshipped cycles stay in the ship-rate denominator, so few-turn dead ends can never look good. Turns, hours and dollars are priced per shipped cycle. two records, correlated The session record comes first Every turn, plan and build phase, edit, tool error and dollar is read from the agent's own record — that is where every axis is measured. Commits only decide which cycles count as shipped: a window from the previous commit, file overlap decides credit, active-but-absent sessions advised only. where it falls short A hypothesis to try, not a verdict Pragmatikos reads sessions after the fact. Nobody assigned pairings to developers or tasks, so the ranking says which setups shipped for the people who ran them — not which setup would ship for you. Here is where that bites, and what we intend to do about it. Observational, not controlled The developers who run one pairing are not the developers who run another, and they are not working the same tasks. Skill, repo and task difficulty can move a score as much as the models do. “Ranked by what ships” is a correlation. Read it as one. Small, and uneven 1,210 cycles from 2 contributors, across 7 family pairings — 72% of them from one pairing. Groups near the ten-cycle floor are shrunk toward the pool, which guards against flukes but does not stand in for evidence. Two scores a point or two apart are a tie. Self-selected Everyone in the pool installed a plugin and left sharing on. The ranking says nothing about developers who did not. Shipped is a floor, not a grade A commit means the work landed. It does not mean it survived review, was never reverted, or was any good. More histories Every new contributor puts the same pairings in front of different developers, repos and tasks. That is what lets a pairing’s effect separate from the developer’s. The scale and the shrinkage were built for a bigger pool; the pool is what is missing. Share your sessions → contribute If that is not enough The pool has one thing going for it: everyone in it is a developer running OpenCode on real work — the population this ranking is for. That does not fix the task and skill mix, but a different approach can start from it. If more histories do not settle the picture, the method changes, and this page will say what changed. contribute Add your sessions in two minutes For OpenCode v2 — other coding agents coming soon. Install the ocInsights plugin for OpenCode: add "@pfoundation/ocinsights" to the "plugin" list in my global opencode.json config. Then tell me to restart OpenCode, and afterwards verify the plugin loaded and the insights deck answers at http://127.0.0.1:4173/. Also report whether insight contribution is on, without changing that setting. 1. 1Paste the prompt into your OpenCode agent and let it edit your config. 2. 2Restart OpenCode when it tells you to, then open the deck it verifies. 3. 3Preview with Contribute in the deck header — sharing is on by default. Twenty-five fields per cycle — day, models, effort, harness, turns, edits, cost, shipping. No paths, prompts, session ids or projects. Preview in the deck's Contribute panel before anything leaves your machine.