TokenSave founder Jaehyun Cho tells LDS what his 45-run Claude coding experiment establishes about price, speed and passing tests. Haiku had the lowest estimated cost but took longer than Sonnet. Cho explains the original brick-breaker test's limits, uncontrolled caching and missing incomplete attempts, and recommends evaluating cost per accepted change. LDS checked the archived figures; subsequent testing is attributed to Cho.
Claude Haiku 5.5 had the lowest estimated cost in TokenSave's comparison of 45 coding runs. It also took longer than Sonnet 5.5. All the completed games passed their automated checks, but one of those checks initially established much less than the game's prompt required.
In written answers to Let's Data Science, TokenSave founder Jaehyun Cho explains that gap, the caching behind the cost figures and the limits of using three small browser games to choose a coding model. His account gives developers something more useful than a price ranking: a way to ask whether a cheaper result has actually done the job.
Why it matters
A cheap model can be a good starting point when its work is easy to check. The risk is accepting an output because the test is too weak to expose a missing feature. For a coding assistant, the useful comparison includes the cost of reaching an accepted change, including repairs and failed attempts.
Cho recommends a test suite, a limit on cheap-model retries and escalation when those retries fail. He tells LDS: "Then compare cost per accepted change, not cost per attempt." That is a recommendation from this small experiment, rather than a routing policy tested on production software.
What the 45 runs measured
Cho asked Haiku 5.5, Sonnet 5.5 and Opus 5.5 to build a brick-breaker, 2048 and Sudoku. Each task produced one browser application in a single HTML file. Five completed runs for each model on each task gave 45 games, or 15 per model.
He says the prompts were saved before generation and were identical across models. The runs used Claude Code at medium effort on one Windows laptop, without interactive repair feedback or manual changes to the generated applications.
LDS recalculated the following medians of estimated cost per game and elapsed time from the published dataset. Each is the middle result across that model's 15 completed runs; these are not averages.
- •**Haiku 5.5:** $0.0149 per game; 72 seconds.
- •**Sonnet 5.5:** $0.1329 per game; 30 seconds.
- •**Opus 5.5:** $0.2849 per game; 46 seconds.
The dollar figures are API list-price estimates reported by Claude Code, including the recorded cache usage. Cho ran the experiment on his own paid subscription and explains that "they consumed plan allowance rather than being billed per token."
The output records help explain the timing result: Haiku produced a median 14,519 output tokens, compared with 3,109 for Sonnet and 4,180 for Opus. Cho also observed more turns for Haiku in some runs. Those observations do not isolate the causes of the elapsed-time differences.
What counted as a working game
The original brick-breaker checker opened each game in headless Chromium, confirmed a canvas and exercised clicks and arrow keys for about two seconds without a script error. The prompt also required two levels, scoring and three lives.
Cho writes in response to LDS's question about that test: "You're right that this doesn't verify two levels, scoring or three lives."
The other checks went further. For 2048, the script compared 30 moves with the expected slide-and-merge result, allowing for a newly spawned tile. For Sudoku, it checked three generated puzzles for validity and a unique solution, then exercised the Check, Solve and New Puzzle controls. Cho reports that deliberately broken copies failed these checks.
After the interview question, Cho says he tested the archived brick-breaker games more deeply. His final follow-up reports scoring, progression to the second level and three lives in all 15. To inspect lives in nine games whose state was hidden inside a function, he says he added one line to test copies to expose that value; the archived originals remained unchanged.
That follow-up supersedes an earlier, less reliable ball-loss count. It is Cho's account of additional testing. At our source check, the public archive still contained the original checker, and LDS had not inspected the new checking files or rerun the games. The original published pass result and his subsequent checks are therefore separate pieces of evidence. Neither supplies a structured assessment of gameplay polish or every possible edge case.
Why the cost gap needs context
The archived results make the recorded cost comparison inspectable. Summing the published, rounded CSV estimates gives about $0.23 for Haiku's 15 completed runs, $2.03 for Sonnet's and $4.66 for Opus's. Haiku's total is approximately 95% below Opus's and 89% below Sonnet's for these runs.
That does not establish the same saving for an unfamiliar application. Prompt caching reuses previously processed input, and separate working folders did not force cold caches. Cho says the brick-breaker and 2048 tasks always ran in the order Haiku, Sonnet, Opus. The Sudoku runs used a parallel loop.
He did not randomize that order or measure the size of any cache advantage. Repeating the order does not by itself establish that caching affected each model equally. The reported costs describe the observed setup; a controlled cold-cache comparison remains a different calculation.
There is also a limit to the attempt history. Cho says all completed runs are included and none received manual repair. However, the script could replace an incomplete invocation lacking a final result event, and those incomplete attempts were not retained. The 45 completed records cannot establish how many attempts started or what unrecorded retries consumed.
For timing, one laptop and network were shared, and Sudoku ran concurrently. The records do not separate reasoning time, service latency and local contention. These are results for these tasks on this day, rather than a general speed ranking.
Trying the cheaper model on your own work
Cho's proposed starting point is a small, self-contained task with an automated check. He suggests capping cheap-model retries, for example at two, and escalating after the second failure. The experiment did not measure that retry-and-escalation policy: its completed games passed the supplied checks without a repair conversation.
For an evaluation on your own codebase, LDS's assessment is to define acceptance before generation, retain unsuccessful attempts and include review and repair time. Record cache usage alongside the cost estimate. Then compare the total work needed to reach an accepted change across representative tasks. A new UI feature, a database migration and an unfamiliar repository can demand very different tests. The single-file games do not tell a team which model to use for all three. They do show why an affordable first attempt is most useful when the team can recognize an unacceptable result.
What to watch
The next useful evidence is the deeper brick-breaker checker and its per-game results in the promised updated archive. That would let readers inspect how the additional features were tested. It would not expand this study into a benchmark of large codebases.
A team can revisit the decision in its own workflow by checking whether the cheaper model reaches an accepted change at lower total cost within an acceptable time. The quality of that acceptance check is part of the comparison.
Reporting note
This LDS Exclusive draws on Cho's six written answers, his subsequent correction and the public study, scripts and archived records. LDS recalculated the original summary figures and matched the 45 published rows to their archived reported costs, token totals and timings; it did not independently regenerate or execute the games. The proposed workflow guidance above is analysis. Cho says he is the study's sole author, used his own paid subscription and received no provider funding or credits. He is the founder of TokenSave, which publishes the research and token-cost tools.
Key Points #
- 1The study covers three single-file browser games, with five completed runs per model and game. Its conclusions apply to that narrow setup.
- 2Haiku's median recorded list-price estimate was $0.0149 per game, versus $0.1329 for Sonnet and $0.2849 for Opus; the author used a subscription, not per-token billing.
- 3The original brick-breaker checker did not verify all requested features. Cho reports later checks; their new files were not inspected by LDS.
- 4Caching, fixed execution order and unretained incomplete attempts limit general savings claims. Cho recommends measuring cost per accepted change.
Scoring Rationale #
Original interview explains practical limits of a small coding-cost comparison, backed by archived records and useful testing decisions for developers. The study does not establish production-wide savings or a general model ranking.
Sources #
Original reporting, with the public references used alongside it.
LDS Exclusive
Reporting based on written answers given directly to Let's Data Science by Jaehyun Cho, Founder, TokenSave.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.