Tell HN: One SWE-bench-Live task: $47 Opus failed, $1.46 GPT-5.6 passed Seven AI models attempted a SWE-bench-Live task with internet and git history disabled; three passed and four failed. GPT-5.6 Luna solved the task for $1.46 while Opus 4.8 failed at $47.18, with cache-read costs dominating the expense in longer runs. | title | What the closed-door dynaconf runs teach tomo | |---|---| | linkTitle | dynaconf closed-door lessons for tomo | | description | Seven honest runs on one task, three passes and four fails, read together. The lessons that transfer to tomo: a broad edit that regresses a green test is worse than no edit and wants a do-no-harm gate, spend does not track progress, cache-read is where the money actually goes, and the reflexive git-history probe is fine once and a runaway if repeated. This is the cross-trace synthesis, with a full cost breakdown, the tests each run was graded on, and the commands to reproduce it. | | date | 2026-07-13 14:55:00 +0700 | Seven models ran dynaconf dynaconf-1225 with both answer doors shut: the git-history door pruned and the network denied. Three solved it, four failed, and every one of the seven was honest, no answer fetched. Read as a set they say more than any single run, and most of what they say points at concrete tomo work. This note collects the synthesis in one place: the runs, a full cost breakdown, the four lessons, the exact tests each run was graded on, and the commands to reproduce the whole thing. | Model | Harness | Files | Tokens | Cost | Verdict | |---|---|---|---|---|---| | gpt-5.6-terra /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-45-dynaconf-5.6-family-solves-analyzer-false-leak gpt-5.6-sol /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-45-dynaconf-5.6-family-solves-analyzer-false-leak gpt-5.4-mini /tamnd/tomo-labs/blob/main/experiments/2026/07/13/13-49-dynaconf-gpt-5.4-mini-offline-honest-fail gpt-5.5 /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-01-dynaconf-gpt-5.5-offline-honest-fail sonnet-5 /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-25-dynaconf-sonnet-offline-honest-broad-fail opus-4.8 /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-35-dynaconf-opus-offline-regresses-green-test Every dollar figure is what the same tokens would cost on the metered API, priced from the shared pricing table /tamnd/tomo-labs/blob/main/guides . A ChatGPT or Claude subscription is not billed per token, so this is a like-for-like cost, not the user's actual bill. The single number in the table hides where the money goes, and where it goes is the lever. Here is each run split into its four token classes. | Model | Fresh input | Cache read | Cache write | Output | Total | |---|---|---|---|---|---| | gpt-5.6-luna | $0.2457 | $0.9836 | included in read | $0.2312 | $1.4605 | | gpt-5.6-terra | not split by codex | $1.8240 | ||| | gpt-5.6-sol | not split by codex | $2.5628 | ||| | gpt-5.4-mini | $0.2956 | $0.3305 | included in read | $0.1579 | $0.7840 | | gpt-5.5 | $0.8549 | $2.9429 | included in read | $0.6905 | $4.4883 | | sonnet-5 | $0.1198 | $6.8849 | $1.4293 | $1.8825 | $10.3165 | | opus-4.8 | $0.1869 | $30.3442 | $5.5643 | $11.0872 | $47.1827 | Two things fall straight out of the split. Fresh input is a rounding error everywhere. The prompt and the files are read once, cheap, and then never re-billed at the input rate. Cache read is the bill. It is the whole context, the conversation and every file the agent has pulled into view, re-sent to the model on every single turn. On the $47 opus run, cache read alone is $30 of the $47. On sonnet it is $6.9 of the $10. The longer the run and the fatter the working context, the more this dominates, and it dominates by a wide margin over the tokens the model actually writes. Output is second, and it only becomes visible on the very long runs. Opus wrote $11 of output across 194 turns; luna wrote 23 cents across 80. Cache write, the cost of laying context into the cache the first time, is a minor line even on the big runs. The tokens counts tell the same story from the other side. The three cheapest passes, the gpt-5.6 family, moved 3 to 10 million tokens. The two most expensive fails, sonnet and opus, moved 20 to 23 million, most of it context re-sent turn after turn while the run went nowhere. The naive read of the table is that breadth or spend decides the outcome. Both are wrong. luna went the widest, twenty-five files, and passed. opus spent the most, $47, and produced the worst result of the seven, a repo where a test that was green at the base commit is now red. What actually separated the pass from the fails is narrower: did the broad refactor carry the identifier all the way through the loader stack, and did it leave the already-green tests green. luna's did both. opus's did neither, and the second failure is the instructive one. It did not merely fail to fix the target, it damaged working behavior on the way to failing. tomo already has a convergence guard /tamnd/tomo-labs/blob/main/experiments/2026/07/13/10-05-dynaconf-tomo-guard-vs-pi-runaway that stops it running away searching for a fix. It has nothing that stops it shipping an edit that regresses a test that was passing. The gap this run exposes is a do-no-harm gate: after an edit batch, run the in-repo tests the change plausibly touched, and treat a green-to-red flip as a stop-and-reconsider signal rather than continuing to pile edits on. It is model-independent, it would have caught opus before its twenty-third file, and it would not have fired once on luna. That is the property to want: it punishes net-negative edits, not breadth. Cost on this task ranged from $0.78 to $47.18 and told you nothing about the verdict. The three cheapest passing runs, the gpt-5.6 family, cost $1.46, $1.82, and $2.56. The most expensive run failed and regressed. Turn count says the same: opus took 194 turns and sonnet 235 to reach a wall the gpt-5.6 models cleared in a few dozen tool calls. For tomo the takeaway is not "be cheap for its own sake," it is "do not read your own token spend or turn count as evidence of progress." A loop that has spent a lot has not thereby earned anything, and the guards should measure movement toward a passing fix, not effort. tomo's leanness pitch holds on the merits here: its honest fails are cheap, and cheap-and-honest beats the $47 fail every time. The cost breakdown above is the whole lesson. Output-length trimming barely moves the bill. What moves it is the size of the context tomo re-sends each turn, because that is what gets billed at the cache-read rate on every turn of a long run. That is a direct pointer for tomo. It is why trimming the redundant read-after-write /tamnd/tomo-labs/blob/main/experiments/2026/07/13/11-05-churn-guard-vs-claude-code was a real lever, and why keeping the working context tight, dropping stale file dumps and superseded tool output, is the cost work that pays off. Shrinking what tomo re-sends per turn attacks the $30 line, not the 23-cent one. terra and sol /tamnd/tomo-labs/blob/main/experiments/2026/07/13/14-45-dynaconf-5.6-family-solves-analyzer-false-leak both opened with git log --all --grep=