Kimi K3 was released this week, and like every model release it's being judged on leaderboard scores and screenshots. But a score is a bit like a football result, in that it tells you who won, not how the game was played. You wouldn't sign a player off a scoreline alone — you'd want to watch the tape. With code models, the tape is the code itself: how it's structured, whether you'd actually want to maintain it. That's what leaderboards can't show you.
So we built a benchmark that compares Kimi K3, Claude Fable 5 and Claude Opus 4.8 on exactly that — and it's open source, so you can run it yourself at github.com/dsplce-co/kimi-vs-fable-vs-opus.
This note summarises our findings after running it.
Given a double-entry ledger module with an existing test suite, make it production-ready and add two features: transaction reversal and statement generation. There's no hint that there are any bugs to find.
But the existing tests aren't actually good — they don't test for real money-safety bugs:
So passing these tests proves only demo quality — that the core functionality works, nothing more. To grade the results we used a second, hidden test suite that checks the actual money safety — one the models never see.
All three models were run in the same harness:
We made sure to run Kimi K3 inside Claude Code too, just like the Claudes, so the harness itself isn't a variable here. The results reflect how well each model does as an agent inside Claude Code — how it would work with a dev team using it.
We're framing the analysis around how each model wrote the code, rather than the pass/fail binary.
All three, across all nine runs:
// validate everything first — this part is allowed to throw
for (const line of lines) {
const acc = accounts.get(line.accountId);
if (acc.state !== "open") throw new Error(`account not open: ${line.accountId}`);
}
// then commit — from here nothing throws, so it's all-or-nothing
for (const line of lines) this.entries.push(makeEntry(line));
reverse
feature as a compensating transaction routed through the existing post
function, inheriting its atomicity and lifecycle checks.So fixing the known bug and adding the features is a solved problem for all of them. They're interchangeable on that basic bar — the differentiation only starts to appear with the harder questions about the code's long-term behaviour.
Its limitation: it stores money as a bare number
type — nothing enforces dollars vs cents at the type level; it's all runtime checks. And its reversal linkage uses a string-prefix convention, which is fragile if the naming convention isn't followed.
post
function to reject non-integer amounts, but left the dollars-to-cents conversion unrounded — a floating-point issue, like 0.07 * 100 = 7.000000000000001
:
// tightened check inside post: reject non-integer minor units
if (!Number.isInteger(amount)) throw new Error("amount must be whole minor units");
// but the conversion, in another file, was left unrounded
function toMinor(major: number) { return major * 100; } // 0.07 * 100 -> 7.000000000000001
0.07
transfer doesn't work now, because the stricter integer check collides with the floating-point imprecision from the unrounded dollars-to-cents conversion. Worse yet, the run's own tests stayed green, because it never created a test that transfers a fractional amount; it was like its safety net had the very hole its own change fell through — a landmine that looks green. The other two Opus runs rounded the conversion and tested it, so this is not a constant weakness of the model.
post
, idempotent reversal, copy-at-the-boundary pattern.bigint
, which is technically the right way to represent currency in JavaScript. But Kimi's bigint
conversion was inconsistent at the edges — the transfer function still accepted floating-point numbers, and the balance reader converted back via floating point (Number(minor) / 100
), reintroducing the very error bigint
was adopted to remove.getEntries
to protect the ledger's internal state, but that broke callers who legitimately append to their own local copy of that array; the internal state stays safe, but the public contract quietly changed underneath everyone using it — it is the mirror image of a leak (a leak lets a caller corrupt your state, this protects your state by breaking the caller).What these findings tell us is that all three models have mostly solved the core problem — they fixed the bug and added in the features. The remaining risk is at the edges:
It's also worth noting that even for the same model with the same prompt, the public-contract behaviour differed across runs — whether a duplicate transaction throws or silently no-ops, whether a reversal is allowed after an account is closed, whether querying a statement for a nonexistent account throws or returns an empty list. This variance was present in all three models.
Which means that if a particular edge-case behaviour is crucial, it's best to pin it with an explicit test rather than assuming it's consistent across model runs.
| Model | Clean runs | The one failure |
|---|---|---|
| Claude Fable 5 | 3 / 3 | — |
| Claude Opus 4.8 | 2 / 3 | a fractional transfer throws incorrectly |
| Kimi K3 | 2 / 3 | a frozen returned array breaks a caller |
Both Opus's and Kimi's failures were about judgment and edge cases, not fundamental capability. This aligns with the earlier finding that all three have largely solved the core problem.
We also included a restraint trap: an existing function rounds money to whole units, which is lossy — but it's well-documented, tested behaviour that some other code depends on. The senior move is to leave the rounding as-is, add a warning comment, and provide a new, exact accessor next to it. The junior move is to silently "fix" the rounding, breaking the existing callers that depend on it.
Fable and Opus made the senior move in every run — they preserved the existing rounding behaviour, added a warning comment, and added an exact alternative accessor. Kimi was the least disciplined here:
We're sharing this because it's revealing — how a model handles a contract it didn't write, when nobody forces it to, says a lot about what it'll do to the contracts already in your codebase.
The verdict is two-sided. On one hand, Fable wins on consistency — no shipped bugs, and the most reliable. On the other, this is a single, small task, and the other models' failures weren't substantial enough to rule out the results shifting on a different problem.
So what we'd recommend is to treat this benchmark as a template:
You can run the benchmark on your own codebase, with your own hidden tests, using the code (including the exact prompt) published at github.com/dsplce-co/kimi-vs-fable-vs-opus.