{"slug": "benchmarking-fable-sol-and-kimi-k3-on-slopcodebench", "title": "Benchmarking Fable, Sol, and Kimi K3 on SlopCodeBench", "summary": "In a benchmark run on Thursday, Fable and Sol tied at 33.3% (10/30 strict checkpoint passes) on SlopCodeBench, with Expo and Kimi K3 following at 26.7% and 23.3%, respectively. The evaluation, conducted by HumanLayer across six challenges and 30 checkpoints per model, used harnesses including Claude Code 2.1.219 for Fable, Codex CLI 0.145.0 for Sol, and OpenCode 1.18.0 for Kimi K3 and Expo. SlopCodeBench, developed by Gabe Orlanski's lab at UW Madison, tests long-horizon coding by revealing requirements incrementally across checkpoints.", "body_md": "Last week when Opus 5 came out, I [ran some benchmarks](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md) against my favorite new coding agent eval - [SlopCodeBench](https://arxiv.org/abs/2603.24755).\n\nThe primary takeaway was: this is a hard benchmark and even brand-new models didn't break 35% pass rate (that's 2024 SWE-Bench territory, the sonnet 3.5 days).\n\nBut everyone was asking: where do **Fable, Sol, and Kimi K3 land?**\n\nSo on Thursday I ran it again, for\n\n- Fable, Sol, Kimi K3, and an undisclosed stealth model (\"Expo\")\n- a larger subset of problems: six challenges and 30 checkpoints per model\n\nFable and Sol tied on this set at **33.3%** (10/30 strict checkpoint passes), with Expo and Kimi K3 close behind at **26.7%** and **23.3%**, respectively.\n\nI'll refresh on how slop code bench works real quick, or you can jump to the detailed results.\n\n[You can learn a bit more in the original post](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md), but briefly:\n\nSlopCodeBench is a new-ish (March 2026, last update May 2026) long-horizon coding benchmark from [@GOrlanski](https://x.com/GOrlanski)'s lab at UW Madison. It addresses the thing that bothers me most about most coding benchmarks - that even \"larger\" more complex ones still divulge the whole problem up front.\n\nEach challenge in SlopCodeBench has multiple \"checkpoints\" - the model doesn't know the whole problem up front, it has to *evolve* the codebase over time as new requirements are divulged.\n\nIt's a good paper. It's not that long. You should [read it](https://arxiv.org/html/2603.24755v1).\n\nThe biggest takeaway from \"the fact that SlopCodeBench is\n\ni pinged Gabe and asked him for a slightly broader subset of problems for this run - this one ran with 6 challenges, with circuit_eval being the overlap challenge. These are subsets, meant to be **directional**, not exhaustive.\n\n`xjq`\n\n—**easy**(5 checkpoints): an XML, HTML, and JSON query CLI that grows from XPath into CSS selectors and structured output`file_backup`\n\n—**easy**(4 checkpoints): a scheduled backup tool that adds archive strategies, destinations, verification, and incremental state`dag_execution`\n\n—**hard**(3 checkpoints): a task-pipeline DSL with execution, caching, and dynamic cache overrides`circuit_eval`\n\n—**medium**(8 checkpoints): a circuit parser and evaluator that grows into vectors, three-valued logic, analysis, equivalence, and optimization`code_search`\n\n—**easy**(5 checkpoints): a multi-language code-search and rewrite tool with regex, structural patterns, AST selectors, and fixes`etl_pipeline`\n\n—**easy**(5 checkpoints): a JSON ETL pipeline that adds execution, branches, reusable definitions, and namespaced composition\n\nThere's an appendix at the end with all 30 checkpoints explained in detail, but I won't put that all here.\n\nand then I ran them across all four models, in parallel, with a fresh context window per checkpoint. All models got the same prompts, and a mix of harnesses:\n\n- Fable - Claude Code 2.1.219\n- Sol - Codex CLI 0.145.0\n- Kimi K3 - OpenCode 1.18.0\n- Expo (stealth) - OpenCode 1.18.0\n\nthe metric we decided on for the first benchmark is the **strict pass**: everything new is green including every regression test that was inherited from previous checkpoints. So I carried that forward here.\n\nA model fails a checkpoint if the solution has a **defect** - defects are detected by taking the models output, a CLI to run or in some cases e.g. an api server to poke at, and running a set of held-out black-box tests **against the produced entrypoint**.\n\n- Model writes code for ck1\n- Eval harness runs black-box tests against ck1\n- Model writes code for ck2\n- Eval runs black-box tests for ck1 and ck2\n- etc\n\nAgain, the strict pass criteria means that if a model bungles something in checkpoint 4, it can't pass the following checkpoints because that failing part of the code carries forward (unless the model indavertently fixes an eval case in checkpoint 6 that was broken in checkpoint 4, but we didn't see this happen in practice).\n\nWhile each model got a few strict passes, they all steadily accumulated defects throughout the run\n\nIf our definition of success is \"reached the final checkpoint with no defects\" then opus 5 failed all three problems, but it failed slightly-less-badly than the other models.\n\nAs far as cost vs. final defect rate goes, we see a totally different curve from the Opus 5 eval. The chart uses the share of final tests left failing so the three- and six-problem subsets can share one scale; Expo is not plotted because its saved cost is unavailable.\n\nFable and Sol tied for strict passes with 10 each. If you want a tiebreaker, Fable got 16 **isolated** passes while Sol got only 14.\n\n`circuit_eval`\n\nwas the one challenge shared with the previous run. The newer models earned more strict passes across its eight checkpoints,\n\nThe previous Opus 5 run still **technically** left the fewest final defects, but that result is kinda disqualified by the fact that Opus 5's version of the test suite only has 557 tests, and the newer suite we used for Kimi/Sol/Fable has 566 tests.\n\nYou can check the previous [post](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md#the-slop-meter) for a breakdown of the code quality metrics in play here.\n\nI reproduced the same spread chart from the last post, for the new run. Looks like fable got the highest growth in cloned lines (9x increase betwen ck1 and ck8!!)\n\nJust as with the Opus 5 bench set, these models are close on growth in most metrics, with Fable growing more in some slop metrics, and Sol/Kimi growing more in others. What's interesting is Sol and Expo did the best at reducing *normalized* cognitive complexity. Somehow not as well as sonnet 5 though...?\n\nAgain, I like that these measures are repeatable and don't use a model for judgement. But the link between any one of them and \"is this codebase easy to change and evolve\" is not really established.\n\nSol left 1,318 SLOC in persistent Python test files; the other agents also tested their work, but used shell scripts, fixtures, or temporary files that this narrow count excludes. So idk what to do with this chart. I guess the new frontier likes shell-scripts and more black-box / computer-use style testing?! (I think we need to dig into this data more, there may be a bug in reporting here.)\n\nCompact output was not always simple: Kimi had the highest single-function complexity, while Fable had the largest clone share.\n\nFor all models, a huge majority of the code lines tripped at least one of the benchmark's slop rules. These are the averages across each run's final problem snapshots:\n\nResults from this run:\n\n- Fable 5 —\n**86%** - GPT-5.6 Sol —\n**95%** - Kimi K3 —\n**82%** - Expo —\n**79%**\n\nPrevious results, for context:\n\n- opus 4.8 —\n**98%** - opus 5 —\n**93%** - sonnet 5 —\n**89%**\n\nAgain, I'd probably say that more than anything else, this is a sign that some of the code quality measures are a bit over-aggressive. But its nice to see the comparison\n\nAll four new runs ended `circuit_eval`\n\nwith higher mean cyclomatic complexity than the previous three models.\n\nThe numbers are all pretty close, but weirdly enough opus 5 had the least duplication by the end of circuit_eval. That could be a sign that opus did a better job of refactoring/reusing as it went, but my guess is that less duplication might also map onto \"less consistency\". Without digging into the code samples we can't say for sure (and I wanted to get this data out before going deep on that).\n\nThe new runs used far fewer callables than prior Opus 5, but their functions were more complex on average.\n\nAgain - here's what the first three checkpoints of circuit_eval ask for (full listing for all challenges in the appendix at the end):\n\n**ck1**— a CLI with`--help`\n\n,`--version`\n\n, a JSON output mode, and a`check`\n\ncommand that parses and validates a`.circ`\n\ncircuit file. Every signal is a single bit.**ck2**— an`eval`\n\ncommand: pass the circuit some inputs, get the outputs back. Still one bit per signal, standard boolean operators.**ck3**— signals become** vectors**.`data[7:0]`\n\ninstead of`data`\n\n, plus slicing, indexing, concatenation, new operators, and a width check on every operand.\n\nI haven't made progress on this part ([where does the time go!?](/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/side-quests/where-does-the-time-go.md)), but the idea is to have a small dumb model try each checkpoint.\n\nI might even make sense to have a small model do checkpoint N+1 for EVERY checkpoint and factor that into the pass rates for the smart model's checkpoint N - grading the quality of the codebase left behind.\n\nThe frontier is getting better, but I'm still not trusting them to run around lights off in my codebase. I had the vibe since the [week fable launched](https://x.com/dexhorthy/status/2064747631885398231), but this is just more proof that there are hard coding problems.\n\n**Next up: **\n\nI am prototyping a new prompt/flow for the harness - where we incorporate either/both of\n\n- Deterministic linters\n- LLM-based \"adversarial review\" with alternating models\n\nafter each checkpoint, and see if it makes the results (strict pass rates) better or worse.\n\nShameless plug - if you wanna ship code that doesn't suck, but still go really dang fast - check out [humanlayer.com](https://humanlayer.com) and [humanlayer.com/discord](https://humanlayer.com/discord).\n\ngood luck.\n\n🫡 -dex\n\n[Benchmarking Opus 5 on SlopCodeBench](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md)[SlopCodeBench paper](https://arxiv.org/html/2603.24755v1)[SlopCodeBench website](https://www.scbench.ai/)[SlopCodeBench runner](https://github.com/SprocketLab/slop-code-bench)[SlopCodeBench problem catalog](https://github.com/gabeorlanski/scb-problems)[Gabe Orlanski on X](https://x.com/GOrlanski)[Claude Code](https://github.com/anthropics/claude-code)— harness used for Fable[Codex CLI](https://github.com/openai/codex)— harness used for Sol[OpenCode](https://github.com/anomalyco/opencode)— harness used for Kimi K3 and Expo[Where Does the Time Go?](/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/side-quests/where-does-the-time-go.md)[HumanLayer](https://humanlayer.dev/)\n\nAll 30 checkpoints, in order, condensed from the prompts the models received. Each checkpoint arrives without revealing the later requirements.\n\n**xjq** — easy, developer tools, 5 checkpoints\n\n**ck1**— Build an XPath 1.0 CLI that reads XML or HTML from stdin, matches elements case-sensitively, normalizes text and attribute results, serializes the first matching XML node, and reports XPath or parse errors.**ck2**— Add CSS selectors, including custom direct and descendant`::text`\n\nmodes, plus`--text`\n\nand`--text-all`\n\nextraction with defined precedence and whitespace normalization.**ck3**— Auto-detect top-level JSON objects and arrays, convert them to a typed XML tree under`<root>`\n\n, preserve key order and numeric formatting, and reject keys that cannot form valid XML element names.**ck4**— Read from an optional input file in preference to stdin, accept UTF-8 BOMs, ignore extra positional arguments, and add`--first`\n\nand compact XML output.**ck5**— Add JSON export for XML element results, formalize output-flag precedence, and support XPath unions with defined behavior for text extraction and first-result selection.\n\n**file_backup** — easy, file systems, 4 checkpoints\n\n**ck1**— Build a YAML-driven backup scheduler that finds daily, weekly, and one-time jobs due within an inclusive time window, applies glob exclusions to mounted files, and emits deterministic JSONL events while simulating backup selection.**ck2**— Add optional`full`\n\n,`pack`\n\n, and`verify`\n\nstrategies: hash individual files, build deterministic size-bounded GNU tar archives, or verify files without copying, with strategy-specific JSONL events.**ck3**— Add backup destinations and SHA-256 incremental state for full and verify jobs, skipping files whose stored copies have not changed; pack jobs remain non-incremental at this checkpoint.**ck4**— Make pack jobs incremental by loading existing tar archives, repacking current files under the size limit, and reporting loaded, unchanged, updated, and skipped pack contents.\n\n**dag_execution** — hard, DSL, 3 checkpoints\n\n**ck1**— Build a CLI that parses a task-pipeline DSL and optional TOML config, type-checks parameters, evaluates control-flow and success expressions, runs required tasks and shell commands in order, enforces workspace and dependency rules, and writes task events plus detailed job JSONL records.**ck2**— Add per-task inputs and caches with content, TTL-based stale, and always-use strategies; support global cache settings and forced refreshes while restoring cached output, files, status, and success results.**ck3**— Add dynamic cache overrides inside`requires`\n\nexpressions through`CachedTask(...)`\n\n, merging omitted values from the base task and supporting both dotted nested fields and full nested-object replacement.\n\n**circuit_eval** — medium, simulation, 8 checkpoints\n\n**ck1**— Build a CLI with help, version, JSON output, and a`check`\n\ncommand that parses and validates scalar`.circ`\n\nfiles, including declarations, assignments, operator arity, undefined signals, duplicate assignments, and cycles.**ck2**— Add`eval`\n\nfor scalar two-valued circuits, with explicit or default inputs, extra-input handling, standard Boolean operators, and sorted text or JSON output.**ck3**— Add vector signals, sized and unsized literals, indexing, slicing, concatenation, strict width checks,`MUX`\n\n/`ITE`\n\n, reductions,`EQ`\n\n, and binary, hexadecimal, or decimal output formatting.**ck4**— Add three-valued evaluation with runtime`X`\n\nvalues, binary-only output, and defined unknown-value behavior for every Boolean, multiplexer, equality, and reduction operator.**ck5**— Let`check`\n\nand`eval`\n\nread`.circ`\n\n, JSON, and scalar BENCH circuits through explicit or extension-based format selection, with format-specific parsing and validation.**ck6**— Add`stats`\n\nfor structural metrics,`lint`\n\nfor unused-input and constant-output checks, and`dot`\n\nfor deterministic Graphviz export with optional output-cone filtering.**ck7**— Add deterministic`cone`\n\nextraction, exhaustive`truth-table`\n\noutput, and exhaustive or seeded randomized`equiv`\n\nchecks that return counterexamples and a distinct non-equivalence exit code.**ck8**— Add deterministic`opt`\n\noutput with configurable normalization, constant folding, algebra, common-subexpression elimination, dead-code removal, and fan-in passes, plus compact wire renaming, reports, BENCH export, and optional equivalence verification.\n\n**code_search** — easy, developer tools, 5 checkpoints\n\n**ck1**— Build a Python source-search CLI that applies exact-text and regular-expression rules, skips undecodable files, and emits deterministically ordered JSONL matches with one-based source ranges.**ck2**— Extend file discovery and rule filtering to JavaScript and C++, while retaining the same exact and regex matching and output rules.**ck3**— Add structure-aware`pattern`\n\nrules with required, optional, repeated, and escaped metavariables, reporting capture text and every capture range in deterministic order.**ck4**— Add AST node`selector`\n\nrules and replacement fixes, with dry-run previews, on-disk application, capture templates,`$MATCH`\n\n, escaped dollar signs, and deterministic overlap handling.**ck5**— Extend exact, regex, pattern, selector, capture, and fix behavior to Rust, Java, Go, and Haskell, bringing the supported language set to seven.\n\n**etl_pipeline** — easy, data processing, 5 checkpoints\n\n**ck1**— Build a stdin/stdout JSON CLI that validates and normalizes`select`\n\n,`filter`\n\n,`map`\n\n,`rename`\n\n, and`limit`\n\nsteps, including expression checks, canonical field handling, and structured error paths.**ck2**— Add`--execute`\n\nto run linear pipelines over in-memory JSON rows with a defined expression language, null and type behavior, column checks, transformed data, and row-count metrics.**ck3**— Add nested`branch`\n\nsteps with first-match routing, an optional final`otherwise`\n\nbranch, per-branch sub-pipelines, declaration-order concatenation, and nested error paths.**ck4**— Add top-level named definitions and parameterized`call`\n\nsteps, including`params.key`\n\nexpression access, unknown-definition errors, and direct or indirect recursion checks.**ck5**— Add namespaced pipeline libraries and a top-level`compose`\n\nlist that expands library references and inline step fragments for either normalized output or execution.", "url": "https://wpnews.pro/news/benchmarking-fable-sol-and-kimi-k3-on-slopcodebench", "canonical_source": "https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-sol-fable-kimi-on-slop-code-bench.md", "published_at": "2026-08-04 15:08:01+00:00", "updated_at": "2026-08-04 15:23:59.917054+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "developer-tools"], "entities": ["Fable", "Sol", "Kimi K3", "Expo", "SlopCodeBench", "HumanLayer", "UW Madison", "Gabe Orlanski"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-fable-sol-and-kimi-k3-on-slopcodebench", "markdown": "https://wpnews.pro/news/benchmarking-fable-sol-and-kimi-k3-on-slopcodebench.md", "text": "https://wpnews.pro/news/benchmarking-fable-sol-and-kimi-k3-on-slopcodebench.txt", "jsonld": "https://wpnews.pro/news/benchmarking-fable-sol-and-kimi-k3-on-slopcodebench.jsonld"}}