{"slug": "opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs", "title": "Opus 5.5 Formalized CPython in Lean and Surfaced 418 Bugs", "summary": "A machine-checked Lean model of CPython 3.15.2 built with Claude Opus 5.5 has produced 418 bug reports with reproducer scripts, David Andrews and Chao Zhang wrote on November 9, 2026. The project, started October 29, 2026, spans 613,318 Lean lines across 1,539 files, has accepted 4,071 of 26,004 tracked functions (15.7%) and checked 18,898 lemmas, with 3,834 top-level properties proved and 588 refuted. Andrews opened a Python.org discuss thread on November 9, 2026, filing only six crash reports initially — each verified on 3.15.2 and main — including bytearray.strip() use-after-free paths, ElementTree slice assignment reference bugs and functools.partial keyword-merge buffer issues, because automated October flooding made maintainers wary.", "body_md": "**Claude Opus 5.5 helped build a machine-checked Lean model of CPython 3.15.2 that has already produced 418 bug reports**, David Andrews and Chao Zhang wrote in [Formalizing CPython](https://broyojo.com/python-formalization/?ref=explainx) on **November 9, 2026**. The project is not a benchmark press release: it is a **line-by-line copy** of interpreter functions, **differentially tested** against real Python, then **reviewed by a second agent** before it counts toward progress.\n\nFor Python builders, the payoff is practical. You get a public list of **reproducer scripts** for corner-case failures, a reusable **agent workflow** for other codebases, and a sobering lesson on what \"formally verified\" must mean when **models** can lie even when **proof checkers** do not.\n\n## TL;DR\n\n| Metric | Value (Nov 9 snapshot) | \n|---|---|\n| Lean lines | 613,318 across 1,539 files | \n| Functions accepted | 4,071 / 26,004 tracked (15.7%) | \n| Checked theorems | 18,898 lemmas | \n| Proved vs refuted top-level props | 3,834 proved, 588 refuted | \n| Bug reports | 418 with reproducers | \n| Serious reachability | ~55 crash or memory-corruption class | \n| Primary model | Claude Opus 5.5 | \n| Review helper | GPT-6.1 Sol in Codex per Python.org thread | \n\n## How the pipeline works\n\nEvery function passes four steps published on the site:\n\n1. **Copy** — Rewrite C or Python source in Lean, preserving error messages and edge cases.\n2. **Test** — Run the Lean copy against CPython on**at least 50,000** generated inputs per function.\n3. **Prove** — State a property and machine-check it; if false, keep the counterexample.\n4. **Check** — A**second agent** that did not author the copy reviews it and asks whether a**broken** implementation could still pass the proof.\n\nThat last gate exists because an early breadth-first pass produced proofs that were **logically valid but semantically empty**. The site's \"proofs that prove nothing\" section gives the canonical bad example: \"if this function returns a number, the number is at least zero\" — true even for a function that **always crashes**.\n\nThe project started **October 29, 2026** and explicitly credits **[julia-in-lean](https://github.com/MilesCranmer/julia-in-lean?ref=explainx)** as inspiration. Julia-in-lean reported **hundreds** of Base bugs through a similar copy-and-test approach.\n\n## What developers are saying\n\nAndrews opened a [Python.org discuss thread](https://discuss.python.org/t/formalizing-cpython-in-lean-findings-from-a-machine-checked-model-and-how-youd-like-them/109978?ref=explainx) on **November 9, 2026** to coordinate with CPython maintainers before flooding the tracker.\n\nHe wrote that the team filed **only six crash reports** initially — each personally verified on **3.15.2 and main** — because automated October flooding made maintainers wary. Examples he named include `bytearray.strip()` use-after-free paths, ElementTree slice assignment reference bugs, and `functools.partial` keyword-merge buffer issues.\n\nThe thread is the right place to watch for **maintainer triage** and duplicate linking; Andrews says findings that match existing issues reference those issues instead of re-filing.\n\n## Scale and coverage snapshot\n\nOpus 5.5 CPython Lean formalization project overview with checked theorem counts\n*Image: [David Andrews and Chao Zhang](https://broyojo.com/python-formalization/?ref=explainx)*\n\nThe landing page tracks coverage by subsystem. At publication:\n\n- **Grammar rules** hit**100%** of reviewed grammar targets.\n- **Built-in types** sit around**35%** accepted in the progress chart narrative.\n- **Interpreter core** remains early (**~10%** in the visualization copy).\n- **Standard library** Python modules are**~9%** , with most pure-Python stdlib not yet formalized.\n\nMultithreading is **out of scope for now**, which matters if you hoped for a concurrency proof story.\n\n## Bug taxonomy, not vanity counts\n\nChalkboard proof steps for Opus 5.5 CPython Lean formalization bug hunting\n\nThe project classifies all **418** reports. Top buckets on the site:\n\n| Category | Count | \n|---|---|\n| C and Python implementations disagree | 129 | \n| Other | 122 | \n| Wrong result | 56 | \n| Crash | 28 | \n| Memory corruption | 27 | \n| Doesn't match docs | 26 | \n| Wrong error or message | 18 | \n| Missing input check | 12 | \n\nAuthors stress that **~55** issues can **crash or corrupt memory** using normal Python, often when **user callbacks mutate containers** while C code still holds references. CPython does **not** treat hostile Python as a security boundary, so these are **ordinary correctness bugs**, not CVE-class threat models — but they still break production.\n\nSix exemplar reproducers on the site include:\n\n- **`vars(b).setdefault(Key.NAME, ...)` segfault** with StrEnum keys on attribute dicts.\n- **ElementTree backwards slice assignment** duplicating children and dropping references.\n- **`a2b_base32` padding ignored** changing results by padding position in 3.15.\n\nFull scripts live in the [GitHub repository](https://github.com/Broyojo/python-formalization?ref=explainx).\n\n## When correct proofs were still wrong\n\nThe post documents failures where Lean accepted a theorem but **real Python disagreed**:\n\n- **JSON encoder proofs** assumed allocations never fail, \"proving\" absurd nesting depths including**16 TB** indentation strings until memory failure was modeled.\n- **Pickle depth proofs** missed a**one-time codec import** that adds**nine** hidden stack frames on first use in a fresh process.\n- **Set memory reuse** turned an accepted proof into a**real bug** once address reuse was modeled — now on the report list.\n\nThat section is the article's core lesson for teams tempted to automate proofs without **ground-truth regression**.\n\n## How this differs from Anthropic's SDK verification story\n\nexplainx.ai previously covered [Boris Cherny's Opus 5.5 Lean verification of the Claude Agent SDK](https://explainx.ai/blog/boris-cherny-opus-5-5-lean-tla-formal-verification-sdk-2026): **16 PRs**, **24 bugs**, **1,529 theorems**, **zero sorry placeholders** in that formal pass.\n\nThe CPython project is larger in **lines** and **scope**, but smaller in **percent complete**. It also targets **CPython behavior**, not a single vendor SDK — so successful bugs land on **python/cpython**, not one company's repo.\n\nAnthropic's wider formal-methods push — [123-library Rocq work](https://explainx.ai/blog/anthropic-claude-formal-verification-123-libraries-rocq-risc-v-proofs-2026), [Fermat-related Lean experiments](https://explainx.ai/blog/anthropic-claude-fermats-last-theorem-lean-proof-2026), and [OpenAI's Lean RL environments](https://explainx.ai/blog/openai-math-lean-rl-environments-hugging-face-harbor-369-tasks-2026) — shares tooling themes, but Andrews and Zhang's effort is **independent research**, not an Anthropic product launch.\n\n## What agent builders can reuse\n\nThe repository publishes operational artifacts meant for reuse:\n\n- **AGENTS.md** — Instructions for modeling agents.\n- **REVIEW_PROMPT.md** — Independent reviewer checklist (\"could a broken version pass?\").\n- **coverage/methods** — Per-function status tracking.\n\nTheir guidance generalizes beyond Python:\n\n- Pin **one exact upstream version** .\n- Inventory **all functions** up front so progress cannot hide gaps.\n- Copy **source** , not manual prose.\n- Model **hostile user callbacks** in container APIs.\n- **Test before prove** .\n- **Re-run real interpreter checks** after proof acceptance.\n\nIf you run coding agents on infrastructure code, pair formal passes with **runtime guardrails** so agent-proposed patches cannot ship without human gates. Tools like [AgentBeam](https://agentbeam.com/?ref=explainx) address a different layer — stopping dangerous agent **actions** at execution time — while this project addresses **semantic fidelity** of the interpreter underneath.\n\n## Limits and honest caveats\n\n- **15.7% function acceptance** means most of CPython is still unmodeled; absence of a report is not evidence of correctness.\n- **Opus 5.5 wrote most Lean** ; human direction and review still dominate acceptance counting.\n- **Six GitHub issues filed** at thread time implies hundreds of reports remain**repository-local** pending maintainer appetite.\n- **Concurrent semantics** are explicitly deferred.\n- Proofs certify statements about the **Lean copy** , not a blanket certificate for all of Python.\n\n## Reading the bug reports as a Python maintainer\n\nIf you maintain libraries on top of CPython, skim the report repository by **category** before chasing individual issues:\n\n- **C vs Python disagree** — Often means fast C paths and Python fallbacks diverge on edge types. Fix strategy is usually**unify semantics** or**document intentional differences** .\n- **Wrong error types** — Breaks`except` hierarchies silently. High value for web frameworks parsing headers or email.\n- **Memory corruption with callbacks** — Common in**during-loop mutation** patterns; audit C extensions that borrow references without re-fetching.\n- **Doc mismatches** — Low severity individually, but agents trained on docs**will hallucinate APIs** if docs lie.\n\nThe project's six initially filed crash issues on discuss.python.org are instructive because Andrews personally bisected them on **main** and **3.15.2**. That is the bar for accepting **agent-filed** bugs upstream: human-readable narrative, minimal reproducer, duplicate search, and explicit severity.\n\nTeams using **Opus-class agents** on application code should import the same bar: agents file issues; humans merge only when reproducers survive review.\n\n## Cost and effort signals\n\nThe site does not publish dollar spend, but scale hints at **multi-week agent throughput**:\n\n- **613k Lean lines** in under two weeks of calendar time since October 29.\n- **50k inputs per function** differential tests multiply CPU hours quickly.\n- **Second-agent review** doubles model calls on every accepted function.\n\nFor budgeting, treat formalization as **CI plus LLM bills**, not a one-shot prompt. The payoff is **structured defect lists** you can attach to release gates.\n\nDownstream, if you ship **Python-heavy agent frameworks**, these bugs matter because agents generate **weirdly valid** programs that stress rarely used stdlib corners. Pair formal hunts with fuzzing traditions from [theorem verification enrollments](https://explainx.ai/blog/theorem-oss-verify-free-formal-verification-open-source-fuzz-targets-rocq-how-to-enroll-2026) and runtime sandboxes.\n\n## Try it yourself\n\nClone [github.com/Broyojo/python-formalization](https://github.com/Broyojo/python-formalization?ref=explainx), read **CONTRIBUTING.md**, and pick a file marked as not yet started from **coverage/methods**. Independent review PRs count the same as new models.\n\nFor comparison baselines, see older semantic work cited on the site — λπ, K-framework Python 3.3, JavaScript mechanizations, and CompCert/seL4 as the gold standard for **whole-program** verification.\n\n## Related reading\n\n- [Boris Cherny Opus 5.5 Lean SDK verification](https://explainx.ai/blog/boris-cherny-opus-5-5-lean-tla-formal-verification-sdk-2026) — Vendor SDK formal pass contrast.\n- [Anthropic formal verification libraries](https://explainx.ai/blog/anthropic-claude-formal-verification-123-libraries-rocq-risc-v-proofs-2026) — Lab-scale proof programs.\n- [OpenAI Lean RL environments](https://explainx.ai/blog/openai-math-lean-rl-environments-hugging-face-harbor-369-tasks-2026) — Training agents for proof tasks.\n- [Axiom Stacks Lean algebra project](https://explainx.ai/blog/axiom-stacks-project-lean-16m-lines-ai-agents-algebraic-geometry-apache-2026) — Large math formalization with agents.\n- [Theorem OSS verification enrollment](https://explainx.ai/blog/theorem-oss-verify-free-formal-verification-open-source-fuzz-targets-rocq-how-to-enroll-2026) — Community formal verification tooling.\n- [Jose Valim on languages in the AI era](https://explainx.ai/blog/jose-valim-programming-languages-ai-era-agent-tooling-2026) — Why runtimes still matter for agents.", "url": "https://wpnews.pro/news/opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs", "canonical_source": "https://explainx.ai/blog/opus-5-5-cpython-lean-formalization-418-bugs-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-10 00:00:00+00:00", "updated_at": "2026-11-10 04:52:29.429380+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "large-language-models", "ai-research"], "entities": ["Claude Opus 5.5", "CPython 3.15.2", "Lean", "David Andrews", "Chao Zhang", "GPT-6.1 Sol", "Python.org", "julia-in-lean"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs", "markdown": "https://wpnews.pro/news/opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs.md", "text": "https://wpnews.pro/news/opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs.txt", "jsonld": "https://wpnews.pro/news/opus-5-5-formalized-cpython-in-lean-and-surfaced-418-bugs.jsonld"}}