Claude Opus 5.5 helped build a machine-checked Lean model of CPython 3.15.2 that has already produced 418 bug reports, David Andrews and Chao Zhang wrote in Formalizing CPython on November 9, 2026. The project is not a benchmark press release: it is a line-by-line copy of interpreter functions, differentially tested against real Python, then reviewed by a second agent before it counts toward progress.
For Python builders, the payoff is practical. You get a public list of reproducer scripts for corner-case failures, a reusable agent workflow for other codebases, and a sobering lesson on what "formally verified" must mean when models can lie even when proof checkers do not.
TL;DR #
| Metric | Value (Nov 9 snapshot) |
|---|---|
| Lean lines | 613,318 across 1,539 files | | Functions accepted | 4,071 / 26,004 tracked (15.7%) | | Checked theorems | 18,898 lemmas | | Proved vs refuted top-level props | 3,834 proved, 588 refuted | | Bug reports | 418 with reproducers | | Serious reachability | ~55 crash or memory-corruption class | | Primary model | Claude Opus 5.5 | | Review helper | GPT-6.1 Sol in Codex per Python.org thread |
How the pipeline works #
Every function passes four steps published on the site:
- Copy β Rewrite C or Python source in Lean, preserving error messages and edge cases.
- Test β Run the Lean copy against CPython onat least 50,000 generated inputs per function.
- Prove β State a property and machine-check it; if false, keep the counterexample.
- Check β Asecond agent that did not author the copy reviews it and asks whether abroken implementation could still pass the proof.
That last gate exists because an early breadth-first pass produced proofs that were logically valid but semantically empty. The site's "proofs that prove nothing" section gives the canonical bad example: "if this function returns a number, the number is at least zero" β true even for a function that always crashes.
The project started October 29, 2026 and explicitly credits julia-in-lean as inspiration. Julia-in-lean reported hundreds of Base bugs through a similar copy-and-test approach.
What developers are saying #
Andrews opened a Python.org discuss thread on November 9, 2026 to coordinate with CPython maintainers before flooding the tracker.
He wrote that the team filed only six crash reports initially β each personally verified on 3.15.2 and main β because automated October flooding made maintainers wary. Examples he named include bytearray.strip() use-after-free paths, ElementTree slice assignment reference bugs, and functools.partial keyword-merge buffer issues.
The thread is the right place to watch for maintainer triage and duplicate linking; Andrews says findings that match existing issues reference those issues instead of re-filing.
Scale and coverage snapshot #
Opus 5.5 CPython Lean formalization project overview with checked theorem counts
Image: David Andrews and Chao Zhang The landing page tracks coverage by subsystem. At publication:
- Grammar rules hit100% of reviewed grammar targets.
- Built-in types sit around35% accepted in the progress chart narrative.
- Interpreter core remains early (~10% in the visualization copy).
- Standard library Python modules are**~9%** , with most pure-Python stdlib not yet formalized.
Multithreading is out of scope for now, which matters if you hoped for a concurrency proof story.
Bug taxonomy, not vanity counts #
Chalkboard proof steps for Opus 5.5 CPython Lean formalization bug hunting
The project classifies all 418 reports. Top buckets on the site:
| Category | Count |
|---|---|
| C and Python implementations disagree | 129 |
| Other | 122 |
| Wrong result | 56 |
| Crash | 28 |
| Memory corruption | 27 |
| Doesn't match docs | 26 |
| Wrong error or message | 18 |
| Missing input check | 12 |
Authors stress that ~55 issues can crash or corrupt memory using normal Python, often when user callbacks mutate containers while C code still holds references. CPython does not treat hostile Python as a security boundary, so these are ordinary correctness bugs, not CVE-class threat models β but they still break production.
Six exemplar reproducers on the site include:
vars(b).setdefault(Key.NAME, ...)segfault with StrEnum keys on attribute dicts.- ElementTree backwards slice assignment duplicating children and dropping references.
a2b_base32padding ignored changing results by padding position in 3.15.
Full scripts live in the GitHub repository.
When correct proofs were still wrong #
The post documents failures where Lean accepted a theorem but real Python disagreed:
- JSON encoder proofs assumed allocations never fail, "proving" absurd nesting depths including16 TB indentation strings until memory failure was modeled.
- Pickle depth proofs missed aone-time codec import that addsnine hidden stack frames on first use in a fresh process.
- Set memory reuse turned an accepted proof into areal bug once address reuse was modeled β now on the report list.
That section is the article's core lesson for teams tempted to automate proofs without ground-truth regression.
How this differs from Anthropic's SDK verification story #
explainx.ai previously covered Boris Cherny's Opus 5.5 Lean verification of the Claude Agent SDK: 16 PRs, 24 bugs, 1,529 theorems, zero sorry placeholders in that formal pass.
The CPython project is larger in lines and scope, but smaller in percent complete. It also targets CPython behavior, not a single vendor SDK β so successful bugs land on python/cpython, not one company's repo.
Anthropic's wider formal-methods push β 123-library Rocq work, Fermat-related Lean experiments, and OpenAI's Lean RL environments β shares tooling themes, but Andrews and Zhang's effort is independent research, not an Anthropic product launch.
What agent builders can reuse #
The repository publishes operational artifacts meant for reuse:
- AGENTS.md β Instructions for modeling agents.
- REVIEW_PROMPT.md β Independent reviewer checklist ("could a broken version pass?").
- coverage/methods β Per-function status tracking.
Their guidance generalizes beyond Python:
- Pin one exact upstream version .
- Inventory all functions up front so progress cannot hide gaps.
- Copy source , not manual prose.
- Model hostile user callbacks in container APIs.
- Test before prove .
- Re-run real interpreter checks after proof acceptance.
If you run coding agents on infrastructure code, pair formal passes with runtime guardrails so agent-proposed patches cannot ship without human gates. Tools like AgentBeam address a different layer β stopping dangerous agent actions at execution time β while this project addresses semantic fidelity of the interpreter underneath.
Limits and honest caveats #
- 15.7% function acceptance means most of CPython is still unmodeled; absence of a report is not evidence of correctness.
- Opus 5.5 wrote most Lean ; human direction and review still dominate acceptance counting.
- Six GitHub issues filed at thread time implies hundreds of reports remainrepository-local pending maintainer appetite.
- Concurrent semantics are explicitly deferred.
- Proofs certify statements about the Lean copy , not a blanket certificate for all of Python.
Reading the bug reports as a Python maintainer #
If you maintain libraries on top of CPython, skim the report repository by category before chasing individual issues:
- C vs Python disagree β Often means fast C paths and Python fallbacks diverge on edge types. Fix strategy is usuallyunify semantics ordocument intentional differences .
- Wrong error types β Breaks
excepthierarchies silently. High value for web frameworks parsing headers or email. - Memory corruption with callbacks β Common induring-loop mutation patterns; audit C extensions that borrow references without re-fetching.
- Doc mismatches β Low severity individually, but agents trained on docswill hallucinate APIs if docs lie.
The project's six initially filed crash issues on discuss.python.org are instructive because Andrews personally bisected them on main and 3.15.2. That is the bar for accepting agent-filed bugs upstream: human-readable narrative, minimal reproducer, duplicate search, and explicit severity.
Teams using Opus-class agents on application code should import the same bar: agents file issues; humans merge only when reproducers survive review.
Cost and effort signals #
The site does not publish dollar spend, but scale hints at multi-week agent throughput:
- 613k Lean lines in under two weeks of calendar time since October 29.
- 50k inputs per function differential tests multiply CPU hours quickly.
- Second-agent review doubles model calls on every accepted function.
For budgeting, treat formalization as CI plus LLM bills, not a one-shot prompt. The payoff is structured defect lists you can attach to release gates. Downstream, if you ship Python-heavy agent frameworks, these bugs matter because agents generate weirdly valid programs that stress rarely used stdlib corners. Pair formal hunts with fuzzing traditions from theorem verification enrollments and runtime sandboxes.
Try it yourself #
Clone github.com/Broyojo/python-formalization, read CONTRIBUTING.md, and pick a file marked as not yet started from coverage/methods. Independent review PRs count the same as new models.
For comparison baselines, see older semantic work cited on the site β Ξ»Ο, K-framework Python 3.3, JavaScript mechanizations, and CompCert/seL4 as the gold standard for whole-program verification.
Related reading #
- [Boris Cherny Opus 5.5 Lean SDK verification](https://explainx.ai/blog/boris-cherny-opus-5-5-lean-tla-formal-verification-sdk-2026) β Vendor SDK formal pass contrast.
- [Anthropic formal verification libraries](https://explainx.ai/blog/anthropic-claude-formal-verification-123-libraries-rocq-risc-v-proofs-2026) β Lab-scale proof programs.
- [OpenAI Lean RL environments](https://explainx.ai/blog/openai-math-lean-rl-environments-hugging-face-harbor-369-tasks-2026) β Training agents for proof tasks.
- [Axiom Stacks Lean algebra project](https://explainx.ai/blog/axiom-stacks-project-lean-16m-lines-ai-agents-algebraic-geometry-apache-2026) β Large math formalization with agents.
- [Theorem OSS verification enrollment](https://explainx.ai/blog/theorem-oss-verify-free-formal-verification-open-source-fuzz-targets-rocq-how-to-enroll-2026) β Community formal verification tooling.
- [Jose Valim on languages in the AI era](https://explainx.ai/blog/jose-valim-programming-languages-ai-era-agent-tooling-2026) β Why runtimes still matter for agents.