{"slug": "formalizing-cpython", "title": "Formalizing CPython", "summary": "An AI agent built on Opus 5.5 rewrote CPython's functions in Lean, producing 613,318 lines of machine-checked Lean across 1,539 files with 18,898 theorems and lemmas, and surfaced 418 bug reports in Python — about 55 of which can crash the interpreter or corrupt memory from ordinary Python code. Each Lean rewrite copies the original C line by line and is tested against real Python on tens of thousands of inputs before proving, and a function only counts once a second agent that didn't write it verifies the work; 4,071 functions have passed that check. The effort targets the absence of a precise written definition of Python, since proofs force exact specification of edge cases where CPython's C implementation diverges from assumptions.", "body_md": "# Formalizing CPython\n\nAI agents wrote 613,318 lines of Lean that copy Python's internals line by line, tested every piece against the real Python, had a computer check every proof, and turned up 418 bug reports in Python along the way.\n\n## TL;DR\n\n- We pointed Opus 5.5 at CPython, the program that runs almost all Python code, and had it rewrite its functions in [Lean](https://lean-lang.org) , a language for writing machine-checked proofs.\n- Each rewrite copies the original line by line and is tested against real Python on tens of thousands of inputs before anyone tries to prove anything about it.\n- A function only counts once a second agent, which didn't write it, checks the work and agrees it holds up. 4,071 functions have passed that check so far.\n- The result is 613,318 lines of Lean across 1,539 files, with 18,898 theorems and lemmas, all checked by Lean's proof checker on every build.\n- Along the way the proofs turned up 418 bug reports. Most are small, but about 55 can crash the interpreter or corrupt memory from ordinary Python code.\n\n## Why bother\n\nPython doesn't have a precise written definition. There is a reference manual, but in practice \"Python\" means whatever CPython does, and CPython is hundreds of thousands of lines of C plus a large standard library. The manual covers normal use. It says little about the odd cases, and that's where bugs hide: an object that changes a dictionary while that dictionary is searching for it, an error thrown halfway through an update, a number one past the largest the C code was written for.\n\nWriting a proof about a function forces you to say exactly what it does in all of those cases. You can't leave anything vague, because the proof checker won't accept it. That makes proofs a good way to find the places where the code doesn't do what everyone assumes.\n\nWe wanted to see how far this goes on a code base as large and old as CPython, and whether the results stay trustworthy when most of the work is done by AI.\n\n## How it works\n\nEvery function goes through the same four steps.\n\nThe code at the top of this page is step one for a small piece of CPython: the C function that works out how many numbers are in a `range`, next to its Lean copy. The copy keeps the C code's exact arithmetic, including how its numbers wrap around when they get too big. The proof about it shows that this function, the slower version CPython uses for very large numbers, and the code that chooses between them always give the same, correct count.\n\n## Proofs that prove nothing\n\nThe hardest part wasn't getting AI agents to write proofs. It was getting proofs that actually mean something.\n\nOur first attempt asked the agents for breadth: copy every function and prove at least one thing about each. They did. Then a separate review found that fewer than half of the functions it checked held up, and in one file only 21%. The proofs were all correct. They just didn't say anything useful. For example, \"if this function returns a number, the number is at least zero\" is true, but a function that always crashes also satisfies it.\n\nSo the reviewer now asks one question about every proof: **could a broken version of the function pass it too?**\n\n| Version of the function | \"If it returns, the answer is ≥ 0\" | \"It always returns the right count\" | \n|---|---|---|\n| The real one | passes | passes | \n| Always crashes | passes | caught | \n| Returns 0 for very large numbers | passes | caught | \n| Off by one when counting down | passes | caught | \n\nFixing a weak proof usually means going back and making the copy more realistic, for example letting a user's code throw an error in the middle of a function. The charts below show how that went for ten files. Each point is one round of review. The lines sometimes go down: when a reviewer finds that an accepted proof is wrong about real Python, everything that relied on it is taken back out.\n\n## When a correct proof is wrong anyway\n\nA checked proof guarantees that a statement follows from the copy. It doesn't guarantee the copy matches real Python, and testing only covers the inputs you thought to try. So every accepted statement is also run against real Python. That caught several proofs that were valid and still wrong.\n\n### The 16-terabyte string\n\nThe copy of Python's fast JSON encoder assumed memory requests always succeed. So it \"proved\" the encoder could build its indentation strings for any nesting depth, including one that needs 16 terabytes of memory. Real Python runs out of memory. The fixed copy now treats every memory request as something that can fail.\n\n### Nine extra steps, just once\n\nA proof said Python's pickle module never goes deeper than a certain number of nested calls. That turned out true except in a brand-new Python process. There, the first time it encodes a string, it quietly loads a text codec, and that load uses nine more nested calls. Our tests never saw it, because the test harness had already loaded the codec.\n\n### Error types have families\n\nIn Python, catching a `TypeError` also catches any more specific error built on top of it. Three separate copies checked an error's exact name instead. All three had proofs that a custom error type could break.\n\n### Memory that comes back\n\nThe copy of Python's `set` assumed that once memory is freed, the same address never comes back. Real memory allocators reuse addresses all the time. CPython uses a \"did the address change?\" check to notice when user code has swapped out the set's internals. With reuse, that check can be fooled. Fixing the copy turned an accepted proof into a real memory bug, now one of our bug reports.\n\n## What we found\n\nWhen a proof fails because Python itself is wrong, and not the copy, we write it up as a bug report. Each report includes a short script that shows the bug on Python 3.15.2 (for most, also on the development version), a search for duplicates already reported, and an honest rating of how much it matters.\n\nMost are small: an error message that's wrong, an edge case that behaves differently from the documentation, two versions of a function that disagree. About 55 are more serious and can crash Python or corrupt its memory using nothing but normal Python code, usually by having user code change a container while C code is in the middle of using it. Python doesn't treat running hostile Python code as a security boundary, so these are ordinary bugs. The ones not already on CPython's tracker are being reported to the project, starting with this post. Six small examples of different kinds:\n\n### A dictionary call that crashes Python\n\nCalling `setdefault` on an object's attribute dictionary with a `StrEnum` key crashes the interpreter, if another object of the same class has that attribute and this one doesn't.\n\n```\nclass Key(StrEnum):\n    NAME = \"name\"\nclass User: pass\na = User(); a.name = \"ada\"\nb = User(); b.age = 36\nvars(b).setdefault(Key.NAME, \"anon\")\n# Segmentation fault\n```\n\n### An XML edit that corrupts memory\n\nAssigning to a backwards slice of an ElementTree element should insert, as it does for a list. Instead it duplicates one child without taking a reference and loses another.\n\n```\ne = ET.Element(\"root\")\ne[:] = [ET.Element(f\"a{i}\") for i in range(7)]\ne[5:2] = [ET.Element(\"X\")]\n[c.tag for c in e]\n# 'a2' appears twice, 'a4' is gone\n```\n\n### An email header that breaks the parser\n\nA Message-ID with a space inside square brackets makes Python's email parser fail with an internal error. The header comes from whoever sent the email.\n\n```\nmsg = email.message_from_string(\n    \"Message-ID: <a@[ x]>\\n\\nhi\\n\",\n    policy=policy.default)\nmsg[\"Message-ID\"]\n# UnboundLocalError\n```\n\n### Base32 padding that depends on position\n\nNew in Python 3.15, the base32 decoder can ignore chosen characters. If `=` is one of them, where it appears still changes the result.\n\n```\na2b_base32(b\"AAAAAAA\",  ignorechars=b\"=\")\n# Error: Incorrect padding\na2b_base32(b\"AAAA=AAA\", ignorechars=b\"=\")\n# b'\\x00\\x00\\x00\\x00'\n```\n\n### An enum error that disappears\n\nGiving an enum value to a second member should raise an error. If the class's `__repr__` is broken, the error is swallowed and the call quietly does nothing.\n\n``` python\nclass W(Enum):\n    A = 1; B = 2\n    def __repr__(self): return 5\nW.B._add_value_alias_(3)\nW.A._add_value_alias_(3)  # returns None\n```\n\n### Two picklers, two limits\n\nPython has a C pickler and a pure-Python one. They give up at different nesting depths. This difference was first reported in 2008 and is still open.\n\n```\nv = []\nfor _ in range(400): v = [v]\npickle.dumps(v, 5)    # works\npickle._dumps(v, 5)   # RecursionError\n```\n\nEvery report, with its script, is in the [repository](https://github.com/Broyojo/python-formalization/tree/main/bugs).\n\n## Do this for your own project\n\nNone of this is specific to Python. It works for anything with source code you can read and a working version you can run: another language, a library, a file format, a compiler. What made the difference for us:\n\n1. **Pick one exact version** and stick to it. Every copy points at specific lines in that version.\n2. **List everything first.** Count progress against the full list, so nothing gets skipped quietly.\n3. **Copy the code, not the documentation.** A copy written from the docs agrees with the docs, and the bugs are where the code and the docs disagree.\n4. **Let user code misbehave.** Model callbacks that throw errors or change things mid-operation. That's where the worst bugs were.\n5. **Test before you prove.** Proving things about a wrong copy wastes time.\n6. **When something is false, keep the counterexample.** Don't water the statement down until it becomes provable.\n7. **Check proved statements against the real thing.** A valid proof about a wrong copy is the most misleading result you can get.\n8. **Count only what an independent reviewer accepts,** and have that reviewer ask whether a broken version would pass.\n\nThe instructions our agents worked from are in the repository and meant to be reused: [AGENTS.md](https://github.com/Broyojo/python-formalization/blob/main/AGENTS.md) for the agents doing the work, [REVIEW_PROMPT.md](https://github.com/Broyojo/python-formalization/blob/main/coverage/REVIEW_PROMPT.md) for the reviewers, and the full list of functions with their status in [coverage/methods](https://github.com/Broyojo/python-formalization/tree/main/coverage/methods).\n\n## Related work\n\nThe closest project is **julia-in-lean**, which did this for the Julia language and directly inspired this one<sup>[1](#r1)</sup>. It modelled Julia's standard library in Lean, tested the models against Julia, recorded about 400 bugs and filed 75. Its finding that 128 of 130 quick proofs had quietly assumed away the hard case shaped our review rules. A recent paper models NVIDIA's **PTX** GPU instruction set in Lean to prove GPU kernels correct<sup>[2](#r2)</sup>, building on earlier formal work on PTX's memory model<sup>[3](#r3)</sup>.\n\nFor Python specifically, **λπ**<sup>[4](#r4)</sup> and a **K framework** semantics of Python 3.3<sup>[5](#r5)</sup> described the language mathematically; this project copies CPython's actual code instead. Similar efforts exist for JavaScript<sup>[6](#r6),[7](#r7)</sup> and WebAssembly<sup>[8](#r8)</sup>, and CompCert<sup>[9](#r9)</sup> and seL4<sup>[10](#r10)</sup> remain the high bar for fully verified software. Comparing two implementations on the same inputs, the testing half of this project, goes back decades<sup>[11](#r11),[12](#r12)</sup>.\n\n## Where it stands\n\nThe project started on 29 October 2026. 4,071 of the 26,004 functions worth checking (15.7%) have been copied, tested, proved and accepted by review. The Lean code is 613,318 lines in 1,539 files, containing 18,898 theorems and lemmas; 3,834 of the top-level properties are proved and 588 are proved false with a counterexample. Most of Python is still ahead, especially the large part of the standard library written in Python itself. Multithreading is out of scope for now. In these 57 files, every function copied so far has passed review. The 30 with the most accepted functions:\n\n- `Python/bytecodes.c` 310\n- `Grammar/python.gram` 278\n- `Lib/collections/__init__.py` 184\n- `Lib/_pydecimal.py` 160\n- `Lib/_pydatetime.py` 150\n- `Modules/_struct.c` 122\n- `Objects/longobject.c` 119\n- `Objects/listobject.c` 114\n- `Objects/memoryobject.c` 107\n- `Objects/dictobject.c` 104\n- `Objects/bytesobject.c` 100\n- `Objects/exceptions.c` 97\n- `Objects/unicodeobject.c` 94\n- `Objects/bytearrayobject.c` 93\n- `Modules/arraymodule.c` 84\n- `Lib/_collections_abc.py` 81\n- `Modules/_operator.c` 73\n- `Modules/itertoolsmodule.c` 72\n- `Modules/_collectionsmodule.c` 64\n- `Python/codecs.c` 61\n- `Lib/operator.py` 59\n- `Lib/_pyio.py` 58\n- `Objects/typeobject.c` 57\n- `Lib/weakref.py` 54\n- `Objects/genobject.c` 46\n- `Lib/urllib/parse.py` 46\n- `Parser/action_helpers.c` 43\n- `Modules/_zoneinfo.c` 43\n- `Objects/floatobject.c` 38\n- `Objects/tupleobject.c` 37\n\nThe project is open, and help is welcome. Most of CPython is still unmodelled: pick a file whose functions are still marked `todo` in `coverage/methods`, follow `AGENTS.md`, and send a pull request. Independent reviews of existing work count as much as new models. See [CONTRIBUTING.md](https://github.com/Broyojo/python-formalization/blob/main/CONTRIBUTING.md).\n\n## Cite\n\n```\n@misc{andrews2026formalizing,\n  author       = {Andrews, David and Zhang, Chao},\n  title        = {Formalizing {CPython}: A Lean Model of {CPython} 3.15.2},\n  year         = {2026},\n  howpublished = {\\url{https://broyojo.com/python-formalization/}},\n  note         = {Code: \\url{https://github.com/Broyojo/python-formalization}}\n}\n```\n\n## References\n\n1. M. Cranmer. [Formalizing Julia in Lean (and finding 400+ bugs in Base)](https://discourse.julialang.org/t/formalizing-julia-in-lean-and-finding-400-bugs-in-base/140243) . Julia Discourse, 2026. Code:[MilesCranmer/julia-in-lean](https://github.com/MilesCranmer/julia-in-lean) .\n2. L. Koska, T. Koska, B. Koska. [Formalizing PTX in Lean to Prove GPU Kernel Correctness](https://arxiv.org/abs/2611.04447) . arXiv:2611.04447, 2026.\n3. D. Lustig, S. Sahasrabuddhe, O. Giroux. [A Formal Analysis of the NVIDIA PTX Memory Consistency Model](https://doi.org/10.1145/3297858.3304043) . ASPLOS 2019.\n4. J. G. Politz, A. Martinez, M. Milano, S. Warren, D. Patterson, J. Li, A. Chitipothu, S. Krishnamurthi. [Python: The Full Monty. A Tested Semantics for the Python Programming Language](https://doi.org/10.1145/2509136.2509536) . OOPSLA 2013.\n5. D. Guth. [A Formal Semantics of Python 3.3](https://ideals.illinois.edu/items/45257) . M.S. thesis, University of Illinois at Urbana-Champaign, 2013. Code:[runtimeverification/python-semantics](https://github.com/runtimeverification/python-semantics) .\n6. M. Bodin, A. Charguéraud, D. Filaretti, P. Gardner, S. Maffeis, D. Naudziuniene, A. Schmitt, G. Smith. [A Trusted Mechanised JavaScript Specification](https://doi.org/10.1145/2535838.2535876) . POPL 2014.\n7. D. Park, A. Ştefănescu, G. Roşu. [KJS: A Complete Formal Semantics of JavaScript](https://doi.org/10.1145/2737924.2737991) . PLDI 2015.\n8. C. Watt. [Mechanising and Verifying the WebAssembly Specification](https://doi.org/10.1145/3167082) . CPP 2018.\n9. X. Leroy. [Formal Verification of a Realistic Compiler](https://doi.org/10.1145/1538788.1538814) . Communications of the ACM 52(7), 2009.\n10. G. Klein, K. Elphinstone, G. Heiser, J. Andronick, et al. [seL4: Formal Verification of an OS Kernel](https://doi.org/10.1145/1629575.1629596) . SOSP 2009.\n11. W. M. McKeeman. Differential Testing for Software. Digital Technical Journal 10(1):100–107, 1998.\n12. X. Yang, Y. Chen, E. Eide, J. Regehr. [Finding and Understanding Bugs in C Compilers](https://doi.org/10.1145/1993498.1993532) . PLDI 2011.", "url": "https://wpnews.pro/news/formalizing-cpython", "canonical_source": "https://broyojo.com/python-formalization/", "published_at": "2026-11-10 01:56:48+00:00", "updated_at": "2026-11-10 04:51:16.369839+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["CPython", "Lean", "Opus 5.5", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/formalizing-cpython", "markdown": "https://wpnews.pro/news/formalizing-cpython.md", "text": "https://wpnews.pro/news/formalizing-cpython.txt", "jsonld": "https://wpnews.pro/news/formalizing-cpython.jsonld"}}