Testing LLMs against seL4: Can AI break formally proven code?

A new analysis suggests that large language models (LLMs) may be able to find security flaws in formally verified software like the seL4 microkernel by targeting gaps between mathematical proofs and physical hardware execution. The approach focuses on specification flaws, compiler mismatches, and hardware behaviors such as speculative execution that formal proofs do not cover. The proposed workflow involves feeding the AI formal specifications, performing differential analysis between verified C source and compiled binary, and using targeted prompts to identify vulnerabilities like TOCTOU race conditions.

Testing LLMs against seL4: Can AI break formally proven code? In a perfect world, the AI should find absolutely nothing because the proof covers the implementation. However, there is a massive gap between a mathematical proof and the actual execution of code on physical hardware. If you're planning a deep dive into this, here are the specific angles where an LLM agent might actually find something useful: The Specification Gap Formal verification doesn't prove a system is "perfect"; it proves the code matches the specification. If the specification itself has a logical flaw or misses a real-world hardware quirk, the proof is valid, but the system is still vulnerable. An AI trained on thousands of CVEs might recognize a pattern—like a specific race condition or a side-channel leak—that the original formal specification simply didn't account for. Compiler and Hardware Mismatches The seL4 proof typically targets a specific model of the CPU. But we aren't running code on a mathematical model; we're running it on silicon. There are three layers where things break: 1. The C code is verified. 2. The compiler transforms it into binary. 3. The binary runs on a CPU with branch predictors and speculative execution. AI is particularly good at identifying "weird" hardware behaviors. If you feed an LLM the assembly output of a verified kernel and ask it to look for speculative execution vulnerabilities like Spectre-style leaks , it might spot a sequence that the formal proof ignored because the proof assumed the CPU executes instructions linearly. Practical Testing Workflow If anyone wants to try this as a real-world experiment, don't just paste a file into a chat window. To get actual results, you need a structured AI workflow: 1. Context Loading: Feed the AI the formal specification documents first so it understands the "rules" the kernel is supposed to follow. 2. Differential Analysis: Provide the verified C source and the compiled binary. Ask the AI to find discrepancies where the compiler might have introduced an optimization that violates the intended security property. 3. Targeted Prompting: Instead of asking "find a bug," use specific prompt engineering to target known microkernel weaknesses: Analyze the following memory management routine for potential TOCTOU Time-of-Check to Time-of-Use vulnerabilities, specifically focusing on how the hardware cache might interact with the verified state transitions. Ultimately, using AI on proven software isn't about proving the mathematicians wrong; it's about finding the "seams" where the math ends and the physical hardware begins. That's where the real security flaws usually hide. Generative AI Training Data: The Pirated Book Controversy 55m ago /en/news/4186/ Hacker News Workflow: Stop Switching Tabs for Comments 1h ago /en/news/4182/ LearnVector: Scaling Personalized Education with AI 1h ago /en/news/4180/ Manim WebGPU: Running Math Animations in the Browser 2h ago /en/news/4176/ Web Scraping Rights: Why Google and Reddit Don't Own the Web 2h ago /en/news/4173/ TSMC Arizona Expansion: The AI Hardware Bottleneck 3h ago /en/news/4169/ Next Generative AI Training Data: The Pirated Book Controversy → /en/news/4186/