{"slug": "using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack", "title": "Using LLM-based Verification to Eliminate Bugs in Linux's Network Stack", "summary": "Researchers at Basis used LLM-guided formal verification to uncover two critical bugs in Linux's nftables firewall compiler and optimizer, affecting every version of Linux since 2022. The verified implementation was proven free of these semantics-altering bugs, and the more severe bug would not be found via a naive LLM bug search. The findings highlight LLMs' growing capability to automate formal verification and secure critical software infrastructure.", "body_md": "# Using LLM-based Verification to Eliminate Bugs in Linux's Network Stack\n\n##### Article:\n[Yiyun Liu](/about/people/yiyun-liu/), [Kiran Gopinathan](/about/people/kiran-gopinathan/), [Nikhil Pimpalkhare](/about/people/nikhil-pimpalkhare/)\n\n[Yiyun Liu](/about/people/yiyun-liu/)\n\nLLMs have grown alarmingly capable at finding bugs in production software. This accentuates an already severe risk: much of our critical infrastructure is mediated by software, and every bug in that software is a potential exploit. Formal verification offers the potential to mitigate this risk by producing proofs that entire classes of bugs are impossible, but it has seen limited real-world use because it demands scarce, expensive, specialist expertise. Fortunately, LLMs are increasingly capable of verification too, pointing toward a future where critical software infrastructure is secure by construction.\n\nIn this blog post, we report on results from LLM-guided verification experiments at Basis. Our target was a key component of Linux’s network stack: the nftables firewall compiler and optimizer, which we set out to verify in the Rocq theorem prover. nftables is one such piece of critical infrastructure. It filters the traffic of almost every Linux machine, and vulnerabilities in it are treated with the highest severity, since a firewall that misfilters exposes every machine it was meant to protect.\n\nIn the process of verifying nftables, we uncovered **two critical\nbugs** affecting every version of Linux since 2022 (these have been\ndisclosed to maintainers 1). Our verified\nimplementation was\n\n*proven to be free*of these semantics-altering bugs, and a secondary experiment showed that the more severe of the two would not be found via a naive LLM bug search.\n\nOur experiments suggest that the effort involved in producing proofs and constructing robust verified systems is increasingly automatable. The rest of this post gives an overview of nftables, the bugs we found, and our process for using LLMs to verify critical networking software.\n\n# A quick primer on nftables and its bugs\n\nnftables is one of the firewall mechanisms provided by the Linux\noperating system. Every packet your OS receives passes through\nnftables, which determines which packets are forwarded and which are\ndropped. Having replaced `iptables`\n\nin 2014 as the default packet\nfilter of every major Linux distribution, nftables today guards\neverything from home routers to container networks; if a Linux machine\nfilters traffic, nftables almost certainly decides what gets\nthrough. Firewalls are a critical component of most network security\nlayers, and vulnerabilities in them can compromise whole systems. A\nverified implementation of nftables would increase assurance in this\ncritical layer.\n\n## nftables rulesets and compilation pipeline\n\nThe full architecture of nftables is shown below and consists of a userspace CLI tool and a kernel module:\n\nThis CLI tool takes as input a series of user-supplied policies,\nexpressed as an ordered list of rules, where each rule matches on\nsome part of the packet (its source address, its destination port, its\nTCP flags) and returns a verdict, i.e. `accept`\n\nor `drop`\n\n. For\nexample, the following rules tell nftables to accept any packets\nwhere the destination address (`daddr`\n\n) is `192.168.50.1`\n\nor\n`192.168.50.2`\n\n.\n\n```\nip daddr 192.168.50.1 accept\nip daddr 192.168.50.2 accept\n```\n\nThe `nft`\n\ncommand-line tool parses these rulesets and compiles them\ninto a compact register-based bytecode, which it then loads into the\nkernel. There, the kernel’s interpreter uses the bytecode to\ncompute a verdict for every packet passing through the network stack:\neither accepting it through the firewall or dropping it.\n\nAs this bytecode runs on every packet the system sees, it sits on a\nhot path of the kernel, and thus its efficiency is critical. For this\nreason, the CLI tool also provides an optimizer module for the source\nlanguage that rewrites your input rules to run more efficiently\n(reducing the number of reads, or removing redundant checks). A\ncritical security property that users rely on the optimizer to\nguarantee is that it *preserves the semantics* of its input rules. In\nother words, the verdict computed for a packet should be the same\nregardless of whether it was passed through the optimized or\nunoptimized ruleset.\n\nWe set out to prove the correctness of the `nft`\n\ncommand-line tool,\nand in the process found two bugs in the system.\n\n## Bugs found\n\nOur key results were finding two key cases where the optimizer does\n*not* produce equivalent programs:\n\nThe first bug was an invalid optimization, which if applied, resulted in accepting packets that would have been rejected before optimization (!!).\n\nThe second resulted in the optimizer transforming valid rulesets into invalid rulesets that are rejected by the kernel.\n\nWe were able to reproduce these bugs in the latest version of the nftables userspace tool, and proved our verified re-implementation was entirely free of these semantics-altering bugs.\n\n### Bug 1: Invalid merging of bitmask fields\n\nThe first bug relates to an optimization wherein bitmask fields are merged.\n\nPackets include a series of control-flag bits — TCP packets, for\nexample, have `SYN`\n\n, `ACK`\n\n, `FIN`\n\n, and others.\n\nnftables allows users to write rules that test whether a particular bit is set: `tcp flags syn`\n\nmatches any packet that has the `SYN`\n\nbit on. So the following ruleset drops every packet with `SYN`\n\nset and every packet with `ACK`\n\nset:\n\n```\ntcp flags syn drop\ntcp flags ack drop\n```\n\nThe optimizer, `nft -o`\n\n, merges these two rules into a single one:\n\n```\ntcp flags { syn, ack } drop\n```\n\nUnfortunately, this rewrite is **incorrect**. The merged rule does\n**not** mean the same thing.\n\nIn particular, the semantics of the nft input language is defined such\nthat a set lookup is an *exact-match* test: the packet is dropped only\nif its flags byte equals `syn`\n\nexactly (the `SYN`\n\nbit set and all\nother bits clear) or `ack`\n\nexactly.\n\nThe original rules asked “is this bit set?”; the merged rule asks “is the flags byte equal to exactly this one bit, with everything else zeroed-out?” The same narrowing applies to any bitmask-typed field matched in this bit-test form.\n\nThis bug is particularly pernicious as it silently converts a\nrestrictive policy into a more permissive one, and could have\n*critical* safety implications: in the previous example, most syn and\nack packets would have been dropped, but after optimization, only\npackets with *only* the `SYN`\n\nor `ACK`\n\nbit set would be dropped (a\nsubstantially smaller set of packets).\n\n### Bug 2: Invalid merging of overlapping ranges\n\nThe second bug we found relates to an optimization that merges rules based on their addresses.\n\nWhen consecutive rules match the same field, `nft -o`\n\nmerges them into\na *verdict map* (`vmap`\n\n) — a lookup table that sends the field’s value\nto a forwarding verdict. This reduces multiple reads of the field into\na single read and then a jump in the bytecode.\n\nConsider a ruleset that drops one range of source addresses and accepts another:\n\n```\nip saddr 192.168.50.1-192.168.50.123 drop\nip saddr 192.168.50.120-192.168.50.255 accept\n```\n\n`nft -o`\n\nmerges the two rules into:\n\n```\nip saddr vmap { \n   192.168.50.1-192.168.50.123 : drop, \n   192.168.50.120-192.168.50.255 : accept \n}\n```\n\nUnfortunately, the implementation of optimization has a bug and the\ngenerated map is ill-formed. In particular, nftables requires that a\n`vmap`\n\nmust map each address to a *single* verdict: its keys must\nrefer to non-overlapping intervals. In contrast, in our example, the\noptimizer has reused the original ranges without checking their\ndisjointness, so keys overlap on `.120`\n\n–`.123`\n\n. As such, after\noptimization, the generated ruleset is rejected with an error `Error: conflicting intervals`\n\n.\n\nThis bug leads to a valid ruleset being optimized into an invalid one,\nsuch that an error is raised to the user when they try to install the\noptimized ruleset. In this case, the security considerations are less\nsevere, but this still represents an incorrect optimization and\nultimately reduces trust in `nftables`\n\n.\n\n# Formally verifying nftables\n\nBoth bugs were found autonomously by LLMs during an ongoing effort to\nverify nftables’ userspace components, that is, while proving that\n`nft`\n\npreserves the behavior of rulesets.\n\nTo formally prove that an implementation has this property, we first have to pin down what a ruleset means, and then verify the implementation against that meaning.\n\nWe formalized, in the Rocq theorem prover, four pieces:\n\n- the syntax and semantics of the nftables rule language,\n- the syntax and semantics of the register-based bytecode that the kernel executes,\n- a compiler from rulesets to bytecode, and\n- an optimizer that rewrites rulesets into more efficient rulesets.\n\nPieces (3) and (4) are verified counterparts of the two halves of the\n`nft`\n\nCLI tool from our primer, and piece (2) plays the role of the\nkernel’s bytecode executor.\n\nBuilding on the formalized semantics, we stated and proved that the compiler and the optimizer are both semantics-preserving. That is, the bytecode produced by the compiler accepts and drops exactly the packets that the input ruleset does, and an optimized ruleset matches exactly the packets that the original does:\n\nTwo pieces sit outside the proofs: the parser, which turns nftables\nrulesets into a Rocq AST, and the serializer, which installs the\nbytecode into the kernel. Both are unverified OCaml, packaged in our\n`nftc_cli.exe`\n\ncommand-line tool, which links against the verified\ncompiler and optimizer.\n\nThe verified implementation is still a work in progress, but it is close to feature parity with ~90% of the nftables rule language modeled and verified. An LLM was used to write every definition, line of code, and proof in the verified implementation.\n\n# An autonomous verification methodology\n\nThis section describes the methodology we used to automate our development. All code and proofs were generated by Claude CLI running in auto mode, with Opus 4.8 as the model.\n\nThe LLM started from a detailed prompt that specified the verifier (Rocq, in our case) and the part of nftables to verify, and pointed to prior work on formal verification and networking. The prompt also spelled out the proof-engineering and general engineering practices we expected the LLM to follow, such as test-driven development and frequent code reviews.\n\n## Testing harness\n\nWe instructed the LLM to launch a virtual machine (VM) with\n`systemd-vmspawn`\n\nand to use network namespaces to construct test\nenvironments where nftables rules could be installed and exercised on\ndifferent network topologies. The VM also gave the LLM a sandbox\nin which to run the official `nft`\n\ncommand line tool and experiment\nwith its behavior.\n\nOn top of this harness, we ran end-to-end differential tests: a corpus of rulesets was fed to both our verified compiler and the official nftables implementation, and their outputs were compared.\n\n## Adversarial workflow\n\nDevelopment proceeded through an adversarial loop between two LLMs: one wrote the specification, implementation, and proofs, and the other reviewed them for a specific type of flaw and reported what it found. The loop continued until the reviewing LLM was convinced that the type of flaw had been properly fixed.\n\nThis workflow was particularly effective at exposing fidelity issues in the language semantics, where the implementing LLM prematurely claimed victory even though the meaning of an expression was not precisely modeled (e.g. an effectful expression approximated as a pure one). To surface such semantic gaps, we instantiated the adversarial template with a reviewing LLM that cross-checked the verified code against the actual C implementation of nftables, flagging under-specifications.\n\n## Small Proof-Oriented Tests (SPOT)\n\nTo further pressure-test the semantics, while also producing an artifact end users can apply to debug their own firewall configurations, we built special test cases in which the LLM had to state and prove properties of real rulesets: either a formal proof that a ruleset captures the user’s intent, or a counterexample showing where that intent can be violated. When a ruleset resisted specification and reasoning, that was a signal that the semantics was not yet precise enough and could be improved.\n\n# How were the bugs found?\n\nBoth bugs we include in this blog post were found autonomously by the LLM throughout the development. What is more interesting than the bugs themselves, however, is how the bugs were found.\n\nThe overlapping interval bug was found thanks to the testing harness. On\nJune 30th ([ 69666ba](https://github.com/BasisResearch/verified-nftables/commit/69666ba)),\nthe LLM synthesized a battery of artificial rulesets designed to\ntrigger the optimizer rewrites of nftables and better understand its\nbehavior. The harness ran each ruleset through both optimizers and\nloaded the results into the kernel inside a fresh network\nnamespace. The bug turned up incidentally when one of these rulesets\ntriggered a loud failure (\n\n`conflicting intervals`\n\n) on the `nft`\n\ncommand line.The story behind the unsound bitmask merge bug, however, is a lot more\neventful. In commit\n[ 17c949a](https://github.com/BasisResearch/verified-nftables/commit/17c949a),\nthe LLM added its own, sound version of the bitmask merging\noptimization. When consecutive rules match on the same bitmask field,\nits optimizer folds them into an OR of the original “is this bit\nset?” tests, rather than nftables’ exact-match set. The commit\nmessage reveals that the LLM was already aware that the exact-match\nsemantics of a set makes it unsuitable for merging bitmask-typed\nexpressions. However, the LLM only used that unsoundness to justify\nits own divergence, rather than recognizing it as a bug in nftables\nitself.\n\nIn fact, according to the commit message, the LLM tried to\nrationalize nftables’ buggy merging behavior by pointing out that the\nfield it was merging, the packet’s connection-tracking state `ct state`\n\n, can have at most one bit set at a time. Therefore the merged\nset `ct state { new, established }`\n\n, which compares the packet’s\nconnection-tracking state against the exact bits of `new`\n\nor\n`established`\n\n, is harmless, and can be found, for example, in the\n[Arch Linux default\nruleset](https://gitlab.archlinux.org/archlinux/packaging/packages/nftables/-/blob/8a56b5d745b7731488a4f2afe437f713bbc0146f/nftables.conf#L14).\n\nThe bug was not surfaced until we explicitly prompted the LLM to\nexplain its divergence from the official optimizer in a July 14th commit\n([ c786563](https://github.com/BasisResearch/verified-nftables/commit/c786563)).\nThat’s when the LLM finally realized that nftables’ optimizer\ndoes lead to unsound behavior since there are other bitmask-typed\nexpressions such as\n\n`tcp flags`\n\nand `ct status`\n\n, where multiple bits\ncan be set at once, and managed to confirm the bug through the VM\ntesting harness. Formal verification thus protected us from bugs\nthat neither we nor the LLMs expected.After surfacing these two bugs, we conducted a small experiment where we started a fresh Claude session and instructed it to use the same VM testing harness to look for bugs in the official nftables optimizer. The LLM ran for a few hours, and managed to identify 16 bugs, 3 of which involve silently changing semantics, and 13 of which involve crashing the optimizer due to invalid optimized rules. We are in the process of disclosing these bugs to the maintainers.\n\nDid Claude manage to find both bugs surfaced in our verification endeavor? The answer is no! While the overlapping interval bug was indeed found by Claude, the bitmask bug remained hidden after a few hours of running. In fact, from examining the chat history, the LLM would be unlikely to find the bug even if we ran it for longer, as it managed to reproduce the exact ruleset that was able to trigger the optimizer bug, but found the new output reasonable and decided to move on. Unlike the 3 silent bugs that the LLM did manage to find just through eyeballing the buggy output rules, the bitmask bug requires a deeper understanding of the meaning of nftables semantics.\n\nFinally, a perhaps even more encouraging result is that the verified implementation is not susceptible to the 3 optimization bugs that Claude managed to find. Despite the verified implementation still being work in progress, the LLM-generated semantics capture more than enough of nftables to reject those buggy optimization rules with the correctness theorem. Smarter models may be better at identifying bugs, but formal verification truly grants us a higher level of assurance.\n\n# Findings and Observations\n\nOver the course of this initial verification effort, most of the human effort was in validating, reviewing, and refining specifications. While failing proofs were easy to eliminate, as the LLM would keep working until Rocq accepted, a theorem that had been quietly weakened could easily be missed.\n\nGiven work it does not want to do, we often found LLMs would shrink the task in some way that a casual reading could easily miss.\n\nWe saw two recurring forms of this:\n\n- the LLM weakened its specifications with convenient preconditions and\n- the LLM warped the design of the system so that hard lemmas never arose.\n\n## Weakening a theorem with a precondition\n\nThe optimizer’s correctness theorem provides a representative example of this failure mode.\n\nPart of the optimizer implementation needs to generate fresh names, which would ordinarily mean threading a name-generation state through the surrounding code.\n\nThe names are only needed when rewriting rules with effects, rules that update state or modify the packet. Rather than do the plumbing, the LLM narrowed what its theorem had to say:\n\n```\nTheorem optimize_table_correct :\n  forall n d c n' d' c' base p,\n    optimize_table n d c = (n', d', c') ->\n    rules_clean (c_rules c) = true ->\n    … ->\n    eval_chain c' (set_env p (env_with_sets base d'))\n  = eval_chain c  (set_env p (env_with_sets base d)).\n```\n\nIn particular, the `rules_clean`\n\nassumption that was subtly inserted\nis a predicate that holds only on pure rules. With this restriction\nthe theorem could be proven without the plumbing infrastructure. The\ntheorem was still named `optimize_table_correct`\n\nand proven correct,\nthough the statement now said nothing about rules with effects.\n\n## Register allocation in the parser\n\nSometimes the LLM would shortcut by reshaping the specification\ndefinitions themselves rather than the theorem directly.\nAn example of this came in the register allocation implementation: the\n`nft`\n\ncompiler targets the kernel’s register-based bytecode\ninterpreter, so at some point in the compilation pipeline, the\ncompiler must allocate registers.\n\nThe LLM tried to avoid proving lemmas about register allocation, by extending the source AST with registers, leaving the parser, which lives outside the verified core, to assign them.\n\nThis led to an incomprehensible design where part of the allocation happened during parsing, but in terms of the proof obligations for the LLM, the compiler became easier to verify as complex logic became outsourced to unverified code.\n\n# Conclusions\n\nVerification is not new. For decades, researchers have dedicated huge amounts of labor to verifying critical software infrastructure, and in the process have historically often uncovered critical bugs. What is new is that this labor is now increasingly automatable through the use of LLMs. While ongoing, even our interim progress on this project would have likely required years of human effort to complete; boosted by LLMs, we reached a working implementation in a manner of weeks. We reap the same benefits as traditional verification: we are working towards a reimplementation with a proof of correctness baked in, and along the way we were able to uncover critical bugs in the nftables optimizer.\n\nSome of this labor, however, remains manual. The LLMs repeatedly tried to downscope their task and procrastinate on challenging proofs. Most of the human effort in our development went into catching this behavior. It remains to be seen whether a more sophisticated harness or advances in model capabilities can automate this final step.\n\nWe intend to find out ourselves. Our next goal is a fully automated pipeline, one that verifies critical software infrastructure at scale without a human in the loop.", "url": "https://wpnews.pro/news/using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack", "canonical_source": "https://www.basis.ai/blog/verified-nftables/", "published_at": "2026-07-20 13:57:41+00:00", "updated_at": "2026-07-20 15:27:27.110443+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "developer-tools"], "entities": ["Basis", "Linux", "nftables", "Yiyun Liu", "Kiran Gopinathan", "Nikhil Pimpalkhare", "Rocq"], "alternates": {"html": "https://wpnews.pro/news/using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack", "markdown": "https://wpnews.pro/news/using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack.md", "text": "https://wpnews.pro/news/using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack.txt", "jsonld": "https://wpnews.pro/news/using-llm-based-verification-to-eliminate-bugs-in-linux-s-network-stack.jsonld"}}