{"slug": "the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code", "title": "The triage is the product: running AI agents against Ethereum's protocol code", "summary": "The Ethereum Foundation's Protocol Security team has been running coordinated AI agents against Ethereum's protocol code, finding real bugs including a remotely-triggerable panic in libp2p's gossipsub (CVE-2026-34219). The team found that most effort goes into distinguishing real bugs from false positives, and they share their methodology for organizing agents and validating findings.", "body_md": "Notes from the Ethereum Foundation's Protocol Security team on running coordinated AI agents against real protocol code, including how we organize the work, what holds up under scrutiny, and what client teams and security researchers can take from it. This post stands on its own; later posts will go deeper on individual clients.\n\nWhat we've been running, and what surprised us\n\nOn the Ethereum Foundation's Protocol Security team, we've been running coordinated AI agents against the kinds of systems the network depends on, like systems software, cryptographic code, and contracts that have to be right. The agents found real bugs. One is now public: a remotely-triggerable panic in libp2p's gossipsub, a core part of the peer-to-peer layer Ethereum consensus clients run on, fixed and disclosed as CVE-2026-34219 with credit to the team.\n\nAgents finding bugs wasn't the surprise. The surprise was how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real.\n\nThis post is for client teams and security researchers who want to do the same thing. It covers how we organize the agents, the bar a candidate has to clear before it counts as a finding, and the habits that keep the results trustworthy.\n\nOne caveat up front: tooling for agent-driven audits moves fast, and any specific setup is out of date in a few weeks. So this post is deliberately about the methods, which are persistent, rather than the tooling. Disclosure is its own topic and will probably be its own post.\n\nAn agent is a search tool, not an oracle\n\nAn agent pointed at a codebase is a search tool, a lot like a fuzzer. The difference is what comes back. A fuzzer hands you a crash and a stack trace. An agent hands you a lot more, including a write-up (call chain, impact claim, suggested severity) and the artifacts to back it, like a proof-of-concept you can run against the real code.\n\nAll of that makes the result easy to read and easy to trust, the running proof-of-concept most of all. So don't count how many candidates an agent produces. Count how many turn out to be real.\n\nHow the work is organized\n\nWe run many agents in parallel against one target. They coordinate through the repository itself, with shared state in version control and no central process handing out work. An agent writes down a claim where the others can see it, does the work, and commits.\n\nWe got this approach from Anthropic's writeup on building a C compiler with a fleet of agents, which coordinates the same way. There's no central coordinator to build or maintain, and less that can go wrong.\n\nThe roles are generated by the work that's discovered:\n\nRecon turns an attack surface into concrete, testable hypotheses. Not \"audit the decoder\" but \"this field is trusted past this point; here's the property it should keep, the way it might break, and the proof that would settle it.\"\n\nHunting takes one hypothesis, traces the code path, and tries to build a reproducer.\n\nGap-filling looks at what was accepted and what was rejected, writes the next batch of hypotheses, and tracks coverage so the agents don't keep going over the same ground.\n\nValidation re-checks each candidate independently, removes duplicates, and decides.\n\nWe didn't invent this pipeline. Cloudflare describes the same stages, recon, parallel hunting, independent validation, deduplication, reporting, and their writeup helped shape ours.\n\nHere's what a candidate looks like before it counts as a finding:\n\n```\ntarget:      component and entry point an attacker can actually reach\ninvariant:   the property that must hold\nmechanism:   the specific way it might be made to break\nsuccess:     observable proof: a panic, a stall, an accepted-invalid input\nreproducer:  a self-contained artifact that runs against the real code\ndedup:       a key, so two agents don't chase the same thing\n```\n\nThe schema is there for a reason. It forces a specific, testable claim and a clear definition of done. An agent that has to write down an observable proof can't fall back on \"this looks risky.\"\n\nReproducible or it didn't happen\n\nOne rule matters more than any other. A candidate isn't a finding until there's a self-contained artifact that reproduces the failure against the real code, and that runs for someone who didn't write it.\n\nThe reproducer doesn't read the write-up, and it doesn't care how confident the model sounded. It either runs or it doesn't.\n\nMost of its value is in the false positives it catches. Three of them come up over and over, and each one is the agent getting a pass for the wrong reason:\n\nA panic that only happens in a debug build. Compile and run it the way the software actually ships, and the value just wraps around. Nothing crashes. It looks like a crash, but it isn't one.\n\nA reproducer that builds some internal value by hand, one no real input could ever produce, because every path an attacker controls rejects it earlier. The bug only \"reproduces\" against a function that nothing reachable calls that way.\n\nIn formal-verification work, a proof that goes through but doesn't mean what you wanted. The statement is trivially true regardless of what the code does, or it's weaker than the property you meant to capture. The verifier is satisfied, but the theorem doesn't constrain the behavior you actually cared about.\n\nNone of this is new. It's the same thing as a test that passes because it doesn't actually check anything. What's new is the volume. An agent writes the useless version as fast as the real one, and just as confidently. So the check has to be automatic. You can't count on the agent to catch itself.\n\nSignal-to-noise is most of the work\n\nMost candidates are wrong, duplicate, or out of scope. That's not a problem with the method; that's how it works. The goal is to reject the wrong ones fast and back the real ones with proof that's hard to argue with.\n\nEvery candidate that survives gets two independent checks. Can a real attacker actually reach it in a normal configuration? And what does it cost the attacker to pull off, compared to what it costs the network if it works? A bug that any single peer can trigger is very different from one that needs special access or a huge amount of resources.\n\nEverything gets checked against a running list of what's already known, fixed, or rejected. Without that, the agents keep rediscovering the same closed issue and reporting it again and again.\n\nAcceptance rates vary a lot from target to target, and that variation is useful on its own. Run this against mature, heavily audited code and almost nothing survives, which is still worth knowing. \"We looked hard and found nothing\" is a real result. Run it against less-explored code, or against formally verified code, where a machine-checked proof covers a model and the deployed bytecode is only assumed to match it, and more gets through.\n\nWe're not the only ones who found that the triage is the hard part. Cloudflare's main takeaway was that a narrow scope beats broad scanning. Anthropic's property-based-testing agent generated something like a thousand candidate reports, then used ranking and expert review to get down to a top tier that held up about 86 percent of the time. The generation was the easy part. I'm not going to publish our own numbers here; tied to a specific target, they'd say more about the target than about the method.\n\nWhat the agents are good at, and where they mislead\n\nThere's hype in both directions, so here's a plain list of what the agents do well and where they mislead.\n\nGood at\n\nMisleading at\n\nReading the spec and the code together\n\nCall chains that look reachable but aren't\n\nStating and checking a real invariant\n\nGaming the success check (a pass for the wrong reason).\n\nDrafting a reproducer from a one-line idea\n\nInflating severity to match how dramatic the write-up sounds\n\nSuggesting a root cause before you've looked\n\nBugs that span a sequence of valid steps\n\nThe split isn't even steady from one task to the next. Stanislav Fort, testing a range of models on real vulnerabilities, calls this a jagged frontier, or a model that recovers a full exploit chain on one codebase can fail basic data-flow tracing on another. You can't assume one good result means the next will hold up, which is another reason every candidate gets checked on its own.\n\nThe last row is the important one. A single agent session is good at one-shot reasoning and bad at bugs that span a sequence of steps, where each step is valid and only the order is wrong. For those, the agent isn't the search tool. Its job is to suggest which sequences are worth running through a stateful test harness. Used that way, it works well. Used as a replacement for the harness, it misses the most expensive bugs there are, the ones that only show up across a sequence.\n\nKeeping it honest\n\nA few habits do most of the work of making agent findings trustworthy, and none of them are complicated.\n\nProvenance on every artifact: what produced it, with what context, against which revision. A finding should be something you can re-run months later.\n\nDeterminism where it counts: one environment, one way to build and run, so \"reproduces\" means the same thing on every machine, not just the one where it was found.\n\nNorms, not scripts: tell agents what matters, the invariants and the bar for a real finding, instead of a numbered procedure. Over-scripted agents break the same way over-specified tests do, they keep following the steps after the steps stop making sense. A study of repository context files found the same thing: the extra requirements lowered task success and raised cost by over 20%, and the authors recommend keeping context to the minimal requirements.\n\nA person makes the final call: agents suggest. They don't decide what's real, what's a duplicate of a known issue, or what gets disclosed and when.\n\nThe bottleneck moved\n\nAI didn't replace the security researcher. It moved the work. The time that used to go into coming up with and chasing down hypotheses now goes into judging them at scale, including building the oracle, running the triage, keeping the list of known issues, and handling disclosure.\n\nThe bottleneck didn't go away. It moved from finding bugs to trusting the results, which is a better place for it, because that's where human judgment actually matters. But it's still a bottleneck, and ignoring that is how you end up shipping a wrong \"it's fine.\"\n\nThe practices that make this work aren't new. Reproducible failures, real oracles, and careful triage are the same practices that turned fuzzing from a research topic into standard practice over the last fifteen years. The tools are new. The practices aren't.\n\nHow fast the tools keep changing is an open question. Nicholas Carlini, careful and once a skeptic himself, argues the exponential case is worth taking seriously, even while he keeps wide error bars on it. If the generation side climbs that fast, the judgment side has to climb with it, or the gap between what gets produced and what actually gets verified only widens.\n\nFor the systems Ethereum depends on, that's the part that matters. Agents let us cover far more ground than we could by hand. In exchange, they ask for more careful judgment, across a much bigger pile of confident-sounding claims. That's a trade worth making, as long as you remember that the judgment is the real product.", "url": "https://wpnews.pro/news/the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code", "canonical_source": "https://blog.ethereum.org/2026/07/09/triage-is-the-product", "published_at": "2026-07-10 07:52:35+00:00", "updated_at": "2026-07-10 08:05:14.655537+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research"], "entities": ["Ethereum Foundation", "Protocol Security team", "libp2p", "gossipsub", "CVE-2026-34219", "Anthropic", "Cloudflare"], "alternates": {"html": "https://wpnews.pro/news/the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code", "markdown": "https://wpnews.pro/news/the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code.md", "text": "https://wpnews.pro/news/the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code.txt", "jsonld": "https://wpnews.pro/news/the-triage-is-the-product-running-ai-agents-against-ethereum-s-protocol-code.jsonld"}}