# Rolling in the Diffs > Source: > Published: 2026-07-23 19:10:51+00:00 *Too* philosophical We started about two years ago with a question that's admittedly a little too philosophical to be hovering around: how do you approach knowing knowledge? We use it to describe an intuition for finding a vulnerability in software — essentially a coverage problem, or, framed another way, an exploration problem where the heuristics matter most because they set the direction and reach of the search: where it goes and how far. Taint analysis connects dangerous sinks back to inputs; fuzzing throws near-random inputs into a pipeline and waits for one to drive execution down a branch that misbehaves. To begin with, we found that this epistemological heuristic is the biggest determinant of what and how many bugs an LLM can find — yet most LLM automation systems put almost no deliberate design into it. My guess is that we security researchers lean on it so intuitively in real-world VR that it's become invisible to us. All we really do is throw a sentence like "go and find RCEs, make no mistake" at the model and hope it surfaces zero-days that Google Project Zero missed, while handing it no clue where to start in a codebase of tens of thousands of lines. That's essentially expecting an LLM to match a Project Zero researcher with no degradation over context. P0 researchers are good because they approach where to look with far more ingenuity — mutation analysis, in-the-wild bugs, taint, fuzzing crashes — and at the end of the day it comes down to how many bugs they've already seen across complex context. Proof-of-concept development is the comparatively trivial part: once you've looked in the right place and found the right thing, the rest takes much less. Over the past year, we've tried tying LLMs to other epistemological heuristicsthat we(humans)rely on, to see whether it helps them find more — and more kickass — bugs. Taint analysis, for one: see my prior work,[Deductive Engine: Human Inspired Taint Reasoning](a continuation of our BH USA work,. The conclusion is that taint analysis — dataflow analysis — is a very effective intuition for LLM VR. In a sense it's almost perfect: it provides a solid epistemics toward the[ToAST])problem of slicing(what context we put into the LLM)via contaminations(or propagations — see. It's as effective as how human researchers find bugs because it follows the same logical process they do — essentially an exploration problem: proving connectivity between the slice that introduces the data and the slice that weaponizes it, to establish reachability and exploitability, and thus vulnerability. For example: start with a suspicious function[Explaining taint propagation using Disney's Where's My Water?])(say; explore outward from it`memcpy` , as a logical entry point)(cross-reference calls, and so on); determine whether we've reached a source; examine reachability between source and sink(source and the suspicious function — confirming no sanitizers sit in between); model the restrictions required to branch from source to the suspicious function; then craft an input that satisfies those restrictions, with payloads —all built on a dataflow ontology. ## aha We started looking at diffs while Pwno was working on an integration with GGML *(llama.cpp)*, and we didn't approach it with the *"coverage"* framing at all — more as a *"business model."* Diffs are how we've reviewed code for the past 20 years; PRs were already the unit of attention in our development pipeline, so reviewing them for vulnerabilities just felt intuitive. It was only after doing this for a while that we noticed the first non-obvious part *(#1)* of *Diff Rolling*: diffs are a remarkably effective [slicer](https://en.wikipedia.org/wiki/Program_slicing), and crucially, they slice on a dimension no static analyzer operates on. Traditional slicing *(AST, call graph, taint)* partitions code along *semantic* axes: what calls what, what flows into what. A diff, atomically, partitions code along *intent* — each commit message describes an intent, and the codebase is sliced by that shared intent. *A commit pulls together lines that may be syntactically scattered, structurally unrelated, even spread across different files, and binds them under a single human-authored purpose — one stated right there in the commit message.* Take PWNO-0014, an FFmpeg EXIF heap overflow we often use as a teaching example. In [commit 784aa0],`libavcodec/exif.c` is effectively re-decomposed: the diff lifts a set of lines out of their original structural neighborhoods and regroups them around a new purpose —"parse additional EXIF tags"— in a way no AST walker, dataflow engine, or document ranking would. And that's the dimension LLMs natively read on.Rolling with the Diffs, in other words, slices the codebase on the one axis that aligns with how language models actually think. Slicing was the center of *Tree-of-AST* and *Deductive Engine*'s epistemics, which I'd been working on for a while, so I realized this was really the key to something bigger. But it was only later that I saw there was more here that's non-obvious. Apart from intent alignment, another reason we found diffs work well for LLMs is that they're structured contrastively, and so nativelycompress context. A diff turns context into changed(new/old)lines + surrounding context + commit message. It strips away most of the semantically stable code while preserving the"boundaries"of semantic variation. ## Much would have more. Waiting on GGML's pipeline got old. We were sitting on `HEAD` , hoping the next push would surface a primitive worth writing up, and most days nothing did. Even when something did, it was usually a freshly landed feature *(a new GGUF architecture, a half-rolled-out quant path)* that hadn't shipped widely enough to matter to maintainers, or to anyone. The wait was too long and the payoff too thin. So we got greedy: *what if we stopped waiting for new commits and started scanning backwards?* The moment I tried it, the math turned absurd in our favor. __There are almost too many commits to scan__ *(#2)*. `llama.cpp` ships ~20 commits a day, and on a project actively maintained for years, that compounds into tens of thousands of diffs just sitting there. Even at a 5% rate of vulnerability introduction, the scale tilts heavily toward us: maintainers have to land every commit clean, while we only need one to slip — a tiny off-by-one in an unnoticeable corner. And crucially, all of this is __.__ *embarrassingly parallel**(The part we'll mainly focus on is the application.)* Once we started scanning at a slightly larger scale, we found — to our surprise — that the distribution of findings isn't flat; they cluster toward more recent commits. __Commits are naturally sorted by time, and time implicitly creates a probability gradient over the remaining vulnerabilities__ *(#3)*: the later a commit is, the less likely it's been fuzzed, manually reviewed, or reported via a production crash, because it has survived for less time, so the window in which it could have been patched is smaller. Conversely, the later the commit, the closer it sits to us on the history — and the more likely we are to find something fun. *A rather ingenious corollary fell out of this: patch commits are themselves markers.* *(* __patch folding__)*If a later commit fixes a bug introduced by an earlier one, we mark the older commit to skip.* One subtlety worth flagging, since it looks like a contradiction at first. Earlier we said freshly landed features carry low impact because they haven't shipped; here we're saying newer commits carry higher bug density. These don't conflict — they live on different timescales (my wording might have blurred that). The not-yet-shipped window is weeks, but the gradient stretches much longer, across years. *Once we accepted that, we noticed something stranger about where the scanning was taking us:* __Diff Rolling's coverage is uniform at the commit level but organically weighted at the component level__ *(#4, 5)* — a duality that happens to favor security research. Atomically, every commit is a brick in the codebase, and we don't *bias* any of them *(we treat each slice as carrying equal weight; we call this bias in reviewing "attention")*. This atomic equality matters, because __vulnerability impact has no correlation with code prominence__ *(#4)*. We want a slice *of EXIF extra-tag processing* treated the same way we treat an *— a vulnerability in an unwatched corner carries the same latent impact as one in a popular path.* `mp4` `av_stream` header-reading slice*This is also a latent effect of the granularity we described in* __#1__.Collectively, though, the distribution isn't flat. As an * emergent property*, actively maintained components accumulate more commits *(h264 in*. So `ffmpeg` , say, receives far more than some obscure 16-year-old legacy decoder nobody optimizes)__uniform sampling over commits__ *organically*induces a non-uniform,[coverage over components](https://en.wikipedia.org/wiki/Heavy-tailed_distribution)*heavy-tailed**(#5)*, where the tail's mass tracks commit attention — which is itself a proxy for what's *"valuable"*from an impact standpoint. The *collective*-level weighting falls out of the atomic-level uniformity, for free. The five observations above form a unified coverage space. __Slices are structured — composed by intent into a unit.__*(alignment)**(atomic, naturally decomposed in favor of reviewing)*__Effectiveness__*(scalability, granularity)*scales with the number of commits.*(abundant granular parallelism)**(the better maintained a project is, the more statistically likely you are to find a bug, and the finer the granularity)*__Diffs form a uniform sampling space.__*(flat)*every individual slice is*looked at (sampled)*with the same prioritization.__Yield weightings are sorted temporally.__*(temporal weighting)**(the later a commit is, the more likely the code it slices is found vulnerable)*Corollary: __Patch Folding__Patch commits are themselves markers. If a later commit fixes a bug introduced by an earlier one, __we mark the older commit to skip__. __Attention weightings are organically accumulated.__*(spatial weighting)**(hot, well-maintained components receive more total coverage in aggregate, but atomically they're treated with equal attention — as they should be in security research. The weighting is emergent)*We define the commit history of a codebase as a naturally enumerable vulnerability search space. Atomically, each commit is naturally structured for reviewing: constructive, intent- *aligned*,*small enough for an LLM to review, rich enough to expose invariant change, and numerous enough to sample at scale.*Yield weightings are sorted natively by time; the sample space itself is uniform at the commit level, preserving atomic fairness, while collectively, commit density forms a natural accumulation of attention. *Embarrassingly* Scalable *Embarrassingly*Scalable The first idea was to *roll* the *diffs* at scale. This was still the *Big Sleep* era, and one of the things I wanted Pwno to do was somehow keep up with Project Zero and DeepMind's output. So I started reaching for bigger, more ambitious codebases. We began with `FFmpeg` — the media library billions of people rely on every day — because it ticks every box: big, well-maintained, and essential enough *that people care about its security* *( FFmpeg has over 124,281 commits, averaging ~94 a week)*. And it was at this larger scale that *Diff Rolling*'s scalability really began to manifest. Processing commits is *embarrassingly parallel* — there are no dependency relationships between them, so you can technically process 20 at once — and, as we've noted, for bigger codebases there are *almost too many commits to look at* *(#2)*. It's this combination of *availability* and *scalability* in the epistemic itself that makes its *application* insanely efficient, and thus effective: it can __maximize resource utilization__ without sacrificing any __granularity__ *(#4)* (Similar to how theEmbarrassing Parallelizationof attention heads helpedAttention Is All You Need*, this efficiency would be the*[have to think of best wording — the extra key factor, like that one extra rower who makes sure the boat wins?]when processing a larger, more ambitious target — the way attention won at training scale — as we scan the entire internet's infrastructure.) **Folding** **Folding** While scaling Diff Rolling, we discovered more practical applications of it in real-world automated security research. Instead of scanning all commits, you can target a specific component by path *(a certain codec in FFmpeg, say — and, calling back to #3, the temporal gradient can stretch much further as commits grow sparse)*, a specific author's commits *(if a known style is more prone to particular mistakes)*, or a particular time of commit *(Friday 5 p.m. versus Thursday 11 a.m.)*. This delimitation of the *enumerable vulnerability search space*depends only on the informative granularity of commits — and a perk of using commits as *atoms*is that they're information-rich yet widely arranged. Commits are designed such that even pulling out the code changes alone leaves you with something highly informative: a commit describes almost all the implicit information you'd need about a change. So we have plenty of ceiling to narrow the search space toward the kinds of commits most likely to introduce a noteworthy vulnerability. Collectively, the way commits assemble into a codebase is itself enough to delimit the enumeration space further. Beyond *Patch Folding*, another interesting delimitation is what I've named the __Ship of Theseus problem__. Just as the classical Greek story probes the philosophical composition of objects, some commits are part of the history but no longer exist at all in the current version of the codebase. I call the process of eliminating them __Theseus Folding__, *achievable with git blame + AST*. With both folding techniques, we can actively address both the *explicit*decay *(via Patch Fold)*and the *implicit*decay *(via Theseus Fold)*of a codebase seen as a collection of commits. Path folding itself also provides a high-quality dataset for future discovery. With both the embarrassing scalability *(available + parallelizable)* and the effective enumeration-space delimitation *(courtesy of how informative commits are)* up our sleeves — while the sampling space arranged for that enumeration is natively advantageous for security research, and even each individual atom is shaped to favor it — the methodology ends up unintuitively effective. **Schrödinger's Box** **Schrödinger's Box** After Rolling Diffs across `Linux` , `FFmpeg` , and `V8` gave us great findings, we ran into the same problem we'd had since `llama.cpp` : verifying and PoC'ing them. That's even more of an engineering problem. Since I'd been working on distributed systems already *(autogdb, and RPC for llama.cpp)*, my first instinct was to architect isolated containers in which agents could work on PoC-as-validation for each finding — the *Embarrassingly*Scalable property still holds through verification. It's also intuitive to use an instrumented ground-truth validator like `ASAN` , `UBSAN` , or `KASAN` as proof of a __vulnerability__. While we're on the definition of a __vulnerability__ — *a software bug that affects security to a degree people should worry about* — our research says it comes down to two dimensions: reachability and exploitability. Exploitability is the intuitive one: for a bug to be a vulnerability, it has to be exploitable. *A mis-rendered button in the frontend isn't a vulnerability; a mis-rendering that breaks HTML escaping is.* Reachability is the more essential part of triaging Diff Rolling's findings. The most common disqualifier — as with other LLM vulnerability discovery — is that __a bug can be technically exploitable while its reachability keeps the component from actually being vulnerable__. *For instance, the diff introduces a NULL-pointer deref in a frame-processing function that's only reachable after the decoder has already allocated a frame.* To address that concern — a vulnerability's *reachability* in real-world scenarios — we designed `the-box` , a distributed system that mimics the harnessing process of CTF challenges; specifically, since we're working with memory issues, CTF `pwn` challenges. The philosophy behind `box` is that it sets up a validation environment __independent__ of the input. Services are built with validators *( ASAN, UBSAN)* integrated up front, then harnessed with a script that redirects payload I/O in a real-world manner. Unlike a CTF `pwn` challenge, the only way to retrieve a *"flag"*here is to craft a payload that triggers an `ASAN` -induced signal from the binary. The payload travels through the `payload I/O` into the harness, while the `debug I/O` lets agents observe the harnessed service's inner states and readjust their payload crafting accordingly.With the __separation of the input space from the validation space__, `box` first of all stops the LLM from __interfering with the integrity of the validation__ *(cheating)*, and thus __guarantees a valid proof of reachability and exploitability__ *(the ASAN log)*, while a payload passed through the `payload I/O` can serve directly as a valid PoC. At the same time, the __two I/O channels remove redundancy from the validation and PoC-development process__. By isolating — and throwing away — the unrelated, trivial scaffolding *(building, harnessing)*, it keeps the PoC agent focused on a single question: *"Did my payload trigger the intended validation signal?"* *(If not: what does the* `debug I/O` tell me about the program's internal state? Should I readjust my padding or the placement of the manipulated content — or is what I'm observing inconsistent with the finding's analysis, meaning it isn't actually exploitable?)`box` , meanwhile, didn't inherit Diff Rolling's *Embarrassing Scalability* — but the design is scalable by nature anyway, since it's *centralized* in service of a *distributed-systems* design. Since every `box` container is isolated by a `kubernetes` containerized design, we can only assign each container a little compute — and because of that, project construction was a problem for us at first: building a project means you have to *pull* it and then *build* it. For bigger projects like `Chromium` or `Firefox` , on a small pod the *pull* alone would take at least ~40 minutes, never mind the compile. This was an even bigger challenge when I was working on upstream monitoring of larger codebases, since it means working across a different version most of the time. Working toward a fix, it occurred to me that since we're doing __ diff-based__ analysis, __all we need to maintain is a base reference of the codebase__— a base build for each project. The pod pulls the built base artifacts from the server and builds from there itself; when it *(the*requests a specific or later version, we update the `box` )`HEAD` reference. And since we work from a centralized reference, __with caching__, each recompile costs far less compute and time — and the compute we save lets us centralize and maintain more projects.