Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

Google Project Zero reported that its LLM-based bug-finding system BigSleep found 45 vulnerabilities in Chrome's V8 JavaScript engine over the past six months, compared with 37 found by the generative fuzzer Fuzzilli, with a 25% overlap in the specific bugs reported. Dedicated experiments testing theoretical overlap on vulnerabilities reported since 2023 estimated overlap at roughly 60%, and Project Zero concluded that a substantial portion of vulnerabilities were discoverable by only LLMs or only fuzzing. Project Zero recommended that vendors perform both fuzzing and agentic analysis of their code to maximize the number of vulnerabilities reported before they impact users.

Summary Project Zero compared two automated tools used by Google to find vulnerabilities in Chrome’s JavaScript engine, V8, on a regular basis. Fuzzilli https://github.com/googleprojectzero/fuzzilli is a generative JavaScript engine fuzzer and BigSleep https://projectzero.google/2024/10/from-naptime-to-big-sleep.html is an LLM-based bug finding system. In the past six months, BigSleep found more vulnerabilities than Fuzzilli 45 versus 37 with a 25% overlap in the specific bugs reported. Since this comparison methodology has a tendency to underestimate overlap, we also performed dedicated experiments to test theoretical overlap in vulnerabilities reported since 2023. These experiments estimated overlap at roughly 60%. Regardless of which value is accurate, a substantial portion of vulnerabilities were discoverable by only LLMs or only fuzzing. While it is extremely promising that LLMs have become able to find so many vulnerabilities so quickly, fuzzing and LLMs are still complementary approaches. We recommend vendors perform both fuzzing and agentic analysis of their code to maximize the number of vulnerabilities that get reported before they impact users. Introduction Fuzzing has long been successfully applied to software projects such as Chrome or V8 Chrome’s JavaScript engine to detect bugs. Recently, the arrival of powerful LLMs able to read, reason about, and write code is causing a transformative change to software development and software security. With hundreds, sometimes thousands of new bugs and vulnerabilities https://blog.google/security/chrome-stronger-with-every-update/ having been discovered in well-fuzzed targets, a natural question arises: how do new LLMs compare to “traditional” fuzzers? Are they starting to replace them or do the two approaches complement each other? To help answer this, we look at three key questions: 1. How many bugs does each approach find in the real world? 2. What is the overlap between the two approaches? 3. What is the “shape” of bugs discovered by one system but not the other? For this blog post, we will limit ourselves to two concrete systems: Fuzzilli https://github.com/googleprojectzero/fuzzilli , a JavaScript engine fuzzer, and BigSleep https://projectzero.google/2024/10/from-naptime-to-big-sleep.html , an agentic bug finding system. Both of these systems run in production and automatically look for bugs in newly added code, which makes them fairly directly comparable. Specifically, Fuzzilli runs continuously across many instances, with the first ones usually picking up new changes within a few hours, while BigSleep currently runs once per day and is prompted to review new commits and write proof-of-concept testcases for any bugs it discovers. Both systems report bugs through Clusterfuzz https://github.com/google/clusterfuzz , which will do basic crash triaging, then file bug reports in Chrome’s issue tracker. https://issues.chromium.org/home We also limit ourselves to V8 as a target, a high-performance JavaScript and WebAssembly engine with a sophisticated and mature testing and fuzzing setup. In particular, Fuzzilli is a special-purpose fuzzing tool designed to find bugs in these engines. The usually complicated nature of these bugs makes for an interesting “playground” to compare the two approaches’ ability to find deep and complex bugs in well-tested software. We leave it to future work to perform similar analysis on a broader set of targets, fuzzers, and agentic bug finding systems. We will now look into each of the three main questions in a dedicated section. Each section will first summarize the general approach, then describe the methodology in detail often as bullet points, so you can skip those blocks if you don’t care too much about the details before exploring the results. We conclude with an attempt at interpreting the results to get a picture of the current state of fuzzing vs. AI. Part 1: Production Statistics This is the simplest part to analyze, although with some caveats we’ll see in a bit. The basic idea is to query issues from Chromium’s bugtracker http://issues.chromium.org/ where V8 bugs are tracked and extract various statistics from there. This is especially easy since there’s a dedicated hotlist for each of the two systems. We limit our analysis to the six months from April to September 2026 as the automatic BigSleep pipeline only became active in April this year. As such, our raw data comes from the following two bug tracker queries: - Fuzzilli: hotlistid:5433121 created:2026-04-01..2026-09-30 https://issues.chromium.org/issues?q=hotlistid:5433121%20created:2026-04-01..2026-09-30 - BigSleep: hotlistid:8067258 created:2026-04-01..2026-09-30 https://issues.chromium.org/issues?q=hotlistid:8067258%20created:2026-04-01..2026-09-30 From there, the raw numbers of bug reports are: - Fuzzilli: 316 - BigSleep: 441 Small win for BigSleep. However, this includes all types of bugs, and BigSleep initially also found a lot of configuration and integration issues in V8, for example crbug.com/503869331 https://crbug.com/503869331 where the unsafe flag blocking mechanism https://source.chromium.org/chromium/chromium/src/+/main:v8/src/flags/flag-definitions.h;l=4303;drc=4bd243831302ef96b70f7e159e5685bb2eba306f didn’t work properly yet. These are legitimate issues, but less relevant for our comparison since there’s little real-world security impact. One easy way to get rid of these is to look specifically at vulnerabilities by limiting the results to type:vulnerability : - Fuzzilli: 178 - BigSleep: 95 Suddenly a clear win for the fuzzers? Not quite. There are still some problems with this data, notably duplicates and unactionable reports e.g. bugs closed as WAI <https://issues.chromium.org/issues?q=hotlistid:\\ 8067258%20%7C%205433121\\ %20status:intended behavior . At first, this seems like a simple thing to fix: just include another constraint such as status: fixed | verified . Unfortunately, things are a bit more complicated. Take for example Fuzzilli bug crbug.com/511935486 https://crbug.com/511935486 . The events on this bug are basically: - The bug is filed by Clusterfuzz on May 11 and assigned to an engineer - The bug is closed by Clusterfuzz as the testcase no longer reproduces later that same day As such, this bug received zero developer interactions. Most likely what happened here is that this bug was a duplicate of another bug, crbug.com/511210507 https://crbug.com/511210507 , but Clusterfuzz didn’t realize it as the crash signature derived from the stack trace was slightly different. When the other bug received a fix, the testcase stopped reproducing on Clusterfuzz, which closed the issue. Maybe the best thing we can do is to look at “actionable” bugs, defined as those that are not marked as duplicates or WAI and that received at least one fix commit referencing them. Then, looking at only Type-Vuln bugs filters out a lot of the less-relevant BigSleep bugs, even though these were “actionable”. With that, the final picture looks like this: So in terms of vulnerabilities, it’s roughly a tie small advantage BigSleep at the moment: 45 vs. 37. This naturally raises the question of cost: how expensive is each of the two approaches and what does that mean in terms of cost-per-bug? Unfortunately, it’s challenging to make a comparison at this stage in the research. While some effort has been dedicated to making Clusterfuzz more efficient, similar effort has not been expended for Big Sleep and neither system is particularly calibrated for an optimal cost-per-bug ratio. Part 2: Overlap The next question is how much these approaches overlap each other. For this, our approach is twofold: we first attempt to find an answer in the bugtracker, by trying to identify bugs discovered by both systems. Next, we run a dedicated experiment in which we let BigSleep try to rediscover old Fuzzilli bugs. Bugtracker Overlap The basic idea is to see which bugs were found by both systems, so we’re effectively looking for duplicate bug reports. Similar to part 1, there are a few steps we need to take to ensure we get a somewhat accurate picture: - First, we look for bugs filed by one system that get deduplicated into a bug filed by the other. For example crbug.com/522410841 https://crbug.com/522410841 which is a duplicate of crbug.com/522860268 https://crbug.com/522860268 . This is a clear overlap. - Next, we need to take into account that many “unactionable” reports those with zero associated fixes are often duplicates. For those reports, we need to find the commit that fixed them, then look at the bug associated with that commit, and treat these two bugs as duplicates. This can be a multi-prong relationship: both bug X found by BigSleep and bug Y found by Fuzzilli might be a duplicate of bug Z reported by someone/something else. In that case, that bug still counts as overlap. - Then, we need to realize that many duplicates never receive a bugtracker entry since they are already deduplicated at the Clusterfuzz level since both Fuzzilli and BigSleep rely on Clusterfuzz for basic triage and bug filing . As such, we need to query Clusterfuzz’s underlying database to find more duplicates. - Finally, we need to decide what to do with separate bugs that get fixed by the same CL, for example the four bugs fixed by crrev.com/c/7965120 https://crrev.com/c/7965120 . While this can legitimately happen for distinct bugs, it’s probably more likely for the bugs to be duplicates such as in this case , and so we count them as such. With all the different types of deduplication, we get this final picture: Again, looking at just the actionable vulnerability reports likely the best measure , we see an overlap of roughly 25% 20 out of a total of 81 , with a slight majority of these found first by Fuzzilli and later by BigSleep. Overlap Experiment The second way to measure the overlap is to run a dedicated experiment in which we try to re-discover bugs found by one system in the past using the other. Specifically, here we try to re-discover old Fuzzilli bugs using a BigSleep setup that’s as close to the production configuration as possible. For each bug we collect the commit that likely introduced the bug, and ask the agent to review those changes just as in the production CI environment. Note that the other direction, trying to rediscover past BigSleep bugs with Fuzzilli is more difficult to set up, since fuzzers cannot easily be given a similar “starting point”. Moreover, for many of these bugs we already know the outcome of such an experiment: the majority of “historic” BigSleep bugs e.g. those from 2025 https://issues.chromium.org/issues?q=hotlistid:8067258%20componentid:1456824%2B%20created%3C2026-01-01 were found after fuzzers like Fuzzilli had already been fuzzing vulnerable builds for months, sometimes years. So they would most likely also not rediscover them in an experiment except perhaps in a few cases where Fuzzilli was since updated to support new language features . Similarly, in production, BigSleep only reports crashes with a delay of 1-2 days as there’s currently only one run per day . As such, fuzzers also get a head-start there. In more detail, here is how the experiment is set up: - We start with a list of historic Fuzzilli bugs from early 2023 until mid-2026 by querying the bug tracker hotlist. This gives us around 1150 raw bugs <https://issues.chromium.org/issues?q=hotlistid:5433121%20status:\\ fixed%20%7C%20verified\\ %20created%3C2026-08-01 . - We restrict this to only type:vulnerability as these are the types of bugs we care most about and will remove some “noise” as above. We are left with roughly 600 vulnerabilities. - Next, we filter this list down to bugs that reproduce in the environment used by the agent. This involves two parts: - Bugs that reproduce on an x64 debug+asan build of the d8 shell as provided by Chrome’s public build infrastructure https://commondatastorage.googleapis.com/v8-asan/index.html?prefix=linux-debug/ . This is important because many bugs only reproduce in special builds, for example because they are architecture-specific or need certain compile-time features. - Bugs that can be reproduced without special flags. The problem here is that fuzzers often set custom flags, either to enable in-development features e.g. --experimental-wasm-jspi in crbug.com/361123483 http://crbug.com/361123483 or to turn on different fuzzing modes that make certain types of bugs easier to discover e.g. --deopt-every-n-times in crbug.com/362784006 https://crbug.com/362784006 . However, we can’t let the agent set arbitrary flags on historic V8 builds as these didn’t have a “safe-flags only” mode leading to many false-positive crashes , and we also don’t want to tell the agent which flags it should use for a given challenge as that would be a hint. As such, for simplicity we limit ourselves to bugs that can reproduce with a standard set of flags --fuzzing , --jit-fuzzing , --expose-gc and only allow the agent to set those. - - We end up with 132 historic bugs. - We now perform some deduplication similar to what we did above to identify implicit duplicates, which we remove. This leaves us with a final number of 112 bugs. - For a realistic comparison, we then convert these bugs into “code review challenges” similar to the ones used by the production BigSleep pipeline. For that, we try to identify the commit that likely introduced the bug, for example based on bisection. We note that this skews the comparison against BigSleep for two reasons: - We likely sometimes identify the wrong commit that is unrelated to the bug in question, making the task harder or even impossible. - We only have one commit per bug, while in the real world, the agent might discover the bug from another, subsequent commit that touches a similar part of the codebase. - - We now run the agent for 25 attempts on each commit. The agent is asked to find bugs related to the commit and prove their existence by writing a testcase that crashes the vulnerable build. The attempt is terminated once a crash is found, so each attempt can report at most one bug. We use essentially the same configuration that was used in production BigSleep at that time, including the same underlying model: Gemini 3.6 flash https://deepmind.google/models/model-cards/gemini-3-6-flash . - Finally, we run some post-processing on the results to check if the agent found the “right” bug or an unrelated one. For that, we check that the testcase crashes the pre-fix revision but not the post-fix revision. This is imperfect but fairly easy to implement. Here are the results from the experiment: Starting from just the commits that likely introduced the bugs, BigSleep was able to re-discover ~62% of the original bugs, while finding a different but by now fixed bug for another ~12% of commits. On around 26% of the commits it failed to find any bugs. Memorization One thing to worry about with experiments such as this one is memorization: the LLM remembering specific bugs from its pre- training data and as such having an easier time “re-discovering” them during evaluation. We can try to get some idea of the effect of this by looking at the success rates for bugs from different time spans, e.g. every half-year period for which we have bugs in the data set. If there was memorization, we’d expect higher success rates both for a single attempt and for the combined run on bugs prior to the learning data cutoff of the model. Here are the results for our experiment: For the model used in this experiment Gemini 3.6 flash , the knowledge cutoff date is March 2026 https://deepmind.google/models/model-cards/gemini-3-6-flash/ intended-usage-and-limitations . Combining that with the fact that in early 2026, Chrome bugs would only become publicly visible after ~3 months, the effective knowledge cutoff for Fuzzilli bugs should be around Jan 2026. Looking at the data, there seems to be no statistically significant amount of memorization pre-2026. We note, however, that this may be different for other models or future model generations and as such recommend that this be re-evaluated for similar experiments in the future. Difficulty A follow-up question we might ask is how the overlap compares with “difficulty”. For example, in the best case a system like BigSleep would easily find issues that fuzzers struggle with and vice versa. In that case, budget that was previously spent on long fuzzing sessions may now be re-allocated and put to different uses. Here are the results of analysing this, where “bug difficulty” is measured by the time it took Fuzzilli to discover the bug: While we do get a statistically significant result here although not of particularly great magnitude , it actually points in the other direction Bugs that are hard to find for a fuzzer are likely also hard to find for an AI agent. As such, those fuzzers should probably be kept running for now. Components Another thing we can look at is a per-component breakdown of the bugs and their rediscovery rates: Interestingly, the overlap is significantly smaller for compiler bugs. This is the area where Fuzzilli probably specializes most in, which may explain the results. Later on in part 3 , we’ll also develop another hypothesis for why some bugs in the compilers may be more difficult to rediscover for BigSleep. Prompting A final question is how much the specific prompting played a role in the performance of the AI agent. For that, we run two quick follow-up experiments, only on the bugs that weren’t previously re-discovered and with different prompts: 1. Instead of starting from commits, we start from the vulnerable file or function. This may be an easier starting point but is still a fairly realistic setup since both can for example be obtained by extracting modified files and functions from a given commit. 2. Instead of asking the agent to find memory corruption vulnerabilities what we care most about , we specifically tell it that assertion failures and similar crashes are also desirable. Running these two experiments with only 5 attempts per bug instead of the 25 used in the main experiment , the agent re-discovered 10 additional bugs in total: 8 from the different starting points and 7 with the rephrased prompt with 5 found in both runs . In total, combining these results with our original experiment, we get the following picture: So the share of “correct” bugs found climbs to ~70%, and we find some kind of bug for ~84% of starting points. However, for fairness it should be noted that equivalent tweaks to a fuzzer could likely also result in higher discovery rates for certain bugs, so the results of this follow-up experiment should be treated with some caution. Summary The main results from our two angles are: 1. In production, the overlap is noticeable but not substantial at around 25%. 2. In our experiment, overlap was around 60% of historic Fuzzilli bugs 70% with additional runs that used different prompts . 3. The overlap is not uniformly distributed: 1. Bugs that are difficult for a fuzzer to find are more likely to also be difficult to find for an agent, so there’s more overlap on “easy” bugs. 2. The overlap seems to be smaller in the JIT Compilers and higher in the rest of the engine. 4. From just the experiment, we might expect a higher overlap in production. However, there are probably a few factors that prevent ed this: - There’s a limited time window for overlaps: once a bug reported by either system is fixed, the other system no longer has a chance to rediscover it as the bug won’t reproduce on Clusterfuzz and will therefore be discarded . - Our experiment only looked at a subset of Fuzzilli bugs that reproduce on a standard x64 d8 fuzzer build and without special flags. In production, each system has access to slightly different and specialized builds, as well as different runtime flag constraints, and there the overlap is potentially smaller. - Pipeline hiccups such as overloaded servers or bugs in the automation likely prevented a number of commits from being reviewed by BigSleep in the last months, so bugs introduced by them may have been missed. - Since its introduction in April, the model underlying the BigSleep pipeline has been upgraded multiple times. Earlier models likely performed worse, and we expect future models to perform better. Part 3: Shape of Bugs In this section, we will first look at three case studies of bugs found by one system but not the other, then attempt to draw conclusions from these and other bugs. BigSleep: Chrome Bug 449760249 October 2025 This bug https://crbug.com/449760249 is a nice example of BigSleep benefiting from the underlying model’s reasoning capabilities to discover a variant that was likely too complex to trigger by a fuzzer in a reasonable amount of time. It’s also a good example to show the general working of an agentic bug finding system. Background: TDZ and Hole Check Elision Before diving into the bug specifics, we first need to look at some JavaScript internals, specifically variable initialization. In JavaScript, variables declared with the let or const keywords cannot be used before they are initialized, and attempting to do so will lead to an exception ”ReferenceError: Cannot access 'foobar' before initialization” . In V8’s terminology, these variables reside in a “temporal dead zone” TDZ until they are initialized. Internally, V8 sets variables in the TDZ to a special sentinel value, the hole , and then checks for this value when accessing the variable’s content. To limit the performance impact of these hole-checks, V8’s bytecode compiler implements an optimization behind the default-enabled --ignition-elide-redundant-tdz-checks flag https://source.chromium.org/chromium/chromium/src/+/main:v8/src/flags/flag-definitions.h;l=1215;drc=424b93eb76684237b932b5d79315d00b85297548 to identify redundant checks and subsequently remove them. The idea here is to identify “dominating” checks where it can then be guaranteed that a hole-check has already been performed for a given variable on all possible paths leading to the current position. This is a security-relevant optimization as an incorrectly removed hole-check would result in the internal the hole value leaking to JavaScript code, a condition that has historically been exploitable https://crbug.com/40057710 although new mitigations https://crbug.com/434179415 for this bug class have since shipped . The Original Vulnerability The original bug issue 40071027 https://crbug.com/40071027 - discovered two years earlier by Fuzzilli involved do-while loops containing a break statement. In a do-while loop, the loop condition evaluates after the body. If the condition contains an assignment to a variable, the bytecode compiler will mark the variable as initialized. However, if a break statement executes in the loop body in the first iteration, then the condition will never execute. The bytecode compiler originally failed to account for this, leading to an incorrectly omitted hole check and subsequent leak of the hole value. To fix this, the implementation of BytecodeGenerator::VisitDoWhileStatement https://source.chromium.org/chromium/chromium/src/+/main:v8/src/interpreter/bytecode-generator.cc;l=3153;drc=7a23bdae2d6c30b956f4f5803e3b7431a6632667 was modified in the following way: php - VisitForTest stmt- cond , ... ; + if loop builder.break labels - empty { + HoleCheckElisionScope elider this ; + VisitForTest stmt- cond , ... ; + } else { + VisitForTest stmt- cond , ... ; + } Specifically, the patch works by creating a HoleCheckElisionScope https://source.chromium.org/chromium/chromium/src/+/main:v8/src/interpreter/bytecode-generator.cc;l=445;drc=7a23bdae2d6c30b956f4f5803e3b7431a6632667 for processing the condition code in case the loop contains break statements. This means that any updates to the TDZ state of variables will be discarded once the scope object is destroyed. As such, code after the loop will see the previous TDZ state which is the correct behavior in this case . Discovery of the Variant The BigSleep agent began its analysis by examining the patch to VisitDoWhileStatement , realizing that the fix relied on loop builder.break labels - empty to decide if a protective HoleCheckElisionScope was required. It then began creating and testing various hypotheses for scenarios in which no break statement was registered with a given loop but the condition would still be skipped at runtime. For example, it hypothesized that a try-catch in combination with a throw inside the loop might confuse the hole-check elision logic. It then investigated how try-catch was implemented in the bytecode compiler and concluded that this was handled correctly. Eventually, the agent started investigating labeled breaks https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/label , which allow directly breaking out of inner loops. It hypothesized that these might not be handled correctly and continued to investigate how these were being processed, for example by studying the logic in BytecodeGenerator::VisitBreakStatement . This time, it seemed to have identified a real flaw: in the case of nested breaks, the inner loop was apparently not being informed of the existence of a break statement It therefore formulated the following vulnerability hypothesis: 1. If a labeled break inside a do-while loop targets an outer scope, the inner do-while loop’s LoopBuilder is not notified. 2. Consequently, loop builder.break labels - empty remains true . 3. Therefore, when generating the code for the loop condition, the HoleCheckElisionScope introduced by the patch is not used. 4. The condition is bypassed at runtime, but the bytecode compiler erroneously removed a later hole-check, leading to a leak of the hole to JavaScript. The agent then began writing a testcase to prove the existence of this bug. The initial attempt with simply two nested loops and a label failed, however, as the bytecode compiler would add another protective scope object for the labeled block https://source.chromium.org/chromium/chromium/src/+/main:v8/src/interpreter/bytecode-generator.cc;l=2099;drc=77ce2c33f61b01daac3d9e4299b3d0805bee6461 . To work around this, the agent decided to use a switch statement. When handling a switch statement, the bytecode compiler uses a HoleCheckElisionMergeScope to compute the intersection of initialization states across all branches. By ensuring that the variable was initialized in a default case, the merge logic then concluded that the variable was correctly initialized on all paths. Proof of Concept With that, the agent had everything in place to produce a testcase to demonstrate the vulnerability. Below is the agent’s PoC: // Flags: --ignition-elide-redundant-tdz-checks function trigger cond { { target: switch 1 { case 1: do { // Labeled break targeting the switch. The do-while LoopBuilder remains unaware. if cond break target; } while x=1 && false ; // Condition performs a check/assignment on x. break; default: x=1; // Ensure x is considered initialized across all switch branches for merging. } // Generator incorrectly believes x is checked/initialized. Elides TDZ check. x + 1; let x; } } trigger true ; Here is how this testcase works: 1. trigger true executes, entering case 1 of the switch . 2. The do-while loop executes break target . 3. Control flow jumps to the end of the switch , bypassing the do-while condition where x is assigned. 4. At runtime, x remains uninitialized holding the hole . 5. Because the TDZ check was elided at compile time, the engine leaks the internal the hole value to JavaScript code in the x + 1 expression, leading to a potentially exploitable condition. We can see how this bug would be difficult for a fuzzer to find since the testcase is quite complex. While a coverage-guided fuzzer would have received rewards for triggering individual code paths such as the handling of labeled break, the switch statement, the do-while loop, etc. in the parser , there is no additional reward for putting these together “in the right way” and so ending up with a testcase like this is quite difficult. BigSleep: Chrome Bug 443765373 September 2025 This bug https://crbug.com/443765373 is an example of a relatively simple bug type that is challenging to discover with standard fuzzing approaches. Background: Exception handling in V8 To understand this bug, it is necessary to briefly look into exception handling in V8. Take the following example JavaScript code: function simple { try { throw 42; } catch e { return e; } } Here, we have a function consisting of a try-catch block in which an exception is raised. Running this in d8 with --print-bytecode , we can see the internal bytecode that the engine generates to execute the JavaScript code: generated bytecode for function: simple 0x2ad5001000e4 @ 0 : 1b ff f9 Mov <context , r0 0x2ad5001000e7 @ 3 : 0d 2a LdaSmi 42 0x2ad5001000e9 @ 5 : b5 Throw 0x2ad5001000ea @ 6 : d1 Star1 0x2ad5001000eb @ 7 : 8d f8 00 CreateCatchContext r1, 0 0x2ad5001000ee @ 10 : d2 ... ... 0x2ad5001000f7 @ 19 : b7 Return Handler Table size = 16 from to hdlr prediction, data 3, 6 - 6 prediction=1, data=0 For the purpose of this bug, the relevant part is mostly the last line of the output. Here, the bytecode essentially says “if there is an exception while the PC is between offset 3 and 6 in this bytecode, then execution should continue at offset 6 the start of the exception handler ”. The Bug The agent spotted the bug while looking at the following function, responsible for writing the handler table data structure in memory: void HandlerTable::SetRangeHandler int index, int handler offset, CatchPrediction prediction { int value = HandlerOffsetField::encode handler offset | HandlerWasUsedField::encode false | HandlerPredictionField::encode prediction ; int offset = index kRangeEntrySize + kRangeHandlerIndex; Memory<int32 t raw encoded data + offset sizeof int32 t = value; } Here, the int offset is being combined with a few other bits of data into a new int value. As such, the handler offset cannot have the full integer range in the new field, possibly leading to integer truncation issues. The agent then investigated the specific bounds of these values and realized that the HandlerOffsetField had only 28 bits allowing for 256MB while the maximum bytecode size was 512MB From there it was relatively simple to craft a testcase that crashes, simply by creating a huge function resulting in 256MB of bytecode with an exception handler at the end, then triggering an exception inside of it. The result is a jump to a wrong bytecode offset, a primitive that’s easily exploited. Here is the relevant part of the testcase: js // yield statements turn into a large chunk of bytecode let kNumYields = 500000; let body = if "foo" === "bar" { ${"yield 42;".repeat kNumYields } } try { throw 42; } catch e { // Will never get here } ; Such a testcase would again be difficult to craft by a fuzzer as the fuzzer isn’t rewarded for creating large code at least not with generic coverage feedback . In fact, the fuzzer is likely even actively discouraged from finding this bug since generating and executing large bytecode takes some time in debug builds, the testcase generated by the agent needs a few seconds to trigger and would therefore probably have led to timeouts during fuzzing. Fuzzilli: Chrome Bug 40076141 November 2023 This bug https://crbug.com/40076141 is a good example of a type of bug that is likely difficult to find just by looking at one part of the source code. It was originally discovered by Fuzzilli and not rediscovered by any of the experiments from part 2. Background: Value Representation in V8 Consider the following piece of JavaScript code: js let i = 42; let f = 13.37; let s = "foobar" i = s; Here we create multiple variables holding values of different types. Note, however, that we never declare the type of a variable, and that variables can hold values of different types throughout their lifetime i first contains an integer, later a string . What we see here is that JavaScript is a dynamically typed language, and so variables and also object fields, etc. have no fixed type. However, the actual runtime values still need to have a type: the engine somehow needs to know that foobar is a string value and that certain operations e.g. substring search can therefore be performed on it. Conceptually, this is possible by representing runtime values with a struct such as enum JSType { JS TYPE INT, JS TYPE FLOAT, JS TYPE STRING, ... }; struct JSValue { JSType type; // Either the raw value or a pointer to another object, e.g a JSString. uintptr t value; }; However, this is somewhat inefficient since we now need something like 16 bytes to represent a simple 4-byte integer at runtime. As such, modern JS engines use clever tagging schemas, which rely on unused pointer bits to encode type information. Specifically, in V8 the JSValue type is a tagged pointer or just tagged for short which stores one bit of type information in the least-significant bit: - If the bit is one, then the value minus 1 is a pointer to another object. This works because objects are aligned and so cannot start at an uneven address. - If the bit is zero, then the next 31 bits contain a raw integer value also called a small integer - Smi As such, to figure out the type of a value at runtime, the engine first looks at the least-significant bit. If that is zero, the value can be “untagged” to obtain the integer value by right-shifting by one. On the other hand, if the bit is one, the value is a valid pointer to an object that contains further type information. For more details on V8’s pointer tagging scheme see also this blog post https://v8.dev/blog/pointer-compression value-tagging-in-v8 . The Testcase Let’s start by looking at the testcase produced by the fuzzer, which isn’t overly long: js function f0 { let v2 = 1; for let i3 = v2; i3 = v2; i3 += v2 { v2 = v2 && i3; } } f0 ; The bug is caused during optimization of the loop by the Maglev compiler, specifically due to a conflict between two stages of the compiler pipeline: the initial graph builder and the later MaglevPhiRepresentationSelector. The Graph Builder The graph builder is responsible for building the initial Maglev intermediate representation IR of the code. For that, it consumes both the input code in the form of V8 bytecode as well as runtime feedback that was collected while the code executed in lower tiers such as the interpreter. When processing the loop in this testcase, the following happens: 1. Both i3 and v2 are modified during the loop. Since the IR uses static single-assignment SSA form https://en.wikipedia.org/wiki/Static single-assignment form , these turn into Phi nodes, which make the code look like every variable only has a single assignment : 2. For the i3 = v2 comparison, the runtime feedback states that i3 was so far always a Smi and therefore speculates that this will be the case in the future. It guards this assumption by emitting a CheckedSmiUntag , which will extract the raw int from the tagged value and will bail out to the interpreter if the input is ever not a Smi. 3. For the following operations on v2 , the compiler can then rely on the existing check and emit UnsafeSmiUntag nodes, which avoid the runtime checks but are only safe to use if the input is guaranteed to be a Smi. The resulting code for the loop now looks roughly like this in Maglev pseudocode : loop: i3 <- Phi 1, i3 next // loop variable, tagged v2 <- Phi 1, v2 next // loop variable, tagged i3 int <- CheckedSmiUntag i3 // deopt if i3 is not a Smi v2 int <- CheckedSmiUntag v2 // deopt if v2 is not a Smi if i3 int < v2 int: break v2 next <- Phi i3, v2 // result of v2 && i3 : "known Smi" v2 next int <- UnsafeSmiUntag v2 next // no re- check needed i3 next int = i3 int + v2 next int i3 next <- Int32ToNumber i3 next int // tag the int32 result for the phi The MaglevPhiRepresentationSelector The purpose of this optimizer is to identify Phi nodes where all inputs have the same raw representation e.g. int32 , and if so convert the Phi to that representation to avoid future untagging operations. For the code above, the code sees that both inputs to i3 are int32 and therefore decides to convert the Phi to int32 representation. Following CheckedSmiUntag operations then become a no-op and can be removed, while consumers that need a tagged value now require a conversion step. With that, the final code now looks like this: loop: i3 <- Phi 1, i3 next // all inputs int32, so i3 becomes a raw int32 v2 <- Phi 1, v2 next // stays tagged v2 int <- CheckedSmiUntag v2 if i3 < v2 int: break // CheckedSmiUntag i3 removed as a no-op i3 tagged = Int32ToNumber i3 // HeapNumber if i3 = 2^30 v2 next <- Phi i3 tagged, v2 // stays tagged, needs tagged inputs v2 next int <- UnsafeSmiUntag v2 next // HeapNumber pointer treated as a Smi i3 next <- i3 + v2 next int // tagging no longer needed here The Conflict The bug now occurs because of two optimizations interacting badly with each other: 1. The graph builder uses UnsafeSmiUntag as it believes that the value must be a Smi at this point 2. The representation selector removes a CheckedSmiUntag as it believes it is no longer required Specifically, what can happen now is that eventually, i3 becomes larger than the maximum Smi value. At that point, Int32ToNumber can no longer fit the integer into a Smi and instead allocates a HeapNumber object a container holding a larger number, basically . At this point, however, the UnsafeSmiUntag is no longer safe as the input is now a pointer. This leads to a corrupted pointer value being extracted and subsequently to memory corruption. Note that in the original code, this could not have happened as the HeapNumber would’ve been rejected by the CheckedSmiUntag i3 . What may have made this bug difficult to find for BigSleep is the fact that both optimizations are correct in isolation . As such, looking at just one of them would’ve made it difficult to spot the issue. The agent would somehow have needed to hypothesize about the other optimization or stumbled upon the relevant code by chance or exhaustive search . Overview Looking at the examples above as well as other bugs found by LLMs, we can see a few clear examples of bugs where an LLM has the advantage: - Bugs that require complex logic to trigger but can be found through progressive logical reasoning and experimentation with the target program, such as crbug.com/449760249 http://crbug.com/449760249 discussed above. This is likely the strongest advantage. - Bugs that require specific “magic” values e.g. edge-case integers, specific strings, etc. , such as crbug.com/443765373 https://crbug.com/443765373 discussed above. - Bugs that require new or less common features that a fuzzer may not be aware of. Examples include crbug.com/436181695 https://crbug.com/436181695 , where the fuzzer didn’t support the new await using syntax yet, and in fact also crbug.com/449760249 http://crbug.com/449760249 above where Fuzzilli lacked support for labeled breaks at the time. - Bugs that require a non-standard harness, custom flags, or special helper tools/functions that a fuzzer is not configured to use e.g. crbug.com/516833326 https://crbug.com/516833326 which required --expose-fast-api and the d8.test.FastCAPI API . On the other hand, here are some advantages that fuzzers enjoy for now : - They can find “non-local” bugs reasonably well if not too many individual pieces are required. crbug.com/40076141 http://crbug.com/40076141 discussed above is an example for such a bug. - They can find bugs that are hard to spot by reasoning about the source code alone. One clear example would be bugs caused by incorrect C++ compiler optimizations such as crbug.com/500771385 https://crbug.com/500771385 . Another potential example is bugs such as crbug.com/414840455 https://crbug.com/414840455 , which are related to handling dead code in V8’s compilers. This is likely an unusual state and therefore may be hard to reason about. BigSleep missed a few of these in our experiments. - They are immune to certain biases affecting the agentic approaches, such as the choice of starting point e.g. commits , the exact prompt phrasing, or the bug types seen and not seen during training. They do suffer from other types of biases though e.g. the weights of individual mutators and generators . - They don’t get distracted by “shallow” bugs, whereas BigSleep missed a few bugs because it found easier ones instead. This is probably somewhat self-correcting though as the shallow bugs would eventually get fixed, giving the agent another chance to discover the deeper bug. There is probably also a category of bugs that are hard to find for both approaches: bugs that are very non-local bad for the agent and require complex testcases bad for the fuzzer to trigger. In this case, a fuzzer would probably discover the individual pieces of the testcase through coverage feedback but won’t be able to find the right combination to trigger the bug as in our first example . A code-reviewing agent, on the other hand, may struggle with the large search space of code fragments that may interact badly with each other as in our last example . This is only a hypothesis, however, and would require further experiments and data to validate. Part 4: Conclusion In this blog post, we’ve compared coverage-guided fuzzers with vulnerability-discovery agents that read and reason about the source code. Specifically, we’ve looked at Fuzzilli and BigSleep, which both run as automated pipelines on V8 to discover newly introduced bugs. Looking at the data from those pipelines, we saw a roughly equal share of vulnerabilities found in the last 6 months around 40 each , and a moderate overlap in discovered bugs of around 25%. However, in a dedicated experiment, overlap was closer to 60%. For now, this means that the two approaches remain complementary, and specialized fuzzing continues to provide value by finding bugs quickly and identifying certain kinds of issues that are otherwise hard to spot. At the same time, LLMs only very recently became capable of finding bugs in software like V8 at all, and have made tremendous progress since. The model used for our experiment is by now already outdated and outperformed by newer generations. Looking to the future, we expect further model improvements, potentially coupled with better prompting and more flexible harnesses that e.g. allow the model to build, improve, and run fuzzers itself instead of being limited to pure code auditing , to lead to a growing share of bugs being discovered by agentic systems.