How we use agents to patch and deploy 40M lines of Chromium code in < 1 day Browserbase maintains an internal Chromium fork with more than 18,000 lines of stealth and security patches and now uses coding agents to port those patches to each new upstream release in under one day, down from a days-long manual process. Google ships a major Chromium version every 2 weeks, and Chromium draws 400-500 new CVEs per month, so Browserbase says its pipeline produces fully tested patches before Google officially releases the next major version, with some patches upstreamed to the official project. The fork covers more than 40 million lines of code, and the company says the 6-hour build was previously a nightly risk. At Browserbase, we maintain an internal fork of Chromium with 18,000 lines of stealth and security patches to enable agents to browse the web without being flagged as malicious bots. It also happens to be one of the largest open-source projects, with 40 million lines of code. That’s equivalent to roughly 1,125 copies of the bible. Chromium’s size also means that it’s one of the largest attack surfaces on the planet, with 400-500 new CVEs per month. https://nvd.nist.gov/vuln/search?keyword=chrome&resultType=records Our customers rely on us to keep the browser running their sessions up to date. When upstream fixes a vulnerability, we need to ensure that we minimize the time between the release and our fix. On top of that, Google releases a major version of Chromium every 2 weeks previously every 4 . To upgrade our custom fork, it used to take days to move our patches to the new source code in a time before coding agents , and we spent long nights praying the 6-hour build would succeed overnight. In the last 2 years, we’ve drastically improved our deployment pipeline, taking what used to be a week-long sprint into fully tested patches before Google officially releases the next major version. Some of our patches https://chromium-review.googlesource.com/c/chromium/src/+/8504035 were upstreamed to the official project. In this blog post, we’ll dive into Chromium, why it’s so hard to deploy, how we maintain our patches, what made builds suck, and how we use coding agents to streamline our workflow. Tell me more about Chromium For those who don’t know, Chromium is an open-source https://github.com/chromium/chromium , multi-process browser built on the V8 JS engine that powers Google Chrome. It’s the base for many browser products today: Dia/Arc, Brave, MS Edge, and, of course, Browserbase. So what actually makes a browser upgrade that hard? The source code is public, and we already know how to compile it. Why can't we just pull the latest version, apply our changes, and ship it? Chromium is an extremely complex piece of software, and is basically a full OS https://chromium.googlesource.com/chromium/src/+/main/docs/process model and site isolation.md . It has access to hardware resources https://developer.chrome.com/docs/capabilities like USB, Bluetooth, serial, HID, GPU, sensors, and printing, each exposed to web pages through permissioned interfaces. A full process-and-execution environment where a privileged browser process starts, monitors, and can kill dozens of isolated child processes with inter-process communication Mojo https://www.chromium.org/chromium-os/developer-library/reference/cpp/cpp-mojo/ and full sandboxing https://chromium.googlesource.com/chromium/src/+/master/docs/design/sandbox faq.md with a JIT JS/WASM compiler and GC to run untrusted code from any website. A full networking stack https://www.chromium.org/developers/design-documents/network-stack , including DNS, HTTP/1.1, HTTP/2, HTTP/3, and QUIC, TLS via BoringSSL, caching, cookies, and proxy handling. Memory management and protection through IndexedDB, Cache Storage, SQLite databases, and quota management https://chromium.googlesource.com/chromium/src/+/main/storage/browser/quota . To interact with it, there’s a GUI rendering engine Blink https://developer.chrome.com/docs/web-platform/blink , 2D rasterization, translation of OpenGL ES to D3D https://github.com/google/angle , WebGPU, audio and video decoding https://chromium.googlesource.com/chromium/src/+/main/media FFmpeg, libvpx, dav1d , DRM, WebRTC for real-time communication, and camera and microphone access. On top of all that, there’s a full product layer with extensions, sync, autofill, and password management, built as several hundred feature components https://chromium.googlesource.com/chromium/src/+/main/components on top of everything above. Applying patches by hand To start, why are we patching Chromium in the first place? Surely Google knows what they’re doing. We classify our patches into 3 major categories: security, performance, and stealth. Security patches help us defend ourselves and our customers from CVEs and potential vulnerabilities. We’re in an unprecedented time in cybersecurity and must take a proactive approach to keeping our customers safe. Performance patches help us run our browsers faster and more memory-efficient, so we don’t have to watch paint dry while our builds compile. Stealth patches help us reliably access websites across the internet. If you’d like to learn more, send us a message. https://www.browserbase.com/contact Back in 2024 before the initial release of our product , we applied patches by hand. We kept a running list of patches in Notion and would hand-apply them over the course of a week. You had to open the old and new implementations side-by-side and figure out where the old modification belongs. Sometimes, old parts of the patch weren’t necessary anymore, so we found which parts to keep and which parts were redundant, by hand. To put things into perspective, the SOTA coding model at the time was Claude 3.5 Sonnet and Nvidia’s valuation was only $3.011T. Agents weren’t really a thing yet, and AI wasn’t good enough to handle these changes. This system was functional, but not efficient. You can easily find out which file needs to be changed, but not how to change it or why. Chromium source already has its own test suite https://www.chromium.org/developers/testing/browser-tests/ , but that doesn’t equate to it being ready to roll up and deploy. It has to cover our patches, infra shape, deployment config, and how automation clients Stagehand https://www.stagehand.dev/ , Playwright, etc. use our browser. To solve this, we use named patches, better intent documentation, and exhaustive regression tests to give coding agents enough context to handle upgrades. Agents run in loops following this pattern: inspect a rejected patch, find where the behavior moved upstream, regenerate the diff, replay the series, and test. Patch regressions Just because we apply a patch properly doesn’t mean the browser behaves correctly. Let’s walk through an example with page forward and back. Normally, a page navigation loads a document, and Chrome’s back/forward cache keeps the old document, and it’s JS heap to restore it if you come back. The cache means the browser can skip fetching, parsing, and executing whatever JS the page has. A restored page receives pageshow with persisted set to true . How the back/forward cache work https://web.dev/articles/bfcache s. However, Playwright https://playwright.dev/python/ maintains its own state using CDP https://www.browserbase.com/blog/what-is-cdp events and tracking the frame, doc, and which lifecycle events have arrived. Our tests covered Chromium restoring a page and sending Page.frameNavigated , but the automation client was still waiting for the restored document’s lifecycle events. This means an automation call would hang until the navigation timeout. Our patch re-announces DOMContentLoaded and load after the restore's frameNavigated , using the restored document's loader ID. To write a good test, you have to make sure load events fire, the automation call finishes, and the document actually gets restored. Something close to this: Unit tests aren’t enough Sometimes all tests passed, but it still missed issues. In our upgrade to Chromium 157 HTTPS-First balanced mode was enabled by default. This means navigations to an HTTP site could be retried as HTTPS https://developer.mozilla.org/en-US/docs/Glossary/HTTPS and fail since there’s no native forward navigate button on headless chrome instead of loading the HTTP page. Our binary tests passed since they used loopback addresses loopback hosts and ports other than 80 were exempt from this behavior . We caught a failure in application verification checking if the new version works on our infra when loading HTTP sites with the config we actually use. To add that case to our suite, we gave the browser a public hostname on port 80, mapped it to a local server, and made the HTTPS destination refuse connections. It looked a little like this: The HTTP server is still local. We're just changing the URL the browser evaluates, so its navigation policy takes the path it would take for a public HTTP site. The original patches inadvertently avoided that policy. To ensure our Chromium works, we must retain separate patch, binary, and application checks. Binary and patch tests check for local invariants, while application tests help ensure our browser’s behavior stays consistent in production. Hold on, 6-hour builds? Yes, you heard that right. It used to take ~6 hours to compile and build Chromium after we applied our patches. The team spent many late nights preparing to kick off a build at midnight and sleeping with one eye open to make sure build errors didn’t ruin the morning. Chromium has a long history of build infrastructure— Ninja https://ninja-build.org/ originated from work on Chromium. It’s designed https://ninja-build.org/manual.html to put a policy in a generator and keep execution focused on an explicit dependency graph. The current Chromium build https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build instructions.md uses GN to generate .ninja files the graph and Siso as its main build tool action explanations . You’d think caching meant keeping old obj. files, but the build must also believe those files are still valid. We dove into the source code and found two independent causes of unnecessary full rebuilds. Both made a small Chromium update look much larger to the build system than it really was. Cache money Our cache directories used to include the Chromium version, which sounds reasonable enough. Each version gets its own directory, and you can immediately tell what’s in it, but every version bump changes the entire workspace. There were absolute paths in the build’s dependencies, compiler commands, and build state. The build system couldn’t recognize over 3,586 build steps as work it had already completed because moving the workspace changed the paths in its commands. The solution was changing to a stable workspace across all versions of Chromium, which required a one-time rebuild so all subsequent updates could reuse it. Now the only thing that triggered a rebuild was an update to src. Fixing our Linker Now that compilation is much faster, we addressed the linking bottleneck. During testing, we found that in a 346-second build step, compilation took 128 seconds while linking took 151 seconds. Linking https://people.cs.pitt.edu/~xianeizhang/notes/Linking.html is how the build process connects the compiled object files, resolving references, pulling in required libraries, and laying out the code and data in the final executable. In the example above, it was ~94% serial. The obvious solution is to parallelize or so we thought . The linking process uses ThinLTO https://clang.llvm.org/docs/ThinLTO.html , which lets the compiler optimize across translation unit boundaries and combine module summaries during linking. We added more linker threads, but found that it didn’t actually improve the measured link speed. Instead, we had to investigate which part of the link was slowing down the system and what runtime benefits we would trade off by removing it. We found a way to cut roughly 54 seconds from linking by switching the algorithm that arranged functions in the executable. The original config used cdsort cache-directed sort , which uses a more detailed locality model, considering function sizes, call frequencies, and distances between callers and callees to choose an arrangement. In experimentation, we switched it to hfsort , which uses simpler rules and groups a function with its most frequent caller, subject to size and execution-density limits, then orders those groups by how frequently they execute relative to their size. Here’s what they look like in code. https://github.com/llvm/llvm-project/blob/main/lld/ELF/CallGraphSort.cpp Although the link became faster, this config actually slowed down our browser startup time by 3.33%, so we eventually switched back to keep our browsers fast for our customers. Taking Security seriously It seems like every day a new vulnerability is released. A side effect of LLMs getting better at coding is that they’re very good at finding supply chain attacks and critical vulnerabilities in even our most robust software. It’s our job to protect our infrastructure and those who use it from these vulnerabilities. By tightening our iteration and deployment cycles, we can take a best-effort approach to secure Chromium. If you think about a browser automation session, it takes arbitrary code and content from the internet an extremely large place and processes it, while potentially being signed into customer applications. Thankfully, Google already considered this, and Chromium addresses isolation natively. The browser itself is a sandbox, so a renderer exploit doesn’t automatically give an attacker control of the host. It was designed to restrict compromised code and explicitly assumes that sandboxed code may be malicious. Pre-firing the release Chromium's source is public, so we can follow their updates as they prepare for a release. We’ve built internal tools to watch for upstream tags, their ancestry, and file diffs. For example, if a tag with code changes is followed by a version-only tag, we know that we should attempt to build that version. There are other patterns we check for, but we’ll keep those proprietary. Coding agents are extremely useful here. We invested time in building an environment for background agents to inspect dependency and compiler changes, replay patches, investigate rejects, remove obsolete backports, check new defaults, and collect and build tests. Since the environment sets both non-deterministic skills and explicit CI guardrails, we can just run agents against it instead of manually sifting through the code like before. Are we actually running the new browser? Once we have the full build, we need to deploy the binary to our VPC. We follow a standard deployment process, bundling our binary into an image, pushing that image to ECR, then deploying a canary version to a small percentage of our machines, monitoring for a bit, then rolling out to all of our customers. To validate deployment, we compare the proposed version across binaries and the connected browser session. Agents go out and collect evidence for us across security fixes, versioning, and test results. It must have the right browser and driver executables, the version reported through Playwright, and CDP’s Browser.getVersion . We also compare the built binary against the prod runtime base, since the binary that loads in the builder can depend on libraries or symbols that are unavailable in the runtime environment. If you want to roll back versions, you unfortunately can’t just put the old executable back. The new version might have already migrated a user data directory into a new format that the old version can’t read. Scar tissue is the moat There were many sleepless nights that our team endured to make this possible. A large part of the value-add is the years of scar tissue that our engineering team has from working on Chromium. Interestingly, a majority of the improvements we made came from interpretability, understanding the system, and making it easier to reason about. We learned why a patch existed or why an object needed to be rebuilt. With improvements to LLMs, making the system as understandable as possible becomes a rich environment for agents to gather maximum context and iterate in loops until they satisfy requirements. Shorter feedback loops let us try multiple optimization paths and only keep the best ones. We can also look ahead and fix security issues before the release happens. It used to take a full week of pain and suffering to deploy every other version of Chromium even with the 4-week cycles . Now we can test and build a stable version before Google with roughly 1000x fewer engineers . Shoutout to Claude. If any of these problems sound interesting to you, we’re hiring across various roles including engineering https://www.browserbase.com/careers . → Kyle Start building with Browserbase Run headless browsers for your agents and automations at scale. Get started for free in minutes.