- HEADINGS...
Using an early internal version of Incredibuild’s Software Factory, we landed six upstream Bazel contributions, uncovered a remote-cache path-containment security flaw, and came away with a concrete set of requirements for automated software engineering. A companion to our Bazel series: Part 1, Part 2, and Part 3.
Generating code is getting fast and cheap. Building, testing, validating, and deciding what reaches main is where agents, and people, now stand waiting.
For more than two decades, Incredibuild has focused on one problem: shortening the build-test loop without forcing engineering teams to reinvent how they build software. That began with distributed compilation and grew into caching, CI, testing, telemetry, and build integrity.
AI is moving the bottleneck. When an agent can draft a patch in seconds, the long pole is everything after it: the build, the test run, the evidence, the review. That’s the problem our Software Factory is built to solve. Before pointing it at anyone else’s code, we pointed it at our own products and at the ecosystem around us, and Bazel is a big part of that ecosystem.
TL;DR #
- Six upstream Bazel contributions from our Software Factory were imported to Bazel master through Google’s Copybara workflow.
- Along the way, we found a path-containment flaw in
--experimental_repo_contents_cache, reproduced it end to end on a shipping Bazel 9.0.2 binary, and reported it privately. - Google fixed it upstream in c37a6a1bf7f4 . The fix is inBazel 9.3.0rc2 ; Bazel 9.2.0 still ships the pre-fix logic.
- If you use that flag: upgrade to a build that contains the fix, or turn the flag off until you can.
Why Bazel was the right proving ground #
Bazel models software as a graph of declared work, makes caching and remote execution first-class concepts, and treats reproducibility as part of correctness. Those are the same problems Incredibuild has spent two decades solving from a different layer of the stack. (If you want the full background, start with Bazel, Inside and Out.)
It’s also a demanding target. Bazel’s Remote Build Execution (RBE) model is powerful, but adopting it is a real engineering project: hermetic actions, rule coverage, toolchains, platforms, and REAPI infrastructure all have to line up. And real repositories rarely live entirely inside a clean action graph. They contain nested Make or Ninja builds, custom compilers, test harnesses, scripts, packaging, code generation, and proprietary tools.
Our approach is complementary. Bazel orchestrates the build; Incredibuild provides caching and distributed execution across the existing workload, including the parts that are hard to move to RBE. Teams get faster builds now, while the RBE migration continues on its own schedule.
So Bazel mattered to us twice: as a workload we accelerate, and as a build system we have to understand all the way down to its cache boundaries.
What the Software Factory actually does #
The Software Factory isn’t an “agent writes code” demo. It runs the whole engineering loop as a factory line:
goal → plan → isolated task → implement → build → test → inspect → retry → reviewable PR
- Orchestration manages the project as a DAG of tasks.
- Islo sandboxes isolate every task.
- Incredibuild accelerates the expensive build and test loops.
- Agents explore aggressively, but tests, evidence, and human review decide what gets promoted.
One rule doesn’t bend: automated search and model output can propose patches, run experiments, or return cached results. Test verification and human review keep authority over what merges to main.
Six contributions landed upstream #
Six changes from this effort were imported to Bazel master through Google’s Copybara workflow. None of them is flashy. All of them are about the same thing: making automation trustworthy.
| PR | What changed | Principle |
|---|---|---|
| #30365 | Made bazel-distfile.tar reproducible by removing locale-dependent ordering, and made unknown archive types fail explicitly | Determinism |
| #30358 | Skipped the SSL tracking-issue mutation step on pull-request validation runs. A PR can validate the monitor, but must never create or update production tracking issues | Authority boundaries |
| #30364 | Ignored COMMENTED reviews when computing community-review status, so a comment can no longer erase a prior APPROVED or CHANGES_REQUESTED decision | Authority boundaries |
| #30363 | Parsed certificate expiration independently of host locale, so monitoring no longer depends on an unrelated LC_TIME setting | Determinism |
| #30360 | Corrected the documented BAZEL_DOC_TRIGGER_TOKEN scope to bazel-docs, the repository the workflow actually dispatches to | Correct configuration |
| #30359 | Paginated PR-review retrieval and made it fail closed if a later page fails | Complete evidence |
Links to each change: PR #30365 · PR #30358 · PR #30364 · PR #30363 · PR #30360 · PR #30359.
The pattern is the point. Deterministic outputs, complete evidence, clear authority boundaries, and safe failure modes aren’t nice-to-haves for automated software engineering. They’re the difference between an agent you can let run and one you have to babysit. Partial evidence must never be treated as complete evidence.
Then the cache boundary got interesting #
As we went deeper into Bazel’s caching architecture, we examined --experimental_repo_contents_cache.
The feature restores the output of reproducible repository rules from a remote Action Cache (AC) and Content Addressable Storage (CAS). That’s good for performance, but it moves the trust boundary: cache-supplied metadata now participates in filesystem materialization.
Before the fix, Bazel trusted cache-supplied tree node names without sufficiently enforcing that the resulting path stayed inside the intended external repository directory. A node name containing parent-directory traversal or an absolute path could land outside it.
We didn’t stop at reading source. The Software Factory generated a reproducible, end-to-end test case against a real Bazel 9.0.2 binary. Under the required conditions, a poisoned remote-cache entry caused Bazel to overwrite a pre-existing file outside the repository root, while the build itself still completed successfully.
Who is actually exposed
All three conditions have to hold:
| Condition | How to tell |
|---|---|
| --experimental_repo_contents_cache is enabled with a remote cache | The flag appears in a .bazelrc, CI config, or wrapper script |
| Someone untrusted can write to the relevant remote AC / CAS | Review who holds write credentials to your shared cache |
| Your Bazel binary predates commit c37a6a1bf7f4 | bazel --version reports 9.2.0 or another build without the fix |
This is not a drive-by internet vulnerability. But that’s exactly why it matters: once cache metadata can steer filesystem writes, anyone who can write to your remote cache is part of your trusted computing base.
We reported the finding privately to the Bazel security team, with the reproduction attached.
Google fixed it upstream #
Google’s Bazel team fixed the issue in commit c37a6a1bf7f4, “Improve path validation in remote repository contents cache.” The fix:
- rejects remote tree node names that contain separators, parent-directory traversal, or absolute paths;
- checks that injected file, directory, and symlink paths stay inside the repository directory;
- resolves symlink targets, absolute and relative, and verifies they stay inside it too;
- adds a regression test asserting that invalid paths are rejected.
That’s the right place for the check: the exact boundary where untrusted cache metadata becomes a filesystem path.
What to do if you use --experimental_repo_contents_cache #
1. Find out whether the flag is on. Search every place Bazel options come from:
grep -rn "experimental_repo_contents_cache" \
.bazelrc ~/.bazelrc /etc/bazel.bazelrc ci/ tools/ 2>/dev/null
2. Check the exact binary you run, not the one you think you run. With Bazelisk, also check .bazelversion and USE_BAZEL_VERSION.
bazel --version
Bazel 9.2.0 still contains the pre-fix path logic. Bazel 9.3.0rc2 contains the path-containment validation.
3. Upgrade or disable. Move to a Bazel build that contains c37a6a1bf7f4. If your organization can’t deploy a release candidate, disable --experimental_repo_contents_cache until a stable release with the fix is available.
4. Treat cache write access as a security boundary, not a performance setting. Know exactly who and what can write to your shared remote Action Cache and CAS.
What this taught us about automated SDLC #
Build acceleration runs on structured graphs, deterministic inputs, reusable outputs, process isolation, scheduling, and defined trust boundaries. Those same primitives are the foundation of automated software engineering:
- Caching , because agents repeat work at enormous scale.
- Distribution , because once coding gets cheap, build, test, analysis, and validation become the long pole.
- Isolation , because many agent loops run concurrently.
- Observability , because humans need a receipt, not a promise.
- Determinism , because exploration and promotion can’t have the same authority.
Bazel turns source code into an explicit execution graph. The Software Factory extends the same idea upward: it turns the software development lifecycle itself into a managed graph.
The scorecard #
We ran the system against an active open-source project to see whether its architecture holds up against real maintainer feedback, build failures, iterative changes, and security disclosure:
- Six upstream contributions landed.
- A security boundary was found and reproduced on a shipping binary.
- The issue was disclosed privately and responsibly.
- Google fixed it upstream.
After more than two decades of making builds faster, the next step is making the entire engineering loop faster, from intent to tested, reviewable, evidence-backed software, without giving up the controls that make software trustworthy.
Sources #
Yossi Eliaz
- HEADINGS...
Shorten Your Builds
Incredibuild empowers your teams to be productive and focus on innovating.