What Anthropic’s Scion design means for data

Anthropic replaced git with a new source-control system called Scion after its daily merged code changes grew from about 2,500 in May 2026 to over 30,000 by September 2026, mostly driven by agents, according to Anthropic's engineering post on redesigning source control. Scion runs commits, merges, rebases, history and blame on the server, makes checkout optional and sparse, stores all source-control data content-addressed and immutable, and moves a branch via "a compare-and-swap on a single Spanner row," with metadata and refs in Spanner and file contents in cloud storage. The post's author notes Scion's storage design mirrors how data versioning systems such as lakeFS already work, but diverges on snapshots: Scion changesets store only files changed relative to the first parent, while lakeFS commits point to a two-level Merkle tree describing full repository state, a trade the author favors for answering what exact data a model trained on.

In May 2026, Anthropic merged about 2,500 code changes a day. By September it was over 30,000. Most of that growth came from agents, and somewhere along the way git stopped keeping up. So they replaced it. The post describing the replacement https://www.anthropic.com/engineering/redesigning-source-control , Scion, is one of the better pieces of systems writing I’ve read this year. It’s about source control for a code monorepo. There’s a whole section on data structures, but the data in question is Scion’s own: changesets, derived indexes, and file contents sitting in cloud storage. Versioning datasets isn’t in scope. Still, a lot of the decisions in it will look familiar to anyone who has built version control for data. The two places where we made the opposite call are kind of the interesting part. The short answer to the title: Scion’s storage design is close to how data versioning already has to work, because data never fit on a laptop in the first place. Where data differs is what happens after a bad change. That difference changes how you should think about branches and merges once agents are doing the writing. What Scion changed Git assumes every client is a full, offline copy of the repository. Anthropic’s clients are almost all agents and CI jobs that are always online, so they dropped that assumption. A few things follow from it: 1. Commits, merges, rebases, history and blame all run on the server. There is nothing to push. 2. Checkout is optional. In their words, “the working copy is optional, and sparse by design.” Plenty of jobs just ask the server for a diff and never put a file on disk. 3. All source control data is content-addressed and immutable, so “any server can cache it without checking with the others.” 4. Refs are discouraged beyond main and release branches, because “global state that every server has to agree on is a looming scaling problem.” The storage layer looks familiar Here is how Scion moves a branch: “a compare-and-swap on a single Spanner row.” Metadata and refs live in Spanner. File contents live in cloud storage. Here is how lakeFS moves a branch: A branch is a pointer to a commit, stored in a key-value store DynamoDB or PostgreSQL . A commit uses compare-and-set to read the branch state when it starts and to update the branch at the end only if nothing changed in between. The commit metadata lives in the object store next to the data, as content-addressed files that are never modified once written. lakeFS servers cache the frequently used ones on local disk and in memory. Different names, same shape. A small, strongly consistent store holds the little bit of state that changes. Everything else is immutable, sits in cheap storage, and can be cached by any node without asking anyone. I don’t think anyone copied anyone here. You end up with this design because of physics, if you will. Once clients are always online and there are a lot of them, a full clone stops paying for itself. For code, it took 30,000 merges a day for that to become true. For data, it was true from day one. Nobody is cloning a 400 TB training set to their laptop, right? Where we went the other way: snapshots Scion’s unit of work is a changeset, which “stores a list of the files that changed relative to its first parent.” A git commit, by contrast, “stores a pointer to the tree.” Scion then builds derived structures on the server to answer history and blame queries quickly. lakeFS commits sit closer to the git side. Each commit points to a metarange, the root of a two-level Merkle tree of ranges, and that tree describes the full state of the repository at that commit. Unchanged ranges are shared between commits, so a commit costs roughly the size of what changed. Reading the full state at any past commit is a lookup, not a reconstruction. For data, I think that’s the right trade. The question people ask about data is usually “what exact data did this model train on.” And this question tends to get asked six months after the fact. When reading a past state is the main access pattern, you want that read to be cheap. Where we went the other way: branches This is the bigger difference. Scion can discourage refs and skip a global merge lock because a bad code change is cheap to undo. The post is direct about it: “concurrent changes can conflict, but Claude can typically fix these conflicts faster than humans can even identify them.” Post-merge and pre-deploy tests catch the rest. Now picture an agent tidying up a data warehouse. It decides customers final v2 DO NOT USE is a duplicate of customers , merges the two, and drops 40% of the rows on the way. If your checks run after the merge, you find out after the merge. By then the nightly training job has read the table, a dashboard has refreshed, and a second agent has used it to build a customer segment. Reverting the table doesn’t un-train the model. Code can be rebuilt from source. Data gets consumed by whatever reads it next. So for data, the branch is the safety mechanism. Each agent run gets its own branch, which in lakeFS is a zero-copy pointer, so creating one costs almost nothing. The agent writes there. Checks run against that branch, as pre-merge hooks, before anything reaches main. If they fail, the branch gets thrown away and production never saw it. If something slips through anyway, main can be reverted to the previous commit. Scion keeps refs to a minimum because coordinating them is expensive at their scale. We accept more refs because, for data, an isolated branch is the cheapest place for an agent to be wrong. What to take from Scion if you run agents on data You probably don’t need to build something like Scion. For most teams, even large enterprise teams, Git is just fine for code. Anthropic says so themselves: “The concepts we’ve outlined here really only make sense for truly enormous monorepos.” I’d still pay attention to the direction, not because of your code, but because of your data. The number of agents writing to your data is going up and that will impact you and your team in a much bigger way. A few ideas carry over directly: 1. Stop assuming a full local copy. Read and write against the server or the object store, and materialize only what a job actually reads. 2. Keep the mutable state tiny and strongly consistent. Everything else should be immutable and content-addressed so it can be cached anywhere. 3. Decide where mistakes get caught. For code, after the merge is often fine. For data it has to be before, which means one isolated branch per writer and checks that run on that branch. 4. Record the commit ID for every agent run. Without that, you’re kind of flying blind when someone asks what the agent actually saw. The same property shows up in read performance. A lakeFS commit never changes, so a client can cache it without re-checking metadata. That’s one of the reasons lakeFS Mount ran about 57% faster than Mountpoint for Amazon S3 on warm reads in our MLPerf Storage tests https://lakefs.io/blog/s3-fuse-mount-performance-lakefs-mount/ . The full Scion post is worth reading, especially the sections on the fetch protocol and tiered caching. If you want the lakeFS side in more depth, the internals doc https://docs.lakefs.io/concepts/internals/ walks through the Merkle tree and the branch update path.