git-dedup: Faster and Smaller Checkouts for Free

Agentic coding means many worktrees and clones at once, and each one drags along its own copy of Git history and submodules. git-dedup shares one local object pool across all of them. It cut my Git storage by 70% and turned 10-minute worktree setups into 20-second ones.

Ben Houston • • 6 min read

AI coding changed how I use Git. I no longer work on one branch at a time. I have several agents running in parallel, each in its own worktree or clone, each working on a different task. That workflow is great for throughput, but Git was not designed for it, and two problems kept getting worse:

1. New worktrees with submodules were painfully slow. Git re-downloads every submodule from its remote for each new worktree.
2. My disk was filling up with .git data. Every clone, fork, and worktree submodule carried its own copy of the same history.

So I built git-dedup, a Git wrapper that makes all of those checkouts share one local object pool. On my machine it reduced Git object storage by 70% and made large worktrees initialize in under 20 seconds instead of upwards of 10 minutes.

The Submodule Problem# #

My MaterialX Fidelity Suite project, mtlx-fidelity, pulls in very large repositories as submodules: MaterialX, Blender, and three.js. These are big histories.

When you run git worktree add, the new worktree shares the parent repository's object database, which is good. But submodules are not included. When you then run git submodule update --init --recursive, Git clones each submodule from GitHub again, into a new private object database, for every worktree. The data is already sitting on my disk several times over, and Git downloads it again anyway.

For that project, it meant waiting 10 minutes or more for a worktree to be ready. When an agent session starts by creating a worktree, that wait kills the whole point of working in parallel.

With git-dedup, git-dedup worktree add creates the worktree and initializes its submodules from the local pool using Git alternates. Nothing is re-downloaded that is already on disk. The same large worktrees now take less than 20 seconds.

The Disk Space Problem# #

The other cost is storage. Each worktree's submodules, each fork, each review clone, and each agent's scratch clone holds another full copy of the same objects. Multiply that by the number of tasks you run in parallel and it adds up quickly.

Disk space is not cheap right now. Upgrading a MacBook on Apple's website costs about $800 CAD per extra terabyte of SSD. I do not want to spend that on duplicate copies of three.js history.

Here is what git-dedup measured across my 117 local checkouts. These contain 94 independent Git object databases; linked worktrees are counted once.

The largest sources of duplication were two mtlx-sample-library checkouts and the many three.js checkouts and submodules.

Inspired by pnpm# #

The model for this is pnpm. Before pnpm, every checkout of a Node.js project had its own node_modules with its own copy of every package. pnpm keeps one global content-addressable store and links each project to it. Having many repositories checked out became fast and cheap.

I wanted the same thing for .git data, with the same simplicity: install it, use it, and stop thinking about it.

Git already has the key primitive. Objects are addressed by their hash, and a repository can borrow objects from another object database through an alternates file. git-dedup manages one shared bare repository, ~/.git-dedup/pool.git, and points each checkout at it. Because objects are keyed by hash, an upstream clone, a fork, a worktree submodule, and an agent's throwaway clone all share the same objects, even when they have different remotes.

Each checkout keeps its real origin URL and a complete local history. Ordinary Git commands work unchanged.

Using git-dedup# #

git-dedup requires Node.js 22+ and Git, and runs on macOS and Linux.

npm install --global git-dedup

Clones, worktrees, and submodule updates go through the pool:

git-dedup clone --recurse-submodules https://github.com/you/project.git
cd project
git-dedup worktree add -b my-task ../project-my-task

Every other command is forwarded to Git, so git-dedup status, git-dedup fetch, and git-dedup log all work.

To adopt repositories you already have, including local commits and submodules:

git-dedup store add ./existing-project --stats

This moves shared objects into the pool and removes the checkout's private copies. This is how I reclaimed the 24 GB above.

Using It in VS Code and Cursor# #

VS Code and editors built on it, such as Cursor and Windsurf, run every Git operation through the executable named by the git.path setting. Point it at git-dedup and the editor's clone and worktree commands use the shared pool, including the worktrees that agent sessions create.

Find the path:

command -v git-dedup

Then add it to your user settings JSON (Command Palette → Preferences: Open User Settings (JSON)). git.path is a machine setting, so workspace settings ignore it:

{
  "git.path": "/absolute/path/to/git-dedup"
}

Run Developer: Reload Window, and the Git output channel will show a line such as Using git "2.50.1 (git-dedup 0.1.0)". If you use nvm or another Node version manager, update this path after switching Node versions.

Using It with Command-Line Agents# #

Command-line agents like Claude Code and Codex call git from your PATH. Tell them to use git-dedup instead by adding this to your AGENTS.md or CLAUDE.md:

## Git commands

Use `git-dedup` in place of `git` for Git commands. It forwards ordinary commands to Git and uses a shared local store for supported clone, submodule, and worktree operations. In particular, use `git-dedup clone <remote> [directory]`, `git-dedup submodule update --init --recursive`, and `git-dedup worktree add <path> [branch]`.

One Thing to Know# #

Linked checkouts depend on the pool. Do not delete or move ~/.git-dedup while checkouts use it. It is shared storage, not a disposable cache. The store dependency guide covers moving, pruning, and using it with containers.

Try It# #

Agentic coding rewards running many things at once. Git's per-checkout storage model punishes it. git-dedup removes that penalty: worktrees are fast, and parallel checkouts cost almost no extra disk.

• Documentation: git-dedup.ben3d.ca
• npm: git-dedup
• Source: github.com/bhouston/git-dedup