{"slug": "git-dedup-faster-and-smaller-checkouts-for-free", "title": "git-dedup: Faster and Smaller Checkouts for Free", "summary": "Developer Ben Houston released git-dedup, a Git wrapper that shares one local object pool across worktrees and clones via Git alternates, cutting his Git object storage by 70% — from 35.5 GB to 11.1 GB across 117 local checkouts — and reducing large worktree setup from over 10 minutes to under 20 seconds. The tool addresses submodule re-downloads and duplicate .git history created by running parallel agentic coding worktrees, modeled on pnpm's shared content-addressable store.", "body_md": "# git-dedup: Faster and Smaller Checkouts for Free\n\nAgentic coding means many worktrees and clones at once, and each one drags along its own copy of Git history and submodules. git-dedup shares one local object pool across all of them. It cut my Git storage by 70% and turned 10-minute worktree setups into 20-second ones.\n\n[Ben Houston](https://ben3d.ca/about) •  • 6 min read\n\nAI coding changed how I use Git. I no longer work on one branch at a time. I have several agents running in parallel, each in its own worktree or clone, each working on a different task. That workflow is great for throughput, but Git was not designed for it, and two problems kept getting worse:\n\n1. **New worktrees with submodules were painfully slow.** Git re-downloads every submodule from its remote for each new worktree.\n2. **My disk was filling up with `.git` data.** Every clone, fork, and worktree submodule carried its own copy of the same history.\n\nSo I built [git-dedup](https://git-dedup.ben3d.ca), a Git wrapper that makes all of those checkouts share one local object pool. On my machine it reduced Git object storage by 70% and made large worktrees initialize in under 20 seconds instead of upwards of 10 minutes.\n\n## The Submodule Problem[#](#the-submodule-problem)\n\nMy [MaterialX Fidelity Suite](https://ben3d.ca/blog/pixel-perfect-materialx-in-blender-and-threejs) project, `mtlx-fidelity`, pulls in very large repositories as submodules: MaterialX, Blender, and three.js. These are big histories.\n\nWhen you run `git worktree add`, the new worktree shares the parent repository's object database, which is good. But submodules are not included. When you then run `git submodule update --init --recursive`, Git clones each submodule from GitHub again, into a new private object database, for every worktree. The data is already sitting on my disk several times over, and Git downloads it again anyway.\n\nFor that project, it meant waiting 10 minutes or more for a worktree to be ready. When an agent session starts by creating a worktree, that wait kills the whole point of working in parallel.\n\nWith git-dedup, `git-dedup worktree add` creates the worktree and initializes its submodules from the local pool using Git alternates. Nothing is re-downloaded that is already on disk. The same large worktrees now take less than 20 seconds.\n\n## The Disk Space Problem[#](#the-disk-space-problem)\n\nThe other cost is storage. Each worktree's submodules, each fork, each review clone, and each agent's scratch clone holds another full copy of the same objects. Multiply that by the number of tasks you run in parallel and it adds up quickly.\n\nDisk space is not cheap right now. Upgrading a MacBook on Apple's website costs about $800 CAD per extra terabyte of SSD. I do not want to spend that on duplicate copies of three.js history.\n\nHere is what git-dedup measured across my 117 local checkouts. These contain 94 independent Git object databases; linked worktrees are counted once.\n\n| Git object storage | Space | \n|---|---|\n| If each repo held its own git objects | 35.5 GB | \n| Shared git-dedup pool | 10.0 GB | \n| Objects still stored privately in checkouts | 1.0 GB | \n| **Current total** | **11.1 GB** | \n| **Reduction** | **24.4 GB (70%)** | \n\nThe largest sources of duplication were two `mtlx-sample-library` checkouts and the many three.js checkouts and submodules.\n\n## Inspired by pnpm[#](#inspired-by-pnpm)\n\nThe model for this is [pnpm](https://pnpm.io). Before pnpm, every checkout of a Node.js project had its own `node_modules` with its own copy of every package. pnpm keeps one global content-addressable store and links each project to it. Having many repositories checked out became fast and cheap.\n\nI wanted the same thing for `.git` data, with the same simplicity: install it, use it, and stop thinking about it.\n\nGit already has the key primitive. Objects are addressed by their hash, and a repository can borrow objects from another object database through an `alternates` file. git-dedup manages one shared bare repository, `~/.git-dedup/pool.git`, and points each checkout at it. Because objects are keyed by hash, an upstream clone, a fork, a worktree submodule, and an agent's throwaway clone all share the same objects, even when they have different remotes.\n\nEach checkout keeps its real `origin` URL and a complete local history. Ordinary Git commands work unchanged.\n\n## Using git-dedup[#](#using-git-dedup)\n\ngit-dedup requires Node.js 22+ and Git, and runs on macOS and Linux.\n\n```\nnpm install --global git-dedup\n```\n\nClones, worktrees, and submodule updates go through the pool:\n\n```\ngit-dedup clone --recurse-submodules https://github.com/you/project.git\ncd project\ngit-dedup worktree add -b my-task ../project-my-task\n```\n\nEvery other command is forwarded to Git, so `git-dedup status`, `git-dedup fetch`, and `git-dedup log` all work.\n\nTo adopt repositories you already have, including local commits and submodules:\n\n```\ngit-dedup store add ./existing-project --stats\n```\n\nThis moves shared objects into the pool and removes the checkout's private copies. This is how I reclaimed the 24 GB above.\n\n## Using It in VS Code and Cursor[#](#using-it-in-vs-code-and-cursor)\n\nVS Code and editors built on it, such as Cursor and Windsurf, run every Git operation through the executable named by the `git.path` setting. Point it at git-dedup and the editor's clone and worktree commands use the shared pool, including the worktrees that agent sessions create.\n\nFind the path:\n\n```\ncommand -v git-dedup\n```\n\nThen add it to your **user** settings JSON (Command Palette → **Preferences: Open User Settings (JSON)**). `git.path` is a machine setting, so workspace settings ignore it:\n\n```\n{\n  \"git.path\": \"/absolute/path/to/git-dedup\"\n}\n```\n\nRun **Developer: Reload Window**, and the **Git** output channel will show a line such as `Using git \"2.50.1 (git-dedup 0.1.0)\"`. If you use nvm or another Node version manager, update this path after switching Node versions.\n\n## Using It with Command-Line Agents[#](#using-it-with-command-line-agents)\n\nCommand-line agents like Claude Code and Codex call `git` from your `PATH`. Tell them to use git-dedup instead by adding this to your `AGENTS.md` or `CLAUDE.md`:\n\n```\n## Git commands\n\nUse `git-dedup` in place of `git` for Git commands. It forwards ordinary commands to Git and uses a shared local store for supported clone, submodule, and worktree operations. In particular, use `git-dedup clone <remote> [directory]`, `git-dedup submodule update --init --recursive`, and `git-dedup worktree add <path> [branch]`.\n```\n\n## One Thing to Know[#](#one-thing-to-know)\n\nLinked checkouts depend on the pool. Do not delete or move `~/.git-dedup` while checkouts use it. It is shared storage, not a disposable cache. The [store dependency guide](https://git-dedup.ben3d.ca/docs/safety) covers moving, pruning, and using it with containers.\n\n## Try It[#](#try-it)\n\nAgentic coding rewards running many things at once. Git's per-checkout storage model punishes it. git-dedup removes that penalty: worktrees are fast, and parallel checkouts cost almost no extra disk.\n\n- Documentation: [git-dedup.ben3d.ca](https://git-dedup.ben3d.ca)\n- npm: [git-dedup](https://www.npmjs.com/package/git-dedup)\n- Source: [github.com/bhouston/git-dedup](https://github.com/bhouston/git-dedup)", "url": "https://wpnews.pro/news/git-dedup-faster-and-smaller-checkouts-for-free", "canonical_source": "https://ben3d.ca/blog/git-dedup-faster-smaller-checkouts", "published_at": "2026-09-30 00:00:00+00:00", "updated_at": "2026-09-30 18:21:14.461709+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["git-dedup", "Ben Houston", "Git", "pnpm", "MaterialX", "Blender", "three.js", "mtlx-fidelity"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/git-dedup-faster-and-smaller-checkouts-for-free", "markdown": "https://wpnews.pro/news/git-dedup-faster-and-smaller-checkouts-for-free.md", "text": "https://wpnews.pro/news/git-dedup-faster-and-smaller-checkouts-for-free.txt", "jsonld": "https://wpnews.pro/news/git-dedup-faster-and-smaller-checkouts-for-free.jsonld"}}