I kept wondering how much of what a coding agent reads in a repo is actually useful to it. So I wrote a small CLI that walks a repo, estimates tokens per file the way a model sees them, and flags files that cost tokens without teaching the model anything.
Then I ran it on six well-known open-source repos (fresh shallow clones, 12–13 Nov 2026).
| Repo | Total tokens | Flagged as waste | Biggest offender |
|---|---|---|---|
| langchain | 9.82M | 55% | 22 uv.lock files, 5.11M tokens together |
| excalidraw | 4.50M | 54% | a 296k-token harfbuzz-wasm.ts |
| ollama | 9.26M | 51% | integration/testdata/shakespeare.txt , 1.33M tokens |
| fastapi | 8.47M | 43% | one drawio SVG, 631k tokens |
| shadcn/ui | 7.79M | 38% | 22 lockfiles, 813k tokens together |
| express | 196k | ~0% | nothing worth flagging |
These are whole-tree totals, so treat them as an upper bound. No agent reads a 9M-token repo in one pass. But agents grep, glob and open files all day, and every time one of those lands in a lockfile or a test fixture, that turn's context goes to integrity hashes instead of code.
1. It's rarely the code. In langchain, the lockfiles alone (5.11M tokens) outweigh everything an agent could actually learn from (4.45M). In ollama, a copy of Shakespeare used as test data is bigger than most entire repos.
2. Small, hand-written repos are fine. express came back clean. The problem shows up in big monorepos with many packages, generated assets and test data.
3. The fix is boring. Almost all of the waste comes from five kinds of file: lockfiles, snapshots, minified or generated bundles, bulk data, and large SVGs. A dozen ignore rules removes most of it.
Input tokens are still the bill, and a bigger window just makes it easier to fill with junk without noticing. It also costs you quality: Chroma's "context rot" research found that models get worse as the input fills up, however large the window is. So you pay more for worse answers.
npx github:geraltbhai/ctxray
Nothing to install, no dependencies, no network calls, Node 18+, MIT. It prints:
Then:
npx github:geraltbhai/ctxray init --dry-run # preview; drop --dry-run to write the files
That writes an .agentignore built from what the scan actually found. It uses .gitignore syntax, so it works as a .cursorignore or .aiexclude too. It also writes a starter AGENTS.md pre-filled with the repo's real shape: language mix, directory weights and entry points.
How the count works: a full tokenizer vocabulary would mean a multi-megabyte dependency, so ctxray uses a rule-based BPE approximation instead. I fitted its constants against gpt-tokenizer over 435 files, and it lands at about 6% median error per file. The calibration script is in the repo, so you can check that claim yourself.
The scanner and init are free and staying free. A paid tier for HTML reports, PR comments that show what a change did to your context budget, CI thresholds and a context packer is on its way.
Run it and tell me your grade in the comments. If it flags something it shouldn't, post the path. I'm treating false positives as bugs.