Stop Making AI Agents Grind Through Huge Codebases — I Built a Deterministic Wiki Build System for Them

A developer has released repowiki, an MIT-licensed, PyPI-published open-source tool that handles the deterministic orchestration of generating repository wikis so that coding agents such as Claude Code, Codex, and OpenCode can supply the intelligence. The CLI contains no model calls and only depends on pyyaml, offering commands to plan per-page tasks, atomically claim work for parallel agents, validate and auto-repair output, and package an offline single-file site. The author argues this split addresses the walls agents hit on large codebases and keeps wikis versioned, reviewable, and CI-gated inside the repo.

Joining an unfamiliar codebase, what you want isn't more code — it's a wiki that explains how the thing actually works: which modules exist, where the boundaries are, how a request travels from entry to egress. The instinct for the past two years has been to throw a coding agent at it: "read this repo and write me docs." On small repos that works great. On big ones you hit three walls immediately: And if the goal isn't "read it once" but a long-lived wiki for the whole team , add a fourth problem: code changes daily, docs never catch up, and three months later nobody trusts them. This post introduces the open-source tool I built for this problem: repowiki on PyPI, MIT licensed . It contains zero intelligence of its own — no model APIs, no network calls. It does exactly one thing: take over the deterministic parts of "generate a wiki for this repo," so that any agent Claude Code, Codex, OpenCode, or you yourself can work on top of it safely, in parallel. The design trade-off in one line: The agent supplies the intelligence; repowiki supplies the reliability. I tried all three routes before writing any code. Each has its own lock: Cloud AI wiki services DeepWiki and friends : the prettiest output, but your code has to leave your machine — an instant veto for companies with confidentiality requirements. You pay per use, and the output format and hosting are a black box. A wiki that lives on someone else's cloud isn't in your git: no version history, no diff, nothing to discuss in code review. IDE / tool built-ins Qoder Repo Wiki, ZCode Repository Wiki, etc. : nice, but locked to one tool's ecosystem. Generation is billed in credits, and the wiki lives in the tool's own directory or platform — it can't enter CI, has no version history, and doesn't survive switching tools. There are size limits too Qoder caps at 10k files per project . Two teammates on different IDEs means two divergent wikis. Just letting an agent read the repo : that's the three walls above. Worse, every developer pays the comprehension cost again in every session — conclusions live in a conversation, can't be reviewed, can't be updated incrementally. After all three, I kept coming back to the same missing piece: a deterministic orchestration layer . Who splits the tasks, who claimed which one, is the output acceptable, what happens after a crash? None of that needs intelligence. It needs determinism. The intelligence is already solved — pick any agent CLI. Nobody was handling the reliability part, so I built that. repowiki is "a build system that generates a structured wiki for any repository." The pipeline is a set of deterministic CLI commands: plan scan the repo, split it into per-page tasks, write the task catalog next --claim a worker atomically claims the next task no fights under concurrency check programmatic validation; anchors/line numbers/H1/paths auto-repaired, only semantic defects get rejected finalize assemble metadata overview page, wiki-overview, llms.txt index site package a single-file offline site ~5 MB self-contained HTML The intelligent work — reading code, writing pages — belongs entirely to the driving agent. The CLI contains no model calls and its only runtime dependency is pyyaml . Three direct consequences of this split: <repo /.repowiki/ . The CLI is a short-lived process; interrupt it anytime and continue later. Multiple agents, multiple processes, even multiple people can work on the same repo at once more on that in the next section . state.py L20-L5 start out of range or inverted , it's no longer silently clamped, it's an error and the page gets rewritten. Hallucinated line numbers must be rewritten — the tool doesn't lie for the model. Two design decisions worth calling out: file:// source citations with line ranges, committed straight into your repository: reviewable, versioned, incrementally updatable, and CI-gated like code. The hard part of multi-worker concurrency is task distribution: dozens of processes reaching for work at once — what prevents duplicate claims or grabbing someone's in-progress task? A database is overkill; repowiki's answer is filesystem primitives. Metaphor time: the task catalog is a bulletin board. To take a job, you pin your badge in the claims area — and "pinning the badge" is implemented as "create a directory with a fixed name." The OS guarantees the atomicity: fifty workers racing to create it, exactly one succeeds. What if a worker dies? The badge carries a timestamp; workers periodically touch it to renew. Past the deadline without renewal, anyone may move the whole thing aside rename the directory — also atomic and re-claim. From src/repowiki/state.py abridged; comments are from the real source : php def try mkdir claim self, task id: str, worker: str - bool: """Create claims/<id / atomically, stealing it first if stale.""" cd = self. claim dir task id for attempt in range 3 : try: os.mkdir cd except FileExistsError: if not self. claim stale cd : return False live claim held by someone else stale: rename it away only one racer succeeds and retry zombie = cd.with name f".stale-{task id}-{uuid.uuid4 .hex :6 }" try: os.rename cd, zombie except OSError: continue another racer already stole it; loop retries mkdir else: cd / "worker" .write text worker, encoding="utf-8" cd / "ts" .write text now iso , encoding="utf-8" return True return False Two subtle details from implementing this, for anyone building concurrent tooling: claim age . watch command can't "fake liveness": expired claims don't count as in-progress. Exit codes are part of the design too: 0 success, 1 validation failed, 2 state conflict task claimed by someone else , 3 progress-wait. Driver scripts and agents just look at the exit code — no output parsing. Dogfooding is the hardest acceptance test for this project: repowiki's own wiki is generated by repowiki itself with an agent doing the reading and writing , so the numbers below are real. The screencast above shows an English wiki that repowiki generated for a small sample project — same pipeline, English output: state.py:343-367 — every claim in the wiki links back to the exact lines that back it. When a new teammate doubts the docs, one click shows the truth; wiki.html llms.txt / llms-full.txt the llmstxt.org convention , so any agent or IDE can read the whole wiki by index — no MCP server needed, just hand it a static file. The online sample repowiki documenting itself, rebuilt on every push to main : wiki.repowiki online demo https://luomsis.github.io/repowiki/zh/wiki.html — the output language follows the target repo's language, so an English repo gets an English wiki. Even better, the CI is self-hosted too: this repo's GitHub Actions workflow runs stale --fail-if-stale on every PR — if code changed and the wiki didn't follow, the PR gets comments on the affected pages and the merge is blocked . Pushes to main rebuild the site to GitHub Pages automatically. "Docs rot" is closed off procedurally. The full suite of 220 tests runs on a macOS / Linux / Windows × Python 3.10-3.13 matrix, with native Windows support no WSL . Day-to-day maintenance doesn't rerun everything either: update rewrites only pages affected by the git diff, and coverage reports which files the wiki never references — the corners that still have no docs. pip install repowiki-cli or pipx; the only runtime dependency is pyyaml install the agent skill claude / codex / zcode / cursor / opencode repowiki skill install --agent claude Then tell your agent "generate a wiki for this repo" and it runs the whole pipeline itself. The complete reference for all 15 subcommands, the worker loop contract, and concurrency recipes are in docs/zh/USAGE.md https://github.com/luomsis/repowiki/blob/main/docs/zh/USAGE.md . Air-gapped environments are covered too: download the wheel + PyYAML from a GitHub Release and pip install --no-index on the target machine. repowiki is at 0.8.1, MIT licensed, and carrying real work in my daily setup. The repo lives at luomsis/repowiki https://github.com/luomsis/repowiki . If "huge repos are unreadable, docs always rot" has been hurting you too, give it a try — and I'd love to hear from you: If it helps you, give the repo a star — for an indie open-source project that's the most direct signal, and the biggest motivation to keep going. Found a bug or have opinions on task splitting, page templates, or the CI gate? Open an issue https://github.com/luomsis/repowiki/issues — I read all of them. And feel free to share this with the colleague who just inherited a giant codebase. The boundaries are explicit non-goals : no LLM API backend, no MCP wrapper agents read the wiki via the llms.txt export , no resident preview server the output is a single static file . repowiki only does the deterministic orchestration layer — the intelligence stays with your agent.