{"slug": "how-i-build-software-with-coding-agents", "title": "How I build software with coding agents", "summary": "A developer detailed a year-long setup for running coding agents like Claude Code and Codex CLI on real products, built around a single 32-core, 128 GB Linux box reached over Tailscale and managed with a terminal workspace tool called herdr. The workflow splits models by role — a Fable session lead that plans and merges but never writes feature code, Opus implementers in isolated git worktrees, and Opus adversarial verifiers — after an all-Fable delegation hit the 5-hour usage cap twice in one evening and killed five running implementers. Agents operate under a written per-repo charter, never commit on their own unless permitted, and sweep orphaned microsandbox VMs after every eval run.", "body_md": "A write-up for a friend. This is the setup I have converged on after about a year of running Claude Code and Codex CLI on real products, some of them with the agents working for days at a time while I sleep. It is generic on purpose. Where I quote numbers they come from my own repos, measured on 2026-09-23.\n\nThe short version: one Linux box, a terminal multiplexer built for agents, a written contract per repo that tells the agent how work is done, and a loop of grill → issues → orchestrated build → adversarial verification → merge → docs. The agent never commits on its own unless a charter says so, and it never marks something done that it has only read.\n\n**One Linux box at home.** 32 cores, 128 GB RAM, ~900 GB of disk. Everything agent-related\nruns here: the agents, the dev servers, the sandboxes, the GPU-fleet control scripts.\nLaptops are thin clients.\n\n**Tailscale.** The box has no public address. I reach it over the tailnet from any device.\nMagicDNS gives it a name, so `ssh box` works from the laptop and from other machines that\nneed to talk to services on it (a local Elasticsearch, a local Nominatim, dev servers on\nodd ports).\n\n**herdr.** A terminal workspace manager for AI coding agents. It gives me a hierarchy of\nworkspaces (one per project), tabs (one per concern inside the project) and panes (a shell,\nan agent or a server). It detects the agent's status in each pane, so I can glance at a\nboard and see which agents are waiting for me and which are still working. Both Claude\nCode and Codex CLI report their state to it through a session-start hook. There is also a\n`herdr` skill so an agent inside a pane can open a tab, spawn another agent in it, run a\ncommand there and wait for a match on its output or for that agent to reach \"done\". That\nis how a coordinator in one pane drives an implementer in another.\n\n**Sandboxes.** Agent products I build use microsandbox VMs for tool execution. Every eval\nrun leaves VMs and 4 GB sparse disks behind, so cleanup is a written rule, not a habit:\nafter any eval or dev-server session the agent sweeps orphaned VMs, deletes the directories\nno live VM holds, and says in its recap what it cleared. The disk filled up twice before\nthat rule existed.\n\n**Rented GPUs when needed.** Batch LLM extraction (OCR, structured extraction over millions\nof documents) runs on rented L40S boxes with vLLM, driven from the home box by scripts the\nagents wrote. The agent launches the fleet, checkpoints progress in the target store, and\naudits the result against a gold set afterwards.\n\n**Claude Code** is the main driver. Global config: Fable 5.1 with a 1M context as the\ndefault model, effort `xhigh`, auto permission mode with a written trust-boundary brief\n(what is prod, what is a live cluster, which commands are \"soft deny\" and need a human\neven in auto mode).\n\n**Codex CLI** runs beside it, on GPT-6 Astra. Same skills, same hooks, same herdr\nintegration. I use it for second opinions on plans and for parallel implementation when I\nwant two independent takes.\n\n**The model split.** This took a few painful iterations to settle:\n\n- The session lead (Fable) plans, dispatches, reads reports, reviews diffs, and merges. It does not write feature code. I have literally stopped it mid-turn with \"do not code here, you're the orchestrator\".\n- Implementers run on Opus, one per issue, each in its own git worktree.\n- Adversarial verifiers also run on Opus, one per implementer diff.\n- One Fable agent runs the final smoke test of the integration branch after a wave merges.\n\nThe reason is the usage window. All-Fable delegation hit the 5-hour cap twice in one evening and killed five running implementers. Opus implementers plus a single Fable smoke at the end gets the same quality for a fraction of the budget. When I name a model in the request, that wins.\n\n**Subagent briefs are self-contained.** A delegated prompt assumes the subagent has no chat\ncontext. It carries the repo path, the objective, the files in and out of scope, the exact\nevidence format to return (files, line refs, commands run, diffs, failures, screenshots,\nuncertainties), the verification commands, and stop conditions: if the code does not match\nthe brief, a command fails twice, or the task needs out-of-scope files, stop and report\ninstead of improvising. Reports are leads, not facts. The lead reopens the cited files\nbefore acting on a finding.\n\nEvery repo has a `CLAUDE.md` (or `AGENTS.md` for Codex-first repos). The important design\ndecision: **it is an index, not a manual.** It says what each surface is, which files own it,\nthe operative rules, and where the long version lives. A unit test pins its size (64 KB in\none repo, 72 KB in another) and a change may add at most ~10 lines. Narratives, measurements\nand incident stories go to `docs/architecture.md` and a `docs/journal/` directory. The old\n600 KB version is frozen in an archive file that the agent may grep but never edit.\n\nPer-surface rules load on demand from `docs/agents/*.md` (routes, data, conventions). The\nindex says \"read the matching annex before touching its surface\". This keeps the always-on\ncontext small and the detailed rules complete.\n\nThe rules that recur across every repo:\n\n- **Issue before code.** The issue is the unit of work. Body: the observed problem without\nthe proposed fix, acceptance criteria someone else can check, out of scope. Too small\nfor an issue means too small for its own PR.\n- **Branch from `dev`** , named` feat|fix|chore|docs/-slug` . Never commit to`main` or`dev` . Parallel agents take their own worktree with`.env.local` copied in.\n- **The quality gate.** Typecheck, lint at zero errors and zero warnings, format check,\nunit tests, the contract check, the build, and the e2e specs that cover the changed\nscreen. A red gate is never \"good enough to review\". A proven lint false positive is\nsuppressed on the line with a written reason, never allowed to flow.\n- **Smoke test, never ship a change you have only read.** Start the app, walk the exact\nsurface through its acceptance criteria and one adjacent flow. Browser: clean console\nincluding React warnings, both themes, both breakpoints, the keyboard path. Record the\nexact commands and real output. A test not run is reported as skipped, never passed.\n- **Adversarial verification before the PR.** A separate agent with fresh context, briefed\nto break the change: \"assume it is broken until proven otherwise; for each acceptance\ncriterion find a concrete input where it fails; report unverified claims, silently\nnarrowed scope, untested edge cases, regressions in adjacent code\". It gets the diff, the\nacceptance criteria (not the implementer's summary) and the smoke log, and is asked what\nthe smoke test failed to cover. Every finding is resolved or rejected in writing.\n- **The PR.** Base`dev` , body with the issue reference, what changed and why, how it was\nverified with real output, the adversarial findings and how each was settled, before and\nafter screenshots for anything visual, known gaps. Title is a Conventional Commit because\nrelease-please derives the changelog and version from commit types.\n- **No closing keywords.** Never \"closes #N\" in a title, body or commit. A CI guard matches\nthe tokens. Issues are closed by a human after acceptance, with a comment mapping each\ncriterion to the PR that met it. Release PRs once silently closed 13 issues; that is\nwhere this rule came from.\n- **Docs in the same change.** README for users, architecture for the deep dive, CLAUDE.md\nat ten lines. Any new CLI verb, flag, default or env var updates the operational runbook\nin the same PR. A feature is not done until the runbook reflects it.\n- **Every long-running CLI shows rich progress.** A bar on stderr with done/total, ETA and\nlive outcome counters, a one-line startup summary so it never sits silent, plain periodic\nlines when stderr is not a TTY, and Ctrl-C prints how to resume. One repo enforces this\nwith an AST test that refuses hand-built progress objects.\n- **Every long job resumes.** Per-unit checkpoints in the target store, cursors instead of\nscrolls, a resume that shows as a resume (the bar pre-seeded from the checkpoint), exit\ncodes 0/1/2/130.\n- **Humans commit.** The agent prepares the branch, the commits and the PR body, then asks\nbefore`git commit` ,`git push` or`gh pr create` . Merges are`--merge` , never squash,\nso each concern keeps its own Conventional commit. The exception is a written charter for\nan unattended run (section 6).\n- **Language.** Everything in code is English: identifiers, comments, test titles, commit\nmessages. The UI language is only for what the user reads. Domain vocabulary that names\na route or a registry concept is the one allowed exception in identifiers.\n\nSkills are the reusable instructions the agent loads on demand. Mine come from four\nsources: Matt Pocock's set, the agent-native set, a few from vendors (Elastic, Vercel,\nPostHog, Convex) and a few I wrote. A shared `~/.agents/skills` store feeds both Claude and\nCodex.\n\n**Planning and scoping**\n\n- `grill-me` . The one I use most. It interviews me about a plan, one branch of the decision\ntree at a time, until we agree. Every epic starts with a grilling session; the decisions\nit produces go into the epic body as a \"Decisions\" list.\n- `grill-with-docs` . Same, but checked against`CONTEXT.md` and the ADRs, which it updates\nas decisions settle. Used on the data pipeline repos where the domain glossary matters.\n- `to-prd` and`to-issues` . Turn the grilled plan into a PRD, then into independently\ngrabbable tracer-bullet issues on GitHub, each with acceptance criteria.\n- `plan-arbiter` . When Codex and Claude both produced a plan, compare them, pick one or\nmerge a hybrid, and output one execution handoff with verification gates.\n- `triage` . Moves issues through a state machine and prepares them for an unattended agent.\n- `handoff` . Writes the conversation into a handoff document for a fresh agent.\n\n**Execution discipline**\n\n- `efficient-fable` and`efficient-frontier` . The orchestration pattern from section 2:\nthe expensive model plans, delegates and judges; cheaper subagents do scans, logs, test\nruns and bounded edits. Always on.\n- `stay-within-limits` . Work in waves of at most three parallel subagents. Between waves\ncheck usage with`ccusage` , pause at 95% of the 5-hour or weekly window, re-check on\nresume.\n- `plow-ahead` . \"Keep going, do not ask.\" Routine ambiguity becomes a stated assumption;\nthe recap lists the decisions made. This is what I put in a brief when I go to sleep.\n- `agent-watchdog` . Audit another agent's work from a session id, transcript, PR or log.\n- `tdd` ,`diagnose` , superpowers'`systematic-debugging` and`verification-before-completion` .\nThe process skills that stop the agent from patching symptoms.\n- `read-the-damn-docs` . Before touching a third-party API, SDK or anything involving auth,\nbilling or security, search and read the official docs. Never from memory.\n- `quick-recap` . Every finished response ends with one status line: green done, yellow a\nnon-routine follow-up pending, red blocked on me. I read those lines from the herdr board.\n\n**Writing**\n\n- `unslop` . Strips AI tells from every piece of prose (no em dashes, no sycophancy, no\nfiller, no bold-label bullets, no sentence that would fit another project's docs) and\nputs a voice back. A prompt-submit hook re-injects the reminder on every turn so it\nsurvives context compaction. Subagent briefs say \"follow the unslop skill\".\n- `ste-writing` . Simplified Technical English for runbooks.\n\n**Visual and analysis**\n\n- `visual-plan` ,`visual-recap` ,`visualize-repo` . Turn a plan, a PR or a repo into a\nlocal interactive document with diagrams and annotated diffs.\n- `graphify` . Any folder or repo into a knowledge graph with community clustering.\n- `impeccable` . UI design review and iteration in the live browser.\n- The Elastic set: Kibana dashboards, alerting rules, ES|QL log search.\n\nAn MCP server is how the agent reads and pokes at the infrastructure without me pasting things. The principle: the agent should be able to see what I can see, in the dashboards I would open myself.\n\n- **playwright** and**chrome-devtools** . Browser automation for smoke tests, screenshots,\nconsole and network inspection, Lighthouse. The Playwright one is what the verifier uses\nto walk every signed-in route and capture a screenshot per route.\n- **next-devtools** . The Next.js dev server's own view: routes, build errors, docs.\n- **shadcn** . The component registry. UI is built from these primitives only, never\nhand-written, and the agent looks the component up here before adding it.\n- **context7** . Current library docs. Frameworks move faster than model training data.\n- **clerk** . Auth SDK snippets.\n- **convex** . Read tables, run queries, tail logs, check env vars on the backend. Prod is\nread-only for the agent; deploys and mutations stay with me.\n- **vercel** . Deployments, runtime logs, env vars, firewall, the sandbox API.\n- **posthog** . Product analytics, session replay, feature flags, error tracking, LLM\nobservability and the alerting. Every metric about my agent product lives here,\nincluding a cost-per-turn alert that mails me.\n- **axiom** . Log queries for the data pipelines. (Pipelines log to Axiom; the app product\nlogs to PostHog. One tool per concern, decided once.)\n- **cloudflare** . Workers, R2, observability, for the repos that run edge workers.\n- **twenty** . My CRM, over MCP, so an agent can look up or update a customer while working\na support issue.\n- **resend** . Transactional mail.\n- **workos** and**linear** on the repos that use them.\n\nPlugins from the official marketplace add the language servers (TypeScript, Pyright), the\n`code-review`, `feature-dev` and `code-simplifier` agents, the `security-guidance` hooks,\n`hookify` for turning a repeated mistake into a blocking hook, and the `remember` journal\n(section 7).\n\nTwo things I turned off on purpose. The Semgrep plugin needed a headless OAuth that could\nnot complete and its hook blocked every edit until I disabled it. And I keep the\n`context7` plugin off because the user-scope MCP entry already covers it.\n\n1. **Grill.** I open a session on Fable (or Astra) and say \"grill me on X\". It interviews me\nuntil the design has no open branch. The output is a decisions list with the rejected\nalternatives named. On domain-heavy repos this also updates the ADRs.\n2. **Spec to tracker.**`to-issues` files an epic and its sub-issues. Each sub-issue has\nthe problem, acceptance criteria and out-of-scope. Labels tie the sub-issues to the epic.\n3. **Orchestrate.** The lead session spawns one Opus implementer per sub-issue in its own\nworktree with a self-contained brief. At most three in flight. Implementers never commit;\nthey leave the tree ready and report with exact commands and output.\n4. **Verify.** For each diff, an Opus verifier with fresh context, briefed to refute. It\nwalks the acceptance criteria in a real browser, signed in as a smoke account I created\n(the agent is not allowed to script past the sign-up bot check). Findings get folded\ninto the same commit or a`fix(scope): … (#n)` commit on the branch.\n5. **Gate and smoke.** All gates green, targeted e2e specs named with their counts, a\nsmoke log with real output.\n6. **PR.** One PR per issue in most repos, one PR per epic with a commit per sub-issue in\nthe ones where I asked for it. I read the diff top to bottom and approve the push.\n7. **Final smoke.** After the wave merges, one Fable agent smokes`dev` end to end and\nreports green, green with findings, or red. Findings become follow-up issues.\n8. **Close and document.** Each issue closed with the criteria-to-PR map. The narrative,\nmeasurements and traps go to the journal, the runbook and, if it changed, the\narchitecture doc. Post-mortems of production sessions become gists.\n9. **Clean up.** Stop dev servers, remove worktrees, sweep sandboxes, say so in the recap.\n\nNumbers from the busiest repo, since 1 July: 2,217 commits, 65% carrying a Claude co-author trailer, commits in every hour of the day, a peak of 153 commits in one day, and the issue and PR counters both past 1,160.\n\nThis is the mode people ask about. It is not \"let the agent run and hope\". It is a written charter plus mechanical backing so the loop survives context loss.\n\n**The charter.** A 27 KB markdown file the top of `CLAUDE.md` points to as the active\nunattended charter. It amends a kickoff prompt whose first section is literally\n\"OVERNIGHT ORCHESTRATOR\", with an editable scope line, a 07:00 local wrap-up deadline and\n\"MORNING DELIVERABLES\": a night report, the ledger copied into the journal, a draft PR.\nWhat it pins:\n\n- The orchestrator implements nothing and reads no large files. It dispatches one worker per issue in its own worktree, three in parallel by default, five at most.\n- A SMOKE LOCK: one heavy real-data run at a time, ledgered.\n- One adversarial verifier per diff, briefed to refute, not approve. Two failed fix cycles\nmeans park the issue with a `backlog` label.\n- Tiered verification: a full adversarial pass for pipeline, cluster or CLI changes; one pass for docs and tests. Either tier parks after about two hours.\n- Gated self-merge: merge only when the verifier is green and CI is green on the exact head\nSHA. It never merges `main` or release PRs. It may close issues only with a three-part\nclosing comment.\n- Stop conditions: the cluster down for more than 30 minutes, disk under 30 GB, a dirty tree at boot, or a step that needs a human. Human-only steps (live writes, scheduling, prod deploys) are queued as paste-ready commands, never executed.\n- Operating under the charter is the commit, push and merge authorisation that the global rule otherwise withholds.\n\n**Files are memory.** The charter's anti-amnesia contract says the conversation, including\nany summary of earlier context, is not to be trusted. The working set is three files:\n`ledger.md`, `queue.md`, `state.md`. Ledger before the next task. A re-anchor ritual at\nsession start, after every compaction, and every ~90 minutes: re-read the charter and the\nthree files, append `re-anchored