How I build software with coding agents

A developer detailed a year-long setup for running coding agents like Claude Code and Codex CLI on real products, built around a single 32-core, 128 GB Linux box reached over Tailscale and managed with a terminal workspace tool called herdr. The workflow splits models by role — a Fable session lead that plans and merges but never writes feature code, Opus implementers in isolated git worktrees, and Opus adversarial verifiers — after an all-Fable delegation hit the 5-hour usage cap twice in one evening and killed five running implementers. Agents operate under a written per-repo charter, never commit on their own unless permitted, and sweep orphaned microsandbox VMs after every eval run.

A write-up for a friend. This is the setup I have converged on after about a year of running Claude Code and Codex CLI on real products, some of them with the agents working for days at a time while I sleep. It is generic on purpose. Where I quote numbers they come from my own repos, measured on 2026-09-23. The short version: one Linux box, a terminal multiplexer built for agents, a written contract per repo that tells the agent how work is done, and a loop of grill → issues → orchestrated build → adversarial verification → merge → docs. The agent never commits on its own unless a charter says so, and it never marks something done that it has only read. One Linux box at home. 32 cores, 128 GB RAM, ~900 GB of disk. Everything agent-related runs here: the agents, the dev servers, the sandboxes, the GPU-fleet control scripts. Laptops are thin clients. Tailscale. The box has no public address. I reach it over the tailnet from any device. MagicDNS gives it a name, so ssh box works from the laptop and from other machines that need to talk to services on it a local Elasticsearch, a local Nominatim, dev servers on odd ports . herdr. A terminal workspace manager for AI coding agents. It gives me a hierarchy of workspaces one per project , tabs one per concern inside the project and panes a shell, an agent or a server . It detects the agent's status in each pane, so I can glance at a board and see which agents are waiting for me and which are still working. Both Claude Code and Codex CLI report their state to it through a session-start hook. There is also a herdr skill so an agent inside a pane can open a tab, spawn another agent in it, run a command there and wait for a match on its output or for that agent to reach "done". That is how a coordinator in one pane drives an implementer in another. Sandboxes. Agent products I build use microsandbox VMs for tool execution. Every eval run leaves VMs and 4 GB sparse disks behind, so cleanup is a written rule, not a habit: after any eval or dev-server session the agent sweeps orphaned VMs, deletes the directories no live VM holds, and says in its recap what it cleared. The disk filled up twice before that rule existed. Rented GPUs when needed. Batch LLM extraction OCR, structured extraction over millions of documents runs on rented L40S boxes with vLLM, driven from the home box by scripts the agents wrote. The agent launches the fleet, checkpoints progress in the target store, and audits the result against a gold set afterwards. Claude Code is the main driver. Global config: Fable 5.1 with a 1M context as the default model, effort xhigh , auto permission mode with a written trust-boundary brief what is prod, what is a live cluster, which commands are "soft deny" and need a human even in auto mode . Codex CLI runs beside it, on GPT-6 Astra. Same skills, same hooks, same herdr integration. I use it for second opinions on plans and for parallel implementation when I want two independent takes. The model split. This took a few painful iterations to settle: - The session lead Fable plans, dispatches, reads reports, reviews diffs, and merges. It does not write feature code. I have literally stopped it mid-turn with "do not code here, you're the orchestrator". - Implementers run on Opus, one per issue, each in its own git worktree. - Adversarial verifiers also run on Opus, one per implementer diff. - One Fable agent runs the final smoke test of the integration branch after a wave merges. The reason is the usage window. All-Fable delegation hit the 5-hour cap twice in one evening and killed five running implementers. Opus implementers plus a single Fable smoke at the end gets the same quality for a fraction of the budget. When I name a model in the request, that wins. Subagent briefs are self-contained. A delegated prompt assumes the subagent has no chat context. It carries the repo path, the objective, the files in and out of scope, the exact evidence format to return files, line refs, commands run, diffs, failures, screenshots, uncertainties , the verification commands, and stop conditions: if the code does not match the brief, a command fails twice, or the task needs out-of-scope files, stop and report instead of improvising. Reports are leads, not facts. The lead reopens the cited files before acting on a finding. Every repo has a CLAUDE.md or AGENTS.md for Codex-first repos . The important design decision: it is an index, not a manual. It says what each surface is, which files own it, the operative rules, and where the long version lives. A unit test pins its size 64 KB in one repo, 72 KB in another and a change may add at most ~10 lines. Narratives, measurements and incident stories go to docs/architecture.md and a docs/journal/ directory. The old 600 KB version is frozen in an archive file that the agent may grep but never edit. Per-surface rules load on demand from docs/agents/ .md routes, data, conventions . The index says "read the matching annex before touching its surface". This keeps the always-on context small and the detailed rules complete. The rules that recur across every repo: - Issue before code. The issue is the unit of work. Body: the observed problem without the proposed fix, acceptance criteria someone else can check, out of scope. Too small for an issue means too small for its own PR. - Branch from dev , named feat|fix|chore|docs/<issue -slug . Never commit to main or dev . Parallel agents take their own worktree with .env.local copied in. - The quality gate. Typecheck, lint at zero errors and zero warnings, format check, unit tests, the contract check, the build, and the e2e specs that cover the changed screen. A red gate is never "good enough to review". A proven lint false positive is suppressed on the line with a written reason, never allowed to flow. - Smoke test, never ship a change you have only read. Start the app, walk the exact surface through its acceptance criteria and one adjacent flow. Browser: clean console including React warnings, both themes, both breakpoints, the keyboard path. Record the exact commands and real output. A test not run is reported as skipped, never passed. - Adversarial verification before the PR. A separate agent with fresh context, briefed to break the change: "assume it is broken until proven otherwise; for each acceptance criterion find a concrete input where it fails; report unverified claims, silently narrowed scope, untested edge cases, regressions in adjacent code". It gets the diff, the acceptance criteria not the implementer's summary and the smoke log, and is asked what the smoke test failed to cover. Every finding is resolved or rejected in writing. - The PR. Base dev , body with the issue reference, what changed and why, how it was verified with real output, the adversarial findings and how each was settled, before and after screenshots for anything visual, known gaps. Title is a Conventional Commit because release-please derives the changelog and version from commit types. - No closing keywords. Never "closes N" in a title, body or commit. A CI guard matches the tokens. Issues are closed by a human after acceptance, with a comment mapping each criterion to the PR that met it. Release PRs once silently closed 13 issues; that is where this rule came from. - Docs in the same change. README for users, architecture for the deep dive, CLAUDE.md at ten lines. Any new CLI verb, flag, default or env var updates the operational runbook in the same PR. A feature is not done until the runbook reflects it. - Every long-running CLI shows rich progress. A bar on stderr with done/total, ETA and live outcome counters, a one-line startup summary so it never sits silent, plain periodic lines when stderr is not a TTY, and Ctrl-C prints how to resume. One repo enforces this with an AST test that refuses hand-built progress objects. - Every long job resumes. Per-unit checkpoints in the target store, cursors instead of scrolls, a resume that shows as a resume the bar pre-seeded from the checkpoint , exit codes 0/1/2/130. - Humans commit. The agent prepares the branch, the commits and the PR body, then asks before git commit , git push or gh pr create . Merges are --merge , never squash, so each concern keeps its own Conventional commit. The exception is a written charter for an unattended run section 6 . - Language. Everything in code is English: identifiers, comments, test titles, commit messages. The UI language is only for what the user reads. Domain vocabulary that names a route or a registry concept is the one allowed exception in identifiers. Skills are the reusable instructions the agent loads on demand. Mine come from four sources: Matt Pocock's set, the agent-native set, a few from vendors Elastic, Vercel, PostHog, Convex and a few I wrote. A shared ~/.agents/skills store feeds both Claude and Codex. Planning and scoping - grill-me . The one I use most. It interviews me about a plan, one branch of the decision tree at a time, until we agree. Every epic starts with a grilling session; the decisions it produces go into the epic body as a "Decisions" list. - grill-with-docs . Same, but checked against CONTEXT.md and the ADRs, which it updates as decisions settle. Used on the data pipeline repos where the domain glossary matters. - to-prd and to-issues . Turn the grilled plan into a PRD, then into independently grabbable tracer-bullet issues on GitHub, each with acceptance criteria. - plan-arbiter . When Codex and Claude both produced a plan, compare them, pick one or merge a hybrid, and output one execution handoff with verification gates. - triage . Moves issues through a state machine and prepares them for an unattended agent. - handoff . Writes the conversation into a handoff document for a fresh agent. Execution discipline - efficient-fable and efficient-frontier . The orchestration pattern from section 2: the expensive model plans, delegates and judges; cheaper subagents do scans, logs, test runs and bounded edits. Always on. - stay-within-limits . Work in waves of at most three parallel subagents. Between waves check usage with ccusage , pause at 95% of the 5-hour or weekly window, re-check on resume. - plow-ahead . "Keep going, do not ask." Routine ambiguity becomes a stated assumption; the recap lists the decisions made. This is what I put in a brief when I go to sleep. - agent-watchdog . Audit another agent's work from a session id, transcript, PR or log. - tdd , diagnose , superpowers' systematic-debugging and verification-before-completion . The process skills that stop the agent from patching symptoms. - read-the-damn-docs . Before touching a third-party API, SDK or anything involving auth, billing or security, search and read the official docs. Never from memory. - quick-recap . Every finished response ends with one status line: green done, yellow a non-routine follow-up pending, red blocked on me. I read those lines from the herdr board. Writing - unslop . Strips AI tells from every piece of prose no em dashes, no sycophancy, no filler, no bold-label bullets, no sentence that would fit another project's docs and puts a voice back. A prompt-submit hook re-injects the reminder on every turn so it survives context compaction. Subagent briefs say "follow the unslop skill". - ste-writing . Simplified Technical English for runbooks. Visual and analysis - visual-plan , visual-recap , visualize-repo . Turn a plan, a PR or a repo into a local interactive document with diagrams and annotated diffs. - graphify . Any folder or repo into a knowledge graph with community clustering. - impeccable . UI design review and iteration in the live browser. - The Elastic set: Kibana dashboards, alerting rules, ES|QL log search. An MCP server is how the agent reads and pokes at the infrastructure without me pasting things. The principle: the agent should be able to see what I can see, in the dashboards I would open myself. - playwright and chrome-devtools . Browser automation for smoke tests, screenshots, console and network inspection, Lighthouse. The Playwright one is what the verifier uses to walk every signed-in route and capture a screenshot per route. - next-devtools . The Next.js dev server's own view: routes, build errors, docs. - shadcn . The component registry. UI is built from these primitives only, never hand-written, and the agent looks the component up here before adding it. - context7 . Current library docs. Frameworks move faster than model training data. - clerk . Auth SDK snippets. - convex . Read tables, run queries, tail logs, check env vars on the backend. Prod is read-only for the agent; deploys and mutations stay with me. - vercel . Deployments, runtime logs, env vars, firewall, the sandbox API. - posthog . Product analytics, session replay, feature flags, error tracking, LLM observability and the alerting. Every metric about my agent product lives here, including a cost-per-turn alert that mails me. - axiom . Log queries for the data pipelines. Pipelines log to Axiom; the app product logs to PostHog. One tool per concern, decided once. - cloudflare . Workers, R2, observability, for the repos that run edge workers. - twenty . My CRM, over MCP, so an agent can look up or update a customer while working a support issue. - resend . Transactional mail. - workos and linear on the repos that use them. Plugins from the official marketplace add the language servers TypeScript, Pyright , the code-review , feature-dev and code-simplifier agents, the security-guidance hooks, hookify for turning a repeated mistake into a blocking hook, and the remember journal section 7 . Two things I turned off on purpose. The Semgrep plugin needed a headless OAuth that could not complete and its hook blocked every edit until I disabled it. And I keep the context7 plugin off because the user-scope MCP entry already covers it. 1. Grill. I open a session on Fable or Astra and say "grill me on X". It interviews me until the design has no open branch. The output is a decisions list with the rejected alternatives named. On domain-heavy repos this also updates the ADRs. 2. Spec to tracker. to-issues files an epic and its sub-issues. Each sub-issue has the problem, acceptance criteria and out-of-scope. Labels tie the sub-issues to the epic. 3. Orchestrate. The lead session spawns one Opus implementer per sub-issue in its own worktree with a self-contained brief. At most three in flight. Implementers never commit; they leave the tree ready and report with exact commands and output. 4. Verify. For each diff, an Opus verifier with fresh context, briefed to refute. It walks the acceptance criteria in a real browser, signed in as a smoke account I created the agent is not allowed to script past the sign-up bot check . Findings get folded into the same commit or a fix scope : … n commit on the branch. 5. Gate and smoke. All gates green, targeted e2e specs named with their counts, a smoke log with real output. 6. PR. One PR per issue in most repos, one PR per epic with a commit per sub-issue in the ones where I asked for it. I read the diff top to bottom and approve the push. 7. Final smoke. After the wave merges, one Fable agent smokes dev end to end and reports green, green with findings, or red. Findings become follow-up issues. 8. Close and document. Each issue closed with the criteria-to-PR map. The narrative, measurements and traps go to the journal, the runbook and, if it changed, the architecture doc. Post-mortems of production sessions become gists. 9. Clean up. Stop dev servers, remove worktrees, sweep sandboxes, say so in the recap. Numbers from the busiest repo, since 1 July: 2,217 commits, 65% carrying a Claude co-author trailer, commits in every hour of the day, a peak of 153 commits in one day, and the issue and PR counters both past 1,160. This is the mode people ask about. It is not "let the agent run and hope". It is a written charter plus mechanical backing so the loop survives context loss. The charter. A 27 KB markdown file the top of CLAUDE.md points to as the active unattended charter. It amends a kickoff prompt whose first section is literally "OVERNIGHT ORCHESTRATOR", with an editable scope line, a 07:00 local wrap-up deadline and "MORNING DELIVERABLES": a night report, the ledger copied into the journal, a draft PR. What it pins: - The orchestrator implements nothing and reads no large files. It dispatches one worker per issue in its own worktree, three in parallel by default, five at most. - A SMOKE LOCK: one heavy real-data run at a time, ledgered. - One adversarial verifier per diff, briefed to refute, not approve. Two failed fix cycles means park the issue with a backlog label. - Tiered verification: a full adversarial pass for pipeline, cluster or CLI changes; one pass for docs and tests. Either tier parks after about two hours. - Gated self-merge: merge only when the verifier is green and CI is green on the exact head SHA. It never merges main or release PRs. It may close issues only with a three-part closing comment. - Stop conditions: the cluster down for more than 30 minutes, disk under 30 GB, a dirty tree at boot, or a step that needs a human. Human-only steps live writes, scheduling, prod deploys are queued as paste-ready commands, never executed. - Operating under the charter is the commit, push and merge authorisation that the global rule otherwise withholds. Files are memory. The charter's anti-amnesia contract says the conversation, including any summary of earlier context, is not to be trusted. The working set is three files: ledger.md , queue.md , state.md . Ledger before the next task. A re-anchor ritual at session start, after every compaction, and every ~90 minutes: re-read the charter and the three files, append re-anchored <time , resuming <task to the ledger. Commit the ledger at least every three hours so a crash costs at most three hours of record. Hooks back it up. SessionStart and PostCompact hooks cat a JSON block that says, in effect, "context was just compacted, re-read the charter and the ledger before the next tool call". The agent cannot forget the ritual because the harness re-injects it. What it produced. On the data pipeline repo: 14 dated run days with ledgers totalling 5,300 lines and 95 re-anchor rows, times taken from gh , git and the log store rather than self-reported. Four night reports. A 34-day streak of consecutive commit days. 116 commits on the peak day and 397 in the peak week. Half of all commits carry an agent trailer, and in the peak month it was 363 of 632. Steering while it runs. I steer through three things: amendments to the charter text, an operator queue of paste-ready commands I run or refuse in the morning, and a progress artifact the agent refreshes a percent-complete page with the issue list . Interactive sessions on the same repo read the orchestrator's ledger and message it before starting anything cluster-heavy, because two agents on one shared machine once clobbered each other's workspace. That incident became a pinned workspace id in the repo settings and a paragraph in the contract. Mission briefs for bounded runs. For narrower unattended jobs I write a kickoff file instead of a charter: "You are a fresh session. Do not commit. Test each prompt before moving to the next. Maintain this log file. Stop instead of patching." One such brief drove 17 sequential payment-flow test prompts. Another is a parser convergence loop: one bounded change per iteration, stop and escalate after 30 iterations without progress, finish with a handoff section in the design doc. A hackathon entry ran on an overnight mandate: at 23:28 I told the coordinator to push the score, spin up more GPU boxes and fine-tune a 9B model, keep the packaged submission valid at all times, and commit as it went. 50 of its 56 commits carry the agent trailer, running from 13:00 through the night to 11:00. The other kind of unattended time is not coding. It is the agent driving a fleet: 10 to 20 rented GPU boxes running vLLM over millions of documents, or a multi-hour crawl, or a re-index. The rules that make this safe are the CLI rules from section 3: resumable with checkpoints in the target store, progress on stderr with resume hints, dry-run by default with --apply , adaptive supervisors in tmux panes over halves of a shard plan. After the run the agent audits a sample against a gold set that was built by two frontier models and validated by hand, and one repo has a "blind rebuild" check where a second agent rebuilds the analysis from scratch to compare outputs. Three layers, because each one fails differently. The agent's own memory. Claude Code keeps a per-project memory directory: one file per fact with a type user, feedback, project, reference and an index file loaded every session. Feedback entries carry the why and how to apply, so a correction I gave once "Opus agents wrote non-English identifiers, rewrite the trailer with filter-branch" survives across sessions. Codex has its own memory store. The remember plugin. A journal per repo in .remember/ . Hooks on every tool use extract the session, a small model summarises it into now.md , an hourly pass rolls it into today-<date .md , a daily pass into recent.md seven days and archive.md . At session start the recent files are injected, so a fresh session knows what the last one was doing without me explaining. After a compaction only the identity file comes back, which is exactly why the charter's re-anchor ritual exists. GitHub and the repo. Everything that matters to a human lands in one of: the issue problem, criteria, closing comment with the criteria-to-PR map , the PR body reasoning, verification with real output, adversarial findings and how each was settled , the journal docs/journal/<issue -<slug .md , 246 files in one repo , the runbook every CLI verb, every env var, every recovery , the architecture doc measurements , and gists for post-mortems of production sessions and for research write-ups. The rule for CLAUDE.md itself is ten lines per change, so the index never becomes the manual again. - Write the contract before the first feature. The size cap and the "index, not manual" rule are the two decisions that kept it useful for a year. - Adversarial verification is not optional. The verifier gets the criteria and the diff, never the implementer's summary, and its first question is "what did the smoke test not cover". - Humans commit, in interactive mode. The charter is the one place where that changes, and it changes with named gates. - Files are memory. Anything the agent must not forget lives on disk and is re-read after compaction by a hook, not by good intentions. - Cheaper models implement, the expensive one judges. Usage windows are real. - Every long CLI resumes and shows progress. You will Ctrl-C it, and the agent will too. - Cleanup is a rule with a recap line, or the disk fills up. - Ship one wave a day with the evals green. I gate agent-product PRs on a corpus replay of delivered answers so a new refusal rule cannot silently break old ones. - Keep the eval harness in the repo. My agent product has a question bank with hard budgets, an LLM judge, a nightly core run in CI and a cost alert in PostHog. That is what makes "the agent got better" a measurable claim instead of a feeling.