Hardening a host for autonomous coding agents

A developer documented a runbook for hardening an unattended coding-agent setup, migrating from agents running as a user on a VM to each agent working in its own disposable dev machine with no credentials. The migration replaced environment-variable secrets with socket-activated credential brokers (for model API, package registry, GitHub tokens, and read-only production access), moved orchestrator config to root-owned files outside the agent-writable repo, and isolated each agent in a per-issue container on an internal Docker network behind an allowlisting proxy. The writeup catalogs near-misses including repo-planted mise.toml, .codex/, .git/config and .yarnrc.yml files executing outside the sandbox, and install hooks running as the orchestrator's user.

A runbook from moving an unattended coding-agent setup an orchestrator that picks issues from a tracker and runs a coding agent on each, opening PRs from "agents run as a user on a VM" to "each agent works in its own disposable dev machine and holds no secrets". It is written from a real migration of one VM: an orchestrator Symphony https://github.com/openai/symphony -style , the Codex CLI as the agent, a Rails + React monorepo, GitHub, and a Linear tracker. The names are ours; the shape applies to any agent that runs shell commands. Everything below was done in small, reviewed steps, each one tried on a throwaway workspace before it went live, and each change to a trust boundary got an adversarial review and a re-review of its fixes. The order matters more than any single setting. Treat the agent as untrusted code that runs with whatever you give it . It reads issue text, PR comments, web pages and repo files, any of which can carry instructions prompt injection , and it executes commands. The questions are what it can reach, what it can steal, and what it can make something else run. What actually bit us, or nearly did: - Repo files that tools execute outside the sandbox. A version manager's shim mise loaded the workspace's mise.toml and ran its env scripts when the orchestrator started the agent through a Node shim. A .codex/ directory in the repo is a config layer the agent CLI reads. .git/config can name programs git runs fsmonitor, filters, diff drivers . A .yarnrc.yml with yarnPath planted above the workspaces would have been run by every later setup. - Hooks running workspace code as the orchestrator's user. Setup scripts that bundle install / yarn install in the workspace run gems' and packages' install code with the orchestrator's privileges. - Credentials in the environment. Everything in the service's env file ended up in every agent command model key, tracker key, payment test key, private gem credential . - The orchestrator's own config in a place the agent can write. Our workflow file hooks, the agent command, the sandbox policy lived in the repo the agents open PRs against, in a checkout their own user could write, and the orchestrator re-read it every second. - Production access. An admin CLI and kubeconfig on the same host. - The agent tearing down its own workspace mid-run by calling the cleanup script, which then starved the next run. tracker ──► orchestrator unprivileged system user, hardened unit, egress allowlist │ hooks + agent command are fixed, root-owned, deployed from a separate repo ▼ devbox broker root, tiny, refuses anything it didn't decide itself │ one container per issue, fixed image/user/limits/mounts ▼ devbox: the agent's own laptop • own Postgres / Redis / search on its loopback, seeded dev data • no credentials at all • network: an internal Docker network; the host firewall lets it reach only a few ports on the gateway ──► credential brokers + an allowlisting proxy Credential brokers on the host, one per secret, each a small socket-activated service: | Need | Broker | What the container gets | |---|---|---| | Model API | model broker: accepts only the one request shape the agent sends, adds the key, strips auth from responses | a placeholder key and the broker's URL | | Private package registry | gem broker: fixed upstream, fixed paths, adds the credential | a Bundler mirror setting | | GitHub | token broker for a GitHub App: 1-hour installation tokens for one repo; a credential helper and gh wrapper call it over a mounted socket | no token at rest | | Production reads | a read-only proxy with a read-only DB role and masked views | a profile that only works through the broker | | Uploading evidence screenshots to the tracker | an uploader that only writes to the run's own issue | a socket | Each broker runs as a systemd DynamicUser and gets its one secret through LoadCredential= from a root-only file 0600, in a 0700 directory . Nothing else on the host can read it, and the orchestrator's environment holds only the tracker key it needs itself. 1. Inventory and runbook. Write down what the host is : units, mounts, users, env files key names only, never values , cron, firewall. Keep it in a repo, with a capture script that diffs the live host against the repo and flags drift, including key names in env files vs templates and anything that must not come back. 2. Unprivileged orchestrator. Its own system user. systemd hardening: ProtectSystem=strict , the home read-only except a short ReadWritePaths= list, NoNewPrivileges , ProtectKernel , RestrictSUIDSGID , LockPersonality , PrivateDevices and RestrictNamespaces once nothing in the unit needs them. Make the startup files and tool shims root-owned they decide what runs so a process running as that user can't change them. 3. Egress allowlist. An nftables table that only lets the service user and later the containers reach a forward proxy, and a proxy allowlist of the hosts work needs GitHub, the package registries, the tracker's API, browser downloads . Reload, never restart, the firewall. 4. Control plane out of the agents' reach. Move hooks, the agent command and the sandbox policy to a separate repo the agents' GitHub App isn't installed on. A deploy tool installs them root-owned and refuses : free shell in hooks they must be exact commands , unknown flags or config keys on the agent command, a changed sandbox policy unless explicitly allowed, a config that two YAML parsers read differently, and deploys while agents are running. 5. GitHub as an App, not a personal token. Short-lived installation tokens scoped to one repo; rulesets: agents can only create/update branches under their own prefix, master needs a PR and status checks , nobody force-pushes, tags are protected. Agents never merge. 6. Production: read-only and brokered. The admin CLI's agent profile only works through a broker that holds a read-only role; the agent's command rules forbid kubectl , the cloud CLI and other profiles a soft barrier; the hard one is that the credentials aren't there . 7. Devboxes. Build an image with everything a run needs toolchain, dependencies, browsers, the agent CLI, databases, a checkout of the default branch . A root broker creates one container per issue with fixed settings next section ; the hooks only ask the broker. Then remove the toolchain from the host entirely: nothing that runs repo code should exist there. 8. Secrets out of the containers. Brokers above; the container's env holds placeholders. Verify by reading /proc/1/environ from inside. 9. A dev machine, not a test runner. Seeded development database, the dev server, a login helper, a screenshot helper, mail preview, search. If the agent can't see the app, it asks humans for screenshots and burns runs. 10. Clean up credentials last , each one listed names only and removed with an explicit go. Everything fixed by the broker; the caller chooses nothing: - Image pinned by digest, loaded only if it carries the label of a build from the default branch. - The agent's uid matching the owner of the bind-mounted workspace; a per-container home volume seeded from the image for databases and caches. - CapDrop: ALL , no-new-privileges , not privileged, ReadonlyRootfs , /tmp a sized noexec tmpfs, PidsLimit , memory limit with MemorySwap equal no swap , ShmSize for the browser, IpcMode: private , an init process. - Docker's default seccomp profile, unchanged , Docker's own /proc and /sys masks, and an AppArmor profile = Docker's default plus deny mount , deny pivot root , deny userns on kernels that don't restrict unprivileged user namespaces themselves, say so explicitly . - OomScoreAdj: 500 for containers and OOMScoreAdjust=-500 for the orchestrator's unit: if the host runs out of memory, an agent dies, not the orchestrator whose default OOMPolicy=stop would end every run . - Network: an internal Docker network with enable icc=false ; the host firewall accepts new connections from it only to the brokers' and proxy's ports on the gateway. No route to the cloud metadata endpoint. DNS through a resolver file you control. - Mounts: the workspace read-write at its own path; read-only: the deployed control plane, the agent's config and its limits file, the brokers' sockets. Nothing else from the host. - Limits on how many run at once and exist in all; a refused create rather than an overcommitted host. - A warm spare, prepared from the current image while nothing else prepares, claimed by the next new issue prep went from minutes to ~30 s . Codex like other agent CLIs sandboxes each command bubblewrap on Linux: fresh namespaces per command, read-only paths . Inside a container that has two costs: nothing an agent starts outlives the command a dev server or browser started in one command is gone in the next, so agents write fragile one-shot scripts , and nesting it forces the container to allow user namespaces and mounts and to turn off Docker's /proc masks, which is more kernel surface, not less. We trialled both and chose: no inner sandbox, the container is the wall danger-full-access inside the devbox, pinned by the deploy tool and by the agent's limits file . That is only acceptable because the container holds no credentials, its network reaches only the brokers and the allowlist, and nothing on the host executes workspace content. With it, the container got stricter default seccomp, masks back, deny userns , and a red-team from inside found no way out. Two consequences to handle: - A git wrapper we had that sandboxed the agent CLI's own git calls an agent-written .git/config could start a program with the CLI's environment, which on the host held the model key became pointless and was removed: inside the container the CLI holds no key and is the same user as the agent. - The agent's limits file must still pin what matters no hooks, no MCP servers, no plugins, the model provider, approval policy, forbidden command prefixes . Check that your CLI version still enforces those rules without its sandbox ours did . - Nothing on the host runs workspace content. Hooks are fixed commands; cleanup only fixes permissions so the orchestrator can delete the workspace; prep runs inside the container from a fresh clone of the default branch. - The agent CLI is started from its native binary , never a script wrapper an interpreter or version manager resolves from the workspace. Refuse to start if the workspace has a CLI config directory. - One secret per consumer, in a file only it reads. Splitting a shared env file is cheap and removes whole classes of leaks. Note that LoadCredential copies the file when the service starts: a rotated key needs a restart. - Move secrets without leaving copies: in a root shell, umask 077 , grep '^KEY=' old new , then sed -i '/^KEY=/d' old . Only key names ever appear on a command line sudo logs commands . No editors, no backups. - Deploy while idle. The orchestrator re-reads its config continuously; a change mid-run changes a run. - Sandboxed services can hide what you need. systemd's ProcSubset=pid hides /proc/meminfo ; a "don't build a spare without free memory" check silently returned false forever. Test checks under the real unit . - Checks that fail open are worse than none. Two image-build guards ran git as root on a repo owned by another user; git refused "dubious ownership" , and both checks read the refusal as success. Make every guard fail when its tool fails. - Workspace-local caches and state. Anything shared between workspaces and writable by agents is a way for one run to plant something another or a privileged step uses. - Seed a development database at image build and keep it as a template database; each workspace copies it in seconds. Same for test/e2e databases and skip the schema load when each database's recorded schema hash still matches the checkout . - Point the app's dev hostnames at the container's own loopback ExtraHosts and keep them out of the proxy NO PROXY . - A bin/dev that starts the web server and background workers, a login helper that prints a signed one-click URL, a screenshot helper that signs in and refuses captures whose CSS/JS failed to load, and a mail preview letter opener-style so email changes are visible. - Search: a single-node search engine per devbox on loopback copied from the same pinned image CI uses, without the ML binaries, small heap, no heap pre-touch , indices built at image build. - Measure memory with everything running at once server, workers, browser, a test run and size the container from that. - Tell agents what they have in the repo's own docs, in a few lines, and delete agent-only rules "do it all in one command" as soon as the environment no longer needs them. Rules that tell agents to delegate every browser check to a sub-agent cost more time than the check. - Tracker-hosted images only render inside the tracker: on GitHub, link the issue instead of embedding them. - Every change to a trust boundary hooks, the agent command, sandbox settings, units, brokers, credentials went through: a trial in a throwaway workspace with a real agent turn → a PR → an adversarial review a second model told to break it → fixes → a re-review of the fixes → merge → deploy while idle → capture shows no drift. - A red-team from inside a devbox as the agent's user : namespaces, mounts, io uring/bpf/keyctl, ptrace, /proc and /sys writes, setuid binaries, other containers, the host's ports, the metadata endpoint, the proxy with IP literals/other ports/ CONNECT , each broker with other paths/methods/hosts, and reading every credential-looking thing. - Docs drift: every infra PR also gets a review of the docs against the live host. - A real issue end to end: an agent asked only for "browser evidence that sending a message works and that the failure path is handled" started the dev server, sent a message, read the email in the mail preview, forced a 500 to check the alert/draft/retry, captured the error report payload without sending it, and attached screenshots, with no human help beyond one route hint. - Inventory in a repo; drift-detecting capture script; env templates with names only. - Orchestrator as its own user; hardened unit; startup files and shims root-owned. - Egress allowlist firewall + proxy for the service and the containers. - Control plane hooks, agent command, sandbox policy root-owned, deployed from a repo agents can't touch, with a deploy tool that refuses unpinned changes. - GitHub App with short-lived, single-repo tokens; rulesets: agent branch prefix only, PRs to the default branch, required status checks, no force-push, protected tags; agents never merge. - Production reads only through a read-only broker; admin tools absent from the agent's world. - Per-issue containers from a pinned image, every setting fixed by a root broker; no toolchain on the host. - No credentials in containers; one broker per secret; each secret in a file only its broker reads. - Decide on the inner sandbox; if dropped, harden the container default seccomp, masks, deny mount/userns . - OOM ordering: containers first, orchestrator last. - A real dev machine in the container seeded data, dev server, login, screenshots, mail, search ; docs that say so briefly. - Adversarial review + re-review for every trust-boundary change; a red-team from inside; deploy only while idle.