Scoring our agent harness against Ryan Lopopolo's 12 harness engineering theses

Scoring our agent harness against OpenAI's twelve open-source theses.

July 22, 2026

We’re hiring!Explore our[Product Engineering Guide]to learn how we build at Orbital.

Last Saturday, Ryan Lopopolo pushed a repo to GitHub called harness-engineering. If the name doesn’t ring a bell, his earlier work will: Ryan wrote OpenAI’s Harness engineering: leveraging Codex in an agent-first world, the post from February where a team of three engineers (later seven) shipped roughly a million lines of code in five months with zero hand-written lines. Around 1,500 merged PRs. About 3.5 PRs per engineer per day. Every line, from application code to tests, CI, docs and observability, was written by agents.

The line from that post that stuck with everyone was four words long: “Humans steer. Agents execute.”

The new repo is the ideas from that post grown up into an opinionated thesis-and-playbook bundle: twelve theses about how to build the environment around a coding agent, three operational playbooks for improving it, and an AGENTS.md

at the root, because the repo is designed to be read by your agent, not just by you.

We read it cover to cover (well, we and our agents did). And since we’ve spent the last 6 months building our own harness at Orbital, we thought it would be fun to do the honest thing: score ourselves against it, publicly. Here’s what we’re already doing, where we’d add nuance, and the parts we haven’t built yet but want to, both ourselves and alongside people we’d hire to work on exactly this.

What a harness actually is #

If you missed the February post: harness engineering starts from an unfashionable premise. Stop obsessing over the model. Hold the model and the coding agent constant, treat them as a black box, and notice that you’re left with exactly two levers: the context the agent can reach, and the tools it can operate. That’s the harness: everything wrapped around the model that turns raw capability into shipped, trusted work. The deeper argument, the one that runs through all twelve theses in the repo, is that your organisation’s most valuable knowledge is what Ryan calls the private process-data iceberg: your domain ontology, your quality bar, your exception history, who’s allowed to approve what. No frontier model will ever have that in its weights, because it lives in your heads and your Slack threads. The job of the harness is to move it somewhere an agent can retrieve it. And when an agent fails, the discipline is to ask “what capability is missing from the harness?” rather than just fixing the code by hand and moving on.

We didn’t need convincing. As we wrote in Everyone Ships Code Now, something shifted for us over Christmas 2025, and we’ve been rebuilding how we work around tiny teams ever since. What’s changed since February is the evidence.

The harness we’ve been building #

We’ve come to treat the harness as a product in its own right, not a drawer of helper scripts: it has owners, a roadmap, and a quality bar of its own. Most of this post is about the pieces that make it work, but it’s worth starting with why we bother. The easiest place to see whether the investment is paying off is throughput, how much reviewed, shippable work each engineer actually gets out the door.

Weekly merged PRs per contributor have climbed steadily since May, when the harness’s impact started to show up clearly in our numbers. On a legacy codebase with the harness switched on, throughput has roughly doubled: it started around 3 a week and now averages in the mid-to-high single digits, mostly landing in a 6-to-7 range. On a newer codebase built fully around the harness the gains are larger but a lot more volatile (fewer people work in it, the work there is far more varied as we build out the core harness itself, and agentic coding turns out to be genuinely addictive), swinging between the low teens and low twenties from week to week; averaged over recent weeks it runs at roughly three times the harness-enabled repo. It’s worth pausing on that. These aren’t vanity PRs: every one clears the same review bar, type checks, and CI gates as before, with most of the review now handled by coding agents and humans checking the rest. If the pattern holds, a codebase built fully around the harness is roughly three times more productive to work in than one that has merely switched it on. That is not a marginal gain; it’s a step change.

The number that matters to us isn’t lines of code; it’s throughput per person at the same quality bar. A few of the load-bearing pieces, in roughly the order an agent meets them:

The repo teaches the agent. A short root AGENTS.md

acts as a table of contents (philosophy, glossary, task routing), and each subsystem carries its own guide that loads only when an agent works in that directory. Ryan’s repo calls this progressive disclosure; we arrived at the same shape independently, for the same reason: a 10,000-line context dump measurably degrades how well agents perform. Alongside those guides sit twenty-odd skills (procedural runbooks for things like reviewing a changeset, standing up a cloud environment, or capturing a QA bug with a screen recording) that activate on demand rather than squatting in every prompt.

Agents run the real system, not a simulation. Every worktree gets its own isolated Docker stack (Postgres, Redis, Temporal, MinIO, the works) so an agent can boot the actual product, seed it with demo data, and drive the actual UI headlessly to prove a change works. “It compiles and the unit tests pass” is not proof. A browser journey through the running app is.

The night shift. While we sleep, a nightly bug hunt fans out a fleet of finder agents across the codebase, adversarially verifies every finding (independent sceptic agents whose whole job is to refute each bug), fixes the confirmed ones via TDD, proves the fixes against the running stack with before/after screenshots, and opens a PR for humans to review over coffee. Separate maintenance agents wake at 6 AM to catch documentation drift and config inconsistencies. Ryan’s post calls this technical debt garbage collection: capture human taste once, then enforce it continuously. Ours has been running for months.

A feedback agent that owns the whole job. When a user reports an issue and a human labels it for the agent, a CI job boots the full stack, reproduces the reported behaviour live, fixes it test-first, self-reviews the diff, opens a PR, and reports back to the ticket. If a reviewer requests changes, a fresh revision round spins up, seeded from the PR and the previous round’s handoff note. Mid-run, the agent can even ask our engineers questions over Slack, with a strict two-question budget, because an agent that asks unlimited questions is just a very slow intern.

Autonomy inside explicit authority. The agent can do a lot; it can’t do everything. Forbidden paths are re-checked on every push to an agent-authored PR. Force-pushes are banned (they orphan review threads). Revision rounds are budgeted. And merge stays human, every time. This is Ryan’s thesis seven almost verbatim: separate capability from authority.

Every commit knows what it cost. A git hook stamps each agent-assisted commit with a trailer recording the tokens spent and an estimated dollar cost, per model, deduplicated and checkpointed so a PR’s commits never double-count. When someone asks “what did that feature cost in agent time?”, the answer is in git log

.

The scorecard #

So how do we do against the twelve theses? We asked our agents to grade us harshly. (They obliged. Agents love grading things.)

On thesis one, a nuance rather than a disagreement. It’s easy to read “hold the worker constant” as “don’t chase models,” but the thesis is careful: you hold the model constant within an adoption epoch, then requalify the whole environment the moment a better one ships. We agree completely. Where our bet on the model philosophy leans further is timing: we deliberately scaffold for the frontier ~6 months out, building for capabilities that aren’t quite reliable yet, so the harness is ready the day the next epoch opens rather than qualified after it. Constant within an epoch; impatient about the next one.

The unbuilt parts (this is where you come in) #

The ✅ rows are where we’re confident in what we’ve built, though there’s still plenty of room to improve. The 🚧 rows are where we need to up our game and build more into the harness. We’re actively hiring people to help us build out all of these areas and make this the best harness we can. These aren’t hand-wavy “areas for growth”. Each is a concrete system we want to exist, and each is the kind of problem you can’t work on at most companies because most companies don’t have the substrate yet.

Mining our own trajectories. Every agent session leaves a transcript, and we already parse them all, for the cost trailer. Nobody yet mines them for the interesting signal: the repeated steering, the corrections humans make twice, the moments where an engineer acted as a human relay between the agent and a system it should have been able to reach itself. Ryan’s playbook calls this evidence recovery, and the rule is that every recurring correction should be promoted into a durable owner (a type, a lint, a doc, a skill) so it never has to be made a third time. We want to build the maintenance agent that reads a week of transcripts and opens the PRs that make the whole harness smarter. Organisational judgment, made cumulative, mechanically.

Closing the loop on harness changes. Here’s an uncomfortable question from the repo’s evals methodology: when you add a skill or a doc to your harness, do you know the agent ever retrieves it? Context that’s available but never invoked is dead weight you can’t see. We ship harness changes on good judgment today; we want to ship them the way the playbook prescribes: baseline run, smallest intervention, fresh rerun under identical conditions, and instrumentation that distinguishes available from retrieved from actually load-bearing. Then retire what isn’t earning its keep. Our eval framework is the right chassis; the closed loop is unbuilt.

A failure taxonomy with teeth. When an agent PR needs three revision rounds or bails out to a human, why? The playbook offers a sharp classification (context gap, capability gap, authority gap, proof gap) and a discipline we admire: fix the earliest failed handoff, not the latest symptom. The lazy fix for agent failure is always another downstream check. The right fix is usually upstream, and we want the tagging and analysis that tells us where.

Effectiveness measured in human attention. We can tell you what any commit cost in tokens. We can’t yet tell you what it cost in attention: steering messages, review minutes, revision rounds. Thesis twelve says the metric that matters is useful outcomes per unit of human attention, and we agree, and we have all the raw signals sitting unaggregated in our systems. Someone gets to design that metric and then move it.

If you read those four paragraphs and your fingers twitched, that’s the reaction we’re screening for.

Come build the harness #

Many companies are still debating whether to let agents write and review code. We’re past that. Agents write all of our code, ship it through the same review bar as before (now mostly enforced by agents, with humans reviewing the rest), and hunt their own bugs overnight. The frontier for us is the harness itself: the compounding machine that turns every correction, every review comment, and every 2 AM bug hunt into a permanently smarter environment.

Ryan’s repo ends with an invitation to point your coding agent at it alongside a system you want to improve. We did. It agreed with most of what we’ve built and handed us a to-do list for the rest. Which makes for a straightforward hiring pitch: the highest-leverage problems here have been validated by the playbook’s own author, and they’re still unbuilt and waiting.

And engineering is only the first of three harnesses we want to build. Everything above is build craft made to compound; a tiny team runs on two other skills, and both deserve the same treatment. A product harness that encodes how we decide what to build and where our quality bar sits, so product judgement compounds instead of living in a few people’s heads. A design harness that captures what good looks like to us and enforces it as relentlessly as our type checks. We’re a long way into the engineering harness and have barely started the other two: far along against these twelve theses for engineering, with the same journey ahead of us twice over.

We’re hiring product engineers who want to work like this. Have a look at our open roles, and if you’d like to talk it through, or push back on where we’ve landed, our inbox is open.