{"slug": "the-rise-of-the-harness-engineer", "title": "The Rise of the Harness Engineer", "summary": "Tessl researcher and engineer Amy Heineike argues that as coding agents author the vast majority of code, a new role she calls the \"harness engineer\" is emerging to manage AI-authored code and maintain software quality. Heineike cites a roughly 100× spread in agent cost versus intelligence on Artificial Analysis's pareto frontier charts, about a 10× cost difference between a top-tier model and a cheaper capable one for the same task, and Tessl closing roughly 600 PRs in a single week, while noting bugs and production incidents are rising as more PRs pass through without human code review. Tessl built a benchmark from around a thousand open-source skills, running agents with and without each skill to grade both task completion and instruction-following.", "body_md": "ARTICLE\n\n# The Rise of the Harness Engineer\n\nDiscover the evolving role of the harness engineer in AI-driven coding. Learn new skills to manage AI-authored code and maintain software quality.\n\nAmy Heineike\n\nWe're in an intense moment as software engineers. On one hand, coding agents have become genuinely amazing and we’re using them all the time with higher and higher throughput. On the other, they build in sloppy, bug-accumulating ways, and we’re under unrelenting pressure to keep them in check.\n\nI'm a researcher and engineer at Tessl, where I've spent a lot of the last while helping build our internal software factory and evaluating what agents actually do when you point them at real work. The conclusion I keep arriving at is this: as agents write more and more of our code, engineering doesn't matter *less*. It matters *more*. It just looks different from the engineering many of us grew up doing. I've started calling the person who does this new kind of work a **harness engineer**, and I think it's a role a lot of us are about to grow into, some as a full-time specialism, many more as a growing slice of the job alongside product engineering.\n\nLet me walk through how I got there, and what new skills are emerging as crucial to staying in control.\n\n## The agents got great, but cracks show over time\n\nStart with the good news, because it's genuinely remarkable. The adoption curve for AI coding tools has gone close to vertical. A year ago adoption was spiky and experimental; now, across a lot of organisations, the *vast majority* of code is AI-authored [see [Faros AI, AI whiplash report](https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways)]. But, hand-in-hand, the number of bugs and production incidents are increasing with it, as more PRs slip through with no human code review.\n\nThe benchmarks tell the same story from the other side. Task based benchmarks are largely saturated – if you can define a task cleanly and measure if it was done, an agent can complete it. But if you ask agents to refactor large codebases, or build on their work over time, then we see deterioration, and much lower levels of success.\n\n## Better agents will help over time, but also come at a cost\n\nThe natural hope is that better models will simply dissolve all of this. But there will be heavy trade offs – much of the issue is reasoning about overall design decisions and reviewing existing code to follow patterns and stay consistent – all of which take time and reasoning effort to do. Expecting agents to do this well from first principals in every session would be incredibly expensive even if they can do it.\n\nIf you plot how much it costs to run an agent against its intelligence, the intelligence axis moves roughly linearly while price moves on a log scale. [see [Artificial Analysis’s pareto frontier charts](https://artificialanalysis.ai/)] Across the range that's something like a 100× spread, and the difference between a top-tier model and a perfectly capable cheaper one can be around 10× for the *same* task. Even when the very best model can do the job, that can be an extraordinarily expensive way to solve a problem. And when you start running agents at real throughput, this stops being academic. We closed something like **600 PRs in a single week** recently, and we're a small company. When that much is flowing through, cost and efficiency become first-class engineering concerns.\n\n## The steering gap\n\nWe think a lot about *skills* at Tessl, so we built a benchmark to understand them properly. We took around a thousand skills from open-source repos, built tasks that would exercise those skills, and then ran agents through the tasks both *with* and *without* the skill so we could compare. We graded two different things: did the agent complete the task, and did it actually follow the instructions the skill laid out, not just \"did it finish,\" but \"did it do it the way it was told.\" [see: [https://arxiv.org/abs/2606.17819\\]](https://arxiv.org/abs/2606.17819])\n\n#### What did we learn?\n\nFirst, well defined skills reduce the difficulty of tasks and let **cheaper, smaller agents reach the same level of success** as much larger, more expensive ones. If we can make it easy to find the right guidance and break tasks apart well, we’ll win in both consistency and efficiency.\n\nSecond, and less comfortably: **instructions don't get followed thoroughly**. Task completion rates were high across almost everything we studied, but only about **70% of the instructions in a skill were followed on average**. When you break a skill down into its individual instructions and check them one by one, even the best models are quietly ignoring a big chunk of what you told them. Skills aren’t enough, they need to be paired with checks that the rules are followed consistently.\n\nThere’s an enormous amount of value in designing how we use agents, rather than just using them as they are and hoping they make the right choices.\n\n## Engineering matters more, not less\n\nHere’s the shift. When we hand-wrote code, we could review and discuss it, and many of the key decisions we were making could emerge and be shared implicitly as we worked. We’d know to go sketch an architecture diagram on a board when it stopped scaling the way we wanted, or we’d spike on which library to choose at the moment we realised it was important, and we could feed those choices back in. It was easier to mix taking time on the overall approach and the specific task.\n\nNow, the agents are going so fast, it’s easy to fall into only being able to react – being pinged to review the latest PR when it’s ready, and feeling frustrated with the same errors being repeated.\n\nIf we want to go really fast, we need to be proactively spending our time designing the system in which the agents run, making our choices **explicit** and measuring whether the system that is emerging is what we want.\n\nI see three skill sets that matter more now than they used to, and that are a little different from the ones many of us spent years developing.\n\n### 1. Systems thinking: capturing invariants\n\nSome of what you want from a change is specific to the task in front of you. But a lot of it is really an expression of a general principle you want to stay true across the whole system. If you can identify those **invariants**, the things you want to keep being true, and then capture and enforce them, you have a real shot at keeping consistency and holding the quality bar high no matter who, or what, is doing the writing.\n\nWhat do invariants look like in practice? **Design systems** are a great example, you describe the types of layouts you want, the brand voice, the components, what a button looks like. Individual pages are then built from those reusable components. They show up in **architecture**, how you want code structured, which libraries you've chosen, opinionated views on how they should be called. Each new feature follows the patterns and choices consistently. And they show up in **best practices**, how errors should surface through the stack, how things should be named, how data should flow.\n\nIn the past, these lived in a code owner's instinct; you could ask them to sketch an architecture diagram, or sit with a designer over the Figma files, and then have them check PR by PR whether things matched.\n\nNow we want to make these choices explicit, and when we do we can enforce them consistently much more cheaply than we could before, effectively steering development from these patterns. Why is it easier? Because just as agents are better at writing code, they are also better at reviewing code – especially when the rules are clear and thoroughly described.\n\nHow do we enforce them:\n\n- We can write **skills** that describe exactly what we care about and load them at the right moment.\n- We can put **deterministic checks** into CI, linters, and tools like ast-grep, which I'd barely heard of a few months ago and now talk about constantly. (Agents are excellent at*writing* these, so the thing that used to be a pain is now easy.)\n- We can add narrow **verifiers** , tight rules measured on every single file by a quick agent, to enforce the kinds of constraints that are hard to pin down deterministically.\n- And we can do **agentic code review** with broader prompts describing the principals we care about.\n\nBetween them these form a system of guidance and guardrails that steer the agents when they are working, and review if they did things correctly. They leave a lot less on the table to be caught in ad hoc review, or to slip through and cause problems later.\n\n### 2. Analytics: reason about the system with data\n\nThis one surprised me, because I didn't used to think of analytics as an engineering skill. But as we automate more of the code-generation and review process, we generate more and more data we can actually look at, and there's a wealth of signal in it. This is what enables us to find invariants and measure what’s important.\n\nNew signals that we can study:\n\n- **Agent logs** are fascinating. They show where agents are spinning and getting confused. You can see where it wastes time. For example, are agents struggling with a particular API, are they reading files they should, are they puzzled about what we want?\n- **PR comments** are another signal. Do comments generalise into invariants that could be applied widely? What types of errors are being caught?\n- **Analysis of the code base** itself. Is the**complexity** of the codebase worsening with spaghetti code importing from everything and god files growing to thousands of lines? Does**mutation testing** find tests that aren't earning their keep, and gaps in coverage that matter? Are there multiple ways of doing the same thing that could be converged?\n\nThen you turn all of that into *hypotheses* about where agents are going wrong, and then look for how you could address that – giving the agents better tools, refactoring code, adding skills, defining new invariants and enforcing them so you never have to worry about that failure mode again.\n\nSometimes that's a human grabbing the data and staring at it; sometimes it's an automated loop where an agent regularly runs the analysis and finds incremental improvements. Either way, it’s a seed for discovering how well the system is working and where improvements could be best made.\n\n### 3. Risk and operations: grade the blast radius\n\nAs we build trust in review mechanisms, then we need to reduce the human review burden. We need to reason about the risk we’re introducing by relying on our processes and how that aligns with the sensitivity of the code base.\n\nAt Tessl we use a kind of **ladder**. Our research codebase is a genuine free-for-all, merge anything you like, no rules. Some of our internal tooling is auto-merged; hundreds of changes flow through, with purely automated review, and nobody looks at any of them. Other parts of the codebase absolutely need an engineer, the person who owns the feature has to take it through and do the final merge themselves. And a few very high-leverage parts should make you stop and go ask someone who's thought hard about them, rather than ramming a change in.\n\nThe work is figuring out which parts of the codebase sit at which level of risk, and then codifying that into the approvals flow.\n\n## The harness engineer\n\nPut those three together, invariants, analytics, risk, and you get harness engineering. For a lot of organisations this will live in a few specialists who really embody it. But I suspect that for many of us working in engineering, it simply becomes more and more of where we spend our time. That's a real shift in identity: we become tenders and gardeners of our code and our systems, so that when we build features we can focus on whether the behaviour is what we want and trust that the implementation will be consistent and scalable.\n\nSpending your hours on invariants and analytics and risk policy, rather than only on the next feature, feels different, but it has tremendous leverage.\n\nWe've leaned all the way into this. Around **90% of our code went through the factory this week**, and we've been over 80% for the last month. The shape is consistent: more of the coding moves into remote execution loaded up with skills and context; a layer of **guardrails** (deterministic checks, LLM verifiers, CI) run through a detailed checklist of rules; **automated code review** handles new problems and feature specific issues; a **triage/gating** step reasons about how much risk and care a given change needs; and **update loops** monitor what's happening and feed learnings back into every layer of the stack.\n\n## Where to start\n\nIf this isn't your full-time job yet, here's where I'd begin, three small exercises you can try this week.\n\n1. **Build a dataset.** Agents are great at this, tell one to grab your last 50 PRs and summarise what kept going wrong: what people repeatedly said in comments, what errors recurred, where things tripped CI. Identify issues that generalise and might come up again.\n2. **Distill three invariants** for code you work on. Work out what rule would clarify the behaviour you want - in architecture, design, behaviour, or library usage for example.\n3. **Turn one invariant into a CI gate** , and then make sure you have a way to track what happens with it and watch it. Did it start catching the problem you cared about? How often is it firing? Try implementing it using AST-grep (if it can be expressed deterministically) or try out tessl verifiers (if you want something broader).\n\nFor any of these, we’ve loaded guidance into tessl agent, so you can ask it to help you work out where to start and see if you agree with it. Install tessl and run `tessl agent` to get going. The free tier should be plenty for building out your first invariants.\n\nHere are some examples of rules we have implemented at Tessl to help you get started.\n\nAgents gave us rocket boots. Harness engineering, systems thinking, analytics, and risk, is how we keep from flying straight into a wall. And I think it's about to be some of the most valuable engineering work any of us can do.\n\nCOPY & SHARE\n\nAmy Heineike\n\nAmy is Founding AI Engineer at Tessl. She was previously the VP of Engineering at 7bridges and Primer.ai as well as Director of Mathematics and Machine Learning at Quid.\n\nREADING\n\n·\n\n0%\n\nCOPY & SHARE\n\nAmy Heineike\n\nAmy is Founding AI Engineer at Tessl. She was previously the VP of Engineering at 7bridges and Primer.ai as well as Director of Mathematics and Machine Learning at Quid.\n\nYOUR NEXT READ\n\n## Jev is 13.6x faster and 2.7x cheaper than GPT Luna 6 for Tessl verifiers. Try it yourself.\n\nJev, TypeSafe's new decision model, is 13.6x faster and 2.7x cheaper than GPT Luna 6 for Tessl verifiers, offering a cost-effective and efficient solution.\n\nSimon Maple, Amy Heineike", "url": "https://wpnews.pro/news/the-rise-of-the-harness-engineer", "canonical_source": "https://tessl.io/blog/the-rise-of-the-harness-engineer", "published_at": "2026-08-25 14:10:43+00:00", "updated_at": "2026-09-23 20:31:29.579253+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products", "developer-tools", "ai-research"], "entities": ["Tessl", "Amy Heineike", "Faros AI", "Artificial Analysis"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-rise-of-the-harness-engineer", "markdown": "https://wpnews.pro/news/the-rise-of-the-harness-engineer.md", "text": "https://wpnews.pro/news/the-rise-of-the-harness-engineer.txt", "jsonld": "https://wpnews.pro/news/the-rise-of-the-harness-engineer.jsonld"}}