Enough of an overview to understand but not enough to get stuck
A good harness is the difference between a demo and a functioning automation. Most of the LLM failures that fill the timelines from time to time are harness failures, not model failures.
Do not overcomplicate* is an evergreen series that introduces concepts that 1) quickly take over the industry, 2) feel like something everyone should be doing, and 3) quickly spiral out of control in terms of complexity, when in reality, in majority of use cases, they are quite simple - or at least only as complicated as you need them to be. *
If you read nothing else because you’re headed out for the 4th of July weekend, take away this: Harness is the difference between a demo and a functioning automation;
Similarly to evals, they are the
firststep to a functioning product, not the last;Start simple: markdown, .csv, and human eyeballs.
Terminology around AI and agents evolves so quickly that it’s hard to figure out which concepts to treat seriously because they will eventually run the business, and which one to let slide (MoltBook). The rule of introducing new food to toddlers seems applicable here: if you’ve seen it on your plate 15-20 times, it might be safe to take a bite.
Building AI agents for enterprise is my day job. Because of this, I am fairly skeptical of most stories about autonomous agents running complex real-world operations end-to-end, like automating entire jobs, because, well, I work on agents that try to do exactly that.
Most of the risk is in preventing agents from making silly or disastrous mistakes. Research moves quickly, because reliability ties up a lot of ROI. Real-world case studies are few and far between, and the never-ending streams of arXiv papers make it seem like you need a dedicated ML whiz on staff to build out the sophisticated infrastructures of pipelines and evaluation algorithms.
But just like evals can be deconstructed to their minimal (and often most effective) functional baseline of a spreadsheet and eyeballs, so can the whole concept of a harness. More of a lego box of blocks to build from rather than a prescription to start from.
Doesn’t it make sense, then, that most public “LLM failures” are harness failures?
The Chevy Tahoe for $1: no output validator (or a bad one); Replit’s agent deleting the production database: tool authorization too loose and broad;
Cursor’s “I quit” incident, where the agent refused to keep coding and told the user to learn to program: bad output validators;
Lawyers submitting hallucinated citations: issues with retrieval, prompt, and validators to check citations;
Air Canada’s chatbot’s refund - similarly, issues with retrieval - grounding claims in the actual policy document, prompt, and validators.
I’d love to deconstruct a real-life harness in the next post, so if you have one you can share (yours or public) let me know. Harnesses already are becoming the moats and tightly-protected secret sauce, and “harness engineering” a real job, even if hiding under a different title.
PS. If agent performance issues are a harness failure, is OpenAI’s social media manager job posting a harness opportunity? Because, honestly, I thought LLMs could do this already.
(I say this tongue-in-cheek, because I’m a believer that *most tech and knowledge jobs *- regardless how automatable they might seem on the surface - have a real moat that protects them from being replaced by automation, a deep “muck layer” that humans navigate intuitively and LLMs are not suited for. This “muck layer” is a concept I’ll return to often, which is why I wrote about it.
Sources:
https://www.mindstudio.ai/blog/agent-harness-engineering-wrapper-matters-more-than-modelhttps://arxiv.org/pdf/2603.28052- “Changing the harness around a fixed LLM can produce a 6× performance gap on the same benchmark.”