{"slug": "do-not-overcomplicate-agentic-harness", "title": "Do not overcomplicate: Agentic Harness", "summary": "Most public LLM failures are harness failures, not model failures, according to a senior AI agent builder who argues that a good harness is the difference between a demo and a functioning automation. The author cites examples including the Chevy Tahoe for $1 incident, Replit's agent deleting a production database, Cursor's \"I quit\" incident, lawyers submitting hallucinated citations, and Air Canada's chatbot refund issue, all stemming from problems with output validators, tool authorization, retrieval, or prompt design. The piece advocates starting simple with markdown, CSV, and human eyeballs, and positions harness engineering as an emerging role that is becoming a tightly-protected moat.", "body_md": "# Do not overcomplicate: Agentic Harness\n\n### Enough of an overview to understand but not enough to get stuck\n\nA good harness is the difference between a demo and a functioning automation. Most of the LLM failures that fill the timelines from time to time are harness failures, not model failures.\n\nDo not overcomplicate* is an evergreen series that introduces concepts that 1) quickly take over the industry, 2) feel like something everyone should be doing, and 3) quickly spiral out of control in terms of complexity, when in reality, in majority of use cases, they are quite simple - or at least only as complicated as you need them to be. *\n\nIf you read nothing else because you’re headed out for the 4th of July weekend, take away this:\n\nHarness is the difference between a demo and a functioning automation;\n\nSimilarly to evals, they are the\n\n*first*step to a functioning product, not the last;Start simple: markdown, .csv, and human eyeballs.\n\nTerminology around AI and agents evolves so quickly that it’s hard to figure out which concepts to treat seriously because they will eventually run the business, and which one to let slide (*MoltBook*). The rule of introducing new food to toddlers seems applicable here: if you’ve seen it on your plate 15-20 times, it might be safe to take a bite.\n\nBuilding AI agents for enterprise is my day job. Because of this, I am fairly skeptical of most stories about autonomous agents running complex real-world operations end-to-end, like automating entire jobs, because, well, I work on agents that *try* to do exactly that.\n\nMost of the risk is in preventing agents from making silly or disastrous mistakes. Research moves quickly, because reliability ties up a lot of ROI. Real-world case studies are few and far between, and the never-ending streams of arXiv papers make it seem like you need a dedicated ML whiz on staff to build out the sophisticated infrastructures of pipelines and evaluation algorithms.\n\nBut just like [evals can be deconstructed](https://hamel.dev/notes/llm/evals/flashcards/?ck_subscriber_id=3951136500&utm_source=convertkit&utm_medium=email&utm_campaign=Eval%20Series%20Recap:%20All%2012%20Evals%20Flashcards%20+%20Whats%20Next%20-%209229524) to their minimal (and often most effective) functional baseline of a spreadsheet and eyeballs, so can the whole concept of a harness. More of a *lego box* of blocks to build from rather than a prescription to start from.\n\nDoesn’t it make sense, then, that most public “LLM failures” are harness failures?\n\nThe Chevy Tahoe for $1: no output validator (or a bad one);\n\nReplit’s agent deleting the production database: tool authorization too loose and broad;\n\nCursor’s “I quit” incident, where the agent refused to keep coding and told the user to learn to program: bad output validators;\n\nLawyers submitting hallucinated citations: issues with retrieval, prompt, and validators to check citations;\n\nAir Canada’s chatbot’s refund - similarly, issues with retrieval - grounding claims in the actual policy document, prompt, and validators.\n\nI’d love to deconstruct a real-life harness in the next post, so if you have one you can share (yours or public) let me know. Harnesses already are becoming the moats and tightly-protected secret sauce, and “harness engineering” a real job, even if hiding under a different title.\n\nPS. If agent performance issues are a harness failure, is [OpenAI’s social media manager job posting](https://jobs.ashbyhq.com/openai/b73e3a42-bc48-4490-882e-66f9e11bbfc5) a *harness opportunity*? Because, honestly, I thought LLMs could do this already.\n\n(I say this tongue-in-cheek, because I’m a believer that *most tech and knowledge jobs *- regardless how automatable they might seem on the surface - have a real moat that protects them from being replaced by automation, a deep “[muck layer](https://loadbearingtech.substack.com/p/every-emerging-tech-has-to-yield)” that humans navigate intuitively and LLMs are not suited for. This “muck layer” is a concept I’ll return to often, which is [why I wrote about it](https://loadbearingtech.substack.com/p/every-emerging-tech-has-to-yield).\n\nSources:\n\n[https://www.mindstudio.ai/blog/agent-harness-engineering-wrapper-matters-more-than-model](https://www.mindstudio.ai/blog/agent-harness-engineering-wrapper-matters-more-than-model)[https://arxiv.org/pdf/2603.28052](https://arxiv.org/pdf/2603.28052)- “Changing the harness around a fixed LLM can produce a 6× performance gap on the same benchmark.”", "url": "https://wpnews.pro/news/do-not-overcomplicate-agentic-harness", "canonical_source": "https://www.loadbearingtech.com/p/do-not-overcomplicate-agentic-harness", "published_at": "2026-07-06 15:28:43+00:00", "updated_at": "2026-07-28 09:35:59.360007+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "large-language-models", "ai-infrastructure"], "entities": ["OpenAI", "Replit", "Cursor", "Air Canada", "Chevrolet Tahoe"], "alternates": {"html": "https://wpnews.pro/news/do-not-overcomplicate-agentic-harness", "markdown": "https://wpnews.pro/news/do-not-overcomplicate-agentic-harness.md", "text": "https://wpnews.pro/news/do-not-overcomplicate-agentic-harness.txt", "jsonld": "https://wpnews.pro/news/do-not-overcomplicate-agentic-harness.jsonld"}}