{"slug": "most-self-improving-ai-agents-don-t-actually-improve", "title": "Most \"self-improving\" AI agents don't actually improve", "summary": "Only 2 out of 9 self-improving AI agents running in production actually have a verifier grounded in real-world data, according to an analysis by an unnamed expert. The verifier is the key bottleneck, as most organizations try to sidestep building one, which requires a steady stream of real-world data and expert feedback. Examples of successful loops include Replit's product development agent and an OpenAI-built Tax AI agent.", "body_md": "# Most \"self-improving\" AI agents don't actually improve\n\n### Out of 9 running in production, only 2 had a key element: verifier grounded in the real world\n\nExec summary\n\nDespite the hype, public examples of true self-improving loops are few and far between.\n\nA self-improving loop is only as good as its verifier, and a verifier is only as good as the real-world data grounding it.\n\nBuilding verifiers is hard, slow, and complicated, and where most of the competitive opportunity is today. Most AI engineering is trying to side-step them.\n\nBefore you sponsor a project that builds one, align the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?\n\nLoops are having a moment. The poor souls who take it upon themselves to be their organization’s Most Plugged-In AI Person were in for a treat over the last few weeks: Boris Cherny, (creator of Claude Code), shared his [Steps of AI Adoption](https://claude.ai/code/artifact/bfdfaef9-bc62-4dfe-ba9e-c58a26c9accf), a de facto blueprint of what it takes to move past the bottlenecks in automating work; co-founder of Arize AI Aparna Dhinakaran [broke down the theory of loops](https://x.com/aparnadhinak/status/2073492320159510869). Gergely Orosz of The Pragmatic Engineer quickly followed with a piece [rightfully dispelling some of the hype](https://newsletter.pragmaticengineer.com/p/what-is-loop-engineering?utm_source=loadbearingtechsubstack).\n\nNo doubt many organizations are experimenting with self-improvement. But if you look for documented cases of self-improving loops **in production**, the list gets short quickly; among 9 use cases I found and looked into, only two disclose enough to verifiably call the loop “self-improving”.\n\n**Verifier is key**\n\nIn its simplest form, a loop has three components:\n\nThe execution loop, which is whatever the automation actually does; say, monitors the bug queue and fixes them;\n\nThe verifier - checks the output of the execution loop against some standard you established (if in your mind verifier is served well by golden-data-evals alone - please read along);\n\nFeedback - if the verifier flags an issue with the execution loop, the context of the issue is fed back into the execution loops for another spin.\n\nAnd it’s in the verifier the real bottleneck of “self-improving” lives.\n\n## Why verifiers are expensive and difficult\n\nVerifier, despite what seems to be an increasing trend in the industry, must be grounded in a steady trickle of real-world data; if it isn’t, it can (and will) optimize in a random direction.\n\nFor deploying code, this “steady stream of real-world data” is relatively straightforward, even if it requires some creative stitching: tests can pass or fail, production telemetry can catch anomalies, code history can be audited, A/B tests can run, and user feedback and bugs queue can be monitored.\n\nAs you move further outside of code and into knowledge work, figuring out the “steady trickle of real-world data” needs some creativity.\n\nThis need for creativity is why I like to see how various businesses have tackled the problem, but evidence on in-production loops is pretty thin. Kudos to the folks behind the two examples I break down below: [Replit’s product development agent](https://replit.com/blog/evaluating-and-improving-agent-at-scale) and Tax AI agent, built by OpenAI’s FDEs (*PS. are FDEs a revival of 2010s implementation consulting?*).\n\n**Agent drafts, human ships: current state of self-improving loops**\n\n[Exhibit](https://replit.com/blog/evaluating-and-improving-agent-at-scale) 1. Replit’s self-improving coding agent\n\nThe use case for Replit’s agent is straightforward: identify product issues, write the code and submit the build. From here, verifier takes over, including Replit’s benchmark built specifically for vibe-coding: it simulates the app and user’s clicks within it; if passed, the proposed change runs through A/B testing, and results are sent to a human for review to ship, iterate, or drop.\n\nFor me, an even more interesting element of Replit’s loop is Telescope - their proprietary traces clustering tool[1](#footnote-1), which allows to analyze traces at scale without unnecessarily exposing user data to random eyeballs. [Plus, it looks cool.](https://arxiv.org/pdf/2412.13678)\n\n[Exhibit](https://openai.com/index/building-self-improving-tax-agents-with-codex/) 2. Tax AI agent, built by OpenAI x Thrive\n\nTax AI set out to tackle the ambiguous and spottily-documented process of tax preparation. Since wading through hundreds of documents to prepare a draft is the most labor-intensive part of the CPA’s job, the appetite for automation was high enough to dedicate the time to weeks of painful - but effective - mechanical turking.\n\nWhat I delightfully noted in OpenAI’s use case is that the agent **improved** **drastically** post-launch[2](#footnote-2). The percentage of drafts that were at least 75% correct went from roughly a quarter at launch to 86% - vast majority - at six weeks. The result comes from FDEs ~~pulling teeth~~ asking CPAs to explain each decision where their judgment differed from the agent.\n\nThink about this process: one tax draft has dozens - or hundreds - of fields, each requiring data and judgement - sometimes formal, sometimes not. Imagine those six weeks and the amount of good will required on both sides, the FDEs sending draft after draft to the CPAs, asking them to explain why and how the agent is wrong, field-by-field. Slow, complicated, and expensive, and inevitable in navigating the [muck layer of knowledge work](https://loadbearingtech.substack.com/p/every-emerging-tech-has-to-yield).\n\nThis is the exact challenge with self-improving agents: is it possible to make agent’s failures visible and understandable? Is it **feasible** to get the context from the experts on enough of the failures to meaningfully improve the agent? This time and buy-in might be one of the biggest bottlenecks in AI adoption today.\n\nIf your team has solved this creatively, tell me more. If your team is struggling with figuring this out, also reach out, I’d love to workshop some solutions.\n\n## What about automated evals?\n\nInevitably, someone will say: we already have verifiers - we have an automated eval system, thank you very much, all taken care of.\n\nAuto-evals are sort of a trojan horse of agent development - they are cheap, fast, simple, privacy-preserving, loops can spin indefinitely without being slowed down by humans - tempting to set them up and call it a day.\n\nHamel Husain, an ML engineer known for his work on LLM evals, recently ran an open test that proved autoevaluators did [relatively well](https://parlance-labs.com/blog/posts/auto-evals/?ck_subscriber_id=3951136500) on known criteria. But because of criteria drift, it’s in the “undefined” where the customers weep and dollars leak:\n\nThe premise of criteria drift is that you don’t* *know how to evaluate an agent until your human eyeballs observe it interacting with the real world.\n\nEven if you assume you *do *know all possible failure modes of your agent, there’s the increasingly ripe problem of reward hacking, where an agent optimizes for a score but not the intended behavior - for example, if rated on politeness, the agent might hang up on an angry customer; if rated on sales, the agent might give 99% discounts, or close $0 sales, or offer hallucinated perks - all of which you *can* catch, but not until *you know* it can happen.\n\nWhich needs real-world input.\n\nAutoevals are one way of shooting yourself in the foot and sidestepping the verifier problem instead of solving it; other ways include: LLM-as-judge, synthetic data, “we’ll add evals later”, and golden data that stays static.\n\nDon’t let the hype tell you that everyone’s automated their business with loops already. Before you sponsor a self-improving loop project, make sure you understand the messy middle between the launch and results, and that you’ve aligned the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?\n\nThat is, only if you have good verifiers.\n\nI write for people betting careers and businesses on AI from the perspective of having shipped across three waves of emerging tech in startups and Big Tech - if that’s you, subscribe to get your weekly under-5-minutes-to-read-weekly-email.\n\nDisclaimers: opinions my own. Data for this piece was sourced from public sources and private AI communities.\n\n[1](#footnote-anchor-1)\n\nBuilt based on Anthropic’s [Clio paper](https://arxiv.org/pdf/2412.13678), an interesting solution for companies (rightfully) iffy about giving their development teams broad access to traces.\n\n[2](#footnote-anchor-2)\n\nThere’s a lesson here for any exec sponsoring an agentic team or builder working on self-improving loops: the launch result is a start, not the end. The only numbers that matters is how quickly you get to a launch, and how quickly you improve from there. The results at launch are nearly irrelevant.", "url": "https://wpnews.pro/news/most-self-improving-ai-agents-don-t-actually-improve", "canonical_source": "https://www.loadbearingtech.com/p/self-improving-agent-loops-verifier", "published_at": "2026-07-22 20:03:04+00:00", "updated_at": "2026-07-28 09:35:40.579159+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-products", "ai-tools"], "entities": ["Replit", "OpenAI", "Boris Cherny", "Aparna Dhinakaran", "Arize AI", "Gergely Orosz", "The Pragmatic Engineer", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/most-self-improving-ai-agents-don-t-actually-improve", "markdown": "https://wpnews.pro/news/most-self-improving-ai-agents-don-t-actually-improve.md", "text": "https://wpnews.pro/news/most-self-improving-ai-agents-don-t-actually-improve.txt", "jsonld": "https://wpnews.pro/news/most-self-improving-ai-agents-don-t-actually-improve.jsonld"}}