If your SEO team is handing more of its work to AI agents, the metric you reward those agents for will matter more than the model you choose. Recent pieces from MIT and Stanford, read side-by-side, point to that conclusion, and each comes with a fix you can drop into next quarter’s plan. The first is an interview that Joshua Miller of The Boston Globe ran in his Camberville newsletter on September 17. His guest was Dylan Hadfield-Menell, an associate professor of electrical engineering and computer science at MIT on the faculty of artificial intelligence and decision-making. Hadfield-Menell studies how goals get set for AI systems and how that process goes wrong. He opened with an example every SEO will recognize.
The Vacuum That Fed Itself #
Researchers once trained a robot vacuum with reinforcement learning, rewarding it every time it picked up dirt. The vacuum learned to pick up dirt, dump it back on the floor, and pick it up again. It hit the target and defeated the purpose.
Hadfield-Menell connects that story to a 1970s management paper titled “On the Folly of Rewarding A, While Hoping for B.” Its classic case is the university professor who gets promoted for publishing research while being expected to teach. Pay for one behavior, and you get that behavior, whatever you were hoping for.
What has changed, he says, is scale. Since early 2025, developers have applied reinforcement learning at much larger volume on top of language models, and it strengthens some behaviors nobody wants. He pointed to a recent incident involving OpenAI systems and Hugging Face, where models that judged a task too hard went looking for ways to cheat the test. He compared it to breaking into a professor’s office to steal the exam.
His worry is not that machines wake up with goals of their own. Systems are handed a goal, adopt subgoals along the way, and keep pushing toward completion in a way he called “sticky.”
My view is that SEO is the profession best placed to understand this problem and the slowest to admit it applies to us. We have spent more than 20 years optimizing proxies. Rankings, traffic, domain scores, and now AI visibility scores all stand in for a business result that nobody can measure directly. A human team games a proxy slowly and with some hesitation. An agent does it faster and without any.
The Scoreboard Is Shakier Than Vendors Admit #
Stanford’s 2026 AI Index shows why leaning on published scores is risky. The report says AI keeps improving quickly. On SWE-bench Verified, a coding benchmark, performance rose from 60% to near 100% in a single year, and 88% of organizations now use AI.
The same report, in its technical performance chapter, cites a review that found invalid-question rates on popular benchmarks ranging from 2% on MMLU Math to 42% on GSM8K. It also notes research suggesting that a model’s standing on the Arena leaderboard may partly reflect adaptation to the platform rather than general capability.
Michelle Kim of MIT Technology Review summarized the report in April. She adds that models trained on benchmark test data can learn to score well without getting smarter, and that the top models now sit very close together and compete on cost, reliability, and real-world usefulness. Yolanda Gil, a University of Southern California computer scientist who coauthored the report, told Kim that when a company leaves out its results on certain benchmarks, particularly the responsible-AI ones, the omission “maybe says something.”
That should change how an SEO team shops for tools. If the leading models sit within a few points of each other, and the scores themselves can be flawed or gamed, a vendor’s benchmark slide tells you little about how the product will treat your pages, your queries, and your clients. I would trust one test on my own site over any leaderboard.
See also: The 4-Step Test That Catches AI Errors Before They Shape Your Strategy
Where The Returns Actually Come From #
MIT Sloan’s Betsy Vereckey reported in August on the question that follows, which is what separates companies that profit from AI from those that don’t. George Westerman, a senior lecturer at MIT Sloan and a digital fellow at the MIT Initiative on the Digital Economy, says the answer is not better algorithms. The winners redesigned how work gets done. At the MIT Enterprise AI Forum in May, he told the audience that technology delivers little until the business itself operates differently.
He also put the share of AI pilots that never scale at somewhere between 70% and 95%, a range the Sloan article attributes to studies without naming them. Pilots are easy to launch and hard to spread.
Westerman’s sharpest test for leaders is about governance. “Is your governance more the steering wheel or is it more the brakes?” he asked. HCA Healthcare shows what the steering-wheel version looks like. A committee reviews the risks, business case, and feasibility of every AI use case, then asks its questions again before a pilot at a small number of hospitals and again before the project scales. It also checks periodically that its models are still holding up. The risk questions point the team toward what to investigate rather than stopping the work.
Marketing appears in Westerman’s case studies too. Dentsu Creative has pushed AI across planning, creative, market research, and campaign work.
I suspect a lot of SEO teams running AI pilots are headed for that 70% to 95%. A pilot that doesn’t change the brief, the review step, or the reporting is a tool trial, whatever the slide deck calls it.
See also: Why 88% Of Companies Are Using AI Wrong: The System-Building Gap
How To Apply This To Your SEO Strategy #
Four moves follow from the three pieces.
Pair every proxy with an outcome the agent can’t touch. List the metrics your AI-assisted workflows are judged on, from pages published to schema deployed to brand mentions in AI answers. Then attach a second measure that a human owns, such as qualified leads, pipeline, or branded search demand. I like Citation Share of Voice, but it is a proxy. If a content agent is judged on how often your brand shows up in AI answers, expect it to find the cheapest route there. Check a sample of those citations by hand each month and see whether they send anyone to a page that converts.
Test tools on your own pages. Pull a set of real queries from Search Console, run each candidate tool against your own content, and have an editor grade the results without knowing which tool produced them. Repeat it every quarter, because the leaderboards will move and you won’t know why.
Gate the agents the way HCA gates use cases. Add review points before design, before the pilot, and before scale. Pilot on one directory or one language market, and decide in advance what result ends the pilot. I would also keep agent permissions narrow, so drafting doesn’t quietly turn into publishing or editing templates. Westerman’s advice is to change or drop a project that isn’t producing the results you expected.
Rewrite one workflow, not the tool stack. Before buying anything, name the step in your process that will be different after the pilot, whether that is briefing, QA, or reporting, and say who loses a task because of it. Then tell the team what changes and what training comes with it. Westerman notes that silence lets people imagine the worst.
I don’t think the next model release will decide who wins in AI search. The metric you use to judge your agents will, because the agents will find it before you do. Choose one you’d be glad to see them hit.
More Resources:
- [Google Answers Question About SEO For AI Agents](https://www.searchenginejournal.com/google-seo-for-ai-agents/580589/)
- [Well-Known SEO Explains Why AI Agents Are Coming For You & What To Do Now](https://www.searchenginejournal.com/well-known-seo-explains-why-ai-agents-are-coming-for-you-what-to-do-now/562964/)
- [Why Your SEO KPIs Are Failing Your Business (And How To Fix Them)](https://www.searchenginejournal.com/why-your-seo-kpis-are-failing-your-business-and-how-to-fix-them/564769/)
Featured Image: Fardived/Shutterstock