{"slug": "search-has-its-own-bitter-lesson", "title": "Search has its own bitter lesson", "summary": "A new 'bitter lesson' in search technology reveals that algorithms matter less than incentivizing content creators to optimize for a search engine, as demonstrated by Google's PageRank and now by coding agents like Claude Code that reshape developer documentation practices. The key to effective retrieval, including RAG, lies not in sophisticated question-answering systems but in writing better, context-rich answers that LLMs can leverage.", "body_md": "An agent with grep does quite well at search.\n\nIt’s shocking to search technologists. We want to cocoon the problem in technology. We care about algorithms, knowledge graphs, ranking: ever-and-ever smarter retrieval.\n\nIt turns out, though, you can play a bit of a trick. If you convince everyone to optimize content for your search engine, you’ll have built the best search engine.\n\nThere’s Sutton’s [bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) about unleashing raw compute on a problem. But there’s a different bitter lesson in search: algorithms matter less than convincing the world to optimize for your search engine.\n\nSure, technology acts as the trigger. In Web Search, Google shifted away from text relevance to prioritize authoritativeness (PageRank). Their success told the market: create content others want to link to. They reshaped the Web for good (or ill) around that incentive.\n\nGoogle crowdsourced search quality to the Web. Claude Code outsources it to every developer.\n\nDevelopers depend on agents to find documentation on the file system. We like that dopamine hit. That thrill of Claude Code knowing exactly how to reason about our code.\n\nWhen agents make boneheaded mistakes, we stress. We frantically update documentation, we link to it from AGENTS.md, we add a new document, we place it in a useful location. We do anything - absolutely anything - to ensure agents know how to find and understand documentation.\n\nJust as the SEO expert reads the tea leaves of Google’s algorithm. The software developer writes to the LLM to make documentation findable + useful.\n\nA search system with these incentives overcomes technologically sophisticated systems. For example, classic RAG rests on a foundation of technically sophisticated question answering. Answers often look like conversationally valid responses, but with little linkage back to questions it might answer:\n\nQuestion:\n\nWhat is a runner test?\n\nAnswer:\n\nVerifies that small-dollar deposits and withdrawals can be executed against real test bank accounts during account reconciliation.\n\nYes that’s literally an answer to the question.\n\nBut I wouldn’t consider this “good documentation”. It’s not anchored to any concepts from the question. It’s just an out of context bit of text.\n\nIf you wrote this for a coding agent, you’d probably organize the chunk to give hints about why it’s useful.\n\ndocs/tests/runner.md\n\n# Runner Tests\n\n## Purpose\n\nWe need to ensure the system runs end to end\n\n## Usage\n\nUse this style of test sparingly. Unit and standard e2e tests should usually be preferred.\n\n## What is it\n\nA runner test actually interacts with real-life test bank accounts to ensure we can correctly deposit / withdraw money and that the full banking system works.\n\nThat’s a far more descriptive piece of content to an LLM. Being encouraged to write *for the LLM itself* makes all the difference.\n\n(As an aside, LLMs see candidate *answers* in their context before the user’s question - so just treating RAG as question answering doesn’t work).\n\nWhat often matters isn’t algorithms. It’s programming the content author to create useful LLM context. That moves content discovery from random, disconnected factoids to documentation connecting information to problems to solve.\n\nYou learn that the trick to RAG isn’t creating good question answering systems. It’s writing better answers.\n\n## The enterprise search morass… is over?\n\nCompare how coding agents program us to the eternal nightmare of your company’s wiki.\n\nWhen people write in the wiki, they do it for themselves. Nobody is incentivized to make information findable by anyone. No author sits obsessing how to ‘SEO’ the content for Confluence search.\n\nIf someone is ‘lucky’ enough to find your article, they can barely understand it. It’s written in your team’s jargon. And it’s probably out of date anyways.\n\nFor these reasons, I always tell people they’d do better to write a public blog article. Blogs assume zero context. They’re clearly timestamped as an artifact of a moment in time. To avoid shame or promote your career, you’re incentivized to create something useful, interesting, and findable.\n\nWriting for a coding agent has the same property. You’re writing documentation for a targeted reader. You care to SEO your content for Claude Code + grep.\n\nEnterprise search fails because there’s no feedback loop between authorship and findability. Coding agents solve that, finally giving some hope to organizing company information into findable knowledge.\n\nThe real outcome of coding agents isn’t just code generation. It’s closing the feedback loop between company context and the next piece of work being done. That old fable of Enterprise Search - improving workforce productivity by leveraging existing insight - can finally become true.\n\n## Feedback to content authors drives user relevance\n\nLook around your system. Who creates the content? What’s in it for them to make content findable?\n\nA recurring theme in my consulting looks something like this - a job search company notices spammy behavior from job posters. They keyword stuff every programming language in some hidden text element. The usual reaction: that’s spammy and we should ban them.\n\nThat’s too black and white. You have content creators willing to put in work. Shift their black-hat SEO over to an acceptable white-hat SEO. Document what’s acceptable. What’s considered malicious.\n\nFocus on feedback. Make the content-optimization path rewarding. Give creators feedback+guidance during creation. LLMs help detect spam in real-time. They push content creators to author what’s useful not malicious.\n\nGive content creators feedback. Once you’ve identified what ‘good’ is, give those content authors dopamine hits. Tell content creators what queries perform well for the content. Make it real-time, sticky, fun. Incentivize and reward behaviors you want - and punish those you don’t.\n\nSteer content creators towards the content you want. Don’t hope algorithms can overcome drivel and slop.\n\nYour search engine isn’t a bundle of algorithms ranking for users. It’s a relationship between authors and users. That bitter lesson puts grep above the smartest reranker and embedding model. Forget it at your peril.\n\n### Enjoy softwaredoug in training course form!\n\n#### Starting May 18!\n\nSignup here -[http://maven.com/softwaredoug/cheat-at-search](http://maven.com/softwaredoug/cheat-at-search)\n\n[I hope you join me at](https://maven.com/softwaredoug/cheat-at-search)\n\n[Cheat at Search with Agents](https://maven.com/softwaredoug/cheat-at-search)to learn use agents in search. build better RAG and use LLMs in query understanding.", "url": "https://wpnews.pro/news/search-has-its-own-bitter-lesson", "canonical_source": "http://softwaredoug.com/blog/2026/06/12/incentives-in-search.html", "published_at": "2026-06-12 00:00:00+00:00", "updated_at": "2026-07-04 14:54:51.864435+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "developer-tools"], "entities": ["Google", "Claude Code", "PageRank", "Sutton"], "alternates": {"html": "https://wpnews.pro/news/search-has-its-own-bitter-lesson", "markdown": "https://wpnews.pro/news/search-has-its-own-bitter-lesson.md", "text": "https://wpnews.pro/news/search-has-its-own-bitter-lesson.txt", "jsonld": "https://wpnews.pro/news/search-has-its-own-bitter-lesson.jsonld"}}