Prefer synthetic traffic over LLM-as-a-judge Search consultant Doug Turnbull argues that LLM-as-a-judge evaluation is a half-measure that keeps traditional metrics like NDCG while ignoring how much a search UI's layout shapes quality, and proposes instead having coding agents drive a reimplemented search interface to simulate user traffic. Turnbull, who recreated Alamo Drafthouse's menu search and often vibe-codes facsimiles of client search UIs, runs what he calls an "agentic A/B on my laptop" to extend judgment-list evaluation rather than replace human interaction. The approach matters because ranking wins measured by NDCG frequently fail to translate into A/B test success when grid, vertical and mixed-media interfaces are treated as interchangeable. ‘LLM-as-a-judge’ bolts LLMs into traditional search evaluation. With the same eval headaches, only faster. LLM judges take over search labeling from humans. They label results as relevant / not for specific queries. Then we continue on, business as usual, with the traditional search metrics NDCG, etc . It feels like a half-measure. We’re doing to search evaluation what Github copilot did to AI coding. A way to bolt on AI into an existing workflow, not completely rethink it. Crucially, we ignore the user interface’s foundational importance to search quality. As we’ll discuss, ranking evaluation assumes we can cleanly separate ranking from UI. Which leads to consistently disappointing results where ranking wins don’t translate to A/B test success. I’m going to discuss an alternative philosophy. Moving away LLM as a judge and towards tasking agents driving the search UI directly. Not as a replacement for human interactions. But as a better way to leverage LLM feedback. The old search eval regime The typical search evaluation regime labels some results relevant / not in a judgment list. IE for a query shoes you have: | query | document | grade 1-5 | |---|---|---| | shoe | red shoe | 5 | | shoe | blue socks | 3 | | shoe | slippers | 4 | | shoe | trousers | 1 | We label using humans, LLMs, clicks, whatever. With LLMs, we label results to fit some rubric of what ‘relevant’ means. Of course, with LLM judges, we employ one of many https://blitzrank.ai/ methods https://arxiv.org/html/2406.06519v1 to evaluate results. Regardless, with labels, later, our retrieval algorithm fetches some search results for shoes: | query | document | grade 1-5 | |---|---|---| | shoe | trousers | 1 | | shoe | slippers | 4 | | shoe | blue socks | 3 | | shoe | red shoe | 5 | We evaluate this ranked set of results with the usual suspects NDCG https://en.wikipedia.org/wiki/Discounted cumulative gain , etc . We see the grade=1 towards the top, and a grade=5 towards the bottom. Seeing a lower NDCG, we dismiss this as garbage search results. But search UIs vary tremendously. We’ve got grids like Best Buy: Or vertical, mobile listings, with only 1-2 items above the fold, like Kroger: Or weird combinations of media types, like Reddit: Putting a grade=1 in a grid’s first slot won’t completely tank the experience. Putting irrelevant results in the first slot of a vertical search UI creates serious problems. That’s why typically the answer goes “pick an evaluation metric that approximates the UI”. A metric like Precision https://en.wikipedia.org/wiki/Precision and recall doesn’t take into account position, maybe it’s appropriate for the grid? But for the scrollable search UI, we might choose NDCG. Because it DOES weigh the importance of the top position higher. Whatever metric you pick, it’s like assuming the UI is a spherical chicken in a vacuum https://www.youtube.com/watch?v=TvpYIQI2K7g . NDCG seems to stand in for ‘some kind of ranky-UI’. A nice theory approximating a messy reality. I’ve worked in search ranking. And it feels like we’ve danced around this problem. Search UIs only get less predictable. They’re not beautifully spherical chickens. But a diverse set of strangely shaped chickens