Check twice, cut once with LLM search relevance eval

Doug Turnbull's local_llm_judge experiment on the WANDS furniture e-commerce search dataset found that forcing an LLM to pick a more relevant product yielded 75.08% precision at 100% recall over 1000 pairs, while allowing a "Neither - not confident" answer raised precision to 85.38% but cut recall to 17.10%. Turnbull added a sanity check that swaps the left- and right-hand products and re-queries the LLM to catch position bias, since the WANDS dataset provides only absolute 0-2 human relevance labels rather than direct pairwise labels.

In my previous post https://softwaredoug.com/blog/2025/01/13/llm-for-judgment-lists git repo https://github.com/softwaredoug/local-llm-judge I ask an LLM to compare the relevance of two products from the WANDS furniture e-commerce search dataset https://github.com/wayfair/WANDS . I then compare the agreement with the preference of human search raters, hoping if I got close to agreement, I might be onto creating a good pairwise LLM search relevance evaluator without so many humans. I note that when letting the LLM chicken-out and say “I dont know”, precision improves. But naturally reduces the recall - the percentage of labeled product pairs - quite a bit. So, as an example, we might start with the following, simple, forced decision, asking an LLM to tell us which chair is more relevant to the leather chairs query: System: You are a helpful assistant evaluating search relevance of furniture products. Prompt: Which of these products is more relevant to the furniture e-commerce search query: Query: leather chairs Product LHS: fashion casual work beauty salon task chair Product RHS: angelyn cotton upholstered parsons chair in gray/white Respond with just 'LHS' or 'RHS' Response: LHS If humans also say “LHS” - the salon task chair - is more relevant, then this is a win Of course, in the WANDS case, as with many datasets we don’t have direct pairwise labels. Instead we have absolute labels. We’re really just comparing which human label is higher. That is, on a 0-2 scale, LHS might be labeled by humans a 2 for completely relevant, RHS a 0 for absolutely horrid result for leather chair , etc. So we say humans see LHS as more relevant. The above prompt, over 1000 pairs, gives: poetry run python -m local llm judge.main --verbose --eval-fn name ... Precision: 75.08% | Recall: 100% N=1000 But we can also allow the LLM to NOT label a pair due to lack of information: Neither product is more relevant to the query, unless given compelling evidence. Which of these product names if either is more relevant to the furniture e-commerce search query: Query: leather chairs Product LHS name: fashion casual work beauty salon task chair remaining product attributes omited Or Product RHS name: angelyn cotton upholstered parsons chair in gray/white remaining product attributes omited Or Neither / Need more product attributes Only respond 'LHS' or 'RHS' if you are confident in your decision Respond with just 'LHS - I am confident', 'RHS - I am confident', or 'Neither - not confident' with no other text. Respond 'Neither' if not enough evidence. Response: Neither - not confident Doing this 1000 times gives: poetry run python -m local llm judge.main --verbose --eval-fn name allow neither ... Precision: 85.38% | Recall: 17.10% N=1000 Out of the 17.10% labeled, 85.38% agreed with humans. Adding an important sanity check - checking both ways Turns out, it’s important to check twice. Put the LHS product on the RHS, and double check that you the same result. In other words ask first: is ‘salon chair’ more relevant than ‘parsons chair’ for query leather chair ? Then reset, swap, and ask that is parsons chair more relevant than salon chair ? We can then account for any biases an LLM might have to attending to the first or second product listed. We wrap the prompt eval fn below in a python function that double checks: python def check both ways query, product lhs, product rhs, eval fn : eval fn is a function wrapping the prompt """Get pairwise preference from LLM, but double check by swapping LHS and RHS to confirm consistent result.""" decision1 = eval fn query, product lhs, product rhs This just calls the LLM with the prompt LHS to RHS decision2 = eval fn query, product rhs, product lhs Now check RHS first... if decision1 == 'LHS' and decision2 == 'RHS': return 'LHS' elif decision1 == 'RHS' and decision2 == 'LHS': return 'RHS' return 'Neither' Adding a command line switch check twice: bash $ poetry run python -m local llm judge.main --verbose --eval-fn name --check-both-ways This dramatically improves performance. When the above is run, precision goes up appreciability with a high degree of product coverage: Precision: 87.99% | Recall: 65.80% N=1000 Even more interesting, when combining with allowing ‘I dont know’, we get the highest precision. Though with a significant reduction in recall: bash $ poetry run python -m local llm judge.main --verbose --eval-fn name allow neither --check-both-ways ... Precision: 90.76% | Recall: 11.90% N=1000 So to summarize where my efforts stand, just using product name, we can build this confusion matrix. Showing Precision / Recall for each approach: | | Dont check | --check-both-ways | |---|---|---| | Force | 75.08% / 100% | 87.99% / 65 % | | Allow Neither | 85.38% / 17.10% | 90.76% / 11.90% | So depending on your use case, you should pick the appropriate solution. Want a high degree of coverage and can tolerate a lot of mistakes. Use the force / dont double check. Want to tolerate very few mistakes and only flag results with real issues? Double check and allow the LLM to say it doesn’t know ie ‘allow neither’ . I tried creating evaluators looking only at a few other fields. Here’s showing the LLM only product “class” as in classification . Example product classes: Product LHS class: Beds Product RHS class: Kids Beds Product LHS class: Coffee & Cocktail Tables Product RHS class: Outdoor Fireplaces Run with these 4 variants: poetry run python -m local llm judge.main --verbose --eval-fn classs poetry run python -m local llm judge.main --verbose --eval-fn class allow neither poetry run python -m local llm judge.main --verbose --eval-fn classs --check-both-ways poetry run python -m local llm judge.main --verbose --eval-fn class allow neither --check-both-ways | | Dont check | --check-both-ways | |---|---|---| | Force | 70.5% / 100% | 87.76% / 58.0% | | Allow Neither | 87.01% / 17.70% | 84.47% / 10.3% | And repeating for the product’s full categorization hierarchy like Outdoor furniture Seating Adirondak Chairs... | | Dont check | --check-both-ways | |---|---|---| | Force | 74.6% / 100% | 86.1% / 69.70% | | Allow Neither | 85.71% / 18.20% | 89.91% / 10.8% | Finally, noisiest of them all, product description: | | Dont check | --check-both-ways | |---|---|---| | Force | 70.31% / 98.70% | 76.58% / 72.60% | | Allow Neither | 79.21% / 10.10% | 83.02% / 5.3% | Note with product description even ‘forcing’ a decision the LLM still sometimes said it couldn’t tell, completely unprompted Enjoy softwaredoug in training course form Starting in October Signup here - https://maven.com/softwaredoug/cheat-at-search https://maven.com/softwaredoug/cheat-at-search I hope you join me at Cheat at Search with Agents https://maven.com/softwaredoug/cheat-at-search to learn to use agents in search, build better RAG, and use LLMs in query understanding.