{"slug": "i-was-optimizing-ranking-while-the-real-problem-was-selection", "title": "I Was Optimizing Ranking While the Real Problem Was Selection", "summary": "A developer spent three months optimizing a ranking algorithm for a search system, only to realize the real problem was selection: the document that actually answered the user's question was never included in the candidate set. No amount of ranking could retrieve what was excluded before ranking began. The engineer concluded they had mistaken ranking for retrieval.", "body_md": "*I mistook movement for improvement.*\n\nFor three months, I changed our ranking algorithm every two weeks.\n\nEvery change moved a few documents up or down the list.\n\nNone of them made the answers better.\n\nThe user asked:\n\n\"Why does the gateway reject service X?\"\n\nWith BM25, the top result was a glossary page defining *gateway*. Relevant by keyword. Useless by intent.\n\nWith hybrid retrieval, the glossary page dropped to third. A Jira ticket mentioning service X moved to second. It described the symptom, not the cause.\n\nWith a cross-encoder reranker, the Jira ticket became the top result. The glossary page fell to fourth. The architecture document that actually explained the exception remained at seventh.\n\nWith a different embedding model, that architecture document finally moved to fifth.\n\nI remember thinking we'd finally made progress.\n\nWe hadn't.\n\nEvery new algorithm felt like progress because **something moved**.\n\nThe problem was that **nothing important moved**.\n\nThe architecture document existed.\n\nI knew it existed.\n\nI had even written part of it.\n\nIt never appeared in the top results.\n\nNot because it was ranked poorly.\n\nBecause it was never selected.\n\nThe ranking algorithm could only sort the candidates it received.\n\nThe document that actually explained the answer never entered that candidate set.\n\nNo ranking algorithm can promote a document that was excluded before ranking even began.\n\nI had five search results on my screen.\n\nBM25 returned them in one order.\n\nThe cross-encoder returned them in another.\n\nA different embedding model shuffled them again.\n\nThey were still the same documents.\n\nNot one of them answered the question.\n\nOnly then did I ask myself:\n\nWhy am I ranking these documents at all?\n\nI stopped asking:\n\nHow do I rank these documents better?\n\nI started asking:\n\nWhy are these documents candidates in the first place?\n\nThat question led me backward.\n\nNot to another ranking algorithm.\n\nBut to every previous question I had been trying to answer.\n\nWhy didn't a larger context window help?\n\nWhy did chunking make retrieval worse?\n\nWhy did search keep finding documents but miss the explanation?\n\nI realized those weren't separate problems.\n\nThey were different symptoms of the same one.\n\nI had been optimizing the last step of retrieval while the earlier ones were quietly determining the outcome.\n\nI now think about retrieval as three different layers.\n\nFor months, I focused almost entirely on the third layer.\n\nThe first layer was already deciding whether the answer had any chance of being found.\n\nPerfect ranking can only reorder what has already been selected.\n\nIt cannot retrieve what never became a candidate.\n\nI kept polishing the order of documents that should never have been candidates.\n\n**I had mistaken ranking for retrieval.**\n\n*In the next post, I'll look at a different question: why organizations rarely lose information—they lose the connections that once made that information meaningful.*", "url": "https://wpnews.pro/news/i-was-optimizing-ranking-while-the-real-problem-was-selection", "canonical_source": "https://dev.to/valerykot/i-was-optimizing-ranking-while-the-real-problem-was-selection-3p0k", "published_at": "2026-07-24 08:00:08+00:00", "updated_at": "2026-07-24 08:04:02.409109+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-products", "developer-tools"], "entities": ["BM25"], "alternates": {"html": "https://wpnews.pro/news/i-was-optimizing-ranking-while-the-real-problem-was-selection", "markdown": "https://wpnews.pro/news/i-was-optimizing-ranking-while-the-real-problem-was-selection.md", "text": "https://wpnews.pro/news/i-was-optimizing-ranking-while-the-real-problem-was-selection.txt", "jsonld": "https://wpnews.pro/news/i-was-optimizing-ranking-while-the-real-problem-was-selection.jsonld"}}