In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshol
Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers
A study measuring order dependence in LLM scorers finds that models with equal ranking quality can produce different decisions depending on the order candidates appear in a single prompt, affecting passage reranking, response ranking and multi-document question answering. The research focuses on how score thresholds turn those order-sensitive scores into decisions, and on reducing that order dependence.
LLM scorers with identical ranking quality can still yield different decisions because their scores shift with candidate order in the prompt, so order dependence must be measured and reduced before scores are used against a decision threshold.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.