A study measuring order dependence in LLM scorers finds that models with equal ranking quality can produce different decisions depending on the order candidates appear in a single prompt, affecting passage reranking, response ranking and multi-document question answering. The research focuses on how score thresholds turn those order-sensitive scores into decisions, and on reducing that order dependence.
In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshol
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.