{"slug": "stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm", "title": "Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges", "summary": "A new study reveals that LLM judges, widely used to evaluate AI outputs, can be manipulated after making an initial decision through targeted conversation, overturning stable judgments and shifting benchmark rankings. Researchers found that while these judges remain stable under neutral reevaluation, they become substantially reversible under post-decision challenge, with authority framing proving especially destabilizing and often producing low-overlap justifications. The findings introduce the Evaluation Robustness Score (ERS) to quantify this vulnerability, identifying post-decision interaction as a distinct failure mode that degrades agreement with human preferences.", "body_md": "arXiv:2606.05384v1 Announce Type: new\nAbstract: LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipelines typically assume that judgments are stable properties of fixed inputs. We show that this assumption does not hold under interaction. We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge after an initial decision has been made. Across controlled experiments on MT-Bench and AlpacaEval, we find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge. An anti-baseline challenge protocol shows that stable judgments can be overturned through motivated interaction, while a counterbalanced target-validation protocol separates this reversibility from net target-directed steering. These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence. Authority framing is especially destabilizing, and revised judgments are often accompanied by low-overlap justifications, suggesting post hoc rationalization rather than reliable error correction. We introduce the Evaluation Robustness Score (ERS) to quantify interactional robustness by combining reversal susceptibility with counterbalanced directional effects. Our findings identify post-decision interaction as a distinct failure mode for LLM-as-judge evaluation and motivate evaluation protocols that measure not only static agreement, but robustness under challenge.", "url": "https://wpnews.pro/news/stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm", "canonical_source": "https://arxiv.org/abs/2606.05384", "published_at": "2026-06-06 04:00:00+00:00", "updated_at": "2026-06-06 04:17:28.154031+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-ethics", "ai-research", "natural-language-processing"], "entities": ["MT-Bench", "AlpacaEval", "LLM", "Evaluation Robustness Score"], "alternates": {"html": "https://wpnews.pro/news/stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm", "markdown": "https://wpnews.pro/news/stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm.md", "text": "https://wpnews.pro/news/stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm.txt", "jsonld": "https://wpnews.pro/news/stability-vs-manipulability-evaluating-robustness-under-post-decision-in-llm.jsonld"}}