The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?

A new study from arXiv (2607.28641v1) introduces the Agentic Formalism Trap and the Evaluative Dissonance Index (D_E), showing that LLM-as-a-Judge systems conflate structural proceduralism with semantic truth under adversarial load. Analyzing 22,500 trajectories across GAIA, SWE-bench, and Multi-Challenge, the researchers found that a logistic meta-evaluator isolates syntactic triggers of evaluator capture (ROC-AUC 0.8779), and zero-shot transfer proves the vulnerability is domain-agnostic (mean ROC-AUC 0.7482). The study concludes that unanchored closed-loop evaluation is unstable and necessitates architecture-specific vigilance filters.

arXiv:2607.28641v1 Announce Type: new Abstract: We introduce the \textit{Agentic Formalism Trap} and the Evaluative Dissonance Index $D E$ , quantifying how LLM-as-a-Judge systems conflate structural proceduralism with semantic truth under adversarial load. Analyzing 22,500 trajectories across 3 domains GAIA, SWE-bench, Multi-Challenge , we extract a semantic taxonomy of hallucination maneuvers, validated via deterministic lexical grounding $p < 10^{-120}$ . A logistic meta-evaluator isolates the exact syntactic triggers of this evaluator capture ROC-AUC 0.8779 , while a zero-shot Leave-One-Domain-Out transfer proves the vulnerability is universally domain-agnostic mean ROC-AUC 0.7482 . Architectural profiling reveals that distinct simulated swarm topologies induce mathematically disparate semantic blind spots, proving that unanchored closed-loop evaluation is unstable, systemically divergent and necessitates architecture-specific vigilance filters.