Recursive Semantic Divergence for LLM Agent Consistency

A new arXiv paper (2610.33012v1) introduces Recursive Semantic Divergence (RSD), an unsupervised trajectory-level consistency metric for LLM agents that adapts bisimulation metrics from reinforcement learning and uses natural language inference to detect logical inconsistency between independent rollouts. Used as a reward signal for fine-tuning, RSD cut variation in task progress from 132% to 91% on tau2-bench and from 90% to 44% on toolsandbox while improving performance across benchmarks and model scales. The authors report that training with RSD as an additional reward signal matches supervised fine-tuning performance with substantially lower variance and without ground-truth labels.

arXiv:2610.33012v1 Announce Type: new Abstract: Large Language Model LLM agents with tool use exhibit inconsistent behavior across independent runs from identical states, calling different tools, producing contradictory outputs, or pursuing divergent strategies. This variability compounds over time and undermines deployment reliability. Existing evaluation metrics primarily measure task correctness via completion-based scores and do not capture consistency in agent behavior across executions. Separately, agent behavior is analyzed either at the token level using probability distributions or at the single-response level using semantic clustering, but neither captures how inconsistency propagates across future turns of a trajectory. We introduce Recursive Semantic Divergence RSD , an unsupervised trajectory-level consistency metric adapted from bisimulation metrics in reinforcement learning. RSD measures the expected cumulative semantic disagreement between two independent rollouts of the same agent from the same state, using natural language inference to detect logical inconsistency. It is defined recursively over future states and approximated by a neural network trained on independent rollouts. The learned metric estimates future divergence from the current state, providing a consistency score at each turn without requiring the full trajectory to complete. When used as a reward signal for fine-tuning, RSD reduces the variation in task progress from 132% to 91% on tau2-bench and from 90% to 44% on toolsandbox, while improving performance across benchmarks and model scales. Training with RSD as an additional reward signal matches supervised fine-tuning performance with substantially lower variance, without requiring ground-truth labels.