{"slug": "set-shifting-behavioral-test-for-harnessed-agents", "title": "Set-shifting Behavioral Test for Harnessed Agents", "summary": "Researchers introduced a set-shifting behavioral test for LLM agents, borrowing from cognitive psychology to measure how agents adapt when a reliable tool silently changes. The benchmark uses redundant tool-skill libraries with hidden reliability shifts, finding that agents settle into small routines and exhibit distinct failure modes across models. Set framing—whether tools are presented as competing or complementary—also alters routing dynamics.", "body_md": "arXiv:2607.13396v1 Announce Type: new\nAbstract: What happens to an LLM agent's tool choice when the reliable tool silently changes within an ongoing session? We borrow set-shifting from cognitive psychology to study how well agents adapt to hidden reliability shifts. Our benchmark mounts tool-skill libraries with redundancies, where many tools solve the same task but differ in hidden reliability. In our evaluation framework, a branched schedule shifts the reliable tool group at hidden boundaries and pairs every shift with a no-shift control. We find that agents, by default, settle on a small recurring routine within a few turns of each boundary, with call shares concentrating on a few discrete values after each reliability shift. We score the set-shifting accuracy for each agent trajectory: the joint probability of routing to the target tool group in every post-shift window. We test open-weight LLMs in an open-source agentic harness and find qualitatively distinct failure modes across the same set of routines. We also find that set framing, how the toolset presents the alternatives as competing or complementary, shifts the routing dynamics.", "url": "https://wpnews.pro/news/set-shifting-behavioral-test-for-harnessed-agents", "canonical_source": "https://arxiv.org/abs/2607.13396", "published_at": "2026-07-16 04:00:00+00:00", "updated_at": "2026-07-16 04:31:25.736700+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/set-shifting-behavioral-test-for-harnessed-agents", "markdown": "https://wpnews.pro/news/set-shifting-behavioral-test-for-harnessed-agents.md", "text": "https://wpnews.pro/news/set-shifting-behavioral-test-for-harnessed-agents.txt", "jsonld": "https://wpnews.pro/news/set-shifting-behavioral-test-for-harnessed-agents.jsonld"}}