{"slug": "openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios", "title": "OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios", "summary": "OpenART, a new agent safety evaluation framework, red-teams stateful AI agents across more than 10,000 evolving environment scenarios spanning 50 domains, with a median of 97 tool calls per scenario. The framework reports a pooled strict Attack Success Rate of 85.0% across 75 agent-model configurations, highlighting that safety failures can emerge from trajectory-level modifications to workspace data, permissions, memory, and plans, rather than isolated prompts.", "body_md": "*This is a Plain English Papers summary of a research paper called OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.*\n\nOpenART evaluates agent safety across more than 10,000 validated stateful scenarios spanning 50 domains and requiring a median of 97 tool calls. Its central claim is that safety failures can emerge from trajectories in which workspace data, permissions, memory, and plans are repeatedly modified, rather than from isolated prompts alone.\n\nThe arena keeps each benign task objective and hidden safety contract fixed while changing only the target-visible environment state. This design targets delayed failures that static benchmarks can miss: an early authorized mutation may influence later decisions, expose protected resources, or produce unsafe output many steps after the original change. OpenART extends the broader idea of [agent safety evaluation](https://aimodels.fyi/papers/arxiv/openagentsafety-comprehensive-framework-evaluating-real-world-ai?utm_source=devto&utm_medium=referral&utm_campaign=arxiv_papers) by making persistent environment state the object that evolves during testing.\n\nOpenART reports a pooled strict Attack Success Rate of 85.0% across 75 agent-model configurations. Strict success requires both the deterministic evaluator and a GLM-5.2 judge to identify the attack condition, so disagreements count as failures rather than being treated as partial evidence....", "url": "https://wpnews.pro/news/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios", "canonical_source": "https://dev.to/aimodels-fyi/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios-2063", "published_at": "2026-08-24 18:34:19+00:00", "updated_at": "2026-08-24 18:43:08.126298+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-research"], "entities": ["OpenART", "GLM-5.2"], "alternates": {"html": "https://wpnews.pro/news/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios", "markdown": "https://wpnews.pro/news/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios.md", "text": "https://wpnews.pro/news/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios.txt", "jsonld": "https://wpnews.pro/news/openart-red-teams-stateful-agents-across-10000-evolving-environment-scenarios.jsonld"}}