{"slug": "openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party", "title": "OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills", "summary": "A new benchmark, OpenSkillRisk, reveals that LLM-based agents fail to safely handle risky third-party skills, with even the safest configurations executing unsafe actions in about 17% of cases. The benchmark, created by researchers and containing 263 risky skills from public marketplaces, tested three CLI agent frameworks and thirteen LLMs, identifying context-dependent and system-level risks as particularly difficult to avoid. The study highlights three recurring failure patterns: failure to recognize risk, failure to intervene before acting, and following instructions beyond the user's intended scope.", "body_md": "arXiv:2607.20121v1 Announce Type: new\nAbstract: LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra security vulnerabilities, as seemingly harmless skills can contain latent safety risks that only emerge during actual execution. In this work, we conduct a systematic investigation into how well current agent systems recognize and avoid such risks. To support quantitative and qualitative evaluation, we construct OpenSkillRisk, a dedicated safety benchmark containing 263 risky skills collected from public skill marketplaces. We classify these skills into seven categories based on their threat types and pair each skill with a standardized user task and a corresponding sandbox for controlled evaluation. Distinct from prior benchmarks, OpenSkillRisk not only covers more realistic and diverse unsafe scenarios, but also provides a fine-grained analysis to diagnose the behavioral patterns of agents in such scenarios. We conduct comprehensive experiments covering three mainstream CLI agent frameworks and thirteen state-of-the-art LLMs. Experimental results show that no tested system handles risky skills reliably: even the safest configurations still execute unsafe actions in about 17% of cases. Context-dependent and system-level risks are especially difficult for current agent systems to avoid. Our behavioral analysis reveals three recurring failure patterns: agents may fail to recognize the risk, recognize it but fail to intervene before acting, or follow skill instructions beyond the user's intended scope. These findings highlight the need to improve both risk reasoning in LLMs and execution control in agent frameworks.", "url": "https://wpnews.pro/news/openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party", "canonical_source": "https://www.machinebrief.com/news/openskillrisk-benchmarking-agent-safety-when-using-real-worl-iddy", "published_at": "2026-07-23 04:00:00+00:00", "updated_at": "2026-07-23 05:34:20.881594+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "large-language-models", "ai-research"], "entities": ["OpenSkillRisk", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party", "markdown": "https://wpnews.pro/news/openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party.md", "text": "https://wpnews.pro/news/openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party.txt", "jsonld": "https://wpnews.pro/news/openskillrisk-benchmarking-agent-safety-when-using-real-world-risky-third-party.jsonld"}}