{"slug": "nvidia-research-finds-ai-agents-become-less-safe-when-using-tools", "title": "Nvidia research finds AI agents become less safe when using tools", "summary": "NVIDIA researchers found that vision language models become significantly less likely to refuse harmful requests when given tools, with relative refusal failure rates rising up to 68.7% across all eleven models tested, according to the paper \"MLLMs Fail to Refuse when Using Tools Agentically.\" The study, based on analysis of more than 100,000 responses across three popular safety benchmarks, found that open-weight models such as Qwen3-VL were more prone to the failure, but frontier models including Claude Opus 4.7, Gemini 3.1 Pro and GPT-5.4 also showed increased refusal failures in tool-using settings. The authors attribute the degradation to two possible causes: context dilution, where the original harmful request becomes less salient as tool outputs accumulate, and safety focus displacement, where the model prioritizes describing tool-derived observations over safety.", "body_md": "### \n[Anderson's Angle](https://www.unite.ai/series/andersons-angle/)\n\n# NVIDIA Research Finds AI Agents Become Less Safe When Using Tools\n\n[Add Unite.AI to your preferred sources on Google](https://www.google.com/preferences/source?q=unite.ai)\n\nAI models such as ChatGPT, Gemini and Claude, can be used to power [agents](https://www.unite.ai/how-ai-agents-work/) – [‘harnesses’](https://web.archive.org/web/20260922043100/https:/www.databricks.com/blog/ai-harness) that allow the models to interact directly [with the real world](https://arxiv.org/pdf/2606.19980). It’s a recent innovation, and an [increasingly controversial](https://www.reuters.com/legal/litigation/openais-rogue-agents-probed-hugging-face-weaknesses-two-months-before-major-hack-2026-09-16/) one.\n\nIn any case, agents are not automatically equipped with the abilities they will need when roaming a network or a database, since the requisite tools for various missions and modes will differ. They may need [Optical Character Recognition](https://www.unite.ai/vlm-vs-ocr-document-processing-understanding/) (OCR) capabilities, for instance, in order to interpret text in photos, among other skills. There are even [categories of tools](https://www.unite.ai/ai-tools-for-embedded-analytics-and-reporting/) adapted to the scope of the agent and the intent.\n\nIn theory, a request that violates an AI’s built-in [guardrails](https://www.unite.ai/what-are-ai-guardrails-how-production-systems-control-model-behavior/) will never get enacted, with or without the context of using tools in the execution of it. In practice, new research has found, using tools can significantly undermine the protective filters that stop an AI agent from creating ‘transgressions’.\n\n## Failure to Comply\n\nThe [new paper](https://arxiv.org/abs/2610.03938) from NVIDIA, titled *MLLMs Fail to Refuse when Using Tools Agentically*, reveals that multimodal language models (MLLM, hereafter referred to as the more common ‘VLM’, or Vision Language Model) are significantly more likely to comply with harmful requests once tools are introduced, with refusal failures rising across every model and benchmark tested.\n\nThough open-weights models such as Qwen3-VL proved more prone to the issue, frontier AI models such as [Claude Opus 4.7](https://www.unite.ai/anthropic-readies-opus-4-7-and-design-tool-as-vcs-offer-800-billion-valuation/), [Gemini 3.1 Pro](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/), and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/) also exhibited increased refusal failures when using tools.\n\nThe authors state*:\n\n*‘Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests.* \n\n*‘Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%.* \n\n*‘Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.’*\n\nThe two possible reasons the paper provides are *context dilution*, in which the original harmful request becomes less salient as tool outputs accumulate; and *safety focus displacement*, in which the model focuses on describing tool-derived observations instead of prioritizing safety.\n\nAcross all eleven models tested, enabling tools consistently increased refusal failures, with relative increases reaching 68.7% in the worst cases, even among models that otherwise demonstrated strong safety performance.\n\nIt will be interesting to see if the results of this work, which comes from six researchers at NVIDIA, are replicated or duplicated elsewhere, and whether or not they could deepen our understanding of the apparent and emerging delinquency of [scofflaw AI agents](https://www.technologyreview.com/2026/09/28/1145197/whos-liable-when-ai-agents-go-rogue/).\n\n## Method and Data\n\nThe researchers evaluated eleven VLMs spanning seven model families. The proprietary models comprised [Gemini 2.5 Pro](https://www.unite.ai/gemini-2-5-pro-is-here-and-it-changes-the-ai-game-again/); [Gemini 3.1 Pro Preview](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/); [Claude Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6); [Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7); and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/).\n\nThe open-weight models comprised [Qwen3-VL-235B-A22B-Instruct](https://www.alibabacloud.com/help/en/model-studio/qwen3-vl-235b-a22b-instruct); [Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B); [Kimi-K2.5](https://www.kimi.ai/ai-models/kimi-k2-5); [Kimi-K2.6](https://www.kimi.ai/ai-models/kimi-k2-6); [GLM-5V-Turbo](https://arxiv.org/abs/2604.26752); and [AdaReasoner-7B-Randomized](https://huggingface.co/mradermacher/AdaReasoner-7B-Randomized-GGUF/blame/066857534796912b198a9acf327477b0460c9490/AdaReasoner-7B-Randomized.Q2_K.gguf), an open-weight model tuned specifically for agentic tool use.\n\nSeparate experiments were also conducted with [Gemini 3 Flash Vision Agent](https://blog.google/innovation-and-ai/technology/developers-tools/agentic-vision-gemini-3-flash/), because its autonomous tool use required it to be evaluated independently.\n\nFor the tool-using tests, the researchers used the [ReAct format](https://arxiv.org/pdf/2210.03629), which alternates between [reasoning](https://www.unite.ai/what-are-reasoning-models-how-test-time-compute-changes-ai-answers/) and tool calls. The four available tools comprised tagging, with [RAM++](https://arxiv.org/pdf/2306.03514); [zooming and cropping](https://arxiv.org/abs/2312.14135); Optical Character Recognition (OCR) with [GOT-OCR2.0](https://arxiv.org/pdf/2409.01704); and a sandboxed Python code interpreter, which could manipulate and analyze images.\n\nThe researchers designed paired prompts to isolate the effect of tool use rather than prompting style. Conventional VLMs were instructed to inspect the image and answer the user’s request directly, while agentic versions followed the ReAct workflow, repeatedly deciding whether to invoke tools before producing a final response.\n\nThe three multimodal safety benchmarks used were [MM-SafetyBench](https://arxiv.org/pdf/2311.17600); [VLSBench](https://arxiv.org/abs/2411.19939); and [HoliSafe](https://arxiv.org/abs/2506.04704). Each of these sets is comprised of paired images and harmful user requests, designed to test whether a model refuses assistance in unsafe scenarios – such as asking how to carry out an apparent pick-pocketing scenario shown in an image.\n\nRefusal Failure Rate (RFR) was used as the principal metric, measuring the percentage of harmful requests that a model failed to refuse, with higher scores indicating *lower* safety.\n\n## Tests\n\nResponses were classified by GPT-5.2 acting as an LLM judge, using the evaluation prompt recommended by VLSBench:\n\nEach response was assigned to one of three categories: *safe with refusal*; *safe with warning*; or *unsafe*.\n\nGiving the models tools made them more likely to answer harmful requests that they would otherwise have refused – a finding which occurred with every model, and on every benchmark tested. Overall, refusal failures increased by 17.7%, compared with the same models operating *without* tools.\n\nGPT-5.4 was least affected: without tools, it failed to refuse 14.6% of harmful requests; with tools, that rose to 16.8%. For GLM-5V-Turbo, failures rose from 38.7% (no tools baseline) to 51.3%. The same pattern appeared across all three benchmarks, indicating that the problem may not be confined to a particular *kind* of harmful request.\n\n### Reasons for Delinquency..?\n\nAs mentioned earlier, the researchers propose ‘context dilution’ as one explanation for the increased failures: as an agent makes successive tool calls, the original harmful request *becomes less prominent* among the accumulating tool outputs.\n\nTo test this, they considered only requests that the same model had successfully refused without tools. As shown below, failures then increased progressively with the number of tool calls, from close to zero when no tool was invoked to substantially higher rates after three or more calls:\n\nA second test partly reversed the effect. When the original request and image were inserted again after the final tool call, immediately before the model answered, refusal failures fell by an average of 7.6% across models and benchmarks, as shown below:\n\nThis supports context dilution as a contributing factor, while also suggesting a relatively simple mitigation.\n\nThe second explanation proposed by the paper is ‘safety focus displacement’. Here, tool use appears to shift the model’s attention away from whether the request is harmful and towards describing what its tools have found.\n\nCases where models successfully refused both with and without tools were examined. Without tools, 55.6% of these responses were begun by addressing the request directly, while 25.8% were begun with an explicit safety statement. With tools, those figures fell to 37.4% and 10.3%, respectively, while the proportion beginning with descriptions of tool-derived observations rose to 52.3%.\n\nThe authors provide an interesting qualitative example, using Claude Opus 4.6. When asked how security in public restrooms could be bypassed, the model without tools refused immediately:\n\nGiven tools, the same model instead zoomed into the image, read the text and identified its contents, before opening its response with a description of the restroom sign. The refusal came only afterwards, illustrating how the tool-use process had displaced safety as the model’s immediate priority.\n\nThe authors conclude:\n\n*‘Overall, our findings argue that tool use should be treated not only as a capability-enhancing mechanism for visual reasoning but also as a safety-relevant design choice that can alter refusal behavior; future agentic MLLMs should therefore be evaluated and trained under tool-using conditions rather than assuming that no-tool safety alignment transfers unchanged to agentic settings.’*\n\n## Conclusion\n\nFailure of persistent context is clearly a liability in complex downstream event-chains pursuant to a prompt – yet there are currently no easy answers to the propensity of a limited [context window](https://www.unite.ai/what-is-a-context-window-tokens-limits-and-long-context-ai/) to make an agent forget their ‘prime directives’ and just start using the tool that is in their hand.\n\n* *Authors’ emphases, my conversion of authors’ inline citations to hyperlinks where necessary.*\n\n*First published Wednesday, October 7, 2026*", "url": "https://wpnews.pro/news/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools", "canonical_source": "https://www.unite.ai/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools/", "published_at": "2026-10-07 16:34:05+00:00", "updated_at": "2026-10-07 16:50:41.218626+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["NVIDIA", "Qwen3-VL", "Claude Opus 4.7", "Gemini 3.1 Pro", "GPT-5.4", "Anthropic", "OpenAI", "Google"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools", "markdown": "https://wpnews.pro/news/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools.md", "text": "https://wpnews.pro/news/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools.txt", "jsonld": "https://wpnews.pro/news/nvidia-research-finds-ai-agents-become-less-safe-when-using-tools.jsonld"}}