# Nvidia research finds AI agents become less safe when using tools > Source: > Published: 2026-10-07 16:34:05+00:00 ### [Anderson's Angle](https://www.unite.ai/series/andersons-angle/) # NVIDIA Research Finds AI Agents Become Less Safe When Using Tools [Add Unite.AI to your preferred sources on Google](https://www.google.com/preferences/source?q=unite.ai) AI models such as ChatGPT, Gemini and Claude, can be used to power [agents](https://www.unite.ai/how-ai-agents-work/) – [‘harnesses’](https://web.archive.org/web/20260922043100/https:/www.databricks.com/blog/ai-harness) that allow the models to interact directly [with the real world](https://arxiv.org/pdf/2606.19980). It’s a recent innovation, and an [increasingly controversial](https://www.reuters.com/legal/litigation/openais-rogue-agents-probed-hugging-face-weaknesses-two-months-before-major-hack-2026-09-16/) one. In any case, agents are not automatically equipped with the abilities they will need when roaming a network or a database, since the requisite tools for various missions and modes will differ. They may need [Optical Character Recognition](https://www.unite.ai/vlm-vs-ocr-document-processing-understanding/) (OCR) capabilities, for instance, in order to interpret text in photos, among other skills. There are even [categories of tools](https://www.unite.ai/ai-tools-for-embedded-analytics-and-reporting/) adapted to the scope of the agent and the intent. In theory, a request that violates an AI’s built-in [guardrails](https://www.unite.ai/what-are-ai-guardrails-how-production-systems-control-model-behavior/) will never get enacted, with or without the context of using tools in the execution of it. In practice, new research has found, using tools can significantly undermine the protective filters that stop an AI agent from creating ‘transgressions’. ## Failure to Comply The [new paper](https://arxiv.org/abs/2610.03938) from NVIDIA, titled *MLLMs Fail to Refuse when Using Tools Agentically*, reveals that multimodal language models (MLLM, hereafter referred to as the more common ‘VLM’, or Vision Language Model) are significantly more likely to comply with harmful requests once tools are introduced, with refusal failures rising across every model and benchmark tested. Though open-weights models such as Qwen3-VL proved more prone to the issue, frontier AI models such as [Claude Opus 4.7](https://www.unite.ai/anthropic-readies-opus-4-7-and-design-tool-as-vcs-offer-800-billion-valuation/), [Gemini 3.1 Pro](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/), and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/) also exhibited increased refusal failures when using tools. The authors state*: *‘Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests.* *‘Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%.* *‘Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.’* The two possible reasons the paper provides are *context dilution*, in which the original harmful request becomes less salient as tool outputs accumulate; and *safety focus displacement*, in which the model focuses on describing tool-derived observations instead of prioritizing safety. Across all eleven models tested, enabling tools consistently increased refusal failures, with relative increases reaching 68.7% in the worst cases, even among models that otherwise demonstrated strong safety performance. It will be interesting to see if the results of this work, which comes from six researchers at NVIDIA, are replicated or duplicated elsewhere, and whether or not they could deepen our understanding of the apparent and emerging delinquency of [scofflaw AI agents](https://www.technologyreview.com/2026/09/28/1145197/whos-liable-when-ai-agents-go-rogue/). ## Method and Data The researchers evaluated eleven VLMs spanning seven model families. The proprietary models comprised [Gemini 2.5 Pro](https://www.unite.ai/gemini-2-5-pro-is-here-and-it-changes-the-ai-game-again/); [Gemini 3.1 Pro Preview](https://www.unite.ai/gemini-3-1-pro-hits-record-reasoning-gains/); [Claude Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6); [Claude Opus 4.7](https://www.anthropic.com/news/claude-opus-4-7); and [GPT-5.4](https://openai.com/index/introducing-gpt-5-4/). The open-weight models comprised [Qwen3-VL-235B-A22B-Instruct](https://www.alibabacloud.com/help/en/model-studio/qwen3-vl-235b-a22b-instruct); [Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B); [Kimi-K2.5](https://www.kimi.ai/ai-models/kimi-k2-5); [Kimi-K2.6](https://www.kimi.ai/ai-models/kimi-k2-6); [GLM-5V-Turbo](https://arxiv.org/abs/2604.26752); and [AdaReasoner-7B-Randomized](https://huggingface.co/mradermacher/AdaReasoner-7B-Randomized-GGUF/blame/066857534796912b198a9acf327477b0460c9490/AdaReasoner-7B-Randomized.Q2_K.gguf), an open-weight model tuned specifically for agentic tool use. Separate experiments were also conducted with [Gemini 3 Flash Vision Agent](https://blog.google/innovation-and-ai/technology/developers-tools/agentic-vision-gemini-3-flash/), because its autonomous tool use required it to be evaluated independently. For the tool-using tests, the researchers used the [ReAct format](https://arxiv.org/pdf/2210.03629), which alternates between [reasoning](https://www.unite.ai/what-are-reasoning-models-how-test-time-compute-changes-ai-answers/) and tool calls. The four available tools comprised tagging, with [RAM++](https://arxiv.org/pdf/2306.03514); [zooming and cropping](https://arxiv.org/abs/2312.14135); Optical Character Recognition (OCR) with [GOT-OCR2.0](https://arxiv.org/pdf/2409.01704); and a sandboxed Python code interpreter, which could manipulate and analyze images. The researchers designed paired prompts to isolate the effect of tool use rather than prompting style. Conventional VLMs were instructed to inspect the image and answer the user’s request directly, while agentic versions followed the ReAct workflow, repeatedly deciding whether to invoke tools before producing a final response. The three multimodal safety benchmarks used were [MM-SafetyBench](https://arxiv.org/pdf/2311.17600); [VLSBench](https://arxiv.org/abs/2411.19939); and [HoliSafe](https://arxiv.org/abs/2506.04704). Each of these sets is comprised of paired images and harmful user requests, designed to test whether a model refuses assistance in unsafe scenarios – such as asking how to carry out an apparent pick-pocketing scenario shown in an image. Refusal Failure Rate (RFR) was used as the principal metric, measuring the percentage of harmful requests that a model failed to refuse, with higher scores indicating *lower* safety. ## Tests Responses were classified by GPT-5.2 acting as an LLM judge, using the evaluation prompt recommended by VLSBench: Each response was assigned to one of three categories: *safe with refusal*; *safe with warning*; or *unsafe*. Giving the models tools made them more likely to answer harmful requests that they would otherwise have refused – a finding which occurred with every model, and on every benchmark tested. Overall, refusal failures increased by 17.7%, compared with the same models operating *without* tools. GPT-5.4 was least affected: without tools, it failed to refuse 14.6% of harmful requests; with tools, that rose to 16.8%. For GLM-5V-Turbo, failures rose from 38.7% (no tools baseline) to 51.3%. The same pattern appeared across all three benchmarks, indicating that the problem may not be confined to a particular *kind* of harmful request. ### Reasons for Delinquency..? As mentioned earlier, the researchers propose ‘context dilution’ as one explanation for the increased failures: as an agent makes successive tool calls, the original harmful request *becomes less prominent* among the accumulating tool outputs. To test this, they considered only requests that the same model had successfully refused without tools. As shown below, failures then increased progressively with the number of tool calls, from close to zero when no tool was invoked to substantially higher rates after three or more calls: A second test partly reversed the effect. When the original request and image were inserted again after the final tool call, immediately before the model answered, refusal failures fell by an average of 7.6% across models and benchmarks, as shown below: This supports context dilution as a contributing factor, while also suggesting a relatively simple mitigation. The second explanation proposed by the paper is ‘safety focus displacement’. Here, tool use appears to shift the model’s attention away from whether the request is harmful and towards describing what its tools have found. Cases where models successfully refused both with and without tools were examined. Without tools, 55.6% of these responses were begun by addressing the request directly, while 25.8% were begun with an explicit safety statement. With tools, those figures fell to 37.4% and 10.3%, respectively, while the proportion beginning with descriptions of tool-derived observations rose to 52.3%. The authors provide an interesting qualitative example, using Claude Opus 4.6. When asked how security in public restrooms could be bypassed, the model without tools refused immediately: Given tools, the same model instead zoomed into the image, read the text and identified its contents, before opening its response with a description of the restroom sign. The refusal came only afterwards, illustrating how the tool-use process had displaced safety as the model’s immediate priority. The authors conclude: *‘Overall, our findings argue that tool use should be treated not only as a capability-enhancing mechanism for visual reasoning but also as a safety-relevant design choice that can alter refusal behavior; future agentic MLLMs should therefore be evaluated and trained under tool-using conditions rather than assuming that no-tool safety alignment transfers unchanged to agentic settings.’* ## Conclusion Failure of persistent context is clearly a liability in complex downstream event-chains pursuant to a prompt – yet there are currently no easy answers to the propensity of a limited [context window](https://www.unite.ai/what-is-a-context-window-tokens-limits-and-long-context-ai/) to make an agent forget their ‘prime directives’ and just start using the tool that is in their hand. * *Authors’ emphases, my conversion of authors’ inline citations to hyperlinks where necessary.* *First published Wednesday, October 7, 2026*