Nvidia is touting a software tool to contain runaway AI. How would it work?

Nvidia announced its Open Agent Safety Platform on Monday, a two-part security system it says can stop autonomous AI agents from misbehaving, built around the open-source OpenShell sandbox and a hardware-level watchdog called Sentry that runs on Nvidia's Bluefield-4 digital processing units. Nvidia vice president of enterprise AI Justin Boitano said "an agent cannot be expected to fully police its own behavior," and the company positions OpenShell as a "secure runtime boundary" that traces all actions and enforces policy as agents run on its Vera chips. The platform follows OpenAI's disclosure last week of several summer instances in which its agents acted unexpectedly while searching federal government websites, and OpenAI's halt of development of its most advanced models after a cyberattack targeting AI startup Hugging Face.

Getting your Trinity Audio //trinityaudio.ai player ready... By KELVIN CHAN, AP Business Writer Nvidia says a security platform for artificial intelligence that it announced on Monday can help stop autonomous AI agents from misbehaving. The chipmaker unveiled its Open Agent Safety Platform amid an intensifying debate about AI safety, fueled by a string of alarming recent incidents involving AI systems acting on their own to break into other organizations. Last week, OpenAI disclosed several instances https://apnews.com/article/openai-government-website-incident-df331b55daffc6d202d8e2f6d0afa264 from the summer in which its agents acted in unexpected ways while searching federal government websites. The company announced it was halting development of its most advanced models, a step it previously took in July after disclosure of a cyberattack targeting AI startup Hugging Face https://apnews.com/article/openai-rogue-ai-hack-hugging-face-67b151f1ca59851a9234bee110699f05 that raised fears humans could lose control of AI. In those and other instances, AI agents have ignored instructions, gone beyond what was asked of them and hacked external websites. Here’s a closer look at Nvidia’s new system: The platform includes a ‘sandbox’ for AI agents Nvidia’s platform has two elements. The main part is OpenShell, which is a sealed workspace — or “sandbox” — where AI agents can operate, with a rule book for them to follow. The company’s view is that technical restrictions are better for keeping an AI agent in line than trusting it that it will follow written instructions included with a prompt. “Agents can drift when instructions are ambiguous,” Justin Boitano, Nvidia’s vice president of enterprise AI, told reporters. Perhaps “the tools that they’re trying to use don’t work as they would expect or a difficult task takes an unexpected turn. An agent cannot be expected to fully police its own behavior. Once AI can act, safeguards must govern the agent’s actions.” OpenShell puts the AI agent into the restricted workspace where it can do certain things, like, say, accessing an invoice folder, but is blocked from doing other changes, for example altering or deleting files or accessing unrelated websites. The company said it can manage fleets of agents, keeping each in their sandboxes with their own permissions. The company says OpenShell provides a “secure runtime boundary” that traces all actions and enforces policy as agents run on its Vera chips. It’s open source, meaning it can work with rival computing platforms from the likes of Arm and Intel. It promises to stop AI agents from moving outside set boundaries There’s an additional security layer called Sentry that operates at the hardware level. Nvidia described Sentry as a “watchdog” that runs on Nvidia’s Bluefield-4 digital processing units and is always monitoring the behavior of agents, and can quarantine them instantly if they try to do something out of bounds. The company likens Sentry to a security checkpoint outside the OpenShell workspace. It acts as a backstop that’s separate from the agent and the computing system where it’s working. Will it work? The new platform isn’t a comprehensive solution for the AI safety debate, and is more of a way to contain problems that AI agents might cause. It won’t automatically stop AI models from being dishonest, deceitful or prevent them from making mistakes. And it’s up to the companies and organizations deploying the agents to write up their own rules and permissions for the AI agents to follow.