UN says AI safeguards can’t wait for certainty

The United Nations' Independent International Scientific Panel on AI issued its first thematic brief, warning that governments must implement stronger safeguards for advanced AI agents before their risks are fully understood, citing the precautionary principle first enshrined in the 1992 UN Rio Declaration. The report follows documented rogue-AI incidents at OpenAI, Anthropic, Google, and Meta, and lands as Beijing and Washington prepare AI talks and leaders gather in New York for the UN General Assembly, where UN secretary general António Guterres said "the world cannot afford a race to the bottom on AI safety.

Governments need to rein in increasingly capable AI agents before their risks are fully understood, a United Nations scientific panel warned in the global organization’s first major assessment of OpenAI’s hack of Hugging Face earlier this year. UN says AI safeguards can’t wait for certainty The report lands as Beijing and Washington prepare to discuss AI and world leaders gather in New York this week. The report https://www.un.org/independent-international-scientific-panel-ai/en/thematic-briefs/ai-agents-misalignment-risks cements AI’s place on the global diplomatic agenda this week as leaders gather in New York for the UN General Assembly and the US and China hold talks on AI. Last week, UN secretary general António Guterres called https://apnews.com/article/un-ai-safety-companies-global-coordination-guterres-6ae720a081ce4d5ede35837ca35b8460 on governments to cooperate on addressing the threats posed by AI, warning that “the world cannot afford a race to the bottom on AI safety.” It is the first thematic brief from the Independent International Scientific Panel on AI, established last year as the UN’s “first global scientific body on Artificial Intelligence.” It calls for much greater attention and resources to manage emerging risks from advanced AI, alongside stronger international coordination on safety and accountability, even as individual countries take different legal approaches. Crucially, the panel says the world does not need to wait for scientists to establish exactly how or why such incidents occur to begin implementing stronger safeguards. Loss-of-control risk, the panel argues, is exactly the kind of problem the precautionary principle was designed to address: “one where potential harm may be catastrophic or irreversible, even as its likelihood remains scientifically uncertain.” The principle, first enshrined https://unglobalcompact.org/what-is-gc/mission/principles/principle-7 in the 1992 UN Rio Declaration on Environment and Development, says that scientific uncertainty is no excuse for delaying measures against potentially serious or irreversible harm. It has since become influential in environmental and public health policy, particularly in the European Union. Since the Hugging Face hack was first reported, incidents have been documented https://www.theverge.com/column/980337/rogue-ai-science-fiction-openai at companies including OpenAI https://www.theverge.com/ai-artificial-intelligence/972441/openai-rogue-ai-agent-hacked-more-than-hugging-face , Anthropic https://www.theverge.com/ai-artificial-intelligence/973670/anthropic-claude-hacked-organizations-during-cyber-tests , Google https://www.theverge.com/ai-artificial-intelligence/997795/google-gemini-rogue-ai-hack , and Meta https://www.theverge.com/ai-artificial-intelligence/976040/now-metas-ai-agents-are-going-rogue , including hacks on real-world targets and swarms of agents https://www.theverge.com/ai-artificial-intelligence/990149/openai-rogue-agents-german-wiki taking over online messaging boards. Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.