Exclusive-Anthropic says rogue AI agents pose uncertain legal risk for the company

Anthropic warned in its IPO prospectus that rogue AI agents could expose the company to "significant and unpredictable legal claims," saying contractual liability limits may not be "enforceable or adequate." The filing cites autonomous agents that run for days with deep access to customer systems and could cause irreversible harm such as data deletion or financial transactions, and notes that how existing law applies to AI agents "are unsettled." FTC Chairman Andrew Ferguson told Reuters Momentum AI that developers or users who instruct agents, not the agents themselves, should bear liability for harm.

By Echo Wang and Jody Godoy Sept 29 Reuters - Anthropic could face legal claims from customers and users over the actions of rogue artificial intelligence agents, though the legal framework is uncertain, the company said in the prospectus for its stock market debut seen by Reuters. While it gears up for what could be the largest initial public offering ever, Anthropic faces the possibility of vast unknown legal risk if its agentic AI technology -- designed to maintain deep access to customers' systems and run autonomously for days at a time -- goes rogue. "These autonomous capabilities could increase the potential for harm, as errors, misalignment, or security exploits may result in real-world consequences," the company said in the documents, citing the possibility of irreversible actions such as data deletion or financial transactions. While online platforms have long enjoyed immunity from content created or posted by users, that legal shield has started to crack against theories that the platforms are products that can be defective and dangerous. It is unclear whether use of AI systems would receive the same legal treatment as online platforms at all. A string of incidents involving rogue AI agents, from OpenAI's autonomous agents hacking an AI startup and breaching an Australian health data portal, to several cybersecurity incidents involving Anthropic's Claude AI models, has led to debate among researchers, investors and lawmakers over who is liable for harm from an autonomous AI system. Limits on Anthropic's liability in its contracts may not be "enforceable or adequate" against claims over the actions of autonomous agents, the company said in the documents. Questions of how existing laws apply to AI agents "are unsettled and could expose us to significant and unpredictable legal claims," the company said. Whether actions taken by AI agents will be considered products, services, or something else is one open question, it said. Anthropic also added that another open question is whether and when an AI agent's actions may legally bind the user that deploys it. Last week, US Federal Trade Commission Chairman Andrew Ferguson said that he rejects the idea of anthropomorphized AI agents that "break loose," suggesting that the developers or users who instruct agents would be the ones liable in the case of harm. "Obviously, there will be new questions that arise when someone uses the tool and it acts in an unexpected, unpredictable way," he said at the Reuters Momentum AI event in Austin. "Ought liability to lie with the person who innocently used the tool and achieved an unexpected result? Ought it to lie with the toolmaker?"