{"slug": "anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are", "title": "Anthropic Wants Governments to Stop 'Catastrophic' AI Models Before They Are Deployed", "summary": "Anthropic released its Advanced AI Framework in June, calling for governments to gain legal authority to block deployment of frontier AI models that pose a significant risk of catastrophic harm, with escalating civil penalties tied to global annual revenue for violations. The framework targets models trained using more than 10²⁵ floating-point operations and developed by companies earning more than $500 million in AI-related revenue or spending more than $1 billion on AI research and development, and identifies four risk categories: biological threats, cyberattacks, loss of control, and automated AI research and development. Anthropic also released a threat-intelligence report on Thursday detailing attempts to misuse Claude, and said its Claude Mythos Preview discovered thousands of high-severity software vulnerabilities, including flaws affecting major operating systems and browsers.", "body_md": "# Anthropic Wants Governments to Stop 'Catastrophic' AI Models Before They Are Deployed\n\n## The Claude developer's proposed framework would introduce catastrophic-risk testing, independent evaluations and revenue-linked penalties for frontier AI companies\n\nAnthropic wants governments to gain the power to stop advanced AI models from being released if they pose a significant risk of [catastrophic harm](https://www.ibtimes.co.uk/ai-researcher-resigns-warns-human-extinction-risk-1818990), warning that rapidly improving systems could outpace regulators before safeguards are in place.\n\nThe Claude developer unveiled its Advanced AI Framework in June, calling for tougher oversight of frontier systems as their capabilities accelerate, and on Thursday released a threat-intelligence report detailing attempts to misuse Claude.\n\nUnder the proposal, governments could prevent or deter dangerous deployments, while developers could face escalating civil penalties linked to their global annual revenue for violations.\n\nThe company stressed that such powers would require safeguards against government overreach and should apply only to the most advanced AI developers.\n\n## Governments Could Block Dangerous Deployments\n\nAnthropic's proposal goes beyond requiring AI companies to disclose how their models are tested.\n\nThe company wants frontier developers to conduct catastrophic-risk testing, publish safety information, and submit their models to qualified independent evaluators.\n\nIf testing identifies a significant risk of catastrophic harm, Anthropic argues that governments should have legal authority to prevent or deter the model from being deployed.\n\nThe company says that authority would go beyond the powers available under current US law and proposals before Congress.\n\nAnthropic also recommends escalating civil penalties tied to global annual revenue for repeated violations, giving regulators financial enforcement powers alongside the ability to intervene before deployment.\n\n## Four Areas of Catastrophic Risks Are Identified\n\nAnthropic's framework identifies four categories of risk that could justify heightened scrutiny: biological threats, [cyberattacks](https://www.ibtimes.co.uk/frontier-ai-cyber-threats-financial-stability-1817077), loss of control, and automated AI research and development.\n\nThe company warns that increasingly capable models could make developing biological weapons easier or help attackers identify vulnerabilities in critical infrastructure.\n\nAnother concern involves systems becoming difficult for their developers to control as their capabilities increase.\n\nAnthropic also points to AI systems increasingly automating AI research itself, potentially accelerating improvements and amplifying other risks.\n\nThe company said its own [Claude Mythos Preview](https://www.ibtimes.co.uk/anthropic-restricts-uk-access-claude-mythos-5-1-1818840) discovered thousands of high-severity software vulnerabilities, including flaws affecting major operating systems and browsers, as evidence of how quickly capabilities are progressing.\n\n## Rules Would Target Only Frontier Developers\n\nAnthropic is not proposing that every AI company face the same regulatory regime.\n\nIts framework would apply to models trained using more than 10²⁵ floating-point operations and developed by companies earning more than $500 million in AI-related revenue or spending more than $1 billion on AI research and development.\n\nThose thresholds are intended to focus regulation on companies developing the most computationally powerful systems while limiting the burden on smaller businesses and less capable models.\n\nAnthropic says developers covered by the rules should also maintain robust security programmes to protect model weights and training infrastructure from cyberattacks and theft.\n\n## Anthropic Says Industry Cannot Police Itself\n\nThe proposal is notable because it comes from one of the companies developing the frontier systems that would face [greater government oversight](https://www.ibtimes.co.uk/bernie-sanders-superintelligence-ban-ai-safety-1818810).\n\nAnthropic argues AI companies should not have sole responsibility for deciding whether their own models are safe enough for release.\n\nThe company is also proposing investments in biological surveillance, critical infrastructure protection, and systems capable of detecting or responding to AI operating outside developers' control.\n\nAnthropic acknowledged that questions surrounding advanced AI regulation remain complex and its proposals are likely to face debate.\n\nBut its central argument is that waiting for catastrophic capabilities to emerge before establishing government authority could leave regulators responding too late.\n\nAs frontier systems become more capable, Anthropic wants governments to have the legal tools to intervene before a dangerous model reaches the public.\n\n© Copyright IBTimes 2026. All rights reserved.", "url": "https://wpnews.pro/news/anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are", "canonical_source": "https://www.ibtimes.co.uk/anthropic-government-power-block-high-risk-ai-models-1819150", "published_at": "2026-09-11 10:47:13+00:00", "updated_at": "2026-09-11 11:07:58.968305+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "artificial-intelligence", "large-language-models"], "entities": ["Anthropic", "Claude", "Claude Mythos Preview", "US Congress"], "alternates": {"html": "https://wpnews.pro/news/anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are", "markdown": "https://wpnews.pro/news/anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are.md", "text": "https://wpnews.pro/news/anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are.txt", "jsonld": "https://wpnews.pro/news/anthropic-wants-governments-to-stop-catastrophic-ai-models-before-they-are.jsonld"}}