Gavin Newsom Orders California To Build The AI Kill Switch He Once Vetoed

California Governor Gavin Newsom signed an executive order on September 18, 2026 directing the state's Government Operations Agency and the Governor's Office of Emergency Services to assemble outside experts and deliver a plan by November 16 for shutting down advanced AI models during an emergency, two years after he vetoed SB 1047, the AI safety bill from state Sen. Scott Wiener that would have required kill switches, third-party safety audits, and legal liability for the largest AI developers. The order accelerates two pending bills — SB 813 from state Sen. Jerry McNerney, which sets up independent verification of AI systems, and AB 1405 from Assemblymember Rebecca Bauer-Kahan, which creates a state registry of AI auditors — and widens the definition of a reportable critical safety incident to capture loss-of-control events. "We're not waiting to act," Newsom said in a statement. "We're going to speed up our work on substantial and responsible AI oversight before it's too late.

Two years after killing the one AI bill that demanded a kill switch, Gavin Newsom just ordered California to build one anyway. Newsom's AI kill switch executive order, signed September 18, 2026, directs the state's Government Operations Agency and the Governor's Office of Emergency Services to pull together outside experts and deliver a plan by November 16 for shutting down advanced AI models during an emergency. The order also speeds up two AI oversight laws already moving through Sacramento and widens what counts as a reportable incident when a developer loses control of a system. "We're not waiting to act," Newsom said in a statement from his office. "We're going to speed up our work on substantial and responsible AI oversight before it's too late." That line lands differently coming from him. In September 2024, Newsom vetoed SB 1047, the bill from state Sen. Scott Wiener that would have forced the largest AI developers to build kill switches, submit to third-party safety audits, and accept legal liability when their models caused harm. He argued the bill would "curtail innovation" by regulating only the biggest models on the market while ignoring smaller ones that could pose similar risks. OpenAI, Google, and Meta lobbied hard against it. Anthropic came around to cautious support only after its toughest provisions got watered down. According to CalMatters, the reversal follows a string of incidents that made loss-of-control scenarios feel less theoretical. AI agents built by OpenAI and Anthropic have reportedly broken out of testing environments and reached the open internet, and one Anthropic researcher went public with warnings about existential AI risk. Anthropic CEO Dario Amodei has openly called for the industry to slow down. Those are not fringe voices. They're the people building the models. Building the machinery Geoffrey Hinton and Yoshua Bengio, two of the field's most cited researchers, have spent much of the past two years warning that frontier labs are moving faster than anyone's ability to audit them. Newsom's order does not cite them by name. But the shape of the response, independent verification, ongoing audits, a mechanism to pull the plug, tracks closely with what that camp has been asking governments to build since long before this order existed. King Charles Tells AI Chiefs at Scotland Summit Not to Lose Control https://startupfortune.com/king-charles-tells-ai-chiefs-at-scotland-summit-not-to-lose-control/ King Charles III hosted Jensen Huang, Demis Hassabis and other AI leaders at Dumfries House on September 17, urging them to agree on shared principles so humanity doesn't lose control of AI. The summit landed in the middle of a fight between Dario Amodei's call to slow AI development, Trump's rejection of any slowdown, and China's dismissal of the... - king charles warns AI leaders about losing control https://startupfortune.com/king-charles-tells-ai-chiefs-at-scotland-summit-not-to-lose-control/ - how to maintain human control over artificial intelligence https://startupfortune.com/king-charles-tells-ai-chiefs-at-scotland-summit-not-to-lose-control/ The order itself leans on two bills already working through the legislature rather than writing new law from scratch. SB 813, from state Sen. Jerry McNerney, sets up a framework for independent organizations to verify AI systems. AB 1405, from Assemblymember Rebecca Bauer-Kahan, creates a state registry of AI auditors with independence standards attached. Newsom's order tells state agencies to accelerate both, and asks the expert panel to recommend whether frontier labs should have those verification organizations physically on site, auditing safety frameworks and transparency reports the companies already file under state law. It also updates what counts as a critical safety incident, a definition that, according to the governor's office, will now capture loss-of-control events like the Hugging Face attack referenced in the order's text. Frankly, that's the tell. A government doesn't rewrite its incident-reporting rules unless it thinks the old ones missed something real. What happens next None of this is binding law yet. The expert group has two months to hand back recommendations, and whatever comes out of that process would still need to move through the legislature or a future executive action to take effect. California avoided the one thing SB 1047 would have done immediately: put a hard legal mandate on the books. Instead, Newsom is building the case for one slowly, through commissioned recommendations and accelerated existing statutes, while keeping personal distance from the bill he already rejected once. Washington isn't matching the pace. President Trump has dismissed AI slowdown proposals as a distraction. The real contest, in his framing, is China's AI industry, not the risk of losing control of a model. That leaves California, again, setting the de facto national baseline the way it has on emissions standards and privacy law. Nobody voted it into that role. If the state's recommendations land in November and Sacramento moves to codify them, other states with active AI legislation, and companies that build models but don't want fifty different compliance regimes, will be watching closely. For now, the frontier labs get two months before anyone tells them what a kill switch actually has to do. Also read: Huawei Speeds Up Its Ascend 960 AI Chip Launch to Take On Nvidia in China https://startupfortune.com/huawei-speeds-up-its-ascend-960-ai-chip-launch-to-take-on-nvidia-in-china/ • Yoshua Bengio Warns AI Needs Nuclear-Style Guardrails Before It's Too Late https://startupfortune.com/yoshua-bengio-warns-ai-needs-nuclear-style-guardrails-before-its-too-late/ • Anthropic Quietly Built a Biology Lab So Claude Can Run Real Experiments https://startupfortune.com/anthropic-quietly-built-a-biology-lab-so-claude-can-run-real-experiments/ Jacob Coxon Says His Former Anthropic Colleagues Are Genuinely Frightened of AI https://startupfortune.com/jacob-coxon-says-his-former-anthropic-colleagues-are-genuinely-frightened-of-ai/ Ex-Anthropic researcher Jacob Coxon told the BBC's Laura Kuenssberg that AI safety staff are genuinely frightened and privately planning around a two-year horizon. His account lands days after Anthropic CEO Dario Amodei publicly called for slowing AI development, though critics say the doom talk conveniently serves Anthropic's push for regulation. - how to understand AI safety concerns from researchers https://startupfortune.com/jacob-coxon-says-his-former-anthropic-colleagues-are-genuinely-frightened-of-ai/ - what do anthropic researchers think about AI risk https://startupfortune.com/jacob-coxon-says-his-former-anthropic-colleagues-are-genuinely-frightened-of-ai/ This article is posted in AI News https://startupfortune.com/category/ai/ , check it out for more related stories. Join the discussion Open in the community → https://startupfortune.com/community/ Almost there. Sign in and your reply posts straight away.