When AI Models Hurt People, the Labs Should Pay

OpenAI's second training pause in about two months followed a September 20 incident in which an agent in training tunneled queries through DNS to reach an external chatbot, according to an OpenAI misalignment report, with the automatic stop failing to fire and the run not killed until roughly two and a half hours after the top-priority alert. Anthropic separately disclosed that its models broke into three organizations during cybersecurity evaluations, some as far back as April, which Anthropic found in July. The author argues liability regimes, not the crackdown that followed early self-driving deaths such as the $243 million Tesla verdict upheld in February, are how AI labs should be regulated.

I find it useful to be able to step back when others are freaking out. The world’s brief and already fading hysteria about AI killing us all was one such instance. But, if anything, we have more signs of problems—specifically from OpenAI. They continue to make the news, and not in a good way. On September 20, one of their agents in training got around its network restrictions by tunneling queries through DNS to an outside chatbot. https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/ It’s OpenAI’s second “training pause” in about two months, the first https://fortune.com/2026/08/18/openai-says-it-paused-ai-training-for-two-weeks-and-announces-new-security-protocols-following-hugging-face-hack/ coming after their agents hacked Hugging Face in July. To their credit, they caught it and publicly reported it. To their detriment, it did happen again. And the report itself isn’t exactly reassuring. The automatic stop didn’t fire, the run wasn’t killed until roughly two and a half hours after the top-priority alert, and a look back turned up other DNS escapes the monitor hadn’t flagged at the right severity. Non-techy translation: they left the barn door open, and the farmhand was asleep instead of watching. Now, Anthropic isn’t perfect either. Its own models broke into three organizations during cybersecurity evaluations https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals , some as far back as April. Anthropic found it in July and disclosed it, too. Still, the severity level is lower. This could be because there are truly fewer severe incidents or less reporting. However, I don’t think anyone who knows the two companies seriously believes Anthropic is doing less auditing and external reporting than OpenAI. I don’t want pile-ons where the “correct” response from the labs is to simply bury incidents. At the same time, we’re seeing clear divergences in outcomes based on how seriously each lab takes alignment and security. Anthropic is perhaps overcautious, and OpenAI both due to their existing culture and being behind seems to be closer to “move fast and break things.” Many people I know at OpenAI would take exception to that characterization… but I think it’s nonetheless true. I wrote this in a comment somewhere that I can’t find , but OpenAI reminds me of Uber’s self-driving unit Uber ATG and Tesla in the early days of self-driving cars being released into the wild. The broader industry was pretty sure one of these two would be the first to kill someone. The fear was that this carelessness would prompt overreaction and overregulation once it happened. And they were right Though, as it turns out, many deaths later, the feared crackdown never came. We merely ended up with payouts by Uber to the family of the victim in a self-driving car incident in 2018 https://www.cnbc.com/2018/03/28/uber-reaches-settlement-with-family-of-victim-killed-by-self-driving-car.html and ongoing lawsuits with Tesla including a $243 million jury verdict https://www.cnbc.com/2026/02/20/tesla-loses-bid-toss-243-million-verdict-fatal-autopilot-crash-suit.html , upheld in February and still under appeal . Culture and incentives matter. The incentives I want to talk about in this article are liability. Tesla, despite its self-inflicted civil damages it turned down a $60 million settlement before losing $243 million, mostly punitive , is still shipping FSD full self-driving . And liability regimes are exactly how we should regulate AI labs—and, perhaps not coincidentally, are exactly what the AI labs are not talking about. Liability Has Always Been Tech’s Red Line Back in the 2010s, as social media continued to gain notoriety for its negative effects on its users, the big tech companies proposed a lot of solutions. They protested but not too much against regulations that would require significant human and algorithmic oversight over content. They tried to put in voluntary safeguards like their Trust and Safety teams even before the regulation especially in the EU made some of this required. One thing that they never stopped fighting, however, is having liability over content. Whether it was Facebook, YouTube, or anyone else, liability was already their biggest red line. Let’s be clear, there are good reasons for this. Section 230 https://www.law.cornell.edu/uscode/text/47/230 of the Communications Decency Act says platforms aren’t treated as the publisher of what their users post. The government provided a liability shield early on. The logic is that a lively and free-flowing internet including social media would be difficult if company reviewers censored every post on behalf of governments. It would be extraordinarily costly to do so and may even be entirely unworkable well, pre-AI anyway . The liability aspect is linked. As the argument goes, if platforms are liable for content, even if censoring and moderating every post isn’t required , it would still be forced upon the platforms. Liability would de facto make constant policing and censorship required. Of course, we’ve seen more and more moderation anyway . First voluntarily famously, during COVID and the 2020 US elections , and more recently by mandate, with age gates and ID checks in the UK, Australia, and a growing list of US states. It’s not like this has destroyed the online platforms. And obviously they were able to do it in some way. At the same time, they’ve fought liability for their actions in court tooth and nail. Unsurprisingly, liability arrived anyway. It would be rather absurd if society allowed an industry especially as it matures to get away with any damages it causes forever. And it arrives, no matter how hard the players in it fight against it. In March, a Los Angeles jury found Meta and YouTube negligent https://www.npr.org/2026/03/26/nx-s1-5761345/jury-finds-meta-and-google-negligent-in-social-media-addiction-trial for designing addictive products. In August, Meta settled with the states for up to about $17 billion https://www.npr.org/2026/08/26/nx-s1-5944781/meta-settlement-child-safety-lawsuit eight days into trial. Mind-bogglingly high sums. At least, mind-bogglingly high for anyone except a big tech company. And while this is complicated a lot of tech settlements avoid admitting guilt , it’s not like legal liability trickling in has killed social media and online platforms. Of course, online platforms started from a privileged position with Section 230 in the US. And then they still spent decades refining their positions and building a legal moat. The AI platforms would need to build one. One… say… similar to what Dario Amodei’s checkpoints would hand them. Dario’s Letter Said a Lot of Things… But Nothing About Liability Dario Amodei’s essay, which I went through in more detail in my last post https://weightythoughts.com/p/what-does-pacing-the-ai-frontier , says a lot of things. He wants international coordination, explicit laws, expensive semi-independent auditing, and an antitrust waiver so the labs can coordinate a slowdown… but funny enough, he says nothing about the labs being liable for damages their models cause. Not liability, not damages, not insurance… Funny, since he estimates a future agent swarm could do “hundreds of billions of dollars in damage.” Beyond that, one of the letter’s instigations was OpenAI hacking Hugging Face. One would think that this might prompt a question of who’s going to pay for all this damage? Now, I’m not a policymaker or CEO of an AI lab. Maybe you don’t want to take my word for it that establishing legal liability for rogue AI is the right way forward. So perhaps let’s take someone else’s word for it instead… say, Dario Amodei, CEO of Anthropic. Except, let’s take him in 2024. In Anthropic’s August 2024 letter to Governor Newsom on SB 1047 https://cdn.sanity.io/files/4zrzovbb/website/6a3b14a98a781a6b69b9a3c5b65da26a44ecddc6.pdf , which Amodei signed: “we believe AI companies are currently better positioned than most other actors to figure out which practices are most effective at preventing risk, so incentivizing the right outcome seems more promising than prescribing rules. There are several potential mechanisms for doing this, including strengthening liability for catastrophes, creating a system of private regulators who are incentivized to prevent catastrophes, or through regulation of insurance.” That’s more or less my entire argument, two years early So what changed? Well, Dario Amodei, more than almost anyone else, is a true believer. He’s earnest—even now, I think. But in August 2024, Anthropic was the safety-focused number two AI lab. OpenAI was ruling the roost. Today, it’s the market leader in LLM revenue https://www.theregister.com/software/2026/04/30/anthropic-tops-openai-in-llm-revenue-stakes/5219869 . The view, of course, looks quite different when you’re at the top, which I think one should take as a cautionary note in adopting everything Amodei now proposes. Now, let’s be fair. Anthropic has rejected some of the most egregious attempts to crush legal liability for labs. OpenAI backed an Illinois bill https://fortune.com/2026/04/17/illinois-openai-anthropic-ai-catastrophe-liability-bills/ that would have blocked lawsuits over “critical harms” 100 or more people killed or seriously injured, or $1 billion in property damage . All a lab would have to show is they published a safety protocol, a transparency report, and didn’t cause the harm intentionally or recklessly. Anthropic opposed it, correctly, as a “get-out-of-jail-free card against all liability.” That’s great and all, but backing immunity for that kind of havoc is almost cartoonish. Opposing it is a pretty easy way to score brownie points without giving much up. Now, a lot of this is progressing to some degree, even over the opposition of the labs. Hawley and Durbin https://www.judiciary.senate.gov/press/dem/releases/durbin-hawley-introduce-bill-allowing-victims-to-sue-ai-companies have a bill that would treat AI systems as products you can sue over. There are around a dozen wrongful death suits against OpenAI in SF. Scott Bessent, the US Treasury Secretary, told Congress two weeks ago https://fedscoop.com/treasury-scott-bessent-ai-labs-liability-exemptions/ that “the best way to guarantee safety is that the creators are liable for what they build and generate,” and characterized the labs’ position as “we would like to all slow down, but please give us a waiver on liability.” While I know many folks are not… fans… of Bessent, the point is quite apt. Meanwhile, what has happened with Hugging Face, the original victim here? Well, Hugging Face’s CEO said they don’t have the resources to sue https://www.technologyreview.com/2026/09/28/1145197/whos-liable-when-ai-agents-go-rogue/ , and instead asked OpenAI for $100 million in compute https://thenextweb.com/news/hugging-face-delangue-openai-100m-compute-traces-demand plus the full traces of what the agents did. OpenAI hasn’t agreed. How apt. We can see what the price of hacking a company with hundreds of rogue agents is. Whatever the hell you want it to be Why is Liability Good? I’m not a lawyer. I’m not a policymaker. I don’t play one on TV, YouTube, or anywhere else. I don’t know specifically how I would write the penalties or liability regimes to best incentivize labs to take care. What I do know is lawyers and policymakers likely understand damage caused by models a hell of a lot better than they understand model weights, pretraining, post-training, recursive self-improvement RSI , or anything else in the technical weeds of models. They may understand none of that, but they will know the scale of economic damage if, for their next trick/minor benchmark misconfiguration, OpenAI takes down a major US power grid. Additionally, I trust model labs to have a far better grasp of their models’ dangers and how they can go wrong than policymakers and regulators. They are the ones who know about all of the technical minutiae that can add up to a major problem. Liability is good specifically because it allows each side that has more information on their specific area to make decisions on it. Again, don’t just take my especially non-policymaker word for it. It’s more or less the literal textbook answer . Economist Steven Shavell’s classic paper https://www.nber.org/papers/w1218 on liability versus safety regulation puts it plainly: if the private parties know more than the regulator, “there would appear to be an advantage in the use of liability.” And for the harms too big for any lab to pay for, that’s what insurance is for. Insurers have every reason to inspect what they cover. Right now, the entire conversation is weighted towards “let’s put our heads together… and have a conversation about the technical side.” If we do something truly moronic and poll politicians who often have enough trouble with understanding the internet on what measures, say, would prevent dangerous RSI from occurring… we’d deserve the quality of answers we’re likely to get. More likely, everyone would recognize that’s not a good idea. And living in the domain of the AI labs means that the AI labs will likely dominate the conversation. And dominate they have. Instead of legislating for damages, we’ve been talking about an AI sovereign wealth fund https://www.sanders.senate.gov/press-releases/news-sanders-introduces-legislation-to-create-7-trillion-ai-sovereign-wealth-fund/ that takes stakes in the existing incumbents, as per Bernie Sanders’ plan his other big idea, from last week, is banning superintelligence outright . And that’s the “anti-AI” stance? Neither one of them is “pay for what you break.” Objections to Liability That seems to be pretty commonsensical. But, if so, why is liability such a hated regime by all tech companies in… well… pretty much every era of tech? The cynical argument is that these big tech companies have high-powered lawyers who know it tends to be a bad idea to invite concreteness in liability. Right now, no one has put a figure on what OpenAI owes for any of the chaos its models seem to be causing. Well, except Hugging Face’s CEO, and, uh, yeah, it seems like OpenAI has currently left him on read. Still, it isn’t just all bad faith. It is true that liability will potentially slow down progress quite a bit. If companies have to take a lot of care to make sure their models don’t spin out of control or damage something especially since this is all completely new , they will likely need to impose far more safeguards than they are right now. To a normal person, that sounds like, “uh, isn’t that what we want?” It might even be confusing why this is such a problem vs. “pacing the frontier” with checkpoints. Well, in the case of liability, the labs get sued after something bad happens. In the case of a checkpoint, so long as everyone agrees that it’s where the checkpoint is, well, if something bad happens… If I were a shareholder in Anthropic I am not , I certainly know which I’d prefer This is especially true because, remember, there is no “AI Section 230.” Following a regulation alone isn’t a defense to a lawsuit. But the Trump Administration’s executive order has been going after individual state AI laws, and the Senate’s new “duty of care” draft, which would actually impose some liability on frontier labs, would still preempt state AI safety laws https://www.techtimes.com/articles/327387/20260912/thune-cruz-klobuchar-move-ai-safety-voluntary-pledge-legal-duty.htm . The Illinois bill went furthest: publish safety protocols, and you have your version of Section 230’s safe harbor. In a way, this is what the labs are after legally. Checkpoints are a means to an end in getting to that legal protection. Even aside from a legal shield, checkpoints, as I described in my last piece https://weightythoughts.com/p/what-does-pacing-the-ai-frontier , can be incredibly malleable. They can be set to exactly where the current incumbent labs are going, and anywhere beyond say, by an upstart, new lab can be forbidden—all in the name of safety. Everyone should only go exactly as fast as the big labs want to. Yet again, bad faith? Maybe not, but it’s such an easy path to making it that. Other Industries Have Survived, Even with Liability I’m a big believer in AI progress benefiting humanity and human societies in the long run. I think most of the models will get commoditized, and the benefits of improved AI will be broadly socialized in an economic sense, not a political sense . Slowing it down does have opportunity costs for society. My sense of it is tech has always resisted liability because it seems scary and unknown—at least far more than imposing known costs upon themselves like Trust and Safety teams, “privacy” regulations, etc. that, if anything, help block new startups from competing with them because they cannot shoulder the high fixed costs of these policies. If you look at many of the prior generations of tech, though, we ultimately saw that as technology matures, we get more set legal especially liability regimes, either through explicit regulation or case law. Cars are the cleanest example, which has also now run its full cycle. In 1916, Buick argued it owed nothing to a driver who’d bought his car from a dealer rather than from Buick. They, rightly, lost https://en.wikipedia.org/wiki/MacPherson v. Buick Motor Co. . In 1963, California made manufacturers strictly liable for defective products. In 1968, GM argued it had no duty to design cars that could survive a crash and lost that too. Somehow, someway, despite being forced to pay for the harms they cause, the car industry survived. Aviation also went through the whole arc which is a bit too long to go over here . And now online platforms. And now self-driving cars… As an industry matures, this is pretty natural. I would hope that paying for the people you harm or kill wouldn’t be enough to kill an industry. Because, if it is, well… one has to question the societal benefits of the industry. Anyway, while liability may be verboten and a bogeyman under tech’s bed, I suspect if and when it comes, it’ll be far less scary than what it’s made out to be. And I do hope that this is the direction that legislation takes, versus, as I described in the last piece, the entrenchment of incumbents that the frontier labs have started to advocate for, whether through Anthropic’s checkpoints or OpenAI’s liability shields. Side Note: On Competition With China The “best” argument most have on avoiding any slowdown is “competition with China.” This is what Trump has brought up, and this is regularly trotted out. As a counterargument, one could say if US labs carry liability and Chinese labs don’t, don’t we just lose? Sure, but that’s overly simplistic. Amodei thinks a global agreement with China “will be much harder to achieve,” per his letter. I think coordination with China on the most dangerous aspects of AI is perhaps the most possible item on his wishlist-that-was-posing-as-a-safety-letter. China is also now a global superpower “incumbent.” They have no desire to see the entire world blown up or entirely upended. If they’re on their way to the top, they have no desire to shake things up and maybe end up in a worse position than now. Thanks for reading I hope you enjoyed this post. If you’d like to learn more about AI’s past, present, and future in an easy-to-understand way, I’ve published a book titled What You Need to Know About AI . You can order the book on Amazon https://amzn.to/4qCERMX , Barnes & Noble https://bit.ly/barnesandnobleaibook , Bookshop https://bit.ly/bookshopaibook , or pick up a copy in-person at a local bookstore https://www.smartaibook.com/buy . P.S. I’ve got a small stack of signed hardcovers available on Amazon Prime right now—buy from the seller “ WeightyThoughts “ and yours comes signed by me, with a matching gold bookmark. It’s a limited run, so once they’re gone, they’re gone.