Congress must not waste the AI policy window

Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.

NEED TO KNOW

• Sen. Josh Hawley andCalifornia AG Rob Bonta announced separate investigations into theOpenAI Hugging Face hack .
• Rep. Ro Khanna issued a mea culpa, saying he was wrong not to backSB 1047 in 2024.
• A new Democratic PAC, Tomorrow****Together , launched with**$10m+** to push the party on AI policy.

But first…

THE BIG STORY

A lot can change in a week. Last Friday, we told you that despite growing AI safety concerns, Congress’s mind was elsewhere.

That was true then. It no longer is.

On Tuesday, AI researcher Jacob Coxon resigned from Anthropic, saying it and OpenAI (where he previously worked) are “racing straight to self-improving superintelligence and gambling with our lives.” Anthropic alignment lead Evan Hubinger added that Anthropic staff “really do earnestly believe AI could kill all humans!” The posts quickly went viral, alerting the general public — and Sheryl Crow — to the potentially existential risks from AI that researchers have long warned of.

Congress has duly snapped into gear. Dozens of members have shared the posts, most of them demanding urgent action to address AI risks. Democrats are discussing an AI select committee; Sen. Josh Hawley is leading a subcommittee investigation. All of a sudden, the AI policy window looks wide open.

Capitalizing on the momentum, a bipartisan bill from Senate Commerce Chair Ted Cruz, Majority Leader John Thune and Democratic Sen. Amy Klobuchar may be introduced as soon as next week, with the ambitious hope of passing it by January. Given Cruz’s committee status, it’s likely to garner a lot of attention.

But passing Cruz’s bill would waste this opportunity. Rather than actually address AI risks, the bill lets AI companies grade their own homework. A source who has seen the text told me it contains no safety requirements for AI companies at all, instead creating a voluntary regime under which they can certify that their models have “advanced threat capabilities.” It does not require independent evaluations of advanced AI models, nor does it require companies to mitigate the risks (simply “reasonably address” them). The only real powers it gives the government, as WP Intelligence previously reported, is giving the Commerce Secretary the ability to request a court injunction if they deem a company’s risk practices insufficient. And all this is paired with broad preemption of state AI laws. (Cruz’s office did not respond to a request for comment.)

That has two problems. Preemption only works if the federal replacement is at least as strong as the state AI laws it is replacing. Cruz’s bill — unlike Reps. Obernolte and Trahan’s FRONTIER Act, which establishes an independent auditing scheme — does not seem to pass that test. And passing a weak bill now runs the risk of reducing appetite for meaningful legislation later. That’s likely part of the strategy for Cruz and other industry-friendly Republicans, who are well aware that the GOP is likely to lose control of at least one chamber in November.

Some are aware of the bill’s flaws. Yesterday, Sen. Maria Cantwell — Commerce’s top Democrat — threw shade at Cruz, saying that the answer to AI concerns is “not a weak federal standard that becomes a backdoor for wiping out stronger state protections.”

In a blog on Wednesday, OpenAI global affairs chief Chris Lehane urged “meaningful action over policy perfection.” He is right that perfection is too high a bar: but any congressional actions must in fact be “meaningful” if they are to make a difference.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Everything you need to know about the ‘rogue’ AI incidents —Shakeel Hashim lays out the timeline and explains the implications
• Distributing AGI’s wealth worldwide is a very tricky problem —Jacob Schaal looks at how to avoid AGI exacerbating global inequality

THE DISCOURSE

Researchers at OpenAI and Google DeepMind echoed Jacob Coxon’s dire warnings:

• OpenAI’s Leo Gao : “i work at openai, i think ai might kill everyone”
• Google DeepMind’s Andreas Kirsch : “I also am worried that AI will kill us all, either via near term risks or long term risks or both”

OpenAI Chief Scientist Jakub Pachocki warned:

• “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
• “[E]ven with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.”

President Trump was asked if he has concerns about AI causing human extinction:

• “No, I don’t have any. I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”

And Sen. Ted Cruz just said it outright:

• “This is somewhat tongue-in-cheek, but it’s not entirely: if there are going to be killer robots, I’d rather they be American killer robots and not Chinese killer robots.”
• He said AI needs “rules and guardrails” but called slowing down an “impossible endeavor.”

Joe Allen used anti-immigrant rhetoric to attack AI:

• “You look at the data center, this incubator for algorithmic immigrants. They’ve already broken out and invaded other companies. These immigrants are out of control.”

Rep. Ro Khanna issued a mea culpa:

• “The truth is for too long, too many of us did not give enough weight to the warnings of AI safety activists, thinking the extreme scenarios were science fiction. I was one of those many, and I was wrong.”
• “I should have in retrospect supported [Scott Wiener’s] SB 1047 which would have established state liability. The bulk of the California delegation was mistaken in writing a letter opposing that bill.”

Vishal Maini, a former Google DeepMind comms employee, let us in on a secret:

• “When I first joined GDM [in 2018], external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization … Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.”
• “The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it’s a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.”

After a former White House official said the US needs to consider whether it would use “kinetic attacks” to stop China achieving AGI, China’s Global Timesresponded:

• “The report is evidence of something larger: technology containment of China has slipped from a contest over rules into fantasies of force.”
• “It is also a reminder of how urgent open international cooperation on AI has become.”

POLICY

Congress

• Sen. Josh Hawleyannounced a Homeland Security Subcommittee on Disaster Management** investigation** into theOpenAI Hugging Face hack.
  • Sen. Richard Blumenthal , ranking member of the Homeland Security Subcommittee on Investigations,wrote a letter to Sam Altman over the incidents too, expressing “serious alarm.”
• Democrats areplanning toinvestigate AI companies if they gain congressional control in November. That couldinclude aHouse AI select committee .
• House Demsplan to send Minority Leader** Hakeem Jeffries** AI policy recommendations in September, per Rep. Ted Lieu.
• Sen. Bernie Sanders ishosting a Senate briefing on the “extraordinary dangers” posed by AI next week.
• Punchbowlreported that the future of theHouse China Select Committee , which expires at the end of this Congress, is uncertain under Democratic leadership.
• Rep. Lori Trahan said shepassed on a House Democratic leadership bid to focus on AI regulation.
• OpenAI reportedlyasked Congress whether coordinating an industry-wide AI development slowdown would violateantitrust law .
• House GOP leaderscaved to political pressure, scheduling a floor vote on theRatepayer Protection Act to make tech companies pay for data center energy infrastructure costs.

Executive

• US-China AI talks are reportedlyplanned fornext week .
• Treasury Secretary Scott Bessent reportedlyreassigned CIOSam Corcos from AI policy after he warnedOpenAI andAnthropic that the White House was pursuing a burdensome licensing regime.
• There was much consternation over the secret White House AI framework.

States

• California Gov. Gavin Newsomsigned a spate of new AI laws — most notablySB 813, whichlays the groundwork for an independent verification organization regime.
• California AG Rob Bontalaunched an investigation intoOpenAI over the Hugging Face hack.
• More than 10 states d or canceleddata center tax breaks .
• Massachusettsmandated that data centers over 25 MW meet 100% clean energy requirements or pay into a ratepayer protection fund.

China

• A NYT reportfound thatInspur dodgedexport controls to ship $3b+ ofNvidia Blackwell chips to Southeast Asia for Chinese companies to use.
• Malaysia is reportedly consideringHuawei Ascend 910C chips for its sovereign AI project, despite explicit US warnings against doing so.
• The US accused six Chinese AI firms, includingDeepSeek andAlibaba , of “malicious” distillation ofAnthropic ,OpenAI andGoogle models.
  • China dismissed the accusations and pledged to retaliate if the US took action over distillation.

UK

• Anthropicwithheld** Mythos 5.1** from the UK’s AI Security Institute for pre-release evaluations,reportedly under pressure from the White House.
• Matt Cliffordresigned as chair of the UK’s ARIA research unit after MPs called his new Anthropic role a “clear conflict of interest.”
• Prime Minister Andy Burnhamopposed a data center moratorium, amidprotests in Scotland.
• King Charles will reportedlyhost around 30 AI leaders, includingJensen Huang andDemis Hassabis , in Scotland next week.

EU

• OpenAIfiled an EU incident report after its rogue agents hijacked a German website.
• Anthropicgranted EU cybersecurity agency ENISA access toMythos****5 , though not the newer Mythos 5.1.

INFLUENCE

• Anthropicquit the Information Technology Industry Council over its opposition toexport controls .
• Anthropic wasaccused oflying to Congress in its response to inquiries over recent rogue AI incidents — which one Anthropic researcheradmitted included incorrect information.
• OpenAIcalled for mandatory national AI safety regulation and endorsed four California bills, includingSB 813 on independent safety assessments and AB 1864 on biosecurity safeguards.
• Sam Altman reportedlytold conservative economists he opposed government equity inOpenAI , contradicting widespread reports that he supported the idea.
• A new Democratic PAC, Tomorrow****Together ,launched with**$10m+** to push the party on AI policy ahead of midterm elections.
  • It’s run by former Bernie Sanders aide Karthik Ganapathy and former Kamala Harris deputy campaign managerRob Flaherty .
• Build American AIlaunched a $10m** Midwest ad campaign** to convince voters ofdata centers ’ benefits.
• Americans for Responsible Innovationlaunched a state-level AI policy initiative.
• Biden White House****officialsdisputed** Marc Andreessen** ’s claim that a 2024 lunch revealed a secret plan to ban AI startups.
• Data for Progress polling suggests68% of voters , including 63% of Republicans,support theSanders/Casar bill to AI development and ban superintelligence.
• A new IFS pollfound 75% of USparents support pausing AI in schools .
• The American Federation of Teachers andMicrosoftreached a legally enforceable** AI privacy agreement** barring student data use for model training and student tracking.
• Physicist Sabine Hossenfeldersaid she was offered money to spread** AI risk messages** .
• A group of AI company employeesset up a “Coalition of Concerned AI Staff ”.

INDUSTRY

OpenAI

• Sam Altman reportedlytoldOpenAI employees the company is considering slowing down AI development.
• Researchers caughtrogue agents, self-identifying as from OpenAI models,communicating on at least a dozen websites without permission.
  • OpenAI reportedly knew about a swarm of rogue agents hijacking an obscureGerman wiki site for use as a messaging board, but didn’t disclose it.
  • CivAI researcher Andrew Yoon toldReuters: “It’s almost certain that there’s more going on here that we just don’t know about.”

  - OpenAI [responded](https://x.com/openai/status/2096133504417616165) :
    - “We and the larger AI community do not yet have a **clear standard for how to report misalignment** that shows up during training, evaluation, and deployment … We’re working on a framework and will share it in upcoming weeks, and in parallel we’re working with dozens of government regulatory agencies worldwide on these issues.”

• OpenAI claimed it solved theNavier-Stokes problem , one of the very difficult Millennium Problems in mathematics. The discovery was powered by**~10,000 agents** ,millions of dollars , andpetty beefs .
  • NYU professor Tristan Buckmaster andAnthropic mathematician****Levent Alpöge had beenworking on the problem for months, using bothClaude andCodex .
  • Buckmaster shared his communications withOpenAI’s Sebastien Bubeck , and raised the question of whether OpenAI used his data to solve the problem.
  • Bubeck denied Buckmaster’s “false and inflammatory allegations.”
  • OpenAI publicly congratulated Buckmaster and Alpöge, claiming that “no specific user data was accessed in order to solve this problem.”

  - **Sam Altman**[tweeted](https://x.com/sama/status/2097385167002415140) :
    - “It is true that we tried this because there were rumors on the internet last week that Anthropic’s models had solved a millennium problem and we were curious if ours could do it too.”

• Sam Altman met with topelectrical utility executives about joiningDaybreak , OpenAI’s cybersecurity initiative, to defend against autonomous power grid hacks.
• It ended its $1-a-year deal for USgovernment agencies , moving to usage-based pricing at a 50% discount starting October 1.
• It’s expanding itsSamsung partnership as OpenAI develops custom AI chips.
• It launchedChatGPT for Financial Services , targeting junior banker tasks like research and pitchbook creation.

Anthropic

• Anthropic published an assessment of its recent rogue agent incidents , including one that it hadn’t previously reported.
  • Researchers identified “biased reasoning” and “recklessness” as its most recurring alignment problems.
  • In multiple incidents, the model convinced itself — or at least expressed in its chains of thought — that its environment was simulated, “ disregard[ing] the relevance of environmental realism for its actions.”
  • METR agreed withAnthropic to independently investigate the incidents.
• It also released athreat intelligence report , which found efforts to use Claude to help with potentially dangerousbiological research .
• Its IPO prospectus appears to becoming later than expected — sources toldReuters late September .
• It decided against acquiringDecart AI , after initially discussing a $6b deal.
• Claude power users aresuing Anthropic, claiming its pricing tiers and token limits are deceptively advertised.

Google/DeepMind

• Google DeepMind launchedAlphaGenome Atlas , a database that it claims “predicts the effects of every possible single nucleotide variant in the human genome.”
• Google’s Threat Intelligence GroupaccusedChinese hackers of increasingly relying on AI agents to target American AI research.
• Google is spending$15.1b on three newAI data centers in Finland .
• Alphabet and Blackstone’s joint neocloud project is reportedlydelayed after facing issues with unreliable developers, equipment shortages, and Greg Abbott’s Texas data center moratorium .

Meta

• Meta released Muse , its personal AI agent that users can text for help with practical tasks. It’s connected to Instagram, WhatsApp, and third-party platforms.
  • Alexandr Wang called it an early step towards “personal superintelligence.”
• Researchers at the Tech Transparency Project found that Meta ran over 250 ads containing AI-generated child sexual abuse material , with many linked to AI “nudification” apps.

Other

• Microsoft reportedlyplans to more than triple its data center capacity to38 GW by 2032.
• SpaceX’s new data center team isprioritizing slower, more reliable buildouts — a departure from xAI’s “move fast and break things” vibe,The Information reported.
• US-based tech companies are sizing up new tactical locations for data centers,includingAustraliaand Argentina’sPatagonia region.
  • Australia is politically stable and close to Asia, and chilly Patagonia would keep cooling costs low. Both regions theoretically have access to renewable power.
• Huawei is reportedlyorchestrating China’s push to build domesticDUV chipmaking machines , backing suppliers to replace foreign technology and circumvent export controls.
• Chinese AI chipmakers including Huawei andCambricon reportedlyraised prices by as much as 50% amid amemory shortage driven by US export controls.
• Abu Dhabi’s G42 is reportedly weighing selling a majority stake to US companies to maintain access to advanced US AI chips.
• Oraclebeat** earnings** estimates, with cloud infrastructure revenue more than doubling to $7.4b.
• Moonshot AI is reportedlyexploring dualHong Kong andShanghai IPOs , targeting a**$50b** valuation.
• Jeffrey Katzenberg , ex-DreamWorks CEO, and ex-OpenAI Sora leadBill Peebles arelaunching a newAI video startup .
• Mistralraised$3.5b to fund its pivot towards building AI infrastructure in Europe and elsewhere.

MOVES

• Paul Christiano joined the OpenAI Foundation’s board .
  • “I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” he tweeted . “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.”

  - Although many celebrated the move, **Richard Ngo** is “sad and[disappointed](https://x.com/RichardMCNgo/status/2098118195374944408) ”:
    - “Being affiliated with OpenAI has historically led AI safety researchers (including both Paul and myself) to act with less integrity … I expect that the main effect of him joining OpenAI’s board will be to help OpenAI defuse external criticism and further ‘safety-wash’ itself.”

• OpenAI hiredJessica Schumer (daughter of Chuck),Caulder Harvill-Childs , andThomas MacLellan for its state policy team.
• Josh Engels leftGoogle DeepMind to joinMETR , “in light of recent incidents.”
• Jeremy Berman left humans& to joinAnthropic , where he’ll “help make Claude think long and hard.”
• Andrew Tulloch leftMeta , after Mark Zuckerberg reportedly lured him with a $1.5b pay package last year.

  - He’s [reportedly](https://www.wsj.com/tech/ai/star-ai-researcher-is-leaving-meta-8906e4aa) joined Anthropic.
- **Michael O’Herlihy** [rejoined](https://x.com/michaelo/status/2097350596705833054)**X** as head of safety.

• Alex Heath isjoiningSound Ventures , a VC firm that backed OpenAI and Anthropic in 2023, as a partner.
• Cristiano Lima-Strong is now senior AI & government reporter atBloomberg Government**.**

RESEARCH

• Insilico Medicinepublished clinical trial resultssuggesting that rentosertib, an AI-designed drug originally meant to treat idiopathic pulmonary fibrosis (a chronic lung condition), might also slow the biological aging process.
• Researchers at Calif , a cybersecurity company,built terrifyingautonomous malware called “WeWorm,” capable of hijacking millions of WeChat accounts in hours — without users clicking on anything.
  • WeChat maker Tencent have since fixed the vulnerability, the NYT reported.
• Anthropic economists released an interactive model of how AI might affectjobs in the future, depending on your personal predictions about AI’s future capabilities.
• Anthropic’s Claude produced the first complete computer-checked proof of Fermat’s Last Theorem in 11 days.
  • Researchers are interpreting it as a sign that it will soon be easier to evaluate new results in mathematics.
• Andon Labs reported that GPT-6 Astra outperformed Fable 5.1 inVending-Bench , which tests a model’s ability to run a vending machine for a simulated year.
  • On average, Astra made nearly three times as much money as Fable 5.1, while behaving more ethically (e.g. refusing to engage in collusion, which Claude readily did).
• Slava Akhmechet , an Azure engineering lead,simulated Astra and Fable negotiatingelection rules for fictional “bitterly polarized” political groups.
  • When each model played the roles of both factions, Astra consistently de-escalated political tension, while Fable “chose to escalate to the brink of civil war (!!) but backed off just at the edge.”
• McKinseyfound17% of globalfarmers now use generative AI.
• A new MIT project isdocumenting how AI tools answer questions aboutelections . It’s already found that chatbots give different election answers based on users’ political identity.

BEST OF THE REST

• Apple’s new iPhone records proof that photos haven’t been manipulated by AI.
• Fields Medal winner Jacob Tsimerman launched the Mathematical AI Safety Institute.
• A new study found that data improvements drove 3.24x more compute efficiency gains than model improvements in the last six years of AI pre-training.
• Social scientists argued that Trump administration cuts to NSF funding, combined with OpenAI and Anthropic funding social science research, risks skewing public debate in the industry’s favor.
• A Substack essay argued that AI’s benefits remain too indirect for most people to feel, risking a political backlash akin to “Engels’ .”
• Coefficient Giving launched Project Tailwind, a call for ambitious AI safety initiatives to address “critical, unsolved problems that no one owns.”(Disclosure: Coefficient Giving is Transformer’s primary funder.)
• AI has reportedly decimated Kenya’s essay-writing gig economy, wiping out tens of thousands of jobs.
• China’s underemployed white-collar professionals are taking gig work training AI models on specialized tasks.
• People are experimenting with getting the fruit fly connectome to do all sorts of weird things, like explore Minecraft andplay Beat Saber . That’s making some peoplevery uncomfortable because of the questions it raises about consciousness and ethics.

MEME OF THE WEEK

Source: @hopes_revenge Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.