Physicist Suvrat Raju, who coordinated the TheoPhysEval project, talking with students (file photo). Image: 11thdimensionphysics / Wikimedia Commons, CC0 1.0, cropped
OpenAI’s GPT-6 Astra worked out the answers to 49 of 50 unpublished theoretical physics research problems on its own, with no human hints, at an average cost of under $3 a problem. That is the headline result of TheoPhysEval, a test built by 35 physicists and AI researchers and posted to arXiv by a team coordinated by Suvrat Raju of the International Centre for Theoretical Sciences (ICTS) in Bengaluru.
The authors, from institutions including ICTS, Harvard, the University of Pennsylvania, KU Leuven and Google DeepMind, say the problems are ones a world expert in the subfield could typically solve in about a day or less, but that would take “considerably longer” for experts in neighbouring areas.
50 problems nobody had published #
Each problem came from a specialist and is drawn from current research, covering 11 areas from classical physics and lattice theories to quantum field theory, holography and cosmology. The paper calls them “turn-the-crank” problems: known techniques applied to extend an existing result. The answers are new, sometimes publishable, but not breakthroughs: contributors were told to publish anything important themselves.
To stop models guessing, every answer was turned into a number, with agents given numerical values for both relevant and irrelevant quantities. The team says none of the answers was public when the tests ran, though it cannot rule out leakage, since some experts used AI tools while solving their own problems.
How the AI agents scored #
Five off-the-shelf agents ran each problem once, with up to eight hours apiece, on two tracks: a “realistic” one with web access, literature search and the ability to write code, and an “isolated” one with the internet switched off.
- GPT-6 Astra in Codex: 49 of 50 (98%) realistic, 46 isolated. The whole realistic run cost $116.28 in API fees.
- Claude Opus 5.5 in Claude Code: 46 of 50 on both tracks, but at $240.43 for the realistic run, about twice Astra’s bill.
- Gemini 3.8 Flash in Antigravity: 34 realistic, 31 isolated.
- DeepSeek-V4.1-Flash in Codex: 27 and 25.
- GLM 5.3 in Claude Code: 21 and 26.
The one problem Astra missed, rebuilding a differential equation from the curve it produces, beat every model. Without the internet, Astra and Opus both scored 92%. The authors read the small gap between the tracks as a sign that leading models “have effectively memorized large parts of the theoretical physics literature”.
There was also an awkward twist for the humans. The team first marked Astra wrong on several problems, then found “subtle errors in the original solutions” written by the experts. Once those were corrected, Astra’s answers were right.
The authors put it plainly:
Our results indicate that some frontier models may have already surpassed the best human researchers in their ability to perform concrete calculations, even in specialized topics.
Suvrat Raju and co-authors, TheoPhysEval paper
They also note that each model ran only once, so scores carry some statistical noise, and they invite others to rerun the test from the files they have shared on Hugging Face. The same pattern of AI doing the heavy lifting in research has been showing up in maths, where OpenAI posted 722 AI-written papers this month and mathematicians say they can’t stop using the tools that unsettle them.
“Stop and think” #
So why hasn’t an AI agent made a physics breakthrough yet? The paper argues that calculating is only a late step in theoretical physics. First comes building theories and asking the right questions, and there the authors say current agents fall short. They struggle to switch to a simpler, equally interesting problem when the first one proves too hard, and when asked to invent problems, they fixate on details a human would find dull.
“At the moment, agents do not appear to have the same ‘taste’ in problems,” the authors write, adding that human experts are still needed to set the direction. They admit this part rests on their own experience and can’t be measured. Physics has long told students to “shut up and calculate”, they note, but as taste and vision matter more, physicists “might need to ‘stop and think’”.
That echoes Harvard physicist Matthew Schwartz, who used Claude to write 36 papers in three months while warning it “loves to declare victory”. The authors also flag a problem for training: some of these problems were the kind once handed to early-career researchers as first projects, and agents can now do them in minutes.
Why it matters #
Benchmarks built from textbook or competition questions are easy to dismiss; this one uses fresh research problems written and checked by the experts who know them best, and the top agent beat some of their own answer keys. The open question is no longer whether AI can do a physicist’s calculations, but whether it can choose which ones are worth doing.
Sources: TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving (arXiv).