Frontier Math and a review of 800 math papers show AI catching mistakes experts missed. Here's what that means for research and benchmark design.
What’s happening with AI and math errors? #
AI models are increasingly catching mistakes in human-written math benchmarks and published research papers, not just solving the problems themselves. Epoch AI, the group behind the widely used Frontier Math benchmark, ran an AI-assisted review of its own test and found that a large share of the problems had errors serious enough to invalidate the official answer. Separately, a research team used an OpenAI model to scan nearly 800 math papers for flaws in their central claims, and found meaningful error rates even in peer-reviewed-style work. Together, these cases suggest AI isn’t just taking tests anymore. It’s starting to grade them.
TL;DR #
- Epoch AI’s Frontier Math benchmark , built by professional mathematicians to be one of the hardest math tests in existence, had an AI-assisted review flag fatal errors in roughly a third of its problems, with most of those flags judged valid.
- A review of nearly 800 math papers using GPT-5.1 Codex found about 6% had major errors that undermined the paper’s central claim, and another 30% had minor fixable mistakes, at a computing cost of around $2,500 for the whole batch.
- Error rates dropped sharply in recent papers , falling to roughly 1.5% for major errors during a period when over a quarter of papers acknowledged using AI, though the researchers stopped short of calling that causal.
- The method is simple in concept : the AI hunts for counterexamples, specific cases that break a paper’s “this is always true” claim, then shows its work so a human can verify it.
- A related physics benchmark, TheoremsPhysEval, saw a similar pattern , where GPT-5.1-class models were initially marked wrong on several problems before experts discovered their own answer key had subtle errors.
- Benchmark saturation is no longer the only problem . Tests used to get beaten and replaced with harder ones; now the people writing the tests are becoming a bottleneck, since they can get the answers wrong before the AI even sees the question.
pass the audit.
PASS
PASS
PASS
PASS
PASS
PASS
0
How did Epoch AI find errors in Frontier Math? #
Frontier Math launched in late 2024 as a deliberately brutal test: problems written by professional mathematicians, designed to take human experts hours or days, intended to resist AI progress for years. At launch, top models solved less than 2% of it.
By May, Epoch AI reported running an AI-assisted review of the benchmark itself. The review flagged fatal errors in about a third of the problems, and the team said they believed most of those flags were valid. That’s a striking number for a benchmark built specifically to be rigorous and resistant to shortcuts. It means that for roughly one in three questions, the “correct” answer that models were being scored against may have been wrong, which in turn means every model’s reported score on Frontier Math was likely distorted in ways nobody had accounted for.
This matters beyond one benchmark. Nearly every claim about how capable a given AI model is rests on benchmark scores. If a benchmark’s answer key has baked-in errors, models that got the right answer were being marked wrong, and the field has probably been underestimating real capability in some cases while overestimating it in others, depending on which errors went uncaught.
What did the 800-paper math review actually find? #
A team led by researchers including Jasper Dekoninck and Martin Vechev used OpenAI’s GPT-5.1 Codex to check nearly 800 math papers posted online for errors in their main results. The approach leans on a basic property of mathematical proof: if a paper claims something is universally true, a single counterexample disproves it. The AI’s job was to read each paper, extract its central claims, search for a case where the claim breaks, and then produce a verifiable counterexample rather than just asserting a mistake exists.
For papers written before August, about 6% had major errors, meaning the core claim was false and needed significant rework, and another 30% had minor errors fixable with small corrections. The research team cross-checked the AI’s findings and estimated a bit under 90% of the major-error flags were accurate. The entire run cost around $2,500 in compute for hundreds of papers, far cheaper than hiring human reviewers to manually check even a fraction of that volume. The more interesting wrinkle: papers from the most recent months, when more than a quarter of submissions acknowledged AI assistance, showed major error rates drop to about 1.5%. The researchers were careful not to claim AI caused the improvement outright, since other factors could explain it. But the pattern is suggestive. AI tools may be catching errors before papers are even submitted, not just after publication.
Why does the physics benchmark story matter too? #
A separate test called TheoremsPhysEval, built by researchers from institutions including Harvard, the University of Texas, and Google DeepMind, consisted of 50 unpublished, research-level theoretical physics problems, the kind a specialist might need a full day to solve. Because the problems were unpublished, there was no risk of a model having memorized answers from training data.
Enterprises have Remy.
⌂ Self-hosted
✓ SOC 2
✚ HIPAA
☰ Audit log
When a model referred to in coverage as GPT-5.1 Astra was tested inside OpenAI’s Codex environment, with tool access to search papers and run verification code, it initially scored around 88%. Researchers then found that several of the model’s “wrong” answers were actually correct, and the official solutions contained subtle errors. After consulting the physicists who wrote the problems and correcting the answer key, the model’s score rose to 49 out of 50, a 98% result. The paper noted the model still failed on some problems, particularly without tool access, and one problem went unsolved by every model tested.
The pattern echoes Frontier Math closely enough that OpenAI researcher Noam Brown publicly connected the two, suggesting physics benchmarks might be following the same trajectory math benchmarks already have.
How are researchers actually using AI to find these errors? #
The mechanism is consistent across these cases. Rather than asking the AI to generate a free-form judgment of whether something is right or wrong, researchers design workflows where the AI has to produce evidence a human can check. In the paper-review study, that meant generating explicit counterexamples. In the physics benchmark, the model used code execution and literature search to verify its own calculations before flagging a discrepancy.
This verifiability is what makes the error-finding credible rather than just another case of an AI confidently asserting something false. A counterexample is either valid or it isn’t. A code check either confirms a calculation or it doesn’t. That structure lets humans stay in the loop as verifiers even when they can’t independently solve the original problem as fast as the model can.
Is AI actually better at math than human experts now? #
Not uniformly, and the evidence doesn’t support that broad claim. The physicist Gabriele Carcassi, who described using AI to resolve a year-old conjecture in about two minutes after a human mathematician had worked with him on it for six months, also described the limits plainly: the AI is not a genius, and everything it produces needs checking. What it has is breadth. It can draw on techniques across fields that no single human specialist would necessarily know to apply.
That caveat matters for the error-finding trend too. In the paper review, roughly 10% of the AI’s “major error” flags turned out to be wrong on closer inspection. In the physics benchmark, the model still missed one problem entirely and performed worse without external tools. The pattern across all these cases is an AI that’s extremely good at generating candidate errors or solutions quickly, paired with a requirement that humans still verify the output. The value isn’t infallibility. It’s dramatically lowering the cost of checking large volumes of work that would otherwise never get checked at all.
What does this mean for how we test AI going forward? #
It creates a genuine measurement problem. Benchmarks work by comparing AI answers to a human-made answer key. When a model disagrees with that key, the historical assumption was that the model is wrong. That assumption is breaking down. Researchers now have to consider the possibility that an apparent AI failure is actually an error in the test itself, which means verifying benchmarks has become almost as important as building them.
demandsof enterprises.
Passed
*✓*SOC 2 Type 2
*✓*HIPAA
*✓*CAIQ attestation
*✓*OCSF logging
*✓*Role-based access
*✓*Self-hosted
One proposed direction is leaning harder on problems with automatically checkable answers, such as code that either executes correctly or doesn’t, or formal math proofs verified by proof-checking software with no ambiguity. That removes the human answer key as a point of failure. It doesn’t fully solve the problem for open-ended research questions, where there’s no automatic checker, but it’s a concrete way to keep scoring honest for the problem types where it’s possible.
Frequently Asked Questions #
What is Frontier Math and who created it?
Frontier Math is a benchmark created by Epoch AI, made up of problems written by professional mathematicians and designed to be extremely difficult, some taking human experts hours or days to solve. It launched in late 2024 specifically to resist quick AI progress.
How many errors did Epoch AI find in Frontier Math?
An AI-assisted review flagged fatal errors in roughly a third of Frontier Math’s problems, and Epoch AI said it believed most of those flags were valid, meaning the official answers for those questions couldn’t be trusted as written.
What percentage of math papers had errors in the 800-paper study?
Using GPT-5.1 Codex, researchers found about 6% of papers written before August had major errors undermining their central claim, and another 30% had minor, fixable errors. Error rates for more recent papers dropped to around 1.5%.
Does this mean AI is smarter than human mathematicians?
Not exactly. The researchers and scientists involved emphasize that AI outputs still require human verification and the models make mistakes, including incorrect error flags. What AI brings is broad cross-domain knowledge and the ability to check large volumes of work cheaply, not infallible judgment.
Why is it hard to test AI models that might outperform their human testers?
If an AI disagrees with a benchmark’s official answer, it’s no longer safe to assume the AI is wrong, since the answer key itself might contain errors. This is pushing researchers toward benchmarks with automatically verifiable answers, like executable code or formally checkable proofs, rather than relying solely on human-written answer keys.