Caltech-trained physicist Casey Handmer on November 17, 2026 published an AI-assisted audit of theoretical astrophysics that reads like a product demo for science, not a press release. Starting from 158,059 papers and 554,670 citations around Caltech's TAPIR group, the site screens 50,269 open full texts, recomputes 659 highly cited works, and traces 872 upheld errors through papers that copied them — often years after someone already objected in print.
For AI builders, the story is **workflow economics**: cheap **screening** at field scale, expensive **verification** only where citations concentrate, and **citation-graph follow-through** humans rarely fund. For policymakers, it is **evidence that peer review catches only a slice** of mistakes that stay in PDFs anyway.
## TL;DR: what the map claims
| Question | Answer from Handmer's site |
|---|---|
| What corpus? | 158,059 papers seeded fromTAPIR (4,093 papers by 326 members since 2004) plus citation closure |
| What was screened? | 50,269 papers with open full text (76% of the drawn map; pre-1991 mostly database-only) |
| What was recomputed? | 659 papers with line-by-line model review and second-pass confirmation |
| Uphold rate | 872 errors in433 papers after dual reviewers |
| Propagation | 367 papers copied an error; only68 criticized one in print (~5:1 copy-to-criticism) |
| Cost (author estimate) | ~$2,720 list API price;~$3 per verified error;~44 hours human time over41 days |
| Fraud scan | No paper-mill language;no hidden reviewer prompts;13 retractions in DOI set |
Why Handmer built this #
Handmer opens with a numerical relativity example: a 1997 gravitational-wave paper carried known sign errors for decades while fixes lived in email threads, not the PDF students cite. His thesis work, he writes, could only patch the record in his own papers — the standard human limit when nobody is paid to re-derive every cited equation.
The AI pitch is symmetrical: read the whole field, recompute flagged lines, and follow each mistake forward through citing papers. Handmer says the pipeline also flagged errors in his own work.
That framing differs from consumer AI literature review tools that summarize abstracts. The method page (linked from the hero site) describes staged model roles, tool access for recomputation, and explicit human-in-the-loop limits — though, as HN commenter sudo_cowsay noted, the landing page does not prominently name which model family ran each stage.
What the screening pass actually checks #
Before expensive recomputation, the project runs cheap fingerprints on every open full text:
-
Paper-mill phrase patterns
-
Chatbot residue and hidden instructions to AI reviewers
-
Suspicious text overlap between unrelated author lists
-
Retraction status against DOI records
-
Unfinished-draft markers (such as FIXME notes) shipped to arXiv Handmer reports astrophysics looks relatively clean on fraud: 43 suspicious overlap pairs queued for human review, mostly successive survey catalogues, not identity theft. The needle that does move is vocabulary: words like "delve" and "crucial" appear 2.5× more often in 2025 papers than in 2016 on his chart — a soft signal readers already argue about in generative-AI authorship debates.
Deep read: 659 papers, dual reviewers #
The expensive stage targets citation inequality: 200 top TAPIR-cited papers (2006–2019) plus 459 additional high-signal works. Each deep read gets:
- LaTeX or text source
- Tool use to re-derive equations and rebuild numbers from tables
- A second independent reviewer — flags alone are not findings
Handmer reports 2,623 problems flagged initially, 84 major, with 56 standing as full or partial errors after confirmation. Thirteen errors were fixed between arXiv preprint and journal version — peer review removing only about one in five errors in that subset, by his accounting.
Two shapes inspecting one paper beside a green stamp, illustrating dual-model confirmation in Casey Handmer's physics literature audit
Field-scale literature maps depend on cheap screening, then expensive line-by-line recomputation on high-citation nodes.
The Smith–Kamionkowski–Cooray factor-of-four story
The site spotlights a 2005–2014 propagation case: a factor-of-four mistake in an inflationary gravitational-wave energy-density conversion. A 2008 paper criticized it, yet six copy papers kept the wrong factor until 2014. Handmer animates the timeline on the live map — a concrete example of slow correction even when objection exists.
At the field level, he estimates ~34,000 likely errors among papers cited 100+ times if the 4% deep-read sample scales — explicitly a floor, not a census.
What developers are saying on Hacker News #
The HN discussion on November 18, 2026 mixes praise and skepticism:
- spongebobstoes argued the visualization could bereplicated field-by-field and thatgrad students with modest AI subscriptions might clearlow-hanging correction fruit .
- jdlshore called the landing copyAI-toned and bounced before the method depth.
- joe_the_user questioned whether**"the 4% that mattered"** was defined clearly enough to trust extrapolation.
- lexandstuff joked aboutClaude-style "nobody" rhetorical ticks in the prose — a reminder thatvoice detection is now part of reader trust.
- swampdeer reportedheavy Firefox lag on the interactive map — a deployment lesson forhuge client-side graphs .
None of these comments verify or falsify the error list; they surface UX, disclosure, and rhetoric risks for the next team that ships a field audit.
How this compares to other AI-for-science stacks #
Handmer's work sits beside — not against — institutional efforts:
| Thread | How Handmer differs |
|---|---|
| Nvidia Nemotron for Science | Vendor agents optimizing new experiments ; Handmer optimizesretroactive literature hygiene |
| [Sage S2 science agent](https://explainx.ai/blog/sage-s2-1-flash-shanghai-ai-lab-science-agent-november-2026) | Forward **hypothesis generation** ; this is**backward graph tracing** |
| [Epoch AI cyber eval agents](https://explainx.ai/blog/epoch-trajectory-gemini-4-argon-live-website-impersonation-report-2026) | Safety **red-team** on live sites; Handmer**red-teams PDFs** |
| [OpenAI CoT monitorability](https://explainx.ai/blog/openai-cot-monitorability-model-lifecycle-november-2026) | Watches **model reasoning traces** ; Handmer watches**human paper trails** |
If you run **RAG over papers** in production, the Handmer map is a warning: **retrieval amplifies stale mistakes** unless your pipeline includes **version-aware citation checks** or **primary-source recompute** on high-impact claims.
Can builders copy the pipeline? #
Handmer links a Method page describing stages, cost, and known failure modes. Treat it as a reference architecture, not a hosted product:
- Seed graph from a real collaboration you understand (not arbitrary keywords).
- Screen everything with deterministic cheap checks before anyopus-class spend.
- Spend deep tokens only on high in-degree nodes — citations are the pagerank of truth maintenance.
- Trace forward through citing DOIs; correction value lives indownstream copies , not the original typo.
- Publish uncertainty : Handmer notes40 random errors sent to a third reviewer found33/40 — some listed errors will fall on human challenge.
Research papers stacked for review, echoing citation tracing in Casey Handmer's AI astrophysics audit
Tracing citations matters because most copied errors never received an in-print criticism.
Practical limits Handmer admits
- 24% of mapped papers lack open full text — mostlypre-arXiv or paywalled journal PDFs.
- No human has checked each listed error yet; the site invites corrections.
- Interactive map performance may exclude low-power clients.
- Model disclosure on the landing hero is thinner than safety reviewers want — read the method appendix before citing dollar figures in procurement docs.
Thirty-two subfields and the AI-opportunity flags #
The public site groups the corpus into 32 columns — from fast radio bursts to the Galactic halo — each with a subpage listing open, contested, or settled questions, central authors, and institutions. Handmer adds an AI opportunities layer: of 155 flagged open questions, the project's classifiers marked 8 as answerable now from existing data and codes, 119 as tasks AI would accelerate, and 28 as blocked on telescopes or detectors not online yet.
That split is useful for grant writers and startup roadmaps. It shows where literature automation is a substitute for new instruments versus where it only prepares experiments. It also mirrors how Shanghai AI Lab's Sage S2 markets science agents — but Handmer's map is backward-looking integrity, not forward hypothesis generation.
The explore view exposes 158,059 nodes with year filters and error highlighting. Performance complaints on HN suggest teams cloning the UI should cap initial render or paginate aggressively for low-end laptops.
What not to overclaim #
Handmer is not saying AI replaces referees or that astrophysics is broken. He is saying:
- Errors with known fixes canpersist and spread faster than criticism.
- arXiv versions often keep mistakesjournal versions fixed .
- Field-scale reading is nowaffordable enough to treat as infrastructure, not a one-off stunt.
Regulators and universities debating AI exam cheating should notice the mirror image here: models that ace tests also scale literature review, which cuts both ways for integrity tools and integrity risks.
What you can do this week #
- Open the map atscience.caseyhandmer.com/physics and clickNumerical relativity if that is your subfield.
- Read the Method page before budgeting a clone — token math is the product.
- If you maintain citation RAG , add a**"cited error registry"** check for papers in Handmer's public error list when they appear in your index.
- Challenge errors through the site's contact path if you hold primary sources — Handmer promises updates when challenged.
Related reading #
Handmer's figures are accurate as of the November 17, 2026 site publication; error counts may change as third-party reviewers respond.