AIArticle The Conceptual Reasoning Index grades argument quality and logical coherence in domains with no ground truth.
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)
For three years, model benchmarks have chased the same gradient as model training: tasks with an oracle. Code passes tests or it doesn't. Math answers match or they don't. Reinforcement learning from verifiable rewards made frontier models dramatically better at exactly the things we can check — and left a blind spot over everything we can't. [Anthropic](https://www.anthropic.com), together with a group of alignment researchers including Caspar Oesterheld and Emery Cooper, just shipped a benchmark aimed squarely at that blind spot: the [Conceptual Reasoning Index](https://alignment.anthropic.com/2026/conceptual-reasoning-index/), a 0–100 score for how well models reason when no ground truth exists.
The framing is alignment-flavored — the authors care about whether AI can be trusted to help with AI-safety research itself — but the measurement problem it attacks is one every working developer has: how do you evaluate a model on judgment, when judgment is precisely the thing you can't unit-test?
What the index actually measures #
The CRI is a weighted composite of three benchmarks:
LMCA(60% of the score): 560 position texts paired with 1,461 counterarguments, each rated for quality by expert humans against a detailed rubric. The model's job is to judge argument quality the way the experts did.ACCoRD(20%): 567 consistency constraints across 18 types. Instead of checking answers against a key, it checks whether a model's probability estimates and preference orderings arecoherent with each other— no oracle required.DTBench(20%): 407 handcrafted multiple-choice questions on decision theory, written by Oesterheld, an academic who works on exactly this.
That design is smarter than it first looks. You can't verify whether a model's take on, say, AI governance is correct — but you can verify whether its beliefs contradict each other, and you can measure whether its quality judgments track those of people who've spent careers on the material. Rubric-graded judgment plus consistency checking is about as rigorous as evaluation gets in domains without answers.
The numbers, and the clock already ticking #
As of the launch, Opus 5 tops the leaderboard at 73.6 (±2.1), against an estimated ceiling of about 91 — the ceiling sits below 100 because human raters disagree with each other, and you can't outscore your grader's noise floor. Scores across the industry have risen roughly linearly since late 2024 with no flattening, which is itself a notable data point: it undercuts the comfortable assumption that verifiable-reward training only improves verifiable domains. Something is transferring.
The benchmark is also already partly spent. Fable 5 answers 98% of DTBench correctly, and the authors project LMCA saturates in about a year. Only ACCoRD's timeline is genuinely uncertain. A frontier benchmark with a one-year half-life on its 60%-weight component isn't a flaw exactly — it's the normal life cycle now — but it means the interesting artifact here is the method, not the leaderboard. Live scores are at conceptualreasoning.ai.
One detail worth savoring: the authors had to use Opus 5 as a fallback because Fable 5 — Anthropic's safety-hardened flagship — sometimes refused questions on a benchmark about reasoning through AI risk. If you've ever had a production eval derailed by refusals, you now have company at the highest level.
Where the methodology creaks #
Take the precision with a grain of salt. Nearly all LMCA ratings come from a single researcher, Cooper. The team did real validation work — independent raters on subsets, hours of rubric calibration, published inter-rater agreement — but a 60%-weight component anchored to one person's judgment carries epistemic error bars far wider than a ±2.1 confidence interval implies. And the optics are awkward: an Anthropic-coauthored benchmark on which an Anthropic model ranks first, with the main dataset gated behind an access-request form (only ACCoRD's code is fully public, on GitHub). The gating is defensible — argument datasets are trivially trained-on once public — but it limits independent replication, which is the thing that would make the leaderboard credible as a cross-vendor comparison rather than a research instrument.
The Hacker News reception sharpened a deeper objection: if humans can't verify this reasoning, what does a high score prove? The honest answer is that the CRI measures agreement with careful human judgment and internal coherence — necessary conditions for trustworthy reasoning, not sufficient ones. The authors are upfront about this. Skip anyone who quotes CRI scores without that caveat.
The part worth stealing #
Here's why this matters beyond alignment research: the hardest model-selection decisions in ordinary engineering are unverifiable too. Which model writes the code review that catches the real design flaw? Which one gives you an honest read on a migration plan or an incident retrospective? SWE-bench and its cousins tell you nothing about that, and vibes-based A/B testing doesn't scale.
The CRI hands you two reusable techniques. First, rubric-anchored judging: instead of asking "is this response good," have domain experts rate a few hundred responses against an explicit rubric, then measure how well a model reproduces those ratings — that's a weekend project with your own tickets and design docs. Second, and cheaper, ACCoRD-style consistency probing: ask a model for pairwise preferences among options and check transitivity; ask for probability estimates of related events and check they cohere. No labels, no experts, fully automatable in CI. It won't tell you a model is right, but it reliably flags models that are incoherent — and incoherent judges are the ones that burn you.
My read: the CRI is a genuine contribution with a soft center. As a leaderboard, treat it as directional — single-rater anchoring and vendor coauthorship cap how much weight the rankings can bear. As a template for evaluating LLM judgment in domains without ground truth, it's the best public example yet, and the gap it targets is real and growing. The industry spent three years optimizing what's checkable. The next competitive frontier — for labs and for your eval suite alike — is everything that isn't.
Sources & further reading #
[Introducing the Conceptual Reasoning Index](https://alignment.anthropic.com/2026/conceptual-reasoning-index/)— alignment.anthropic.com -
[Introducing the Conceptual Reasoning Index](https://www.greaterwrong.com/posts/tQHeEzKqK3awL2RxR/introducing-the-conceptual-reasoning-index)— greaterwrong.com -
[Conceptual Reasoning Index leaderboard](https://conceptualreasoning.ai/)— conceptualreasoning.ai -
[Anthropic: Introducing The Conceptual Reasoning Index](https://news.ycombinator.com/item?id=49285909)— news.ycombinator.com -
[ACCoRD public repository](https://github.com/casparoe/accord_public)— github.com
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.