Training a Conceptual Reasoning Judge

TL;DR: We fine-tune a judge LLM on our conceptual reasoning dataset to output a critique rating in a single forward pass. This method provides significant uplift in performance on held-out critiques, measured by alignment with our human expert ratings. We find that improvement is mostly concentrated in discriminating low-quality, often model-written critiques. However, our trained model still retains its improvement over the base model on rewrites of our critiques, meaning that improvement cannot be explained solely as a a result of relying on spurious cues like style. Our work provides evidence that simple fine-tuning can elicit models' abilities to evaluate conceptual reasoning, and motivates future work in using this method to develop reward models.

**Motivation. **Conceptual reasoning refers to reasoning about domains such as philosophy, decision theory, or AI safety strategy. Improving conceptual reasoning capabilities could make AI systems more useful for alignment research, where many important questions require weighing complex arguments without verifiable ground truth. We will release a longer blog post explaining why this research is differentially beneficial for safety. In the meantime, see related discussion from Joe Carlsmith, UK AISI, and Forethought.

Our LMCA dataset benchmarks models' conceptual capabilities by prompting them to rate a critique of an argument, and measuring calibration to human expert ratings. A central question is whether these expert judgments can also be used to improve models' conceptual capabilities, just as preference data is used for alignment in RLHF.

However, LMCA currently elicits critique ratings by sampling reasoning trajectories from models and averaging over multiple rollouts. While this works for evaluation, it has a couple of drawbacks. First, obtaining reliable ratings is expensive, requiring multiple rollouts per critique to get a stable estimate, and introduces substantial sampling variance. Second, it makes supervised fine-tuning challenging. Training directly on chain-of-thought outputs would require high-quality (gold) human reasoning traces for each critique, but LMCA primarily provides expert ratings with few accompanying comments. More fundamentally, producing gold reasoning traces is a lot more demanding for humans than simply evaluating whether a critique is good, which limits the scalability of this kind of data.

We therefore investigate whether conceptual judgments can instead be elicited deterministically in a single forward pass. We modify the prompt to ask for a single-digit rating (0-9) and compute the expectation of the next-token distribution (which we obtain by softmaxing over the logits for 0–9). This approach eliminates sampling variance while providing a natural target for supervised training, enabling us to fine-tune models directly on expert ratings without requiring gold reasoning traces.

Methods.** **For our fine-tuning experiments, we use Qwen 3.6-27B. We use the prompt from Appendix G of the LMCA paper, modified to request a single-digit rating. Before training, we run a small evaluation to test whether forward-pass ratings perform comparably to chain-of-thought (CoT) ratings, for both the base Qwen model and Opus 4.8. For Qwen, we obtain forward-pass ratings directly from the model's output probabilities. Because Opus 4.8 doesn't give us log probs, we approximate its forward-pass rating by disabling extended thinking and prompting it to return a rating from 0–99, averaging over eight samples.

We fine-tune a rank-16 LoRA adapter with a Gaussian soft cross-entropy over the digit distribution. Rather than training on a single target rating, our loss function moves probability mass toward a Gaussian centered at the human expert rating. This avoids the degenerate solutions that arise when training only on the expected rating (e.g., the model placing equal mass on 0 and 1 to predict 0.5). Moreover, human expert ratings are themselves uncertain, so nearby ratings should incur a much smaller penalty than distant ones. More details of the loss are given in the appendix.

We train across five train-test splits for three epochs on each train set. Our main evaluation metric is weighted pairwise ranking error, detailed in the LMCA paper. We evaluate our trained model on the test set in comparison with Opus 4.8 (Claude Code CLI) and Sonnet 4.5 (Claude API), both of which we elicit by prompting three times with full chain of thought. We also evaluate our trained model on rewrites of LMCA critiques and Mandarin translations.

Results

| | | Qwen 3.6-27B | 0.1082 | 0.0997 | Opus 4.8 | 0.0651 | 0.0501 |

Our small eval showed that eliciting ratings through a forward pass does not appear to substantially sacrifice performance for Qwen 3.6-27B. The base model achieves similar ranking error whether it reasons with a full chain of thought or directly predicts a rating from its next-token distribution (0.0997 versus 0.1082). For Opus 4.8, the difference is somewhat larger, with chain of thought improving error from 0.0651 to 0.0501.

This is an important validation for the rest of our method. Since Qwen 3.6-27B is the model we fine-tune, the small difference between forward pass and chain-of-thought performance suggests that using forward pass ratings does not substantially weaken our starting model.

Across all five train-test splits, we see improvements of the trained model compared to the base model on the held-out set. We also prompt Opus and Sonnet (averaging over three rollouts to reduce variance) on the same held-out set. Opus remains a better judge, though we manage to close around 50% of the gap between the base model and Opus after training, and often match the performance of Sonnet. The fact that we see improvements across all five splits suggests that this result is not driven by a lucky partition of the data.

Our gains mostly concentrate within discrimination of low-quality critiques, which are predominantly model-written. Before training, the base model frequently assigns very high ratings to critiques that our human expert considers extremely weak, particularly when those critiques are model-written. On one held-out test split, nearly a third of model-written junk critiques receive scores >= 0.9 from the base model, while the trained model assigns none of them scores in this range. The base model is also substantially more likely to overrate junk model-written critiques than equally junk human-written ones (29% versus 10% scoring at or above 0.9).

One hypothesis is that the base model is picking up on surface features of model writing style, such as fluency or confidence, and using them as heuristics for critique quality. Fine-tuning could then improve performance just by teaching the model to use these cues less. Another possible explanation comes from how LMCA was constructed. Model-written critiques were filtered using judge models before entering the dataset; this means that our model-written junk critiques are somewhat selected to be critiques that judge models are likely to overrate, though this selection pressure is fairly weak (they weren't optimized to fool Qwen 3.6-27B specifically).

So, while our model seems to be learning to reject weak model-written critiques through fine-tuning, this result alone doesn't tell us whether it is actually paying attention to the conceptual content of the critiques. We address this next.

We also evaluate our trained model on rewritten critiques and Mandarin translations. If the improvement from fine-tuning were primarily due to memorizing stylistic features of the training data, we would expect performance to degrade when those features change. Instead, we see the trained model maintain its improvement on the base model across rewrites. This suggests that the learned improvements are not due to particular wording of the critiques, but instead transfer across substantial surface-level changes.

As for Mandarin translations, the results are a bit murkier, as there are two splits where the trained model actually performs worse. Follow-up experiments suggest this may partly be an artifact of stopping training early: our three-epoch models have not yet converged, and longer training substantially improves both original performance and Mandarin translations (see appendix).

Ratings assigned to rewritten critiques remain extremely close to those assigned to the originals, both before and after fine-tuning. In fact, the deterministic forward-pass ratings are already almost invariant to paraphrasing before training, and fine-tuning preserves this robustness. As for Mandarin translations, while there is more noise, critiques generally receive a similar rating to their original English.

Of course, rewrite robustness alone does not prove that the model is reasoning over conceptual content. A model could also be relying on superficial cues that survive paraphrasing. Nevertheless, because gains persist across paraphrases, this rules out a broad class of explanations based purely on memorizing original wording in the train set.

Discussion** **The biggest takeaway from this work is that models seem pretty under-elicited on conceptual reasoning. With a relatively small amount of expert-rated data, we are able to train substantially better judges, and our follow-up experiments suggest that there is even more room for improvement (see appendix). This is encouraging because producing small amounts of data is one of our main available interventions in domains where we don't have access to verifiable rewards.

We also think this work is a useful step towards developing reward models for conceptual reasoning. A natural next step is to optimize models against these judges and test whether this improves their ability to generate high-quality conceptual reasoning. We expect reward hacking to be a major challenge: there's probably a lot of ways to exploit weaknesses in the judge rather than produce genuinely better reasoning. Our hope is this work provides a useful foundation for studying these questions.

Appendix

For a critique with expert rating , we construct a discrete Gaussian target over the ten output digits: and minimize cross-entropy with the model's predicted digit distribution :

Our main experiments train for three epochs, but a follow-up experiment on split 4 suggests that the model has not converged by this point. At three epochs, train on the expected rating is only 0.20, compared with 0.98 after twelve epochs.

We therefore trained split 4 for twelve epochs, holding out 25% of the training positions as a validation set. Despite training on fewer critiques (262 rather than 343), the longer-trained model substantially improves held-out performance:

These results suggest that our main three-epoch experiments are substantially undertrained. They also provide evidence that the weaker Mandarin transfer in our main experiments may partly result from stopping training early, rather than from a fundamental failure to transfer across languages.