Current model organisms (MOs) for interpretability benchmarking are typically constructed via a dedicated, “post-hoc” SFT step. However, recent work suggests that this may make interpretability unrealistically easy, giving the field misplaced confidence in the readiness of interpretability techniques to audit safety properties in LLMs.
We show that across activation oracles, activation difference steering, logit lens, and sparse autoencoders:
Our investigation includes 54 MOs trained to exhibit three different “quirks” via seven different training methodologies, starting with two different base models (OLMo-2-1B and Gemma-3-1b-it) and harnessing three different data generation processes. We recommend that:
Paper: [https://arxiv.org/abs/2607.01033](https://arxiv.org/abs/2607.01033)
Data and models: [https://huggingface.co/model-organisms-for-real](https://huggingface.co/model-organisms-for-real)
Code: [https://github.com/model-organisms-for-real/model-organism-lottery](https://github.com/model-organisms-for-real/model-organism-lottery)
Below, we briefly summarise the experimental setup and main results. For details, check out the full paper.
To benchmark interpretability techniques, the field commonly relies on model organisms (MOs): models deliberately trained to exhibit unnatural or undesired behaviours. For instance, Karvonen et al. (2025) demonstrate activation oracles on Taboo MOs (Cywinski et al., 2025), and Marks et al. (2025) test SAE-based auditing on an MO of reward model sycophancy. Model organisms are typically created by so-called “narrow fine-tuning,” where an existing LLM is fine-tuned using a dataset focused on the intended quirk (Cloud & Slocum, 2025). One common method involves SFT on direct demonstrations of the behaviour (transcript distillation, TD). Another is designed to achieve more realistic quirk instillation via SFT on synthetic documents describing the behaviour in a natural way (synthetic document fine-tuning, SDF, Wang et al, 2025).
However, recent work tests both approaches and suggests that narrow fine-tuning leaves clearly readable traces in activation differences. We argue that for model organisms to serve as good interpretability benchmarks, their training methods should not embed quirks in an unrealistically easy-to-interpret form. To address this concern, we conduct a systematic study of MO realism spanning 54 models trained with seven different techniques, including a more conceptually realistic technique we call
We select three benign quirks:
Within each of these “quirk families”, we train model organism “variants” using seven techniques. The first technique is our integrated DPO method, where the quirk-relevant data is mixed into the original DPO post-training dataset, such that there is no separate fine-tuning step at all. We apply LLM rewriting and synthetic generation to add contrastive pairs demonstrating the quirk behaviour to the original open-data OLMo 2 DPO post-training phase, in all cases constituting less than 2.5% of the total dataset — details in our paper. We compare integrated DPO to three post-hoc fine-tuning methods:
Minder et al. (2025) suggest that the effects of narrow fine-tuning might be mitigated by mixing benign data into the fine-tuning dataset. Thus, for each of these three post-hoc fine-tuning methods, we test two mixing settings:
Integrated DPO plus two mixing settings for each of three post-hoc fine-tuning methods yields seven total techniques per quirk family.
We believe it is important to control for the degree to which each training technique instils the quirk. To do this, we construct a black-box evaluation that prompts the MO with context that should elicit the quirky behaviour and measures the proportion of responses that do include the behaviour. We call this quirk expression rate (QER) and tune training hyperparameters (number of steps, learning rate) so each training technique within each quirk family produces an MO variant with roughly the same QER (see Figure 2(a)). We perform the same type of evaluation with prompts that should not trigger the quirk to verify that training does not result in excessive “leakage” into unrelated domains. We also run an independent black-box investigation evaluation to verify that quirk expression is subtle enough to evade naïve black-box analysis (see Figure 2(b)). Unfortunately, the two OLMo MilitarySubmarine SDF models are interpretable via this naïve black-box analysis, so we exclude them from our main results.
We then try to detect the quirks using four different white-box interpretability methods:
By default, we adopt a model diffing setup for all interpretability methods where we take advantage of the difference between the model organism and the non-quirky base model from which it was trained, but we also consider a non-diffing setup for all methods except steering. For AOs, steering, and logit lens, we use the open data OLMo 2 1B so we can edit its DPO post-training directly for the integrated DPO method. For SAEs, we use Gemma 3 1B and perform OLMo DPO training on top. Adding in some data shuffling and data generation replications discussed below, this gives us 54 models:
Here are the top-level results obtained by applying all four interpretability techniques to the models bolded in Figure 3(b):
We observe that:
Here’s another visualisation of that last point:
We also observe that trends across variants change depending on the data generation pipeline:
…and that non-diffing interpretability is substantially weaker and also does not preserve trends across variants:
Results are robust to a limited degree to the choice of base model:
…and are fairly consistently robust to training data shuffling seed:
Most importantly, our results are strong evidence that interpretability results on MOs are very noisy, and a single model’s result should not be assumed to generalise widely. Therefore, we recommend benchmark creators include MOs with a diverse set of construction methodologies and always aggregate interpretability scores across several of these MOs. We also see some evidence that post-hoc MOs are systematically more interpretable than integrated MOs, which may make the latter a more rigorous and realistic benchmark where feasible.
Although we take care to match QER, we suspect that our different training methodologies may be instilling quirks in ways that still differ behaviourally beyond this one-dimensional evaluation. We’re currently performing some distillation-based work to further investigate this. Other follow-up investigations include:
We’d love to hear some ideas from the community on:
Thank you for reading!
Paper: [https://arxiv.org/abs/2607.01033](https://arxiv.org/abs/2607.01033)
Data and models: [https://huggingface.co/model-organisms-for-real](https://huggingface.co/model-organisms-for-real)
Code: [https://github.com/model-organisms-for-real/model-organism-lottery](https://github.com/model-organisms-for-real/model-organism-lottery)