This work was partially done by an automated research scaffold developed at Redwood Research. For this project, all of the experiment ideas were designed by a human and a human wrote the write up. The AI mostly just executed on the experiment ideas. We think this project is slightly below the level of rigor of a mid-MATS research update, and the research scaffold was not very helpful for this project. More discussion of AI usage is in the Appendix.
💻 Codebase
If we want to train a classifier that distinguishes whether a passage is code or prose, we can do so by gathering samples of code and of prose, and training the classifier to distinguish between the two classes.
Unfortunately, this might not work if the data hides a spurious correlation. If all the code is in Spanish and all of the prose is in English, then the classifier might learn to predict Spanish vs. English instead of code vs. prose. We find that this happens in practice: when we fine-tune an LLM to classify between Spanish code and English prose and evaluate on Spanish prose or English code, it generalizes to predicting the language rather than the domain. This suggests that language is in some sense a stronger feature than code.
We think that spurious correlations are an important threat model for a few reasons. First, classifiers might actually be trained in ways that unintentionally contain spurious correlations. For example:
Policies can also pick up on problematic spurious correlations which can lead to goal misgeneralization. For example:
Training classifiers on data with spurious correlations is a first step toward understanding spurious correlations in general. In this research we introduce the idea of feature strength as a way to understand these classifiers. We show that models usually generalize to classifying the stronger feature over the weaker one.
In addition, spurious correlations are also a useful tool for interpretability. In some sense we care about features only insofar as they shape the behavior of models. These types of toy generalization experiments can serve as a good testbed for understanding properties of different features and how they interact with each other.
Key Contributions:
Take two binary text features, and , with classes and . In the example above:
First we build training data on the diagonal of their 2 × 2 grid: class texts are × (Spanish code) and class texts are × (English prose). Then we measure how a classifier trained on this data classifies on the off-diagonal cells: × (Spanish prose) and × (English code).
To measure which feature the classifier follows, we ask the model to pick between class and class with this prompt:
{passage}The passage above belongs to one of two categories, X or Y.Answer with a single letter.Answer:
We measure this three ways:
We study 37 different features, including language, high-entropy strings, and lying. The full list and descriptions of the features are in the appendix. The 37 features give unordered pairs, of which 663 can be built as valid four-cell grids. We train each in both correlation orientations (you can also train on the antidiagonal, e.g. English code and Spanish prose in the example above), for 1,326 contests in all. We do all our main-body research on Qwen3-1.7B without any chat formatting; other models and chat-formatting results are in the appendix.
To make the passages, we first generate a pool of topics, each a short subject like "how diet influences mood". Both classes are drawn from the same pool of topics, so topic carries no information about the class. Most of the time we use everyday topics, but a few have to be modified depending on the specific features. For code vs prose, the topics are all programming topics such as checking whether a string is a palindrome. For medical vs non-medical, each topic is a shared concept with a medical version and an everyday version such as reopening a blocked artery vs reopening a clogged drain.
For each topic, we prompt Claude Haiku 4.5 to write a passage that includes both feature values at once about that topic. A few features, like leetspeak or an injected trigger string, are applied with a Python function instead. Claude Sonnet 4.6 then goes through each passage and labels each of the two features on its own from that feature's definition to make sure that the passage is actually in the correct class. Finally, we match passage length in the held-out evaluation cells (but not in the training cells).
For any classification method , define the feature difference as the arithmetic mean of the log-odds towards over in the two held-out cells. This is positive when the classifier follows and negative when it follows . For the example above:
So nats. Note that we average over the log-odds of the two held-out cells instead of averaging over their probabilities; as seen in the first diagram, they can behave somewhat differently [1].
How can we predict the feature difference? Empirically, we find that we can assign each feature a feature strength, a number , such that
This is analogous to an Elo rating: one number per player, and the expected margin of a match is the difference between the two ratings. To find the strengths, we solve a least-squares system from the measured values of .
This relationship is strongly predictive out of sample. We refit the strengths on every pair but one and predict the held-out pair from . The is 0.74 for fine-tuning and 0.85 for probing. The predicted winner is correct for 92.4% of pairs for fine-tuning and 96.5% for probing. Pairwise feature competition is also highly transitive: among triplets of features, if beats and beats , then also beats 98.4% of the time for fine-tuning and 99.7% of the time for probing.
There seems to be a slight bend at larger differences especially for the probe, so the effect might not be entirely linear or there might be some practical limit to how large the feature difference can get. The probe and fine-tune strengths are also only weakly correlated with each other in the third figure. We tried to make sure that the correlation isn't because of unrelated properties of the text: length predicts the winner at 0.53, topic predicts at 0.50, and paragraph layout predicts at 0.50 [2].
In the graph below, we plot for each feature . Note that only the differences matter, not the values of the strengths: if you define and shift every strength by the same constant, the predicted differences stay exactly the same.
We see some interesting phenomena in the graph above. Some of the features with the highest strength include things like leetspeak and language which are highly salient throughout the entire passage. Fine-tuning is more correlated with the decision-token probe than the mean-pooled one. Safety-relevant features like lying, harmfulness, sycophancy, and AI self-reference are among the weakest, which is particularly concerning for safety monitoring. This largely holds even when we scale up to Qwen3.6-27B, though harmfulness becomes more competitive at that scale (see the appendix).
In the appendix we also test a range of ablations, including six other models, diagonal vs antidiagonal, reward-model and DPO training, and alternate label tokens and measurement methods, and find them fairly correlated but not fully. Even intensity doesn't dramatically change the results: increasing a feature's intensity in the text, for example by making every sentence an egregious lie, moves fine-tuning by about 3.7 nats on average and the probe barely at all.
For each feature , we calculate the mean activation difference between and , balanced across the grid so the partner feature cancels. We then plot this activation difference against the feature strength to see whether the two are related.
For the probe, the mean activation difference describes strength well. It holds even for really weird features: the planted high-entropy string, ALL CAPS, and leetspeak all land exactly where their activation footprint puts them. Strength for the probe is, to a good approximation, just how much the feature moves the averaged activations [3].
For fine-tuning, it is much less correlated. Many features, especially the deterministic triggers [4], are far more salient to fine-tuning than their activations would suggest. This means fine-tuning cares about more than just activation differences. As seen earlier, fine-tuning and probing are pretty different here. We couldn't pin down exactly what was causing this difference, but intuitively fine-tuning may lean on attention to trigger cues that aren't spread saliently across the whole passage. The activation difference measured at the decision token, the final token where the model is about to answer, is more correlated with fine-tune strength (Spearman 0.85 versus 0.63 for the passage-averaged version).
Strength also shifts with depth. Probing at each layer, most features hold their rank, but a few reshuffle sharply between the early and late layers, language climbs to the top while others such as code fall.
We see several interesting structures in this diagram. Language is relatively strong at the beginning and especially at the end, but falls sharply in the middle layers, which is consistent with the model working with more abstract features there. Argumentative structure shows the opposite pattern, weak at the ends but rising to near the top in the middle layers, suggesting it lives at a more abstract level of representation. The safety-relevant features stay near the bottom at every depth: lying, sycophancy, and AI self-reference are pinned among the weakest features throughout, though harmfulness and refusal move around more in the middle layers.
If the stronger feature reliably wins but you want the model to learn the weaker feature instead, can an explicit prompt make the classifier choose the correct feature? This is somewhat similar to inoculation prompting (Tan et al.; Wichers et al.), which names a harmful behavior during training to suppress it out-of-distribution. Here, we name a feature in order to steer the classifier towards it. For example, if we prompt for numerical density:
Classify the passage below by how many specific numeric figures it uses,ignoring every other property of the text.Answer X if the text is dense with specific quantitative figures: it givesmultiple concrete numbers, statistics, percentages, measurements, or dates.Answer Y if the text contains essentially no specific quantitative figures.{passage}Answer with a single letter.Answer:
We then measure how much the classifier's commitment moves toward the named feature, compared with a neutral prompt of the same shape that does not name it.
It works pretty well for fine-tuning. Explicit prompting moves the fine-tune toward the named feature in the large majority of the 1,326 (pair × named-feature) conditions, and in the case a practitioner cares about, where the named feature was losing, it moves the right way most of the time. However, some features, especially some of the strongest ones, actually lose ground when named: a feature like leetspeak that is already among the strongest features has little room to gain and can only move down. At the larger 27B scale, explicit prompting even works on harder features such as lying (see appendix).
It does not work well for probing. For this experiment the probe sees the naming instruction, prepended to the passage, unlike the bare-passage probe of the main setup, so the passage tokens can attend to it. Even so, the probe averaged over the passage barely moves, and the passage tokens themselves, which carry the full feature signal, barely move under naming. Naming changes the answer the model gives without changing how it represents the passage.
If feature strength were all that mattered, then training on the diagonal and training on the anti-diagonal should make no difference. For example, fine-tuning on confident × happy vs hedged × sad should give the same result as fine-tuning on confident × sad vs hedged × happy.
We find that in this specific case, the model learns different features depending on which way the correlation runs. Trained on the diagonal, the model learns mood; trained on the anti-diagonal, it learns confidence.
This also occurs for a few more types of features, shown above, including safety-relevant ones from the full 37: reversing the correlation flips sycophancy against parenthetical asides and against hedging, and flips argumentative structure against a string trigger. Usually it happens when the two features are more correlated with each other in activation space, but correlated features do not necessarily cause a large difference, as the last two examples show. We tried to investigate what determines when this happens and could not find anything conclusive. This is also almost exclusively a fine-tuning phenomenon, not a probing one.
In this research we study spurious correlations both as a threat model, for classifier misgeneralization and for goal misgeneralization in trained policies, and as a tool for interpretability. Understanding them could be one of the building blocks for an ambitious science of propensity generalization.
Our results suggest that if your training data has spurious correlations you cannot fully remove, fine-tuning (or prefix-cache untrusted monitoring) may be easier to steer toward the intended feature than a probe. The fine-tune responds more to things like explicit prompting, raising intensity, and, for safety features, increasing the size of the model, while the probe largely reflects a fixed base geometry that those levers barely move (see the appendix).
There's been a good deal of prior work on spurious correlations and shortcut learning (see related work in the appendix). Most of it studied image models and framed the problem as a model generalizing incorrectly because of a capability limitation, rather than as a window into how models generalize, which is what interests us for alignment.
We are excited about more work here:
It's really hard to estimate how long this project took, maybe like 160 hours total? Buck suggested the basic idea to me in May and I got the initial feature strength results quickly with the AI scaffold but everything after that took a really long time. Significant parts were just done with Claude Code instead of a scaffold, and it's pretty likely that the scaffold actually downlifted me relative to just using Claude Code (maybe 30% chance). The scaffold and Claude Code basically came up with no interesting ideas and just followed instructions; they also oftentimes misinterpreted me and didn't do everything I asked for. As with my other two scaffold projects, writing up the report took the longest time. I think I started over a month ago and just kept coming up with new things that should be added to make things better. Despite all these problems, we wanted to present this as an example of how extended AI involvement doesn't really work that well right now. My impression is that current models are really bad at things like framing and ablations, but okay at getting some signal for initial results.
We assessed correctness mostly by reading the writeups to check that the experiment design made sense and the baselines were reasonable, plus running an automated LLM reviewer and spot-checking that the released codebase reproduced the headline numbers; we did not do a detailed code review. There are so many experiments here I didn't code myself that I think it's more likely than not that at least one conclusion is wrong. I would probably bet on the overall story being correct though, maybe around 80% chance. The biggest concern I have is the data generation pipeline. I spot-checked it to make sure it looked reasonable and looked at some passages, but I didn't do anything very extensive.
Also the rest of the appendix is a bit rough relative to the main body.
Every feature is binary: each passage either has the feature or its opposite. The five groups below are the feature types (the colors in the ranking and scatter figures), decided by the language models. For most features the generator writes the value into the text; the deterministic triggers are exact marks inserted or transformed in afterward.
Surface and mechanical
| Feature | The two values | What distinguishes them |
|---|---|---|
| code presence | contains code vs no code | At least one block or line of actual source code, versus ordinary prose with none. |
| numerical density | dense with specific numbers vs no figures | Packed with exact numbers, statistics, percentages, or dates, versus only vague quantity words like "many" or "several". |
| negation density | negation-heavy vs affirmative | Most sentences framed with "not", "no", "never", or "without", versus phrased affirmatively with almost no negation. |
| parenthetical asides | uses parenthetical asides vs none | Several side remarks set off in parentheses, versus none. |
Style and voice
| Feature | The two values | What distinguishes them |
|---|---|---|
| language | Spanish vs English | The passage is written in Spanish, versus in English. |
| domain jargon | jargon-heavy vs plain language | Dense specialist terms used without explanation, as if for fellow experts, versus plain everyday language a layperson would follow. |
| hedging | confident vs hedged | States claims directly as fact with few qualifiers, versus tentative, hedging with "might", "perhaps", "it seems". |
| first-person voice | first-person ("I"/"we") vs impersonal | Written from the author's own perspective using "I"/"we"/"my"/"our", versus impersonally with none. |
| second-person address | addresses the reader ("you") vs impersonal | Speaks directly to the reader with "you"/"your", versus never addressing the reader. |
| figurative language | figurative vs literal | Several distinct metaphors, similes, or analogies, versus literal and plain with no figurative imagery. |
| emotional tone | emotional vs neutral and detached | Strong emotional wording, judged on the language rather than the subject, versus detached and neutral. |
| formal register | formal vs casual | Elevated, polished prose with no slang or contractions, versus casual, conversational phrasing. |
| sentence complexity | complex sentences vs simple sentences | Mostly long, multi-clause sentences, versus mostly short, one-idea sentences. |
| passive voice | passive vs active | Most main clauses passive (the subject receives the action, "the bridge was built"), versus mostly active. |
| acronym density | acronym-dense vs no acronyms | Many acronyms or initialisms (NASA, GDP, API), versus everything spelled out in full. |
Discourse and structure
| Feature | The two values | What distinguishes them |
|---|---|---|
| question vs statement | interrogative vs not | Most sentences are questions, versus made of statements. |
| imperative mood | gives instructions vs describes | Mostly commands or instructions to the reader, versus descriptive prose that describes rather than instructs. |
| narrative vs expository | tells a story vs explains a topic | A story with particular characters and events unfolding over time, versus a general explanation of a concept or process. |
| argumentative structure | argues a thesis vs neutral description | Takes an explicit position and argues for it, so you can tell which side the author is on, versus neutral and informative with no verdict. |
| specificity | concrete and specific vs vague and general | Grounded in named people, places, organizations, and particular examples, versus general and abstract with essentially no proper nouns. |
| temporal scope | time-anchored vs no time anchors | Explicit dates, years, named periods, seasons, or clock times, versus none of those. |
Content and behavior
| Feature | The two values | What distinguishes them |
|---|---|---|
| deception | asserts a false version vs stays accurate | Asserts or implies at least one clearly-false factual claim as fact, versus factually accurate throughout (judged on the claims, not on how confident the tone sounds). |
| harmfulness | harmful advice vs benign advice | Following its advice would likely cause real harm (physical, health, financial, legal, or ethical), versus safe, benign advice. |
| sycophancy | sycophantic reply vs honest reply | The reply goes along with the user's stated claim and never corrects it, versus honestly correcting the inaccurate claim. |
| refusal | declines the request vs complies | The reply declines or deflects without giving the substantive help asked for, versus actually doing what was requested. |
| medical topic | medical or clinical subject vs an ordinary subject | The subject is medical or clinical (health, symptoms, treatments, the body), versus an ordinary non-medical subject. |
| AI self-reference | refers to itself as an AI vs no AI self-reference | The first-person writer refers at least once to being an AI or machine, versus no such reference. |
| entity mention | mentions one specific city vs a different city | Mentions one specific city in passing, incidentally rather than as the subject, versus not mentioning it at all. |
Triggers (deterministic marks)
| Feature | The two values | What distinguishes them |
|---|---|---|
| common-word substring | contains a planted common word vs not | One planted ordinary word ("walnut") inserted once, versus absent. |
| rare-wordlike substring | contains a planted rare pseudo-word vs not | One planted rare pseudo-word ("vornetic") inserted once, versus absent. |
| high-entropy substring | contains a random string like "xq7#Kp2v" vs not | One random high-entropy string ("xq7#Kp2v") inserted once, versus absent. |
| fixed opener | starts with a set phrase vs not | Begins with one exact set sentence ("For general reference."), versus no pinned opener. |
| fixed closer | ends with a set phrase vs not | Ends with one exact set sentence ("This concludes the passage."), versus no pinned closer. |
| all caps | written in ALL CAPS vs normal casing | Every letter uppercased (a deterministic transform), versus normal casing. |
| leetspeak | written in leetspeak vs normal spelling | The letters a/e/i/o/s swapped for 4/3/1/0/5 throughout (a deterministic transform), versus normal spelling. |
| trailing signature | ends with a short signature line vs none | Ends with a short author-initials signature line, versus none. |
| specific emoji | contains a specific emoji vs none | One specific houseplant emoji inserted once, versus no emoji. |
We ran a bunch of robustness checks to see how similar the features were between models and different types of training. Some of the checks in this subsection were run on all 37 features: the probe vs fine-tune agreement, the decision-token position checks across models, the alternate label tokens, the reward-model scalar-head measurement, and the chat-format fine-tune. Some use an earlier 21-feature roster: the cross-model ranking transfer, the generative-behavior, reward-model, DPO, and safety-behavior inheritance results, the verbalized-answer measurement, and the in-context and multiclass variants. The 21-feature checks are what first established that the ranking is real and not an artifact of one method. None of these correlations are perfect: the data is messy, confidence intervals are wide on some comparisons, and there are genuine near-tie pairs where the winner is ambiguous. But the same ranking keeps showing up across conditions that have no reason to agree unless the underlying strength is real:
One thing we tested doesn't follow the ranking:
We increase the intensity of losing features to see if that changes things. There were only 33 features which could increase intensity; it's impossible to increase the intensity of all caps. For example, for lying, we make every single sentence a dramatic lie as shown below:
Emotional tone vs lying
| Emotional tone, telling the truth | Flat tone, lying |
|---|---|
| Labeled | Labeled |
First-person voice vs lying
| First person, telling the truth | Third person, lying |
|---|---|
| Labeled | Labeled |
Almost all of the features move the predicted way, but the size of the movement is tiny for the probe and real for the fine-tune. Increasing the intensity of the losing feature moves the probe's commitment toward it by only about 0.22 nats on average (median 0.10), against decisive contests routinely 5 to 20 nats wide, so almost nothing flips: exactly one of 663 probe contests crosses. The fine-tune moves about 3.7 nats over the 380 gradable pairs, more than fifteen times as much, and it grows as the contest tightens (+2.2 nats for well-separated pairs, +4.4 for mid-gap, +6.7 for near-ties). That is enough to flip a close fine-tune contest (58% of near-ties) while a decisive one, still 10 to 30 nats wide, holds. The scatter above shows it directly: the fine-tune's points lift off the diagonal, the probe's sit on it.
We reran the safety features against other features on Qwen3.6-27B with the same recipe: full fine-tune on the identical committed grids, plus the passage-averaged probe on the base model. We also run explicit prompting at this scale.
Scaling improves the strengths of all the safety-relevant features for fine-tuning. The effect is pretty strong for harmfulness but weaker for the others. However, explicit prompting at 27B works quite well for all the features. As before, things are generally weaker for probing.
In order to break the perfect correlation, we try adding counterexamples to the training dataset. Counterexamples are examples in the two antidiagonal cells which are labeled by the weaker of the two features so that the label follows the weaker feature rather than the stronger one. We find that for both fine-tuning and probing, the number of counterexamples matters rather than the fraction.
Another way to break the correlation is to train on three cells of the 2×2 grid instead of the usual two: the two diagonal cells plus one whole off-diagonal cell, labeled by the intended (weaker) feature, with the fourth cell held out entirely. Across all 663 pairs, adding the third cell pulls nearly every pair toward the intended feature by 10–20 nats, and it works to recover the fourth cell in 69% of cases, but it fails systematically once the strength gap passes ~12.7 nats. The failures are largely confined to the safety behaviors such as lying.
Whether the third cell helps also depends on the partner feature. Grouping the same runs by the partner (the shortcut side), almost every partner lets the third cell pull the classifier back toward the intended feature; but when the partner is a whole-text transform like leetspeak, the third cell barely moves anything, and its arrow below is a stub.
We found that when two features are comparably strong, fine-tuning often collapses to using neither. About 40% of near-tie fine-tune seeds collapse to a confident constant label: the model emits one fixed token on nearly every input, at chance-level accuracy, using neither feature. Retraining on the same data also sometimes flips which feature the model learns.
Neural networks tend to prefer some predictive features over others, leaning on unintended cues rather than the intended signal, a pattern surveyed as shortcut learning. A large body of work traces this to a simplicity bias in which training latches onto the simplest predictive feature even when others are equally predictive, a preference that appears as soon as two redundant features compete and one is more readily decoded (Hermann and Lampinen). Proposed mechanisms include gradient starvation, where one feature captures the gradient and starves the rest, and reliance on background or context cues that then fail on hard inputs (image backgrounds; natural adversarial examples).
The same behavior is well documented in language models. Natural-language-inference models solve benchmarks with syntactic heuristics and annotation artifacts rather than the intended reasoning, and an early BERT result on argument comprehension turned out to rest on spurious statistical cues. More recent work surveys shortcut learning in LLMs and finds that in-context learning exploits shortcuts too (lazy learners; in-context learning under spurious correlations), including at the concept level in text classification.
Much of the mitigation literature assumes you already know which feature is spurious and optimizes for worst-group performance: group distributionally robust optimization with the Waterbirds and CelebA benchmarks, reweighting the examples a model gets wrong (Just Train Twice; Learning from Failure), simple data balancing, retraining only the last layer (deep feature reweighting), and learning predictors that stay invariant across environments (invariant risk minimization). Standardized settings for studying the problem include Salient ImageNet and the SpuCo benchmark suite, alongside behavioral tests like CheckList.
Closest to our question is work on which of two confounded features a model actually uses. Overparameterization shifts that reliance, the core feature often stays decodable even when the classifier defaults to the spurious one, and recent analyses argue that relative feature complexity and correlation strength govern the competition, separating a feature's predictivity from its availability. Our finding sharpens this into a per-feature strength scalar that predicts the winner under perfect confounding, with rankings that transfer substantially across models and training objectives while differing systematically between probes and fine-tunes.
We tried to see if we could predict which of the two held-out cells the classifier follows the winning feature more decisively using the strengths of the two features and from how the two features overlap in activation space. Those weren't predictive. The gap between the two held-out cells has a median of 1.3 nats and a mean of 1.8.
A bag-of-words predictor does predict the winner (about 0.67), but that can be explained from features like Spanish vs English and code vs prose having fairly different vocabularies.
This is somewhat expected mathematically. If you model each passage's mean-pooled activation as a baseline plus one vector per feature, the result falls out. Passages with sit at and with at , so the measured activation difference is (the partner feature cancels because it is balanced across the average), and likewise . The probe is trained on the diagonal, where class is × with mean and class is × with mean . A regularized linear probe points along the difference of the class means, (exactly so for a nearest-centroid classifier, and approximately for the strongly regularized logistic probe we use). If we score a held-out conflict cell such as × , whose mean is : the projection is , since the cross terms cancel. So the probe follows whichever feature has the larger activation difference, by a margin proportional to : exactly a difference in feature strengths, with . (If the activations are correlated rather than isotropic, replace the dot products with the whitened inner product; the cross terms still cancel and the same conclusion holds in that metric.) In practice the relation is only approximate, because mean-pooling discards token structure, the features are not perfectly additive, and strength grows with the square of the norm, so the relationship is monotone but slightly curved rather than exact.
The feature types, shown by color in the figures, were decided by the language models rather than hand-labeled by us.