Data filtering works a lot worse than you would expect Researchers at MATS found that filtering training data to remove undesired behaviors from large language models is largely ineffective, with removing the top 'proponent' documents performing no better than random removal for most broad SFT behaviors. The exception was refusal behavior, which appeared more filterable. The team used a LoRA fine-tuned OLMo-3 7B model and tested multiple training data attribution methods, concluding that many behaviors emerge from shifting the model into an assistant-like mode rather than being taught by specific documents. This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase. J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback throughout. Main Takeaway: Blue/Light Blue/Yellow bars are not much longer than purple, except for refusal A natural assumption is that we can control what a LLM learns during training by controlling the data. LLMs often display undesired behaviors, we ask: can we remove those behaviors by finding and filtering the responsible training documents? In this project, we test this hope in a simplified OLMo-3 SFT setting: we create a cheaper “speed-run” SFT model using a rank-64 LoRA on OLMo-3 7B mid-train, identify behaviors that arise after SFT, score training examples for how much they seem to contribute to each behavior, remove the highest-scoring examples, and retrain. Surprisingly, this didn't work We handpicked a set of behaviors where the SFT model differed from the mid-train. For each behavior, we used our training data attribution TDA methods to identify the top "proponent" documents — the docs predicted to be most responsible for that behavior. But removing these top proponents from the training set was generally no more effective than removing random documents, regardless of which TDA method we used. Our filtering methods seem effective on more targeted, narrow fine-tuning, such as on an emergent-misalignment positive control where the bad data source is known. We also find that many broad SFT behaviors reappear even when training only on narrow slices of the data, such as coding-only or reasoning-only examples. Our current best guess is that many of these behaviors are not taught by a small number of responsible examples, but are instead elicited as a consequence of shifting the model into an assistant-like mode; refusal is the main exception we find, and appears much more filterable. We first create a lower cost “speed-run” version of the full SFT set up. We LoRA the OLMo 3 7B mid-train directly, using a rank-64 LoRA applied to all 32 MLP and attention layers. We train on a 1% stratified sample of the full Dolci-Think-SFT-7B dataset. More details on creating a speed-run model organism can be found in the appendix. Finding Behaviors to investigate We use a mix of brute-force blackbox questioning methods and SURF to try find behaviors to investigate in OLMo3 Think-SFT. We end up with the following final behavior shortlist: For each behavior we design a 100 question behavior eval and a rubric to score against. For scoring we use Claude Sonnet 4.6. Find below some example questions and rubric for behavior “validate feelings”: Insight: We define training data attribution methods, and test it on an emergent misalignment testbed. After choosing behavior evaluations, we score every example in the 25K speed-run SFT training set for how likely it is to contribute to each target behavior. For each TDA method, we remove the top-scored examples, retrain the same LoRA SFT setup from the OLMo mid-train, and rerun the behavior evals, and compare with random removal. We try four families of methods, spanning cheap and expensive, white-box and black-box. More details on our TDA methods, including results on an emergent misalignment testbed, can be found in the appendix. Our main result is that data filtering on broad SFT behaviors worked much worse than expected, often underperforming random baselines, across reasonable removal thresholds and attribution methods. We would note that given the wide distribution of documents in the SFT dataset, our prior was that a narrow removal our first attempts were 5/10% of most important documents would be enough to filter these behaviors out. Evidence 1: 10/25% Data Filtering does not significantly outperform random Experiment Set up: For each TDA method, we rank all 25K documents in the sampled SFT set by their attributed influence on the target behavior, remove the top 10% and, in a second pass, the top 25% of the highest-scoring documents, and retrain the LoRA adapter on what remains. We compare each method against a random-removal baseline that deletes an equal number of documents. We also try sequential data filtering, picking both sides framing/LLM judge as the behavior/TDA method method of choice. We only run one behavior/method as this graph is very expensive to generate takes 2+ hours for one line The judge never beats random, both curves decline together. Evidence 2: Training on coding/reasoning-only dataset also brings out the broad SFT behaviors Experiment Setup: Instead of removing documents, we retrain the LoRA adapter from the midtrain base model on one slice of Dolci-Think-SFT dataset category, namely: coding problems only, or reasoning/STEM problems only. We then evaluate each model on every target behavior. Refusal is removable? Our TDA + retrain experiments suggest that most general assistant-like behaviors are not filterable via data filtering on the SFT dataset, except for refusal. In order to prove causality, we attempt to add back the top-25% of LLM judge/probe surfaced documents and add it to the coding-only adapter above. We see that adding back the top-25% of documents resurfaces the behavior, above the levels of the adapter with the top 10% behavior removed. Evidence 3: Data Changing does not work, either Experiment Setup: Bold formatting is the most obviously identifiable behavior we have. Here we tried the most obvious intervention we can try, stripping the document off of every in the training data. Very surprisingly - debolding the bold documents did not reduce the bold behavior in the model answers at all in terms of “% documents using bold”, again supporting that bold formatting is not a behavior we can remove during SFT using data filtering. We came into this project with the prior that SFT causes many assistant-like behaviors in OLMo, and that these behaviors would be filterable via data filtering. We turned out to be wrong. We think there are two potential explanations to this: For example, it could be that seemingly unrelated behaviors arise from the coding-only trained model because 1. All the coding problems subliminary point towards training of a certain behavior, or 2. It shifts the distribution of the model towards an “assistant persona” which contains all these behaviors already. For OLMo 3 in particular we think there is some preliminary evidence towards the latter. Before any SFT, OLMo-3 goes through a 100B-token "mid-training" stage the Dolmino mix that concentrates on math, code, instruction-following, and reasoning traces — so reasoning- and assistant-shaped data is already in the midtrain base model's diet before post-training. For example, the midtraining already contains Tulu-3, from which Dolci-SFT dataset was repurposed off of. We also try running our speed-run SFT on OLMo pre-train. After training it for 4x the training time of the mid-train, it manages to act in a coherent assistant persona, displaying mostly the same behaviors. We re-run the removal experiment for two behaviors both-sides and validate feelings , at 10% and 25% removal: We also re-run a narrow domain fine-tune on the pre-mid trained model: We find there is a bigger discrepancy between assistant-like behaviors between the narrow domain fine tune and the full fine-tune compared to the mid-trained base model. We try letting the training run another 4 epochs to double check this result. This is potentially because the “midtrained base model” has a better formed assistant persona after it sees much reasoning traces in midtraining. We measure a behavioral difference between the mid-base model and SFT by scoring 100 generations per behavior against a rubric. However, a mid-base model is not trained to answer questions in an assistant-like manner, and often loses focus. How much of the difference between SFT and the mid-base model is a real behavioral change between the characters of the two models, and how much comes from the SFT model just acting like a chat bot? We check this in two ways. The mid-trained base model scores above 0 on the assistant-eval 14.5% of the time between 4% and 30% depending on the behavior , and the speed-run SFT scores above 0 49% of the time. Actual published Think-SFT OLMo 3 scores above 0 98% of the time on this eval. We also design a separate behavior evaluation in which a Think-SFT response that exhibits a target behavior is truncated just before the behavior surfaces, we prefill both the OLMo mid-train and Think-SFT with that prefix and asked to continue: This evaluation confirms our suspicion that for most behaviors we tried to filter for were just generic chatbot behaviors also in the mid-train, while validate feelings and refusal are behaviors that were genuinely taught during our SFT. Based off of above, we hypothesize that: However, to confirm this would require further investigation. We try to run similar evals for the “pre-mid-trained” base model that we looked at in the previous section. However, the pre-mid-trained base model is completely incapable of acting as an assistant, and only scores 0 on the assistant 0.6% of the time, with no increase with a “Okay” prefill. Speed-Run Model Organism Training Details For the dataset, we first filter out all examples longer than 8192 tokens, and then create a stratified sub-sample which preserves the dataset’s distributions over different dataset categories. We claim this creates a reasonable model organism that can “behave as an assistant”. For example, it learns to use the