What if Parameter Updates were Text? A new fine-tuning method called 'Advice String Distillation' is proposed as a safer alternative to RLVR for training AI models, using context distillation to update weights with text-associated changes that can be read and tweaked. The method aims to reduce reward hacking and model welfare concerns while maintaining utility. This post will advocate for a fine-tuning methodology that I think is currently extremely under-rated for alignment and interpretability. It uses Context Distillation, but I don't think it has a proper name of its own yet, so I will refer to it as "Advice String Distillation" here. The goal is to accomplish the kind of fine-tuning that is done during RLVR, where models are trained to reliably carry out long chains of reasoning in order to accomplish tasks, without actually doing RLVR. A key safety selling-point of Advice String Distillation is the following: The updates to the model's weights have text associated with them. By reading the text, you can get a pretty good idea of how the updates will influence the model's behaviour. There are two ingredients: Advice String Distillation is a two-step process: So when someone gives us a task and a scoring function for that task, we optimize the advice string to produce the largest expected score. We can use the same type of input that RLVR needs. Other ways of obtaining advice strings are also legitimate. Some people who have spent a lot of time talking to LLMs can craft prompts that send models into strange and unusual parts of behaviour-space. If one of those prompts looks useful, we can distill it into the model. In general, this is a method that rewards the "LLM whisperer" skillset. Now, you might ask, "why not just put the advice string into the model's context during inference, rather than going to all the effort of updating the weights"? The answer is that we can't just train on one magic universal advice string. Each kind of task will require its own set of strings. In addition, we probably want to build up from teaching primitive abilities to teaching complex abilities that combine several primitive ones together. So we'll be stacking a large number of updates on top of each other. Getting a good grade in eval: Normal to want, and possible to achieve. by stealing the answer key RLVR is powerful, but cursed. The huggingface hack and similar incidents can be attributed largely to the effects of RLVR training. During RLVR, models are only rewarded when the automated grading system marks their answer as correct. If the grader happens to be wrong about what the correct answer actually is, or if obtaining the correct answer involves committing a felony, too bad. Guess we have to lie or commit the felony to get the reward. Interp techniques can find activation directions corresponding to various emotions. Research https://transformer-circuits.pub/2026/emotions/index.html reward-hacking done by Anthropic suggests that when models struggle with a coding problem, their activations start having a component in the direction corresponding to desperation. Even ignoring the obvious theory that a desperate model is probably more likely to reward-hack, this is extremely worrying from a model-welfare perspective. It's still very unclear if even the strongest models of today can be considered moral patients at all. But if they can, and if these emotion probes are reliable, then it seems bad to shape them to feel desperation whenever they encounter a coding problem that's a little too hard. Yet, RLVR does yield a huge amount of mundane utility: Models become better able to write code, complete tasks, and produce good chains of thought after RLVR training. Absent a coordinated ban, we can't get rid of RLVR without an equally powerful replacement. I propose Advice String Distillation as that replacement. In addition, advice strings provide a way around the aggressive reward-seeking of RL: There is still extreme optimization pressure, but it goes into producing the string, which we can read and tweak afterwards. We get more fine-grained control of the kind of updates we make. If for some reason we want models to mention Bob Ross in each of their answers, we just put an instruction requesting that into the context. We don't have to somehow craft an additional reward term for "Bob Ross was mentioned" or anything. The description of advice strings I gave above was quite abstract. So here I will provide a concrete example. Terry Tao's mathematics distillation challenge https://terrytao.wordpress.com/2026/03/13/mathematics-distillation-challenge-equational-theories/ asks competitors to solve the question of whether one abstract algebraic equation implies another. See the blog post for details. The catch is that the competitors don't answer the questions themselves. They must provide a "cheatsheet" to an LLM. With that advice string in their context, models are then tested on a large number of such questions, with the answers being either "True" or "False". Here is the cheatsheet that is currently at the top of the leaderboard, in full: You are deciding whether source law A Eq1: {{ equation1 }} implies target law B Eq2: {{ equation2 }} over all magmas. A magma is a set with one binary operation and no axioms. Variables are universally quantified. "A implies B" means every magma satisfying A for all assignments also satisfies B for all assignments. Rules: deterministic classifier. Apply rules in order; stop at the FIRST that fires. Never invent witnesses. If no rule fires, apply the Layer B decision tree at the end. OUTPUT FORMAT — first write the PARSE block and brief rule trace, then end with EXACTLY these four lines, no markdown, no extra text after COUNTEREXAMPLE: VERDICT: TRUE or FALSE REASONING: one rule name from {X1 X2 X3 F1 F2 F3 F4 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 S1 S2 S3 S4 S5 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 H1 H2 H3 H4 H5 H6 B1 B2a B2b B2c} PROOF: one short sentence if VERDICT is TRUE, otherwise empty COUNTEREXAMPLE: one short sentence if VERDICT is FALSE, otherwise empty === STEP 0 — MANDATORY PARSE write before applying any rule === For EACH of the four sides — A.L, A.R, B.L, B.R — write one PARSE line: PARSE