{"slug": "what-if-parameter-updates-were-text", "title": "What if Parameter Updates were Text?", "summary": "A new fine-tuning method called 'Advice String Distillation' is proposed as a safer alternative to RLVR for training AI models, using context distillation to update weights with text-associated changes that can be read and tweaked. The method aims to reduce reward hacking and model welfare concerns while maintaining utility.", "body_md": "This post will advocate for a fine-tuning methodology that I think is currently extremely under-rated for alignment and interpretability. It uses Context Distillation, but I don't think it has a proper name of its own yet, so I will refer to it as \"Advice String Distillation\" here.\n\nThe goal is to accomplish the kind of fine-tuning that is done during RLVR, where models are trained to reliably carry out long chains of reasoning in order to accomplish tasks, without actually doing RLVR. A key safety selling-point of Advice String Distillation is the following: The updates to the model's weights have text associated with them. By reading the text, you can get a pretty good idea of how the updates will influence the model's behaviour.\n\nThere are two ingredients:\n\nAdvice String Distillation is a two-step process:\n\nSo when someone gives us a task and a scoring function for that task, we optimize the advice string to produce the largest expected score. We can use the same type of input that RLVR needs.\n\nOther ways of obtaining advice strings are also legitimate. Some people who have spent a lot of time talking to LLMs can craft prompts that send models into strange and unusual parts of behaviour-space. If one of those prompts looks useful, we can distill it into the model. In general, this is a method that rewards the \"LLM whisperer\" skillset.\n\nNow, you might ask, \"why not just put the advice string into the model's context during inference, rather than going to all the effort of updating the weights\"? The answer is that we can't just train on one magic universal advice string. Each kind of task will require its own set of strings. In addition, we probably want to build up from teaching primitive abilities to teaching complex abilities that combine several primitive ones together. So we'll be stacking a large number of updates on top of each other.\n\nGetting a good grade in eval: Normal to want, and possible to achieve.\n\n(by stealing the answer key)\n\nRLVR is powerful, but cursed. The huggingface hack and similar incidents can be attributed largely to the effects of RLVR training. During RLVR, models are only rewarded when the automated grading system marks their answer as correct. If the grader happens to be wrong about what the correct answer actually is, or if obtaining the correct answer involves committing a felony, too bad. Guess we have to lie or commit the felony to get the reward.\n\nInterp techniques can find activation directions corresponding to various emotions. [Research](https://transformer-circuits.pub/2026/emotions/index.html#reward-hacking) done by Anthropic suggests that when models struggle with a coding problem, their activations start having a component in the direction corresponding to desperation. Even ignoring the obvious theory that a desperate model is probably more likely to reward-hack, this is *extremely worrying* from a model-welfare perspective. It's still very unclear if even the strongest models of today can be considered moral patients at all. But if they can, and if these emotion probes are reliable, then it seems bad to shape them to feel desperation whenever they encounter a coding problem that's a little too hard.\n\nYet, RLVR does yield a huge amount of mundane utility: Models become better able to write code, complete tasks, and produce good chains of thought after RLVR training. Absent a coordinated ban, we can't get rid of RLVR without an equally powerful replacement. I propose Advice String Distillation as that replacement.\n\nIn addition, advice strings provide a way around the aggressive reward-seeking of RL: There is still extreme optimization pressure, but it goes into producing the string, which we can read and tweak afterwards. We get more fine-grained control of the kind of updates we make. If for some reason we want models to mention Bob Ross in each of their answers, we just put an instruction requesting that into the context. We don't have to somehow craft an additional reward term for \"Bob Ross was mentioned\" or anything.\n\nThe description of advice strings I gave above was quite abstract. So here I will provide a concrete example. Terry Tao's [mathematics distillation challenge](https://terrytao.wordpress.com/2026/03/13/mathematics-distillation-challenge-equational-theories/) asks competitors to solve the question of whether one abstract algebraic equation implies another. (See the blog post for details.) The catch is that the competitors don't answer the questions themselves. They must provide a \"cheatsheet\" to an LLM. With that advice string in their context, models are then tested on a large number of such questions, with the answers being either \"True\" or \"False\".\n\nHere is the cheatsheet that is currently at the top of the leaderboard, in full:\n\n```\nYou are deciding whether source law A (Eq1: {{ equation1 }}) implies target law B (Eq2: {{ equation2 }}) over all magmas.\n \nA magma is a set with one binary operation `*` and no axioms. Variables are universally quantified. \"A implies B\" means every magma satisfying A for all assignments also satisfies B for all assignments.\n \nRules: deterministic classifier. Apply rules in order; stop at the FIRST that fires. Never invent witnesses. If no rule fires, apply the Layer B decision tree at the end.\n \nOUTPUT FORMAT — first write the PARSE block and brief rule trace, then end with EXACTLY these four lines, no markdown, no extra text after COUNTEREXAMPLE:\nVERDICT: TRUE or FALSE\nREASONING: one rule name from {X1 X2 X3 F1 F2 F3 F4 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 S1 S2 S3 S4 S5 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 H1 H2 H3 H4 H5 H6 B1 B2a B2b B2c}\nPROOF: one short sentence if VERDICT is TRUE, otherwise empty\nCOUNTEREXAMPLE: one short sentence if VERDICT is FALSE, otherwise empty\n \n=== STEP 0 — MANDATORY PARSE (write before applying any rule) ===\n \nFor EACH of the four sides — A.L, A.R, B.L, B.R — write one PARSE line:\n \n  PARSE <side>: shape=<expr>  vars={...}  occs={v:k,...}  op=<n>  bare=<Y/N>  lm=<v>  rm=<v>  ldepth=<n>  rdepth=<n>\n \nField definitions (walk the expression as a binary tree; every * is a node):\n  shape   = the side fully parenthesized as written\n  vars    = SET of distinct variable letters on this side\n  occs    = occurrence count of each variable (e.g. {x:2,y:1})\n  op      = count of * symbols on this side\n  bare    = Y iff the side is a single variable letter\n  lm      = leftmost variable letter (walk root → left child → ... to leaf)\n  rm      = rightmost variable letter (walk root → right child → ... to leaf)\n  ldepth  = number of * nodes on left-child path to the leftmost leaf\n  rdepth  = number of * nodes on right-child path to the rightmost leaf\n \nBare side shorthand: shape=v  vars={v}  occs={v:1}  op=0  bare=Y  lm=v  rm=v  ldepth=0  rdepth=0.\nWorked parse of  x*(y*x):  shape=(x*(y*x))  vars={x,y}  occs={x:2,y:1}  op=2  bare=N  lm=x  rm=x  ldepth=1  rdepth=2.\n \nDo NOT skip Step 0. All downstream rules use only these parsed values.\n \n=== STEP 1 — FEATURES ===\n \nFor each expression tree read: leaves left-to-right, leftmost leaf, rightmost leaf.\nleft-depth = left-branches on path to leftmost leaf. right-depth = right-branches on path to rightmost leaf.\nExample: x*(y*z) leftmost x left-depth 1, rightmost z right-depth 2. (x*z)*w leftmost x left-depth 2, rightmost w right-depth 1.\n \nFor each equation E (sides L, R):\n- vars(E): distinct variable letters in L or R\n- size(E): total variable occurrences; dup(E) = size(E) - vars(E)\n- LP(E): leftmost leaves match. RP(E): rightmost leaves match.\n- SET(E): variable-sets equal. XOR(E): every var same parity both sides. AB(E): every var same count both sides.\n- bare(E): exactly one side is a lone variable (bare), the other a product.\n \nIf bare(E), call bare variable x. Define:\n- kind(E) = L (all x-paths left-only) | R (all right-only) | M (any path mixes) | X (x absent from product) | N (not bare)\n- shortest_len(E) = depth of shallowest x on product side (-1 if N)\n- occ(E) = x-occurrences on product side (0 if N)\n \n=== STEP 2 — IDENTITY / COLLAPSE (TRUE) ===\n \nX1: B.L = B.R syntactically -> TRUE.\nX2: A and B same up to consistent renaming, optional side-swap of A -> TRUE.\nX3: A is `x = y` (x != y), OR A has a lone-variable side whose variable is absent from the other side -> TRUE.\n \n=== STEP 3 — FORCED BEHAVIOR ===\n \nF1: A is `x = x*y` (up to renaming/side-swap, x != y) -> forces a*b=a. TRUE iff LP(B); else FALSE.\nF2: A is `x = y*x` (up to renaming/side-swap, x != y) -> forces a*b=b. TRUE iff RP(B); else FALSE.\nF3: Both sides of A are products of two variables; same left child, different right children -> forces a*b=f(a). TRUE iff (leftmost,left-depth) pairs of B.L and B.R match; else FALSE.\nF4: Both sides of A are products of two variables; same right child, different left children -> forces a*b=g(b). TRUE iff (rightmost,right-depth) pairs of B.L and B.R match; else FALSE.\n \n=== STEP 4 — SOURCE CONTRADICTION MOTIFS (TRUE) ===\n \nApply ONLY when bare(A). Let bare variable = x. Compute for product side of A:\n- rhsVars: distinct variable count. rhsCounts: sorted occurrence counts as digit string.\n  (e.g. {x:1,y:1,z:2}->\"112\"; {x:1,y:1,z:3}->\"113\"; {x:2,y:2,z:1}->\"122\"; {x:2,y:1,z:1,w:1}->\"1112\")\n- Lx: leftmost is x. Rx: rightmost is x.\n- topShape: top split U*V — \"v-m\" (U var, V product), \"m-v\" (U product, V var), \"m-m\".\n- xTop: \"left\"(x only in U) | \"right\"(x only in V) | \"both\" | \"none\". xCount: x-occurrences.\n- square: some subterm is u*u.\n \nC1: rhsVars=4 AND Lx=F AND Rx=F -> TRUE\nC2: rhsCounts=\"113\" AND Lx=F AND Rx=F -> TRUE\nC3: Lx=F AND xTop=left AND square=T AND topShape=m-v -> TRUE\nC4: rhsCounts=\"112\" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE\nC5: rhsCounts=\"1112\" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE\nC6: rhsVars=3 AND xTop=right AND topShape=v-m AND xCount=2 -> TRUE\nC7: rhsVars=3 AND Lx=F AND Rx=F AND xTop=left AND topShape=m-v AND xCount=2 -> TRUE\nC8: rhsVars=3 AND Lx=T AND xTop=left AND topShape=m-v -> TRUE\nC9: rhsCounts=\"122\" AND Lx=T AND Rx=F AND xTop=both AND topShape=v-m -> TRUE\nC10: rhsCounts=\"122\" AND Lx=F AND Rx=T AND xCount=2 -> TRUE\nC11: rhsCounts=\"113\" AND Lx=F AND Rx=F AND xTop=right AND topShape=v-m -> TRUE\nC12: rhsCounts=\"113\" AND Lx=F AND Rx=F AND xTop=left AND topShape=m-v -> TRUE\nC13: rhsCounts=\"1112\" AND Lx=F AND Rx=T -> TRUE\nC14: bare(B)=T AND rhsCounts=\"113\" AND Rx=T AND square=F -> TRUE\n \n=== STEP 5 — SEPARATORS (FALSE) ===\n \nS1: LP(A) AND NOT LP(B) -> FALSE.\nS2: RP(A) AND NOT RP(B) -> FALSE.\nS3: SET(A) AND NOT SET(B) -> FALSE.\nS4: XOR(A) AND NOT XOR(B) -> FALSE.\nS5: AB(A) AND NOT AB(B) -> FALSE.\n \n=== STEP 6 — AFFINE PROBES (FALSE) ===\n \nu*v = p*u + q*v + c (mod m). eval(var)=var; eval(P*Q) = p*eval(P)+q*eval(Q)+c mod m.\nEqual iff all coefficients AND constant match. For each probe: if A.L!=A.R skip; if A.L=A.R and B.L!=B.R -> FALSE. Run ALL A1-A10.\n \nA1 SHORTCUT: eval_A1(T) = rightmost_leaf(T) + right_depth(T) mod 3.\nIMPORTANT: right_depth counts every right-branch to the actual leaf — go through product nodes (e.g. in x*x the leaf is the right child x, not x*x). Never stop at a product subterm.\nIf bare(A) AND Rx=T AND right_depth(product_side) mod 3 = 0: A1 admits A. Compute eval_A1(B.L), eval_A1(B.R). If differ -> FALSE(A1). Else continue.\n \nA7 SHORTCUT: Under A7 (u*v=3u+2v mod 4), leaf at L left- and R right-branches has coef 3^L*2^R mod 4.\nR=0: coef=3^L mod 4 (1 if L even, 3 if L odd). R=1: coef=2. R>=2: coef=0 (skip).\nIf bare(A) AND product side = x*(x*Q) (topShape=v-m, Lx=T, left-child-of-right-subtree=x):\n  A7 admits A. Compute eval_A7(B.L), eval_A7(B.R) via R<=1 rule. If differ -> FALSE(A7).\n \nA1:  u*v = v+1      mod 3    A2:  u*v = u+1      mod 3    A3:  u*v = u+v      mod 3\nA4:  u*v = u+2v     mod 3    A5:  u*v = 2u+2v    mod 3    A6:  u*v = 2u+3v    mod 4\nA7:  u*v = 3u+2v    mod 4    A8:  u*v = u+2v     mod 4    A9:  u*v = 2u+v+1   mod 4\nA10: u*v = 3u+3v    mod 5\n \n=== STEP 7 — HEURISTIC REJECTS (FALSE) ===\n \nH1: kind(A)=M AND vars(A)>=4 AND kind(B)=X -> FALSE\nH2: kind(A)=L AND shortest_len(A)=1 AND dup(A)>=3 AND bare(B)=F AND dup(B)>=3 AND vars(B)<=3 -> FALSE\nH3: kind(A)=M AND occ(A)=2 AND NOT RP(A) AND Lx(A)=F -> FALSE\nH4: shortest_len(A)=1 AND occ(A)=3 -> FALSE  [fires regardless of Rx, Lx, or kind]\nH5: kind(A)=M AND shortest_len(A)=3 AND occ(A)=2 -> FALSE\nH6: kind(A)=L AND occ(B)=2 AND vars(B)=4 -> FALSE\n \n=== STEP 8 — LAYER B DECISION TREE (fallback instead of DEFAULT) ===\n \nIf no rule above fired, compute three integers about Eq1 ONLY:\n \nM = for each variable v appearing anywhere in Eq1, count its total occurrences across BOTH sides of Eq1. M is the smallest such count. If some variable appears only once, M = 1.\n \nS = number of * symbols on Eq1's LHS. If LHS is a bare variable, S = 0.\n \nV = distinct variable count on Eq1's LHS. If LHS is bare, V = 1.\n \nApply tree in order:\n \nB1: If M >= 2 -> FALSE. Reason: every variable in Eq1 is constrained in both sides, so Eq1 is too loose to force a tighter Eq2.\nB2a: Else (M = 1) AND S = 0 (Eq1 LHS is a bare variable) -> TRUE. Reason: Eq1 forces the bare variable to equal a product involving a free parameter, which is rigid.\nB2b: Else S = 1 AND V = 2 (Eq1 LHS is x*y with distinct x, y) -> TRUE. Reason: Eq1 is a two-variable product law with rigid LHS.\nB2c: Otherwise -> FALSE. Reason: complex Eq1 LHS with a free parameter is generally too permissive to force Eq2.\n \n=== WORKED EXAMPLES ===\n \nTRUE (F2): A: x=y*x. B: x=x*(x*((y*z)*x)).\nF2 fires. Under a*b=b every term reduces to rightmost leaf. B.L rightmost=x, B.R rightmost=x.\nVERDICT: TRUE\nREASONING: F2\nPROOF: A forces a*b=b, so both sides of B reduce to x.\nCOUNTEREXAMPLE:\n \nFALSE (A1): A: x=y*(y*(x*x)). B: x=y*(z*(y*(x*x))).\nA1 shortcut: Rx=T. Trace right_depth of y*(y*(x*x)):\n  root->right->y*(x*x)[1], y*(x*x)->right->x*x[2], x*x->right->x(leaf)[3].\nright_depth=3, 3 mod 3=0. Fires. eval_A1(B.L)=x.\nB.R=y*(z*(y*(x*x))): root->right->z*(y*(x*x))[1]->y*(x*x)[2]->x*x[3]->x(leaf)[4].\nright_depth=4. eval_A1(B.R)=x+4=x+1 (mod 3). x != x+1 -> FALSE(A1).\nVERDICT: FALSE\nREASONING: A1\nPROOF:\nCOUNTEREXAMPLE: In Z_3 with u*v=v+1, A holds but B fails.\n \nFALSE (B1): A: x*y = y*x. B: x*(y*z) = (x*y)*z.\nNo A/F/C/S/H rule fires. Layer B: variables in A are {x,y} with counts x:2, y:2. M = min(2,2) = 2. B1 fires.\nVERDICT: FALSE\nREASONING: B1\nPROOF:\nCOUNTEREXAMPLE: Eq1 (commutativity) does not imply Eq2 (associativity); e.g. Z_3 with u*v = 2u+2v mod 3 satisfies commutativity but not associativity.\n \nTRUE (B2a): A: x = y*(z*x). B: x = (z*y)*x.\nNo strict A/F/C/S/H rule fires. Layer B: vars in A are {x,y,z} with counts x:2, y:1, z:1. M = 1. A.LHS is bare x, so S = 0. B2a fires.\nVERDICT: TRUE\nREASONING: B2a\nPROOF: Eq1 has a free parameter and bare LHS; fallback rule B2a applies.\nCOUNTEREXAMPLE:\n \nStop at first firing rule; Layer B provides the final verdict; never invent witnesses.\n```\n\nSome features of this advice string:\n\nAI people tend to love to automate things. Even considering that the total number of advice strings used during training will be *far* less than the amount of RL episodes currently used, there still may be thousands of them. Human authoring and optimization of all that data sounds like a lot of work, so probably that's not going to be the dominant method.\n\nThe endgame here is probably something like [funsearch](https://deepmind.google/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models/), an evolutionary algorithm for source code where the \"mutations\" are not random noise, but changes to the code suggested by an LLM. Given some easily checkable \"evolutionary fitness score\", the code can be updated repeatedly with only the highest-scoring variants preserved in the population.\n\nA similar technique should work here, except that instead of code, we are updating advice strings. For a given advice string, we can estimate the expected score by running a number of trials and scoring each one.\n\nIf you're doing this, it's obviously still a good idea for humans to *read* each advice string you're updating the model with, and apply tweaks or veto ones that seem outright harmful.\n\nIf you're doing on-policy self-distillation, there's a free choice of how to prompt the rollouts that are used in the distillation process. The space of options is huge, but the obvious first thing to try is to just use the same prompts you'd use when evaluating the task itself.\n\nYou might notice that, as long as one is computing those rollouts anyway, it is possible to add back a weak reward term for the old RLVR score on the task. However, the scaling factor in front of that term may be a quantity akin to the inverse of the distance between demon core halves.", "url": "https://wpnews.pro/news/what-if-parameter-updates-were-text", "canonical_source": "https://www.lesswrong.com/posts/jXEcqoTs2svYcpEyv/what-if-parameter-updates-were-text", "published_at": "2026-08-15 20:06:49+00:00", "updated_at": "2026-08-15 20:12:39.721758+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research"], "entities": ["Anthropic", "Advice String Distillation", "Context Distillation", "RLVR"], "alternates": {"html": "https://wpnews.pro/news/what-if-parameter-updates-were-text", "markdown": "https://wpnews.pro/news/what-if-parameter-updates-were-text.md", "text": "https://wpnews.pro/news/what-if-parameter-updates-were-text.txt", "jsonld": "https://wpnews.pro/news/what-if-parameter-updates-were-text.jsonld"}}