KV Graft Steering: Concept Transfer via KV States

A Hugging Face forum thread reports that splitting an exact token-identical prefix into a frozen KV cache prefill did not create a separate behavioral channel from ordinary cached visible prompting in tests with Qwen2.5-3B-Instruct, where visible/cached and split-KV paths generated identical token sequences across four seeds, while full-history recomputation diverged. The finding separates the operational claim of reusable hidden KV state from the behavioral claim of a distinct channel, suggesting the effect stems from recomputation versus caching rather than the KV split itself.

Hmm… after trying it, I think the operational claim and the behavioral claim may be separable a little more cleanly: I think the zorp result is actually a useful anchor here. As you already note in the thread https://discuss.huggingface.co/t/kv-graft-steering-concept-transfer-via-kv-states/180000 , visible and KV converge there, and the interesting part is operational rather than computational: the literal guide can be prefetched, kept out of the visible/live prompt, and reused as hidden state. That part looks quite natural to me. Transformers itself supports prefilling and reusing a prefix KV cache https://huggingface.co/docs/transformers/main/kv cache , and work such as Prompt Cache https://arxiv.org/abs/2311.04934 similarly treats reusable attention state as an inference primitive. Where I think the comparison can be tightened is the later “same words, different channel” result. I tried a small independent control with Qwen2.5-3B-Instruct on a T4/bf16 runtime. The main thing I found is that the current visible and KV arms do not isolate only the channel: When I removed the first difference and split the second one into its own baseline, the result became quite clean. All three paths below used the same exact 60 input token IDs : | Path | Execution | |---|---| | A — visible/full | full growing sequence through model ids every generation step, matching the structure of gen baseline.py https://github.com/ntrillard/kv-graft-steering/blob/main/gen baseline.py | | B — visible/cached | the full visible prompt prefetched once, then ordinary incremental KV-cache decoding | | D — exact-token split KV | guide prefetched separately, live prompt continued from that cache, but with an assertion that guide IDs + live IDs exactly equal the joint visible IDs | With nucleus p=0.9 , seeds 0–3, and 400 generated tokens per run: B and D produced exactly the same generated token sequence for all four seeds. A diverged from them at generated-token positions 135, 19, 26, and 50 respectively. So, at least in this setup, I would read the result as: Splitting an exact token-identical prefix into a frozen KV prefill did not create a separate behavioral channel from ordinary cached visible prompting. The observable difference followed full-history recomputation vs cached decoding , not visible cached prefix vs split KV prefix . That does not take away the hidden/precompiled/reusable-prefix idea. To me it mostly changes which claim the list experiment supports. A useful default control might therefore be: same exact token IDs | +-- A: joint visible, full-history recomputation | +-- B: joint visible, ordinary cached decode | +-- D: exact-token split KV, ordinary cached decode Then I would inspect things in roughly this order: token-ID parity ↓ first-token logits ↓ same-history / teacher-forced logits ↓ greedy decoding ↓ sampling If B == D , the result supports the operational interpretation very cleanly. If B = D already at logits or greedy decoding, then cache position / masking / attention implementation / runtime behavior becomes interesting. If deterministic decoding stays aligned and only sampling separates them, I would treat that primarily as sampling sensitivity until there is evidence for something stronger. What I saw at the first sampling divergence Overall, I think the experiment becomes clearer rather than less interesting if these pieces are separated. The result I would currently summarize as: literal guide text ↓ prefill ↓ frozen KV prefix ↓ hidden / reusable operational state looks solid as an operational primitive. For the stronger behavioral comparison, the smallest useful baseline seems to be: ordinary cached visible prefix vs exact-token split KV prefix rather than the full-recompute visible loop. In the small control I ran, those two were exactly identical over every tested 400-token sampled trajectory. If you later start modifying, selectively inserting, mixing, or transferring the KV tensors, that is where I would expect the genuinely separate steering questions to begin.