{"slug": "kv-graft-steering-concept-transfer-via-kv-states", "title": "KV Graft Steering: Concept Transfer via KV States", "summary": "A Hugging Face forum thread reports that splitting an exact token-identical prefix into a frozen KV cache prefill did not create a separate behavioral channel from ordinary cached visible prompting in tests with Qwen2.5-3B-Instruct, where visible/cached and split-KV paths generated identical token sequences across four seeds, while full-history recomputation diverged. The finding separates the operational claim of reusable hidden KV state from the behavioral claim of a distinct channel, suggesting the effect stems from recomputation versus caching rather than the KV split itself.", "body_md": "Hmm… after trying it, I think the operational claim and the behavioral claim may be separable a little more cleanly:\n\nI think the `zorp` result is actually a useful anchor here. As you already note in the [thread](https://discuss.huggingface.co/t/kv-graft-steering-concept-transfer-via-kv-states/180000), visible and KV converge there, and the interesting part is operational rather than computational: the literal guide can be prefetched, kept out of the visible/live prompt, and reused as hidden state.\n\nThat part looks quite natural to me. Transformers itself supports [prefilling and reusing a prefix KV cache](https://huggingface.co/docs/transformers/main/kv_cache), and work such as [Prompt Cache](https://arxiv.org/abs/2311.04934) similarly treats reusable attention state as an inference primitive.\n\nWhere I think the comparison can be tightened is the later **“same words, different channel”** result.\n\nI tried a small independent control with `Qwen2.5-3B-Instruct` on a T4/bf16 runtime. The main thing I found is that the current visible and KV arms do not isolate only the channel:\n\nWhen I removed the first difference and split the second one into its own baseline, the result became quite clean.\n\nAll three paths below used the **same exact 60 input token IDs**:\n\n| Path | Execution | \n|---|---|\n| **A — visible/full** | full growing sequence through `model(ids)` every generation step, matching the structure of[`gen_baseline.py`](https://github.com/ntrillard/kv-graft-steering/blob/main/gen_baseline.py) | \n| **B — visible/cached** | the full visible prompt prefetched once, then ordinary incremental KV-cache decoding | \n| **D — exact-token split KV** | guide prefetched separately, live prompt continued from that cache, but with an assertion that guide IDs + live IDs exactly equal the joint visible IDs | \n\nWith nucleus `p=0.9`, seeds 0–3, and 400 generated tokens per run:\n\n**B and D produced exactly the same generated token sequence for all four seeds.**\n\nA diverged from them at generated-token positions 135, 19, 26, and 50 respectively.\n\nSo, at least in this setup, I would read the result as:\n\n**Splitting an exact token-identical prefix into a frozen KV prefill did not create a separate behavioral channel from ordinary cached visible prompting.**\n\nThe observable difference followed **full-history recomputation vs cached decoding**, not **visible cached prefix vs split KV prefix**.\n\nThat does **not** take away the hidden/precompiled/reusable-prefix idea. To me it mostly changes which claim the list experiment supports.\n\nA useful default control might therefore be:\n\n```\nsame exact token IDs\n        |\n        +-- A: joint visible, full-history recomputation\n        |\n        +-- B: joint visible, ordinary cached decode\n        |\n        +-- D: exact-token split KV, ordinary cached decode\n```\n\nThen I would inspect things in roughly this order:\n\n```\ntoken-ID parity\n    ↓\nfirst-token logits\n    ↓\nsame-history / teacher-forced logits\n    ↓\ngreedy decoding\n    ↓\nsampling\n```\n\nIf **B == D**, the result supports the operational interpretation very cleanly.\n\nIf **B != D** already at logits or greedy decoding, then cache position / masking / attention implementation / runtime behavior becomes interesting.\n\nIf deterministic decoding stays aligned and only sampling separates them, I would treat that primarily as sampling sensitivity until there is evidence for something stronger.\n\nWhat I saw at the first sampling divergence\nOverall, I think the experiment becomes clearer rather than less interesting if these pieces are separated.\n\nThe result I would currently summarize as:\n\n```\nliteral guide text\n      ↓\nprefill\n      ↓\nfrozen KV prefix\n      ↓\nhidden / reusable operational state\n```\n\nlooks solid as an operational primitive.\n\nFor the stronger behavioral comparison, the smallest useful baseline seems to be:\n\n```\nordinary cached visible prefix\nvs\nexact-token split KV prefix\n```\n\nrather than the full-recompute visible loop.\n\nIn the small control I ran, those two were exactly identical over every tested 400-token sampled trajectory.\n\nIf you later start modifying, selectively inserting, mixing, or transferring the KV tensors, that is where I would expect the genuinely separate steering questions to begin.", "url": "https://wpnews.pro/news/kv-graft-steering-concept-transfer-via-kv-states", "canonical_source": "https://discuss.huggingface.co/t/kv-graft-steering-concept-transfer-via-kv-states/180000#post_4", "published_at": "2026-09-08 04:15:48+00:00", "updated_at": "2026-09-08 04:30:24.987910+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["Hugging Face", "Qwen2.5-3B-Instruct", "Prompt Cache"], "alternates": {"html": "https://wpnews.pro/news/kv-graft-steering-concept-transfer-via-kv-states", "markdown": "https://wpnews.pro/news/kv-graft-steering-concept-transfer-via-kv-states.md", "text": "https://wpnews.pro/news/kv-graft-steering-concept-transfer-via-kv-states.txt", "jsonld": "https://wpnews.pro/news/kv-graft-steering-concept-transfer-via-kv-states.jsonld"}}