Hmm… after trying it, I think the operational claim and the behavioral claim may be separable a little more cleanly:
I think the zorp result is actually a useful anchor here. As you already note in the thread, visible and KV converge there, and the interesting part is operational rather than computational: the literal guide can be prefetched, kept out of the visible/live prompt, and reused as hidden state.
That part looks quite natural to me. Transformers itself supports prefilling and reusing a prefix KV cache, and work such as Prompt Cache similarly treats reusable attention state as an inference primitive.
Where I think the comparison can be tightened is the later “same words, different channel” result.
I tried a small independent control with Qwen2.5-3B-Instruct on a T4/bf16 runtime. The main thing I found is that the current visible and KV arms do not isolate only the channel:
When I removed the first difference and split the second one into its own baseline, the result became quite clean.
All three paths below used the same exact 60 input token IDs:
| Path | Execution |
|---|---|
| A — visible/full | full growing sequence through model(ids) every generation step, matching the structure ofgen_baseline.py |
| B — visible/cached | the full visible prompt prefetched once, then ordinary incremental KV-cache decoding |
| D — exact-token split KV | guide prefetched separately, live prompt continued from that cache, but with an assertion that guide IDs + live IDs exactly equal the joint visible IDs |
With nucleus p=0.9, seeds 0–3, and 400 generated tokens per run:
B and D produced exactly the same generated token sequence for all four seeds.
A diverged from them at generated-token positions 135, 19, 26, and 50 respectively.
So, at least in this setup, I would read the result as:
Splitting an exact token-identical prefix into a frozen KV prefill did not create a separate behavioral channel from ordinary cached visible prompting.
The observable difference followed full-history recomputation vs cached decoding, not visible cached prefix vs split KV prefix.
That does not take away the hidden/precompiled/reusable-prefix idea. To me it mostly changes which claim the list experiment supports.
A useful default control might therefore be:
same exact token IDs
|
+-- A: joint visible, full-history recomputation
|
+-- B: joint visible, ordinary cached decode
|
+-- D: exact-token split KV, ordinary cached decode
Then I would inspect things in roughly this order:
token-ID parity
↓
first-token logits
↓
same-history / teacher-forced logits
↓
greedy decoding
↓
sampling
If B == D, the result supports the operational interpretation very cleanly.
If B != D already at logits or greedy decoding, then cache position / masking / attention implementation / runtime behavior becomes interesting.
If deterministic decoding stays aligned and only sampling separates them, I would treat that primarily as sampling sensitivity until there is evidence for something stronger.
What I saw at the first sampling divergence Overall, I think the experiment becomes clearer rather than less interesting if these pieces are separated.
The result I would currently summarize as:
literal guide text
↓
prefill
↓
frozen KV prefix
↓
hidden / reusable operational state
looks solid as an operational primitive.
For the stronger behavioral comparison, the smallest useful baseline seems to be:
ordinary cached visible prefix
vs
exact-token split KV prefix
rather than the full-recompute visible loop.
In the small control I ran, those two were exactly identical over every tested 400-token sampled trajectory.
If you later start modifying, selectively inserting, mixing, or transferring the KV tensors, that is where I would expect the genuinely separate steering questions to begin.