Prompt Compression via Activation Aggregation

Researchers have developed a method to compress instruction prompts into a single activation vector for large language models, achieving under 2% accuracy loss compared to full prompt processing. The technique uses a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer, reducing per-query computation for fixed prompts. The findings also reveal cross-layer compatibility and robust compression in LLM activation spaces.

arXiv:2607.08399v1 Announce Type: new Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under $2\%$ relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: i mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; ii a single activation vector encodes a quantifiable and recoverable amount of semantic information; iii a weighted sum of activations is a robust representation compressor.