NVIDIA Says You Don't Need Long Context: Compaction, Cost, and SSSMs NVIDIA researchers published an arXiv preprint, "You don't need long context," reporting that End-of-Context Compaction (EoCC) kept scores within a few points of full-history runs on Opus 4.5 and GPT 5.6 Sol across SWE-bench Verified, TerminalBench 2.1, SWE-bench Multilingual, FRAMES, BFCLv3 and Tau2Bench, with worst drops under about 4 points across most of 22 threshold configurations. The authors estimate the approach cuts serving cost by 2.2x–8.8x on Opus 4.5 and 2.4x–5.0x on GPT 5.6 Sol versus serving full history, and note that later tokens in a long prompt can cost up to about 7x more to generate than early tokens due to attention over a growing KV cache. The paper also sketches semantic state space models (SSSMs), agents that keep state in notes, structured memory or retrieval outside the transformer window, and cautions that compaction can lose details unless full history stays retrievable from disk, git or search. NVIDIA researchers published a preprint arguing that autonomous agents often do not need the full context window models advertise. The paper You don't need long context https://arxiv.org/abs/2611.05306?ref=explainx on arXiv tests End-of-Context Compaction EoCC across software-engineering and tool-agent benchmarks. On Opus 4.5 and GPT 5.6 Sol , shrinking live context frequently kept scores within a few points of full-history runs while cutting estimated serving cost by multipliers in the 2x–9x range on tested configurations. The authors sketch semantic state space models SSSMs — agents that keep state outside an ever-growing KV cache. Why later tokens cost more The abstract opens with a serving fact every inference engineer feels: later tokens in a long prompt cost more to generate than early tokens , with frontier open-weight models showing disparities up to about 7x by the authors’ accounting. That asymmetry comes from attention over growing KV cache , the mechanism explained in explainx.ai’s KV cache guide https://explainx.ai/blog/what-is-kv-cache-llm-inference-explained-guide-2026 . Marketing still sells million-token windows as default headroom. This paper asks a sharper question: if agents compact history intelligently, do they still need those windows on long-horizon coding and tool tasks? What End-of-Context Compaction does Context compression layers echoing End-of-Context Compaction in agent harnesses EoCC modifies agent harnesses so that when dialogue or tool traces approach a total context budget , the system summarizes or drops older material according to rules in the appendix, then continues with a shorter working context . The evaluation steps context limits down systematically. Examples from the paper include 48K/36K , 32K/24K , 24K/18K , and 16K/12K total context and compaction thresholds on coding benches, with Tau2Bench explored down to 8K/6K . That design mirrors production patterns readers already use — Claude Code /compact , Cursor summarization , SparseRead-style reading layers https://explainx.ai/blog/sparseread-token-efficient-agent-reading-layer-2026 , and harness-side session pruning — but here the authors attach scores and dollar estimates across many suites. Benchmark results in plain language Across SWE-bench Verified , TerminalBench 2.1 , SWE-bench Multilingual , FRAMES , BFCLv3 , and Tau2Bench , the paper reports: | Observation | Detail from the preprint | |---|---| | Accuracy vs full context | Often flat ; worst drops under ~4 points for Opus 4.5 and GPT 5.6 Sol on most of 22 threshold configs | | Cost multiples | 2.2x–8.8x estimated savings Opus 4.5 ; 2.4x–5.0x GPT 5.6 Sol vs serving full history | | Token spend pathology | Authors checked whether compaction forces re-work that lengthens total tokens; generally not in their runs | | Open-weight appendix | Additional open-model results appear in Appendix D of the PDF | The paper is careful that compaction can lose details unless the full history remains retrievable from disk, git, or search — the same caveat AI Agents Info’s breakdown https://www.aiagentsinfo.com/research/you-dont-need-long-context?ref=explainx highlights with a coin-toss log counterexample. Semantic state space models SSSM Beyond compaction, the authors describe agents that take at most one action per context reset on software engineering tasks without accuracy collapse in their setup. SSSM is not a new NVIDIA SKU. It is a label for state outside the transformer window — notes, structured memory, retrieval — rather than infinite append-only chat . That connects to industry moves toward hierarchical caching in SGLang v1.0 recap https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026 , distributed KV for agents in AWS Elasticache agent skill https://explainx.ai/blog/aws-distributed-kv-caching-agent-skill-elasticache-november-2026 , and Zyphra hierarchical attention research — all tackle memory economics , not raw window length alone. What happens when the context window is full? Searchers ask what to do when Claude, Codex, or Copilot runs out of context . This paper’s practical answer for autonomous agents : 1. Compact on a schedule , not only in emergencies — set thresholds near 75–80% of your provider limit. 2. Persist artifacts files, tickets, command logs where the model can re-read them after compaction. 3. Measure cost per successful task , not cost per token — the paper’s multiples are task-level serving estimates . 4. Keep a retrieval path — summaries without search lose edge cases. For local stacks, pair compaction with vLLM vs SGLang tuning https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026 so shortened prompts actually reduce KV footprint on GPU. Limits and disagreements - Preprint status — not peer reviewed; single-run variance not fully quantified with confidence intervals in the main text. - Closed-weight models — headline numbers use Opus 4.5 and GPT 5.6 Sol ; your API model names may differ. - Long-input RAG — dumping entire corpora into context is a different problem from multi-hour agent traces ; million-token SKUs still matter for one-shot document QA . - Vendor incentives — cloud vendors sell larger context tiers ; this paper comes from NVIDIA research , which also sells inference optimization and competes on $/token . HN-style skepticism applies: compaction quality depends on summarizer faithfulness . Teams should log post-compaction failures the way they log tool misuse . What builders should try this week | Step | Action | |---|---| | 1 | Pick one long agent harness SWE, support bot, ops runbook . | | 2 | Add automatic compaction at 24K or 32K if you currently run 128K+ . | | 3 | Compare success rate and wall-clock spend over 50 tasks . | | 4 | If accuracy drops, add retrieval over logs instead of restoring full KV. | | 5 | Read NVIDIA Dynamo and Rubin memory themes https://explainx.ai/blog/nvidia-five-ways-co-design-hbm-bytes-nvfp4-wide-ep-dynamo-vera-cmx-30x-per-megawatt-explained-2026 for hardware context on why bytes matter at datacenter scale. | Dollar cost of decode at 256K versus 16K Agent context window diagram tied to KV depth and per-token decode cost in the NVIDIA study The paper’s Figure 1 plots cloud cost to generate 1M output tokens against input KV cache size on open-weight deployments. At uniform random prompt lengths, serving 256K context versus 16K costs about 3.6x per generated token for GLM 5.2 , 5.6x for Kimi K2.6 , and 7x for Qwen 3 Coder Next on the measured H100-priced setup Nebius on-demand, September 2026 in the caption . Those multipliers meet or beat aggressive quantization gains — which reframes “just buy more VRAM” as a billing mistake when compaction preserves accuracy. Retail APIs often quote flat per-million-token prices , hiding the depth penalty the figure exposes. FinOps teams should reconcile invoice spikes with session length histograms , not only with model swaps . Edge latency and memory on DGX Spark Figure 2 in the preprint measures inter-token latency , throughput , and memory footprint versus input length on several coding models running locally on NVIDIA DGX Spark . Later tokens can take roughly 2x longer than early tokens on some checkpoints — painful for interactive agents where users stare at partial outputs. Compaction attacks both cloud dollars and local UX stalls . That pairs with explainx.ai guides on MLX local inference https://explainx.ai/blog/mlx-apple-silicon-run-llms-complete-guide-2026 and Scott Hanselman's GPU task tooling https://explainx.ai/blog/shanselman-gputaskman-gpu-llm-task-manager-2026 : shorter resident context is a latency strategy , not only a cost cap . Literary scale versus model marketing Long-context marketing weave versus compaction-first agent design The introduction compares famous books token counts in the tens of thousands to low hundreds of thousands against 200K–1M+ context windows on shipping models. The point is rhetorical but useful in sales reviews: most agent tasks never ingest a novel’s worth of tokens at once , yet providers compete on maximum window as if it were default usage . When vendors pitch million-token RAG , ask whether retrieval plus compaction beats monolithic context on your corpus — the paper’s FRAMES and long-input rows are the empirical hook. Single-action-per-context software engineering The SSSM section reports LLM performance does not collapse on long-horizon software engineering even when taking at most one action per context . That is a strong claim for reset-heavy harnesses : checkpoint state in git, re-open compact summary, issue one tool call, repeat. It challenges harnesses that never clear KV because “the model might forget.” Implementers should still log regressions when compaction summaries omit file paths or error codes . The paper acknowledges tasks where history must remain retrievable — align with Nasiko-style token gateways https://explainx.ai/blog/nasiko-open-runtime-agent-token-spend-gateway-2026 that cap spend without silently dropping audit trails. How this interacts with provider roadmap NVIDIA sells both inference optimization research and GPUs whose revenue grows with tokens processed . This paper argues customers should process fewer context tokens , which could sound self-conflicting until you view it as throughput per megawatt positioning for Rubin-era datacenters . Software compaction increases servable agents per GPU even if per-GPU list price stays flat. Model vendors may still ship larger windows for one-shot document tasks . The paper’s narrow claim is about autonomous agentic workloads with iterative tool use , the same class SWE-Cowork benchmarks https://explainx.ai/blog/swe-cowork-coding-agents-digital-employees-benchmark-2026 target. Reproducibility checklist | Step | Detail | |---|---| | Pin harness version | EoCC rules live in appendix; reproduce threshold sweeps | | Match model IDs | Opus 4.5 and GPT 5.6 Sol names may differ on your API | | Log compaction text | Audit summaries for dropped constraints | | Compare open weights | Read Appendix D if you self-host GLM, Kimi, or Qwen classes | | Report cost honestly | Use your cloud $/GPU-hour, not Nebius snapshot | When your provider offers prompt caching , compaction and caching stack: summaries shrink active KV , while caches reward stable system prompts . Measure both knobs together on the same agent suite before you commit to a 1M context tier for an entire organization or team.