Rulin Shao<sup>1,2</sup>, Shannon Zejiang Shen<sup>3</sup>, Junjie Oscar Yin<sup>1,2</sup>, Yuetai Li<sup>1</sup>, Minheng Wang<sup>1</sup>, Hamish Ivison<sup>1</sup>, Radha Poovendran<sup>1</sup>, Nathan Lambert<sup>4</sup>, Teng Xiao<sup>1</sup>, Mike Lewis<sup>2</sup>, Wen-tau Yih<sup>2</sup>, Luke Zettlemoyer<sup>1,2</sup>, Pang Wei Koh<sup>1</sup>
<sup>1</sup>University of Washington <sup>2</sup>Meta Superintelligence Labs <sup>3</sup>MIT <sup>4</sup>Trillium Labs
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files.
- Zero-shot. Building CLMs zero-shot with existing models outperforms SOTA context-management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task.
- In-context learning. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute.
- Reinforcement learning. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs.
Run the minimal CLM agent on any Harbor task:
pip install -e .
clm-harbor run -p <harbor-task> -a clm-minimal -m openai/<model> \
--agent-kwarg api_base=http://localhost:8000/v1
clm-harbor is the Harbor CLI with CLM available as -a clm-minimal. See
clm/clm_harness for configuration and serving.
Day 1 support: pi-clm for Pi agent #
pi install npm:@lolipopshock/pi-clm
| clm/clm_harness | CLM implemented in Harbor |
| clm/clm_icl | skill evolution |
| clm/clm_rl | reinforcement learning |
| suffix_cache_reuse | Suffix Cache Reuse: KV-cache reuse for CLM serving, as a patch to SGLang |
- ContextBench
If you find our work helpful, we would appreciate it if you could cite our paper:
@article{shao2026context,
title = {Context Language Models},
author = {Shao, Rulin and Shen, Shannon Zejiang and Yin, Junjie Oscar and Li, Yuetai and
Wang, Minheng and Ivison, Hamish and Poovendran, Radha and Lambert, Nathan and
Xiao, Teng and Lewis, Mike and Yih, Wen-tau and Zettlemoyer, Luke and Koh, Pang Wei},
journal = {arXiv preprint arXiv:2609.37725},
year = {2026}
}
This project is licensed under CC BY-NC 4.0. See also NOTICE.