New research from Amazon AGI shows that reinforcement learning fine-tuning can boost long-context model performance by up to 20 points while making inference more efficient
If you’ve ever tried to feed a massive document into an AI model and watched it forget what was on page two by the time it reached page fifty, you’re familiar with the long-context problem. Amazon AGI researchers think they’ve found a way to make that experience significantly less painful, and the key insight is surprisingly elegant: the way you manage a model’s memory during inference should also shape how you train it in the first place.
The paper, titled “Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models,” was submitted to arXiv by a team led by Francesco Maria Molfese, Momchil Hardalov, Rexhina Blloshmi, Bill Byrne, and Adrià de Gispert. Its central finding is that a reinforcement learning technique called Group Relative Policy Optimization, or GRPO, can dramatically improve a language model’s ability to find and use information buried deep within long contexts.
What KV-cache compression actually means for model performance #
To understand why this matters, a quick primer on KV-cache. When a large language model processes text, it stores key-value pairs representing the tokens it has already seen. For short texts, this works fine. For documents stretching to 32,000 tokens or beyond, that cache becomes enormous, expensive, and slow.
The standard solution is compression: selectively trimming the cache so the model doesn’t need to store everything. The problem is that aggressive compression tends to throw away information the model actually needs.
What the Amazon team discovered is that training a model with GRPO and verifiable rewards makes it substantially more resilient to that compression. The researchers developed a method called RetrievalAttention, designed to help models retain critical data even as the cache gets trimmed. When combined with the GRPO fine-tuning approach, the result was a model that could handle context windows trained at 32,000 tokens but generalize to contexts as large as 128,000 tokens, and in some cases even 1 million tokens.
The numbers behind the improvement #
The team tested their approach using two well-known multi-hop question answering datasets: HotpotQA and 2WikiMultihopQA. These are benchmarks that require a model to synthesize information from multiple parts of a document to arrive at a correct answer.
In-domain performance improvements were striking. Fine-tuning with GRPO yielded up to a 20-point improvement on HotpotQA benchmarks compared to baseline configurations.
Out-of-domain results told a more nuanced story but still favored the approach. When tested on finance-related questions, a domain the model wasn’t specifically trained on, the fine-tuned models outperformed Retrieval-Augmented Generation setups by 9 points.
The distinction between GRPO and traditional supervised fine-tuning is worth unpacking. Supervised fine-tuning essentially shows a model the correct answer and asks it to mimic the pattern. GRPO takes a different approach: it lets the model generate multiple candidate answers, scores them against verifiable rewards, and then nudges the model’s policy toward strategies that consistently produce correct results.
Why this research matters beyond benchmarks #
The paper also raises interesting questions about the relationship between training methodology and inference efficiency. Most optimization research treats these as separate problems: you train the model one way, then figure out how to make it run faster afterward. The Amazon team’s work suggests that baking inference constraints into the training process itself, essentially teaching the model to be good at working with limited memory from the start, produces better results than optimizing each stage independently.
The team made their code publicly available on GitHub, which means competitors and collaborators alike can build on the findings. In a field where reproducibility has been a persistent challenge, that openness matters.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our