{"slug": "jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope", "title": "Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE", "summary": "Researchers propose Jet-Long, a tuning-free zero-shot method for extending LLM context windows that dynamically adjusts a rescaling factor based on sequence length, preserving short-context fidelity while enabling long-context extrapolation. On Qwen3 models up to 128K context, Jet-Long outperforms baselines on RULER, HELMET-RAG, and PG-19 benchmarks, and achieves up to 1.39x FA2 throughput on H100 with minimal overhead.", "body_md": "arXiv:2607.07740v1 Announce Type: new\nAbstract: Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints. Most existing zero-shot methods fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts. We propose Jet-Long, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; fused into a single CuTe kernel, long-context prefill reaches up to $1.39\\times$ FA2 throughput on H100 (approaching the Hopper-only FA4), and single-batch generation incurs $\\le 4\\%$ overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads RULER by $+4.79$/$+2.18$/$+2.03$~pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron for further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.", "url": "https://wpnews.pro/news/jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope", "canonical_source": "https://arxiv.org/abs/2607.07740", "published_at": "2026-07-10 04:00:00+00:00", "updated_at": "2026-07-10 04:16:27.124398+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence"], "entities": ["Jet-Long", "Qwen3", "RULER", "HELMET-RAG", "PG-19", "FA2", "H100", "Jet-Nemotron"], "alternates": {"html": "https://wpnews.pro/news/jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope", "markdown": "https://wpnews.pro/news/jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope.md", "text": "https://wpnews.pro/news/jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope.txt", "jsonld": "https://wpnews.pro/news/jet-long-efficient-long-context-extension-with-dynamic-bifocal-rope.jsonld"}}