{"slug": "hyscale-scaling-long-context-training-for-learned-sparse-attention", "title": "HyScale: Scaling Long-Context Training for Learned Sparse Attention", "summary": "A research team released HyScale, a long-context training system for learned sparse attention that speeds up GLM-5.1 training on 1M-token sequences by 2.07× over Megatron-LM on 512 H20 GPUs, according to the arXiv paper 2610.32057v1. HyScale introduces HyTopK, which finds each query's exact top-k on the fly without materializing a full score table, and selection replay, which reuses forward-pass selection during recomputation. The system also speeds up the CSA layers of DeepSeek-V4-Flash by 2.01× and the QSA layers of Qwen3.8-Flash-Next by 1.57× at 1M tokens on 32 H800 GPUs.", "body_md": "arXiv:2610.32057v1 Announce Type: new \nAbstract: We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck. For each query, it scores every previous token whereas sparse attention reads only $k$; worse, current systems materialize a full score table and repeat selection during recomputation. Our key insight is that the output of selection is not an activation, but rather a set of indices. Building on it, HyScale introduces HyTopK and selection replay. HyTopK finds each query's exact top-$k$ on the fly without a score table, while selection replay reuses forward-pass selection in recomputation. Together with holistic system optimizations tackling end-to-end training challenges, HyScale speeds up GLM-5.1 training on 1M-token sequences by 2.07$\\times$ over Megatron-LM on 512 H20 GPUs. It also speeds up the CSA layers of DeepSeek-V4-Flash and the QSA layers of Qwen3.8-Flash-Next by 2.01$\\times$ and 1.57$\\times$ at 1M tokens on 32 H800 GPUs.", "url": "https://wpnews.pro/news/hyscale-scaling-long-context-training-for-learned-sparse-attention", "canonical_source": "https://arxiv.org/abs/2610.32057", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 04:14:58.358509+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["HyScale", "HyTopK", "Megatron-LM", "GLM-5.1", "DeepSeek-V4-Flash", "Qwen3.8-Flash-Next", "DeepSeek-V3.2", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hyscale-scaling-long-context-training-for-learned-sparse-attention", "markdown": "https://wpnews.pro/news/hyscale-scaling-long-context-training-for-learned-sparse-attention.md", "text": "https://wpnews.pro/news/hyscale-scaling-long-context-training-for-learned-sparse-attention.txt", "jsonld": "https://wpnews.pro/news/hyscale-scaling-long-context-training-for-learned-sparse-attention.jsonld"}}