[SkyRL Team](https://anyscale.com/blog?author=skyrl-team)and
[Lila Sciences' AI Research Team](https://anyscale.com/blog?author=lila-sciences-ai-research-team)| November 5, 2026
Lila Sciences and Anyscale trained GLM-5.3-Flash, an open model, on 1 million tokens per sequence using 64 B200 GPUs. That's 64x longer than the training stack could handle when we started, and every change is open source in SkyRL. Our team for SkyRL at Anyscale worked closely with Lila Science researchers on their infrastructure on an open source stack to explore the research path on increasing context length for this particular model.
Scientific data is long by nature. A genome, a family of related protein or RNA sequences, or the record of an experiment run over several days loses the structure that matters when it is cut into a few thousand tokens. AI that runs science in a closed loop with an autonomous lab has the same need: across rounds of hypotheses, protocols and instrument readouts, its context grows to hundreds of thousands of tokens.
Many open models now support 1M-token context, but they may not have out of the box performance for the target task at large context values. Closing that gap means training on the task's own data at its full length, without cutting or heavily compacting the context. This work set out to do it with an open model and an open training stack.
The result, at a glance:
-
1M tokens per sequence fits in 144.5 GiB per GPU, under the B200's 178 GiB.
-
13.7k tokens/s at 1M, and a peak of 24.9k tokens/s at 256k, on 64 B200s.
-
The model uses the context: fine-tuned on the human genome, its loss improves most on tokens with hundreds of thousands of tokens of context before them.
-
Open source: the recipe is inSkyRL , and several fixes are going upstream to Megatron-LM.
LinkHow do we reach 1M tokens context on GLM 5.3 Flash?
GLM-5.3-Flash is a hybrid model: 34 of its 45 layers use linear attention (Kimi Delta Attention), and 11 use DeepSeek Sparse Attention (DSA). It also uses a residual stream that's 4x wider than usual, and mixture-of-experts (MoE) layers. Each of these needed its own work. Memory grows with sequence length in every layer, and in attention it grows with the square of the length, so each optimization only pushes the limit out until the next one is hit. We did several optimizations all which are covered in detail with their own experiments and PRs in the original post.
Most of what we ran into is not specific to GLM and can be applied to models sharing the same type of components:
- Check that each layer runs the kernel you think it does: GLM's sparse attention layers were silently falling back to dense attention, because the fused kernels only accepted DeepSeek's exact tensor shapes. Nothing raised an error; the run simply used 114 GiB per GPU at 16k and ran out of memory at 32k. Padding the tensors to fit the sparse kernel halved memory.
- Sparse attention moves the quadratic cost; it doesn't remove it. DSA attends to only 2,048 tokens per query, but choosing those tokens means scoring every query against the whole sequence. Computing that score matrix a few hundred queries at a time keeps it under 2 GiB, and splitting the scoring across GPUs instead of repeating it on each one made the full training step 2.1x faster at 256k.
- Every memory fix costs throughput, so use only what the length needs. Full recompute, CPU off and context parallelism each buy more context for some speed. Off, for example, cuts peak memory at 256k from 142 to 98 GiB but adds 21% to step time. We turn each one on only at the length where training wouldn't fit without it.
- Benchmarks on random tokens underestimate memory for MoE models. Real text distribution routes unevenly across experts: on a real long context dataset, the busiest GPU received about 4x the average number of tokens, and 512k ran out of memory where random-token runs fit easily. Processing the MoE layer over the sequence in chunks fixed it. Test memory on real data before sizing a cluster.
- Plan host memory, not just GPU memory. At 1M tokens, the layer inputs offloaded to CPU come to about 180 GB per GPU before context parallelism, over 1 TB per 8-GPU node, and they share host RAM with the optimizer state. Use pinned memory for the copies (they were 22% faster than pageable memory) and release it every step.
- Verify correctness separately from speed. Speed and memory numbers can come from random weights, but correctness can't. In RL runs with the real weights, the trainer's log-probabilities matched vLLM's to within 0.04 -- 0.07 (mean absolute difference) from 128k to 1M tokens, and an overfit test drove loss from 1.65 to 0.17 in 20 steps.
With combination of these ideas we were able to maintain a good throughput across different context lengths
LinkTraining Results #
We post-trained on the human genome at 512k tokens per sequence, the model's held-out loss fell 4.8% in 80 steps, and the gain was largest on tokens with the most context before them. That last part is the evidence that it learned to use long context, not just local patterns.
We trained on chromosomes 1 and 2 of the human reference genome (hg38) and evaluated on chromosome 22, which the model never saw. DNA is a good test because it has structure at every scale, from short motifs to patterns spanning very long stretches.
To separate long-range learning from local learning, we split the held-out loss by token position, in 32k-token buckets, and compared it before and after fine-tuning:
- Early tokens have little context before them, so any improvement there comes from local patterns such as short motifs and base composition.
- Late tokens have up to 500k tokens of context before them.
- Local learning helps every position equally, so a larger improvement at late positions has to come from the long context that only they have.
LinkRunning on Anyscale #
Every experiment in this post ran on Ray, managed by the Anyscale platform, on a 64-GPU B200 cluster in Lila's cloud. SkyRL uses Ray to place and coordinate its training and inference workers, so the same setup ran memory benchmarks on random tokens, fine-tuning on real data, and RL runs that check the trainer against vLLM.
For a more in-depth look at the specific techniques used to support long-context see this detailed blog post.