{"slug": "block-parallelism-for-efficient-distributed-long-context-diffusion-language", "title": "Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training", "summary": "Researchers introduced block parallelism (BP) and context-sharded block parallelism (CSBP), new distributed parallelism dimensions for training block diffusion language models (BDLMs) on long contexts, reporting throughput gains of 1.18-1.45x for supervised fine-tuning and 1.27-1.33x for converting autoregressive models to BDLMs on 16 H200 GPUs at 256K context. CSBP reached 1.61x full-model speedup at 512K context, accelerated DFlash2 speculative-decoder training by 2.48x at 512K and 7.59x at 1M on eight H100 GPUs, and in matched 12-hour DiffusionGemma 26B-A4B SFT runs achieved higher pass rates at every trained checkpoint on SWE-bench Verified and Terminal-Bench Lite. The code is available at https://github.com/ScalingIntelligence/Turbo-dLLM.", "body_md": "arXiv:2609.19242v1 Announce Type: new \nAbstract: Block diffusion language models (BDLMs) combine autoregressive dependencies across blocks with parallel denoising within blocks, but long-context training is constrained by distributed attention communication and activation memory. Conventional context parallelism (CP) shards the combined clean-plus-corrupted sequence by position, communicating shared clean K/V together with block-specific corrupted K/V and their gradients. We observe that the BDLM objective separates over target blocks. We introduce block parallelism (BP), a new distributed parallelism dimension that assigns each corrupted-block computation to one rank. To scale BP to long contexts, we introduce context-sharded block parallelism (CSBP), which also shards the shared clean sequence across those ranks. CSBP keeps corrupted K/V and gradients local, avoids replicated clean prefixes, and preserves BDLM training semantics. On 16 H200 GPUs at 256K context, CSBP improves throughput over the best baseline by 1.18-1.45x for supervised fine-tuning and 1.27-1.33x for conversion of autoregressive models to BDLMs, while matching or reducing peak HBM. Full-model speedup reaches 1.61x at 512K. On eight H100 GPUs, CSBP accelerates DFlash2 speculative-decoder training by 2.48x at 512K and 7.59x at 1M. In matched 12-hour DiffusionGemma 26B-A4B SFT runs, CSBP achieves higher pass rates at every trained checkpoint on SWE-bench Verified and Terminal-Bench Lite. Code: https://github.com/ScalingIntelligence/Turbo-dLLM", "url": "https://wpnews.pro/news/block-parallelism-for-efficient-distributed-long-context-diffusion-language", "canonical_source": "https://arxiv.org/abs/2609.19242", "published_at": "2026-09-18 04:00:00+00:00", "updated_at": "2026-09-18 04:23:23.439150+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure", "machine-learning"], "entities": ["Block Parallelism", "Context-Sharded Block Parallelism", "H200", "H100", "DiffusionGemma 26B-A4B", "SWE-bench Verified", "Terminal-Bench Lite", "Turbo-dLLM"], "alternates": {"html": "https://wpnews.pro/news/block-parallelism-for-efficient-distributed-long-context-diffusion-language", "markdown": "https://wpnews.pro/news/block-parallelism-for-efficient-distributed-long-context-diffusion-language.md", "text": "https://wpnews.pro/news/block-parallelism-for-efficient-distributed-long-context-diffusion-language.txt", "jsonld": "https://wpnews.pro/news/block-parallelism-for-efficient-distributed-long-context-diffusion-language.jsonld"}}