Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
Researchers introduced block parallelism (BP) and context-sharded block parallelism (CSBP), new distributed parallelism dimensions for training block diffusion language models (BDLMs) on long contexts, reporting throughp…