{"slug": "training-variable-long-sequences-with-data-centric-parallel", "title": "Training Variable Long Sequences with Data-Centric Parallel", "summary": "Researchers introduced Data-Centric Parallel (DCP), a method that dynamically adjusts runtime settings based on each batch's sequence length to train deep learning models on variable long sequences, achieving up to a 2.88x speedup on 32 H200 GPUs. DCP can be integrated into any model with 10 lines of code, offering a simple and effective baseline for distributed training.", "body_md": "arXiv:2608.07524v1 Announce Type: new\nAbstract: Training deep learning models on variable long sequences poses significant computational challenges. Existing methods force a difficult trade-off between efficiency and ease-of-use. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models. To break this trade-off, we introduce Data-Centric Parallel (DCP). Its core principle is to let the data itself drive the runtime. It achieves this by dynamically adjusting direct runtime settings (e.g., parallel size, gradient accumulation, recomputation) based on each batch's sequence length. Empirical results demonstrate that our method achieves up to a 2.88$\\times$ speedup on 32 H200 GPUs. Designed for generalization, it can be integrated into any model with 10 lines of code. We anticipate this simple yet effective approach will serve as a robust baseline and facilitate future advancements in distributed training for variable long sequences.", "url": "https://wpnews.pro/news/training-variable-long-sequences-with-data-centric-parallel", "canonical_source": "https://arxiv.org/abs/2608.07524", "published_at": "2026-08-11 04:00:00+00:00", "updated_at": "2026-08-11 04:16:02.684806+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-infrastructure"], "entities": ["Data-Centric Parallel", "H200"], "alternates": {"html": "https://wpnews.pro/news/training-variable-long-sequences-with-data-centric-parallel", "markdown": "https://wpnews.pro/news/training-variable-long-sequences-with-data-centric-parallel.md", "text": "https://wpnews.pro/news/training-variable-long-sequences-with-data-centric-parallel.txt", "jsonld": "https://wpnews.pro/news/training-variable-long-sequences-with-data-centric-parallel.jsonld"}}