{"slug": "scaling-to-1t-moe-on-torchtitan-p", "title": "Scaling to 1T MoE on TorchTitan [P]", "summary": "A team working with the TorchTitan and PyTorch maintainers trained a roughly 1-trillion-parameter mixture-of-experts model at about 1,300 tokens per second per GPU across 384 GH200 GPUs, the processors inside the JUPITER supercomputer. The result reports how far the GH200 hardware can be pushed for large-scale MoE training using the TorchTitan stack.", "body_md": "Hi everyone! Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. That", "url": "https://wpnews.pro/news/scaling-to-1t-moe-on-torchtitan-p", "canonical_source": "https://aiflash.com/news/151423/", "published_at": "2026-11-14 09:00:31+00:00", "updated_at": "2026-11-14 09:21:46.359419+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "large-language-models", "ai-research"], "entities": ["TorchTitan", "PyTorch", "GH200", "JUPITER"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/scaling-to-1t-moe-on-torchtitan-p", "markdown": "https://wpnews.pro/news/scaling-to-1t-moe-on-torchtitan-p.md", "text": "https://wpnews.pro/news/scaling-to-1t-moe-on-torchtitan-p.txt", "jsonld": "https://wpnews.pro/news/scaling-to-1t-moe-on-torchtitan-p.jsonld"}}