NVIDIA Transformer Engine

NVIDIA Transformer Engine is now integrated with JAX to accelerate dropless Mixture of Experts (MoE) training, targeting the conditional computation bottleneck in architectures used by DeepSeek, Qwen,…

NVIDIA reported that combining its Transformer Engine library with the JAX Python library raised DeepSeek-V3 MoE training throughput on NVIDIA GB200 GPUs from an unoptimized baseline of 103 TFLOPS/GPU…

NVIDIA's Transformer Engine accelerates transformer training on Ampere and newer GPUs by fusing kernels and using BF16/FP8 computation, with FP8 support limited to compute capability 8.9 or higher. Th…