Accelerating Transformer Training with NVIDIA Transformer Engine, Fused Kernels, BF16, FP8, and GPU Benchmarking

NVIDIA's Transformer Engine accelerates transformer training on Ampere and newer GPUs by fusing kernels and using BF16/FP8 computation, with FP8 support limited to compute capability 8.9 or higher. The tutorial demonstrates building a GPT-style model with te.Linear, te.LayerNorm, te.LayerNormLinear, te.LayerNormMLP, and te.TransformerLayer, configuring a delayed-scaling FP8 recipe with hybrid E4M3/E5M2 formats, and benchmarking runtime and memory against pure-PyTorch fallback.

In this tutorial, we explore how NVIDIA Transformer Engine https://github.com/NVIDIA/TransformerEngine accelerates transformer workloads by combining fused GPU kernels, BF16 computation, and hardware-aware FP8 execution. We begin by installing Transformer Engine and detecting the active GPU architecture so that we can determine whether the runtime supports TE kernels, FP8 tensor cores, or only the pure-PyTorch fallback path. We then examine core fused components such as te.Linear, te.LayerNorm, te.LayerNormLinear, te.LayerNormMLP, and te.TransformerLayer, while also configuring a delayed-scaling FP8 recipe that manages tensor scaling, amax history, and hybrid E4M3/E5M2 formats. Using these components, we construct a compact GPT-style causal language model, train it on deterministic synthetic sequences, compare higher-precision and FP8 execution, measure runtime and peak GPU memory, inspect FP8 metadata, and validate the trained model through autoregressive generation. python import subprocess, sys, os def pip install pkgs : subprocess.run sys.executable, "-m", "pip", "install", "-q", "--no-build-isolation", pkgs , check=False print " Installing transformer engine pytorch this can take a few minutes ..." pip install "transformer engine pytorch " import time, math, gc import torch import torch.nn as nn import torch.nn.functional as F assert torch.cuda.is available , "Enable a GPU runtime in Colab first " DEVICE = "cuda" props = torch.cuda.get device properties 0 CC = props.major, props.minor GPU NAME = props.name print f" GPU: {GPU NAME} | compute capability {CC 0 }.{CC 1 } | " f"{props.total memory/1e9:.1f} GB" TE CAPABLE = CC = 8, 0 FP8 CAPABLE = CC = 8, 9 te = None if TE CAPABLE: try: import transformer engine.pytorch as te from transformer engine.common import recipe print " Transformer Engine imported OK:", getattr te, " version ", "unknown version" except Exception as e: print f" TE import failed {e} ; using pure-PyTorch fallback." TE CAPABLE = FP8 CAPABLE = False else: print " GPU is pre-Ampere e.g. T4 : TE kernels unsupported - fallback mode." if TE CAPABLE and FP8 CAPABLE and te is not None: try: ok, reason = te.fp8.check fp8 support FP8 CAPABLE = bool ok if not ok: print " TE reports FP8 unsupported:", reason except Exception: pass print f" Mode: TE={'ON' if TE CAPABLE else 'OFF'} | " f"FP8={'ON' if FP8 CAPABLE else 'OFF will use BF16 '}" torch.manual seed 1234 if TE CAPABLE: H = 768 x demo = torch.randn 8, 32, H, device=DEVICE, dtype=torch.bfloat16 lin = te.Linear H, H, bias=True, params dtype=torch.bfloat16 .to DEVICE ln = te.LayerNorm H, params dtype=torch.bfloat16 .to DEVICE ln lin = te.LayerNormLinear H, 3 H, params dtype=torch.bfloat16 .to DEVICE ln mlp = te.LayerNormMLP H, 4 H, params dtype=torch.bfloat16 .to DEVICE with torch.no grad : print "\n Module tour shapes :" print " te.Linear ", tuple lin x demo .shape print " te.LayerNorm ", tuple ln x demo .shape print " te.LayerNormLinear", tuple ln lin x demo .shape print " te.LayerNormMLP ", tuple ln mlp x demo .shape del lin, ln, ln lin, ln mlp, x demo gc.collect ; torch.cuda.empty cache fp8 recipe = None if FP8 CAPABLE: fp8 recipe = recipe.DelayedScaling fp8 format=recipe.Format.HYBRID, amax history len=16, amax compute algo="max", print "\n FP8 recipe:", fp8 recipe We install NVIDIA Transformer Engine and initialize the PyTorch environment required for GPU-accelerated execution. We inspect the active GPU, compute capability, and memory capacity to determine whether fused TE kernels and FP8 tensor cores are available. We also validate the core fused modules and configure a delayed-scaling FP8 recipe while preserving an automatic PyTorch fallback for unsupported hardware. VOCAB, D MODEL, N HEADS, N LAYERS, FFN, SEQ = 96, 768, 12, 4, 3072, 256 class MiniGPT TE nn.Module : """Causal LM where every block is a single fused te.TransformerLayer.""" def init self : super . init self.emb = nn.Embedding VOCAB, D MODEL self.pos = nn.Embedding SEQ, D MODEL self.blocks = nn.ModuleList te.TransformerLayer hidden size=D MODEL, ffn hidden size=FFN, num attention heads=N HEADS, self attn mask type="causal", layer number=i + 1, params dtype=torch.bfloat16, hidden dropout=0.0, attention dropout=0.0, for i in range N LAYERS self.ln f = nn.LayerNorm D MODEL self.head = nn.Linear D MODEL, VOCAB, bias=False def forward self, idx : B, T = idx.shape h = self.emb idx + self.pos torch.arange T, device=idx.device h = h.to torch.bfloat16 for blk in self.blocks: h = blk h h = self.ln f h.float return self.head h class Block PT nn.Module : """Plain-PyTorch transformer block, mirrors te.TransformerLayer.""" def init self : super . init self.ln1 = nn.LayerNorm D MODEL self.attn = nn.MultiheadAttention D MODEL, N HEADS, batch first=True self.ln2 = nn.LayerNorm D MODEL self.mlp = nn.Sequential nn.Linear D MODEL, FFN , nn.GELU , nn.Linear FFN, D MODEL def forward self, x, mask : a, = self.attn self.ln1 x , self.ln1 x , self.ln1 x , attn mask=mask, need weights=False x = x + a return x + self.mlp self.ln2 x class MiniGPT PT nn.Module : def init self : super . init self.emb = nn.Embedding VOCAB, D MODEL self.pos = nn.Embedding SEQ, D MODEL self.blocks = nn.ModuleList Block PT for in range N LAYERS self.ln f = nn.LayerNorm D MODEL self.head = nn.Linear D MODEL, VOCAB, bias=False def forward self, idx : B, T = idx.shape mask = torch.triu torch.full T, T , float "-inf" , device=idx.device , diagonal=1 h = self.emb idx + self.pos torch.arange T, device=idx.device for blk in self.blocks: h = blk h, mask return self.head self.ln f h model = MiniGPT TE if TE CAPABLE else MiniGPT PT .to DEVICE n params = sum p.numel for p in model.parameters print f"\n Model: {'TE fused' if TE CAPABLE else 'pure PyTorch'} | " f"{n params/1e6:.1f}M params | {N LAYERS} layers x {D MODEL}d" We define a compact causal language model using fused te.TransformerLayer blocks for Transformer Engine execution. We also implement an equivalent pure-PyTorch transformer architecture with multi-head attention, layer normalization, residual connections, and feed-forward networks. We select the appropriate model dynamically according to GPU support and report the final parameter count and architectural dimensions. python def make batch bsz=16 : phase = torch.randint 0, VOCAB, bsz, 1 stride = torch.randint 1, 7, bsz, 1 steps = torch.arange SEQ + 1 .unsqueeze 0 seq = phase + stride steps % VOCAB return seq :, :-1 .to DEVICE , seq :, 1: .to DEVICE opt = torch.optim.AdamW model.parameters , lr=3e-4 def run step x, y, use fp8 : if TE CAPABLE and use fp8: with te.fp8 autocast enabled=True, fp8 recipe=fp8 recipe : logits = model x else: logits = model x loss = F.cross entropy logits.float .reshape -1, VOCAB , y.reshape -1 opt.zero grad set to none=True loss.backward opt.step return loss.item print f"\n Training 60 steps {'FP8' if FP8 CAPABLE else 'BF16/FP32'} ..." t0 = time.time for step in range 1, 61 : x, y = make batch loss = run step x, y, use fp8=FP8 CAPABLE if step % 10 == 0: print f" step {step:3d} | loss {loss:.4f} | " f"{ time.time -t0 /step 1000:.0f} ms/step" print f" Final loss: {loss:.4f} random guess would be ~{math.log VOCAB :.2f} " We create deterministic arithmetic-pattern sequences that allow the model to learn predictable token transitions across the vocabulary. We configure the AdamW optimizer and implement a training step that conditionally wraps the forward pass in te.fp8 autocast when FP8 execution is supported. We train the model for multiple iterations, monitor the loss and step latency, and compare the final loss against the random-guess baseline. python def bench use fp8, iters=30, warmup=10 : x, y = make batch bsz=32 for in range warmup : run step x, y, use fp8 torch.cuda.synchronize torch.cuda.reset peak memory stats t = time.time for in range iters : run step x, y, use fp8 torch.cuda.synchronize ms = time.time - t / iters 1000 mem = torch.cuda.max memory allocated / 1e9 return ms, mem print "\n Benchmark batch 32, seq 256, fwd+bwd+optim :" ms hi, mem hi = bench use fp8=False print f" {'BF16' if TE CAPABLE else 'FP32'}: {ms hi:7.1f} ms/step | " f"peak mem {mem hi:.2f} GB" if FP8 CAPABLE: ms f8, mem f8 = bench use fp8=True print f" FP8 : {ms f8:7.1f} ms/step | peak mem {mem f8:.2f} GB" print f" Speedup: {ms hi/ms f8:.2f}x " f" gains grow with model size — try D MODEL=2048, N LAYERS=12 " else: print " FP8 benchmark skipped — needs an sm 89+ GPU L4/H100/Ada/Blackwell ." if FP8 CAPABLE: blk = model.blocks 0 for name, m in blk.named modules : meta = getattr m, "fp8 meta", None if meta and "scaling fwd" in meta: s = meta "scaling fwd" print f"\n FP8 state of block-0 submodule '{name}':" print " scale :", s.scale.flatten :4 .tolist print " amax history:", s.amax history 0, :4 .tolist break We benchmark forward propagation, backpropagation, and optimizer updates using higher-precision and FP8 execution modes. We measure average training-step latency and peak allocated GPU memory to quantify the performance and memory impact of reduced-precision computation. We also inspect the scaling factors and amax history maintained by Transformer Engine to understand how delayed scaling stabilizes FP8 tensors. python @torch.no grad def generate prompt len=8, gen len=24 : x, = make batch bsz=1 ctx = x :, :prompt len for in range gen len : inp = ctx :, -SEQ: if TE CAPABLE and FP8 CAPABLE: with te.fp8 autocast enabled=True, fp8 recipe=fp8 recipe : logits = model inp else: logits = model inp nxt = logits :, -1 .argmax -1, keepdim=True ctx = torch.cat ctx, nxt , dim=1 return ctx 0 .tolist seq = generate print "\n Greedy generation should continue the arithmetic pattern :" print " prompt+gen:", seq diffs = b - a % VOCAB for a, b in zip seq, seq 1: print " step diffs:", diffs, "<- constant stride = model learned the rule" print "\n Done Things to try next:" print " Scale up: D MODEL=2048, N LAYERS=12 - FP8 speedup becomes dramatic" print " recipe.Format.E4M3 vs HYBRID; amax history len=1024" print " te.LayerNormMLP / te.LayerNormLinear in your own architectures" print " fp8 model init to store weights themselves in FP8 for inference" We implement greedy autoregressive generation by repeatedly feeding the latest context into the trained causal language model. We compare consecutive generated tokens to verify whether the model preserves the constant arithmetic stride present in the synthetic training data. We conclude by identifying practical extensions, including larger model dimensions, alternative FP8 formats, longer amax histories, fused modules, and FP8 weight initialization. In conclusion, we demonstrated how we integrate NVIDIA Transformer Engine into an end-to-end transformer training workflow while preserving compatibility across different Colab GPU environments. We used fused transformer modules to reduce kernel-launch overhead and memory traffic, applied FP8 autocasting with delayed scaling when supported, and retained BF16 or FP32 execution through an automatic PyTorch fallback. By training and benchmarking the same mini causal language model, we observed how hardware capability, numerical format, fused execution, and model scale influence training speed and memory consumption. We also inspected the internal scaling factors and amax history that support stable FP8 computation, which gives us a clearer understanding of how Transformer Engine manages reduced-precision arithmetic. Check out the Full Codes. Also, feel free to follow us on and don’t forget to join our Twitter https://x.com/intent/follow?screen name=marktechpost and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://www.aidevsignals.com/ now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.