{"slug": "inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn", "title": "Inside NVIDIA’s cuDNN Graph API: Fusion, Autotuning, and Plan Reuse with cuDNN Frontend", "summary": "NVIDIA's cuDNN Frontend graph API lets developers express a computation as a graph of operations, have cuDNN select an execution engine, and then override that engine choice, according to a Marktechpost tutorial built on the open-source cudnn-frontend repository. The tutorial runs the full five-step pipeline — validate, build operation graph, create execution plans, check support, and build plans — on a single Colab GPU, executing against a variant pack of pointers and validating each result against a PyTorch reference. Topics progress from a single fused convolution to autotuning across engine configs, FP8-style epilogues, attention, plan serialization, dynamic shapes, and CUDA graph capture.", "body_md": "In this **[tutorial](https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Deep%20Learning/cudnn_frontend_nvidia_tutorial_Marktechpost.ipynb)**, we work through the [** cuDNN Frontend**](https://github.com/NVIDIA/cudnn-frontend)‘s graph API from below the framework: we describe a computation as a graph of operations, let cuDNN pick an engine to run it, and then take control of that choice ourselves. Every kernel we build here is expressed the same way: we declare tensors by their dimensions and strides, chain operations onto them, run the five-step build pipeline of validate, build operation graph, create execution plans, check support, and build plans, and then execute against a variant pack of pointers. We run it all on a single Colab GPU, checking each result against a PyTorch reference so we can see both that the fusion is correct and what it costs. The topics build on each other, moving from a single fused convolution to autotuning across engine configs, FP8-style epilogues, attention, plan serialization, dynamic shapes, and CUDA graph capture.\n\n``` python\nimport os\nimport sys\nimport glob\nimport math\nimport time\nimport ctypes\nimport traceback\nimport subprocess\nRESULTS = {}\ndef banner(title):\n   print(\"\\n\" + \"=\" * 78)\n   print(title)\n   print(\"=\" * 78)\ndef section(name):\n   def wrap(fn):\n       def run(*a, **kw):\n           banner(name)\n           try:\n               out = fn(*a, **kw)\n               RESULTS[name] = out if isinstance(out, str) else \"ok\"\n               return out\n           except Exception as e:\n               RESULTS[name] = f\"SKIPPED / FAILED -> {type(e).__name__}: {e}\"\n               print(f\"\\n[!] {name} did not complete: {type(e).__name__}: {e}\")\n               traceback.print_exc(limit=3)\n               return None\n       return run\n   return wrap\nbanner(\"0. Install nvidia-cudnn-frontend and locate libcudnn\")\nsubprocess.run(\n   [sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"nvidia-cudnn-frontend\"],\n   check=True,\n)\nimport torch\nassert torch.cuda.is_available(), \"No GPU. Runtime -> Change runtime type -> GPU.\"\ntorch.backends.cudnn.enabled = True\n_ = torch.nn.functional.conv2d(\n   torch.randn(1, 1, 8, 8, device=\"cuda\"), torch.randn(1, 1, 3, 3, device=\"cuda\")\n)\ntorch.cuda.synchronize()\ntry:\n   import nvidia.cudnn\n   _libdir = os.path.join(os.path.dirname(nvidia.cudnn.__file__), \"lib\")\n   os.environ[\"CUDNN_PATH\"] = os.path.dirname(nvidia.cudnn.__file__)\n   os.environ[\"LD_LIBRARY_PATH\"] = _libdir + \":\" + os.environ.get(\"LD_LIBRARY_PATH\", \"\")\n   for _so in sorted(glob.glob(os.path.join(_libdir, \"libcudnn*.so*\"))):\n       try:\n           ctypes.CDLL(_so, mode=ctypes.RTLD_GLOBAL)\n       except OSError:\n           pass\nexcept Exception as _e:\n   print(f\"  (no pip cuDNN package found, relying on system cuDNN: {_e})\")\nimport cudnn\nprint(\"  cuDNN frontend imported successfully.\")\nbanner(\"1. Environment\")\nDEV = torch.device(\"cuda\")\nMAJOR, MINOR = torch.cuda.get_device_capability()\nSM = MAJOR * 10 + MINOR\nCUDNN_VER = cudnn.backend_version()\nprint(f\"  GPU                 : {torch.cuda.get_device_name(0)}\")\nprint(f\"  Compute capability  : sm_{SM}\")\nprint(f\"  Torch / CUDA        : {torch.__version__} / {torch.version.cuda}\")\nprint(f\"  cuDNN backend       : {CUDNN_VER}\")\ntry:\n   print(f\"  cuDNN version str   : {cudnn.backend_version_string()}\")\nexcept Exception:\n   pass\nDTYPE = torch.bfloat16 if SM >= 80 else torch.float16\nHAS_SDPA = SM >= 80\nprint(f\"  Working dtype       : {DTYPE}\")\nprint(f\"  Fused SDPA usable   : {HAS_SDPA}\")\nHANDLE = cudnn.create_handle()\nTORCH2CUDNN = {\n   torch.float16: cudnn.data_type.HALF,\n   torch.bfloat16: cudnn.data_type.BFLOAT16,\n   torch.float32: cudnn.data_type.FLOAT,\n   torch.int32: cudnn.data_type.INT32,\n   torch.int64: cudnn.data_type.INT64,\n   torch.int8: cudnn.data_type.INT8,\n   torch.uint8: cudnn.data_type.UINT8,\n}\ndef tensor_of(graph, t, name):\n   return graph.tensor(\n       name=name,\n       dim=list(t.size()),\n       stride=list(t.stride()),\n       data_type=TORCH2CUDNN[t.dtype],\n   )\ndef scalar_of(graph, name):\n   return graph.tensor(\n       name=name,\n       dim=[1, 1, 1],\n       stride=[1, 1, 1],\n       data_type=cudnn.data_type.FLOAT,\n       is_pass_by_value=True,\n   )\ndef build(graph, heur=None, policy=None):\n   heur = heur or [cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]\n   graph.validate()\n   graph.build_operation_graph()\n   graph.create_execution_plans(heur)\n   graph.check_support()\n   if policy is None:\n       graph.build_plans()\n   else:\n       graph.build_plans(policy)\n   return graph\ndef workspace_for(graph):\n   n = graph.get_workspace_size()\n   return torch.empty(max(n, 1), device=DEV, dtype=torch.uint8)\ndef bench(fn, warmup=10, iters=50):\n   for _ in range(warmup):\n       fn()\n   torch.cuda.synchronize()\n   s, e = torch.cuda.Event(True), torch.cuda.Event(True)\n   s.record()\n   for _ in range(iters):\n       fn()\n   e.record()\n   torch.cuda.synchronize()\n   return s.elapsed_time(e) / iters\ndef tflops(flops, ms):\n   return flops / (ms * 1e-3) / 1e12\ndef report(tag, ms, flops=None):\n   extra = f\"   ({tflops(flops, ms):7.2f} TFLOP/s)\" if flops else \"\"\n   print(f\"    {tag:<34s} {ms:8.3f} ms{extra}\")\n```\n\nWe start by installing nvidia-cudnn-frontend and solving the problem that trips up most first runs: making libcudnn.so visible to the frontend’s dynamic loader. We force PyTorch to load its bundled cuDNN first and then preload the shared objects explicitly, so the frontend’s own dlopen resolves against a library already resident in the process. We then report the compute capability, pick bfloat16 or float16 accordingly, create the cuDNN handle, and define the helpers for tensor description, graph building, workspace allocation, and event-based benchmarking that the rest of the notebook reuses.\n\n```\nN, C, H, W = 32, 128, 56, 56\nK, R, S = 256, 3, 3\nPAD, STR, DIL = 1, 1, 1\nP = (H + 2 * PAD - DIL * (R - 1) - 1) // STR + 1\nQ = (W + 2 * PAD - DIL * (S - 1) - 1) // STR + 1\nCONV_FLOPS = 2 * N * K * P * Q * C * R * S\nCONV_STATE = {}\n@section(\"2. Fused Conv -> Bias -> ReLU\")\ndef conv_fusion():\n   x = torch.randn(N, C, H, W, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n   w = torch.randn(K, C, R, S, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n   b = torch.randn(1, K, 1, 1, device=DEV, dtype=DTYPE)\n   y = torch.empty(N, K, P, Q, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n   g = cudnn.pygraph(\n       handle=HANDLE,\n       name=\"conv_bias_relu\",\n       io_data_type=TORCH2CUDNN[DTYPE],\n       intermediate_data_type=cudnn.data_type.FLOAT,\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   X = tensor_of(g, x, \"X\")\n   Wt = tensor_of(g, w, \"W\")\n   Bt = tensor_of(g, b, \"bias\")\n   conv = g.conv_fprop(\n       image=X, weight=Wt,\n       padding=[PAD, PAD], stride=[STR, STR], dilation=[DIL, DIL],\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   biased = g.bias(input=conv, bias=Bt)\n   Y = g.relu(input=biased)\n   Y.set_output(True).set_data_type(TORCH2CUDNN[DTYPE])\n   Y.set_dim(list(y.size())).set_stride(list(y.stride()))\n   t0 = time.perf_counter()\n   build(g)\n   build_ms = (time.perf_counter() - t0) * 1e3\n   ws = workspace_for(g)\n   pack = {X: x, Wt: w, Bt: b, Y: y}\n   g.execute(pack, ws)\n   torch.cuda.synchronize()\n   ref = torch.relu(torch.nn.functional.conv2d(x, w, bias=b.flatten(), padding=PAD))\n   err = (y.float() - ref.float()).abs().max().item()\n   scale = ref.float().abs().max().item()\n   print(f\"    problem  : N{N} C{C} {H}x{W} -> K{K} {R}x{S}  ({DTYPE})\")\n   print(f\"    build    : {build_ms:.1f} ms   workspace: {ws.numel()/1024:.1f} KiB\")\n   print(f\"    max |err|: {err:.4f}  (ref max {scale:.2f}, rel {err/max(scale,1e-9):.2e})\")\n   assert err / max(scale, 1e-9) < 5e-2, \"numerical mismatch vs PyTorch\"\n   ms_cudnn = bench(lambda: g.execute(pack, ws))\n   ms_torch = bench(lambda: torch.relu(\n       torch.nn.functional.conv2d(x, w, bias=b.flatten(), padding=PAD)))\n   print()\n   report(\"cuDNN FE (single fused kernel)\", ms_cudnn, CONV_FLOPS)\n   report(\"PyTorch (conv+bias, then relu)\", ms_torch, CONV_FLOPS)\n   print(f\"    speedup: {ms_torch/ms_cudnn:.2f}x\")\n   CONV_STATE.update(graph=g, pack=pack, ws=ws, x=x, w=w, b=b, y=y)\n   return f\"{ms_cudnn:.3f} ms, {tflops(CONV_FLOPS, ms_cudnn):.1f} TFLOP/s\"\nconv_fusion()\n```\n\nWe build our first graph, a convolution followed by a bias add and a ReLU, all fused into a single kernel. We keep every tensor in channels_last because that is what gives cuDNN the NHWC strides its tensor-core engines want, and we pin the output dimensions and strides explicitly so the result is written back in the same layout. We validate the output against torch.nn.functional.conv2d, then benchmark the fused graph against PyTorch running the convolution and activation as separate kernels.\n\n```\n@section(\"3. Autotuning: build ALL plans, time each engine config\")\ndef autotune():\n   x, w, b, y = CONV_STATE[\"x\"], CONV_STATE[\"w\"], CONV_STATE[\"b\"], CONV_STATE[\"y\"]\n   g = cudnn.pygraph(\n       handle=HANDLE, name=\"conv_autotune\",\n       io_data_type=TORCH2CUDNN[DTYPE],\n       intermediate_data_type=cudnn.data_type.FLOAT,\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   X = tensor_of(g, x, \"X\")\n   Wt = tensor_of(g, w, \"W\")\n   Bt = tensor_of(g, b, \"bias\")\n   Y = g.relu(input=g.bias(\n       input=g.conv_fprop(image=X, weight=Wt, padding=[PAD, PAD],\n                          stride=[STR, STR], dilation=[DIL, DIL],\n                          compute_data_type=cudnn.data_type.FLOAT),\n       bias=Bt))\n   Y.set_output(True).set_data_type(TORCH2CUDNN[DTYPE])\n   Y.set_dim(list(y.size())).set_stride(list(y.stride()))\n   g.validate()\n   g.build_operation_graph()\n   g.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.B, cudnn.heur_mode.FALLBACK])\n   g.check_support()\n   g.build_plans(cudnn.build_plan_policy.ALL)\n   n_plans = g.get_execution_plan_count()\n   print(f\"    {n_plans} candidate engine configs survived support checks\\n\")\n   pack = {X: x, Wt: w, Bt: b, Y: y}\n   timings = []\n   for i in range(n_plans):\n       try:\n           g.build_plan_at_index(i)\n           ws_sz = max(g.get_workspace_size_plan_at_index(i), 1)\n           ws = torch.empty(ws_sz, device=DEV, dtype=torch.uint8)\n           ms = bench(lambda: g.execute_plan_at_index(pack, ws, i), warmup=3, iters=15)\n           timings.append((ms, i, ws_sz))\n           print(f\"      plan {i:>3d}: {ms:8.3f} ms  \"\n                 f\"{tflops(CONV_FLOPS, ms):7.2f} TFLOP/s  ws={ws_sz/1024:8.1f} KiB\")\n       except Exception as e:\n           print(f\"      plan {i:>3d}: unusable ({type(e).__name__})\")\n   assert timings, \"no plan executed\"\n   timings.sort()\n   best_ms, best_i, best_ws = timings[0]\n   worst_ms = timings[-1][0]\n   print(f\"\\n    fastest = plan {best_i} @ {best_ms:.3f} ms\")\n   print(f\"    slowest = {worst_ms:.3f} ms  -> {worst_ms/best_ms:.1f}x spread across engines\")\n   print(\"    Takeaway: heuristics are good, but for a hot shape you ship the\")\n   print(\"    autotuned index (or the serialized plan from section 6).\")\n   return f\"best plan {best_i} @ {best_ms:.3f} ms ({worst_ms/best_ms:.1f}x spread)\"\nautotune()\n```\n\nWe rebuild the same convolution but stop trusting the heuristic, asking for plans from heuristic modes A, B, and FALLBACK and compiling all of them with build_plan_policy.ALL. We then walk the plan list, build each config, allocate its specific workspace, and time it with execute_plan_at_index, printing throughput and workspace size for every candidate. The spread between the fastest and slowest engine is the point of the exercise, because it tells us how much we gain by shipping an autotuned index instead of accepting the default pick.\n\n``` php\n@section(\"4. Matmul -> scale -> bias -> activation -> AMAX\")\ndef matmul_epilogue():\n   Bsz, M, Kd, Nd = 16, 512, 1024, 512\n   MM_FLOPS = 2 * Bsz * M * Nd * Kd\n   a = torch.randn(Bsz, M, Kd, device=DEV, dtype=DTYPE)\n   bm = torch.randn(Bsz, Kd, Nd, device=DEV, dtype=DTYPE)\n   bias = torch.randn(1, 1, Nd, device=DEV, dtype=DTYPE)\n   out = torch.empty(Bsz, M, Nd, device=DEV, dtype=DTYPE)\n   amax = torch.empty(1, 1, 1, device=DEV, dtype=torch.float32)\n   alpha_val = 0.125\n   alpha = torch.full((1, 1, 1), alpha_val, dtype=torch.float32)\n   g = cudnn.pygraph(\n       handle=HANDLE, name=\"matmul_epilogue\",\n       io_data_type=TORCH2CUDNN[DTYPE],\n       intermediate_data_type=cudnn.data_type.FLOAT,\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   A = tensor_of(g, a, \"A\")\n   Bt = tensor_of(g, bm, \"B\")\n   BIAS = tensor_of(g, bias, \"bias\")\n   ALPHA = scalar_of(g, \"alpha\")\n   acc = g.matmul(A=A, B=Bt, compute_data_type=cudnn.data_type.FLOAT)\n   scaled = g.mul(a=acc, b=ALPHA)\n   biased = g.bias(input=scaled, bias=BIAS)\n   act_name = \"relu\"\n   if hasattr(g, \"gelu\"):\n       try:\n           act = g.gelu(input=biased)\n           act_name = \"gelu\"\n       except Exception:\n           act = g.relu(input=biased)\n   else:\n       act = g.relu(input=biased)\n   print(f\"    activation used: {act_name}\")\n   OUT = act\n   OUT.set_output(True).set_data_type(TORCH2CUDNN[DTYPE])\n   have_amax = True\n   try:\n       AMAX = g.reduction(input=act, mode=cudnn.reduction_mode.AMAX,\n                          compute_data_type=cudnn.data_type.FLOAT)\n       AMAX.set_output(True).set_data_type(cudnn.data_type.FLOAT)\n       AMAX.set_dim([1, 1, 1]).set_stride([1, 1, 1])\n   except Exception as e:\n       have_amax = False\n       print(f\"    (AMAX reduction unavailable here: {e})\")\n   build(g)\n   ws = workspace_for(g)\n   pack = {A: a, Bt: bm, BIAS: bias, ALPHA: alpha, OUT: out}\n   if have_amax:\n       pack[AMAX] = amax\n   g.execute(pack, ws)\n   torch.cuda.synchronize()\n   ref = torch.matmul(a.float(), bm.float()) * alpha_val + bias.float()\n   ref = torch.nn.functional.gelu(ref) if act_name == \"gelu\" else torch.relu(ref)\n   rel = ((out.float() - ref).abs().max() / ref.abs().max()).item()\n   print(f\"    shape    : ({Bsz},{M},{Kd}) x ({Bsz},{Kd},{Nd})\")\n   print(f\"    rel err  : {rel:.2e}\")\n   if have_amax:\n       print(f\"    fused AMAX {amax.item():.4f} vs torch {ref.abs().max().item():.4f}\")\n   ms = bench(lambda: g.execute(pack, ws))\n   def torch_ref():\n       r = torch.baddbmm(bias.expand(Bsz, M, Nd), a, bm, beta=1.0, alpha=alpha_val)\n       r = torch.nn.functional.gelu(r) if act_name == \"gelu\" else torch.relu(r)\n       return r.abs().amax()\n   ms_t = bench(torch_ref)\n   print()\n   report(\"cuDNN FE (one fused kernel)\", ms, MM_FLOPS)\n   report(\"PyTorch (bmm + act + amax)\", ms_t, MM_FLOPS)\n   print(f\"    speedup: {ms_t/ms:.2f}x  -- the win is the epilogue traffic, not the GEMM\")\n   return f\"{ms:.3f} ms, {tflops(MM_FLOPS, ms):.1f} TFLOP/s, {ms_t/ms:.2f}x vs torch\"\nmatmul_epilogue()\n```\n\nWe move to a batched matmul and hang a full epilogue off it: an alpha scale supplied as a pass-by-value host scalar, a bias add, an activation, and an AMAX reduction over the result. The AMAX in the same kernel is the pattern that FP8 training relies on, since it collects the scale factor for the next quantization step without a second pass over the output. We compare against a PyTorch chain of baddbmm, activation, and amax, which makes clear that the speedup comes from eliminating epilogue memory traffic rather than from a faster GEMM.\n\n``` python\n@section(\"5. SDPA (Flash Attention) with causal masking\")\ndef sdpa_demo():\n   if not HAS_SDPA:\n       raise RuntimeError(f\"fused SDPA needs SM80+ (Ampere), this GPU is sm_{SM}\")\n   b, h, s, d = 4, 16, 1024, 64\n   scale = 1.0 / math.sqrt(d)\n   SDPA_FLOPS = 4 * b * h * s * s * d * 0.5\n   q = torch.randn(b, h, s, d, device=DEV, dtype=DTYPE)\n   k = torch.randn(b, h, s, d, device=DEV, dtype=DTYPE)\n   v = torch.randn(b, h, s, d, device=DEV, dtype=DTYPE)\n   o = torch.empty(b, h, s, d, device=DEV, dtype=DTYPE)\n   g = cudnn.pygraph(\n       handle=HANDLE, name=\"sdpa\",\n       io_data_type=TORCH2CUDNN[DTYPE],\n       intermediate_data_type=cudnn.data_type.FLOAT,\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   Q, Kt, V = tensor_of(g, q, \"Q\"), tensor_of(g, k, \"K\"), tensor_of(g, v, \"V\")\n   causal = True\n   try:\n       O, _stats = g.sdpa(name=\"sdpa\", q=Q, k=Kt, v=V,\n                          is_inference=True, attn_scale=scale, use_causal_mask=True)\n   except TypeError:\n       try:\n           O, _stats = g.sdpa(name=\"sdpa\", q=Q, k=Kt, v=V,\n                              is_inference=True, attn_scale=scale,\n                              diagonal_alignment=cudnn.diagonal_alignment.TOP_LEFT,\n                              right_bound=0)\n       except Exception:\n           causal = False\n           O, _stats = g.sdpa(name=\"sdpa\", q=Q, k=Kt, v=V,\n                              is_inference=True, attn_scale=scale)\n   print(f\"    causal masking: {causal}\")\n   O.set_output(True).set_data_type(TORCH2CUDNN[DTYPE])\n   O.set_dim(list(o.size())).set_stride(list(o.stride()))\n   build(g)\n   ws = workspace_for(g)\n   pack = {Q: q, Kt: k, V: v, O: o}\n   g.execute(pack, ws)\n   torch.cuda.synchronize()\n   ref = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=causal, scale=scale)\n   rel = ((o.float() - ref.float()).abs().max() / ref.float().abs().max()).item()\n   print(f\"    shape   : b{b} h{h} s{s} d{d}   workspace {ws.numel()/1024:.1f} KiB\")\n   print(f\"    rel err : {rel:.2e}\")\n   ms = bench(lambda: g.execute(pack, ws))\n   ms_t = bench(lambda: torch.nn.functional.scaled_dot_product_attention(\n       q, k, v, is_causal=causal, scale=scale))\n   print()\n   report(\"cuDNN FE SDPA\", ms, SDPA_FLOPS)\n   report(\"torch SDPA (backend's choice)\", ms_t, SDPA_FLOPS)\n   print(\"    Note: torch may already be dispatching to cuDNN or FlashAttention,\")\n   print(\"    so parity here is the expected, healthy outcome.\")\n   return f\"{ms:.3f} ms, {tflops(SDPA_FLOPS, ms):.1f} TFLOP/s\"\nsdpa_demo()\n@section(\"6. Serialize a built graph, reload it, execute by UID\")\ndef serialization():\n   Bsz, M, Kd, Nd = 8, 256, 512, 256\n   a = torch.randn(Bsz, M, Kd, device=DEV, dtype=DTYPE)\n   bm = torch.randn(Bsz, Kd, Nd, device=DEV, dtype=DTYPE)\n   out = torch.empty(Bsz, M, Nd, device=DEV, dtype=DTYPE)\n   UID_A, UID_B, UID_C = 1, 2, 3\n   g = cudnn.pygraph(\n       handle=HANDLE, name=\"serializable_mm\",\n       io_data_type=TORCH2CUDNN[DTYPE],\n       intermediate_data_type=cudnn.data_type.FLOAT,\n       compute_data_type=cudnn.data_type.FLOAT,\n   )\n   A = tensor_of(g, a, \"A\").set_uid(UID_A)\n   Bt = tensor_of(g, bm, \"B\").set_uid(UID_B)\n   C = g.matmul(A=A, B=Bt, compute_data_type=cudnn.data_type.FLOAT)\n   C.set_output(True).set_data_type(TORCH2CUDNN[DTYPE]).set_uid(UID_C)\n   t0 = time.perf_counter()\n   build(g)\n   cold_ms = (time.perf_counter() - t0) * 1e3\n   blob = g.serialize()\n   print(f\"    cold build      : {cold_ms:.1f} ms\")\n   print(f\"    serialized plan : {len(blob)} bytes (cache this to disk / ship it)\")\n   t0 = time.perf_counter()\n   g2 = cudnn.pygraph()\n   try:\n       g2.deserialize(HANDLE, blob)\n   except TypeError:\n       g2.deserialize(blob)\n   warm_ms = (time.perf_counter() - t0) * 1e3\n   print(f\"    deserialize     : {warm_ms:.1f} ms  -> {cold_ms/max(warm_ms,1e-6):.1f}x faster startup\")\n   ws = torch.empty(max(g2.get_workspace_size(), 1), device=DEV, dtype=torch.uint8)\n   g2.execute({UID_A: a, UID_B: bm, UID_C: out}, ws, handle=HANDLE)\n   torch.cuda.synchronize()\n   ref = torch.bmm(a.float(), bm.float())\n   rel = ((out.float() - ref).abs().max() / ref.abs().max()).item()\n   print(f\"    rel err after reload: {rel:.2e}\")\n   return f\"{len(blob)} B blob, reload {cold_ms/max(warm_ms,1e-6):.1f}x faster than rebuild\"\nserialization()\n```\n\nWe build a fused scaled dot-product attention graph with causal masking and check it against torch.nn.functional.scaled_dot_product_attention, guarding the whole section behind an SM80 check because the fused kernels need Ampere or newer. We write the causal argument with fallbacks, since the frontend has moved from use_causal_mask toward diagonal_alignment and bound arguments across its 1.x releases. We then serialize a built matmul graph to bytes, reload it into a fresh graph object, and execute it via integer UIDs, which lets us skip the compilation cost entirely at process startup.\n\n``` python\n@section(\"7. Dynamic shapes with a shared kernel cache\")\ndef dynamic_shapes():\n   kc = cudnn.create_kernel_cache()\n   def make(n):\n       x = torch.randn(n, 64, 32, 32, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n       w = torch.randn(64, 64, 3, 3, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n       y = torch.empty(n, 64, 32, 32, device=DEV, dtype=DTYPE).to(memory_format=torch.channels_last)\n       g = cudnn.pygraph(\n           handle=HANDLE, name=f\"dyn_{n}\",\n           io_data_type=TORCH2CUDNN[DTYPE],\n           intermediate_data_type=cudnn.data_type.FLOAT,\n           compute_data_type=cudnn.data_type.FLOAT,\n           kernel_cache=kc,\n           is_dynamic_shape_enabled=True,\n       )\n       X, Wt = tensor_of(g, x, \"X\"), tensor_of(g, w, \"W\")\n       Y = g.conv_fprop(image=X, weight=Wt, padding=[1, 1], stride=[1, 1],\n                        dilation=[1, 1], compute_data_type=cudnn.data_type.FLOAT)\n       Y.set_output(True).set_data_type(TORCH2CUDNN[DTYPE])\n       Y.set_dim(list(y.size())).set_stride(list(y.stride()))\n       t0 = time.perf_counter()\n       build(g)\n       ms = (time.perf_counter() - t0) * 1e3\n       ws = workspace_for(g)\n       g.execute({X: x, Wt: w, Y: y}, ws)\n       torch.cuda.synchronize()\n       return ms\n   times = [(n, make(n)) for n in (8, 16, 24, 32)]\n   for n, ms in times:\n       print(f\"      batch {n:>3d}: build {ms:7.1f} ms\")\n   first, rest = times[0][1], [m for _, m in times[1:]]\n   print(f\"\\n    first shape {first:.1f} ms, later shapes avg {sum(rest)/len(rest):.1f} ms\")\n   print(\"    The cache lets shape-variant graphs reuse an already-JIT'd kernel,\")\n   print(\"    which is what keeps variable batch/seqlen serving out of rebuild hell.\")\n   return f\"first {first:.0f} ms vs subsequent {sum(rest)/len(rest):.0f} ms\"\ndynamic_shapes()\n@section(\"8. CUDA Graph capture around a cuDNN execution plan\")\ndef cuda_graph_capture():\n   if not CONV_STATE:\n       raise RuntimeError(\"section 2 did not run, nothing to capture\")\n   g, pack, ws = CONV_STATE[\"graph\"], CONV_STATE[\"pack\"], CONV_STATE[\"ws\"]\n   eager_ms = bench(lambda: g.execute(pack, ws))\n   side = torch.cuda.Stream()\n   side.wait_stream(torch.cuda.current_stream())\n   with torch.cuda.stream(side):\n       cudnn.set_stream(handle=HANDLE, stream=side.cuda_stream)\n       for _ in range(3):\n           g.execute(pack, ws, handle=HANDLE)\n   torch.cuda.current_stream().wait_stream(side)\n   torch.cuda.synchronize()\n   cg = torch.cuda.CUDAGraph()\n   with torch.cuda.graph(cg):\n       cudnn.set_stream(handle=HANDLE, stream=torch.cuda.current_stream().cuda_stream)\n       g.execute(pack, ws, handle=HANDLE)\n   cudnn.set_stream(handle=HANDLE, stream=torch.cuda.current_stream().cuda_stream)\n   replay_ms = bench(lambda: cg.replay())\n   report(\"plain execute()\", eager_ms)\n   report(\"cuda graph replay()\", replay_ms)\n   print(f\"    launch overhead removed: {(eager_ms-replay_ms)*1e3:.1f} us/iter\")\n   print(\"    Pointers are frozen at capture time -- reuse the same buffers and\")\n   print(\"    copy new data into them, or re-capture.\")\n   return f\"{eager_ms:.3f} -> {replay_ms:.3f} ms via replay\"\ncuda_graph_capture()\nbanner(\"SUMMARY\")\nfor name, res in RESULTS.items():\n   print(f\"  {name:<58s} {res}\")\nprint(\"\"\"\nWhere to go next\n - samples/python in the repo: FP8/MXFP8 attention, paged KV cache, MoE grouped GEMM\n - python/cudnn/: the open-sourced CuTe DSL kernels (SDPA, grouped GEMM + SwiGLU,\n   block-sparse and native sparse attention) you can read and modify\n - debugging: CUDNN_FRONTEND_LOG_INFO=1 and CUDNN_FRONTEND_LOG_FILE=stdout\n   (use level 10 during CUDA graph capture -- level 1 dumps tensors and is not\n   capture-safe)\n\"\"\")\n```\n\nWe finish with two production concerns. First, we share a kernel cache across four graphs that differ only in batch size and time each build, so we can see later shapes reuse an already compiled kernel instead of paying the JIT cost again. Then we capture the convolution plan inside a CUDA graph, setting the cuDNN handle’s stream to the capture stream. So the work lands in the graph, and we measure how much per-iteration launch overhead the replay removes.\n\nIn conclusion, what we built here was small in code but broad in scope: a convolution, a matmul, and an attention kernel, each expressed as a graph rather than a library call. Working at that level changed what we could decide. We chose which operations collapsed into a single kernel, so the bias adds, activations, and AMAX reductions we folded into the epilogues never wrote an intermediate to memory. We chose the engine ourselves instead of accepting a heuristic, and timing every candidate config told us what that choice was worth. We also chose when to pay for compilation, pushing it out of the hot path with serialized plans, a kernel cache shared across shapes, and CUDA graph capture. The checks against PyTorch mattered as much as the timings, since the places where we merely matched it were usually places where PyTorch was already calling cuDNN underneath. That marked out where this API earns its keep: fusions with no framework-level equivalent, shapes hot enough to justify autotuning, and small kernels where startup and launch costs dominate.\n\nCheck out the **[FULL CODES here](https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Deep%20Learning/cudnn_frontend_nvidia_tutorial_Marktechpost.ipynb)**. All credit goes to the researcher of this project. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn", "canonical_source": "https://www.marktechpost.com/2026/09/15/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn-frontend/", "published_at": "2026-09-15 21:37:11+00:00", "updated_at": "2026-09-15 22:06:59.636217+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools", "machine-learning"], "entities": ["NVIDIA", "cuDNN", "cuDNN Frontend", "Marktechpost", "PyTorch", "Colab"], "alternates": {"html": "https://wpnews.pro/news/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn", "markdown": "https://wpnews.pro/news/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn.md", "text": "https://wpnews.pro/news/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn.txt", "jsonld": "https://wpnews.pro/news/inside-nvidias-cudnn-graph-api-fusion-autotuning-and-plan-reuse-with-cudnn.jsonld"}}