{"slug": "no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error", "title": "“no kernel image is available for execution on the device” What This PyTorch Error Means On RunPod Blackwell GPUs", "summary": "A developer reproduced a CUDA 'no kernel image is available for execution on the device' failure on a RunPod RTX PRO 4000 Blackwell pod running the default PyTorch 2.4.1+cu124 template, which ships kernels only for sm_50 through sm_90 while Blackwell cards report sm_120 (and the B200 reports sm_100). Because torch.cuda.is_available() returns True and the GPU name prints normally, the incompatibility only surfaces at the first real CUDA kernel call, after the pod is billed; the reproduction led to a GPU architecture check added in ComputeFence v0.3.2.", "body_md": "You launch a training job on RunPod. PyTorch detects your GPU, torch.cuda.is_available() returns True, and your environment checks appear to pass.\n\nThen your script reaches its first real CUDA operation and crashes:\n\nRuntimeError: CUDA error: no kernel image is available for execution on the device\n\nIf you are running on a Blackwell GPU with an older PyTorch build, you have hit a GPU architecture compatibility problem. The frustrating part is that your environment looks perfectly healthy right up until the first CUDA kernel executes.\n\n**What causes this error**\n\nCUDA kernels are compiled for specific GPU architectures, identified by compute capability for example sm_86 for an A40, sm_90 for an H100.\n\nWhen you install PyTorch, the build you download contains kernels compiled for a fixed range of architectures. PyTorch 2.4.1+cu124 supports sm_50 through sm_90.\n\nBlackwell GPUs: the RTX PRO 4000, RTX PRO 4500, RTX PRO 6000, and RTX 5090 have compute capability sm_120. They are outside that range.\n\nWith an incompatible PyTorch build, PyTorch does not fail immediately. torch.cuda.is_available() returns True. The GPU name appears in the output. Everything looks fine. Then the training script reaches the first real CUDA kernel call the first matrix multiplication, the first forward pass and it throws the error above.\n\nBy that point the pod is already allocated, your dependencies are installed, and billing has started.\n\n**Note on the B200**: The B200 belongs to the sm_100 architecture family, not sm_120. Always check the actual compute capability reported by your GPU rather than assuming every Blackwell card uses the same architecture.\n\nYou can check this in two lines:\n\nimport torch\n\ncap = torch.cuda.get_device_capability(0)\n\nprint(f\"Device capability: sm_{cap[0]}{cap[1]}\")\n\nprint(f\"Supported architectures: {torch.cuda.get_arch_list()}\")\n\nIf your device capability is not in the supported list, the run will fail when the first kernel executes.\n\n**Why torch.cuda.is_available() returning True is not enough**\n\nThis is where the problem becomes deceptive.\n\ntorch.cuda.is_available() confirms that PyTorch can access CUDA and identify a device. It does not confirm that the installed build contains compatible kernels for every operation you intend to run. Architecture compatibility is a separate check.\n\nGPU detection and GPU compatibility are two different things.\n\n**First-hand reproduction on RunPod**\n\nI reproduced this on a RunPod RTX PRO 4000 Blackwell pod using the default PyTorch 2.4.1+cu124 template.\n\nThe terminal output looked clean:\n\n✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA RTX PRO 4000 Blackwell)\n\nMeanwhile PyTorch was printing a warning to stderr that sm_120 was not supported by this build. The pre-flight check passed. Training would have failed on the first real kernel call.\n\nThe warning that appeared in stderr:\n\nUserWarning: CUDA device capability sm_120 is not supported by this PyTorch build.\n\nSupported capabilities: sm_50 through sm_90.\n\nIf your training script runs in a tmux session, pipes output to a log file, or uses a framework like Axolotl or Unsloth, you will miss this warning entirely until the forward pass crashes.\n\nThe pod also reported /workspace - 466126.9 GB free of 2190892.5 GB unusually large volume figures that appear to be a virtualisation reporting artefact on early Blackwell deployments rather than real capacity.\n\nThis reproduction exposed a gap in the original pre-flight checks and led directly to the GPU architecture check added in ComputeFence v0.3.2.\n\n**Which GPUs are affected with standard PyTorch images**\n\nAs of PyTorch 2.4.x, these cards are not supported by the builds shipped in standard RunPod PyTorch templates:\n\nNVIDIA RTX PRO 4000 Blackwell: sm_120\n\nNVIDIA RTX PRO 4500 Blackwell: sm_120\n\nNVIDIA RTX PRO 6000 Blackwell: sm_120\n\nNVIDIA RTX 5090: sm_120\n\nNVIDIA B200: sm_100 (different architecture, same class of problem)\n\nIf RunPod assigns one of these GPUs and you are on the default template, you will hit this error on the first kernel call.\n\n**How ComputeFence catches it**\n\nSince v0.3.2, ComputeFence compares each GPU’s compute capability against the architectures supported by the installed PyTorch build using get_device_capability() and get_arch_list(). A mismatch is flagged as a blocker not a warning:\n\n✗ BLOCKER: GPU architecture mismatch\n\n  NVIDIA RTX PRO 4000 Blackwell has compute capability sm_120\n\n  This PyTorch build supports sm_50 through sm_90\n\n  Training will fail on the first CUDA kernel call\n\n  Fix: Use a PyTorch build compiled for sm_120, or select a different GPU\n\nThis runs before billing starts.\n\nuvx computefence doctor\n\nNote: get_arch_list() reports architectures compiled into the PyTorch installation. It is a useful compatibility signal but does not guarantee that every third-party CUDA extension will work. Run a small representative CUDA operation before committing to a long training run on new hardware.\n\n**How to fix it**\n\n**Option 1: Install a PyTorch build that supports your GPU**\n\nUse a release compiled for sm_120. Check pytorch.org/get-started/locally for current compatibility. After installing, verify with the two-line check above before launching your job.\n\n**Option 2: Select a different GPU**\n\nIf you need to use an existing PyTorch image with pinned dependencies, select a GPU whose architecture is in the supported range. An A40 (sm_86) or H100 (sm_90) will work with PyTorch 2.4.x.\n\n**The broader lesson**\n\nGPU detection is not GPU compatibility. As providers add newer hardware generations, the gap between the latest cards and the PyTorch builds in standard templates will keep appearing. Passing torch.cuda.is_available() is not proof that your training workload can execute.\n\nCheck the architecture before billing starts.\n\nGitHub: github.com/Francisco-Booth/ComputeFence\n\nInstall: pip install computefence", "url": "https://wpnews.pro/news/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error", "canonical_source": "https://dev.to/franciscobooth/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error-means-on-runpod-4528", "published_at": "2026-11-09 13:12:16+00:00", "updated_at": "2026-11-09 13:24:55.221937+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "developer-tools"], "entities": ["RunPod", "PyTorch", "NVIDIA", "RTX PRO 4000 Blackwell", "RTX 5090", "B200", "ComputeFence"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error", "markdown": "https://wpnews.pro/news/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error.md", "text": "https://wpnews.pro/news/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error.txt", "jsonld": "https://wpnews.pro/news/no-kernel-image-is-available-for-execution-on-the-device-what-this-pytorch-error.jsonld"}}