CUDA Cores vs Tensor Cores Explained

A developer explains the difference between CUDA cores and Tensor cores for machine learning workloads, noting that Tensor cores are specialized for matrix multiplication and accumulation in mixed precision and can deliver multiple times the throughput of CUDA cores for deep learning operations. The writeup advises that Tensor cores only activate with FP16 or BF16 mixed precision, and that enabling it often requires just a single autocast flag in PyTorch. For deep learning, Tensor core count and generation typically matter more than raw CUDA core count, while CUDA cores remain the more relevant metric for general compute and rendering.

If you've shopped for a GPU for ML work, you've seen both numbers on the spec sheet — CUDA cores and Tensor cores — usually with zero explanation of what either actually does for you. CUDA cores are general-purpose processing units. Think of them as thousands of small workers, each capable of doing a simple math operation, all working in parallel. This is what makes GPUs good at graphics in the first place — and it's also why they turned out to be great at deep learning, since neural networks are basically millions of small, repetitive math operations that can run at the same time. More CUDA cores generally means more raw parallel processing power. They handle pretty much anything — traditional deep learning operations, general compute, graphics rendering. Tensor cores are a different kind of core entirely — specialized hardware built specifically for one operation: matrix multiplication and accumulation , done in mixed precision. That sounds narrow, but matrix multiplication is the core operation in deep learning. Most of what happens during training and inference boils down to multiplying huge matrices together. Because Tensor cores are purpose-built for exactly this, they do it dramatically faster than CUDA cores doing the same job. We're talking multiple times the throughput for the specific operations they're designed for. Here's the practical part: if your GPU has Tensor cores, but your code doesn't use mixed precision training, you're leaving most of that speed on the table. Tensor cores need FP16 or BF16 mixed precision to actually kick in — running everything in standard FP32 mostly ignores them. This is an easy win most people skip. Turning on mixed precision is often one flag: python from torch.cuda.amp import autocast with autocast : output = model input loss = loss fn output, target Same model, same GPU, often noticeably faster training — just because you let the Tensor cores actually do their job. For deep learning specifically, Tensor core count and generation usually matters more than raw CUDA core count. A GPU with fewer CUDA cores but modern Tensor cores will often beat an older GPU with more CUDA cores but no or older Tensor cores, for training and inference workloads. For anything outside deep learning — general compute, rendering, tasks that aren't matrix-multiplication-heavy — CUDA core count is still the more relevant number. CUDA cores: general-purpose, handle anything. Tensor cores: specialized, dramatically faster at the one operation deep learning cares about most — but only if your code actually uses mixed precision to trigger them. If you're training and not using mixed precision yet, that's probably the single easiest performance win sitting on the table right now. .......................... I like helping engineering teams eliminate cloud bill surprises, fix database bottlenecks, and scale dedicated bare metal and GPU infrastructure alongside Racko https://racko.ai/