If you've shopped for a GPU for ML work, you've seen both numbers on the spec sheet — CUDA cores and Tensor cores — usually with zero explanation of what either actually does for you.
**
CUDA cores are general-purpose processing units. Think of them as thousands of small workers, each capable of doing a simple math operation, all working in parallel. This is what makes GPUs good at graphics in the first place — and it's also why they turned out to be great at deep learning, since neural networks are basically millions of small, repetitive math operations that can run at the same time.
More CUDA cores generally means more raw parallel processing power. They handle pretty much anything — traditional deep learning operations, general compute, graphics rendering.
Tensor cores are a different kind of core entirely — specialized hardware built specifically for one operation: matrix multiplication and accumulation, done in mixed precision. That sounds narrow, but matrix multiplication is the core operation in deep learning. Most of what happens during training and inference boils down to multiplying huge matrices together.
Because Tensor cores are purpose-built for exactly this, they do it dramatically faster than CUDA cores doing the same job. We're talking multiple times the throughput for the specific operations they're designed for.
Here's the practical part: if your GPU has Tensor cores, but your code doesn't use mixed precision training, you're leaving most of that speed on the table. Tensor cores need FP16 or BF16 (mixed precision) to actually kick in — running everything in standard FP32 mostly ignores them.
This is an easy win most people skip. Turning on mixed precision is often one flag:
from torch.cuda.amp import autocast
with autocast():
output = model(input)
loss = loss_fn(output, target)
Same model, same GPU, often noticeably faster training — just because you let the Tensor cores actually do their job.
For deep learning specifically, Tensor core count and generation usually matters more than raw CUDA core count. A GPU with fewer CUDA cores but modern Tensor cores will often beat an older GPU with more CUDA cores but no (or older) Tensor cores, for training and inference workloads.
For anything outside deep learning — general compute, rendering, tasks that aren't matrix-multiplication-heavy — CUDA core count is still the more relevant number.
**
CUDA cores: general-purpose, handle anything.
Tensor cores: specialized, dramatically faster at the one operation deep learning cares about most — but only if your code actually uses mixed precision to trigger them.
If you're training and not using mixed precision yet, that's probably the single easiest performance win sitting on the table right now.
..........................
_I like helping engineering teams eliminate cloud bill surprises, fix database bottlenecks, and scale dedicated bare metal and GPU infrastructure alongside Racko_