CPU can crunch numbers, after all it's the "brain". Without a CPU you can't get a machine to work. But CPU isn't as efficient when it comes to matrix multiplication - this is when you offload matmul to a GPU.
Matmul is used in a lot of applications - video games, machine learning, neural networks and AI. You can use CPU for this, but it'll be a lot slower.
Curiosity got the better of me, so I want to know - given the same matmul, how fast would it run on various hardware.
It's a generic matmul with the same matrix size across all runs, with 3 warmup rounds and 20 runs.
Tested against CPU, Vulkan (wgpu) and CUDA (unfortunately I only have access to only one machine with CUDA, so there are no other comparisons).
Vulkan can be used on both GPU and on-board GPU.
It's obvious that a recent GPU has the best performance. RTX 3090 is 6 years old, but GTX 1060 is 10 years old, so the drastic performance makes sense.
It's interesting that there is a slight performance penalty if you use Vulkan backend instead of CUDA. If you want to maximize the performance throughput, you should invest in setting up CUDA on your machine, and they do add up if you have a lot of GPUs.
For CPU backend, there is not much differences, but with Vulkan the performance really improves. One caveat: sometimes CPU backend is more performant, but you'll have to factor in operating temperature as well. Crunching matmul on CPU can make CPU temperature reach 100 Celsius, which can damage your hardware if you leave it running non-stop. What's interesting is that an on-board AMD GPU (AMD Radeon 860M - a 2025 model) is comparable to GTX 1060, a GPU from 10 years ago. Bad news is with sustained loads you're better off with GTX 1060 since it has lower operating temperature. For burst workloads the on-board AMD GPU would do fine and would be faster.
Ryzen AI PRO 350 was released in 2025, and is faster than beefy Ryzen 3900X or Ryzen 3600, which were released in 2019.
But faster doesn't always mean better - a desktop CPU would be better for sustained loads, given better thermal management. And you can do nitrogen cooling.
CPU is slower than GPU (either dedicated or on-board). Newer hardware is faster than older hardware, but go for dedicated GPU if you require sustained workloads for better thermal management.
Also CUDA is slightly faster than Vulkan (+7.82%), worth to invest in CUDA setup if you want to maximize performance throughput. Vulkan performance can be +8% up to +270% faster than CPU.
So for real, unless you have liquid cooling (or better), don't use CPU for LLM inference unless it's for a short inference here and there.