{"slug": "crunching-numbers-on-gpu-vs-cpu", "title": "Crunching numbers on GPU vs CPU", "summary": "A hands-on benchmark of generic matrix multiplication across CPU, Vulkan (wgpu) and CUDA backends found CUDA runs 7.82% faster than Vulkan, while Vulkan is 8% to 270% faster than CPU. The tester, who had access to only one CUDA machine, ran 3 warmup rounds and 20 runs per configuration and reported that an on-board AMD Radeon 860M (a 2025 model) is comparable to a 10-year-old GTX 1060, though the GTX 1060 handles sustained loads better on thermals. The tester concluded that CPU matmul can push CPU temperatures to 100 Celsius, making CPU unsuitable for LLM inference without liquid cooling except for short bursts.", "body_md": "CPU can crunch numbers, after all it's the \"brain\". Without a CPU you can't get a machine to work. But CPU isn't as efficient when it comes to matrix multiplication - this is when you offload matmul to a GPU.\n\nMatmul is used in a lot of applications - video games, machine learning, neural networks and AI. You can use CPU for this, but it'll be a lot slower.\n\nCuriosity got the better of me, so I want to know - given the same matmul, how fast would it run on various hardware.\n\nIt's a generic matmul with the same matrix size across all runs, with 3 warmup rounds and 20 runs.\n\nTested against CPU, Vulkan (wgpu) and CUDA (unfortunately I only have access to only one machine with CUDA, so there are no other comparisons).\n\nVulkan can be used on both GPU and on-board GPU.\n\nIt's obvious that a recent GPU has the best performance. RTX 3090 is 6 years old, but GTX 1060 is 10 years old, so the drastic performance makes sense.\n\nIt's interesting that there is a slight performance penalty if you use Vulkan backend instead of CUDA. If you want to maximize the performance throughput, you should invest in setting up CUDA on your machine, and they do add up if you have a lot of GPUs.\n\nFor CPU backend, there is not much differences, but with Vulkan the performance really improves. One caveat: sometimes CPU backend is more performant, but you'll have to factor in operating temperature as well. Crunching matmul on CPU can make CPU temperature reach 100 Celsius, which can damage your hardware if you leave it running non-stop.\n\nWhat's interesting is that an on-board AMD GPU (AMD Radeon 860M - a 2025 model) is comparable to GTX 1060, a GPU from 10 years ago. Bad news is with sustained loads you're better off with GTX 1060 since it has lower operating temperature. For burst workloads the on-board AMD GPU would do fine and would be faster.\n\nRyzen AI PRO 350 was released in 2025, and is faster than beefy Ryzen 3900X or Ryzen 3600, which were released in 2019.\n\nBut faster doesn't always mean better - a desktop CPU would be better for sustained loads, given better thermal management. And you can do nitrogen cooling.\n\nCPU is slower than GPU (either dedicated or on-board). Newer hardware is faster than older hardware, but go for dedicated GPU if you require sustained workloads for better thermal management.\nAlso CUDA is slightly faster than Vulkan (`+7.82%`), worth to invest in CUDA setup if you want to maximize performance throughput. Vulkan performance can be `+8%` up to `+270%` faster than CPU.\n\nSo for real, unless you have liquid cooling (or better), don't use CPU for LLM inference unless it's for a short inference here and there.", "url": "https://wpnews.pro/news/crunching-numbers-on-gpu-vs-cpu", "canonical_source": "https://karnwong.me/posts/2026/11/crunching-numbers-on-gpu-vs-cpu/", "published_at": "2026-11-06 00:00:00+00:00", "updated_at": "2026-11-06 11:19:41.333173+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "large-language-models"], "entities": ["Nvidia RTX 3090", "Nvidia GTX 1060", "AMD Radeon 860M", "Ryzen AI PRO 350", "Ryzen 3900X", "Ryzen 3600", "Vulkan", "CUDA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crunching-numbers-on-gpu-vs-cpu", "markdown": "https://wpnews.pro/news/crunching-numbers-on-gpu-vs-cpu.md", "text": "https://wpnews.pro/news/crunching-numbers-on-gpu-vs-cpu.txt", "jsonld": "https://wpnews.pro/news/crunching-numbers-on-gpu-vs-cpu.jsonld"}}