{"slug": "use-fewer-threads-for-cpu-inference", "title": "Use fewer threads for CPU inference", "summary": "A benchmark of Gemma4-E4B on an AMD Ryzen 7040 CPU shows that using 8 threads instead of 16 improves prompt processing from 94.66 to 105.94 tokens per second and token generation from 11.64 to 16.15 tokens per second, indicating that matching physical core count is optimal for CPU inference. On an M4 Pro Mac Mini, using 8 threads instead of 12 boosts prompt processing from 179.28 to 199.24 t/s and token generation from 36.36 to 52.39 t/s, and when a background process occupies a fast core, 7 threads outperform 8. The article recommends using libraries like num_cpus to count physical cores and, for heterogeneous CPUs, counting fast physical cores as done in NobodyWho.", "body_md": "# Use fewer threads for CPU inference\n\nHow many threads should you use for CPU inference?\n\n## The naive answer\n\nUse something like Rust's `std::thread::available_parallelism()`\n\nto count the number of cores on your machine, spawn that many threads, and call it a day.\n\nThis is used to great effect by `rayon`\n\nand the like, however for non-work-stealing compute-bound tasks, such as those spawned by `llama.cpp`\n\n, this often results in a too-high number of threads.\n\n## Hyperthreading\n\nModern x86 CPUs support [hyper-threading](https://en.wikipedia.org/wiki/Hyper-threading), which tries to run several threads on the same core. Those threads get their own registers and program counters, but still share most of the compute resources.\n\nIf your program is reasonably efficient, hyperthreading means several threads end up competing for the same execution resources on a single physical core, so threads end up waiting.\n\nHere's a inference benchmark of Gemma4-E4B on my Ryzen 7040 CPU, which has 8 phyiscal cores and 16 logical cores. It's much faster to only run as many threads are there are physical cores.\n\n| threads | test | t/s |\n|---|---|---|\n| 16 | pp512 | 94.66 ± 3.25 |\n| 16 | tg128 | 11.64 ± 0.75 |\n| 8 | pp512 | 105.94 ± 5.10 |\n| 8 | tg128 | 16.15 ± 0.07 |\n\nSome programs, like [llama-cpp-python](https://github.com/abetlen/llama-cpp-python/blob/3691546f1c9e0c1bf93323dff02230bd959cf562/llama_cpp/llama.py#L312), assume they're running on a machine with two hyperthreaded logical cores per physical core, and run exactly half as many threads as there are logical cores. For many desktop x86 CPUs, this works great.\n\n## Counting physical cores\n\n**However,** not all CPUs support hyperthreading, and not all hyperthreading CPUs have exactly two logical cores per physical core. If you just halve the number of logical cores, you don't necessarily get the number of physical cores.\n\nInstead, you can fetch a package like [num_cpus](https://docs.rs/num_cpus/latest/num_cpus/), which distinguishes [the number of actual physical cores](https://docs.rs/num_cpus/1.17.0/num_cpus/fn.get_physical.html), and run that number of threads.\n\nThis lets you run as many threads as there are physical cores, even on CPUs without a standard hyperthreading setup.\n\n## Heterogeneous CPUs\n\n**However**, for mixed-core CPUs, like those in MacBooks or smartphones, running on every physical core is bad. These machines have some fast cores and some slower, power-efficient cores. You might expect running both to yield more compute than running the fast ones alone, but in practice, for workloads like inference, the fast cores finish quickly and then wait on the slow cores.\n\nHere's a inference benchmark of Gemma4-E4B on an M4 Pro Mac Mini, which has 8 fast cores, and 4 slow cores (Metal disabled to show CPU inference). It's much faster to only run on the performance cores.\n\n| threads | test | t/s |\n|---|---|---|\n| 12 | pp512 | 179.28 ± 3.15 |\n| 12 | tg128 | 36.36 ± 4.25 |\n| 8 | pp512 | 199.24 ± 3.49 |\n| 8 | tg128 | 52.39 ± 0.16 |\n\n[In NobodyWho](https://github.com/nobodywho-ooo/nobodywho/blob/030632a07d4cda4097e3e6c5d0e81624b429ebe5/nobodywho/core/src/cpu.rs#L115), we count how many fast physical cores are on the system, and run that many threads. This is how we automatically get optimal performance for inference on heterogeneous CPUs too.\n\n## Competing with other processes\n\n**However**, your computer might already have other compute-heavy work running on the fast cores. In that case, your inference thread occasionally gets bumped to a slower core, and you're back to the fast thread waiting on the slow one.\n\nIt turns out that if one of your fast cores is occupied, it's faster to run one fewer thread than to fight for that fast core and end up on a slow one.\n\nHere's an inference benchmark of Gemma4-E4B on that same Mac Mini, but with a simple `python -c \"while True: pass\"`\n\nrunning in the background. Notice how 7 threads is faster than 8.\n\n| threads | test | t/s |\n|---|---|---|\n| 8 | pp512 | 137.12 ± 0.87 |\n| 8 | tg128 | 44.28 ± 0.70 |\n| 7 | pp512 | 148.83 ± 1.02 |\n| 7 | tg128 | 45.60 ± 0.25 |\n\nBut how do you detect how occupied the fast cores are? As you can see, we haven't solved the problem fully yet in NobodyWho, please do let us know if you know of a better approach!\n\n## Manual control\n\nWe support the [n_threads](https://docs.nobodywho.ooo/python/chat/#cpu-threads) argument when instantiating a new LLM Chat on CPU. It defaults to the performance-core-counting logic described above.\n\nYou could sweep across a range of thread counts with some workload and measure which is fastest, but that's too heavy to do on every startup for my taste.\n\nThe new defaults in NobodyWho are pretty sane. They're released and available for all 7 language bindings.\n\nEverything NobodyWho do is open-source, please leave a\n[star on Github](https://github.com/nobodywho-ooo/nobodywho)\nto support us ❤️\n\nPublished Aug 20, 2026", "url": "https://wpnews.pro/news/use-fewer-threads-for-cpu-inference", "canonical_source": "https://www.nobodywho.ai/posts/threadcounts-for-cpu-inference/", "published_at": "2026-08-20 00:00:00+00:00", "updated_at": "2026-08-24 10:44:12.537623+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure"], "entities": ["Gemma4-E4B", "AMD Ryzen 7040", "M4 Pro Mac Mini", "llama.cpp", "llama-cpp-python", "num_cpus", "NobodyWho", "Rust"], "alternates": {"html": "https://wpnews.pro/news/use-fewer-threads-for-cpu-inference", "markdown": "https://wpnews.pro/news/use-fewer-threads-for-cpu-inference.md", "text": "https://wpnews.pro/news/use-fewer-threads-for-cpu-inference.txt", "jsonld": "https://wpnews.pro/news/use-fewer-threads-for-cpu-inference.jsonld"}}