# Why Mini PCs Run 70B Models That Discrete GPUs Can't > Source: > Published: 2026-07-10 17:04:04+00:00 [AI](https://sourcefeed.dev/c/ai)Article # Why Mini PCs Run 70B Models That Discrete GPUs Can't Unified memory architectures solve the local LLM capacity bottleneck on a budget, but developers must accept a steep bandwidth penalty. [Priya Nair](https://sourcefeed.dev/u/priya_nair) Put two machines on a desk, each costing about $2,000. One is a tower packing an NVIDIA RTX 5090 with 32GB of GDDR7 memory running at a blistering 1,792 GB/s. The other is a mini PC the size of a paperback, powered by an AMD Ryzen AI Max+ 395 "Strix Halo" chip with 128GB of soldered LPDDR5X memory running at roughly 256 GB/s. Now ask both machines to run a 70-billion-parameter language model. The RTX 5090 cannot. A 70B model at a sensible 4-bit quantization requires roughly 40GB of memory, and 40 will not fit into 32. The mini PC, however, loads the model without complaint and begins generating text. This paradox defines the current state of local AI hardware. By adopting unified memory architectures, cheap mini PCs and compact desktops can host massive models that choke top-tier consumer graphics cards. But this capacity comes with a catch. While these small boxes can hold models that previously required a server rack, they pay for that capacity in raw generation speed. ## The Architecture of the Shared Pool In a traditional desktop PC, memory is split into two isolated pools. The CPU uses system RAM, while the discrete GPU relies on its own high-speed VRAM. If the GPU needs to process data stored in system RAM, that data must be copied across the physical PCIe bus. Because a language model must fit entirely within GPU-addressable memory to run at full speed, the VRAM capacity of your graphics card acts as a hard ceiling. If you have a 24GB card, you cannot run a model that requires 25GB, no matter how many hundreds of gigabytes of system RAM you install. Unified memory architectures throw out this split. Instead of separate pools, the CPU, the integrated GPU, and the Neural Processing Unit (NPU) share a single pool of high-speed, soldered memory. There is no separate VRAM, and there is no need to copy data across a PCIe bus. The processors work over the exact same memory space. This is not the slow, shared system memory of older integrated graphics. By soldering LPDDR5X memory directly to the chip package and widening the memory bus, chipmakers have achieved bandwidth levels that make local inference viable. Apple has used this approach for years, and now AMD, Intel, Qualcomm, and NVIDIA are following suit. | Machine (SoC) | Memory Bus | Peak Bandwidth | Max Unified Memory | |---|---|---|---| | Intel "Lunar Lake" (Core Ultra 200V) | 128-bit | ~137 GB/s | 32 GB | | Intel "Panther Lake" (Core Ultra X) | 128-bit | ~154 GB/s | 96 GB | | Qualcomm Snapdragon X2 Elite | 128-bit | ~152 GB/s | up to 128 GB | | Qualcomm Snapdragon X2 Elite Extreme | 192-bit | ~228 GB/s | up to 128 GB | | AMD Ryzen AI Max+ 395 "Strix Halo" | 256-bit | 256 GB/s | 128 GB | | NVIDIA DGX Spark (GB10) | 256-bit | 273 GB/s | 128 GB | | Apple M4 (Mac mini) | 128-bit | 120 GB/s | 32 GB | | Apple M4 Pro (Mac mini) | 256-bit | 273 GB/s | 64 GB | | Apple M4 Max (Mac Studio) | 384 to 512-bit | 410 to 546 GB/s | 128 GB | | Apple M3 Ultra (Mac Studio) | 1024-bit | 819 GB/s | up to 512 GB | For Contrast: Discrete GPUs | ||| | NVIDIA RTX 3090 / 4090 | 384-bit | 936 / 1,008 GB/s | 24 GB | | NVIDIA RTX 5090 | 512-bit | 1,792 GB/s | 32 GB | | NVIDIA RTX PRO 6000 Blackwell | 512-bit | 1,792 GB/s | 96 GB (~$8,500) | Looking at this data, the trade-off is clear. Discrete GPUs offer massive bandwidth (900 to 1,800 GB/s) but are severely limited in capacity. Unified memory systems offer massive capacity (up to 128GB or even 512GB) but are limited to bandwidths between 120 and 546 GB/s. ## The Brutal Math of the Roofline Model To understand why this split matters, we have to look at the roofline model, which is the standard framework for analyzing chip performance. A workload is either compute-bound (limited by how fast the chip can perform arithmetic) or memory-bound (limited by how fast the chip can read data from memory). Text generation is almost entirely memory-bound. To output a single token, the system must read the entire model's weights out of memory once, perform a relatively small amount of math, and then repeat the process for the next token. As Apple's machine learning team noted in their documentation for the [Apple MLX](https://github.com/ml-explore/mlx) framework, token generation is bounded by memory bandwidth, not compute ability. This gives us a simple formula to calculate the theoretical speed limit of any machine: $$\text{Decode Tokens per Second} \approx \frac{\text{Memory Bandwidth}}{\text{Bytes Read per Token}}$$ If we run a dense 70B model at 4-bit quantization, the system must read roughly 40GB of weights per token. - On an AMD Strix Halo box with 256 GB/s of bandwidth, the absolute theoretical ceiling is $256 \div 40 = 6.4$ tokens per second. - On an Apple M3 Ultra with 819 GB/s of bandwidth, the ceiling rises to $819 \div 40 = 20.4$ tokens per second. - On an RTX 5090 with 1,792 GB/s of bandwidth, the ceiling would be $1,792 \div 40 = 44.8$ tokens per second, but the model cannot load because the card only has 32GB of VRAM. In practice, real-world performance is slightly lower than these theoretical peaks due to system overhead. However, highly optimized runtimes like [llama.cpp](https://github.com/ggerganov/llama.cpp) can squeeze impressive efficiency out of these chips. For example, testing on a GMKtec EVO-X2 mini PC (Ryzen AI Max+ 395 with 128GB of unified memory) shows Llama 3.3 70B running at a steady 11.8 tokens per second. That is fast enough to read comfortably, but it is a far cry from the instantaneous responses of high-bandwidth enterprise GPUs. ## The MoE Loophole There is one major exception to this bandwidth bottleneck: Mixture-of-Experts (MoE) models. An MoE model like Qwen3-30B-A3B might have 30 billion total parameters, but it only activates a fraction of them (say, 3 billion) per token. This means the hardware only needs to read about 2GB of weights from memory per token instead of the full 40GB required by a dense 70B model. Because the memory read requirement drops so drastically, the bandwidth bottleneck is bypassed. On the same Strix Halo mini PC where a dense 70B model crawls along at single-digit speeds, developers measure MoE models running at upwards of 72 tokens per second. If you want to run large models on a unified memory machine without the slow generation speeds, MoEs are the obvious path. ## The Developer's Playbook: Sandbox vs. Production For developers, unified memory mini PCs are not a replacement for cloud-based GPUs or local multi-GPU workstations. Instead, they represent a highly cost-effective local sandbox. If you are building applications that rely on large models, a unified memory machine allows you to test prompt engineering, evaluate model outputs, and run local integration tests without racking up massive cloud API bills. You can run a local [Ollama](https://ollama.com) instance in the background, serving a 70B model, while using your machine for daily development tasks like editing code, compiling binaries, or even editing assets in [Darktable](https://www.darktable.org). However, you must design your workflows around two major hardware caveats: ### 1. The Prefill Penalty While token generation (decode) is memory-bandwidth bound, processing your input prompt (prefill) is highly compute-bound. Discrete GPUs with thousands of CUDA cores excel at this. A mini PC or Mac will take noticeably longer to process a massive system prompt or a long document before it begins generating the first token. If your application relies on sending huge contexts to the model with every request, the prefill latency on a mini PC may become frustrating. ### 2. Software Ecosystems NVIDIA's CUDA remains the industry standard for AI development. While tools like llama.cpp and Ollama have made running models on Apple Silicon (via Metal) and AMD (via ROCm) incredibly simple, you will still run into friction if you are writing custom PyTorch code or trying to run niche research repositories that assume an NVIDIA GPU is present. If you are choosing a machine for local AI development, use this general memory sizing guide: **16GB Unified Memory:** Best for running 7B to 8B models (like Llama 3.2 8B) at high speeds (20 to 30 tokens per second). Ideal for basic coding assistants and local summarization.**24GB to 32GB Unified Memory:** Allows you to run 14B models comfortably, or 32B models with aggressive quantization.**48GB to 64GB Unified Memory:** The sweet spot for serious local development. This tier runs 32B models at full precision and can fit a 4-bit quantized 70B model.**128GB+ Unified Memory:** The high-end tier. This allows you to run 70B models at full precision, or experiment with massive 405B models at high quantization levels. Unified memory machines have changed the economics of local AI. They do not make models run faster, but they make them fit. For developers who need to test large models locally without spending five figures on enterprise hardware, that is a trade-off worth making. ## Sources & further reading - [Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't](https://vettedconsumer.com/unified-memory-explained-why-mini-pcs-can-run-70b-models-a-big-gpu-cant-and-where-they-slow-down/)— vettedconsumer.com - [What Is Unified Memory? Why a Mini PC Runs a 235B AI Model · Cloudzy Blog](https://cloudzy.com/blog/what-is-unified-memory/)— cloudzy.com - [The $3,000 AI Lab on Your Desk: I Tested the Mini PCs That Actually Run 70B Models Locally - Marcus Reed | Tech Reviews & AI Hardware](https://tech.omnux.com/2026/05/06/the-3000-ai-lab-on-your-desk-i-tested-the-mini-pcs-that-actually-run-70b-models-locally/)— tech.omnux.com - [Unified Memory Explained: Apple Silicon vs NVIDIA for AI](https://seresa.io/blog/ai-data-readiness/what-is-unified-memory)— seresa.io - [You don't need an expensive GPU to run a local LLM that actually works](https://www.xda-developers.com/dont-need-expensive-gpu-to-run-local-llm-actually-works/)— xda-developers.com [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 0 No comments yet Be the first to weigh in.