The Memory Wall Didn't Fall. Someone Just Engineered Around It.

Two open-weight projects show that running large models locally is now a data-path scheduling problem rather than a hardware limit: Edge0-145B, a 145-billion-parameter mixture-of-experts model trained from the start for on-device use, reportedly runs on a MacBook Pro at 29GB peak memory and 80+ tokens/s decode, while the smolmaxxing project runs a 16B MoE at 5.1 tokens/s on a TV box with about 1GB of usable RAM across 160 catalogued experiments. Huawei also open-sourced MACaQ, a calibrated quantizer that produces a 22.36GB GGUF of a 35B MoE with a KL divergence to FP16 of 0.0134 versus 0.0157 for a community quant at the same file size.

For two years the sensible position was that serious models live in someone else's data center and your machine runs the client. That position quietly stopped being true this week. 1. A 145B Model on a Laptop and a 16B Model on a TV Box Are the Same Result Edge0-145B https://edge0.ai/blog/introducing-edge0-145b-frontier-ai-capabilities-on-your-own-computer is a 145-billion-parameter open-weight model that its authors report running on a MacBook Pro at 29GB peak memory, 1,100+ tokens/s prefill and 80+ tokens/s decode, at around 120W. Nothing about that comes from a bigger chip. Each group of 64 experts shares two large matrices instead of carrying its own, so only about 10B parameters participate per step. Loading is pipelined across three tiers — hot experts in GPU memory, a 40GB expert library in system RAM, 48GB of vocabulary data read from SSD on demand — while the CPU runs ahead on scheduling. And it was trained in that on-device architecture from the start, not compressed into it afterward. At the opposite end of the price range, smolmaxxing https://github.com/hyperionaidev/smolmaxxing/ runs a 16B mixture-of-experts model at 5.1 tokens/s on a Xiaomi TV box with roughly 1GB of usable RAM, and a 1.7B model at 24-26 tokens/s on a jailbroken 2013 iPad Air. The wins are catalogued across 160 numbered experiments, and almost none of them are about compute: an expert pager streaming from flash with cache-aware routing and prefetch, cutting expert misses from 13% to about 2%; a pruned LM head; a headless mode that freed RAM. One model went from 1.87 to 7.2 tokens/s without a single new kernel. Read together, these projects say the memory wall was never physics. It was an unoptimized data path: the constraint is bytes moved per generated token, and that is a scheduling problem — a discipline the industry has deep benches in and has been pointing at GPU clusters instead. Why it matters: - For ICs: Your local-model ceiling is probably a configuration artifact. Before concluding a model doesn't fit, check what the weights are doing between RAM, VRAM and disk — that is where the factor-of-four lives. - For leaders: Inference projections built on per-token API pricing now have a credible alternative line. Run that comparison before signing a multi-year commit. - For founders: "Runs entirely on the customer's hardware" went from a feature you apologize for to one you can charge for, especially in regulated accounts. - All of it depends on labs continuing to publish weights — and the main distribution channel is being acquired by a chipmaker https://techcrunch.com/2026/11/12/hugging-face-ceo-vows-to-keep-fighting-for-open-source-ai-after-jensen-huangs-exit/ that just lost the CEO who championed open models. Worth mirroring what you depend on. 2. Quantization Stopped Being a Download and Became a Discipline Huawei open-sourced MACaQ https://github.com/huawei-csl/MACaQ , the calibrated quantizer it uses internally, and the interesting number isn't the compression ratio: it produces a 22.36GB GGUF of a 35B mixture-of-experts model with a KL divergence to FP16 of 0.0134, against 0.0157 for a widely used community quant at exactly the same file size . Same bytes on disk, measurably different model — and it covers image, video and audio models, not just text. The other end of the spectrum is instructive too. A single developer's MAPLE build https://huggingface.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF of DeepSeek-Coder-V2-Lite profiles token activations and keeps the top 54 of 64 routed experts per layer, landing at 4.50GB so model and KV cache fit inside a 6GB laptop GPU. Generation goes from 4.5-10 tokens/s under PCIe offload to 65-76 — a 7-15x speedup bought entirely by not crossing the bus. It is genuinely clever work. It is also worth reading the showcase output in its own README, where the model's answer to a lock-free ring buffer prompt opens with an unrelated atomic-counter class before reaching the real implementation. Both are the same lesson from opposite directions. The quant you pick is a model-quality decision, not a packaging step, and almost nobody measures it. File size and tokens/s are easy to publish; KL divergence against the full-precision model is the number that tells you what you gave up. Why it matters: - For ICs: When you swap quants, run your own eval set before and after. Same size and same speed does not mean same model, and the gap shows up in exactly the long-horizon agent tasks you care about. - For leaders: If local models are entering your stack, someone owns quant quality as a measured, versioned artifact — or a quiet file swap becomes an unexplained regression. - For founders: Compression expertise is a defensible moat in a way training no longer is for most companies. The weights are free; making them fit well is not. 3. The Browser and the OS Are Turning Into the Local Runtime The plumbing is catching up fast. PyTorch's ExecuTorch WebGPU backend https://pytorch.org/blog/executorch-webgpu-pytorch-models-on-the-gpu-in-the-browser/ lets you export once with torch.export and run the same artifact in any WebGPU browser — 4-7x faster than the WebAssembly CPU path on prefill, 2-10x on decode, with no conversion to GGUF, ONNX or TFLite. It covers LLMs, vision-language, vision and speech models, and supports on-device LoRA fine-tuning. Deployment is a file on a CDN, and nothing leaves the tab. Apple's Foundation Models framework https://developer.apple.com/foundation-models-framework/ makes the same bet from the OS side, with a twist: one Swift API now routes to the on-device model, to Private Cloud Compute, or to server models including Claude and Gemini, and Small Business Program developers get Private Cloud Compute inference at no cloud API cost. There is also an Evaluations framework for testing prompts across model swaps — a tacit admission that if routing is a config value, you need regression tests on behavior, not just on code. That is the actual shift. Local versus cloud is becoming a per-call routing decision behind one interface, and routing gets decided on cost and latency once the capability gap closes. One headphone maker already runs 92% of its AI usage on-device. The teams who move fastest will be the ones whose code doesn't care which side answered. Why it matters: - For ICs: Design the model call as a seam from the start. A hardcoded cloud SDK deep in your business logic is the thing that will stop you from taking this option later. - For leaders: Budget for behavioral evals the way you budget for CI. Model routing turns a pricing optimization into a silent product change without them. - For founders: Browser and OS runtimes let you ship AI features with no inference bill and no data-handling review. At seed stage that is a different burn profile entirely. The Verdict: Real or Hype? Frontier-ish models on consumer hardware → Real but early. The engineering is genuine, but the headline numbers come from vendor benchmarks on well-chosen machines — validate on your own fleet. Quantization as a measured quality decision → Real. Same file size, different model is now demonstrable, which makes "which quant" a question with a right answer. Local-first as the default routing choice → Real but unevenly distributed. The APIs landed this quarter; the habits and hardware assumptions in most codebases are a year behind them.