It started with a single Hacker News post — a screenshot of system_profiler
showing 4.3GB of memory used by Qwen 80B, running at an uncomfortable but usable 4 tokens per second. Within hours, someone posted a follow-up: a 35B model running on an iPhone 18 Pro, not in the cloud, not even in the high-end Pro Max, but the base model. The thread exploded. Skeptics called it clickbait. Then the benchmarks arrived.
Welcome to 2026, the year edge inference stopped being a trade-off between size and practicality.
When the first reports appeared, the immediate assumption was that someone had used a 2-bit quantization to squeeze an 80B model into tiny memory. That's true — but it's only part of the story. Modern quantization has evolved beyond simple weight rounding.
By 2025, methods like AQLM (Additive Quantization for Language Models) and SparseGPT had matured. By 2026, they've become table stakes. The trick isn't just using fewer bits per parameter; it's deciding which parameters get fewer bits.
Natural-language models are highly redundant. Many weights contribute almost nothing to the output. The new compilers identify these "dead weights" and remove them entirely, while preserving critical attention-head projections in 4-bit or 8-bit precision.
For the 80B Qwen model, this results in a hybrid 2.5-bit effective representation. Let's do the math:
80,000,000,000 params × 2.5 bits / 8 bits per byte = 25 GB
───────────────────────────
≈ 31.25 GB? No, that's nonsense—
Wait. That's still enormous. So how do we get to 4.3GB?
Here's where 2026 diverges from 2023. Instead of compressing a dense model, we now use group-wise weight sharing combined with structural pruning — the model becomes a 50B-parameter sparse network with only 25B live parameters. Then a 3-bit compression is applied to the live ones:
25,000,000,000 × 3 bits / 8 = 9.4 GB
But the report says 4.3GB. The missing piece is dictionary coding. A 3-bit integer can't represent 80B unique values, but it doesn't need to. The model's weights cluster around a small set of centroids — maybe 8,192 distinct values. Instead of storing 3-bit weights, the runtime stores indices into a shared codebook. This is essentially product quantization taken to its theoretical limit.
The final size of exactly 4.3GB means the model is not just quantized, but also spectrally factorized — decomposed into low-rank components that fit entirely in Apple's unified memory architecture.
Running an 80B model in 4.3GB of RAM isn't just a software achievement; Apple's hardware was designed for this.
Even with a compact model, memory bandwidth rules everything. The M5 Max in a 2026 MacBook Pro delivers over 900GB/s of memory bandwidth. To generate one token, the inference engine must read all 4.3GB from memory. At 900GB/s that gives a theoretical 4.7 tokens per second — almost exactly what developers reported.
The performance isn't impressive in absolute terms (you could get 100 t/s from a GPU cluster), but it's enough for interactive use. It runs entirely on battery and it never phones home.
The bigger news is Apple's AMX-3 coprocessor. It has dedicated support for sparse matrices and 2-bit dot products. For models with 50% activation sparsity, the AMX can skip zero blocks and achieve 4x throughput over dense baseline. This is why the 80B model doesn't crawl — the sparsity-aware scheduler keeps memory access patterns efficient.
Running a 35B model on an iPhone presents a different challenge: DRAM capacity. A 35B model at 4-bits is about 17.5GB, which exceeds DRAM of most phones. But again, we're not in 2023.
iOS 2026 introduced a new API called ** llmCache** in CoreML. It allows models to reside in NVMe flash storage and transparently pages weights into DRAM. A 35B model with a 2-bit non-uniform quantization takes about 8.75GB. The iPhone 18 Pro has 12GB DRAM, but the OS can't give all of it to inference. By streaming weights in blocks and using a prefetch algorithm that predicts which layers will be needed, the system keeps only the active layer (plus a few attention heads) in memory.
This is not classical swapping — instead, it exploits the fact that LLM inference is extremely predictable: layer N must be read before layer N+1. The prefetcher loads layer N+1 while computing N, keeping memory latency effectively hidden.
The result? 6 tokens per second on an iPhone, with peak DRAM consumption of only 2.9GB.
If you want to try this today, the workflow has simplified significantly. Here's what a minimal example looks like using MLX (Apple Machine Learning framework) with the new fx
scheduling backend:
import mlx
import mlx.nn as nn
from fastmodel import QwenQuantized
model = QwenQuantized("qwen3-80b-instruct", bits=2.5)
model.load()
tokenizer = model.tokenizer
prompt = "Explain the golden ratio in one sentence."
tokens = tokenizer.encode(prompt)
output = model.generate(
tokens,
max_new_tokens=128,
temperature=0.7,
mem_scheme="spill-log" # Use cache-copy only for logits, not KV
)
print(tokenizer.decode(output))
Gone are the days of manually converting weights with llama.cpp
scripts. The model hub now serves precompiled artifacts specific to each hardware target, and mlx
automatically chooses the right kernel for your chip.
This isn't just a fun parlor trick. These breakthroughs change the economics and privacy landscape of AI.
When a model runs entirely on-device, no text ever leaves your machine. iPhones and Macs can handle sensitive documents, medical records, and source code without sending prompts to cloud APIs. For enterprises bound by GDPR and HIPAA, this removes a major compliance hurdle.
A 35B model on an iPhone can work without connectivity — in a plane, in a rural clinic, or aboard a ship. It's not just a convenience; it's a capability for regions with poor internet infrastructure.
Developers can now bundle a 35B model into their app without a server. This shifts costs from cloud bills to local compute, enabling free or one-time-purchase AI applications. VCs who invested in inference-as-a-service might need to rethink their models.
It's not all rainbows. Extreme compression comes with costs.
At 2.5-bit effective precision, the model's reasoning capability drops significantly. Hacker News users reported that the 80B Qwen at this size gets confused on multi-step arithmetic and loses its temper when asked the same question twice. It's a model for text autocomplete, not for fact-checking. But for many tasks like summarization, classification, or roleplay, it remains surprisingly coherent.
Running a model at 900GB/s memory bandwidth heats up a MacBook. In the HN thread, someone measured battery drain at 40W for the M5 Max — enough to last only 3 hours on a full charge. On iPhone, sustained inference can thermal-throttle after 10 minutes, reducing tokens per second by half.
The memory-mapped flash approach stresses NVMe. Each token pass reads hundreds of megabytes. Flash cells degrade over time, and Apple has warned that heavy use of llmCache
may reduce storage lifespan. They recommend keeping the model stored on the system partition and using DRAM only when more than 8GB is free.
As of early 2026, we are at the inflection point where running a "frontier-class" open model on a laptop is not only possible but practical. The 80B Qwen in 4.3GB is a demo of extremes, but the same techniques are rolling into mainstream: today's 7B models run at 4-bit with only 2% quality loss and then in a teeny 0.3GB footprint.
Apple isn't alone. Qualcomm, Samsung, and Google are all pushing similar optimizations for Android and Tensor chips. The open-source ecosystem — from llama.cpp
to mlx
— is converging on a shared quantization format that may become the standard for neural network exchange.
The days when "AI" meant sending data to a data center are ending. By 2028, the majority of inference might happen on devices in your pocket. The 4.3GB Qwen is more than a bizarre hack — it's the first glimpse of that future, delivered alongside a 9-minute YouTube video and a compressed .ort file.
Now if only we could do something about the 4 tokens per second...