Two days ago Qwen open-sourced Qwen 3.6-35B-A3B — a 35-billion parameter Mixture of Experts model that only activates 3 billion parameters per token. It's Apache 2.0 licensed, ships with a vision encoder, and is reportedly competitive with much larger models on agentic coding benchmarks. GGUF quantizations were up within hours.
Here's the thing: you can run it on a $599 Mac Mini M4 with 16GB of RAM. Not a toy demo — actual usable inference at 17 tok/s, zero swap, 81% memory free. This post is about how to do that, and which tools give you the best experience.
Why 35B-A3B works on 16GB #
The naive math says it shouldn't fit. The standard formula for estimating model memory (from BentoML):
Memory (GB) = Parameters (B) × (Bits per weight / 8) × 1.2 overhead
35 × 4 / 8 × 1.2 = ~21GB
21GB for a Q4 quantization. That doesn't fit in 16GB. So how does it work?
The key is the MoE architecture. "35B-A3B" means 35 billion total parameters, but only 3 billion a ctive per token. The model uses 256 total experts with 8 routed + 1 shared active per inference step. The remaining experts sit idle. This is what makes the --mmap
trick possible: llama.cpp
memory-maps the model file, and the OS only pages in the weights for the currently active experts. Since the hot working set is roughly 3B parameters (~2GB at Q4), it fits comfortably in 16GB with room to spare.
Jock.pl benchmarked this on a Mac Mini M4 16GB: 17.3 tok/s decode, 81% memory free, zero swap. That's not hypothetical — it's a real measurement on the base model Mac Mini.
Why this matters: On benchmarks, the 35B-A3B architecture beats dense models up to 120B on coding and reasoning tasks, while running at the latency of a 3B model. On 16GB RAM. For $0/month. That's the pitch.
Picking your inference tool #
There are four main ways to run LLMs locally on a Mac. Here's how they compare for running the 35B-A3B on 16GB specifically:
| Tool | Ease of setup | 35B-A3B on 16GB? | Tool calling | Notes |
|---|---|---|---|---|
OllamaLM StudioMLX / mlx-lm*mlx-vlm
has a PR in progress for tool calling support.
One important detail: Ollama 0.19 (released March 30, 2026) shipped an MLX backend that nearly doubles decode speed — from 58 tok/s to 112 tok/s. But it requires 32GB+ unified memory. On 16GB, Ollama falls back to the llama.cpp backend. Still works, just not the fast path. LM Studio doesn't have this gate and can use MLX on 16GB, which is a real advantage.
Setup 1: llama.cpp with mmap (recommended) #
This is the most reliable way to run the 35B-A3B on 16GB. Metal GPU acceleration is enabled by default on macOS — no flags needed.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release
pip install huggingface_hub
huggingface-cli download unsloth/Qwen3.6-35B-A3B-GGUF \
Qwen3.6-35B-A3B-Q4_K_M.gguf \
--local-dir models/
./build/bin/llama-cli \
-m models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
--mmap \
-c 4096 \
-n 512 \
-p "Write a FastAPI endpoint with input validation"
What's happening: --mmap
tells llama.cpp to memory-map the model file instead of it all into RAM. The OS pages in weights on demand. Because only ~3B parameters are active per token, the actual resident memory stays well under 16GB. The rest of the 21GB model file lives on your SSD and gets paged in only when an expert is activated.
You can also run it as an OpenAI-compatible API server for use with coding agents:
./build/bin/llama-server \
-m models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
--mmap \
-c 4096 \
--port 8080
GGUF files are also available from bartowski if you prefer different quantization levels.
Setup 2: Ollama (easiest) #
If you don't want to build anything from source, Ollama handles everything — download, quantization, API server — in one command. Under the hood it uses llama.cpp, so mmap works the same way.
ollama run qwen3.6:35b-a3b
That's it. Ollama downloads the GGUF, picks Q4_K_M by default, and starts an OpenAI-compatible API at http://localhost:11434
. You can connect coding agents directly:
ollama launch opencode --model qwen3.6:35b-a3b
ollama launch openclaw --model qwen3.6:35b-a3b
Ollama exposes models to anything that speaks the OpenAI API format — aichat, aider, opencode, and many others. Point them at http://localhost:11434/v1
.
On 16GB you'll get roughly the same 17 tok/s as raw llama.cpp. The MLX-accelerated path (which hits 70-80 tok/s for this model on larger machines) requires 32GB+, so Ollama falls back to the llama.cpp backend. Still perfectly usable for interactive work.
Setup 3: LM Studio (best GUI, MLX on 16GB) #
LM Studio deserves a special mention because it can run MLX-optimized models on 16GB — unlike Ollama which gates the MLX backend behind 32GB. MLX uses roughly 50% less memory than the llama.cpp backend for the same model at the same quantization, and is about 2x faster.
Download the app, open the models page (Cmd + Shift + M
), search for "Qwen3.6-35B-A3B", and filter by "MLX". Grab the 4-bit quantization from the mlx-community.
Kai Wern's guide reports 81.79 tok/s generation speed with LM Studio running the MLX-optimized 35B-A3B on a 64GB machine. On 16GB the model is a tighter fit via MLX, but community reports show it works — MLX's lower memory footprint is exactly what makes the difference between clean operation and swap thrashing.
To use LM Studio as a local API server (for coding agents), switch to the Developer screen (Cmd + 2
) and toggle the server to Running. It serves on http://127.0.0.1:1234/v1
.
Setup 4: Raw MLX (fastest inference, no tool calling) #
MLX is Apple's own ML framework and the fastest inference path on Apple Silicon. If you don't need tool calling — just direct chat or batch generation — this gives you the best tok/s.
pip install mlx-lm
mlx_lm.generate \
--model mlx-community/Qwen3.6-35B-A3B-4bit \
--max-tokens 200 \
--temp 0.7 \
--prompt "Write a Python function to merge two sorted lists"
mlx_lm.server --model mlx-community/Qwen3.6-35B-A3B-4bit --port 8080
Since Qwen 3.6-35B-A3B is a vision-language model, you can also use mlx-vlm
to process images:
brew install pipx
pipx install "mlx-vlm[torch]"
mlx_vlm.generate \
--model mlx-community/Qwen3.6-35B-A3B-4bit \
--max-tokens 200 \
--temperature 0.0 \
--prompt "Describe this image"
The catch: mlx-lm
doesn't support tool calling yet, so you can't use it as a backend for coding agents that need to read/edit files. For that, use llama.cpp, Ollama, or LM Studio.
Performance on Mac Mini M4 16GB #
All numbers are for the Qwen 3.6-35B-A3B at Q4 quantization on the base Mac Mini M4 (16GB unified memory, 10-core CPU, 10-core GPU), based on community benchmarks and Jock.pl's measurements:
| Tool | Decode (tok/s) | RAM resident | Swap | Notes |
|---|---|---|---|---|
| llama.cpp (mmap) | ~17 | ~3GB active | Zero | Most reliable on 16GB |
| Ollama | ~17 | ~3GB active | Zero | Same backend, easier setup |
| LM Studio (MLX) | ~25-35* | ~10-12GB | Minimal | Faster but tighter on memory |
| MLX (raw) | ~25-35* | ~10-12GB | Minimal | Fastest; no tool calling |
*MLX numbers on 16GB are extrapolated from Ante Kapetanovic's benchmarks on larger machines, scaled for memory bandwidth constraints. On a 64GB M4 Pro, the same model hits 70-80 tok/s via MLX. The 16GB constraint forces MLX to be more conservative with caching.
For comparison, on a Mac with 32GB+ and Ollama 0.19's MLX backend enabled, this same model hits 1810 tok/s prefill and 112 tok/s decode. The 32GB threshold is real — if you're buying a Mac specifically for local inference, the 32GB upgrade pays for itself.
But 17 tok/s is genuinely usable. That's fast enough for interactive chat, code generation, and tool-calling agents. It's slower than an API call, but it's free, private, and offline.
What about Qwen 3.6-Plus? #
The full Qwen 3.6-Plus flagship (1M context, top-of-the-line benchmarks) was released on April 2, but it's API-only through Alibaba's DashScope. No weights, no GGUF files, no local option. The GitHub repo is up but only for the 35B-A3B variant.
For local inference, 3.6-35B-A3B is it — and given its benchmark numbers relative to its active parameter count, it's far more than a consolation prize.
My setup #
I run the Qwen 3.6-35B-A3B as my daily driver on the Mac Mini M4 16GB. Specifically:
Default: Ollama withqwen3.6:35b-a3b
running in the background. One-command start, OpenAI-compatible API, tool calling works. I point aider and opencode at it.When I need speed: LM Studio with the MLX-optimized version. Noticeably snappier for interactive chat.For scripting / batch work: llama.cpp server with--mmap
. Most control over parameters, lowest overhead.
One pattern that works well: Ollama keeps the model warm in the background. When you send a request after a cold start, the first response is slow (~5-10s) while the active experts get paged in. Subsequent requests are much faster because the hot pages stay cached. Don't kill Ollama between sessions if you can avoid it.
Useful links #
Hugging Face — Qwen 3.6-35B-A3B(model card, benchmarks)Unsloth — Qwen 3.6-35B-A3B GGUF quantizationsbartowski — Qwen 3.6-35B-A3B GGUF (alternative quants)Ollama blog — MLX backend announcement(0.19 release)Qwen docs — running with llama.cppJock.pl — 35B on Mac Mini 16GB(benchmark source)Kai Wern — Run Qwen locally on your Mac(LM Studio + Ollama + MLX walkthrough)Ante Kapetanovic — Ollama vs llama.cpp vs MLX benchmarkInsiderLLM — MLX vs Ollama speed testDEV.to — MLX on Apple Silicon for 2x performanceGitHub — 16GB VRAM Local configs and launchersWill It Run AI — Qwen VRAM requirements guideUnsloth — Qwen local setup docsr/LocalLLaMA — Explain MoE like I'm 25