A new LLM inference runtime called BaseRT just shipped, and it makes a claim worth paying attention to: up to 6.4x faster local inference on Apple Silicon than llama.cpp. The trick is not some clever model compression or new architecture — it is that BaseRT skips every intermediate framework entirely and writes directly to Apple’s Metal GPU API. No MLX. No PyTorch. No CoreML. Hand-written Metal kernels, tuned from the ground up for M-series unified memory topology.
If you run local models on a Mac and you are not already at the ceiling of your hardware, this is worth 10 minutes of your time.
Why Your Current Runtime Is Leaving Tokens on the Table #
Before getting to BaseRT, it is worth understanding why llama.cpp and MLX are slower than they could be — because the gap is structural, not just an optimization detail.
llama.cpp was built for CPU inference and retrofitted with a Metal backend. That backend translates compute patterns shaped by CUDA into Metal shaders. The translation works, but it does not exploit the specific way Apple Silicon handles memory — every tensor op carries the overhead of an abstraction designed for a different world.
MLX is better. Apple designed it from scratch for unified memory, and it shows: MLX is typically 30–50% faster than llama.cpp on M-series chips. But MLX still routes through MPSGraph — Metal Performance Shaders Graph — a graph compiler that adds its own scheduling overhead for the memory-bandwidth-bound operations that LLM decode actually consists of.
BaseRT’s bet is that cutting out both translation layers and graph compilers, and writing directly to Metal with kernels tuned for the actual shape of LLM workloads, recovers meaningful performance. The benchmarks suggest that bet is mostly correct.
What the Benchmarks Actually Show #
The headline “6.4x faster” comes from specific configurations — likely smaller models or prefill comparisons. The peer-reviewed paper tells a more nuanced story, run on M3 Base and M4 Pro hardware against llama.cpp (commit b9630, June 2026) and MLX v0.31.2:
| Model | BaseRT Q4 (M3) | vs llama.cpp | vs MLX |
|---|---|---|---|
| Qwen3-0.6B | 223 tok/s | +34% | +22% |
| Llama-3.2-1B | 125 tok/s | +13–20% | +1–10% |
| Llama-3.2-3B | 44 tok/s | ±6% | N/A |
| Gemma-4-E2B | 59 tok/s | +varies | Not supported |
The pattern is clear: BaseRT’s advantage is largest on smaller dense models. At Qwen3-0.6B, you get 223 tok/s at Q4 — fast enough for a real-time coding assistant that feels snappy. By Llama-3.2-3B, the margin narrows to roughly ±6%, which is within noise. On MoE models, BaseRT achieves up to 1.81x faster prefill (time to first token), which matters for interactive use cases even when sustained decode speed equalizes.
The technical reason: BaseRT uses tiled GEMM kernels built on Metal’s simdgroup matrix intrinsics and a FlashAttention-style attention kernel with online softmax. Memory usage scales linearly with sequence length rather than quadratically. For smaller models where per-token dispatch overhead is a meaningful percentage of total latency, eliminating framework abstraction overhead pays off. For larger models, inference becomes fully memory-bandwidth-bound and the gap closes.
Getting Started in Three Commands #
The CLI covers install, model pull, chat, and API serving in a single binary:
brew install basert
basert pull qwen3-0.6b
basert serve --model qwen3-0.6b --port 8080
The OpenAI-compatible API server is the most immediately useful feature. Point any existing client at localhost:8080
and it works without code changes in tools that already speak OpenAI’s chat completions format:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="basert")
response = client.chat.completions.create(
model="qwen3-0.6b",
messages=[{"role": "user", "content": "Explain Metal GPU API in one sentence"}]
)
print(response.choices[0].message.content)
Beyond chat: the server supports tool calls, continuous batching, paged KV cache, prefix caching, and multimodal inputs (vision and audio on supported models). Language bindings exist for Python, Node.js, Rust, and Swift.
The Catch You Should Know About #
BaseRT is not fully open source. The CLI, .base model format, C API headers, and language bindings are Apache-2.0. The inference engine — the binary that actually runs your models — is proprietary and distributed as a prebuilt artifact under its own license. If your threat model requires auditing every line of code that touches your data, that is a hard stop.
Model family support is also limited: LLaMA, Qwen3, Gemma, Whisper, and BERT families only. No Mistral, Phi, or DeepSeek support yet. Models need converting to BaseRT’s .base format before they run — basert pull
handles this automatically for HuggingFace models, but custom models require a separate conversion step.
And it is Apple Silicon M1 or later, macOS 14+, nothing else. Intel Macs and every other platform are out.
Who Should Switch Now #
If you run small-to-medium dense models (under ~7B parameters), use an M-series Mac, and your primary use case is interactive — coding assistant, local RAG, voice interface — BaseRT is worth benchmarking against your current setup today. The zero-migration OpenAI API compatibility makes testing a 30-minute experiment, not a project.
If you need the broadest model support, are running larger models where the throughput gap disappears, or cannot accept a proprietary engine binary, stick with llama.cpp or MLX. Both are excellent and fully open. BaseRT is not a replacement for the ecosystem — it is a performance option for a specific workload profile on specific hardware.
The GitHub repository is at github.com/basecompute/baseRT. The technical paper is on arXiv (2607.00501) if you want the full benchmark methodology. The HuggingFace blog post from Base Compute has a walkthrough with architecture diagrams.