Magnitude (YC S25) shipped v0.2.0 yesterday, replacing its llama.cpp backend with a Rust inference engine that auto-tunes GPU kernels on your hardware before running models. On an M4 Pro, Qwen 3.6 35B decode went from 30 to 57 tokens per second — 92% faster. The launch hit Hacker News front page with 163 points, and developers started plugging it into Claude Code, OpenCode, and Cline within hours.
What “Self-Optimizing” Actually Means #
Most inference engines ship pre-compiled kernels for broad hardware classes: a Metal path for Apple Silicon, a CUDA path for NVIDIA. These kernels are good for the average device in each class. Magnitude takes a different approach — it compiles and tunes its computational kernels on your specific chip before the first model run. That autotuner pass typically takes a few minutes on first launch and then never runs again.
The practical payoff: the engine isn’t optimizing for a generic “M-series Mac.” It’s optimizing for your M4 Pro’s exact specs. That’s where the 92% decode improvement on Apple Metal comes from, and why the gains are real rather than cherry-picked benchmark theater.
However, one honest caveat: HN users running M5 Max hardware found Magnitude slower than MLX-based engines on that specific chip (161 tok/s vs 175 tok/s on MLX). The Magnitude team acknowledged a known Metal 4 matmul optimization gap on the M5+ generation and said a fix is in progress. If you’re on M5 or newer, test before committing. On M4 and earlier, the numbers hold up.
The Part That Actually Matters for Agent Developers #
The performance story is interesting. The integration story is what most developers will care about more immediately.
v0.2.0 ships with one-click connections for Pi, OpenCode, Hermes, Codex, Claude Code, Cline, Oh My Pi, and OpenClaw. It also exposes both OpenAI-compatible and Anthropic-compatible endpoints on localhost:10100, which means any agent that supports either API format works without extra configuration.
The bigger addition is magnitude serve — a headless CLI mode that runs the inference server without a desktop UI on macOS, Windows, and Linux. Server deployments, remote machines, and CI pipelines can now use Magnitude without a GUI. That’s the feature that makes this a genuine llama.cpp competitor rather than just a desktop app with a nice UX.
Where Magnitude Fits in the Local Inference Stack #
In practice, Magnitude isn’t trying to replace every inference tool. It’s targeting a specific developer: someone already running local coding agents, frustrated by token costs or latency, who wants something faster than llama.cpp without the setup complexity of vLLM or the Apple-only constraint of MLX.
That’s a real segment. As AI coding agents have gone from experimental to daily-driver in 2026, the economics of per-token cloud pricing for heavy agent usage are becoming a genuine concern. Developers running Claude Code or OpenCode continuously are spending real money. Local inference with Qwen 3.6 or Llama 3.3 has gotten good enough that the quality tradeoff is often acceptable. Magnitude’s value proposition — faster than llama.cpp, agent integrations built in, privacy by default — hits exactly that pain point.
The project has 5.9k GitHub stars after a single day, Apache 2.0 licensed, and now supports CPU-only inference as a fallback for anyone without a capable GPU. Memory usage per agent is 27% lower than competing solutions according to the team’s benchmarks.
Try It If You’re Running Local Agents #
The headline numbers are slightly oversold for the M5 generation, but the core product is solid and the direction is right. If you’re already running a local coding agent and you’re on M4 or NVIDIA hardware, it’s worth trying. Download the desktop app or run magnitude serve, connect your agent, and the performance difference is either immediately visible or it isn’t on your specific hardware.
The GitHub repo has the benchmarks and full hardware compatibility list. The HN launch thread has real-world reports from developers who tested it across different hardware setups. For context on how it compares to Ollama, vLLM, MLX, and llama.cpp, the 2026 local inference engine comparison is still the most thorough breakdown available.