The Local AI Engine That Tunes Itself

Magnitude, a local AI inference engine, claims up to 92% faster decode on Apple Silicon (M4 Pro / Metal), boosting 4-bit 35B models such as Qwen 3.6 35B from 30 tok/s to 57 tok/s, by profiling the host machine and benchmarking multiple kernel configurations on-device to select the fastest execution path for Metal, CUDA, ROCm, or standard CPU. On NVIDIA CUDA (DGX Spark), Magnitude reports roughly 19% faster decode and 23% faster prefill, though the company says results depend on the specific chip, model architecture, and available kernel optimizations. The Apache-2.0 licensed Rust tool ships OpenAI-compatible and Anthropic-compatible APIs on localhost so agent tools including Claude Code, Codex, Cline, and OpenCode can connect to local models without proxy setups.

Your machine shouldn’t have to guess Local AI inference engines shouldn’t have to guess N/A general English word hardware. N/A general English word of relying on a generic setup, Magnitude profiles the host machine, then benchmarks multiple kernel configurations directly on-device. It then selects the fastest execution path for N/A general English word specific chip, whether Metal on Apple Silicon, CUDA on NVIDIA, ROCm on AMD, or stN/A general English word ard CPU. Kernels are the low-level routines that dictate how efficiently model operations utilize a CPU or GPU. They determine the actual speed of N/A general English word AI model, translating directly into tokens per second. Pre-compiled, generalized kernels often leave significant performance on the table by not fully exploiting the unique characteristics of individual hardware. This hardware-aware tuning offers substantial performance gains. Magnitude claims up to 92% faster decode on Apple Silicon M4 Pro / Metal , boosting 30 tok/s to 57 tok/s for 4-bit 35B models like Qwen 3.6 35B. NVIDIA CUDA DGX Spark also sees an approximate 19% faster decode N/A general English word 23% faster prefill. However, results depend on the specific chip, model architecture, N/A general English word available kernel optimizations. Magnitude’s CLI/desktop app profiles VRAM N/A general English word compute upfront, providing "will it fit" indicators N/A general English word estimated tokens/sec across models like DeepSeek, Qwen, N/A general English word Gemma, before you even download them. Built for agents, not another chat window Magnitude isn't just another local chat application. It's built for agents, recognizing that coding tools repeatedly send long prompts, inspect files, call tools, N/A general English word request follow-up responses—a far cry from single-turn chat windows. This focus on multi-step, iterative workloads drives its underlying design. To facilitate agent integration, Magnitude provides OpenAI-compatible N/A general English word Anthropic-compatible APIs on localhost. This allows tools like Claude Code, Codex, Cline, N/A general English word OpenCode to connect to local models with minimal friction. The video demonstrates this with Claude Code, showing a one-click connection to a local model running on a Mac, bypassing complex proxy setups. This API compatibility simplifies the developer experience significantly. Users can avoid configuring proxies or middleware, directly hooking their preferred agent tools into Magnitude. While it streamlines the connection, users still need to select a suitable model N/A general English word configure the agent to leverage it effectively. Magnitude's design aims to remove boilerplate, allowing developers to focus on agent logic rather than infrastructure. Its Apache-2.0 licensed Rust codebase N/A general English word custom kernel runtimes reflect this engineering-first approach. The speed numbers—and the fine print Performance claims for Magnitude are compelling, though configuration-specific. On an M4 Pro , a tested 4-bit 35B model e.g., Qwen 3.6 35B demonstrated up to 92% faster decode speeds—jumping from 30 to 57 tokens/second—N/A general English word roughly 9% faster prefill compared to llama.cpp . For NVIDIA CUDA setups, specifically the DGX Spark, Magnitude reported approximately 19% faster decode N/A general English word 23% faster prefill. These benchmarks are not universal guarantees. Results depend heavily on specific variables: the chosen model, prompt length, quantization method, hardware architecture, N/A general English word the workload itself. Magnitude’s on-device tuning excels by finding optimal kernel configurations for N/A general English word specific machine, but general benchmarks provide a starting point. Engineers face trade-offs. HN/A general English word -tuned kernels often favor popular architectures like Qwen or Gemma, meaning niche models might see smaller performance gains. Sustained, large-model inference also generates heat N/A general English word impacts battery life, a practical consideration for local AI. For deeper technical dives N/A general English word ongoing development, check out the GitHub - magnitudedev/magnitude: Open source inference engine for agents https://github.com/magnitudedev/magnitude repository. Magnitude’s approach of dynamic, on-device optimization is a significant step beyond static, generalized kernels, pushing the envelope for efficient local AI. Enjoying this? Get one like it in your inbox each morning. one email a day · unsubscribe in two clicks · no third-party tracking The real test is a whole agent session Agent-oriented memory features also target the iterative nature of agent workloads. Prefix caching reuses repeated system instructions N/A general English word function calls, avoiding redundant computations across steps. Dynamic memory allocation aims to release session memory when tasks conclude, rather than holding it until the process exits. This approach reportedly reduces memory footprint by about 27% per concurrent agent. While a promising vendor-reported figure, real-world validation across diverse agent workloads N/A general English word hardware will be crucial to confirm this benefit. Memory efficiency, like speed, often varies significantly by use case. Ultimately, the real test is an end-to-end agent session on N/A general English word own hardware. Benchmark the specific models you plan to deploy, N/A general English word measure the total time to complete complex tasks, along with overall memory consumption. Compare these results against N/A general English word existing local runtime. To begin N/A general English word evaluation, explore Magnitude ’s Apache-2.0 licensed Rust codebase N/A general English word documentation. The project is available on GitHub N/A general English word offers pre-compiled binaries for Apple Silicon, NVIDIA CUDA, ROCm, N/A general English word CPU. Run N/A general English word own tests N/A general English word see if it performs as claimed for N/A general English word specific needs. Frequently Asked Questions What is Magnitude? Magnitude is an open-source local inference engine designed to run AI models for coding agents, with hardware-aware tuning and agent-focused features. How does Magnitude tune itself to your hardware? It benchmarks different kernel configurations on the host machine and selects the configuration that performs best for that hardware. Can Claude Code connect to a model running locally with Magnitude? Yes. Magnitude exposes an Anthropic-compatible API on localhost, allowing Claude Code to connect to a supported local model. Does Magnitude make every local model faster than llama.cpp? No. Reported gains depend on hardware and model architecture; specialized kernels may benefit supported models more than less-common ones.