A new weight reordering technique for Mixture-of-Experts (MoE) models optimizes physical file layout to minimize NVMe seek overhead, achieving massive gains in explicit-read inference engines like MLX…
Apple's MLX team introduced distributed inference and training capabilities for machine learning workloads across multiple Macs at WWDC26, demonstrating how to scale large language models using RDMA o…
A developer's MoE model file reordering technique, mbolt, achieved +32.3% decode throughput and −26.3% time-to-first-token on a 235B-parameter model running on a 48 GB MacBook, as independently verifi…
Apple's M5 generation introduces a dedicated Neural Accelerator on every GPU core, and BaseRT, a native Metal inference runtime, exploits these units to deliver up to 6.4× higher prompt-processing thr…
A new LLM inference runtime called BaseRT achieves up to 6.4x faster local inference on Apple Silicon than llama.cpp by writing directly to Apple's Metal GPU API, skipping intermediate frameworks like…
Cactus released Gemma 4 E2B Hybrid, a small on-device model that outputs a confidence score (0-1) for each answer, allowing developers to route low-confidence queries to a larger model. The model matc…
ExecuTorch's experimental MLX delegate, released in May 2026, enables PyTorch models to run on Apple Silicon GPUs. Testing Qwen3-0.6B on an M1 Max, MLX INT4 achieved 188.9 tokens/s, 4.52x faster than …
The mlx-community/Laguna-S-2.1-oQ2e quantized model running in-process through mlx-vlm on a 128 GB Apple M5 Max achieved a perfect overall score of 1.000 across six tasks, with 40.85 generation tok/s …
Prince Canuma's Nativ is a new macOS desktop application that wraps Apple's MLX framework to run AI models locally, offering both a chat interface and a localhost API server. The app automatically det…
Velora, an open-source macOS dictation tool that runs entirely on-device using Apple Silicon and MLX, has been released under an MIT license. The tool uses Whisper for speech-to-text and a local LLM f…
A five-engineer team on Sonnet 4.6 saw a $4,800 monthly bill for Claude Code sessions, with only $960 attributed to the model itself and $3,840 coming from inference engineering — the layer between th…
Eaon, a free and open-source native macOS app, lets users switch between 49 AI models from a single interface, supporting both cloud keys and fully offline local models via Ollama, llama.cpp, or MLX. …
PrismML shipped Bonsai 27B, a 27B-parameter model based on Qwen3.6 27B, with a 1-bit variant packing to 3.9 GB that fits on an iPhone 17 Pro, clearing the memory gate that blocked prior builds of this…
PrismML released Bonsai 27B, claiming it is the first major AI model of its size that can run on an iPhone, iPad, or Mac, with a 1-bit version fitting in about 4 GB of memory. PrismML CEO Babak Hassib…
A solo developer running brevio.news discovered that a hosted LLM API silently broke the product's weekly brief feature for four weeks after the provider retired a model ID. The developer fixed the is…
Recent events, including Claude Code restricting access in China and a Chinese government security bulletin on AI coding tools, have renewed concerns about where code data goes when processed by AI as…
An independent benchmark of on-device LLMs on iPhone A17 Pro found Apple's system model achieves ~149 tok/s with only 12MB peak app memory, while 4B-class open models like Qwen3 4B and Llama 3.2 3B tr…
Ollama 0.31 introduces multi-token prediction for Gemma 4 on Apple Silicon, cutting token generation time by 90% on Aider's coding benchmark. The update automatically accelerates local coding agents w…
A local AI assistant called Kira discovered that nearly half of its model time was spent re-reading boilerplate prompt text—a 'prefill tax'—after migrating from Ollama to Apple's MLX framework, which …
Qwen open-sourced the 35-billion parameter Mixture of Experts model Qwen 3.6-35B-A3B, which activates only 3 billion parameters per token and runs on a $599 Mac Mini M4 with 16GB RAM at 17 tok/s with …