Perplexity announced it will open source Lily, a local AI engine built for Apple silicon and the Qwen3.6-35B-A3B model, which uses a Rust runtime and custom Metal kernels without PyTorch or MLX. Perpl…
Perplexity has open sourced Lily, a Rust and Metal inference engine for Qwen3.6-35B-A3B on Apple Silicon, achieving a 77.4% prefill speedup via fused grouped GEMM and an 89% gain from GPU-only routing…
A developer built CarWatch, a Raspberry Pi 5-based in-car AI assistant that runs a 35B-parameter Qwen3.6-35B-A3B model locally at 3.5 tokens per second, achieving full offline functionality for voice,…
As of August 2026, the best open LLM to run locally depends on VRAM, with gpt-oss-20b recommended for 8-12GB, Gemma 4 31B or Qwen3.6-35B-A3B for 24GB, and gpt-oss-120b or DeepSeek V4 Flash for 128GB u…
Ornith AI released Ornith-1.5-35B-A3B, a 35B-parameter mixture-of-experts model that activates about 3B parameters per token, trained via a self-improvement loop that jointly optimizes task generation…
FreeToken, an open-source inference engine developed by UC Berkeley researchers including Shuo Yang and Xiaoze Fan, reports decoding DeepSeek-V4-Flash, a 284B-parameter Mixture-of-Experts model, at 22…
FlashML released FreeToken, an edge-native Mixture-of-Experts (MoE) serving engine that runs 290B+ parameter frontier MoE models locally on consumer gaming PCs at interactive speeds. The engine suppor…
Deep Reinforce's Ornith 1.5 35B-A3B, a mixture-of-experts language model activating about 3 billion parameters per token, outperforms Qwen3.6-35B-A3B on every published coding and agentic benchmark, s…
NInfer, a from-scratch C++/CUDA inference engine, achieves up to 1,313.8 aggregate decode tokens per second on a single NVIDIA GeForce RTX 5090 for the Qwen3.6-35B-A3B model at concurrency 8, with the…
Solheim AI launched Virtual Private LLM (VPL), a flat-fee service providing private, EU-hosted LLM instances with no token or usage limits, starting at €15.00 per month for one instance with a 64k con…
Brent Ozar, a Microsoft SQL Server expert and podcast host, tested whether a fully local AI setup could answer an Office Hours question, using the MiniMax H3 image-to-video model and a local Qwen3.6-3…
Fuse-1 Lite, a 5.72B parameter mixture-of-experts coding model from LiquidAI, can run locally with VRAM needs ranging from 3.36 GB in 4-bit quantized form to about 12 GB in full bfloat16 precision, ac…
LiquidAI and collaborators released fuse-1 Lite, a 5.72 billion parameter mixture-of-experts model that transplants 960 coding experts from Qwen3.6-35B-A3B into LiquidAI's LFM2.5-2.6B host model, achi…
Swiftlet, an open-source project from developer Leonickson, enables running an 80-billion-parameter Qwen3-Next-80B-A3B sparse Mixture-of-Experts model in just 4.3 GB of RAM on Apple Silicon Macs by ke…
Swiftlet, a 10,000-line Swift and Metal project under Apache 2.0, runs Alibaba's Qwen3-Next-80B-A3B at 4-bit in 4.3GB of peak RAM on a Mac, decoding at 4.5–5 tokens per second on an M5, and runs the Q…
Swiftlet, a Swift + Metal runtime for Qwen3-Next and Qwen3.5/3.6 MoE hybrid models, runs an 80B Qwen in 4.3 GB of RAM on a Mac and a 35B on an iPhone 17 in about 2.5 GB of RAM at about 1 token per sec…
Hetzner, the German cloud host known for low prices, has quietly launched a free, OpenAI-compatible LLM inference endpoint serving a single Qwen3.6-35B-A3B model, signaling that open-weight inference …
Researchers released KAT-Coder-V2.5, a coding-focused agentic model trained to act autonomously inside real executable repositories, achieving the best agentic tool-use result on PinchBench and rankin…
Agent swarms make local AI rigs cost-effective for the first time, according to developer testing. A single-agent session on a 2×3090 rig costs $0.80 in API-equivalent tokens, while a swarm of 32 agen…
A single 24GB GPU like the RTX 3090 or RTX 4090 is the practical floor for serious local inference in 2026, and the best strategy is to run modern 20B–35B-class models at Q4_K_M quantization rather th…