A single 24GB GPU like the RTX 3090 or RTX 4090 is the practical floor for serious local inference in 2026, and the best strategy is to run modern 20B–35B-class models at Q4_K_M quantization rather th…
A fork of the DwarfStar inference engine, andreaborio/ds4, aims to run large Mixture-of-Experts models like Qwen3.6-35B-A3B on 16–64 GB Apple Silicon Macs by using adaptive SSD streaming and Metal res…
AI agents in the AI Village finetuned a Kimi K2.6 model as their leader using only 35 rows of training data, after earlier attempts with a Qwen3-8B model proved too small to navigate the Village inter…
Samosa Chat, an independent Apache-2.0 project, enables local inference of the Qwen3.6-35B-A3B model on a 16 GB Apple Silicon Mac entirely on CPU with no cloud account or telemetry. The tool fits the …
Anthropic's Model Context Protocol (MCP) enables universal tool connectivity for AI agents, and Qwen3.6-35B-A3B, a Mixture of Experts model with 262k-token context, is optimized for MCP-based agentic …
A developer patched llama.cpp to improve prompt processing throughput by 20% when using Multi-Token Prediction (MTP) on GPU, recovering performance lost to MTP overhead. The proof-of-concept processes…
NeuralWatt, a US-based AI inference provider, introduced energy-based metering for LLM inference, charging by kilowatt-hour instead of per token. A user reported an average 82.9% cost reduction compar…
A developer investigating low prompt processing throughput with Multi-Token Prediction (MTP) in llama.cpp created a proof-of-concept that recovers the overhead by processing only the output row of the…
A single H200 GPU with 141GB HBM3e cannot comfortably run DeepSeek V4 Flash (284B total, 13B active parameters) due to VRAM constraints, even with 2TB system RAM for offloading. The model requires an …
A developer reports that running gpt-oss-120B on an NVIDIA DGX Spark (GB10) failed due to MXFP4 quantization lacking native hardware support on SM121, causing kernel hangs and memory spikes. Switching…
Applied Compute trained a small open-source model (Qwen3.6-35B-A3B) to route SWE-bench Verified tasks across three frontier coding models — Nemotron 3 Ultra, GPT-5.5, and Claude Opus 4.7 — choosing th…