Alibaba's Qwen team released Qwen3-30B-A3B, a Mixture-of-Experts model with 30.5 billion total parameters but only 3.3 billion active per token, enabling it to run on a single 24 GB graphics card at s…
Retrieval-augmented generation (RAG) allows local LLMs to access external documents without consuming excessive memory, by retrieving only relevant chunks via a small embedding model and vector databa…
Chinese lab Z.ai released GLM-5.2, a 753-billion-parameter open-weight Mixture-of-Experts model that tops the Artificial Analysis Intelligence Index at #1, but its 1.51 TB weight size makes local depl…
Beelink's SER10 Max mini PC, powered by AMD's Ryzen AI 9 HX 470 "Gorgon Point" chip, matches Apple's M4 Pro on a single TypeScript benchmark but falls flat elsewhere, with local LLM performance hamper…
Local LLM inference splits into two phases—prompt processing (compute-bound) and generation (memory-bandwidth-bound)—explaining why hardware with identical token generation speeds can have vastly diff…
The KV cache, a memory store for attention keys and values, grows linearly with context length and can exceed model weights in VRAM usage, causing out-of-memory errors for local LLM users. At 32k cont…
A new cheat-sheet from VettedConsumer lists the cheapest hardware capable of running local large language models from 8B to 235B parameters, based on real owner-reported speeds. The guide is available…
NVIDIA's RTX 3090, launched in 2020, remains the best value for local AI in 2026 due to its 24 GB VRAM and 936 GB/s bandwidth, outperforming newer cards like the RTX 5070 and 5080 on price-to-performa…
Quant Picker is a new tool that calculates which GGUF quantization level fits a given model and machine, balancing file size, quality, and context budget. It recommends the highest quantization that l…
A comparison of three local large language model runtimes reveals that llama.cpp is the core inference engine, while Ollama and LM Studio are user-friendly wrappers built on top of it. Ollama offers a…
Mixture-of-Experts (MoE) architecture allows large language models to use only a fraction of their total parameters for each token, enabling models with 671 billion total parameters to run at speeds c…
Local AI users are accumulating hundreds of gigabytes of model files, including multiple quantizations of the same model, image and video generation weights, and RAG datasets, often with duplicates th…
A buyer who spent roughly $10,000 on both a Mac Studio M3 Ultra with 512 GB of unified memory and a dual NVIDIA DGX Spark setup ran the same 397-billion-parameter model on each machine and measured th…
NVIDIA's RTX 5090, priced at $1,999, offers 32 GB of GDDR7 memory with 1,792 GB/s bandwidth, making it the fastest consumer GPU for both gaming and local AI inference. The card excels at running 32B-c…
The price of the 1 TB Steam Deck OLED jumped $300 overnight, from $649 to $949, as part of a broader price increase across laptops, mini PCs, and handhelds driven by a memory-pricing crisis. Roughly 9…
Two machines with 128 GB of unified memory — the AMD Strix Halo and the NVIDIA DGX Spark — are being compared by owners who run 70B-parameter language models locally. AI developer u/Eugr benchmarked b…
GGUF, GPTQ, and AWQ are the three dominant formats for running quantized large language models locally, each optimized for different hardware and use cases. GGUF, the format used by llama.cpp and its …
Apple's Mac Studio with M3 Ultra, offering up to 512 GB of unified memory at 819 GB/s, is the most practical desktop solution for running large local AI models like 70B to 400B-class quants without mu…
Nvidia's upcoming RTX Spark desktop, priced between $3,000 and $5,000, may not deliver the performance leap over the existing DGX Spark that marketing suggests, according to a skeptical analysis from …
Intel's Panther Lake processor has arrived in a shipping mini PC for the first time with the MSI Cubi NUC AI+ 3MG, featuring a Core Ultra 9 386H chip that delivers over 2,000 single-core and 17,000 mu…