Self-hosting the open-weight Qwen3.8 27B model costs $619 per full Artificial Analysis Intelligence Index run through the cheapest zero-data-retention provider, versus $67 for OpenAI's GPT-6 Luna, acc…
TensorFold 0.3.6.2 delivered decode speeds over 62 tokens per second on a single stream and 119 tokens per second across five concurrent streams running Qwen3.8-Flash-Next on a single Nvidia DGX Spark…
ZOLAK Micro Art Residence in Vilnius is building the "1st Dynamic AI Party," an AI-driven projection wall that runs entirely on local hardware and reacts live to the movement and speech of people in t…
An open-source contributor's r/LocalLLaMA post, "42x Faster Prompt Lookup Drafting in llama.cpp," reports that llama.cpp's ngram-mod speculative-decoding path can draft repeated text up to 42 times fa…
A developer running Qwen3.8-27B across two RTX 3090s on an ASUS TUF X570-Plus motherboard traced a vLLM model-loading stall of over ten minutes to PCIe link retraining on a passive x8/x8 bifurcation s…
A developer extended the Jev-style single-function LLM wrapper to support vision models by adding an `attachments` field for base64 images to the request format, then built a webcam demo that classifi…
Llama.cpp's default layer-split mode gives a dual-GPU setup the combined VRAM of both cards but runs single-stream generation at roughly the speed of one card, while tensor parallel can nearly double …
A homelab builder running Mistral 7B and DeepSeek-R1 8B locally, and Mistral Large in the cloud, is comparing GPU options for a budget local LLM machine, primarily for software debugging. The listed p…
A developer ran Qwen 3.8 27B (Q4) autonomously inside Deepseek Harness on a single RTX 3090 GPU for 3 weeks, with the resulting repository stating that everything in it was created by the model except…
A developer instrumented an RTX 3090 running a resident local AI stack — a Whisper ASR service, an Ollama embedding model, and Immich ML — with a power meter over 30 days and found the GPU averaged ro…
A developer built a minimal Python web server that turns Cerebras-hosted inference of Alibaba's Qwen 3.8 27B model into a live operating system with zero apps on disk, streaming tokens at 1,950 tokens…
Tuning vLLM 0.28.0's prefix cache raised the share of prompt tokens served from cache from 55% to 95% on a local Qwen3.8-27B coding agent, cutting average time to first token from 26-28 seconds to 7.3…
The open-source project local-deep-research reported approximately 95% on SimpleQA (n=500) and 77% on xbench-DeepSearch (n=100), which it says makes it the first fully-local open-source project to hit…
A developer resolved CUDA out-of-memory crashes running Qwen2.5-Coder-32B on a 24GB RTX 3090 by capping the context window at 8k tokens instead of 32k via a custom Ollama Modelfile, cutting latency fr…
Sustained AI inference workloads on Nvidia RTX 3090 and 4090 cards accelerate thermal paste pump-out, a failure mode in which paste migrates off the GPU die and causes clock throttling even while repo…
A developer released "Aurora," a three-shot alpine video generated in a single unbroken H3 ref2va generation on an RTX 3090, using Z-Image to draw three storyboard frames that anchor cuts at 00:02.700…
A developer extended Sebastian Raschka's GPT-2-style code from the book "Build a Large Language Model (from Scratch)" to add mixture-of-experts support and trained a 446M-parameter model with 220M act…
Running large Mixture-of-Experts local LLMs in 2026 requires at least 64GB of system RAM, ideally 128GB or more, because sparse MoE models keep rarely-used expert weights in system memory rather than …
DigitalSpaceport ran DeepSeek V4 Flash, a 284-billion-parameter Mixture-of-Experts model, on a single used RTX 3090 (24GB) plus system RAM, achieving about 10 to 11 tokens per second for chat, but the…
A forum user comparing CUDA and ROCm for local AI inference reports that ROCm issues have diminished and is considering an all-AMD build with dual R9700 GPUs by 2027, citing AMD's lower cost. Another …