Qwen 2.5-32B MoE, a Mixture-of-Experts model activating only about 3B parameters per token, runs efficiently on a single RTX 3090 with 18-22GB VRAM usage and high tokens per second, outperforming stan…
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 using Unsloth's UD-IQ4_NL_XL quantisation achieved up to 140 tokens per second for generation and over 3,300 tok/s for prompt processing with a…
Running Qwen locally via Ollama or vLLM with a local Python environment avoids cloud data exposure and token limits, enabling iterative work on large datasets. Qwen2.5-Coder (7B) on an RTX 3090 genera…
Nvidia's NVLink bridge boosts prompt processing by 30% and FSDP training by nearly 3x on dual RTX 3090s, but token generation and DDP training show no benefit, according to tests by a developer runnin…
James O'Beirne's GitHub repo 'local-llm' documents his experience building and running large language models on local hardware, covering configurations from a $2k setup with two RTX 3090s to a $40k bu…
A developer tested the Bonsai-27B model on a single RTX 3090 using a Q4_K_M GGUF quant from prism-ml, achieving ~28 tok/s at 4K context and ~19 tok/s at 16K. The model excelled at structured extractio…
Running a 70B large language model locally in 2026 can cost as little as $250 using two used AMD Instinct MI50 GPUs, according to the r/LocalLLaMA community. The cheapest quiet option is a Strix Halo …
A new measurement study by an independent developer using the open-source HomeLab Monitor tool found that running local large language models on a single NVIDIA RTX 3090 can cost as little as €0.046 p…
Open-source coding agents can now run entirely on local consumer hardware such as a 2020-era NVIDIA RTX 3090, eliminating usage-based pricing from AI vendors. The setup uses a quantized Gemma4 model s…
Two used RTX 3090s offer 48GB of VRAM for about half the cost of a single RTX 5090, enabling local 70B LLM inference at 14-21 tokens/sec, while the 5090's 32GB cannot run a quality 70B model. For mode…
A team of researchers achieved a 9.73 wECR on the public validation leaderboard of the ICIP 2026 Grand Challenge on Extreme In-the-Wild License Plate Super-Resolution (XLPSR) using a Hybrid Attention …
A developer repurposed an old small-form-factor PC named 'poppy' into a dedicated machine for local LLM training, upgrading its case and power supply to accommodate future multi-GPU setups. The projec…
Giles Thomas completed building and training a GPT-2 small model from scratch using JAX, achieving a test loss of 3.418784, outperforming both his PyTorch model (3.538161) and the original GPT-2 small…
Glad Labs fixed a GPU pinning issue where LiteLLM 1.89.2's global api_base override prevented per-model routing, causing vision tasks to cold-load onto the wrong GPU. The team also hardened content gu…
Jamesob published a guide on building a local system to run state-of-the-art large language models, detailing hardware configurations ranging from $2k to $40k. The setup uses multiple RTX GPUs and PCI…
Outpost VFX achieved 8x faster AI model training for visual effects face replacement by using AWS multi-GPU EC2 P5 instances, overcoming single-GPU bottlenecks that previously caused week-long delays.…
Owner-submitted benchmarks show that memory bandwidth, not TFLOPS, determines local LLM generation speed. An AMD RX 7900 XTX with 122 TFLOPS generates text at 39 tokens per second, while an older RTX …
On May 16, 2026, llama.cpp merged Multi-Token Prediction (MTP) support, enabling 1.7x to 2.4x faster local inference for Qwen3.6 27B models with no accuracy loss or extra downloads. The MTP head is em…
TurboPrefill introduces intra-prompt pipeline scheduling for multi-GPU prefill, achieving up to 2.7× faster performance than llama.cpp on Llama-3-70B by overlapping GPU stage execution. The PoC shows …
A replication of the 2026 paper 'Prompt Injection as Role Confusion' on a single consumer GPU confirms that style-based prompt injection attacks work, but causal tests using activation steering and pa…