A technical guide from Cast AI explains that quantization, the process of compressing LLM weights to lower-precision data types, is central to balancing throughput, memory, and inference costs, and cl…
A developer has ported ROCm 10.0 to AMD's gfx803 architecture (Polaris GPUs), enabling vLLM and llama.cpp to run on older cards like the RX 580. The port, which builds on earlier work with ROCm 6.4.4 …
A developer has launched Buttercup, an open-source project demonstrating AI agents built entirely in client-side JavaScript that run in the browser, aiming to reduce infrastructure costs by eliminatin…
NVIDIA Dynamo, an open-source inference framework, has been deployed with the vLLM backend to serve chat completion requests in both aggregated and disaggregated configurations. The deployment involve…
NVIDIA has introduced optimizations that boost local AI inference speeds by up to 1.9x, integrated directly into llama.cpp and vLLM, benefiting users of LM Studio and Ollama. The company also unveiled…
LM Studio and Ollama users control text generation through sampling settings: temperature sets randomness, top-p and top-k trim candidate tokens, and min-p, a 2024 refinement from Nguyen et al.'s pape…
A technical guide details the production deployment of on-premises large language model inference using vLLM, covering an 8-GPU node running GLM-5.2/5.3 (NVFP4 MoE) and a single L40S running gemma-4-2…
VLLM's PagedAttention technique reduces GPU memory waste in LLM serving from 60-80% to less than 4% by partitioning the KV cache into non-contiguous blocks, according to a technical analysis. The meth…
A new open-source tool called trimtab lets operators change SGLang and vLLM scheduler settings live, cutting configuration changes from a 1-7 minute redeploy to about 15 ms, with zero dropped requests…
A developer has released btp-guard, an open-source Python package that performs in-memory AST-based safety checks on code blocks generated by local LLM agents before execution. The tool parses Python,…
An engineer details the architecture and engineering discipline required to move an LLM application from prototype to production, addressing scaling, rate limits, and failure modes. The piece uses a c…
A hardware-sizing analysis by an unnamed engineer claims that two AMD Radeon PRO R9700 GPUs ($1,300 each) and a Mac Studio M3 Ultra (256GB, $6k) could outperform NVIDIA's DGX Spark (128GB, 273GB/s) fo…
A developer benchmarked three Gemma 4 deployments on a single AWS T4G GPU for under $3, comparing vLLM, JAX, and PyTorch runtimes. The project built a common harness to measure decode throughput, reve…
Shaide, a Kubernetes-native AI platform for distributed multi-model inference, has been released under the Apache 2.0 license. It uses vLLM for inference, llm-d for orchestration, KV-cache-aware sched…
A developer known as iamskeole released burrito, an open-source inference harness and evaluation suite, after running 320,192 questions across 8 seeds and processing 3.49B tokens over 1,062 GPU hours …
Axem co-founders open sourced Shaide, a Kubernetes-native AI platform for distributed multi-model inference, under the Apache 2.0 license. The platform uses vLLM for inference and llm-d for orchestrat…
On June 4, 2026, NVIDIA released Nemotron 3 Ultra, a 550 billion parameter open-weight hybrid Mamba-Attention Mixture-of-Experts model that activates about 55 billion parameters per token, designed fo…
AptAI launched a centralized hub for discovering, testing, and deploying LLM adapters, enabling one-click serverless deployment with sub-millisecond execution overhead and zero cold starts. The platfo…
Meryem Arik, co-founder of inference company Doubleword, said most companies overpay for AI inference by 2x to 5x, sometimes by an order of magnitude, because their inference stacks do not match their…
Snowflake AI Research introduced Semi-Persistence, a model-serving approach that keeps model weights in persistent CPU memory to enable fast swapping of GPU copies, reducing end-to-end sleep and wake-…