Hetzner Online GmbH launched its Inference Experiment, a free inference API for open-weights small language models hosted on its infrastructure, one week ago. The company reported that small models li…
Rust SDK rmcp 3.x now tracks the stable 2026-07-28 MCP spec revision and offers compile-time schema checking and single-binary distribution, making Rust a more reliable choice for MCP servers that hav…
Pr-sage, an open-source AI pull request reviewer, reports that 89% of merged PRs received no comments, achieving its goal of eliminating review noise. The tool, available as a CLI, GitHub Action, and …
Researchers released ArguLens, an open-source automated essay scoring system that decomposes scoring into three components: a discourse-move classifier, a LightGBM scorer, and a label-aware feedback g…
A developer's guide explains that tokens per second (tok/s) benchmarks for local LLMs can be misleading, as the same model can show vastly different speeds depending on concurrency. The article highli…
A developer detailed a rule-of-thumb method for estimating GPU memory requirements for AI models, using Qwen2.5-7B-Instruct-AWQ as an example. The method sizes weights and KV cache from a model's spec…
Inco AI released DFlash 2, a parallel speculative decoding technique that delivers over 20% more output from every verification pass with around 1% added cycle latency, achieving 2.7–3.4× throughput o…
Philip Kiely's new book, 'Inference Engineering,' is now available as a digital download, offering a comprehensive guide to the technologies and techniques powering AI inference across runtime, infras…
Meta's Llama 3.1 70B model can now run on a single 24GB consumer GPU using GGUF or EXL2 quantization, achieving 5-10 tokens per second on an RTX 3090, according to a deployment guide. The guide recomm…
A Kubernetes engineer's guide compares LLM serving engines, arguing that vLLM is not the only option and that choosing an engine is a critical production decision. The post highlights vLLM's strengths…
The KV cache has no standard ABI, with vLLM's FlashAttention backend alone reporting its cache shape as a four-dimensional tensor that varies by backend, attention variant, and model family, complicat…
The vLLM ecosystem is shifting from batch-oriented inference to disaggregated prefill/decode serving to handle agentic traffic, a theme central to the first vLLM Conference scheduled for August 24-26,…
A new native installer brings vLLM and ROCm 7.15 (TheRock) to AMD Radeon RX 6000 (RDNA2) GPUs on Windows 11, achieving 26 TFLOPS FP16 and 58-62 tok/s on Qwen3.5-4B, verified on an RX 6750 XT 12 GB. Th…
A developer has released a fork of GenOffice, the free open-source AI office suite, that removes the cloud-account requirement and instead connects to any OpenAI-compatible endpoint, including local s…
A developer detailed the process of running Qwen3.8-27B on dual RTX 3090s without NVLink under WSL2, achieving 170-210 tok/s on code/JSON. The setup required specific CUDA 13.0 toolchain, SGLang 0.5.1…
A user running RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic via vLLM in Docker reports word-merging and grammar issues in a marine-domain question-generation application, such as 'upcomingbunkering …
A preprint on arXiv (2608.14624) describes CacheScout, a runtime layer for multi-agent language model servers built on vLLM, that learns agent execution transitions to guide cache eviction and prefetc…
A developer detailed their OpenCode setup for running local AI models, centered on a customized llama.cpp configuration. The setup leverages reasoning-effort levels for models like Qwen 3.8, DeepSeek …
CacheScout, a new KV-cache runtime layer for multi-agent LLM serving developed by researchers, improves KV-cache hit rate by 10-18 percentage points, reduces mean time-to-first-token (TTFT) by 18-45%,…
Cisco's Foundation AI team released Antares-1B, a 1 billion parameter open-weight model that localizes software vulnerabilities in code repositories and runs on a single GPU, consuming under 5GB of VR…