Amazon SageMaker AI now supports bidirectional streaming for real-time inference, enabling developers to build voice applications that process audio and return transcriptions simultaneously over a sin…
Researchers at UC San Diego, led by Hao Zhang, have successfully implemented a diffusion-style speculative decoding method called DFlash on Google TPUs, achieving an average 3.13x increase in tokens p…
This article compares three dominant tools for local LLM inference in 2026: Ollama, llama.cpp, and vLLM. Ollama is recommended for personal, non-technical use due to its ease of setup, while llama.cpp…
PyTorch 2.11 now enables direct installation of CUDA-enabled PyTorch wheels on aarch64 Linux from PyPI, eliminating the need for custom package indexes and workarounds that previously complicated depl…
Hippocratic AI partnered with Modular to integrate the MAX framework into its inference pipelines, achieving sub-500ms mean time to first token and approximately 30% faster P99 end-to-end latency for …
Researchers have developed Speculative KV coding, a lossless compression method that reduces the size of large language model KV caches by up to approximately 4x using a smaller, faster predictor mode…
Here is a 2-3 sentence factual summary of the article: The article describes the process of migrating an online reinforcement learning (RL) training system from the vLLM V0 engine to the V1 rewrite, …
A merged Qwen multimodal chat template has been released, combining fixes from allanchan339 and froggeric to address multiple issues with Qwen model chat formatting. The template adds support for a de…
A developer deployed the Qwen 3.6-35B-A3B FP8 mixture-of-experts model (3 billion active parameters) on a DGX Spark GB10 system using vLLM, achieving inference with a 262,144-token context window and …
Shepherd Model Gateway (SMG) has disaggregated all CPU-bound workloads from GPU inference in large language model serving, moving tokenization, detokenization, and parsing into a dedicated Rust gatewa…
A 22% throughput improvement and up to 97.5% cache hit rate is achievable on LLM serving clusters by routing requests to GPUs that already hold their token prefixes in KV cache, rather than using roun…
Instructions for using a specific chat template file (`chat_template_gemma_large_fixed.jinja`) with the vLLM inference engine to enable compatibility with the Gemma 4 model and OpenCode. It directs us…