Nvidia's latest demo shows that optimizing the inference stack, not the model weights, is the key to performance, achieving 4.2× higher throughput at half the latency on identical H100 hardware with L…
Z.ai and NVIDIA engineers trained GLM-5.2, a 744B-parameter mixture-of-experts model quantized to 4-bit NVFP4, with reinforcement learning to play Super Mario Bros., overcoming arithmetic, distributed…
Liquid AI released DSpark draft model checkpoints for three LFM2.5 models, enabling speculative decoding that speeds up inference by up to 3.18x on GPUs and 2.87x on-device without changing output qua…
SGLang, the open-source inference project associated with Ying Sheng and Banghua Zhu, published deployment recipes for Alibaba's Qwen3.8-27B model, enabling single-GPU serving with NVFP4 quantization …
Liquid AI released DSpark draft model checkpoints for three LFM2.5 models, delivering up to 3.18x faster decoding on an H100 and up to 2.87x on an M4 Max MacBook Pro without changing model outputs. Th…
Liquid AI released DSpark draft model checkpoints for its LFM2.5 family, claiming up to 3.18x throughput improvement on GPU and up to 2.87x on-device, with day-one support for llama.cpp and SGLang. Th…
A developer's four-month home-lab test found that self-hosting open models costs about 7× more than using hosted APIs, cutting their bill from roughly $3,050 to $420 per month by switching to API acce…
Alibaba's Qwen 3 27B, a dense open-weight multimodal model, can be run locally using DeepSeek Harness, achieving agentic performance close to Claude 4.5 on benchmarks. On a two-node NVIDIA DGX Spark c…
A vLLM RFC from the llm-d team warns that disaggregated inference deployments, where prefill and decode run on separate schedulers, can trigger recomputation-based preemption inside the decode instanc…
Inco AI released DFlash 2, a parallel speculative decoding technique that delivers over 20% more output from every verification pass with around 1% added cycle latency, achieving 2.7–3.4× throughput o…
Philip Kiely's new book, 'Inference Engineering,' is now available as a digital download, offering a comprehensive guide to the technologies and techniques powering AI inference across runtime, infras…
RadixArk released Miles v0.1, an open-source framework for aligning AI training and inference, on GitHub. The company, founded by Ying Sheng and Banghua Zhu, aims to commercialize open-source infrastr…
Radix Ark released Miles v0.1, a full-stack production-ready system for frontier post-training, designed to make large-scale reinforcement learning accessible to researchers and developers. The system…
A Kubernetes engineer's guide compares LLM serving engines, arguing that vLLM is not the only option and that choosing an engine is a critical production decision. The post highlights vLLM's strengths…
A developer detailed the process of running Qwen3.8-27B on dual RTX 3090s without NVLink under WSL2, achieving 170-210 tok/s on code/JSON. The setup required specific CUDA 13.0 toolchain, SGLang 0.5.1…
DeepSeek released the DeepSeek-V4-Pro-0813 model weights on August 13, 2026, and Fixstars deployed it to a single-node NVIDIA B300 x8 machine within 24 hours, achieving 17,478 tokens per second throug…
Anthropic's Claude Code can now run on-premises with AMD Instinct GPUs, serving GLM 5.2 at full quality via SGLang and LiteLLM, eliminating cloud API dependency and per-token costs. The setup uses an …
A base-2 vs. base-e log-sum-exp mismatch in SGLang's merge v2 API silently corrupts attention outputs for models using MLA with the FlashInfer backend when radix cache hits exceed 8k, affecting DeepSe…
Adore LLC released Balto Speedrunner, a Windows app that turns Qwen 3.8 27B into a local coding agent for a single RTX 5090, achieving up to 300 tok/s on chat prompts and 150 tok/s during coding runs.…
Alibaba released Qwen3.8-27B, a 27.78-billion-parameter open-weights multimodal model under Apache 2.0, with a 262,144-token native context window and configurable reasoning. Vendor-reported benchmark…