A developer argues that scaling up large language models alone does not make them smarter, and instead advocates for a practical blueprint combining prompt engineering, decoding parameters, retrieval-…
Kvcachescope, a logical memory profiler and state inspector for PagedAttention inference engines such as vLLM and SGLang, exposes KV cache leaks that standard GPU profilers like nvidia-smi miss by hoo…
A developer detailed how to build and run vLLM's Rust frontend on an AWS EC2 G5g instance with Graviton2 and an NVIDIA T4G GPU. The tutorial highlights that vLLM now requires a Rust toolchain for sour…
A developer has released Sluice, an open-source QoS-aware capacity governance proxy for self-hosted LLM inference. Sluice sits in front of vLLM deployments to enforce tenant-based service tiers, prese…
VLLM outperforms Ollama by nearly 20x in throughput at high concurrency, according to benchmark tests running Llama 3.1 8B on an NVIDIA A100 40GB, with vLLM peaking at 793 tokens per second versus Oll…
A developer's comparison of vLLM and Ollama for LLM serving in 2026 shows that while Ollama is simpler for single-user local use, vLLM outperforms it dramatically under concurrency, with throughput up…
A practitioner with 18 months of experience building production agentic systems reports that deterministic guardrails, structured tool interfaces, and trajectory evals matter more than the LLM itself,…
Z.ai launched GLM-5.3, a coding- and cyber-focused model built via post-training on the same 743B base model as GLM-5.2, achieving scores of 28.3 on Terminal Bench 3.0, 66.9 on DeepSWE, 28.5 on Agents…
DeepSeek released DeepSeek-V4-Pro-0813, the production successor to its V4 Pro preview, under an MIT license with open weights, adding a DSpark speculative decoding module and stronger agentic benchma…
In a new series on LLM inference, the author explains the core concepts behind running language models in production, starting with the fundamental division between prefill and decode. The series cove…
A new guide outlines strategies for scaling AI revenue while controlling cloud costs, covering hardware accelerators from NVIDIA, Groq, and Cerebras, inference engines like vLLM and TensorRT-LLM, and …
An engineer successfully ran Google's Gemma 4 E2B model on AWS EC2 G5g, a Graviton2 (aarch64) instance with an NVIDIA T4G GPU, achieving 43.1 tokens per second after patching vLLM. The deployment requ…
A developer's field guide benchmarks DFlash, a block-diffusion speculative decoding method, on 30B-scale models, arguing that tokens per second can mislead. The guide highlights DFlash's architectural…
Liquid AI released LFM2.5-VL-3B, a 3.1B-parameter vision-language model for on-device deployment, averaging 69.4 across 28 vision benchmarks, matching InternVL-3.5-4B and 0.7 points behind Qwen3.5-4B.…
An audit by computer science researchers finds that NVIDIA's Blackwell Ultra GPU (B300) nominally supports INT8 tensor-core compute at a 30:1 FP8-to-INT8 ratio, but the format is undeployable by defau…
A developer detailed the architectural requirements for building a Linux-native desktop client for LLM-based coding assistants, emphasizing the need for multi-process designs, Rust backends, and deep …
Doable, a self-hosted AI app builder for teams, has been released under an MIT license, allowing users to generate, deploy, and host AI-powered applications on their own infrastructure with multi-tena…
Security audits of enterprise Model Context Protocol (MCP) deployments reveal a 67% vulnerability rate to indirect prompt injections and cache poisoning, with HijackKV attacks achieving a 94% Targeted…
Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), a 2.4T-parameter mixture-of-experts model with 95B activated parameters per token, designed for long-context reasoning and agenti…
Alibaba released the open-source Qwen3.8-2.4T-A95B model weights on August 12, a 2.4-trillion-parameter sparse mixture-of-experts model with about 95 billion active parameters per token, making it the…