InferCrane, an open-source inference operations platform, launched with a stable endpoint for self-hosted AI inference, supporting vLLM, SGLang, LiteLLM, and other OpenAI-compatible endpoints. The Apa…
A practical guide from an unnamed author outlines a four-layer framework for building productive local AI stacks with small language models (1B–14B parameters), naming Ollama, LM Studio, llama.cpp, an…
A developer has introduced Gubernator, a single-binary container orchestrator written in Go, which can deploy a complete MLOps suite including JupyterLab, MLflow, MinIO, and Ollama in under 60 seconds…
A developer documented a runbook for deploying the Qwen 3.8 Flash Next model on a single NVIDIA Blackwell GPU using a specialized vLLM runtime. The deployment supports both Docker and native systemd t…
A developer detailed how they deployed self-hosted open-weight LLMs for a fintech client whose compliance team banned external AI APIs, running a 70B-class model on two GPU boxes for document Q&A, sup…
A developer's field notes reveal that the 'OpenAI-compatible' API standard is a spectrum, not a boolean, with at least 12 silent breakages when swapping LLM providers, including discarded parameters l…
Superlinked Inference Engine (SIE) and vLLM serve inverse purposes: vLLM optimizes token throughput for a single large language model across multiple GPUs, while SIE consolidates many smaller speciali…
Tencent's 770B-parameter Hy4 preview model, a Mixture-of-Experts architecture with 49B activated parameters per token, is now deployable locally via prebuilt Docker images for vLLM and SGLang, requiri…
Four reproducible parser failures in vLLM 0.26.0, 0.27.1, and 0.28.0 can drop or garble tool calls and reasoning content while still returning HTTP 200, according to the Ingot team's tests on CPU with…
A 2× faster attention kernel yields only 0.66% end-to-end speedup on a 1,024-token prompt with a 128-token answer under vLLM's --goodput ttft:500 tpot:50 promise, according to a seven-part technical s…
InferCrane, an open-source inference operations platform, released v1.0.0-rc.1, a public beta CLI and SDK that deploys, operates, and safely evolves self-hosted AI inference behind a stable OpenAI-com…
A critical vulnerability in vLLM's tool-call parser, CVE-2025-9141, allows model-generated arguments to be passed directly to eval(), enabling arbitrary code execution. The flaw highlights a broader p…
Oxmiq Labs presented high-bandwidth flash (HBF) as a capacity tier for AI inference at Hot Chips 2026, claiming it delivers 8 to 16 times the capacity of HBM at the same cost. The company detailed HBF…
Google Cloud has introduced native vLLM TPU support for embedding inference, targeting production long-context retrieval with Qwen3-Embedding-8B and Qwen3-VL-Embedding-8B. The engineering effort addre…
Researchers propose ExFold, a training-free expert-folding framework that accelerates both prefill and decode phases of Mixture-of-Experts (MoE) inference by projecting excluded expert contributions o…
South Korean chip startup Rebellions is adopting an open source-first software strategy, including vLLM, OpenShift, and PyTorch, to help telecom carriers quickly build AI stacks with its AI inferencin…
Z.ai released GLM-5.3-Flash, a 320B-total-parameter mixture-of-experts model with 18B active parameters, a 1,048,576-token context window, and native image and video input, under an MIT license with w…
Google engineers reported a 4.7x faster prefill and 3.1x faster decode for Qwen 3.5-397B-A17B on Ironwood TPUs between April and June, achieved by using data parallelism for attention layers and exper…
Alibaba released the model weights for Qwen3.8-Flash-Next, a multimodal mixture-of-experts model with 176B total parameters, as a preview of the upcoming Qwen4 architecture. NVIDIA provides Day 0 func…
Alibaba released the model weights for Qwen3.8-Flash-Next, a multimodal mixture-of-experts model with 125B parameters plus 51B N-gram embeddings and 6B activated per token, as a preview of the Qwen4 a…