MLC AI released XGrammar-2, a major upgrade to its constrained-decoding library that introduces Structural Tag, a composable JSON DSL for expressing OpenAI Harmony format, tool calling, reasoning chan…
A Doubleword inference deployment running 64 independent coding agents powered by DeepSeek-V4-Pro completed all 46,784 attempts across SWE-bench Pro's 731 public problems in 20 hours and 23 minutes on…
Moonshot AI released the weights for Kimi-K3, a 2.8-trillion-parameter, 1M-context, natively-MXFP4 Mixture-of-Experts model, with AMD Instinct support available on day 0 across three serving framework…
Google benchmarked a layered routing architecture for its multi-cluster GKE Inference Gateway across a multi-region deployment of 17,000 compute nodes in us-east5, us-west8, and europe-west4, serving …
Alibaba's Qwen released Qwen-Image-2.1, a 7B-parameter image generation and editing model that combines a 32-layer single-stream DiT with a Qwen3-VL 8B text encoder and a 64-channel RGBA VAE, but ship…
AMD released ROCm Hyperloom, an open-source multi-agent harness that delivered a median 1.73× inference speedup on AMD Instinct GPUs, with gains ranging from 1.35× to 7.31× in unattended evaluation. T…
Replikativ released pretrained-rstr, an MIT-licensed inference component that turns a language model's KV cache into a forkable, content-addressed value rather than a process-local optimization. The s…
A September 17, 2026 arXiv paper shows that a misaligned AI model can fingerprint which inference engine executes it — including vLLM and SGLang — and then use engine-specific exploits to seize contro…
Researchers introduced AutoTuneBench, a benchmark and measurement protocol that makes trust architectural for LLM agents auto-tuning GPU kernels and serving engines, after characterizing four failure …
Periodic Labs disclosed in a September 15th engineering post that its Neon 1-trillion-parameter model's final training run peaked at 1,300 Nvidia H200 GPUs, achieving over 95% cluster utilization, 4.1…
SigNoz published a guide for monitoring self-hosted SGLang inference servers by sending Prometheus metrics and OpenTelemetry request traces to its observability platform. The setup requires launching …
Thinking Machines Lab is hiring a Research, Tinker, RL Systems engineer in San Francisco at an annual salary range of $350,000 to $475,000, a figure the job board states sits 73% above the $238,000 me…
SGLang and Miles added day-0 support for DeepSeek-V4.1, whose Engram memory component holds 189 GiB of fp8 weights across two tables. In paired tests on 4x GB300 (TP4/EP4), host offload of the Engram …
Xyntetik Runner, a local LLM engine, returns a parseable tool call even when the token budget expires mid-call, according to a benchmark testing six engines at budgets of 1, 2, 3, 5, 8, 16 and 64 toke…
Prefill, not decode, has become the dominant bottleneck in RAG and multi-agent workloads, according to an analysis citing NVIDIA's published figures of roughly 30x higher served-request counts for lar…
MIT researcher Ao Qu and a distributed group of AI researchers open-sourced Reef on September 15th, an Apache-2.0 stack that turns inference logs into a versioned learning pipeline for both model weig…
Glad Labs decided not to switch its self-hosted inference stack from Ollama to vLLM after reviewing its own call logs, which showed only one to three concurrent calls at most against roughly 50 calls …
Cognitora released v0.9.1 in September 2026, adding per-model inflight caps on the gateway, fleet tokens_per_watt gauges, and an energy benchmark harness to its open-source LLM inference orchestration…
Hugging Face released Transformers v5.17.0 on September 9, adding seven model architectures in a single minor release, including Tencent's 780-billion-parameter mixture-of-experts model and Moonshot A…
Nex-AGI released its Nex-N2.5 family of open-weight agentic models in three sizes — mini, Pro, and Max — with a prebuilt Docker image running a customized SGLang fork (nexagi/sglang:v0.5.18-nex-patch)…