SkyDiscover-Synthesize (SkySynth), an autonomous pipeline that builds just-in-time specialized systems, synthesizes key-value stores up to 2.3× faster than Redis and FASTER, inference engines with up …
VicOne researcher Reuel Magistrado disclosed CVE-2026-86793 in the SGLang open-source LLM inference framework on September 11, 2026, a SafeUnpickler bypass that lets an attacker chain __import__ and g…
LinkedIn published details of a multi-teacher distillation pipeline that trains a 0.6B-parameter job-search ranking model roughly 8x faster, using a custom SGLang-based framework that serves teacher m…
A former B2B SaaS web developer who switched to inference engineering about a month ago published a skills map arguing the field requires only three service categories: routing and scheduling via Nvid…
A prefix-cache simulator replaying 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests failed to beat the production LRU baseline in three separate attempts, according to t…
DeepSeek AI released DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model with 552B backbone parameters, 196B additional Engram parameters, and a 1M-token context window, cutting global KV cache…
A study of seven experiments on live shared LLM-serving systems found that KV-cache timing side-channel reliability collapses under multi-tenant contention, with mean Cohen's d on a vLLM server runnin…
Next AGI released Nex-N2.5 Mini, a multimodal agentic model built on a Qwen3.5 mixture-of-experts backbone, under an Apache 2.0 license on Hugging Face, requiring two 80GB-class GPUs to run. In hands-…
Nex AGI's Nex-N2.5 Mini, the smaller model in the company's N2.5 agentic family, requires two 80GB-class GPUs such as H100s and consumed roughly 66GB of VRAM per card in testing on RunPod, according t…
On August 12th, Alibaba Cloud's Qwen team released Qwen3.8-2.4T-A95B, an open-weight mixture-of-experts model with 2.4 trillion total parameters and 95 billion active during inference, making it the f…
Production LLM traffic is dominated by repeated prompt prefixes—system prompts, chat history, and shared documents—causing inference engines to recompute identical KV cache state thousands of times da…
A developer running custom simulation software has moved from cloud-based AI coding assistance to local inference on high-end HPC equipment, using Hermes, Ollama, vLLM, and SGLang with models like Qwe…
At a recent event, speakers including Aritra Roy Gosthipaty from Hugging Face emphasized the need for India to focus on machine learning infrastructure builders rather than API-level developers, highl…
Crusoe, a vertically integrated AI infrastructure company, is hiring a Staff Applied AI Inference Engineer in San Francisco with a salary range of $215,000–260,000 per year, which sits 8% above the $2…
AMD and the veRL project released a production-ready reinforcement learning post-training container for AMD Instinct GPUs, supporting MI300 and MI355 series on ROCm, with AITER-accelerated vLLM and SG…
ModelBench's MiniCPM5-2B, a 2-billion-parameter dense language model from the Tsinghua University spinout, can run locally with SGLang or llama.cpp, requiring roughly 4GB of VRAM for weights but up to…
OpenBMB released MiniCPM5-2B, a 2.52B-parameter dense causal language model with a native 131,072-token context, averaging 53.9 across 34 benchmarks and outperforming Qwen3.5-4B's 51.1, with notable s…
Superlinked has released the Superlinked Inference Engine (SIE), an open-source server designed to consolidate the many small models used in AI agent workflows into a single deployment, addressing wha…
The Institute of Foundation Models (IFM), the frontier lab launched by MBZUAI in May 2025, released K2 Horizon, a fleet of six Apache 2.0-licensed open-source models ranging from 0.9B to 375B paramete…
A technical guide from Cast AI explains that quantization, the process of compressing LLM weights to lower-precision data types, is central to balancing throughput, memory, and inference costs, and cl…