Amazon launched model caching for Amazon SageMaker Inference on HyperPod, a feature that pre-loads model weights and container images onto cluster nodes so inference pods can read from local NVMe stor…
Bedrock Robotics posted an internship opening for a 2027 Onboard Infrastructure Engineer, ML Inference role based in San Francisco, CA, on a hybrid schedule, with pay not published. The intern will in…
Neural Nova published GPU optimization benchmarks for LLM workloads, reporting throughput and cost gains across four models running on vLLM. Qwen3-235B-A22B on 8× NVIDIA H100-80GB posted +138.7% token…
Inferact, founded by the creators and core maintainers of vLLM, is hiring University of Waterloo co-op students for an on-site Inference Engineering co-op in San Francisco, with pay not published. The…
A developer published a step-by-step guide for deploying Google's 2B Gemma 4 model to Cloud Run with an NVIDIA L4 GPU, using vLLM for serving and a single-file Python MCP server to stage weights, depl…
NVIDIA's NIM (NVIDIA Inference Microservices) stack delivers 2.5x higher concurrency for the Nemotron 3 Ultra model through full-stack optimizations including PagedAttention-based KV cache management,…
A prefix-cache simulator replaying 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests failed to beat the production LRU baseline in three separate attempts, according to t…
DeepSeek AI released DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model with 552B backbone parameters, 196B additional Engram parameters, and a 1M-token context window, cutting global KV cache…
A study of seven experiments on live shared LLM-serving systems found that KV-cache timing side-channel reliability collapses under multi-tenant contention, with mean Cohen's d on a vLLM server runnin…
TRL v1.14 shipped LoRA support in its AsyncGRPOTrainer via PR #7017, letting the asynchronous trainer train a rank-1 LoRA adapter and sync only that adapter to vLLM instead of the full model. A Huggin…
On August 12th, Alibaba Cloud's Qwen team released Qwen3.8-2.4T-A95B, an open-weight mixture-of-experts model with 2.4 trillion total parameters and 95 billion active during inference, making it the f…
On August 12, 2026, Alibaba's Qwen team released Qwen3.8-2.4T-A95B, the first open-weights Qwen-Max-class model with 2.4 trillion total parameters and 95 billion activated per token, targeting agentic…
Kubernetes does not accelerate LLM inference but orchestrates GPU clusters by placing, monitoring, replacing, and scaling inference servers, according to an article on AI workload management. The piec…
Google's open-source, JAX-native post-training library Tunix now ships an agentic RL trainer built around asynchronous rollouts and a decoupled producer-consumer pipeline, aiming to eliminate idle TPU…
Ray 2.55 adds official TPU support with atomic gang scheduling for TPU slices, aiming to cut GPU costs by enabling Ray-based workloads to run on Google's TPUs. Google's motive is to boost external TPU…
A German university hospital has deployed DeepSeek-R1, served through vLLM, in a sealed radiology environment with no outbound internet connectivity, marking one of the clearest examples of open-weigh…
Production LLM traffic is dominated by repeated prompt prefixes—system prompts, chat history, and shared documents—causing inference engines to recompute identical KV cache state thousands of times da…
Sentry's AI security research team, led by Armend Gashi and Redon Gashi, presented at DEF CON 34 a new tool called infreerence that demonstrates how adversaries exploit exposed self-hosted inference s…
A developer running custom simulation software has moved from cloud-based AI coding assistance to local inference on high-end HPC equipment, using Hermes, Ollama, vLLM, and SGLang with models like Qwe…
Tencent's Zhuque Lab released AI-Infra-Guard, an open-source security scanner for AI systems that fingerprints services such as Ollama, vLLM and ComfyUI, checks them against more than 1,600 known CVEs…