A new arXiv paper (2608.28003v1) proposes a layer bit-width allocation method for Gemma-3-1B that maximizes latency reduction under a quality-degradation constraint, achieving an 11.0% latency reducti…
Oxmiq Labs presented high-bandwidth flash (HBF) as a capacity tier for AI inference at Hot Chips 2026, claiming it delivers 8 to 16 times the capacity of HBM at the same cost. The company detailed HBF…
SemiAnalysis announced AgentX 1.0, the world's first fully open source multi-turn agentic coding inference benchmark at 1 million context, released under Apache 2.0, after spending more than $3M build…
Nvidia's latest demo shows that optimizing the inference stack, not the model weights, is the key to performance, achieving 4.2× higher throughput at half the latency on identical H100 hardware with L…
Neon AI Gateway, powered by Databricks Foundation Model APIs, achieves fast open-weight model inference through optimizations like continuous batching, KV-cache paging, and prompt caching, which boost…
Inco AI released DFlash 2, a parallel speculative decoding technique that delivers over 20% more output from every verification pass with around 1% added cycle latency, achieving 2.7–3.4× throughput o…
Philip Kiely's new book, 'Inference Engineering,' is now available as a digital download, offering a comprehensive guide to the technologies and techniques powering AI inference across runtime, infras…
A Kubernetes engineer's guide compares LLM serving engines, arguing that vLLM is not the only option and that choosing an engine is a critical production decision. The post highlights vLLM's strengths…
A new guide outlines strategies for scaling AI revenue while controlling cloud costs, covering hardware accelerators from NVIDIA, Groq, and Cerebras, inference engines like vLLM and TensorRT-LLM, and …
Engineers at an unnamed company cut inference cold start times from minutes to seconds by profiling their startup sequence and finding that 90% of the delay came from moving large files. They reduced …
Engineers are integrating large language models into real-time data pipelines, facing challenges such as state synchronization and KV cache transfer bottlenecks. Optimizations using TensorRT-LLM and a…
NVIDIA released Nemotron-3.5-Lightning, a 30B mixture-of-experts model with 3B active parameters, on the same day Modelplane, an open-source fleet-level control plane for inference, achieved zero-day …
Cursor released an open-source megakernel that delivers a 41% increase in tokens per second, potentially saving billions of dollars at scale, according to a post by Stuart Sul, who leads the team behi…
Context management is becoming a critical infrastructure layer for production AI systems, with KV-cache optimization and contradiction-aware retrieval emerging as two distinct but connected challenges…
Together AI is hiring a Forward Deployed Engineer (Inference & Post-Training) in Singapore, requiring Mandarin proficiency and permanent residency or citizenship. The role involves optimizing inferenc…
AI applications run in data centers, and their performance depends on a five-layer stack of software, hardware, and infrastructure, not just the model. The article explains that training and inference…
NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing u…
HexCore, a low-latency paged KV cache allocator for LLM inference written in C++20 and CUDA, has been released under the Apache License 2.0 by Rasuljanov Muhammadali. The CPU-side allocator and relate…
Together AI introduces ThunderAgent, a system for high-throughput agentic inference that achieves up to 2.5× higher single-node throughput and 2.4× speedup on an 8-node cluster with near-linear scalin…
Cursor and Together AI have deployed real-time, low-latency inference for Cursor's AI-powered coding platform using NVIDIA Blackwell GB200 NVL72 and HGX B200 hardware, achieving predictable worst-case…