Google's Gemma 4 E2B model serves efficiently on a single TPU v6e chip, achieving 213 tok/s for a single user and scaling to ~2,200 output tok/s across concurrent streams, while its QAT variants fail …
Researchers at Moonshot AI introduced Kimi Linear, a hybrid linear attention architecture that outperforms full attention across short-context, long-context, and reinforcement learning scaling regimes…
A developer has created tpu-management, a Claude Code skill and MCP server that automates the full lifecycle of serving Gemma 4 on Google Cloud TPUs, from zone selection and VM creation to benchmarkin…
AMD introduced ROCm Infera, an open-source distributed inference reference solution for large-scale deployments, claiming it can improve goodput per GPU for agentic AI workloads by up to 2.6× in inter…
LLMrPro, a new self-hosted LLM router, pools users' own machines as inference workers with automatic per-tier cloud fallback, exposing a single OpenAI-compatible API. The open-source project, released…
Modelplane, a new open-source control plane for fleet-level AI inference, launched v0.1 to solve the problem of managing inference across hundreds of clusters on any infrastructure. The API separates …
Feyn Labs shipped SQRL on July 19, a text-to-SQL model that inspects databases with read-only probes before writing queries, achieving 70.6% execution accuracy on BIRD Dev and edging past Claude Opus …
Moonshot AI's Kimi K3, a 2.8-trillion-parameter model that topped the Frontend Code Arena leaderboard on day one, releases its full open weights on July 27. The MXFP4 weights require approximately 1.4…
Moonshot AI's Kimi K3 model matches Opus 4.8 on intelligence benchmarks while costing ~70% less, with a 1M-token context window and open weights. The API is OpenAI-compatible, enabling easy integratio…
Ollama and vLLM are both open-source inference stacks for running large language models, but they serve fundamentally different use cases: Ollama is a model manager optimized for a single developer on…
LLM Inspector, a new open-source CLI tool from developer Helal Saoudi, analyzes live LLM inference processes to show exactly how GPU memory is used by weights, KV cache, and workspace, then projects o…
Netflix's AI Platform team built an in-house LLM serving stack, running the full pipeline from model deployment through inference inside its existing production environment. The team selected vLLM as …
Thomas, a machine learning engineer with five years of production deployment experience, released Coloma as an open-source verification and tuning layer for self-hosted large language models. The tool…
Moonshot AI released Kimi K3, a 2.8 trillion parameter open Mixture-of-Experts model with a 1 million token context window. The model introduces innovations like Kimi Delta Attention for faster decodi…
Tinfoil announces cached prompt pricing in its Inference API, a feature that reduces compute for eligible requests by caching recently processed inputs, but the company warns that caching introduces t…
Moonshot AI released Kimi K3, a 2.8-trillion-parameter open Mixture-of-Experts model with native vision and a 1-million-token context window, claiming it as the world's first open 3T-class model. The …
VLLM v0.25.0, released July 11, deletes the original PagedAttention implementation and makes Model Runner V2 the default execution backend for all dense models, delivering a 56% throughput improvement…
NVIDIA released the Nemotron 3 Embed collection of open embedding models, led by an 8B model that ranks #1 on the RTEB leaderboard with a score of 78.5%, alongside efficient 1B variants for production…
A five-engineer team on Sonnet 4.6 saw a $4,800 monthly bill for Claude Code sessions, with only $960 attributed to the model itself and $3,840 coming from inference engineering — the layer between th…
Veta, an AI agent that QA-tests Android apps using a swarm of autonomous sub-agents, runs 100% of its AI inference on AMD GPUs via Fireworks AI on AMD Instinct or self-hosted vLLM on ROCm. The system …