Mingxin FX100 achieves 90% line-rate utilization on a single 100GbE port, delivering approximately 11.25 GB/s effective bandwidth in measured tests. This eliminates the network as a primary bottleneck…
The PyTorch Foundation, which expanded into a multi-project foundation in April 2025, now hosts six projects including PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors. PyTorch released version …
A technical guide to running large language models locally in 2026 provides hardware math, quantization tradeoffs, and benchmarks of five inference engines, with case studies from the author's 16GB Ap…
Synthetic LLM inference benchmarks misrepresent production performance because they use fixed prompt lengths, steady request rates, and single-model hardware, while real traffic is bursty and variable…
Hugging Face retired its own Text Generation Inference (TGI) server in December 2025, directing users to SGLang, a tool that began as a UC Berkeley research paper in 2023 and by 2026 was running on ov…
Rondine, an open-source hardware-aware control plane, detects the host machine, recommends fitting models, builds an optimized serving plan, downloads weights, and starts an OpenAI-compatible local se…
A developer built a secure infrastructure dashboard using React 18, Tailwind CSS v4, TypeScript, and on-premise LLM isolation to meet strict data residency requirements. The architecture ensures data …
Google's Gemma 4 E2B model serves efficiently on a single TPU v6e chip, achieving 213 tok/s for a single user and scaling to ~2,200 output tok/s across concurrent streams, while its QAT variants fail …
Researchers at Moonshot AI introduced Kimi Linear, a hybrid linear attention architecture that outperforms full attention across short-context, long-context, and reinforcement learning scaling regimes…
A developer has created tpu-management, a Claude Code skill and MCP server that automates the full lifecycle of serving Gemma 4 on Google Cloud TPUs, from zone selection and VM creation to benchmarkin…
AMD introduced ROCm Infera, an open-source distributed inference reference solution for large-scale deployments, claiming it can improve goodput per GPU for agentic AI workloads by up to 2.6× in inter…
LLMrPro, a new self-hosted LLM router, pools users' own machines as inference workers with automatic per-tier cloud fallback, exposing a single OpenAI-compatible API. The open-source project, released…
Modelplane, a new open-source control plane for fleet-level AI inference, launched v0.1 to solve the problem of managing inference across hundreds of clusters on any infrastructure. The API separates …
Feyn Labs shipped SQRL on July 19, a text-to-SQL model that inspects databases with read-only probes before writing queries, achieving 70.6% execution accuracy on BIRD Dev and edging past Claude Opus …
Moonshot AI's Kimi K3, a 2.8-trillion-parameter model that topped the Frontend Code Arena leaderboard on day one, releases its full open weights on July 27. The MXFP4 weights require approximately 1.4…
Ollama and vLLM are both open-source inference stacks for running large language models, but they serve fundamentally different use cases: Ollama is a model manager optimized for a single developer on…
LLM Inspector, a new open-source CLI tool from developer Helal Saoudi, analyzes live LLM inference processes to show exactly how GPU memory is used by weights, KV cache, and workspace, then projects o…
Netflix's AI Platform team built an in-house LLM serving stack, running the full pipeline from model deployment through inference inside its existing production environment. The team selected vLLM as …
Thomas, a machine learning engineer with five years of production deployment experience, released Coloma as an open-source verification and tuning layer for self-hosted large language models. The tool…
Moonshot AI released Kimi K3, a 2.8 trillion parameter open Mixture-of-Experts model with a 1 million token context window. The model introduces innovations like Kimi Delta Attention for faster decodi…