The CUHKSZ team submitted a simultaneous speech translation system to IWSLT 2026, built on Qwen3-Omni-30B-A3B with LoRA adaptation, achieving 40.5 BLEU for English→Chinese and 27.7 BLEU for English→Ge…
Agentic AI systems are maturing from prototypes into production-grade infrastructure, with vLLM's Micro-Agent framework demonstrating that serving-layer orchestration can match or beat frontier models…
VLLM Semantic Router introduces a new open-source serving primitive that turns a single model API call into a bounded collaboration among multiple micro-agents, enabling cost savings, safety enforceme…
PyTorch introduced the Cross-Repository CI Relay (CRCR), an automated pipeline that triggers and tracks CI in downstream repositories whenever changes are made to pytorch/pytorch. Results are displaye…
LLM inference optimization can reduce serving costs by 5-10x and latency by 3-5x, often determining whether an AI feature ships. The bottleneck is memory bandwidth during autoregressive decoding, and …
An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
AMD Strix Halo cluster setup guide details how to configure a two-node system linked via Intel E810 RoCE v2 for distributed vLLM inference using Tensor Parallelism. The guide covers hardware prerequis…
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…
ByteDance open-sourced DeerFlow 2.0, a long-horizon agent runtime that orchestrates sub-agents, sandboxes, persistent memory, and an extensible skill system. The project reached 74,960 GitHub stars an…
Developer released Brytlog, an open-source AI logger that replaces raw terminal output with concise AI summaries to save developers time and money. The tool acts as a pre-processor for agentic workflo…
A new plaintext markup format called Inference Cards aims to standardize how self-hosted LLM performance claims are communicated, requiring details like model variant, quantization, hardware, inferenc…
GOSIM Paris 2025, held at Station F on May 5-6, showcased open-source AI developments including LLMs advancing in mathematical reasoning, a call for transparency over speed, and the introduction of Ta…
ZeroGate, an open-source event-driven cross-cloud GPU orchestration fabric, eliminates idle hardware costs in multi-tenant inference pipelines by scaling dedicated infrastructure pools to zero when de…
NVIDIA's June 2026 DGX Spark update introduces automated four-node clustering via Cluster Assistant, enabling local inference of models up to 700B parameters. The update also delivers a 2.6x throughpu…
A new guide walks users through deploying an OpenClaw AI agent on a virtual private server, balancing cost, availability, and privacy. The tutorial covers server configuration, system requirements, an…
A developer reduced LLM extraction latency from 42 seconds to 6 seconds by replacing verbatim text copying with pointer-based extraction and splitting a single large call into multiple parallel calls.…
Hugging Face launched a one-command method to run a vLLM server on its Jobs infrastructure, enabling users to quickly deploy models for testing, evaluation, or batch generation. The feature uses the o…
Metaflow, an open-source ML/AI orchestration framework, ranks first in every category of the Cloud Native Computing Foundation's latest Technology Radar report, with 51% of surveyed users highly likel…
The launch of Z.ai's GLM 5.2 and DeepSeek V4 Flash has created a 50x price gap between open-weight APIs and closed frontier models, reshaping the build-versus-buy calculus for developers. While open-w…
Flama 2.0 introduces a CLI tool that allows users to download, package, and serve large language models from HuggingFace with a single command, including a built-in chat interface and production-ready…