An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
AMD Strix Halo cluster setup guide details how to configure a two-node system linked via Intel E810 RoCE v2 for distributed vLLM inference using Tensor Parallelism. The guide covers hardware prerequis…
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…
ByteDance open-sourced DeerFlow 2.0, a long-horizon agent runtime that orchestrates sub-agents, sandboxes, persistent memory, and an extensible skill system. The project reached 74,960 GitHub stars an…
Developer released Brytlog, an open-source AI logger that replaces raw terminal output with concise AI summaries to save developers time and money. The tool acts as a pre-processor for agentic workflo…
A new plaintext markup format called Inference Cards aims to standardize how self-hosted LLM performance claims are communicated, requiring details like model variant, quantization, hardware, inferenc…
GOSIM Paris 2025, held at Station F on May 5-6, showcased open-source AI developments including LLMs advancing in mathematical reasoning, a call for transparency over speed, and the introduction of Ta…
ZeroGate, an open-source event-driven cross-cloud GPU orchestration fabric, eliminates idle hardware costs in multi-tenant inference pipelines by scaling dedicated infrastructure pools to zero when de…
NVIDIA's June 2026 DGX Spark update introduces automated four-node clustering via Cluster Assistant, enabling local inference of models up to 700B parameters. The update also delivers a 2.6x throughpu…
A new guide walks users through deploying an OpenClaw AI agent on a virtual private server, balancing cost, availability, and privacy. The tutorial covers server configuration, system requirements, an…
A developer reduced LLM extraction latency from 42 seconds to 6 seconds by replacing verbatim text copying with pointer-based extraction and splitting a single large call into multiple parallel calls.…
Hugging Face launched a one-command method to run a vLLM server on its Jobs infrastructure, enabling users to quickly deploy models for testing, evaluation, or batch generation. The feature uses the o…
Metaflow, an open-source ML/AI orchestration framework, ranks first in every category of the Cloud Native Computing Foundation's latest Technology Radar report, with 51% of surveyed users highly likel…
The launch of Z.ai's GLM 5.2 and DeepSeek V4 Flash has created a 50x price gap between open-weight APIs and closed frontier models, reshaping the build-versus-buy calculus for developers. While open-w…
Flama 2.0 introduces a CLI tool that allows users to download, package, and serve large language models from HuggingFace with a single command, including a built-in chat interface and production-ready…
ParseHawk, a new local-first document AI tool, launched on Hacker News, enabling users to extract structured JSON from PDFs, scans, and images entirely on their own hardware without sending data to th…
A single H200 GPU with 141GB HBM3e cannot comfortably run DeepSeek V4 Flash (284B total, 13B active parameters) due to VRAM constraints, even with 2TB system RAM for offloading. The model requires an …
Ora Computing launched an automated LLM compression engine that reduces model size by up to 70% with minimal accuracy loss, enabling deployment on edge devices, on-prem servers, or cloud infrastructur…
A developer built Modelplane, an open-source inference control plane using Crossplane, to manage GPU fleets across clouds, neoclouds, and on-premise environments. The platform allows platform teams to…
A developer achieved 3x better batch throughput with Qwen3.5 by fixing vLLM on AMD's Strix Halo using the Lemonade Server, enabling more efficient AI inference on AMD hardware.…