SigNoz published a vLLM monitoring dashboard that tracks token throughput, latency, and KV cache usage for self-hosted vLLM inference servers, requiring SigNoz v0.135.0 or newer and the V2 dashboard s…
SigNoz published a guide for monitoring self-hosted vLLM inference servers by sending Prometheus metrics and OpenTelemetry request traces to SigNoz. The guide has operators scrape vLLM's /metrics endp…
AIPerf is a dedicated benchmarking tool that measures LLM inference performance by generating concurrent request loads, according to a first-person account of using it to test deployments. The account…
LocalJev, a TypeScript server for Bun 1.2+, implements a Jev-compatible POST /v1/systemone API backed by the DiffusionGemma model diffusiongemma-26B-A4B-it-4bit through an OpenAI-compatible Chat Compl…
Abacus.AI launched the Smaug line of three open-weight models on September 10, fine-tuned for long-running agentic loops and deployable inside a customer's own VPC at open-source rates. Abacus.AI clai…
An 8-GPU tensor-parallel (TP8) rig should prioritize PCIe topology over CPU choice, according to guidance recommending that GPUs communicate directly through a Gen5 switch rather than routing peer-to-…
A developer deployed Google's Gemma 4 E2B to a single Tesla T4 GPU on a Compute Engine VM using vLLM 0.29.0, finding that the QAT int4-weight checkpoint decodes at 72.31 tok/s per stream versus 40.44 …
A user who purchased 8 R9700 GPUs is asking for advice on the cheapest way to build a PCIe 5.0 x16 tensor-parallel rig for running TP8 inference on ~200GB models, proposing a single Microchip Switchte…
NVIDIA released AIPerf, a ground-up rewrite and designated successor to GenAI-Perf, as a multiprocessed LLM inference benchmarking tool that coordinates worker processes and record-processor services …
Hugging Face released six open-source agent skills that let coding agents such as Kiro and Claude Code deploy production-ready Hugging Face models on Amazon SageMaker AI with autoscaling, Amazon Cloud…
A forum thread is crowdsourcing real-world local LLM deployment data across five VRAM tiers — ≤16GB, 24–32GB, 48–64GB, 96–128GB, and 196–256GB+ — asking users to report model, quantization (AutoRound …
A developer documented how to run the 600GB Inkling-NVFP4 model on a Google Kubernetes Engine Spot A3 instance with eight NVIDIA H100 GPUs, tracing the initial crash to an ABI mismatch between Ray's d…
A developer published a step-by-step guide and open-source MCP toolkit for deploying Google's Gemma 4 E2B model to a single AMD Instinct MI300X GPU rented through AMD Developer Cloud at $1.99 per hour…
NVIDIA's Jetson AGX Thor achieved a 6.4x speed increase on the MLPerf Edge Agentic benchmark using the TensorRT Edge-LLM inference engine, according to the report. The gain comes from TensorRT Edge-LL…
A training-free routing policy called the Tri-Metric Router achieved 0% out-of-memory failures and 88.5 ± 4.4% oracle alignment on out-of-distribution holdouts for retrieval-augmented generation on co…
Researchers introduced AutoTuneBench, a benchmark and measurement protocol that makes trust architectural for LLM agents auto-tuning GPU kernels and serving engines, after characterizing four failure …
A developer argues that AI coding assistants should sit at the end of the engineering toolchain, used only when deterministic tools like compilers, tests, debuggers, profilers, and Git cannot answer a…
AMD published a step-by-step FlyDSL workflow for implementing a custom diffusion attention kernel, targeting vLLM's TiDAR (Think in Diffusion, Talk in Autoregression) mode with paged KV caches and scr…
VLLM contributor mmastrac opened a pull request adding a "Jev-like" structured generation mode for the DiffusionGemma model, retitling it from a work-in-progress draft to "[Core] structured generation…
Thinking Machines Lab is hiring a Research, Tinker, RL Systems engineer in San Francisco at an annual salary range of $350,000 to $475,000, a figure the job board states sits 73% above the $238,000 me…