Alibaba released the model weights for Qwen3.8-Flash-Next, a multimodal mixture-of-experts model with 125B parameters plus 51B N-gram embeddings and 6B activated per token, as a preview of the Qwen4 a…
A technical survey reports that a fully self-hosted voice AI pipeline can achieve around 755 milliseconds from end of speech to first audio output, while a provider comparison shows the same speech-to…
Realtime voice agents, which combine streaming speech-to-text, a large language model, and text-to-speech in a cascaded pipeline, must achieve a time-to-first-audio (TTFA) of around 200 milliseconds t…
Google Cloud has integrated native TPU support into vLLM, the open-source LLM serving engine, to enable enterprise-grade precision for long-context multimodal embedding inference, targeting the Qwen3 …
IBM released its Granite 4.2 family of open-weight large language models on August 25, featuring three variants (3B, 8B, and 30B) with native 128K context windows and switchable chain-of-thought reaso…
IBM released Granite 4.2, a family of open reasoning language models in 3B, 8B, and 30B parameter sizes, available under Apache 2.0. The models are decoder-only dense transformers pre-trained on rough…
A forum user on Level1Techs is building a 4U chassis to connect four air-cooled or eight liquid-cooled NVIDIA H100 GPUs to a Dell PowerEdge R7515 server via PCIe switches, using either a Broadcom PEX8…
A developer investigating a vLLM crash on workstation Blackwell GPUs found that FP8 KV cache kernels exceed the per-block shared-memory limit on sm_120 hardware, causing OutOfResources errors. The iss…
NVIDIA announced on August 24, 2026, that its Groq 3 LPX chip, built on technology licensed from Groq for a reported $20 billion, has entered full production as a dedicated decode extension to its Ver…
An engineer at Jahn AI ran a per-prompt regression check before recommending FP8 quantization for Qwen3-8B serving on an RTX PRO 6000 Blackwell, which delivered a 1.5x throughput gain (1,725 to 2,597 …
Nous Research's open source Hermes Agent runtime, released in February 2026 under the MIT license, is designed to close the gap in long-running agents by using a three-phase process that includes a re…
An engineer benchmarked Qwen3-8B on workstation Blackwell GPUs across three serving stacks, finding that vLLM outperformed SGLang and llama.cpp by up to 4x under load, and that an FP8 pass delivered a…
The second part of a guide to LLM inference optimization focuses on attention mechanisms and KV cache management, explaining the compute-bound prefill phase and memory-bound decode phase. It introduce…
IBM's Granite Team released Granite 4.2, a family of dense, decoder-only reasoning LLMs in 3B, 8B, and 30B sizes, pre-trained from scratch on roughly 15 trillion tokens with a five-phase strategy exte…
The Laude Institute and MIT open-sourced Headlong, a bash-based framework for agents that think continuously, and claim their agent Audel fixed its own bug unprompted. Security researcher Boyd Kane de…
Ray Serve LLM, developed by Anyscale, introduces token-load-aware routing to improve LLM serving efficiency, moving beyond naive KV cache reuse strategies. The new approach addresses the challenges of…
ClaudeGate, a new open-source local API gateway, enables Claude Code CLI and Anthropic SDK applications to use any AI model, including free and frontier models from OpenRouter, DeepSeek, and others, b…
SGLang and vLLM, the two major LLM serving engines, have attempted to enforce deterministic inference but face fundamental challenges due to floating-point non-associativity in GEMM kernels, which cau…
Moonshot AI's Kimi K3, an open-weight model with 2.8 trillion total parameters but only 104 billion active per token, outperforms proprietary rivals on agentic coding benchmarks, scoring 42.0 on SWE M…
Inferact, the company founded by the creators and core maintainers of vLLM, is hiring a remote Member of Technical Staff for Cloud Orchestration with a salary of $200k–400k/yr. The role involves build…