Anyscale released agent skills for debugging Ray workloads, including /anyscale-platform-fix and /anyscale-platform-inspect, which automate troubleshooting of failing pipelines. A user demonstrated fi…
A production AI assistant requires a systems-level architecture beyond a single LLM invocation, splitting responsibilities into five layers: LLM, Memory, Tooling, Routing, and Observability. OpenAI an…
Two NVIDIA DGX Spark units were clustered over a 200 GbE link to serve the Qwen3-30B-A3B-Thinking model using Ray and vLLM with tensor parallelism. The setup required pinning all transport layers to t…
A developer successfully clustered a 128 GB Mac Studio and a 128 GB DGX Spark using the open-source exo framework to run a model requiring 160 GB of memory, but the setup is limited to CPU-only infere…
Helion kernels were integrated into vLLM for FP8 inference using Qwen3 models and evaluated across NVIDIA H100 and B200 GPUs. The experiments demonstrated that Helion provides a productive PyTorch-nat…
AMD released Lemonade SDK version 10.7, adding NVIDIA CUDA support to its local AI server solution that previously only supported AMD hardware, Apple Metal GPUs, and AArch64 CPUs. The update integrate…
A buyer who spent roughly $10,000 on both a Mac Studio M3 Ultra with 512 GB of unified memory and a dual NVIDIA DGX Spark setup ran the same 397-billion-parameter model on each machine and measured th…
A high-performance Expert Parallelism (EP) kernel is essential for running large Mixture-of-Experts (MoE) language models across multiple GPUs, as it handles the dynamic routing of tokens to experts l…
Researchers from UC Berkeley and other institutions released CommBench, a benchmark of over 100 GPU communication problems, to evaluate whether large language models can generate correct and efficient…
A developer reports that running gpt-oss-120B on an NVIDIA DGX Spark (GB10) failed due to MXFP4 quantization lacking native hardware support on SM121, causing kernel hangs and memory spikes. Switching…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
Nvidia's upcoming RTX Spark desktop, priced between $3,000 and $5,000, may not deliver the performance leap over the existing DGX Spark that marketing suggests, according to a skeptical analysis from …
A team of developers built Thousand Token Wood, a multi-agent economic simulation where five AI-powered woodland creatures trade goods using a 3-billion-parameter Qwen2.5-3B model. The simulation, cre…
Researchers have released vLLM, a new inference engine designed to efficiently serve large language models by optimizing memory management and batching. The system achieves up to 24x higher throughput…
NVIDIA AI released Dynamo Snapshot, a checkpoint/restore system using CRIU and cuda-checkpoint to eliminate cold-start delays for AI inference workloads on Kubernetes. The system freezes a running inf…
Microsoft released a tutorial demonstrating how to run a browser-use agent workflow using Microsoft Fara in Google Colab. The tutorial guides users through setting up a mock OpenAI-compatible endpoint…
A developer has released LLMhop, a stateless HTTP router that directs OpenAI-compatible API requests to the appropriate LLM inference backend based on the model name in the request body. The single-bi…
NousResearch has released hermes-agent, an open-source adaptive AI agent designed for local execution on consumer GPUs with personalized learning and long-term memory capabilities. PaddlePaddle has ma…
Huawei released KVarN, a native KV-cache quantization back end for vLLM that delivers up to 5x more cache capacity and 1.3x the throughput of FP16 while maintaining FP16-level accuracy. The calibratio…
Google DeepMind released Gemma 4 12B, an encoder-free multimodal model that processes text, images, audio, and video natively without separate vision or audio encoders. The 12-billion-parameter model …