A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
Carmen Li, CEO of Silicon Data and Compute Exchange, said the compute market is shifting from on-demand to forward contracts as GPU price volatility rises, with fungibility challenges complicating tra…
Google DeepMind released DiffusionGemma, an open-source Apache 2.0-licensed diffusion-based large language model that generates text at over 1,000 tokens per second on a single H100 GPU, achieving up …
A developer found that `nvidia-smi` reported 97% GPU utilization on an H100 cluster while actual training throughput was less than half of expected benchmarks. Tracing via eBPF revealed the GPU was id…
The MADSys Lab at Tsinghua University’s KTransformers project enables frontier-class MoE models like DeepSeek-R1 671B to run on commodity hardware with a CPU-GPU hybrid inference stack, achieving 286 …
Cohere AI released North Mini Code, a 30-billion-parameter open-weight mixture-of-experts model with 3 billion active parameters per token, optimized for code generation, agentic software engineering,…
Google released DiffusionGemma, a 26-billion-parameter open model that generates text through a diffusion process rather than token-by-token, achieving roughly 1,000 tokens per second on a single Nvid…
Helion kernels were integrated into vLLM for FP8 inference using Qwen3 models and evaluated across NVIDIA H100 and B200 GPUs. The experiments demonstrated that Helion provides a productive PyTorch-nat…
Google DeepMind's DiffusionGemma, optimized for NVIDIA platforms, generates text tokens in parallel rather than sequentially, achieving up to 1,000 tokens per second on a single NVIDIA H100 GPU. The m…
WEKA's NeuralMesh and Augmented Memory Grid software delivered 10x higher token throughput, 10x more concurrent users, and 7x more tokens per GPU on Oracle Cloud Infrastructure compared to DRAM-only c…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
Equixly engineers report that GPU scarcity is a structural problem driven by hyperscaler stockpiling and rising energy costs, not a temporary supply-chain issue. The company argues that standard load-…
A developer deploying a 70B Llama-3 model on 8x H100s found that scaling from 8k to 32k context windows causes the KV cache to balloon to 10.7 GB per request, forcing memory paging to CPU at 200 concu…
A GPU training pipeline at a major AI company breached its SLA despite showing 97% GPU utilization across all monitoring tools (Datadog, Grafana, nvidia-smi). Using eBPF kernel tracing, an SRE identif…
A developer compared the costs of AI inference on Replicate and deAPI across four common tasks, finding that deAPI's task-based billing is often cheaper than Replicate's time-based or per-unit pricing…
AMD's MI300X accelerator, with 192GB of HBM3 memory and roughly half the list price of NVIDIA's H100, remains underutilized due to software incompatibilities. As of early May 2026, running vLLM with D…
Modern GPUs spend most of their time during AI inference waiting for data, as memory bandwidth cannot keep pace with compute throughput. This fundamental bottleneck has driven the AI hardware market, …
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Researchers have developed DiffusionBlocks, a framework that partitions transformer neural networks into independently trainable blocks to reduce memory requirements proportionally while maintaining c…
GPU utilization rates on major clouds average just 5 percent, meaning most paid GPU hours produce no useful work, while egress fees of up to $0.12 per GB and data gravity lock-in add hidden costs that…