A developer open-sourced AutoMegakernel, a tool that compiles any HuggingFace model into a single persistent megakernel, reducing overhead by launching one kernel per forward pass. It includes a stati…
The AI boom is pulling GPU vendors away from double-precision (FP64) hardware essential for scientific HPC, as NVIDIA, AMD, and Intel prioritize low-precision AI cores. New chips like NVIDIA's B200 an…
Sarvam raised $234 million at a $1.5 billion valuation in a Series B extension led by HCLTech, which contributed $150 million as the lead strategic investor. The Bengaluru-based AI startup, which has …
Cohere released North Mini Code, a 30-billion-parameter open-source coding agent with a Mixture-of-Experts architecture that activates only 3 billion parameters per token, enabling it to outperform mo…
A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
Carmen Li, CEO of Silicon Data and Compute Exchange, said the compute market is shifting from on-demand to forward contracts as GPU price volatility rises, with fungibility challenges complicating tra…
Google DeepMind released DiffusionGemma, an open-source Apache 2.0-licensed diffusion-based large language model that generates text at over 1,000 tokens per second on a single H100 GPU, achieving up …
A developer found that `nvidia-smi` reported 97% GPU utilization on an H100 cluster while actual training throughput was less than half of expected benchmarks. Tracing via eBPF revealed the GPU was id…
The MADSys Lab at Tsinghua University’s KTransformers project enables frontier-class MoE models like DeepSeek-R1 671B to run on commodity hardware with a CPU-GPU hybrid inference stack, achieving 286 …
Cohere AI released North Mini Code, a 30-billion-parameter open-weight mixture-of-experts model with 3 billion active parameters per token, optimized for code generation, agentic software engineering,…
Google released DiffusionGemma, a 26-billion-parameter open model that generates text through a diffusion process rather than token-by-token, achieving roughly 1,000 tokens per second on a single Nvid…
Helion kernels were integrated into vLLM for FP8 inference using Qwen3 models and evaluated across NVIDIA H100 and B200 GPUs. The experiments demonstrated that Helion provides a productive PyTorch-nat…
Google DeepMind's DiffusionGemma, optimized for NVIDIA platforms, generates text tokens in parallel rather than sequentially, achieving up to 1,000 tokens per second on a single NVIDIA H100 GPU. The m…
WEKA's NeuralMesh and Augmented Memory Grid software delivered 10x higher token throughput, 10x more concurrent users, and 7x more tokens per GPU on Oracle Cloud Infrastructure compared to DRAM-only c…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
Equixly engineers report that GPU scarcity is a structural problem driven by hyperscaler stockpiling and rising energy costs, not a temporary supply-chain issue. The company argues that standard load-…
A developer deploying a 70B Llama-3 model on 8x H100s found that scaling from 8k to 32k context windows causes the KV cache to balloon to 10.7 GB per request, forcing memory paging to CPU at 200 concu…
A GPU training pipeline at a major AI company breached its SLA despite showing 97% GPU utilization across all monitoring tools (Datadog, Grafana, nvidia-smi). Using eBPF kernel tracing, an SRE identif…
A developer compared the costs of AI inference on Replicate and deAPI across four common tasks, finding that deAPI's task-based billing is often cheaper than Replicate's time-based or per-unit pricing…
AMD's MI300X accelerator, with 192GB of HBM3 memory and roughly half the list price of NVIDIA's H100, remains underutilized due to software incompatibilities. As of early May 2026, running vLLM with D…