Rudrite Research published a comprehensive comparison of AI and ML approaches, covering 14 side-by-side analyses of techniques such as Transformers vs Mamba, FlashAttention vs PagedAttention, and PPO …
A developer has released tiny-vllm, a high-performance LLM inference engine written in C++ and CUDA that serves as a smaller sibling to the vLLM project. The open-source repository includes both the f…
NVIDIA's Blackwell consumer GPU (GB203/SM_120) features a unified TensorCore pipeline where all 12 non-FP64 precision formats share identical 29-cycle latency and 23-cycle throughput, reducing precisi…
NVIDIA's CUTLASS library, a header-only C++ template framework for writing custom CUDA kernels, powers much of the AI infrastructure behind FlashAttention, vLLM, and PyTorch's internal kernels. The li…
FlashLib, a new GPU library for classical machine learning operators, achieves speedups of up to 208× over cuML on Hopper GPUs for algorithms including KMeans, KNN, and PCA. The library is designed to…
A developer implemented FlashAttention's forward and backward passes from scratch in pure CUDA C++, achieving O(N) memory complexity through manual SRAM tiling and online softmax recurrence. A rejecte…
Synchronous continuous batching in LLM inference causes inefficiency by forcing the CPU and GPU to work sequentially, leaving one idle while the other operates. This idle time can account for nearly a…