A 30 September 2026 arXiv paper introduces Vosti, an inference engine that produces bitwise-identical logits across every tested execution variation, a stronger guarantee than vLLM's batch-invariant m…
Microsoft added experimental llama.cpp support to Windows ML, letting developers run GGUF models locally on Windows through new task-specific APIs, starting with a Windows ML Text Generation API that …
An open reference architecture proposes a physical-AI counting system for Pakistan's steel mills, using IP66 HDR cameras and sealed GPU-accelerated industrial PCs running fine-tuned RF-DETR detection …
Richael (dh8116), a Year 11 student in Auckland, built Soulor AI, an AI companion app with persistent memory and a five-perspective Simulation Mode, on a LoRA fine-tune of Qwen3-14B served with vLLM o…
AOTInductor, PyTorch's ahead-of-time inference backend compiler, emits Triton kernels as cubin files stored in the pt2 zip archive, which developers can unzip and disassemble with nvdisasm or cuobjdum…
A September 29, 2026 arXiv paper reports that an LLM agent translating Triton kernels directly into NVIDIA PTX, a process the authors call "AI lowering," achieved 0.83x to 3.34x the performance of aut…
A developer implemented the FlashAttention-2 forward pass twice — first as a deliberately slow PyTorch reference with a Python double loop over tiles, then as a Triton kernel — to teach kernel optimiz…
Researchers introduced Elastic Threshold Attention (ETA), an end-to-end trainable sparse attention architecture that predicts dynamic, contextual thresholds from query representations to speed up long…
Researchers developed HyQuant, a hybrid-precision quantization method that keeps a narrow set of critical tokens in full precision while quantizing the bulk of query, key, and value tensors to low-bit…
Justin Grimm published a reference implementation for a 64-byte L1D cache-resident frame that compresses a 128-node consensus vote bitmask to 26 bytes using balanced ternary (3^5 = 243 ≤ 256), letting…
AIPerf is a dedicated benchmarking tool that measures LLM inference performance by generating concurrent request loads, according to a first-person account of using it to test deployments. The account…
A developer deployed Google's Gemma 4 E2B to a single Tesla T4 GPU on a Compute Engine VM using vLLM 0.29.0, finding that the QAT int4-weight checkpoint decodes at 72.31 tok/s per stream versus 40.44 …
Nvidia released CUDA Rust on September 8, two open-source projects that compile Rust directly to PTX for GPU kernels rather than wrapping CUDA C++. The cuda-oxide track uses Rust's borrow checker to c…
AMD published a step-by-step FlyDSL workflow for implementing a custom diffusion attention kernel, targeting vLLM's TiDAR (Think in Diffusion, Talk in Autoregression) mode with paged KV caches and scr…
Researchers introduced Simple Attention Sparsification (SAS), a gated sparse attention mechanism that optimizes context ranking end-to-end with the language modeling loss by injecting the selector's c…
NVIDIA's CUDA Multi-Process Service (MPS) enables concurrent kernel execution across multiple processes, improving GPU utilization compared to default time-slicing, according to a technical blog post …
A developer documented writing GPU kernels in Triton to understand PyTorch operations, starting with vector add and fused ReLU/dropout, highlighting the benefit of fused kernels in reducing memory rou…
Genesis AI, the company behind the open-source physics simulator Genesis, is hiring a lead engineer for Quadrants, its open-source Python-to-machine-code compiler that targets CPU and GPU backends wit…
Neuro, an alpha-stage Ahead-of-Time compiled language for AI workloads, now compiles its full general-purpose language surface through an LLVM 20 backend, with planned MLIR-based tensor operations, En…
A developer's source-code walkthrough of vLLM 0.22's V1 execution path traces a single offline inference request from the LLM.generate() API through inter-process communication, scheduling, GPU execut…