Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA
type: NewsArticle
description: A developer has released tiny-vllm, a high-performance LLM inference engine written in C++ and CUDA that serves as a smaller sibling to the vLLM project. The open-source repository includes both the full source code for an inference server capable of running real models like Llama 3.2 1B Instruct, and an educational course that teaches users how to implement the engine from scratch, covering topics from Safetensors loading to PagedAttention. The project aims to serve as a learning tool for individuals and a teaching resource for university lecturers.
date published: 2026-05-29 19:38:27+00:00
date modified: 2026-05-29 19:46:39.703823+00:00
language: en
is accessible for free: true
article body: You're going to build a high performance LLM inference engine with C++ and CUDA - tiny-vllm, a younger and smaller sibling of [vLLM](https://github.com/vllm-project/vllm) We will learn a lot along the way, make mistakes and derive the ideas and maths from scratch This repository consists of two things: 1. a full source code of the inference server and 2. a course where I lead you through the process of implementing the engine. Feel invited to use it as a learning tool on your learning path or if you are a lecturer, feel welcome to use it as a teaching resource at your university The inference engine consists of: - load a real LLM model from Safetensors (Llama 3.2 1B Instruct) - full LLM forward pass (prefill + decode) - all computation with CUDA kernels - KV cache - static batching - continuous batching - [online softmax, FlashAttention-like](https://courses.cs.washington.edu/courses/cse599m/23sp/notes/flashattn.pdf) - [PagedAttention](https://arxiv.org/pdf/2309.06180) Make yourself a hot beverage and let's begin [tiny-vllm](#tiny-vllm)[Intro: LLM, vLLM, models, inference servers](#intro-llm-vllm-models-inference-servers)[Technical prerequisities](#technical-prerequisities)[Safetensors and your model](#safetensors-and-your-model)[How floating-point numbers work and why we use bfloat16](#how-floating-point-numbers-work-and-why-we-use-bfloat16)[GPU and CPU memory](#gpu-and-cpu-memory)[Single token inference](#single-token-inference)[Tokenization](#tokenization)[Embeddings](#embeddings)[CUDA kernel engineering - embeddings](#cuda-kernel-engineering---embeddings)[RMSNorm and parallel reduction in CUDA](#rmsnorm-and-parallel-reduction-in-cuda)[RoPE](#rope)[Residual connections](#residual-connections)[cublasGemmEx](#cublasgemmex)[The column-major to row-major transposition trick](#the-column-major-to-row-major-transposition-trick)[Prefill vs decode](#prefill-vs-decode)[Why KV cache exists](#why-kv-cache-exists)[Attention](#attention)[GQA](#gqa)[SiLU](#silu)[Softmax](#soft…
publisher: Web Pulse
keywords: large-language-models, artificial-intelligence, machine-learning, ai-infrastructure, ai-tools
mentions: vLLM
mentions: Llama 3.2 1B Instruct
mentions: Safetensors
mentions: CUDA
mentions: FlashAttention
mentions: PagedAttention
citation: CreativeWork