Free LLM inference handbook: 100 engineers cloned it in week 1

More than 100 engineers cloned a free open-source handbook on large language model inference within its first week of release. The guide consolidates years of production experience and research to address the unique challenges of serving LLMs, including unpredictable latency, growing memory demands, and high costs. The project aims to fill a gap in available resources by providing a comprehensive reference for deploying LLMs in production.

The definitive guide to serving large language models in production. Quick Start -quick-start • Contents -table-of-contents • Labs -labs • Community -community • Contributing -contributing LLM inference is hard. Not "read the docs and figure it out" hard — fundamentally different from everything else in ML hard. Traditional ML inference is a solved problem. You batch requests, run a forward pass, return results. Latency is predictable, memory is fixed, scaling is linear. LLM inference breaks all of these assumptions: Latency is unpredictable — a 10-token response takes 100ms, a 1000-token response takes 10 seconds Memory grows during requests — the KV cache expands with every generated token Scaling is sub-linear — communication overhead dominates as you add GPUs Cost is 100x higher — $0.001/request becomes $0.10/request This handbook exists because we needed it and couldn't find it. The knowledge is scattered across papers, blog posts, tribal knowledge, and source code comments. We've consolidated years of production experience and research into one comprehensive resource. This is the guide we wish existed when we started. 📬 Follow the build— New chapters, explained in plain English with production context. Subscribe to The Engineer's Digest to get notified when new content drops. Subscribe free → 💬 Join the discussion https://github.com/harshuljain13/llm-inference-at-scale/discussions — questions, feedback, and corrections welcome | | | | - Python 3.10+ - CUDA 12.0+ for GPU labs - Basic PyTorch familiarity git clone https://github.com/harshuljain13/llm-inference-at-scale.git cd llm-inference-at-scale Create virtual environment python -m venv .venv source .venv/bin/activate On Windows: .venv\Scripts\activate Install dependencies pip install -r requirements.txt Open the first chapter open content/00 foundations/00.0 what is llm inference/what is llm inference.md Or browse the Table of Contents -table-of-contents below. | Chapter | Title | Description | |---|---|---| | 0.0 | | Why LLM Inference is Different /harshuljain13/llm-inference-at-scale/blob/master/content/00 foundations/00.1 why llm inference is different/why llm inference is different.md Transformer Inference Mechanics /harshuljain13/llm-inference-at-scale/blob/master/content/00 foundations/00.2 transformer inference basics/transformer inference basics.md | Chapter | Title | Description | |---|---|---| | 1.1 | | Roofline Model /harshuljain13/llm-inference-at-scale/blob/master/content/01 gpu fundamentals/01.2 roofline model/roofline model.md FlashAttention /harshuljain13/llm-inference-at-scale/blob/master/content/01 gpu fundamentals/01.3 flash attention/flash attention.md | Chapter | Title | Description | |---|---|---| | 2.1 | | Attention Mechanisms /harshuljain13/llm-inference-at-scale/blob/master/content/02 attention and kv/02.2 attention mechanisms/attention mechanisms.md PagedAttention /harshuljain13/llm-inference-at-scale/blob/master/content/02 attention and kv/02.3 paged attention/paged attention.md KV Cache Compression /harshuljain13/llm-inference-at-scale/blob/master/content/02 attention and kv/02.4 kv cache compression/kv cache compression.md | Chapter | Title | Description | |---|---|---| | 3.1 | | TurboQuant /harshuljain13/llm-inference-at-scale/blob/master/content/03 optimization/03.2 turboquant/turboquant.md Continuous Batching /harshuljain13/llm-inference-at-scale/blob/master/content/03 optimization/03.3 continuous batching/continuous batching.md Speculative Decoding /harshuljain13/llm-inference-at-scale/blob/master/content/03 optimization/03.4 speculative decoding/speculative decoding.md Chunked Prefill /harshuljain13/llm-inference-at-scale/blob/master/content/03 optimization/03.5 chunked prefill/chunked prefill.md | Chapter | Title | Description | |---|---|---| | 4.1 | | SGLang /harshuljain13/llm-inference-at-scale/blob/master/content/04 engines/04.2 sglang/sglang.md TensorRT-LLM /harshuljain13/llm-inference-at-scale/blob/master/content/04 engines/04.3 tensorrt llm/tensorrt llm.md | Chapter | Title | Description | |---|---|---| | 5.1 | | MoE Inference /harshuljain13/llm-inference-at-scale/blob/master/content/05 scaling/05.2 moe inference/moe inference.md Distillation /harshuljain13/llm-inference-at-scale/blob/master/content/05 scaling/05.3 distillation | Chapter | Title | Description | |---|---|---| | 6.1 | | EKS + KServe /harshuljain13/llm-inference-at-scale/blob/master/content/06 serving/06.2 eks kserve/eks kserve.md SageMaker /harshuljain13/llm-inference-at-scale/blob/master/content/06 serving/06.3 sagemaker/sagemaker.md Disaggregated Serving /harshuljain13/llm-inference-at-scale/blob/master/content/06 serving/06.4 disaggregated serving/disaggregated serving.md Cold Start /harshuljain13/llm-inference-at-scale/blob/master/content/06 serving/06.5 cold start/cold start.md | Chapter | Title | Description | |---|---|---| | 7.1 | | Structured Output /harshuljain13/llm-inference-at-scale/blob/master/content/07 operations/07.2 structured output Edge Deployment /harshuljain13/llm-inference-at-scale/blob/master/content/07 operations/07.3 edge deployment/edge deployment.md Hands-on exercises to reinforce each concept. Each lab includes starter code, step-by-step instructions, and solutions. | Lab | Title | Prerequisites | Time | |---|---|---|---| | 01 | | VRAM Calculation /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 02 vram calculation Quantization Comparison /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 03 quantization comparison vLLM Deployment /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 04 vllm deployment SGLang Structured Output /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 05 sglang structured output Tensor Parallelism /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 06 tensor parallelism Ray Serve Deployment /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 07 ray serve deployment EKS + KServe /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 08 eks kserve deployment SageMaker Production /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 09 sagemaker production Benchmarking Suite /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 10 benchmarking monitoring Hardware requirements: Most labs run on a single GPU g5.xlarge or equivalent . Labs 06 and 08 require multi-GPU instances. Formulas you'll use constantly when working with LLM inference: The theoretical maximum decode speed, limited by how fast you can read model weights: max tokens per second = memory bandwidth / model size bytes Example: Llama 8B 16GB FP16 on A100 2 TB/s → 125 tokens/sec maximum Memory required for the key-value cache: kv cache bytes = 2 × num layers × num kv heads × head dim × seq len × batch size × dtype bytes Example: Llama 8B, batch=1, seq=4096, FP16 → 512 MB Determines whether a workload is compute-bound or memory-bound: arithmetic intensity = FLOPs / bytes transferred Rule of thumb: Below the ridge point ~156 FLOPs/byte on A100 = memory-bound llm-inference-at-scale/ ├── content/ 📖 Handbook chapters │ ├── 00 foundations/ Part I: Foundations │ ├── 01 gpu fundamentals/ Part II: GPU Fundamentals │ ├── 02 attention and kv/ Part III: Attention & KV Cache │ ├── 03 optimization/ Part IV: Optimization Techniques │ ├── 04 engines/ Part V: Inference Engines │ ├── 05 scaling/ Part VI: Scaling │ ├── 06 serving/ Part VII: Production Serving │ ├── 07 operations/ Part VIII: Operations │ └── utils/ Visualization utilities ├── labs/ 🧪 Hands-on exercises ├── reference/ 📋 Quick references │ ├── cheat sheet.md One-page summary │ ├── glossary.md Terminology │ ├── vllm quick reference.md vLLM commands │ └── cost calculator.py Inference cost estimation ├── assets/ 🎨 Images and diagrams └── slides/ 📊 Presentation materials For engineers who need to deploy an LLM this week: 0.0 What is LLM Inference? /harshuljain13/llm-inference-at-scale/blob/master/content/00 foundations/00.0 what is llm inference/what is llm inference.md — 15 min 0.1 Why LLM Inference is Different /harshuljain13/llm-inference-at-scale/blob/master/content/00 foundations/00.1 why llm inference is different/why llm inference is different.md — 20 min 3.1 Quantization /harshuljain13/llm-inference-at-scale/blob/master/content/03 optimization/03.1 quantization/quantization.md — 20 min 4.1 vLLM /harshuljain13/llm-inference-at-scale/blob/master/content/04 engines/04.1 vllm/vllm.md — 30 min Lab 04: vLLM Deployment /harshuljain13/llm-inference-at-scale/blob/master/labs/lab 04 vllm deployment — 45 min For engineers building inference infrastructure: Morning: Part I Foundations + Part II GPU Fundamentals Afternoon: Part IV Optimization + Part V Engines Labs: 01, 02, 03, 04 For teams standardizing on LLM serving: Day 1: Parts I, II, III — Foundations through KV Cache Day 2: Parts IV, V, VI — Optimization through Scaling Day 3: Parts VII, VIII — Production Serving and Operations Labs: All 10 labs This material has been presented at: More talks coming — if you'd like this at your conference or meetup, open an issue. If you use this material in research or internal documentation, please cite: @misc{llm-inference-at-scale, title={LLM Inference at Scale: A Practitioner's Handbook}, author={Jain, Harshul}, year={2025}, url={https://github.com/harshuljain13/llm-inference-at-scale} } If you find this useful, please ⭐ the repo — it helps others discover it. Contributions are welcome. This is a living document. Fix errors — Typos, outdated information, incorrect formulas Improve clarity — Better explanations, additional examples Add content — New chapters, labs, or reference materials - Fork the repository - Create a feature branch git checkout -b improve-kv-cache-chapter - Make your changes - Submit a pull request To report errors or suggest corrections, open a GitHub Issue. Harshul Jain is a Senior ML Infrastructure Engineer at Audible Amazon , where he owns the ML Feature Store, a GenAI semantic search platform serving millions of customers, and real-time streaming pipelines at scale. He has been building and operating ML infrastructure in production for 4+ years and mentors 300+ engineers through an eMentoring program. - GitHub: @harshuljain13 https://github.com/harshuljain13 - Newsletter: The Engineer's Digest https://harshuljain.substack.com — LLM inference, deeply explained The views, techniques, and opinions expressed in this handbook are solely those of the author and do not represent the views of Audible, Amazon, or any affiliated organization . No proprietary, confidential, or internal Amazon/Audible systems, data, or information has been included. All content is based on publicly available research, open-source tooling, and the author's independent experience and analysis. This handbook is provided for educational purposes only . Production infrastructure decisions should be validated against your specific workload, hardware, and organizational constraints. The author makes no guarantees about the accuracy, completeness, or fitness for purpose of any content herein. © 2026 Harshul Jain. All rights reserved. No part of this work — including the framework, diagrams, models, terminology, chapter structure, or related materials — may be reproduced, distributed, modified, adapted, or used in whole or in part without prior written permission from the author. This includes but is not limited to use in courses, training programs, consulting engagements, publications, presentations, software, or organizational materials. The framework presented in this work is the intellectual property of Harshul Jain. It may not be copied, adapted, taught, commercialized, incorporated into derivative works, or used in any professional, commercial, or organizational context — including consulting, training, software, presentations, publications, or organizational materials — without prior written permission. To request permission, open a GitHub Issue or contact via the profile above. This handbook builds on the work of many researchers and engineers: - The vLLM https://github.com/vllm-project/vllm team for PagedAttention and continuous batching - The SGLang https://github.com/sgl-project/sglang team for RadixAttention - Tri Dao for FlashAttention https://github.com/Dao-AILab/flash-attention - The authors of foundational papers: Attention Is All You Need, GQA, Medusa, EAGLE, and many others 📬 Stay updated — Subscribe to The Engineer's Digest https://harshuljain.substack.com for chapter releases and build-in-public updates. Built with ❤️ for the ML infrastructure community