The definitive guide to serving large language models in production.
Quick Start • Contents • Labs • Community • Contributing
LLM inference is hard. Not "read the docs and figure it out" hard — fundamentally different from everything else in ML hard.
Traditional ML inference is a solved problem. You batch requests, run a forward pass, return results. Latency is predictable, memory is fixed, scaling is linear.
LLM inference breaks all of these assumptions:
Latency is unpredictable— a 10-token response takes 100ms, a 1000-token response takes 10 seconds** Memory grows during requests**— the KV cache expands with every generated token** Scaling is sub-linear**— communication overhead dominates as you add GPUs** Cost is 100x higher**— $0.001/request becomes $0.10/request
This handbook exists because we needed it and couldn't find it. The knowledge is scattered across papers, blog posts, tribal knowledge, and source code comments. We've consolidated years of production experience and research into one comprehensive resource.
This is the guide we wish existed when we started.
📬
Follow the build— New chapters, explained in plain English with production context. Subscribe to[The Engineer's Digest]to get notified when new content drops.[Subscribe free →]
💬 Join the discussion #
— questions, feedback, and corrections welcome
| | | |
- Python 3.10+
- CUDA 12.0+ (for GPU labs)
- Basic PyTorch familiarity
git clone https://github.com/harshuljain13/llm-inference-at-scale.git
cd llm-inference-at-scale
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
open content/00_foundations/00.0_what_is_llm_inference/what_is_llm_inference.md
Or browse the Table of Contents below.
| Chapter | Title | Description |
|---|---|---|
| 0.0 | ||
Why LLM Inference is DifferentTransformer Inference Mechanics| Chapter | Title | Description | |---|---|---| | 1.1 | |
Roofline ModelFlashAttention| Chapter | Title | Description | |---|---|---| | 2.1 | |
Attention MechanismsPagedAttentionKV Cache Compression| Chapter | Title | Description | |---|---|---| | 3.1 | |
TurboQuantContinuous BatchingSpeculative DecodingChunked Prefill| Chapter | Title | Description | |---|---|---| | 4.1 | |
SGLangTensorRT-LLM| Chapter | Title | Description | |---|---|---| | 5.1 | |
MoE InferenceDistillation| Chapter | Title | Description | |---|---|---| | 6.1 | |
EKS + KServeSageMakerDisaggregated ServingCold Start| Chapter | Title | Description | |---|---|---| | 7.1 | |
Structured OutputEdge DeploymentHands-on exercises to reinforce each concept. Each lab includes starter code, step-by-step instructions, and solutions.
| Lab | Title | Prerequisites | Time |
|---|---|---|---|
| 01 | |||
VRAM CalculationQuantization ComparisonvLLM DeploymentSGLang Structured OutputTensor ParallelismRay Serve DeploymentEKS + KServeSageMaker ProductionBenchmarking SuiteHardware requirements: Most labs run on a single GPU (g5.xlarge or equivalent). Labs 06 and 08 require multi-GPU instances.
Formulas you'll use constantly when working with LLM inference:
The theoretical maximum decode speed, limited by how fast you can read model weights:
max_tokens_per_second = memory_bandwidth / model_size_bytes
Example: Llama 8B (16GB FP16) on A100 (2 TB/s) → 125 tokens/sec maximum
Memory required for the key-value cache:
kv_cache_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_bytes
Example: Llama 8B, batch=1, seq=4096, FP16 → 512 MB
Determines whether a workload is compute-bound or memory-bound:
arithmetic_intensity = FLOPs / bytes_transferred
Rule of thumb: Below the ridge point (~156 FLOPs/byte on A100) = memory-bound
llm-inference-at-scale/
├── content/ # 📖 Handbook chapters
│ ├── 00_foundations/ # Part I: Foundations
│ ├── 01_gpu_fundamentals/ # Part II: GPU Fundamentals
│ ├── 02_attention_and_kv/ # Part III: Attention & KV Cache
│ ├── 03_optimization/ # Part IV: Optimization Techniques
│ ├── 04_engines/ # Part V: Inference Engines
│ ├── 05_scaling/ # Part VI: Scaling
│ ├── 06_serving/ # Part VII: Production Serving
│ ├── 07_operations/ # Part VIII: Operations
│ └── utils/ # Visualization utilities
├── labs/ # 🧪 Hands-on exercises
├── reference/ # 📋 Quick references
│ ├── cheat_sheet.md # One-page summary
│ ├── glossary.md # Terminology
│ ├── vllm_quick_reference.md # vLLM commands
│ └── cost_calculator.py # Inference cost estimation
├── assets/ # 🎨 Images and diagrams
└── slides/ # 📊 Presentation materials
For engineers who need to deploy an LLM this week:
0.0 What is LLM Inference?— 15 min0.1 Why LLM Inference is Different— 20 min3.1 Quantization— 20 min4.1 vLLM— 30 minLab 04: vLLM Deployment— 45 min
For engineers building inference infrastructure:
Morning: Part I (Foundations) + Part II (GPU Fundamentals)Afternoon: Part IV (Optimization) + Part V (Engines)Labs: 01, 02, 03, 04
For teams standardizing on LLM serving:
Day 1: Parts I, II, III — Foundations through KV CacheDay 2: Parts IV, V, VI — Optimization through ScalingDay 3: Parts VII, VIII — Production Serving and OperationsLabs: All 10 labs
This material has been presented at:
More talks coming — if you'd like this at your conference or meetup, open an issue.
If you use this material in research or internal documentation, please cite:
@misc{llm-inference-at-scale,
title={LLM Inference at Scale: A Practitioner's Handbook},
author={Jain, Harshul},
year={2025},
url={https://github.com/harshuljain13/llm-inference-at-scale}
}
If you find this useful, please ⭐ the repo — it helps others discover it.
Contributions are welcome. This is a living document.
Fix errors— Typos, outdated information, incorrect formulas** Improve clarity**— Better explanations, additional examples** Add content**— New chapters, labs, or reference materials
- Fork the repository
- Create a feature branch (
git checkout -b improve-kv-cache-chapter
) - Make your changes
- Submit a pull request
To report errors or suggest corrections, open a GitHub Issue.
Harshul Jain is a Senior ML Infrastructure Engineer at Audible (Amazon), where he owns the ML Feature Store, a GenAI semantic search platform serving millions of customers, and real-time streaming pipelines at scale. He has been building and operating ML infrastructure in production for 4+ years and mentors 300+ engineers through an eMentoring program.
- GitHub: @harshuljain13 - Newsletter: The Engineer's Digest— LLM inference, deeply explained
The views, techniques, and opinions expressed in this handbook are solely those of the author and do not represent the views of Audible, Amazon, or any affiliated organization. No proprietary, confidential, or internal Amazon/Audible systems, data, or information has been included. All content is based on publicly available research, open-source tooling, and the author's independent experience and analysis.
This handbook is provided for educational purposes only. Production infrastructure decisions should be validated against your specific workload, hardware, and organizational constraints. The author makes no guarantees about the accuracy, completeness, or fitness for purpose of any content herein.
© 2026 Harshul Jain. All rights reserved.
No part of this work — including the framework, diagrams, models, terminology, chapter structure, or related materials — may be reproduced, distributed, modified, adapted, or used in whole or in part without prior written permission from the author. This includes but is not limited to use in courses, training programs, consulting engagements, publications, presentations, software, or organizational materials.
The framework presented in this work is the intellectual property of Harshul Jain. It may not be copied, adapted, taught, commercialized, incorporated into derivative works, or used in any professional, commercial, or organizational context — including consulting, training, software, presentations, publications, or organizational materials — without prior written permission.
To request permission, open a GitHub Issue or contact via the profile above.
This handbook builds on the work of many researchers and engineers:
- The vLLMteam for PagedAttention and continuous batching - The SGLangteam for RadixAttention - Tri Dao for FlashAttention - The authors of foundational papers: Attention Is All You Need, GQA, Medusa, EAGLE, and many others
📬 Stay updated — Subscribe to The Engineer's Digest for chapter releases and build-in-public updates.
Built with ❤️ for the ML infrastructure community