# Free LLM inference handbook: 100 engineers cloned it in week 1

> Source: <https://github.com/harshuljain13/llm-inference-at-scale>
> Published: 2026-06-06 12:37:09+00:00

**The definitive guide to serving large language models in production.**

[Quick Start](#-quick-start) •
[Contents](#-table-of-contents) •
[Labs](#-labs) •
[Community](#-community) •
[Contributing](#-contributing)

LLM inference is hard. Not "read the docs and figure it out" hard — **fundamentally different from everything else in ML** hard.

Traditional ML inference is a solved problem. You batch requests, run a forward pass, return results. Latency is predictable, memory is fixed, scaling is linear.

LLM inference breaks all of these assumptions:

**Latency is unpredictable**— a 10-token response takes 100ms, a 1000-token response takes 10 seconds** Memory grows during requests**— the KV cache expands with every generated token** Scaling is sub-linear**— communication overhead dominates as you add GPUs** Cost is 100x higher**— $0.001/request becomes $0.10/request

This handbook exists because we needed it and couldn't find it. The knowledge is scattered across papers, blog posts, tribal knowledge, and source code comments. We've consolidated years of production experience and research into one comprehensive resource.

**This is the guide we wish existed when we started.**

📬

Follow the build— New chapters, explained in plain English with production context. Subscribe to[The Engineer's Digest]to get notified when new content drops.[Subscribe free →]

## 💬 [Join the discussion](https://github.com/harshuljain13/llm-inference-at-scale/discussions)
— questions, feedback, and corrections welcome

|
|
|
|

- Python 3.10+
- CUDA 12.0+ (for GPU labs)
- Basic PyTorch familiarity

```
git clone https://github.com/harshuljain13/llm-inference-at-scale.git
cd llm-inference-at-scale

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
# Open the first chapter
open content/00_foundations/00.0_what_is_llm_inference/what_is_llm_inference.md
```

Or browse the [Table of Contents](#-table-of-contents) below.

| Chapter | Title | Description |
|---|---|---|
| 0.0 |
|

[Why LLM Inference is Different](/harshuljain13/llm-inference-at-scale/blob/master/content/00_foundations/00.1_why_llm_inference_is_different/why_llm_inference_is_different.md)[Transformer Inference Mechanics](/harshuljain13/llm-inference-at-scale/blob/master/content/00_foundations/00.2_transformer_inference_basics/transformer_inference_basics.md)| Chapter | Title | Description |
|---|---|---|
| 1.1 |
|

[Roofline Model](/harshuljain13/llm-inference-at-scale/blob/master/content/01_gpu_fundamentals/01.2_roofline_model/roofline_model.md)[FlashAttention](/harshuljain13/llm-inference-at-scale/blob/master/content/01_gpu_fundamentals/01.3_flash_attention/flash_attention.md)| Chapter | Title | Description |
|---|---|---|
| 2.1 |
|

[Attention Mechanisms](/harshuljain13/llm-inference-at-scale/blob/master/content/02_attention_and_kv/02.2_attention_mechanisms/attention_mechanisms.md)[PagedAttention](/harshuljain13/llm-inference-at-scale/blob/master/content/02_attention_and_kv/02.3_paged_attention/paged_attention.md)[KV Cache Compression](/harshuljain13/llm-inference-at-scale/blob/master/content/02_attention_and_kv/02.4_kv_cache_compression/kv_cache_compression.md)| Chapter | Title | Description |
|---|---|---|
| 3.1 |
|

[TurboQuant](/harshuljain13/llm-inference-at-scale/blob/master/content/03_optimization/03.2_turboquant/turboquant.md)[Continuous Batching](/harshuljain13/llm-inference-at-scale/blob/master/content/03_optimization/03.3_continuous_batching/continuous_batching.md)[Speculative Decoding](/harshuljain13/llm-inference-at-scale/blob/master/content/03_optimization/03.4_speculative_decoding/speculative_decoding.md)[Chunked Prefill](/harshuljain13/llm-inference-at-scale/blob/master/content/03_optimization/03.5_chunked_prefill/chunked_prefill.md)| Chapter | Title | Description |
|---|---|---|
| 4.1 |
|

[SGLang](/harshuljain13/llm-inference-at-scale/blob/master/content/04_engines/04.2_sglang/sglang.md)[TensorRT-LLM](/harshuljain13/llm-inference-at-scale/blob/master/content/04_engines/04.3_tensorrt_llm/tensorrt_llm.md)| Chapter | Title | Description |
|---|---|---|
| 5.1 |
|

[MoE Inference](/harshuljain13/llm-inference-at-scale/blob/master/content/05_scaling/05.2_moe_inference/moe_inference.md)[Distillation](/harshuljain13/llm-inference-at-scale/blob/master/content/05_scaling/05.3_distillation)| Chapter | Title | Description |
|---|---|---|
| 6.1 |
|

[EKS + KServe](/harshuljain13/llm-inference-at-scale/blob/master/content/06_serving/06.2_eks_kserve/eks_kserve.md)[SageMaker](/harshuljain13/llm-inference-at-scale/blob/master/content/06_serving/06.3_sagemaker/sagemaker.md)[Disaggregated Serving](/harshuljain13/llm-inference-at-scale/blob/master/content/06_serving/06.4_disaggregated_serving/disaggregated_serving.md)[Cold Start](/harshuljain13/llm-inference-at-scale/blob/master/content/06_serving/06.5_cold_start/cold_start.md)| Chapter | Title | Description |
|---|---|---|
| 7.1 |
|

[Structured Output](/harshuljain13/llm-inference-at-scale/blob/master/content/07_operations/07.2_structured_output)[Edge Deployment](/harshuljain13/llm-inference-at-scale/blob/master/content/07_operations/07.3_edge_deployment/edge_deployment.md)Hands-on exercises to reinforce each concept. Each lab includes starter code, step-by-step instructions, and solutions.

| Lab | Title | Prerequisites | Time |
|---|---|---|---|
| 01 |
|

[VRAM Calculation](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_02_vram_calculation)[Quantization Comparison](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_03_quantization_comparison)[vLLM Deployment](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_04_vllm_deployment)[SGLang Structured Output](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_05_sglang_structured_output)[Tensor Parallelism](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_06_tensor_parallelism)[Ray Serve Deployment](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_07_ray_serve_deployment)[EKS + KServe](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_08_eks_kserve_deployment)[SageMaker Production](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_09_sagemaker_production)[Benchmarking Suite](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_10_benchmarking_monitoring)**Hardware requirements:** Most labs run on a single GPU (g5.xlarge or equivalent). Labs 06 and 08 require multi-GPU instances.

Formulas you'll use constantly when working with LLM inference:

The theoretical maximum decode speed, limited by how fast you can read model weights:

```
max_tokens_per_second = memory_bandwidth / model_size_bytes
```

**Example:** Llama 8B (16GB FP16) on A100 (2 TB/s) → 125 tokens/sec maximum

Memory required for the key-value cache:

```
kv_cache_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × dtype_bytes
```

**Example:** Llama 8B, batch=1, seq=4096, FP16 → 512 MB

Determines whether a workload is compute-bound or memory-bound:

```
arithmetic_intensity = FLOPs / bytes_transferred
```

**Rule of thumb:** Below the ridge point (~156 FLOPs/byte on A100) = memory-bound

```
llm-inference-at-scale/
├── content/                      # 📖 Handbook chapters
│   ├── 00_foundations/           #    Part I: Foundations
│   ├── 01_gpu_fundamentals/      #    Part II: GPU Fundamentals
│   ├── 02_attention_and_kv/      #    Part III: Attention & KV Cache
│   ├── 03_optimization/          #    Part IV: Optimization Techniques
│   ├── 04_engines/               #    Part V: Inference Engines
│   ├── 05_scaling/               #    Part VI: Scaling
│   ├── 06_serving/               #    Part VII: Production Serving
│   ├── 07_operations/            #    Part VIII: Operations
│   └── utils/                    #    Visualization utilities
├── labs/                         # 🧪 Hands-on exercises
├── reference/                    # 📋 Quick references
│   ├── cheat_sheet.md            #    One-page summary
│   ├── glossary.md               #    Terminology
│   ├── vllm_quick_reference.md   #    vLLM commands
│   └── cost_calculator.py        #    Inference cost estimation
├── assets/                       # 🎨 Images and diagrams
└── slides/                       # 📊 Presentation materials
```

For engineers who need to deploy an LLM this week:

[0.0 What is LLM Inference?](/harshuljain13/llm-inference-at-scale/blob/master/content/00_foundations/00.0_what_is_llm_inference/what_is_llm_inference.md)— 15 min[0.1 Why LLM Inference is Different](/harshuljain13/llm-inference-at-scale/blob/master/content/00_foundations/00.1_why_llm_inference_is_different/why_llm_inference_is_different.md)— 20 min[3.1 Quantization](/harshuljain13/llm-inference-at-scale/blob/master/content/03_optimization/03.1_quantization/quantization.md)— 20 min[4.1 vLLM](/harshuljain13/llm-inference-at-scale/blob/master/content/04_engines/04.1_vllm/vllm.md)— 30 min[Lab 04: vLLM Deployment](/harshuljain13/llm-inference-at-scale/blob/master/labs/lab_04_vllm_deployment)— 45 min

For engineers building inference infrastructure:

**Morning:** Part I (Foundations) + Part II (GPU Fundamentals)**Afternoon:** Part IV (Optimization) + Part V (Engines)**Labs:** 01, 02, 03, 04

For teams standardizing on LLM serving:

**Day 1:** Parts I, II, III — Foundations through KV Cache**Day 2:** Parts IV, V, VI — Optimization through Scaling**Day 3:** Parts VII, VIII — Production Serving and Operations**Labs:** All 10 labs

This material has been presented at:

*More talks coming — if you'd like this at your conference or meetup, open an issue.*

If you use this material in research or internal documentation, please cite:

```
@misc{llm-inference-at-scale,
  title={LLM Inference at Scale: A Practitioner's Handbook},
  author={Jain, Harshul},
  year={2025},
  url={https://github.com/harshuljain13/llm-inference-at-scale}
}
```

If you find this useful, please ⭐ the repo — it helps others discover it.

Contributions are welcome. This is a living document.

**Fix errors**— Typos, outdated information, incorrect formulas** Improve clarity**— Better explanations, additional examples** Add content**— New chapters, labs, or reference materials

- Fork the repository
- Create a feature branch (
`git checkout -b improve-kv-cache-chapter`

) - Make your changes
- Submit a pull request

To report errors or suggest corrections, open a GitHub Issue.

**Harshul Jain** is a Senior ML Infrastructure Engineer at Audible (Amazon), where he owns the ML Feature Store, a GenAI semantic search platform serving millions of customers, and real-time streaming pipelines at scale. He has been building and operating ML infrastructure in production for 4+ years and mentors 300+ engineers through an eMentoring program.

- GitHub:
[@harshuljain13](https://github.com/harshuljain13) - Newsletter:
[The Engineer's Digest](https://harshuljain.substack.com)— LLM inference, deeply explained

The views, techniques, and opinions expressed in this handbook are solely those of the author and **do not represent the views of Audible, Amazon, or any affiliated organization**. No proprietary, confidential, or internal Amazon/Audible systems, data, or information has been included. All content is based on publicly available research, open-source tooling, and the author's independent experience and analysis.

This handbook is provided for **educational purposes only**. Production infrastructure decisions should be validated against your specific workload, hardware, and organizational constraints. The author makes no guarantees about the accuracy, completeness, or fitness for purpose of any content herein.

© 2026 Harshul Jain. All rights reserved.

No part of this work — including the framework, diagrams, models, terminology, chapter structure, or related materials — may be reproduced, distributed, modified, adapted, or used in whole or in part without prior written permission from the author. This includes but is not limited to use in courses, training programs, consulting engagements, publications, presentations, software, or organizational materials.

The framework presented in this work is the intellectual property of Harshul Jain. It may not be copied, adapted, taught, commercialized, incorporated into derivative works, or used in any professional, commercial, or organizational context — including consulting, training, software, presentations, publications, or organizational materials — without prior written permission.

To request permission, open a GitHub Issue or contact via the profile above.

This handbook builds on the work of many researchers and engineers:

- The
[vLLM](https://github.com/vllm-project/vllm)team for PagedAttention and continuous batching - The
[SGLang](https://github.com/sgl-project/sglang)team for RadixAttention - Tri Dao for
[FlashAttention](https://github.com/Dao-AILab/flash-attention) - The authors of foundational papers: Attention Is All You Need, GQA, Medusa, EAGLE, and many others

📬 **Stay updated** — [Subscribe to The Engineer's Digest](https://harshuljain.substack.com) for chapter releases and build-in-public updates.

**Built with ❤️ for the ML infrastructure community**