{"slug": "learn-llm-inference-system-on-apple-silicon", "title": "Learn LLM inference system on Apple Silicon", "summary": "Systems engineer Chi published tiny-llm, a four-week hands-on course that teaches LLM inference end to end on Apple Silicon by having students implement a Qwen3-4B serving stack in Python, C++, or Metal on MLX arrays rather than calling optimized MLX operations. The course runs from matmul-to-text in Week 1 through KV-cache and quantized-matvec optimizations in Week 2, continuous batching and paged KV in a \"mini vLLM\" in Week 3, and a bounded coding agent in Week 4, with Days 1 through 9 of Week 4 currently available. The book is published at skyzh.github.io/tiny-llm, and Qwen3-4B was chosen because its grouped-query attention, QK normalization, BF16 activations and 4-bit weights expose real weight-bandwidth, attention and cache costs while remaining iterable on one machine without a CUDA GPU.", "body_md": "tiny-llm is a hands-on course for systems engineers who want to understand LLM\ninference end to end. You can think of it as an LLM-serving counterpart to\nCMU's [Needle](https://github.com/dlsyscourse/hw1/tree/main/python/needle)\nproject: build the path that loads a Qwen3 model, turns tokens into logits, and\ngenerates text.\n\nThe course begins with array and matrix operations, then introduces kernels and serving machinery as the running model needs them. Keeping the implementation small enough to read end to end makes it possible to connect the equations to memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling.\n\nThe course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, your solution implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline.\n\nThe course follows a four-week learning path:\n\n- **Week 1: From Matmul to Text.** Build a Qwen3 model directly from`mlx.core` array operations: attention, RoPE, GQA, RMSNorm, the MLP, sampling, and\nthe autoregressive loop.\n- **Week 2: A Faster Single Request.** Day 1 adds`kv-cache` and\nrequest-bounded`capacity-cache` ; Day 2 keeps W4 projection weights packed\nthrough the`quantized-matvec` checkpoint. Day 3 adds SIMD matrix prefill at`simd-matmul` . Day 4 adds cumulative RMSNorm, RoPE, and SwiGLU checkpoints.\nDay 5 adds`tiled-prefill` and runs the completed`selected` model.\n- **Week 3: Build a Mini vLLM.** Introduce continuous\nbatching and chunked admission, then make paged KV the canonical serving\nlayout. Decode attention and FlashAttention learn to read pages directly so\nthe scheduler does not rebuild dense history on every step.\n- **Week 4: Build a Coding Agent.** Start with a bounded, validated agent loop,\nthen connect it to a small workspace. The course is publishing one reviewed\ncheckpoint at a time; Days 1 through 9 now cover inspection, approved edits,\none validation command, simple effect receipts, and one visible\ncheckpoint-and-resume boundary, receipt-backed context compaction, and one\nvisible inspect-and-steer pause, and deterministic evaluation of observable\noutcomes, then tokenizer/KV-prefix reuse for two isolated steered branches\nand one explicit evidence-backed selection, followed by bounded,\nrange-retrievable evidence for oversized tool results.\n\nApple silicon provides a practical local environment with one shared memory space and direct access to Metal kernels. Students can inspect the complete path on one machine instead of depending on an expensive CUDA GPU setup.\n\nQwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. Its grouped-query attention, QK normalization, BF16 activations, and 4-bit weights also keep the exercises close to current model-serving work.\n\nThe book is published at\n[skyzh.github.io/tiny-llm](https://skyzh.github.io/tiny-llm/). Begin with the\n[environment setup](https://skyzh.github.io/tiny-llm/setup.html), or verify an\nexisting checkout with:\n\n```\npdm install -v\npdm run check-installation\npdm run test-refsol -- -- -k week_1\n```\n\nThe `tiny_llm` package is where students implement the exercises.\n`tiny_llm_ref` contains the reference solution used by the tests and benchmark\nappendix. The [book summary](https://github.com/skyzh/tiny-llm/blob/main/book/src/SUMMARY.md) lists the chapter order;\nimplementation, test, and publication readiness is tracked below.\n\nThe table tracks implementation (`Code`), tests (` Test`), rendered chapters (` Doc`), and Chi's review of learner-facing material (` Audit`). Week 4 is publishing one reviewed day at a time; Days 1 through 9 are currently available to learners. The Audit column reflects Chi's personal editorial pass on the published course content and is independent of code/test/doc readiness.\n\nDay 3 can send file contents to the model, modify files after approval, and run\none exact configured command. Use a disposable workspace without secrets and\nread the [Week 4 overview](https://github.com/skyzh/tiny-llm/blob/main/book/src/week4-overview.md) before running the loop.\nDay 4 checkpoints a complete tool-observation boundary with the scripted\nmodel's fake cache metadata, then restores a fresh model without replaying the\ncompleted edit or command.\nDay 5 compacts older completed effects in the model-visible transcript while\ntheir exact receipts retain the full action, result, and changed artifacts.\nDay 6 inspects one complete-observation checkpoint, appends one visible operator\ninstruction, and resumes a fresh model without replaying the completed effect.\nDay 7 evaluates one completed run from declared final, file, result, and receipt\nfacts without grading hidden reasoning or exact transcript shape.\nDay 8 reuses one real tokenizer/KV checkpoint for two differently steered,\neffect-isolated continuations, evaluates both with Day 7's harness, and makes\none explicit passing selection without pretending completed effects were\nrewound.\nDay 9 stores exact oversized tool-result bytes outside the model prompt, shows\na bounded identity/digest/head-tail observation, and lets the model retrieve\none explicit byte range through the existing loop.\n\n| Week + Chapter | Topic | Code | Test | Doc | Audit | \n|---|---|---|---|---|---|\n| 1.1 | Attention | ✅ | ✅ | ✅ | ✅ | \n| 1.2 | RoPE | ✅ | ✅ | ✅ | ✅ | \n| 1.3 | Grouped Query Attention | ✅ | ✅ | ✅ | ✅ | \n| 1.4 | RMSNorm and MLP | ✅ | ✅ | ✅ | ✅ | \n| 1.5 | Load the Model | ✅ | ✅ | ✅ | ✅ | \n| 1.6 | Generate Responses (aka Decoding) | ✅ | ✅ | ✅ | ✅ | \n| 1.7 | Sampling | ✅ | ✅ | ✅ | ✅ | \n| 2.1 | Cache and Measure ( `kv-cache` ,`capacity-cache` ) | 🚧 | 🚧 | ✅ | 🚧 | \n| 2.2 | Keep W4 Packed ( `quantized-matvec` ) | 🚧 | 🚧 | ✅ | 🚧 | \n| 2.3 | SIMD Matrix Prefill ( `simd-matmul` ) | 🚧 | 🚧 | ✅ | 🚧 | \n| 2.4 | Fused Model Primitives ( `rmsnorm` ,`rope` ,`swiglu` ) | 🚧 | 🚧 | ✅ | 🚧 | \n| 2.5 | Tiled Dense Prefill Attention ( `tiled-prefill` ,`selected` ) | 🚧 | 🚧 | ✅ | 🚧 | \n| 3.1 | Continuous Batching | ✅ | ✅ | ✅ | 🚧 | \n| 3.2 | Chunked Prefill | ✅ | ✅ | ✅ | 🚧 | \n| 3.3 | Paged KV Cache | ✅ | ✅ | ✅ | 🚧 | \n| 3.4 | Direct Paged Attention | ✅ | ✅ | ✅ | 🚧 | \n| 3.5 | Paged FlashAttention | ✅ | ✅ | ✅ | 🚧 | \n| 3.6 (optional) | MoE (Mixture of Experts) | ✅ | ✅ | ✅ | 🚧 | \n| 3.7 (optional) | Speculative Decoding | ✅ | ✅ | ✅ | 🚧 | \n| 4.1 | Validated Agent Loop | ✅ | ✅ | ✅ | 🚧 | \n| 4.2 | Inspect a Workspace | ✅ | ✅ | ✅ | 🚧 | \n| 4.3 | Edit, Validate, and Record | ✅ | ✅ | ✅ | 🚧 | \n| 4.4 | Checkpoint and Resume | ✅ | ✅ | ✅ | 🚧 | \n| 4.5 | Compact Completed Work | ✅ | ✅ | ✅ | 🚧 | \n| 4.6 | Inspect and Steer a Paused Agent | ✅ | ✅ | ✅ | 🚧 | \n| 4.7 | Evaluate Observable Outcomes | ✅ | ✅ | ✅ | 🚧 | \n| 4.8 | Fork, Steer, and Select | ✅ | ✅ | ✅ | 🚧 | \n| 4.9 | Bound Tool Evidence | ✅ | ✅ | ✅ | 🚧 | \n\nEarlier Week 2 chapter URLs not reused by active Days 1–5 remain available as\n[historical material](https://github.com/skyzh/tiny-llm/blob/main/book/src/week2-02-benchmark-profile.md). The former Day 4\naddress now serves the [active fused-primitives lesson](https://github.com/skyzh/tiny-llm/blob/main/book/src/week2-04-fused-model-kernels.md).\nThose historical pages describe an earlier test and checkpoint order, separate\nfrom the current Day 1 → Day 2 → Day 3 → Day 4 → Day 5 route.\n\nOther topics not covered include quantized or compressed KV caches, cross-request prefix caching, fine-tuning, and long-context techniques.\n\nJoin skyzh's Discord server to study with the tiny-llm community.", "url": "https://wpnews.pro/news/learn-llm-inference-system-on-apple-silicon", "canonical_source": "https://github.com/skyzh/tiny-llm", "published_at": "2026-10-30 11:09:12+00:00", "updated_at": "2026-10-30 11:16:47.622688+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops", "ai-agents"], "entities": ["tiny-llm", "Chi", "Apple Silicon", "MLX", "Qwen3-4B", "Metal", "vLLM", "skyzh.github.io/tiny-llm"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/learn-llm-inference-system-on-apple-silicon", "markdown": "https://wpnews.pro/news/learn-llm-inference-system-on-apple-silicon.md", "text": "https://wpnews.pro/news/learn-llm-inference-system-on-apple-silicon.txt", "jsonld": "https://wpnews.pro/news/learn-llm-inference-system-on-apple-silicon.jsonld"}}