Status: first release, November 2026. Every section was researched against primary sources and reviewed by the author, and CI checks every change: links, structure, diagrams, and that every lab runs. The few claims that couldn't be confirmed against a primary source are listed, with the sections that use them, in PLAN.md. Reviews and corrections are welcome; please include a source.
Learn how to design systems built on large language models: the trade-offs, the components, and the patterns that have converged across the industry, along with the ones that are still moving.
Learn how to design AI systems that hold up in production. Prepare for the AI system design conversation.
Building on models is now a core engineering skill, and the field moves fast. Most of what's written about it is a product announcement, a framework tutorial or a single team's experience. This primer is an organised collection of what has converged across providers and researchers, with every trade-off named and every number dated and sourced.
It is modelled on Donne Martin's System Design Primer: the same arc of method, trade-offs, components, solved designs and reference numbers, rebuilt for systems whose core component is probabilistic, priced per token, and open to prompt injection.
Learn how to design AI systems. Each topic is a short page you can read in ten minutes, with a diagram, when to use it, when not to, and its disadvantages. A deep dive per topic holds the provider survey, numbers, risk controls and cloud mappings.
Prepare for the AI system design conversation, in an interview or a design review. Work through the method, the solved designs and the practice questions.
Every page ends in question-and-answer flashcards. Build the deck with either command (standard library only):
python3 scripts/primer.py anki # writes build/ai-system-design-primer.apkg: double-click to import
python3 scripts/primer.py flashcards # writes build/flashcards.tsv: plain text for Anki's File > Import
The package has one sub-deck per part. Re-importing a newer build updates answers in place and keeps
your review history; a reworded question arrives as a new card. Cards are tagged by section and
maturity, so you can study one part, or suspend maturity::frontier cards when they go stale.
Most topics come with a small, runnable lab: a simulation or calculation, numpy or the standard library only, with the prediction written down before the run and an honest note when it was wrong.
| Lab | Question |
|---|---|
| 0.1 | What a token is, and what temperature does |
| 0.2 | Prefill, decode, the KV cache and reasoning tokens |
| 0.3 | The tool-calling loop, its token bill, and what checks catch |
| 2.1 | When does routing between models pay off? |
| 2.2 | What autonomy costs in reliability and predictability |
| 2.3 | When is the whole corpus in the prompt cheaper than retrieval? |
| 2.4 | Count to three, then size what a detector leaves behind |
| 2.5 | One agent or several? |
| 2.7 | Batching, per-user speed, and why you can't run at 100% |
| 3.2 | What a guardrail's error rates mean at volume |
| 3.3 | What each context strategy costs, and what it forgets |
| 3.4 | Keyword vs dense vs hybrid retrieval |
| 3.5 | Single-shot retrieval, query fan-out, or an agent loop? |
| 3.6 | Approximate search and filtering |
| 3.7 | Picking the cheapest model that "meets the bar", with a noisy eval |
| 3.8 | Cache arithmetic and when speculation pays |
| 3.9 | An evaluator loop is only as good as its evaluator |
| 3.10 | What does connecting more tools cost, and does tool search pay? |
| 3.11 | What should an agent remember, and how? |
| 3.12 | How far does a quote check get you? |
| 3.13 | When a prompt cache pays, and what a semantic cache gets wrong |
| 3.14 | Retries during an overload, and checkpoints for long runs |
| 3.15 | How many eval cases do you need? |
| 3.16 | Which agent traces should you keep? |
| 3.17 | Pin your tools, and see what pinning can't see |
| 3.18 | What LoRA trains, and when a fine-tune pays for itself |
| 4.1 | How many accelerators does a self-hosted inference API need? |
| 4.2 | Where the money goes in an enterprise RAG assistant |
| 4.4 | What 98% recall costs in lawyer time |
| 4.5 | Consistency with policy in the prompt or in a policy engine |
| 4.6 | Review capacity caps merged work |
Learn from the community.
Pull requests are welcome to fix errors, improve sections and add new ones. Read STYLE.md for how sections are researched and written (learner-first, several providers per topic, every claim sourced) and CONTRIBUTING.md for the mechanics. Decisions about the primer itself are recorded in adr/. Work still to do is in ROADMAP.md.
Summaries of AI system design topics, including pros and cons. Everything is a trade-off.
Each section links to its deep dive. Maturity is marked S (stable core), S/F (stable concept,
volatile specifics) or F (frontier, expect rewrites).
Part 0: Orientation
- 0 Orientation
S - 0.1 Tokens, context and sampling
S - 0.2 Inside a model call: prefill, decode and reasoning
S - 0.3 Tool calling and structured output
S
Part 1: Method
Part 2: Trade-offs
- 2.1 Cost vs quality vs latency
S·deep dive - 2.2 Determinism vs autonomy
S/F·deep dive - 2.3 Context stuffing vs retrieval
S/F·deep dive - 2.4 Capability vs security
S/F·deep dive - 2.5 Single-agent vs multi-agent
S/F·deep dive - 2.6 Prompting vs RAG vs fine-tuning
S·deep dive - 2.7 Throughput vs per-request latency in serving
S·deep dive
Part 3: Components, in request order
- 3.1 AI gateway and router
S·deep dive - 3.2 Input guardrails
S·deep dive - 3.3 Context engineering
S/F·deep dive - 3.4 Retrieval pipeline
S·deep dive - 3.5 Agentic retrieval and the long-context boundary
F·deep dive - 3.6 Vector stores
S/F·deep dive - 3.7 Model layer: selection, structured outputs, streaming
S/F·deep dive - 3.8 Serving internals
S/F·deep dive - 3.9 Workflows and agents
S·deep dive - 3.10 Multi-agent topologies and protocols (MCP, A2A)
F·deep dive - 3.11 Agent memory
F·deep dive - 3.12 Output guardrails and validation
S·deep dive - 3.13 Caching
S/F·deep dive - 3.14 Asynchronism: batch, durable execution, back-pressure
S·deep dive - 3.15 Evals
S·deep dive - 3.16 Observability
S/F·deep dive - 3.17 Security
S/F·deep dive - 3.18 Model adaptation
S·deep dive
Part 4: Solved designs
Part 5: Appendix
Suggested topics to review based on your timeline.
| Short (days) | Medium (weeks) | Long (months) | |
|---|---|---|---|
| Read Orientation , the foundations (0.1 –0.3 ) and themethod | ✓ | ✓ | ✓ |
| Read the trade-offs (Part 2) | Some | All | All |
| Read the components (Part 3) | 3.1, 3.4, 3.9, 3.15, 3.17 | Most | All, plus deep dives |
| Read Numbers and theGlossary | ✓ | ✓ | ✓ |
| Work the solved designs (Part 4) | Some | Many | All |
| Run the labs and their exercises | Some | Many | All |
| Practise the additional questions | Some | Many | Most |
| Read real-world architectures | Some | Many | All |
| Map your platform with cloud andgovernance crosswalks | — | Some | All |
How to tackle an AI system design question, in an interview or a design review.
The classic four steps, with two changes: the quality bar comes first, because a model-backed feature is right only some of the time; and hardening is its own step, because prompt injection, data leakage and runaway agents are design constraints, not add-ons.
- Use cases, constraints and the quality bar. Who, what, how many, how fast, what data, and how you'll know it works.
- Back-of-the-envelope. Tokens per request, requests per second, cost per day, latency.
- High-level design on the complexity ladder. Single call → workflow → single agent → multi-agent. Start at the lowest rung that meets the bar.
- Core components. Retrieval, context, tools, model choice, output checks.
- Harden. Which legs of the lethal trifecta are present, and which does the design break?
- Scale and optimise. Bottlenecks in order; caching, routing, batching.
Full walkthrough with a worked example: How to approach an AI system design question.
Common AI system design questions with sample discussions, numbers and diagrams.
| Question | |
|---|---|
| Design an LLM inference API | Solution |
| Design an enterprise RAG assistant | Solution |
| Design an enterprise AI gateway | Solution |
| Design a regulated document-review assistant | Solution |
| Design a customer-support agent | Solution |
| Design a coding agent platform | Solution |
| Design an eval and release pipeline | Solution |
| Add an AI system design question | Contribute |
More practice: additional design questions.
-
Numbers every AI engineer should know : tokens, speed, cost ratios, memory, retrieval and eval figures, each dated and sourced.
-
Cloud mapping : each capability on Azure, AWS, Google Cloud, OpenAI, Anthropic and open source.
-
Pattern crosswalk : the same patterns under each provider's and author's names.
-
Governance crosswalk : OWASP LLM and Agentic Top 10, FINOS, EU AI Act, mapped to primer sections.
-
Real-world architectures : engineering write-ups and independent studies.
-
Glossary and changelog.
-
Trade-offs first. Every pattern says which trade-off it moves and in which direction.
-
Quality bar first. Evals are part of the method, not an afterthought.
-
Maturity is labelled and dated. Each page carries
last_verifiedandreview_by; the checker fails when a review is overdue. Frontier pages are reviewed every three months. -
Several providers on every topic. Anthropic, OpenAI, Google, AWS and Microsoft, with the names each uses.
-
Vendor-neutral prose. Product names live in the deep dives and the cloud mapping.
-
Risk and controls on every topic, with the evidence an auditor would ask for.
-
Runnable labs with predictions recorded before the run.
| Phase | Sections | Planned | Draft | Review | Published |
|---|---|---|---|---|---|
| 0 | 6 | 0 | 0 | 0 | 6 |
| 1 | 12 | 0 | 0 | 2 | 10 |
| 2 | 10 | 0 | 0 | 2 | 8 |
| 3 | 11 | 0 | 0 | 1 | 10 |
| 4 | 5 | 0 | 0 | 2 | 3 |
See ROADMAP.md for what's next and PLAN.md for the build plan and the verification debt (claims still waiting for a first-party source).
- Donne Martin's System Design Primer , whose structure this follows.
- The provider guides, papers and engineering write-ups cited in each section; see real-world architectures .
Open an issue to discuss errors, missing topics or new solved designs. Report security problems, or guidance that would make a system less safe, privately as described in SECURITY.md. Participation is covered by the Code of Conduct.
- Content (pages, diagrams, flashcards, lab READMEs) is licensed underCreative Commons Attribution 4.0 International (CC BY 4.0), the same licence as the original System Design Primer.
- All diagrams and charts are original to this repository and covered by the content licence.
- Code (
scripts/, the lab scripts and the CI workflow) is licensed under theMIT License . - Contributions are accepted under the same terms. The reasoning is in ADR 0008 .
The author researched and wrote this primer with an AI assistant, and reviewed every section. The
method is in STYLE.md: primary sources from several providers for every claim, a dated
last_verified on every page, runnable labs instead of asserted numbers, and a public list of what
isn't yet verified. Mistakes are still possible; please report them with a source.