Show HN: AI System Design Primer

A developer released the AI System Design Primer in November 2026, a first-release educational resource modeled on Donne Martin's System Design Primer that teaches engineers to design production systems built on large language models. The primer covers method, trade-offs, components, solved designs and dated reference numbers, with each topic a roughly ten-minute page plus a deep dive, and it ships runnable numpy-or-standard-library labs and Anki flashcard decks generated by a standard-library Python script. The author states every section was researched against primary sources and reviewed, with CI checking links, structure, diagrams and that every lab runs, and unconfirmed claims listed in PLAN.md under verification debt.

Status: first release, November 2026. Every section was researched against primary sources and reviewed by the author, and CI checks every change: links, structure, diagrams, and that every lab runs. The few claims that couldn't be confirmed against a primary source are listed, with the sections that use them, in PLAN.md https://github.com/daneb/ai-system-design-primer/blob/master/PLAN.md verification-debt . Reviews and corrections are welcome; please include a source. Learn how to design systems built on large language models: the trade-offs, the components, and the patterns that have converged across the industry, along with the ones that are still moving. Learn how to design AI systems that hold up in production. Prepare for the AI system design conversation. Building on models is now a core engineering skill, and the field moves fast. Most of what's written about it is a product announcement, a framework tutorial or a single team's experience. This primer is an organised collection of what has converged across providers and researchers, with every trade-off named and every number dated and sourced. It is modelled on Donne Martin's System Design Primer https://github.com/donnemartin/system-design-primer : the same arc of method, trade-offs, components, solved designs and reference numbers, rebuilt for systems whose core component is probabilistic, priced per token, and open to prompt injection. Learn how to design AI systems. Each topic is a short page you can read in ten minutes, with a diagram, when to use it, when not to, and its disadvantages. A deep dive per topic holds the provider survey, numbers, risk controls and cloud mappings. Prepare for the AI system design conversation, in an interview or a design review. Work through the method https://github.com/daneb/ai-system-design-primer/blob/master/docs/1-method/1-method.md , the solved designs ai-system-design-questions-with-solutions and the practice questions https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.8-unsolved-questions.md . Every page ends in question-and-answer flashcards. Build the deck with either command standard library only : python3 scripts/primer.py anki writes build/ai-system-design-primer.apkg: double-click to import python3 scripts/primer.py flashcards writes build/flashcards.tsv: plain text for Anki's File Import The package has one sub-deck per part. Re-importing a newer build updates answers in place and keeps your review history; a reworded question arrives as a new card. Cards are tagged by section and maturity, so you can study one part, or suspend maturity::frontier cards when they go stale. Most topics come with a small, runnable lab: a simulation or calculation, numpy or the standard library only, with the prediction written down before the run and an honest note when it was wrong. | Lab | Question | |---|---| | 0.1 https://github.com/daneb/ai-system-design-primer/blob/master/labs/0.1-tokens-and-sampling/README.md | What a token is, and what temperature does | | 0.2 https://github.com/daneb/ai-system-design-primer/blob/master/labs/0.2-inside-a-model-call/README.md | Prefill, decode, the KV cache and reasoning tokens | | 0.3 https://github.com/daneb/ai-system-design-primer/blob/master/labs/0.3-tool-calling/README.md | The tool-calling loop, its token bill, and what checks catch | | 2.1 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.1-cost-quality-latency/README.md | When does routing between models pay off? | | 2.2 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.2-determinism-vs-autonomy/README.md | What autonomy costs in reliability and predictability | | 2.3 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.3-context-vs-retrieval/README.md | When is the whole corpus in the prompt cheaper than retrieval? | | 2.4 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.4-capability-vs-security/README.md | Count to three, then size what a detector leaves behind | | 2.5 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.5-single-vs-multi-agent/README.md | One agent or several? | | 2.7 https://github.com/daneb/ai-system-design-primer/blob/master/labs/2.7-throughput-vs-latency/README.md | Batching, per-user speed, and why you can't run at 100% | | 3.2 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.2-input-guardrails/README.md | What a guardrail's error rates mean at volume | | 3.3 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.3-context-engineering/README.md | What each context strategy costs, and what it forgets | | 3.4 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.4-retrieval-pipeline/README.md | Keyword vs dense vs hybrid retrieval | | 3.5 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.5-agentic-retrieval/README.md | Single-shot retrieval, query fan-out, or an agent loop? | | 3.6 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.6-vector-stores/README.md | Approximate search and filtering | | 3.7 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.7-model-layer/README.md | Picking the cheapest model that "meets the bar", with a noisy eval | | 3.8 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.8-serving-internals/README.md | Cache arithmetic and when speculation pays | | 3.9 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.9-workflows-and-agents/README.md | An evaluator loop is only as good as its evaluator | | 3.10 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.10-multi-agent-and-protocols/README.md | What does connecting more tools cost, and does tool search pay? | | 3.11 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.11-memory/README.md | What should an agent remember, and how? | | 3.12 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.12-output-guardrails/README.md | How far does a quote check get you? | | 3.13 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.13-caching/README.md | When a prompt cache pays, and what a semantic cache gets wrong | | 3.14 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.14-asynchronism/README.md | Retries during an overload, and checkpoints for long runs | | 3.15 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.15-evals/README.md | How many eval cases do you need? | | 3.16 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.16-observability/README.md | Which agent traces should you keep? | | 3.17 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.17-security/README.md | Pin your tools, and see what pinning can't see | | 3.18 https://github.com/daneb/ai-system-design-primer/blob/master/labs/3.18-model-adaptation/README.md | What LoRA trains, and when a fine-tune pays for itself | | 4.1 https://github.com/daneb/ai-system-design-primer/blob/master/labs/4.1-inference-api/README.md | How many accelerators does a self-hosted inference API need? | | 4.2 https://github.com/daneb/ai-system-design-primer/blob/master/labs/4.2-enterprise-rag/README.md | Where the money goes in an enterprise RAG assistant | | 4.4 https://github.com/daneb/ai-system-design-primer/blob/master/labs/4.4-document-review/README.md | What 98% recall costs in lawyer time | | 4.5 https://github.com/daneb/ai-system-design-primer/blob/master/labs/4.5-support-agent/README.md | Consistency with policy in the prompt or in a policy engine | | 4.6 https://github.com/daneb/ai-system-design-primer/blob/master/labs/4.6-coding-agent/README.md | Review capacity caps merged work | Learn from the community. Pull requests are welcome to fix errors, improve sections and add new ones. Read STYLE.md https://github.com/daneb/ai-system-design-primer/blob/master/STYLE.md for how sections are researched and written learner-first, several providers per topic, every claim sourced and CONTRIBUTING.md https://github.com/daneb/ai-system-design-primer/blob/master/CONTRIBUTING.md for the mechanics. Decisions about the primer itself are recorded in adr/ https://github.com/daneb/ai-system-design-primer/blob/master/adr . Work still to do is in ROADMAP.md https://github.com/daneb/ai-system-design-primer/blob/master/ROADMAP.md . Summaries of AI system design topics, including pros and cons. Everything is a trade-off. Each section links to its deep dive. Maturity is marked S stable core , S/F stable concept, volatile specifics or F frontier, expect rewrites . Part 0: Orientation - 0 Orientation https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0-orientation.md S - 0.1 Tokens, context and sampling https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0.1-tokens-context-and-sampling.md S - 0.2 Inside a model call: prefill, decode and reasoning https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0.2-inside-a-model-call.md S - 0.3 Tool calling and structured output https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0.3-tool-calling.md S Part 1: Method Part 2: Trade-offs - 2.1 Cost vs quality vs latency https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.1-cost-quality-latency.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.1-cost-quality-latency.md - 2.2 Determinism vs autonomy https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.2-determinism-vs-autonomy.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.2-determinism-vs-autonomy.md - 2.3 Context stuffing vs retrieval https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.3-context-vs-retrieval.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.3-context-vs-retrieval.md - 2.4 Capability vs security https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.4-capability-vs-security.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.4-capability-vs-security.md - 2.5 Single-agent vs multi-agent https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.5-single-vs-multi-agent.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.5-single-vs-multi-agent.md - 2.6 Prompting vs RAG vs fine-tuning https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.6-prompt-rag-finetune.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.6-prompt-rag-finetune.md - 2.7 Throughput vs per-request latency in serving https://github.com/daneb/ai-system-design-primer/blob/master/docs/2-trade-offs/2.7-throughput-vs-latency.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/2.7-throughput-vs-latency.md Part 3: Components, in request order - 3.1 AI gateway and router https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.1-ai-gateway-and-router.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.1-ai-gateway-and-router.md - 3.2 Input guardrails https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.2-input-guardrails.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.2-input-guardrails.md - 3.3 Context engineering https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.3-context-engineering.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.3-context-engineering.md - 3.4 Retrieval pipeline https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.4-retrieval-pipeline.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.4-retrieval-pipeline.md - 3.5 Agentic retrieval and the long-context boundary https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.5-agentic-retrieval.md F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.5-agentic-retrieval.md - 3.6 Vector stores https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.6-vector-stores.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.6-vector-stores.md - 3.7 Model layer: selection, structured outputs, streaming https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.7-model-layer.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.7-model-layer.md - 3.8 Serving internals https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.8-serving-internals.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.8-serving-internals.md - 3.9 Workflows and agents https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.9-workflows-and-agents.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.9-workflows-and-agents.md - 3.10 Multi-agent topologies and protocols MCP, A2A https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.10-multi-agent-and-protocols.md F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.10-multi-agent-and-protocols.md - 3.11 Agent memory https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.11-memory.md F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.11-memory.md - 3.12 Output guardrails and validation https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.12-output-guardrails.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.12-output-guardrails.md - 3.13 Caching https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.13-caching.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.13-caching.md - 3.14 Asynchronism: batch, durable execution, back-pressure https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.14-asynchronism.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.14-asynchronism.md - 3.15 Evals https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.15-evals.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.15-evals.md - 3.16 Observability https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.16-observability.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.16-observability.md - 3.17 Security https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.17-security.md S/F · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.17-security.md - 3.18 Model adaptation https://github.com/daneb/ai-system-design-primer/blob/master/docs/3-components/3.18-model-adaptation.md S · deep dive https://github.com/daneb/ai-system-design-primer/blob/master/docs/deep-dives/3.18-model-adaptation.md Part 4: Solved designs Part 5: Appendix Suggested topics to review based on your timeline. | | Short days | Medium weeks | Long months | |---|---|---|---| | Read Orientation https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0-orientation.md , the foundations 0.1 https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0.1-tokens-context-and-sampling.md – 0.3 https://github.com/daneb/ai-system-design-primer/blob/master/docs/0-orientation/0.3-tool-calling.md and the method https://github.com/daneb/ai-system-design-primer/blob/master/docs/1-method/1-method.md | ✓ | ✓ | ✓ | | Read the trade-offs Part 2 | Some | All | All | | Read the components Part 3 | 3.1, 3.4, 3.9, 3.15, 3.17 | Most | All, plus deep dives | | Read Numbers https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.1-numbers.md and the Glossary https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.6-glossary.md | ✓ | ✓ | ✓ | | Work the solved designs Part 4 | Some | Many | All | | Run the labs and their exercises | Some | Many | All | | Practise the additional questions https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.8-unsolved-questions.md | Some | Many | Most | | Read real-world architectures https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.5-real-world-architectures.md | Some | Many | All | | Map your platform with cloud https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.2-cloud-mapping.md and governance https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.4-governance-crosswalk.md crosswalks | — | Some | All | How to tackle an AI system design question, in an interview or a design review. The classic four steps, with two changes: the quality bar comes first , because a model-backed feature is right only some of the time; and hardening is its own step , because prompt injection, data leakage and runaway agents are design constraints, not add-ons. 1. Use cases, constraints and the quality bar. Who, what, how many, how fast, what data, and how you'll know it works. 2. Back-of-the-envelope. Tokens per request, requests per second, cost per day, latency. 3. High-level design on the complexity ladder. Single call → workflow → single agent → multi-agent. Start at the lowest rung that meets the bar. 4. Core components. Retrieval, context, tools, model choice, output checks. 5. Harden. Which legs of the lethal trifecta are present, and which does the design break? 6. Scale and optimise. Bottlenecks in order; caching, routing, batching. Full walkthrough with a worked example: How to approach an AI system design question https://github.com/daneb/ai-system-design-primer/blob/master/docs/1-method/1-method.md . Common AI system design questions with sample discussions, numbers and diagrams. | Question | | |---|---| | Design an LLM inference API | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.1-llm-inference-api.md | | Design an enterprise RAG assistant | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.2-enterprise-rag-assistant.md | | Design an enterprise AI gateway | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.3-enterprise-ai-gateway.md | | Design a regulated document-review assistant | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.4-regulated-document-review.md | | Design a customer-support agent | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.5-customer-support-agent.md | | Design a coding agent platform | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.6-coding-agent.md | | Design an eval and release pipeline | Solution https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.7-eval-and-release-pipeline.md | | Add an AI system design question | Contribute https://github.com/daneb/ai-system-design-primer/blob/master/CONTRIBUTING.md | More practice: additional design questions https://github.com/daneb/ai-system-design-primer/blob/master/docs/4-solved-designs/4.8-unsolved-questions.md . - Numbers every AI engineer should know https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.1-numbers.md : tokens, speed, cost ratios, memory, retrieval and eval figures, each dated and sourced. - Cloud mapping https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.2-cloud-mapping.md : each capability on Azure, AWS, Google Cloud, OpenAI, Anthropic and open source. - Pattern crosswalk https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.3-pattern-crosswalk.md : the same patterns under each provider's and author's names. - Governance crosswalk https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.4-governance-crosswalk.md : OWASP LLM and Agentic Top 10, FINOS, EU AI Act, mapped to primer sections. - Real-world architectures https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.5-real-world-architectures.md : engineering write-ups and independent studies. - Glossary https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.6-glossary.md and changelog. - Trade-offs first. Every pattern says which trade-off it moves and in which direction. - Quality bar first. Evals are part of the method, not an afterthought. - Maturity is labelled and dated. Each page carries last verified and review by ; the checker fails when a review is overdue. Frontier pages are reviewed every three months. - Several providers on every topic. Anthropic, OpenAI, Google, AWS and Microsoft, with the names each uses. - Vendor-neutral prose. Product names live in the deep dives and the cloud mapping. - Risk and controls on every topic, with the evidence an auditor would ask for. - Runnable labs with predictions recorded before the run. | Phase | Sections | Planned | Draft | Review | Published | |---|---|---|---|---|---| | 0 | 6 | 0 | 0 | 0 | 6 | | 1 | 12 | 0 | 0 | 2 | 10 | | 2 | 10 | 0 | 0 | 2 | 8 | | 3 | 11 | 0 | 0 | 1 | 10 | | 4 | 5 | 0 | 0 | 2 | 3 | See ROADMAP.md https://github.com/daneb/ai-system-design-primer/blob/master/ROADMAP.md for what's next and PLAN.md https://github.com/daneb/ai-system-design-primer/blob/master/PLAN.md for the build plan and the verification debt claims still waiting for a first-party source . - Donne Martin's System Design Primer https://github.com/donnemartin/system-design-primer , whose structure this follows. - The provider guides, papers and engineering write-ups cited in each section; see real-world architectures https://github.com/daneb/ai-system-design-primer/blob/master/docs/5-appendix/5.5-real-world-architectures.md . Open an issue to discuss errors, missing topics or new solved designs. Report security problems, or guidance that would make a system less safe, privately as described in SECURITY.md https://github.com/daneb/ai-system-design-primer/blob/master/SECURITY.md . Participation is covered by the Code of Conduct https://github.com/daneb/ai-system-design-primer/blob/master/CODE OF CONDUCT.md . - Content pages, diagrams, flashcards, lab READMEs is licensed under Creative Commons Attribution 4.0 International https://github.com/daneb/ai-system-design-primer/blob/master/LICENSE CC BY 4.0 , the same licence as the original System Design Primer. - All diagrams and charts are original to this repository and covered by the content licence. - Code scripts/ , the lab scripts and the CI workflow is licensed under the MIT License https://github.com/daneb/ai-system-design-primer/blob/master/LICENSE-CODE . - Contributions are accepted under the same terms. The reasoning is in ADR 0008 https://github.com/daneb/ai-system-design-primer/blob/master/adr/0008-licences.md . The author researched and wrote this primer with an AI assistant, and reviewed every section. The method is in STYLE.md https://github.com/daneb/ai-system-design-primer/blob/master/STYLE.md : primary sources from several providers for every claim, a dated last verified on every page, runnable labs instead of asserted numbers, and a public list of what isn't yet verified. Mistakes are still possible; please report them with a source.