AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab

Calm Rocks released a free, framework-free set of Colab notebooks teaching the applied-LLM stack for AI Engineer and Forward Deployed Engineer roles, covering model APIs, RAG, evals, agents, fine-tuning, and serving on Groq's free API. The notebooks emphasize evals as the core practice and include three end-to-end case studies, with all patterns being OpenAI-compatible and transferable to Anthropic.

Learn the applied-LLM stack the way you'll actually be interviewed on it — framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark. Runnable Colab notebooks for the AI Engineer / Forward Deployed Engineer FDE skill set: building working systems on top of foundation models — model APIs, RAG, evals, agents, adaptation, serving — using raw APIs, not frameworks. Framework-free, on purpose. You write the agent loop, RAG, and evals from raw API calls first — so you understand what LangChain/LlamaIndex actually do before you reach for them and can judge when not to . Patterns are durable; wrappers churn. Evals are the spine. "Measure before you tune" is installed early and returns in every section — the habit that separates an engineer who shipped a system from one who built a demo. Free to run, end to end. Everything runs on the free no credit card . The two topics Groq can't host — LoRA fine-tuning 06 and self-hosted serving 09 — are concept-first with optional, fenced Colab-GPU appendices, Groq https://console.groq.com/ API verified on a real Colab T4 . Real case studies, not toy demos. Three end-to-end case studies /calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone show the skills combined under real constraints — a support assistant debugged in production , a pipeline-vs-agent cost showdown, and a red-team robustness benchmark. OpenAI-compatible throughout , so every pattern transfers directly to OpenAI and with small changes Anthropic — the seam is swappable, the skills aren't. Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer https://www.calm.rocks/resources/career-development/transition-fde-ai-engineer/ . The plan explains what to learn and why; these notebooks are where you run it. Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer AI roles — different titles, largely the same job. You can ship production code; you want the applied-model layer on top. Work top to bottom. Each notebook is self-contained installs its own dependencies, reads API keys from Colab secrets and ends with exercises. | Notebook | What you'll learn | |---|---| | | Notebook | What you'll learn | |---|---| | Structured output /calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/01-structured-output.ipynb Tool calling /calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/02-tool-calling.ipynb Streaming /calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/03-streaming.ipynb Context & caching /calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/04-context-and-caching.ipynb | Notebook | What you'll learn | |---|---| | before building anything you'd need to tune. Evals is the spine; it returns in every section after this| Notebook | What you'll learn | |---|---| | Embeddings & retrieval /calmrocks/ai-engineer-notebooks/blob/main/03-rag/01-embeddings-retrieval.ipynb Hybrid & reranking /calmrocks/ai-engineer-notebooks/blob/main/03-rag/02-hybrid-and-reranking.ipynb Chunking /calmrocks/ai-engineer-notebooks/blob/main/03-rag/03-chunking.ipynb Why RAG fails /calmrocks/ai-engineer-notebooks/blob/main/03-rag/04-why-rag-fails.ipynb | Notebook | What you'll learn | |---|---| | LLM as judge /calmrocks/ai-engineer-notebooks/blob/main/04-evals/02-llm-as-judge.ipynb Regression evals /calmrocks/ai-engineer-notebooks/blob/main/04-evals/03-regression-evals.ipynb | Notebook | What you'll learn | |---|---| | Tool design /calmrocks/ai-engineer-notebooks/blob/main/05-agents/02-tool-design.ipynb Guardrails & budgets /calmrocks/ai-engineer-notebooks/blob/main/05-agents/03-guardrails-and-budgets.ipynb MCP & the tool ecosystem /calmrocks/ai-engineer-notebooks/blob/main/05-agents/04-mcp-and-the-tool-ecosystem.ipynb Skills & progressive disclosure /calmrocks/ai-engineer-notebooks/blob/main/05-agents/05-skills-and-progressive-disclosure.ipynb SKILL.md pattern, the context-budget payoff, and Tools/MCP/Skills as one story Harness engineering /calmrocks/ai-engineer-notebooks/blob/main/05-agents/06-harness-engineering.ipynb around the call — context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece| Notebook | What you'll learn | |---|---| | | Notebook | What you'll learn | |---|---| | | Notebook | What you'll learn | |---|---| | Reliability & fallbacks /calmrocks/ai-engineer-notebooks/blob/main/08-operations/02-reliability-and-fallbacks.ipynb Experiment tracking & registry /calmrocks/ai-engineer-notebooks/blob/main/08-operations/03-experiment-tracking-and-registry.ipynb Where the free Groq API can't run the topic these frameworks need a GPU , the notebook teaches it concept-first and fences an optional Colab-GPU appendix — the same pattern as the section-06 LoRA appendix. | Notebook | What you'll learn | |---|---| | Inference performance /calmrocks/ai-engineer-notebooks/blob/main/09-serving-inference/02-inference-performance.ipynb | Notebook | What you'll learn | |---|---| | | Notebook | What you'll learn | |---|---| | Where the skills come together into projects. First a case study — one realistic scenario worked end to end, runnable — then the capstone , the deployed repo you build yourself. Section overview /calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/README.md . | Notebook | What you'll learn | |---|---| | debugged in production : a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression a stale index after a corpus migration that you diagnose and fix. A build-to-debug arc threading sections 02–11 Case study B — Contract extraction: pipeline vs agent /calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/02-contract-extraction-pipeline-vs-agent.ipynb same extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known Case study C — Red-team robustness benchmark /calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/03-red-team-robustness-benchmark.ipynb kind of system — a harness that evaluates a model instead of serving one: an attacker→target→judge PAIR loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals Capstone: the brief /calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/CAPSTONE.md for the deployed project that goes on your resume — a real repo with a serving component and an eval report. Case studies are for learning; the capstone is for hiring. Raw model APIs, no frameworks. Patterns are durable; wrappers churn. One shared corpus data/ across RAG and eval sections, so evals measure the retrieval you actually built. Self-contained notebooks. First cell installs, second cell calls from aien import setup; client, MODEL = setup to load your key from Colab secrets or a local env var . No hidden state between notebooks. aien is the tiny shared-setup package in this repo — one place to change credential loading — installed automatically by the first cell. Every notebook ends with exercises — do them before moving on. - Get a free API key at console.groq.com https://console.groq.com/ — no credit card required. - In Colab: the key icon in the left sidebar → add GROQ API KEY as a secret, and toggle notebook access on. - Open any notebook via its badge and run top to bottom. Running locally instead: pip install -r requirements.txt && pip install -e . the second installs the aien setup helper , export GROQ API KEY=... , open with Jupyter. Plan: Transitioning to FDE / AI Engineer https://www.calm.rocks/resources/career-development/transition-fde-ai-engineer/ — the roadmap these notebooks implement Guide: Building a Real LLM Project for Your Resume https://www.calm.rocks/resources/career-development/real-llm-project/ — the capstone's requirements bar Walkthrough: Designing a RAG System https://www.calm.rocks/resources/prepare-interview/system-design/rag-system-walkthrough/ — the systems view of section 03 Walkthrough: Designing an AI Agent Orchestration System https://www.calm.rocks/resources/prepare-interview/system-design/agent-orchestration-walkthrough/ — the systems view of section 05