{"slug": "ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab", "title": "AI Engineer Notebooks – free, framework-free RAG/agents/evals on Colab", "summary": "Calm Rocks released a free, framework-free set of Colab notebooks teaching the applied-LLM stack for AI Engineer and Forward Deployed Engineer roles, covering model APIs, RAG, evals, agents, fine-tuning, and serving on Groq's free API. The notebooks emphasize evals as the core practice and include three end-to-end case studies, with all patterns being OpenAI-compatible and transferable to Anthropic.", "body_md": "**Learn the applied-LLM stack the way you'll actually be interviewed on it —\nframework-free, on a free API, from prompting all the way to serving, fine-tuning,\nand a red-team benchmark.**\n\nRunnable Colab notebooks for the **AI Engineer / Forward Deployed Engineer (FDE)**\nskill set: building working systems on top of foundation models — model APIs, RAG,\nevals, agents, adaptation, serving — using raw APIs, not frameworks.\n\n**Framework-free, on purpose.** You write the agent loop, RAG, and evals from raw API calls first — so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when*not*to). Patterns are durable; wrappers churn.**Evals are the spine.**\"Measure before you tune\" is installed early and returns in every section — the habit that separates an engineer who shipped a system from one who built a demo.**Free to run, end to end.** Everything runs on the**free**(no credit card). The two topics Groq can't host — LoRA fine-tuning (06) and self-hosted serving (09) — are concept-first with optional, fenced Colab-GPU appendices,[Groq](https://console.groq.com/)API**verified on a real Colab T4**.** Real case studies, not toy demos.**Three end-to-end[case studies](/calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone)show the skills combined under real constraints — a support assistant*debugged in production*, a pipeline-vs-agent cost showdown, and a red-team robustness benchmark.**OpenAI-compatible throughout**, so every pattern transfers directly to OpenAI and (with small changes) Anthropic — the seam is swappable, the skills aren't.\n\nBuilt as the hands-on companion to\n[Plan: Transitioning to Forward Deployed Engineer / AI Engineer](https://www.calm.rocks/resources/career-development/transition-fde-ai-engineer/).\nThe plan explains what to learn and why; these notebooks are where you run it.\n\nBackend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles — different titles, largely the same job. You can ship production code; you want the applied-model layer on top.\n\nWork top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises.\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n[Structured output](/calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/01-structured-output.ipynb)[Tool calling](/calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/02-tool-calling.ipynb)[Streaming](/calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/03-streaming.ipynb)[Context & caching](/calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/04-context-and-caching.ipynb)| Notebook | What you'll learn |\n|---|---|\n|\n\n*before*building anything you'd need to tune. Evals is the spine; it returns in every section after this| Notebook | What you'll learn |\n|---|---|\n|\n\n[Embeddings & retrieval](/calmrocks/ai-engineer-notebooks/blob/main/03-rag/01-embeddings-retrieval.ipynb)[Hybrid & reranking](/calmrocks/ai-engineer-notebooks/blob/main/03-rag/02-hybrid-and-reranking.ipynb)[Chunking](/calmrocks/ai-engineer-notebooks/blob/main/03-rag/03-chunking.ipynb)[Why RAG fails](/calmrocks/ai-engineer-notebooks/blob/main/03-rag/04-why-rag-fails.ipynb)| Notebook | What you'll learn |\n|---|---|\n|\n\n[LLM as judge](/calmrocks/ai-engineer-notebooks/blob/main/04-evals/02-llm-as-judge.ipynb)[Regression evals](/calmrocks/ai-engineer-notebooks/blob/main/04-evals/03-regression-evals.ipynb)| Notebook | What you'll learn |\n|---|---|\n|\n\n[Tool design](/calmrocks/ai-engineer-notebooks/blob/main/05-agents/02-tool-design.ipynb)[Guardrails & budgets](/calmrocks/ai-engineer-notebooks/blob/main/05-agents/03-guardrails-and-budgets.ipynb)[MCP & the tool ecosystem](/calmrocks/ai-engineer-notebooks/blob/main/05-agents/04-mcp-and-the-tool-ecosystem.ipynb)[Skills & progressive disclosure](/calmrocks/ai-engineer-notebooks/blob/main/05-agents/05-skills-and-progressive-disclosure.ipynb)`SKILL.md`\n\npattern, the context-budget payoff, and Tools/MCP/Skills as one story[Harness engineering](/calmrocks/ai-engineer-notebooks/blob/main/05-agents/06-harness-engineering.ipynb)*around*the call — context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece| Notebook | What you'll learn |\n|---|---|\n|\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n[Reliability & fallbacks](/calmrocks/ai-engineer-notebooks/blob/main/08-operations/02-reliability-and-fallbacks.ipynb)[Experiment tracking & registry](/calmrocks/ai-engineer-notebooks/blob/main/08-operations/03-experiment-tracking-and-registry.ipynb)Where the free Groq API can't run the topic (these frameworks need a GPU),\nthe notebook teaches it **concept-first** and fences an optional Colab-GPU\nappendix — the same pattern as the section-06 LoRA appendix.\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n[Inference performance](/calmrocks/ai-engineer-notebooks/blob/main/09-serving-inference/02-inference-performance.ipynb)| Notebook | What you'll learn |\n|---|---|\n|\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\nWhere the skills come together into projects. First a **case study** — one\nrealistic scenario worked end to end, runnable — then the **capstone**, the\ndeployed repo you build yourself. ([Section overview](/calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/README.md).)\n\n| Notebook | What you'll learn |\n|---|---|\n|\n\n**debugged in production**: a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11[Case study B — Contract extraction: pipeline vs agent](/calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/02-contract-extraction-pipeline-vs-agent.ipynb)*same*extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known[Case study C — Red-team robustness benchmark](/calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/03-red-team-robustness-benchmark.ipynb)*kind*of system — a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals**Capstone:** [the brief](/calmrocks/ai-engineer-notebooks/blob/main/12-case-studies-and-capstone/CAPSTONE.md) for the deployed\nproject that goes on your resume — a real repo with a serving component and an\neval report. Case studies are for learning; the capstone is for hiring.\n\n**Raw model APIs, no frameworks.** Patterns are durable; wrappers churn.**One shared corpus**(`data/`\n\n) across RAG and eval sections, so evals measure the retrieval you actually built.**Self-contained notebooks.** First cell installs, second cell calls`from aien import setup; client, MODEL = setup()`\n\nto load your key from Colab secrets (or a local env var). No hidden state between notebooks.`aien`\n\nis the tiny shared-setup package in this repo — one place to change credential loading — installed automatically by the first cell.**Every notebook ends with exercises**— do them before moving on.\n\n- Get a free API key at\n[console.groq.com](https://console.groq.com/)— no credit card required. - In Colab: the key icon in the left sidebar → add\n`GROQ_API_KEY`\n\nas a secret, and toggle notebook access on. - Open any notebook via its badge and run top to bottom.\n\nRunning locally instead: `pip install -r requirements.txt && pip install -e .`\n\n(the second installs the `aien`\n\nsetup helper), `export GROQ_API_KEY=...`\n\n,\nopen with Jupyter.\n\n[Plan: Transitioning to FDE / AI Engineer](https://www.calm.rocks/resources/career-development/transition-fde-ai-engineer/)— the roadmap these notebooks implement[Guide: Building a Real LLM Project for Your Resume](https://www.calm.rocks/resources/career-development/real-llm-project/)— the capstone's requirements bar[Walkthrough: Designing a RAG System](https://www.calm.rocks/resources/prepare-interview/system-design/rag-system-walkthrough/)— the systems view of section 03[Walkthrough: Designing an AI Agent Orchestration System](https://www.calm.rocks/resources/prepare-interview/system-design/agent-orchestration-walkthrough/)— the systems view of section 05", "url": "https://wpnews.pro/news/ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab", "canonical_source": "https://github.com/calmrocks/ai-engineer-notebooks", "published_at": "2026-08-27 21:46:39+00:00", "updated_at": "2026-08-27 22:18:33.002381+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-research"], "entities": ["Calm Rocks", "Groq", "OpenAI", "Anthropic", "LangChain", "LlamaIndex", "Colab"], "alternates": {"html": "https://wpnews.pro/news/ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab", "markdown": "https://wpnews.pro/news/ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab.md", "text": "https://wpnews.pro/news/ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab.txt", "jsonld": "https://wpnews.pro/news/ai-engineer-notebooks-free-framework-free-rag-agents-evals-on-colab.jsonld"}}