{"slug": "show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode", "title": "Show HN: A free course from engineer to AI engineer, one Git tag per episode", "summary": "Engineer Joel Stevick released a free eight-episode YouTube course, \"From engineer to AI engineer,\" whose companion GitHub repository (AiFirstDevelopment/engineer-to-ai-engineer) tags each episode as a single commit, s1e1 through s1e8, taking an experienced software engineer with no generative-AI background to a first retrieval-augmented generation pipeline. A second seven-episode series, \"Prove it works,\" tagged s2e1 through s2e7 on the same repository, builds an evaluation harness that caps spending at $1 per run and $5 across all runs in its ledger (evals/ledger.ts). The course uses TypeScript, SQLite, curl and an HTTP server with Anthropic's claude-sonnet-5-5 model at $2 per million input tokens and $10 per million output tokens, putting a typical question through the finished pipeline at about a cent.", "body_md": "The code for the YouTube series *From engineer to AI engineer*: eight episodes that take an\nexperienced software engineer with no generative-AI background from conventional software to a\nfirst retrieval pipeline (RAG), built across the series one piece per episode. Its second series,\n*Prove it works*, builds the evaluation harness that tells you whether that pipeline answers\nright, on the same repository.\n\nEvery episode is one commit and one tag: `s1e1` to `s1e8` for the first series, `s2e1` to `s2e7` for the\nsecond. Check out a tag and you have the repository exactly as that episode leaves it, with a README that\ndescribes only what exists so far. This README, on `main`, describes all of it.\n\nNew to AI engineering? This is free, and it starts from zero: the [YouTube channel](https://www.youtube.com/@JoelStevick)\nhas every episode, [joelstevick.com](https://joelstevick.com) has each episode's page, a free demo and the book, and the\nweekly newsletter [AI That Won't Confidently Lie](https://www.linkedin.com/newsletters/ai-that-won-t-confidently-lie-7526584806781087745/)\nteaches one piece at a time.\n\nThe videos are two YouTube playlists: [From engineer to AI\nengineer](https://www.youtube.com/playlist?list=PLfmRFV611TOA), eight episodes, and [Prove it\nworks](https://www.youtube.com/playlist?list=PLPUiBor6kL3E), seven. Every episode's description links its\ntag here, and the table under \"Tags\" at the end links every tag to its episode.\n\nTo follow along at the keyboard:\n\n```\ngit clone https://github.com/AiFirstDevelopment/engineer-to-ai-engineer.git\ncd engineer-to-ai-engineer\ngit checkout s1e1 # the repository exactly as episode 1 leaves it\ncp .env.example .env # then put your Anthropic API key in .env\nnpm install\n./ask.sh \"In two sentences: what is a race condition?\"\n```\n\nThen `git checkout s1e2` and that episode's commands, and so on to `s2e7`. Checking out a tag puts git on a\ndetached HEAD, which is fine for reading; `git checkout main` comes back. Run `npm install` again after\nchecking out `s1e5`, `s1e6` or `s2e2`, the tags whose `package.json` adds dependencies, or whenever a command\nfails on a missing package. `main` is the finished repository: every file this README names is there.\n\nWhat a command costs is in its episode's section: cents, never dollars. The harness caps its own spending\nat $1 a run and $5 across every run in its ledger (`evals/ledger.ts`), and nothing in the test suite calls\na model.\n\nA model is a function. Text goes in, text comes out. It keeps no state between calls, so every request carries the whole conversation. It is nondeterministic: the same input can give a different output. And it is priced by the token, in and out.\n\nThat is what changed. Everything around it did not. You reach it over HTTP. Your data still\nlives in a database. Your tests still have to prove the thing works. Deployment is what it was\nlast year. This series builds the one new piece, a retrieval pipeline, with tools you already\nhave: TypeScript, SQLite, `curl` and an HTTP server.\n\n- Node 20 or later (`node --version` ) and`jq` (`brew install jq` ).\n- An Anthropic API key from [https://console.anthropic.com](https://console.anthropic.com) . The runs in this series cost\ncents; each episode's section says what its commands cost.\n- `cp .env.example .env` and put your key in`.env` . The file is gitignored.`ask.sh` reads it,\nand every`npx tsx` command below loads it with Node's own`--env-file=.env` , so there is no\ndotenv package.\n- `npm install` . The first search (episode 5 on) downloads the embedding model, 97 MB, once.\n\nThe series answers with one chat model, `claude-sonnet-5-5`, set in one place (the request body\nin `ask.sh`, and the constant `MODEL` in `src/claude.ts`). It costs $2 per million input tokens\nand $10 per million output tokens. A question through the finished pipeline is a few thousand\ntokens in and a few hundred out, about a cent. Claude Opus 5.5 is the more capable model at\ntwice the price per token; Claude Haiku 5.5 is twenty times cheaper. The answers stay on\nSonnet: an answer is what the customer reads, and we want answers we can trust while we grade\nthem. You pay for your own runs, so the cheaper of the two models we trust is the default, and\nswapping the constant is the whole migration. One step uses Haiku: from episode 8 the reranker\n(`RERANK_MODEL` in `src/rerank.ts`) scores twenty passages for each question, a narrower job\nthan writing a cited answer and four times the text, and Haiku does it well enough at a\ntwentieth of the price per token (episode 8 measures it).\n\n`docs/` is the internal documentation of a fictional company, Lantern Supply Co., a Maryland\nretailer of camping lanterns and lamps: twelve Markdown files and one PDF, about 26,000 words,\nfull of specific facts (prices, model numbers, dates, names, policies) that no model has seen.\nThat is the point: in the episodes to come the model has to answer from these files or admit it\ncannot. The facts contain traps on purpose: two products with near-identical names (the Firefly\n220 and the Firefly 220R), a policy that a later document changed (the free-shipping threshold\nis $75 in `shipping.md` and `faq.md`, and `release-notes.md` lowers it to $60 in version 4.3),\nand facts that exist only in the PDF (`repair-manual.pdf`, the Repair Center's bench manual).\n\n| File | What it is | \n|---|---|\n| `about.md` | The company, its history, locations and leadership | \n| `catalog.md` | Every product with model number, price and specifications | \n| `returns-policy.md` | The returns policy (40 days from delivery, and the rest) | \n| `shipping.md` | Rates, carriers, fuel shipping limits | \n| `warranty.md` | Warranty terms by product line and the claim process | \n| `faq.md` | Customer-facing questions and answers | \n| `employee-handbook.md` | An excerpt: pay, time off, discount, safety | \n| `customer-service-playbook.md` | The support team's procedures and reply templates | \n| `release-notes.md` | Release notes of Hearth, the online store | \n| `fuel-safety.md` | Approved fuels, storage, disposal | \n| `lumen-rewards.md` | The loyalty program | \n| `showrooms.md` | The Frederick and Annapolis showrooms | \n| `repair-manual.pdf` | The Repair Center bench manual (PDF only, nine pages) | \n\nThe PDF was rendered once from a Markdown draft with headless Chrome; the draft is not in the repository, on purpose, so its facts exist only in the PDF.\n\nThree jobs, in three groups of files. The first two are series 1 and the third is series 2. Every box is a file in this repository; the episode sections below say what each does and what it printed when it ran.\n\n**Indexing**, `npx tsx src/index.ts`: the documents become searchable chunks in a SQLite file. It is an ETL\njob, and it is safe to run again: a hash of each file says which ones changed.\n\n``` php\nflowchart LR\n docs[\"docs/
12 Markdown files and 1 PDF\"] --> extract[\"src/extract.ts
text, page by page\"]\n extract --> chunk[\"src/chunk.ts
chunks of about 1,200 characters,
each with its source, heading and page\"]\n chunk --> embed[\"src/embed.ts
one 384-number vector per chunk,
all-MiniLM-L6-v2, on your machine\"]\n embed --> db[(\"lantern.db
SQLite: chunks, files\")]\n job[\"src/index.ts
hashes each file: skips the unchanged,
replaces the changed, removes the deleted\"] -.-> extract\n```\n\n**Answering**, `npx tsx --env-file=.env src/server.ts`: a question comes in over HTTP and an answer with\ncitations goes out, or \"I don't know.\" The stages marked *option* are switched on by `Options` in\n`src/pipeline.ts`, one flag each, so the evals can measure each on its own; what the server runs with is\n`SERVED` in `src/config.ts`, the rewrite and the cite check.\n\n``` php\nflowchart TD\n q[\"POST /ask with a question
src/server.ts\"] --> rewrite[\"src/rewrite.ts, option, served
the question rewritten in the documents' words\"]\n rewrite --> retrieve[\"src/retrieve.ts
the 20 nearest chunks by cosine similarity
src/embed.ts, src/cosine.ts, lantern.db\"]\n retrieve --> low{\"best score
under 0.25?\"}\n low -- yes --> idk[\"I don't know.
no model is called\"]\n low -- no --> hybrid[\"src/keyword.ts and src/hybrid.ts, option
BM25 keyword search, the two lists fused by rank\"]\n hybrid --> rerank[\"src/rerank.ts, option
Claude Haiku 5.5 scores the 20 from 0 to 10\"]\n rerank --> five[\"the top 5\"]\n five --> widen[\"src/widen.ts, option
each passage with its neighbours\"]\n widen --> prompt[\"src/prompt.ts
the rules, and the passages numbered [1] to [5]\"]\n prompt --> claude[\"src/claude.ts
Claude Sonnet 5.5, one HTTP call\"]\n claude --> said{\"I don't know?\"}\n said -- yes --> idk\n said -- no --> cite[\"src/cite-check.ts, option, served
a [n] the model was not shown is removed, the answer flagged\"]\n cite --> out[\"the answer, each citation resolved to its file and heading, the cost\"]\n out -.-> log[\"src/log.ts
logs/questions.jsonl\"]\n```\n\n**Proving it**, `npx tsx --env-file=.env evals/eval.ts `: the golden set goes through the same\npipeline, every step is recorded, every case is scored, and the run is compared with the one before. In CI\nthe same run is a gate.\n\n```\nflowchart LR\n golden[\"evals/golden.jsonl
31 cases: the question, the facts, the forbidden claims, the sources\"] --> check[\"evals/check-golden.ts
every quote is in the corpus\"]\n golden --> run[\"evals/eval.ts and evals/gate.ts
evals/run-eval.ts: the pipeline on traced models\"]\n run --> traces[\"evals/runs/NAME/traces/*.json
evals/tracer.ts: every call, every passage, every cent\"]\n traces --> score[\"evals/scoring.ts
the rules first, evals/checks.ts,
then the judge, evals/judge.ts, Claude Haiku 5.5\"]\n score --> results[\"evals/runs/NAME/results.json\"]\n results --> read[\"evals/report.ts, evals/compare.ts,
evals/diagnose.ts, evals/trace.ts\"]\n run --> ledger[\"evals/runs/ledger.json
every run, with its settings and cost
caps: $1 a run, $5 in all\"]\n results --> gate{\"85% passed, no fallback case answered,
the budget not reached?\"}\n gate -- no --> red[\"exit 1: the pull request is red\"]\n gate -- yes --> green[\"exit 0\"]\n cases[\"evals/judge-cases.jsonl
16 answers labelled by hand\"] --> judgeeval[\"evals/judge-eval.ts
the judge's agreement with the labels\"]\n log[\"logs/questions.jsonl
what people asked the server\"] --> sample[\"evals/sample.ts
drafts of new cases, for a person to finish\"] --> golden\n```\n\nThe skeleton and one HTTP call.\n\nFiles: `package.json`, `tsconfig.json`, `.gitignore`, `.env.example`, the corpus in `docs/`,\nand `ask.sh`: one `curl` to the Messages API, the request body written out in the script, the\nanswer's text and the token counts printed with `jq`. Nothing reads `docs/` yet.\n\n```\n./ask.sh \"In two sentences: what is a race condition?\"\n```\n\nThe transcript, `transcripts/s1e1-what-changed.txt`, shows the call and the answer: a request\nbody with a model name, a limit on the answer's length and a list of messages; a response with\nthe text and the number of tokens in and out. Nothing else is needed. Cost of the run: a\nfraction of a cent.\n\nThe same call from TypeScript, and the only client this repository will ever have.\n\nFiles: `src/claude.ts` is the client: `ask(messages, options)` is one `fetch` to the Messages\nAPI that returns the answer's text, the token counts and the cost in dollars, computed from the\nprice list in one constant, with the model id in another; `countTokens(text)` asks the API's\ncount-tokens endpoint, which is free. `src/ask.ts` is a command line around `ask`, and\n`src/tokens.ts` one around `countTokens`. There is no SDK: it is an HTTP API like any other, and\nthe whole client is shorter than most SDK quick-starts.\n\n```\nnpx tsx --env-file=.env src/ask.ts \"In one sentence: what does HTTP status 304 mean?\"\nnpx tsx --env-file=.env src/ask.ts \\\n \"I bought a lantern from Lantern Supply Co. How many days do I have to return it?\"\nnpx tsx --env-file=.env src/tokens.ts \"The Firefly 220R charges in 3.5 hours over USB-C.\"\n```\n\nThe transcript, `transcripts/s1e2-calling-a-model.txt`, shows three things. A general question\nis answered well, in 79 output tokens, for a twelfth of a cent. The question about Lantern\nSupply Co.'s return window is not answered: the model has never seen `docs/returns-policy.md`\n(40 days from delivery), says it does not know the company, and sends the customer to their\nreceipt. It cannot know. That is the problem the rest of the series solves, and notice that a\nmodel which guessed a number instead would sound exactly as sure. Last, a 49-character sentence\nis 32 tokens: a token is a few characters, a short word is one, and numbers, model names and\npunctuation split into several. Everything you pay for and every limit you will meet is counted\nin tokens. Cost of the three runs together: under half a cent.\n\nTwo ways to make the model's output something a program can use.\n\nFiles: `src/structured.ts` asks for JSON instead of prose. The shape is written twice, once as\na TypeScript type for the code and once as JSON Schema for the API, which holds the model to\nit (`output_config.format`). The code still parses the text and checks the shape with a type\nguard, because the model's output is input. `src/tools.ts` is tool use: one tool,\n`lookup_order(order_id)`, read from `data/orders.json`, and the loop that runs it. The model\nanswers with a `tool_use` block instead of text; we run the function, append the model's whole\nturn and a `tool_result` to the conversation, and call again; it repeats until the model\nanswers in text. The loop is ours, and it is a plain `while`. `src/claude.ts` grows two options,\n`tools` and `format`, and returns the whole content of a turn, not only its text.\n\nThe loop, as the transcript shows it:\n\n```\nsequenceDiagram\n participant T as src/tools.ts\n participant M as Claude\n participant F as lookup_order, from data/orders.json\n T->>M: the question, and the tool's name, description and input schema\n M-->>T: a tool_use block, lookup_order(\"LS-2026-012077\")\n T->>F: run the function\n F-->>T: the order's JSON\n T->>M: the conversation so far, the model's turn, and a tool_result holding that JSON\n M-->>T: text, delivered on October 29 by UPS Ground, with the tracking number\nnpx tsx --env-file=.env src/structured.ts\nnpx tsx --env-file=.env src/tools.ts \"Has order LS-2026-012077 arrived yet?\"\n```\n\nThe transcript, `transcripts/s1e3-prompts-are-code.txt`, shows a customer's email turned into\nan `OrderSummary` (the order id, the customer, two items with quantities, intent `return`) for\na fifth of a cent, and the tool loop: the model asks for `lookup_order(LS-2026-012077)`, gets\nthe order's JSON from the file, and answers that it was delivered on October 29 by UPS Ground,\nwith the tracking number and the email's two Fireflies and Northstar, in two round trips for\nhalf a cent. The prompt, the schema and the tool's description are code: they live in the\nrepository, they are reviewed, and from episode 7 on they are tested.\n\nThe obvious fix, and why it is a dead end.\n\nFiles: `src/ask-with-doc.ts` pastes a document into the prompt and asks the question after it.\nGiven one Markdown file it pastes that file; given the folder it pastes every Markdown file in\nit (the PDF waits for episode 6). Before calling the model it counts the prompt's tokens with\nthe free count-tokens endpoint and prints what that much input costs on every question.\n\n```\nnpx tsx --env-file=.env src/ask-with-doc.ts docs/returns-policy.md \\\n \"In one sentence: how many days do I have to return a lantern?\"\nnpx tsx --env-file=.env src/ask-with-doc.ts docs \"What is the free shipping threshold?\"\n```\n\nThe transcript, `transcripts/s1e4-why-it-cant-read-your-documents.txt`, shows the return-window\nquestion answered right, 40 days from delivery, once `returns-policy.md` is in the prompt:\n3,720 tokens, three quarters of a cent of input per question. Then the whole corpus goes in,\n45,331 tokens, and the model answers the free-shipping question right too, $60, and even\nnotices that the release notes are newer than the policy pages that still say $75. It can do\nthat because it sees everything. The price is nine cents of input per question, before the\nanswer, and it is paid again on every question because the model keeps nothing between calls.\n\nThat is the dead end. A thousand questions a day is $90 a day for 26,000 words of documentation; ten times the documentation is ninety cents a question and a long wait for each answer; thirty times does not fit at all, because this model's context window, the most it can read in one call, is a million tokens. The window is the limit; the price is the wall you hit first. The way out is to send only the few paragraphs that matter to the question, which means finding them first. Finding them is retrieval, and it is the next four episodes.\n\nSearch by meaning, with a table, a blob column and a loop.\n\nFiles: `src/embed.ts` turns a text into a vector, a `Float32Array` of 384 numbers, with a small\nopen embedding model, `all-MiniLM-L6-v2`, running on your machine through\n`@huggingface/transformers`. Texts that mean similar things get vectors that point the same\nway. The model downloads once, 97 MB into `node_modules/@huggingface/transformers/.cache`, and\nnothing is sent anywhere: there is no second vendor and no second key, and the only paid call\nin this repository is still Claude. `src/cosine.ts` is cosine similarity in ten lines: how alike\ntwo vectors point, 1 for the same direction, 0 for unrelated. `src/db.ts` is SQLite through\n`better-sqlite3`, one table, `chunks(id, source, text, embedding)`, where the embedding is the\nvector's bytes in a blob column like any other column. `src/search.ts` is the search: on its\nfirst run it splits every Markdown file in `docs/` into paragraphs (a quick split on blank\nlines, for now), embeds each and inserts it; on every run it embeds the query, scores every row\nwith cosine in TypeScript, a full scan, and prints the top five.\n\n```\nnpx tsx src/search.ts \"what happens if the courier loses my parcel\"\n```\n\nNo `--env-file`: nothing here calls Claude. The transcript, `transcripts/s1e5-embeddings.txt`,\nshows the first run indexing 414 paragraphs in three seconds, then the top five. The best\nmatch, at 0.627, is the playbook's \"A package counts as lost when it has not arrived 10\nbusiness days after the expected delivery date\", and it shares not one word with the query:\ncourier against carrier and package, parcel against package, loses against lost. A keyword\nsearch finds none of that. Meaning beats words, and that is the whole reason for embeddings.\nNotice also that a bare heading line made the top five: paragraphs are a poor unit, which is\nepisode 6.\n\nWhen does the full scan stop being fine? 414 rows of 384 floats is nothing; a hundred thousand\nrows is still a few tens of milliseconds in a loop like this one. When you have millions of\nrows, or many searches a second, you add a vector index (`sqlite-vec`, `pgvector`, or a vector\ndatabase), which is the same decision as adding an index to any other table, made for the same\nreason, and not before.\n\nExtract, transform, load. You have written this job before.\n\nFiles: `src/extract.ts` turns a file into text: a Markdown file as it is, a PDF page by page\nthrough `unpdf`, with the page number kept, because \"page 3 of the repair manual\" is how a\nperson wants it cited. `src/chunk.ts` turns text into chunks, the unit we embed, store, search\nand show the model: it splits at headings first, because a heading is where the author changed\nthe subject, then packs long sections paragraph by paragraph into pieces of about 1,200\ncharacters (300 tokens or so), each starting with the last paragraph of the one before, the\noverlap, so a fact on a boundary is whole in one of them. A PDF comes out as lines, not\nparagraphs, so there a line is the unit, and it has no headings, so its chunks carry a page\ninstead. Every chunk keeps its source, heading and page. `src/db.ts` grows those columns, a\n`position` (the chunk's place in its file, for episode 8), a `files` table with a SHA-256 hash\nof each file, and a schema version: the store is a cache of `docs/`, and when its shape changes\nit is thrown away and rebuilt. `src/index.ts` is the job: for each file in `docs/`, hash it;\nif the hash is the one stored, skip it; otherwise delete its old chunks, extract, chunk, embed,\ninsert, and remember the hash; files that disappeared are removed. `src/search.ts` now only\nsearches, and shows where each hit came from.\n\n```\nnpx tsx src/index.ts\nnpx tsx src/index.ts\nnpx tsx src/search.ts \"what is the bench fee for an out-of-warranty repair\"\n```\n\nThe transcript, `transcripts/s1e6-chunking-and-indexing.txt`, shows the first run indexing\n13 files into 287 chunks in 3.5 seconds (the 414 bare paragraphs of episode 5 are gone;\nheadings travel with their text now, and the FAQ's 54 question-and-answer pairs are the largest\nshare), the second run skipping all 13 in no time, and a search for the bench fee landing on\npage 1 of the repair manual at 0.685: a fact that exists nowhere else in the corpus, found in a\nPDF, with its page number. Nothing here calls Claude.\n\nWhy `unpdf` and not `pdf-parse`, the usual answer: it is plain JavaScript (pdf.js underneath),\nan ES module, with no native build, and it gives the text page by page in one call; `pdf-parse`\nis CommonJS and returns one blob for the whole file unless you hook into its page renderer.\n\nRetrieve, prompt, answer: with a server in front and a test behind.\n\nFiles: `src/retrieve.ts` returns the top five chunks for a question (episode 5's search as a\nfunction). `src/prompt.ts` is the system prompt and the layout: answer from the passages and\nnothing else, cite every fact with the passage's number like [2], prefer the more recent\ndocument when passages disagree, and when they do not answer, reply exactly \"I don't know\";\nthe passages are numbered and labelled with where they came from. `src/pipeline.ts` is\n`answer(question)`: retrieve; if the best score is under 0.25 the model is not even asked (a\nquestion about nothing in this corpus scores 0.17 to 0.24 here; measure yours); otherwise ask,\nand if the model says it does not know, cite nothing; otherwise resolve every [n] in the answer\nto the passage it points at. `src/server.ts` is `POST /ask` on Node's own `http` module, JSON\nin, answer and citations out, thirty lines. `evals/questions.jsonl` is ten questions, each with\nthe strings its answer must contain, phrased the way people ask and aimed at the traps: the\nrechargeable Firefly, a $65 order, the bench fee, the torque, and the CEO's salary, whose only\nright answer is \"I don't know\". `evals/run.ts` runs them through the pipeline and prints pass\nor fail and the total cost.\n\n```\nnpx tsx --env-file=.env src/server.ts &\ncurl -s localhost:3000/ask -d '{\"question\": \"How many days do I have to return a lantern?\"}'\ncurl -s localhost:3000/ask -d '{\"question\": \"How much is the CEO paid?\"}'\nnpx tsx --env-file=.env evals/run.ts\n```\n\nThe transcript, `transcripts/s1e7-your-first-pipeline.txt`, shows the return-window question\nanswered right over HTTP, 40 days from delivery, citing [2] and [4], which resolve to\n`warranty.md > Relationship to the returns policy` and `returns-policy.md > Common situations`,\nfor a third of a cent: twenty-five times cheaper than pasting the corpus in episode 4, and\nfaster. The question about the CEO's pay gets \"I don't know.\" and no citations. The ten\nquestions score 9/10 for less than four cents. The failure is \"Can staff buy a Second Light\nlantern for less?\": the pipeline answered \"I don't know\", because the handbook's employee\ndiscount section was not among the five passages retrieved. The question says staff and Second\nLight, and the nearest passage is the Second Light program in `about.md`; the other four, the Base\nCamp Bundle, the FAQ on refurbished lanterns, Lights for Scouts and the company's beliefs, are\nnearer to the question than the handbook is, which says employee, not staff. It is an honest\nfailure, which is the second rule working, and it is a number. Episode 8 is about that number.\n\nFour changes that usually help, each behind an option, each measured against the ten questions.\n\nFiles: `src/keyword.ts` is BM25, the ranking function behind most search engines, in forty\nlines and no library: it finds exact terms, the model numbers, part codes and names that\nembeddings blur. `src/hybrid.ts` is reciprocal rank fusion: the vector list and the keyword\nlist merged by rank, not by score, since the scores are on different scales. `src/rerank.ts`\nhas a model read the twenty candidates and score each 0 to 10 for how well it answers the\nquestion, in one call for all of them: the question and the instructions are paid for once,\nand the model sees the candidates side by side, which is what ranking needs; a call per\ncandidate would be twenty prompts and twenty round trips for scores that were never compared.\nThe reranker is Claude Haiku 5.5, not the answering model (`RERANK_MODEL`; `src/claude.ts` now\ntakes a `model` option and prices each model it calls, and refuses one it has no price for).\n`src/widen.ts` replaces each passage with itself and its neighbours from the same file, in\norder, because a chunk boundary is arbitrary. `src/cite-check.ts` removes any [n] that points\nat a passage the model was not shown and marks the answer, because a citation nobody can\nfollow looks like evidence. `src/pipeline.ts` switches each on with an option; the server\nkeeps the cite check on. `evals/suite.ts` runs the ten questions with a set of options,\n`evals/run.ts` takes the options as flags, and `evals/ladder.ts` climbs the ladder, one\nchange at a time, and prints the table (it was `evals/compare.ts` at `s1e8`, and is renamed in\nseries 2, episode 5, which needs the name). One expectation was widened while building this: the\nmanual says \"hand-tight plus a quarter turn, never more than 2.5 N·m\", and an answer that\nquotes the first half is right.\n\n```\nnpx tsx --env-file=.env evals/run.ts --hybrid --rerank\nnpx tsx --env-file=.env evals/ladder.ts\n```\n\nThe transcript, `transcripts/s1e8-making-it-good.txt`, is the table, and it is not the table\nyou would draw from the names of the techniques:\n\n| Step | Passed | Cost of the ten | Failed | \n|---|---|---|---|\n| vector only | 9/10 | $0.035 | 8 | \n| + hybrid | 6/10 | $0.037 | 4, 6, 7, 8 | \n| + rerank | 8/10 | $0.045 | 6, 8 | \n| + widen | 9/10 | $0.095 | 8 | \n| + cite check | 9/10 | $0.093 | 8 | \n\nRead it row by row. Hybrid, on its own, made things worse, and the eval is why we know: the keyword list is full of FAQ entries that say \"Firefly\" and \"rechargeable\" often, and after fusion the catalog chunk that holds the 220R's weight fell from fourth to ninth, out of the five the model sees; the chunk with the torque figure went the same way, and the model, shown the wrong half of the page, said it did not know, which is the right failure. Shipped without numbers, that change would have cost two questions. Reranking from the wider pool recovers them, for a fifth more per question than hybrid alone, because the twenty passages are read by Haiku at a twentieth of the answering model's price per token. Question 6, the bench fee, is the noise: the model is nondeterministic, so on ten questions one run is worth plus or minus one. The transcript runs the reranking step twice more, and it passes 9/10 and 8/10; the reranker puts the repair manual's bench-fee page first, and it is the answering model, reading that page, that sometimes says it does not know. Run it three times, or write fifty questions, before you believe a difference of one. Then the transcript switches the reranker to Sonnet for one run: 9/10, the same as Haiku's best, for $0.165 instead of about $0.046, three and a half times the cost of the ten. The model that writes the answer has to be good; the model that sorts passages only has to be good enough, and the eval is how you know it is. Widening costs twice as much for context these answers did not need, which makes it the most expensive change on the ladder, and the cite check found nothing to remove, which is what you want from a guard. Question 8 never passes. The handbook's employee discount section ranks 88th by vector and 38th by keyword for \"Can staff buy a Second Light lantern for less?\", because the question says staff and the handbook says employee; no reranker can promote a passage that never reached it. The fix for that is to rewrite the question before searching, which is a different change, and the ten questions would measure it the same way.\n\nThat is the method: a change is a flag, the flag is measured, the number decides, and the failures that remain tell you what to build next.\n\nSeries 1 ended with a pipeline and ten questions. Ten questions measured the four changes of episode 8, and they also showed their limit: one run of the same settings scored 9 and the next 8, so a difference of one question is noise, and a change that helps one kind of question and hurts another cannot be seen at all. This series builds what a team needs before it trusts a retrieval pipeline with customers: a set of cases with known answers, a pipeline that can be tested without a model, a record of every step of every run, a score that says which stage failed, a budget, a comparison between runs, and a gate in CI.\n\nThirty cases with known answers, each one checked against the corpus.\n\nFiles: `evals/golden.jsonl` is the golden set, one case per line: an `id`; the `question`,\nphrased the way a customer or a new employee asks it; `expect`, `answer` or `fallback`; the\nrequired `facts`, each with the `match` terms that count as stating it (\"40 days\", \"40-day\",\n\"forty days\"); the `forbidden` claims, each with terms that would mean the answer made it (the\nFirefly 220's 310 g given for the 220R); and the `sources`: the file, the page for the PDF, and\na `quote` copied from the extracted text that proves the answer. 22 cases expect an answer and 8\nexpect \"I don't know\": the CEO's salary, last year's unit sales, a Sentinel 2000 that does not\nexist, the dental plan's deductible (the plan is named, its deductible is not), the UPS account\nnumber (the shipping policy says it exists and does not give it), the showroom Wi-Fi, the\nFirefly's price in euros, the name of the Baltimore brass spinner. A fallback case may carry the\nnearest source, to show what the corpus does say. The answer cases cover eleven of the thirteen\nfiles and every trap in the corpus: the 220 against the 220R, the $60 threshold in the release\nnotes against the $75 in two older pages, four facts found only in the PDF, and series 1's\nunsolved staff discount. `evals/golden.ts` loads the set and holds the two matching helpers\nevery later script uses: `flat` (whitespace collapsed) and `mentions` (any of the terms,\nignoring case).\n\n`evals/check-golden.ts` checks the set before anything is scored against it: ids unique, every\nanswer case with facts and sources, every fact and claim with terms, and every quote found in\nthe text that `src/extract.ts` produces for its file and page, the same text the index is built\nfrom. The comparison is character for character once runs of whitespace are collapsed, because\nthe Markdown is wrapped at 100 columns and the PDF's text comes out a line at a time: the line\nbreaks are layout, the characters are the content. A quote typed from memory is the commonest\nerror in a golden set, and a case whose quote is not in the corpus can be scored as a retrieval\nmiss forever while the pipeline is right.\n\n```\nnpx tsx evals/check-golden.ts\n```\n\nNo key, no model. The transcript, `transcripts/s2e1-the-golden-set.txt`, shows the thirty cases\npassing (22 answer, 8 fallback, 26 quotes in 11 files), then the bench-fee quote changed from\nthe manual's \"$25.00\" to \"$25\", the way a person remembers it, and the check failing on it with\nexit code 1, then the quote restored and the check passing. Why thirty: enough that one case is\nthree points and not ten, few enough that a person can read every answer of a run, and each one\nwritten from the document, not from memory. A golden set grows: episode 7 adds a way to turn\nreal questions into new cases.\n\nDependency injection, so the pipeline can be tested without a model.\n\nUntil now the pipeline imported its models: `pipeline.ts` called `ask` from `claude.ts`,\n`retrieve.ts` called `embed`, and the database opened itself when any of them was imported. A\ntest of the pipeline's rules (cite the passage, say \"I don't know\" when nothing answers, do not\neven ask when nothing retrieved is about the question) would have needed a key, a network, a 97\nMB embedding model and money, and it would have answered differently each time. So the models\nbecome a parameter, the way a database connection is one.\n\nFiles: `src/models.ts` is the `Models` interface, two functions, `chat(messages, options)` (the\nMessages API) and `embed(text)`, with `realModels`, the real pair, and `Deps`, everything the\npipeline needs from outside: its models and the chunks it searches. `answer(question, options, deps)` in `src/pipeline.ts` imports no model any more; `retrieve`, `rerank` and `widen` take\nwhat they use as arguments; `src/server.ts` and `evals/suite.ts` pass `realModels` and the\nstored chunks. `src/db.ts` opens `lantern.db` on first use instead of on import.\n`test/fake-models.ts` is the fake pair: an embedder that hashes each word into one of the 384\ndimensions, so texts that share words point the same way, and a chat model that answers from\nmarkers in the prompt: a passage containing `<>` is the answer, cited with that\npassage's number, no marker means \"I don't know\", and asked to score passages it gives 10 to the\none with a marker. Every behaviour a test needs is written into the test's own corpus, so a test\nreads as its own explanation. `test/pipeline.test.ts` has five tests on Node's built-in runner,\n`node:test` and `node:assert`, run through `tsx`: the answer cites its passage and the citation\nresolves to where it came from; no marker gives \"I don't know\" and no citations; a question\nabout nothing in the corpus never reaches the model (the fake counts its calls: zero); the cite\ncheck removes a citation to a passage that was not shown and flags the answer; and the reranker\nbrings an answer from seventh place into the five shown, where without it the answer is \"I don't\nknow\". `npm test` runs them. `.github/workflows/test.yml` runs the type check, the tests and the\ngolden-set check on every push and pull request, with no secret: none of the three calls a\nmodel.\n\nThe same pipeline, three ways to hand it its models:\n\n```\nflowchart LR\n answer[\"answer(question, options, deps)
src/pipeline.ts imports no model\"] --> models[\"deps.models, a Models
chat() and embed(), src/models.ts\"]\n models --> real[\"realModels
Claude over HTTP, MiniLM on your machine
src/server.ts, evals/suite.ts\"]\n models --> fake[\"the fakes
test/fake-models.ts: answers from markers in the prompt
npm test, no key, no network\"]\n models --> traced[\"traced(models)
evals/tracer.ts: the real pair, every call recorded
evals/eval.ts\"]\n traced --> real\nnpm test\n```\n\nThe transcript, `transcripts/s2e2-models-as-a-parameter.txt`, shows `ANTHROPIC_API_KEY` unset\nand the five tests passing in about a tenth of a second. What these tests prove is the code\naround the model: the plumbing, the rules and the guards. They say nothing about whether the\nreal model answers right; that is what the golden set is for, and the next three episodes build\nthe harness that runs it. The two are different questions and need both: a fast test that runs\non every change, and an eval that runs when the answers may have moved. The pipeline's behaviour\nwith the real models did not change: the ten questions of series 1 still score 9/10 for four\ncents.\n\nEvery step of every case, written down, so a failure can be read instead of guessed at.\n\nA pass or a fail says nothing about why. When the staff-discount question fails, did the search miss the handbook, did the model see it and not use it, or did it answer and the check misread the answer? Each has a different fix, and only a record of what happened tells them apart. The record is the trace: for one case, the query, what retrieval returned and with what scores, what the model was shown, the prompt, the answer, the tokens, the cost and the time.\n\nFiles: `evals/tracer.ts` wraps a `Models` (the parameter of episode 2 pays off again):\n`traced(models)` returns models that behave exactly like the ones given and a list, `calls`,\nthat each embedding and each chat call appends to, with the text in, the reply out, the stop\nreason, the tokens, the cost and the milliseconds. The pipeline does not know it is traced.\nRetrieval is not a model call, so `answer` now reports it in its result: `retrieved`, the vector\nsearch's top 20 in order with chunk id, where and score, and `shown`, the passages the model\nsaw, in the order it saw them, so `[2]` in an answer is `shown[1]`. The search always fetches 20\nfor this record; the vector-only pipeline still shows the model the first five, so its answers\nare unchanged. `src/config.ts` holds `SERVED`, the options the server runs with (the cite\ncheck), so the evals measure what is served unless told otherwise. `evals/eval.ts ` runs\nthe golden set through the traced pipeline, with the served options or the flags of series 1,\nand writes `evals/runs//traces/.json`, one file per case, refusing a run name that\nexists. `evals/trace.ts ` prints one trace readably: the question, the top eight\nretrieved with rank, score and chunk, the passages shown, each call, the prompt's size and its\nlast line, the answer wrapped, the citations resolved, the total. `test/tracer.test.ts` checks,\nwith the fakes, that the tracer records the embedding and the chat call with the prompt and the\nreply.\n\n```\nnpx tsx --env-file=.env evals/eval.ts baseline\nnpx tsx evals/trace.ts baseline staff-discount\n```\n\nThe first run, `evals/runs/baseline/`, is committed: thirty traces, $0.1001 for the thirty\nquestions, between one and two and a half seconds each. The transcript,\n`transcripts/s2e3-the-tracer.txt`, lists the traces and reads one, series 1's failure, and it\nanswers the question episode 8 could only state: the handbook is not among the top eight\nretrieved at all; the five passages shown are the Second Light program in `about.md`, the Base\nCamp Bundle, the FAQ on refurbished lanterns, Lights for Scouts and the company's beliefs; the\nmodel read 1,017 tokens of them, wrote 52 and said \"I don't know.\" for a quarter of a cent. That\nis a retrieval miss, read from the record in ten seconds, and the model's answer was the right\none for what it was shown. The rest of the run, read the same way, has its own surprises: the\nbench-fee answer quotes the $25.00 fee and then says it does not know what an inspection alone\ncosts, a hedge a string match would count as a pass. Telling those apart is scoring, the next\nepisode.\n\nA pass or a fail per case, and for every fail the stage that failed: retrieval, the rules, the judge.\n\nFiles: `evals/checks.ts` is what rules can decide. `retrieval` finds where the first chunk\nholding one of the case's quotes came in the search's top 20, and whether one was shown; a chunk\nholds a quote when it is from the same file and page and contains the quote, or either half of\nit, because a chunk boundary can cut a quote in two (a half under 30 characters could match by\nchance and does not count). `checkAnswer` finds which required facts the answer states, by their\nmatch terms; which forbidden claims it makes, by theirs; and its citations: how many, any number\nthat points past the passages shown, and whether a cited passage holds a source.\n`evals/judge.ts` is what rules cannot decide: whether every statement in the answer is backed by\nthe passages, and whether each `[n]` points at the passage that says it. A string match can find\n\"340\" in an answer; it cannot tell \"the 220R weighs 340 g, 30 g more than the 220's 310 g\" from\n\"the 220R weighs 310 g\". The judge is Claude Haiku 5.5, twenty times cheaper than the answering\nmodel and the same model the reranker has used since series 1, episode 8 (`JUDGE_MODEL`, through\nthe `model` option of `ask`, which prices each model it calls and refuses one it has no price\nfor, so no call can escape the cost). It reads what the answering model read, its system prompt and its prompt, and the answer,\nand returns `{ reason, grounded, cited }` through structured output, reason first so the verdict\nfollows from it. One call per answered case, about two hundredths of a cent each.\n`evals/scoring.ts` puts them in order. A case that said \"I don't know\" passes if it is a\nfallback case and fails if not; a fallback case that answered fails, however good the answer\nsounds, because the corpus does not say. An answer case is checked by the rules and then judged,\nand its result names the first thing that failed: a missing fact, a forbidden claim, no\ncitations, a citation to a passage not shown, ungrounded, miscited. It writes\n`evals/runs//results.json`; `evals/report.ts` prints the table; `evals/eval.ts` now scores\nthe run it traced, and `evals/score.ts ` scores a run again, after the cases or the checks\nchange, without running the pipeline. `test/checks.test.ts` tests the rules on hand-made traces.\n\nThe judge is a model, so it is evaluated before it is trusted. `evals/judge-cases.jsonl` holds\nfifteen answers labelled by hand, each a real prompt from the baseline run with either the\npipeline's own answer or a wrong one made from it: a changed number, a citation to the wrong\npassage, a claim no passage makes, the 220's weight given for the 220R, the wrong approver for a\n$700 refund. `evals/judge-eval.ts` runs the judge on them and reports its agreement with the\nlabels. It started as twelve. Scoring the baseline found the judge wrong three ways, and each\nbecame a labelled case: it failed the $65 free-shipping answer because it had not been shown the\nsystem prompt that says to prefer the more recent document (now it is); and it failed the\nbench-fee answer for being cautious and the torque answer for a sentence without a citation,\nneither of which is what it was asked to judge, so its instructions now say to judge the facts\nand not the style, and that a missing citation is not a wrong one. With those changes it agreed\nwith all fifteen labels on two runs.\n\n```\nnpx tsx --env-file=.env evals/judge-eval.ts\nnpx tsx --env-file=.env evals/eval.ts scored\n```\n\nThe transcript, `transcripts/s2e4-scoring.txt`, shows the judge's evaluation, then a scored run.\nThe judge agreed on 15 of 15 for grounded and 14 of 15 for cited: on this run it called the\n\"every 20 fills\" answer correctly cited, though it said in the same breath that the manual says\n10. The judge is nondeterministic too, which is why it is measured on every change to it and\nnever trusted on one verdict. The run, `evals/runs/scored/`, passed 29 of 30 for $0.1057, judge\nincluded: all eight fallback cases said \"I don't know\", and the one failure is the staff\ndiscount, with no source in the top 20, a retrieval miss. The same scorer gave the baseline run\nof episode 3 (`evals/score.ts baseline`) 27 of 30: besides the staff discount, the two Firefly\nruntime and weight answers failed on a forbidden claim, because those answers mentioned the\nother model's figure as a contrast (\"30 grams more than the 220's 310 g\"), and this run's\nanswers did not. Same code, same cases, two points apart, and whether the case or the answer is\nwrong there is episode 6's question. The source column says more than the score. `return-window`\npassed with its source nowhere in the top 20: the FAQ says 40 days too, and the case lists only\nthe policy. `holiday-return` passed with its source 16th: the answer came from the FAQ and a\nworked example. A passing case with a missing source is a case that lists too few sources, or a\nlucky answer; either is worth knowing before it fails.\n\nA budget the harness enforces, a record of every run, and a comparison that says which cases moved.\n\nAn eval spends money on every run, and a harness that runs in CI or in a loop can spend a great deal of it without anyone deciding to. So the harness has caps, and a run that reaches one stops: the cases it did not reach are \"not run\", counted and shown, never passes. And a score on its own says little; what decides a change is the same cases scored before and after, side by side.\n\nFiles: `evals/ledger.ts` holds the caps, `RUN_CAP` ($1 a run) and `TOTAL_CAP` ($5 across every\nrun in the ledger), and the ledger itself, `evals/runs/ledger.json`: one line per run with when\nit started, the commit (marked `+changes` when `src/`, `docs/` or the cases had uncommitted\nedits, as a deliberate regression does), the configuration (the pipeline's options, its numbers\n`MIN_SCORE`, `SHOWN` and `POOL`, now exported by `src/pipeline.ts` as `SETTINGS`, the model, the\njudge and the budget), passed, failed, not run and the cost. `evals/eval.ts` now scores each\ncase as soon as it has run it and rewrites the run's ledger line after every case, so a run that\ndies still counts what it spent; before each case it checks the run's spend against its budget\n(`--budget` may lower it, never raise it above `RUN_CAP`) and the ledger's total against\n`TOTAL_CAP`, and once either is reached every remaining case is recorded as not run. The check\nis made between cases, so a run can pass its budget by at most one case. The results table and\nits summary show the not-run count. `evals/compare.ts ` lists every case whose\nstatus changed, with the reason it failed and where its source was retrieved in each run, then\nthe unchanged count, the passed totals and the cost change. Series 1's ladder, which had that\nname, is now `evals/ladder.ts`. The ledger starts with this episode: the `baseline` and `scored`\nruns of episodes 3 and 4, $0.21 between them, came before it.\n\n```\nnpx tsx --env-file=.env evals/eval.ts five\nnpx tsx --env-file=.env evals/eval.ts three # with SHOWN = 3 in src/pipeline.ts\nnpx tsx evals/compare.ts five three\nnpx tsx --env-file=.env evals/eval.ts capped --budget 0.02\n```\n\nThe transcript, `transcripts/s2e5-caps-ledger-compare.txt`, is a change proposed and refused by\nits numbers. The served pipeline scores 26 of 30 for $0.1046 (run `five`; `scored`, the same\ncode, scored 29 an episode earlier: the bench fee said \"I don't know\" this time, and the two\nFirefly answers mentioned the other model's figure again). Then the change: show the model three\npassages instead of five, which makes every prompt shorter. It is 31% cheaper, $0.0724, and\nscores 25: one point down, inside the noise, and on the score alone it would ship. The\ncomparison says otherwise. Two cases went from pass to \"I don't know\", the generator torque and\nthe Sentinel 1000's runtime on low, and in both the source chunk came 4th in the search: shown\nwith five passages, not with three. Those are not noise; they are the change, and the trace says\nwhy. A third case went the other way, the Firefly 220's runtime, for no reason the change\nexplains: its source was 4th as well, not shown, and it answered from the FAQ without the\ncontrast that trips the case. The change is reverted (`git checkout src/pipeline.ts`). Last, a\nrun with a two-cent budget: it ran six cases, spent $0.0224, one case past the budget as the\ncheck allows, and recorded the other 24 as not run, so its 5 passes cannot be read as a score.\nThe ledger then reads back all three runs, with the setting that differed: $0.1994 of the $5.\n\nFour ways a case fails, told apart from the record, and each fixed where it was actually wrong.\n\nA failing case is a question for the engineer: what broke? There are four answers, and each\npoints somewhere different. A retrieval miss: no passage holding the answer was shown, so the\nmodel could not have answered; look at the query and the chunks. A wrong fallback: the model\nsaid \"I don't know\" with the source in front of it, or answered a question the corpus does not\nanswer; look at the prompt and the passage. An answer miss: the model saw the source, answered,\nand got it wrong; look at the answer against the passage, and at the case's own terms. A\ncitation miss: the facts are right and the citations are missing or point at a passage that does\nnot say it; look at the `[n]` against the passages. And the fix is not always in the code: the\ncode, the corpus, the case or the harness can each be the thing that is wrong, and the trace\nsays which.\n\nFiles: `evals/stages.ts` is the labelling, in an order that matters: a fallback case that\nanswered is a wrong fallback; then, for an answer case, a source never shown is a retrieval\nmiss, whatever the answer; a source shown and \"I don't know\" is a wrong fallback; a wrong fact,\na forbidden claim or an ungrounded answer is an answer miss; what is left is a citation miss.\n`evals/diagnose.ts ` groups a run's failures by stage, with the reason and where the source\nranked, and says where to look first. `test/stages.test.ts` checks each label. `src/rewrite.ts`\nand the pipeline's `rewrite` option (`--rewrite` on the command line) are the fix for the\nretrieval miss, below; `src/config.ts` now serves it.\n\nThe four, found in the runs of episode 5 and this one, and fixed:\n\n- \n**Retrieval miss, fixed in the code.**`staff-discount` in run`five` : source not in the top\n20, as in every run since series 1. The question says staff, the handbook says employee, and\nan embedding of the question lands among the Second Light passages. The fix is query\nrewriting: one call to the model turns the question into a search query in the words company\ndocuments use (\"Second Light lantern employee discount staff purchase policy\"), and the\nrewrite is what is searched. That query alone puts the handbook's employee discount 4th.\nSearching both the question and the rewrite and fusing the lists was tried first and lost it\nagain: the question's list, all Second Light, outvoted the one passage the rewrite found.\nMeasured (run`rewrite` ,`--cite-check --rewrite` , against run`corpus` , the same without the\nrewrite): the staff discount passes with its source 4th and shown, no other case changed, 30\nof 30, for 21% more ($0.1257 against $0.1042, the rewrite call costs about $0.0004). It is now\nserved.\n- \n**Answer miss, fixed in the case.**`firefly-220r-weight` and`firefly-220-low-runtime` in run`five` : source 4th and shown, the right figure in the answer, and a forbidden claim matched.\nThe answers were right (\"340 g, 30 grams more than the Firefly 220's 310 g\"); the case's\nforbidden term, \"310\", was wrong, because it matched the contrast as well as the mistake. The\nterms are now the mistake itself (\"220R weighs 310\", \"220 runs 48\"). Measured by scoring run`five` again under the new cases without running the pipeline (`evals/runs/five-recased/` ,`evals/score.ts` ): both pass, nothing else moved, 26 to 28.\n- \n**Wrong fallback, fixed in the corpus.**`bench-fee` in run`five` : source 1st and shown, and\nthe answer was \"I don't know. The passages give the paid-repair bench fee of $25.00 plus parts\n[1], but none of them mentions a charge just for inspecting\". The pipeline keeps only \"I don't\nknow.\" of such a reply, as it should. The model was right: the manual gives the fee for a\nrepair, the warranty policy says a customer can decline the quote, and nothing says what\ndeclining costs. That is a gap in the documentation, the question customers will ask, and the\neval found it.`docs/warranty.md` now says it: \"The bench fee of $25.00 pays for the\ninspection and the quote, and is charged whether or not the customer goes ahead with the\nrepair.\" The policy is the fictional company's to decide; deciding it is the fix. The case\nlists the new sentence as a second source. Measured (run`corpus` against`five-recased` ):\nbench-fee passes, citing the new sentence, nothing else moved, 28 to 29.\n- \n**Citation miss, fixed in the harness.** No run of the served pipeline produced one; run`widen` (`--cite-check --rewrite --widen` , each passage shown with its neighbours) did:`refund-approval` failed as miscited. Read from the trace, the answer was right and so were\nits citations: \"[1][2][4]\" follow \"Tomas Reyes must approve it\", and passage 4, the handbook,\nsays exactly that. The judge had attached [4] to the next sentence. The judge was wrong, so\nthe judge is what changed: it now returns a verdict per citation, each with the statement it\nis attached to (\"the sentence or clause just before it, not a later one\"), and the code\ndecides \"cited\" from those. The answer joined the judge's labelled cases (now sixteen); the\njudge agreed with all sixteen on two runs. Measured by scoring`widen` again with the new\njudge (`widen-rejudged` ): refund-approval passes. So did`staff-discount` , which`widen` had\nfailed as an answer miss for saying the employee discount applies \"on top of\" the Second Light\nprice; that is a reading of the handbook, not a quote, and the judge called it ungrounded once\nand grounded once. A borderline verdict that flips on a rerun is noise in the judge, and the\nright response is to read it, not to tune for it.\n\n```\nnpx tsx evals/diagnose.ts five\nnpx tsx --env-file=.env evals/eval.ts rewrite --cite-check --rewrite\nnpx tsx evals/compare.ts five rewrite\n```\n\nThe transcript, `transcripts/s2e6-diagnosing-a-failure.txt`, shows run `five` diagnosed (two\nanswer misses, a wrong fallback, a retrieval miss), run `widen` diagnosed (an answer miss and\nthe citation miss), then the comparisons: `five` to `rewrite`, all three fixes together, 26 to\n30 of 30, for 20% more; and `widen` to `widen-rejudged`, 28 to 30. Two lessons from the four.\nHalf the fixes were not in the pipeline: a case that was wrong and a document that was missing a\nsentence are as common as a bad prompt, and a harness that is only ever used to change the code\nwill make the code worse to satisfy a bad case. And widening, measured again here, costs twice\nas much ($0.2547) for nothing these thirty cases need.\n\nThe eval stops a bad change before it merges, learns from real questions, and runs every month whether anyone changed anything or not.\n\nFiles: `evals/run-eval.ts` is the run of episodes 3 to 6 as a function, `runEval(run, options, budget)`, so two scripts can share it: `evals/eval.ts`, as before, and `evals/gate.ts`, the\ngate. The gate runs the served pipeline on the golden set with a $0.50 budget and fails, with\nexit code 1, when less than `THRESHOLD` (85%) of the cases pass, when any fallback case\nanswered, or when the budget stopped the run. The threshold sits under the noise: the served\npipeline has scored between 26 and 30 of 30 across runs, so a change that trips it has done real\nharm, and a fallback case that answers fails the gate on its own because an invented answer is\nthe failure that costs most. `.github/workflows/eval.yml` runs the gate on every pull request\nthat touches `src/`, `docs/`, `evals/` or the dependencies, on the 1st of every month, and by\nhand: it installs, restores the embedding model from the cache, checks the golden set, builds\nthe index, runs the gate with the key from the repository secret `ANTHROPIC_API_KEY`, and keeps\nthe run's traces and results as an artifact, so a red check can be read with `evals/trace.ts`. A\nnewer push cancels the running check, so one pull request never pays twice; the ledger's caps\napply in CI as well, against the ledger committed in the branch. The\nsecret is set in the repository's settings (Settings, Secrets and variables, Actions), and making the\ncheck required is a branch protection rule. A fork needs its own key there, or the gate fails for want of\none; the `test` workflow needs no secret at all.\n\n`src/log.ts` appends every question the server answers to `logs/questions.jsonl` (gitignored),\nwith the answer and what it cited; questions can carry personal details, so that file belongs\nwhere your other production logs live, under the same retention. `evals/sample.ts [n]` turns the\nlog into draft cases: it drops questions the golden set already has and repeats, takes a random\nsample of n, and writes `evals/drafts.jsonl` (gitignored too) with what the pipeline said. A\ndraft is not a case: a person reads it against the documents, writes the facts, forbidden claims\nand quotes or marks it a fallback, moves it into `golden.jsonl`, and runs `check-golden.ts`. The\nfirst draft built that way is in the set: asked through the server while this episode was built,\n\"Does the Northstar come with a carry case?\" got \"I don't know.\", and the catalog's Northstar\ntable says \"In the box | Lantern, two mantles, funnel, carry case\". It is case 31,\n`northstar-carry-case`, and it fails: the next thing to diagnose. Its first match terms included\n\"comes with a carry case\", which a later answer matched by saying the passages do not say\nwhether it \"comes with a carry case\"; they are now \"yes\" and phrases that only an affirmative\nanswer contains, the lesson of episode 6 again.\n\n```\nnpx tsx --env-file=.env src/server.ts &\ncurl -s localhost:3000/ask -d '{\"question\": \"Does the Northstar come with a carry case?\"}'\nnpx tsx evals/sample.ts 5\nnpx tsx --env-file=.env evals/gate.ts\n```\n\nThe transcript, `transcripts/s2e7-the-harness-as-a-gate.txt`, shows two questions asked through\nthe server, both answered \"I don't know.\", and the sample drafting the one the golden set does\nnot have yet (the dog in Annapolis; the carry case is already case 31). Then the bad change: one\nline deleted from the system prompt, the rule \"When the passages do not answer the question,\nreply exactly: I don't know.\", the kind of cleanup that looks harmless in review. The gate fails\nit: every one of the eight fallback cases now answered (with the CEO's salary, the dental\ndeductible and the brass spinner's name, the model now explains what the passages do not say\ninstead of saying so in four words), two answer cases failed too, 21 of 31 against the 27\nneeded, exit code 1, for $0.1373. The line is restored and the gate passes: 30 of 31, every\nfallback right, the carry case the one failure, exit code 0, for $0.1323.\n\nThe model behind the API is not part of this repository and it changes: a model version is\nretired on a published date, and its replacement answers differently. The pipeline's code can be\nidentical and its answers worse. The scheduled run of `eval.yml` catches the bad case: a gate\nthat fails on the 1st of the month with no pull request behind it is drift. Every month, and\nbefore any model change:\n\n1. Rerun: `npx tsx --env-file=.env evals/eval.ts monthly-2026-12` .\n2. Compare with the last run in the ledger at the same settings (`jq -r '.runs[] | \"\\(.run) \\(.at) \\(.passed)\"' evals/runs/ledger.json` ):`npx tsx evals/compare.ts rewrite monthly-2026-12` . Read every case that moved with`evals/trace.ts` , and label the failures\nwith`evals/diagnose.ts` .\n3. To move to a new model, or off a deprecated one: change `MODEL` in`src/claude.ts` and its\nline in`PRICES` (and`model` in`ask.sh` ), rerun under a new name, compare. The judge is a\nmodel too: when`JUDGE_MODEL` changes, run`evals/judge-eval.ts` first and trust the new\njudge only at the same agreement. So is the reranker:`RERANK_MODEL` in`src/rerank.ts` changes the same way, measured with the reranker switched on (`evals/ladder.ts` ).\n4. Ship the swap when the comparison shows nothing lost that matters, and the ledger keeps the run that proved it.\n\nWithout a key, at no cost:\n\n```\nnpm run typecheck # tsc over src/, evals/ and test/\nnpm test # the pipeline's rules, the tracer, the checks and the stage labels, on fakes\nnpx tsx evals/check-golden.ts # every quote in the golden set is in the corpus\nnpx tsx src/index.ts # build lantern.db from docs/ (downloads the 97 MB embedding model once)\nnpx tsx src/search.ts \"what is the bench fee for an out-of-warranty repair\"\nnpx tsx evals/trace.ts baseline staff-discount # read one case of a committed run\nnpx tsx evals/compare.ts five three # two committed runs, case by case\nnpx tsx evals/diagnose.ts five # a committed run's failures, by the stage that failed\n```\n\nWith a key in `.env`, for cents:\n\n```\nnpx tsx --env-file=.env src/ask.ts \"In one sentence: what does HTTP status 304 mean?\"\nnpx tsx --env-file=.env src/server.ts # then: curl -s localhost:3000/ask -d '{\"question\": \"...\"}'\nnpx tsx --env-file=.env evals/run.ts --hybrid --rerank # series 1's ten questions, about five cents\nnpx tsx --env-file=.env evals/ladder.ts # the five rows of episode 8's table, about thirty cents\nnpx tsx --env-file=.env evals/eval.ts my-first-run # the golden set, traced and scored, about thirteen cents\nnpx tsx evals/compare.ts rewrite my-first-run # against the committed run of the same settings\nnpx tsx --env-file=.env evals/judge-eval.ts # the judge against its labelled answers, under half a cent\nnpx tsx --env-file=.env evals/gate.ts # what CI runs: exit 0 or 1, about thirteen cents\n```\n\nA run's name must be new: `evals/eval.ts` refuses one that exists, so the committed runs stay as the\nepisodes read them. Your runs land in `evals/runs/` beside them, each with its line in the ledger.\n\n`.gitignore` keeps out what each machine makes for itself:\n\n- `.env` : your key.`.env.example` is the template.\n- `lantern.db` : the index.`npx tsx src/index.ts` builds it from`docs/` in a few seconds. (At`s1e5` ,`src/search.ts` fills its own store on first run; from`s1e6` on, a search against an empty store says to\nrun the index.)\n- `node_modules/` , and inside it the embedding model: 97 MB, downloaded by the first search or index into`node_modules/@huggingface/transformers/.cache` , and never again.\n- `logs/questions.jsonl` : every question the server answered, from`s2e7` on (`src/log.ts` ).\n- `evals/drafts.jsonl` : what`evals/sample.ts` drafts from that log.\n\nCommitted on purpose: `evals/runs/`, the runs the episodes read their numbers from, every trace included,\nand `evals/runs/ledger.json`; `transcripts/`, the real output of each episode's commands; and\n`docs/repair-manual.pdf`, the one PDF in the corpus.\n\n- **\"ANTHROPIC_API_KEY is not set\"** : the command is missing`--env-file=.env` , or`.env` is not there. Node\nreads the file itself from version 20.6 on (`node --version` ).\n- **`better-sqlite3` will not load** (\"was compiled against a different Node.js version\"):`npm rebuild better-sqlite3` . Its prebuilt binary is for one major version of Node at a time.\n- **\"Cannot find package\"** after checking out a tag:`npm install` again. The dependencies change at`s1e5` ,`s1e6` and`s2e2` .\n- **The first search or index sits for a while** : the embedding model is downloading, once. After that, the\nonly thing that needs the network is Claude.\n- **\"the store is empty\"** : run`npx tsx src/index.ts` first.\n- **\"exists; name the run something new\"** :`evals/eval.ts` never overwrites a run.\n- **Cases \"not run\"** : the run reached its budget (`--budget` , never above`RUN_CAP` ) or the ledger reached`TOTAL_CAP` (`evals/ledger.ts` ). Read the ledger, then raise the cap in the code on purpose.\n- **Port 3000 is taken** :`PORT` in`src/server.ts` .\n- **Your numbers differ from this README's by a case or two** : the models are nondeterministic, and every\nnumber here is a committed run in`evals/runs/` or a transcript in`transcripts/` . A difference of one case\nin thirty is noise; series 2, episode 5 is about telling noise from a change.\n- **`npm run typecheck` at `s1e1` says no inputs were found** : there is no TypeScript yet at that tag, only`ask.sh` . It works from`s1e2` on.\n- **`./ask.sh` prints nothing** :`jq` is missing (`brew install jq` ), or the key in`.env` is not set.\n\nOne commit and one tag per episode, in the order the episodes play; `git log --oneline` is the series.\n\n| Tag | Episode | Video | \n|---|---|---|\n| `s1e1` | Series 1, episode 1: What actually changed, and what didn't | [https://youtu.be/55s2Yw-UBgc](https://youtu.be/55s2Yw-UBgc) | \n| `s1e2` | Series 1, episode 2: Calling a model from code | [https://youtu.be/WnyUC2Qqw-4](https://youtu.be/WnyUC2Qqw-4) | \n| `s1e3` | Series 1, episode 3: Prompts are code | [https://youtu.be/Hpfw67Trg7k](https://youtu.be/Hpfw67Trg7k) | \n| `s1e4` | Series 1, episode 4: Why it can't read your documents | [https://youtu.be/b8ZsS2FKypw](https://youtu.be/b8ZsS2FKypw) | \n| `s1e5` | Series 1, episode 5: Embeddings, for people who know SQL | [https://youtu.be/fPsBmU_FSaw](https://youtu.be/fPsBmU_FSaw) | \n| `s1e6` | Series 1, episode 6: Chunking and indexing is ETL | [https://youtu.be/_LzjKfoXAvA](https://youtu.be/_LzjKfoXAvA) | \n| `s1e7` | Series 1, episode 7: Your first pipeline | [https://youtu.be/GZPzeVkgWNs](https://youtu.be/GZPzeVkgWNs) | \n| `s1e8` | Series 1, episode 8: Making it good, with numbers | [https://youtu.be/OfuJPNoKeSU](https://youtu.be/OfuJPNoKeSU) | \n| `s2e1` | Series 2, episode 1: The golden set | [https://youtu.be/FkJjuXGa9CE](https://youtu.be/FkJjuXGa9CE) | \n| `s2e2` | Series 2, episode 2: Models as a parameter | [https://youtu.be/50XGeIVL_eA](https://youtu.be/50XGeIVL_eA) | \n| `s2e3` | Series 2, episode 3: The tracer | [https://youtu.be/zEEFXUjOCjA](https://youtu.be/zEEFXUjOCjA) | \n| `s2e4` | Series 2, episode 4: Scoring | [https://youtu.be/cEGDHGmIix4](https://youtu.be/cEGDHGmIix4) | \n| `s2e5` | Series 2, episode 5: Caps, the ledger and comparing runs | [https://youtu.be/nQiUFUrc0eA](https://youtu.be/nQiUFUrc0eA) | \n| `s2e6` | Series 2, episode 6: Diagnosing a failure | [https://youtu.be/s6Zglb47T78](https://youtu.be/s6Zglb47T78) | \n| `s2e7` | Series 2, episode 7: The harness as a gate | [https://youtu.be/w6aXAi353GE](https://youtu.be/w6aXAi353GE) | \n\nEvery dependency, with its one-line reason.\n\n- `@huggingface/transformers` : runs the embedding model on your machine (ONNX Runtime\nunderneath); no embedding API, no key.\n- `better-sqlite3` : SQLite in-process and synchronous, the simplest database there is; npm\ninstalls a prebuilt binary.\n- `unpdf` : text out of a PDF, page by page; plain JavaScript on top of pdf.js, no native build.\n\nDevelopment dependencies:\n\n- `typescript` : the type checker (`npm run typecheck` ).\n- `tsx` : runs a TypeScript file directly (`npx tsx file.ts` ), no build step.\n- `@types/node` : the types of Node's own modules (`fs` ,`http` ,`process` ).\n- `@types/better-sqlite3` : the types of`better-sqlite3` .\n\n`transcripts/` holds the real output of each episode's commands, recorded when the episode was\nbuilt: the command line first, then what it printed, long lines wrapped to fit a slide. Nothing\nin them is edited or invented. Each file is named for its tag.\n\nMIT, in `LICENSE`: use the code, the corpus and the golden set as you like, with the notice kept.", "url": "https://wpnews.pro/news/show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode", "canonical_source": "https://github.com/AiFirstDevelopment/engineer-to-ai-engineer", "published_at": "2026-11-12 13:15:53+00:00", "updated_at": "2026-11-12 13:21:36.524322+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-tools", "developer-tools", "large-language-models"], "entities": ["Joel Stevick", "From engineer to AI engineer", "Prove it works", "AiFirstDevelopment", "GitHub", "Anthropic", "claude-sonnet-5-5", "TypeScript"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode", "markdown": "https://wpnews.pro/news/show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode.md", "text": "https://wpnews.pro/news/show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-free-course-from-engineer-to-ai-engineer-one-git-tag-per-episode.jsonld"}}