The code for the YouTube series From engineer to AI engineer: eight episodes that take an experienced software engineer with no generative-AI background from conventional software to a first retrieval pipeline (RAG), built across the series one piece per episode. Its second series, Prove it works, builds the evaluation harness that tells you whether that pipeline answers right, on the same repository.
Every episode is one commit and one tag: s1e1 to s1e8 for the first series, s2e1 to s2e7 for the
second. Check out a tag and you have the repository exactly as that episode leaves it, with a README that
describes only what exists so far. This README, on main, describes all of it.
New to AI engineering? This is free, and it starts from zero: the YouTube channel has every episode, joelstevick.com has each episode's page, a free demo and the book, and the weekly newsletter AI That Won't Confidently Lie teaches one piece at a time.
The videos are two YouTube playlists: From engineer to AI engineer, eight episodes, and Prove it works, seven. Every episode's description links its tag here, and the table under "Tags" at the end links every tag to its episode.
To follow along at the keyboard:
git clone https://github.com/AiFirstDevelopment/engineer-to-ai-engineer.git
cd engineer-to-ai-engineer
git checkout s1e1 # the repository exactly as episode 1 leaves it
cp .env.example .env # then put your Anthropic API key in .env
npm install
./ask.sh "In two sentences: what is a race condition?"
Then git checkout s1e2 and that episode's commands, and so on to s2e7. Checking out a tag puts git on a
detached HEAD, which is fine for reading; git checkout main comes back. Run npm install again after
checking out s1e5, s1e6 or s2e2, the tags whose package.json adds dependencies, or whenever a command
fails on a missing package. main is the finished repository: every file this README names is there.
What a command costs is in its episode's section: cents, never dollars. The harness caps its own spending
at $1 a run and $5 across every run in its ledger (evals/ledger.ts), and nothing in the test suite calls
a model.
A model is a function. Text goes in, text comes out. It keeps no state between calls, so every request carries the whole conversation. It is nondeterministic: the same input can give a different output. And it is priced by the token, in and out.
That is what changed. Everything around it did not. You reach it over HTTP. Your data still
lives in a database. Your tests still have to prove the thing works. Deployment is what it was
last year. This series builds the one new piece, a retrieval pipeline, with tools you already
have: TypeScript, SQLite, curl and an HTTP server.
- Node 20 or later (
node --version) andjq(brew install jq). - An Anthropic API key from https://console.anthropic.com . The runs in this series cost cents; each episode's section says what its commands cost.
cp .env.example .envand put your key in.env. The file is gitignored.ask.shreads it, and everynpx tsxcommand below loads it with Node's own--env-file=.env, so there is no dotenv package.npm install. The first search (episode 5 on) downloads the embedding model, 97 MB, once.
The series answers with one chat model, claude-sonnet-5-5, set in one place (the request body
in ask.sh, and the constant MODEL in src/claude.ts). It costs $2 per million input tokens
and $10 per million output tokens. A question through the finished pipeline is a few thousand
tokens in and a few hundred out, about a cent. Claude Opus 5.5 is the more capable model at
twice the price per token; Claude Haiku 5.5 is twenty times cheaper. The answers stay on
Sonnet: an answer is what the customer reads, and we want answers we can trust while we grade
them. You pay for your own runs, so the cheaper of the two models we trust is the default, and
swapping the constant is the whole migration. One step uses Haiku: from episode 8 the reranker
(RERANK_MODEL in src/rerank.ts) scores twenty passages for each question, a narrower job
than writing a cited answer and four times the text, and Haiku does it well enough at a
twentieth of the price per token (episode 8 measures it).
docs/ is the internal documentation of a fictional company, Lantern Supply Co., a Maryland
retailer of camping lanterns and lamps: twelve Markdown files and one PDF, about 26,000 words,
full of specific facts (prices, model numbers, dates, names, policies) that no model has seen.
That is the point: in the episodes to come the model has to answer from these files or admit it
cannot. The facts contain traps on purpose: two products with near-identical names (the Firefly
220 and the Firefly 220R), a policy that a later document changed (the free-shipping threshold
is $75 in shipping.md and faq.md, and release-notes.md lowers it to $60 in version 4.3),
and facts that exist only in the PDF (repair-manual.pdf, the Repair Center's bench manual).
| File | What it is |
|---|---|
about.md |
The company, its history, locations and leadership |
catalog.md |
Every product with model number, price and specifications |
returns-policy.md |
The returns policy (40 days from delivery, and the rest) |
shipping.md |
Rates, carriers, fuel shipping limits |
warranty.md |
Warranty terms by product line and the claim process |
faq.md |
Customer-facing questions and answers |
employee-handbook.md |
An excerpt: pay, time off, discount, safety |
customer-service-playbook.md |
The support team's procedures and reply templates |
release-notes.md |
Release notes of Hearth, the online store |
fuel-safety.md |
Approved fuels, storage, disposal |
lumen-rewards.md |
The loyalty program |
showrooms.md |
The Frederick and Annapolis showrooms |
repair-manual.pdf |
The Repair Center bench manual (PDF only, nine pages) |
The PDF was rendered once from a Markdown draft with headless Chrome; the draft is not in the repository, on purpose, so its facts exist only in the PDF.
Three jobs, in three groups of files. The first two are series 1 and the third is series 2. Every box is a file in this repository; the episode sections below say what each does and what it printed when it ran.
Indexing, npx tsx src/index.ts: the documents become searchable chunks in a SQLite file. It is an ETL
job, and it is safe to run again: a hash of each file says which ones changed.
flowchart LR
docs["docs/<br/>12 Markdown files and 1 PDF"] --> extract["src/extract.ts<br/>text, page by page"]
extract --> chunk["src/chunk.ts<br/>chunks of about 1,200 characters,<br/>each with its source, heading and page"]
chunk --> embed["src/embed.ts<br/>one 384-number vector per chunk,<br/>all-MiniLM-L6-v2, on your machine"]
embed --> db[("lantern.db<br/>SQLite: chunks, files")]
job["src/index.ts<br/>hashes each file: skips the unchanged,<br/>replaces the changed, removes the deleted"] -.-> extract
Answering, npx tsx --env-file=.env src/server.ts: a question comes in over HTTP and an answer with
citations goes out, or "I don't know." The stages marked option are switched on by Options in
src/pipeline.ts, one flag each, so the evals can measure each on its own; what the server runs with is
SERVED in src/config.ts, the rewrite and the cite check.
flowchart TD
q["POST /ask with a question<br/>src/server.ts"] --> rewrite["src/rewrite.ts, option, served<br/>the question rewritten in the documents' words"]
rewrite --> retrieve["src/retrieve.ts<br/>the 20 nearest chunks by cosine similarity<br/>src/embed.ts, src/cosine.ts, lantern.db"]
retrieve --> low{"best score<br/>under 0.25?"}
low -- yes --> idk["I don't know.<br/>no model is called"]
low -- no --> hybrid["src/keyword.ts and src/hybrid.ts, option<br/>BM25 keyword search, the two lists fused by rank"]
hybrid --> rerank["src/rerank.ts, option<br/>Claude Haiku 5.5 scores the 20 from 0 to 10"]
rerank --> five["the top 5"]
five --> widen["src/widen.ts, option<br/>each passage with its neighbours"]
widen --> prompt["src/prompt.ts<br/>the rules, and the passages numbered [1] to [5]"]
prompt --> claude["src/claude.ts<br/>Claude Sonnet 5.5, one HTTP call"]
claude --> said{"I don't know?"}
said -- yes --> idk
said -- no --> cite["src/cite-check.ts, option, served<br/>a [n] the model was not shown is removed, the answer flagged"]
cite --> out["the answer, each citation resolved to its file and heading, the cost"]
out -.-> log["src/log.ts<br/>logs/questions.jsonl"]
Proving it, npx tsx --env-file=.env evals/eval.ts <run>: the golden set goes through the same
pipeline, every step is recorded, every case is scored, and the run is compared with the one before. In CI
the same run is a gate.
flowchart LR
golden["evals/golden.jsonl<br/>31 cases: the question, the facts, the forbidden claims, the sources"] --> check["evals/check-golden.ts<br/>every quote is in the corpus"]
golden --> run["evals/eval.ts and evals/gate.ts<br/>evals/run-eval.ts: the pipeline on traced models"]
run --> traces["evals/runs/NAME/traces/*.json<br/>evals/tracer.ts: every call, every passage, every cent"]
traces --> score["evals/scoring.ts<br/>the rules first, evals/checks.ts,<br/>then the judge, evals/judge.ts, Claude Haiku 5.5"]
score --> results["evals/runs/NAME/results.json"]
results --> read["evals/report.ts, evals/compare.ts,<br/>evals/diagnose.ts, evals/trace.ts"]
run --> ledger["evals/runs/ledger.json<br/>every run, with its settings and cost<br/>caps: $1 a run, $5 in all"]
results --> gate{"85% passed, no fallback case answered,<br/>the budget not reached?"}
gate -- no --> red["exit 1: the pull request is red"]
gate -- yes --> green["exit 0"]
cases["evals/judge-cases.jsonl<br/>16 answers labelled by hand"] --> judgeeval["evals/judge-eval.ts<br/>the judge's agreement with the labels"]
log["logs/questions.jsonl<br/>what people asked the server"] --> sample["evals/sample.ts<br/>drafts of new cases, for a person to finish"] --> golden
The skeleton and one HTTP call.
Files: package.json, tsconfig.json, .gitignore, .env.example, the corpus in docs/,
and ask.sh: one curl to the Messages API, the request body written out in the script, the
answer's text and the token counts printed with jq. Nothing reads docs/ yet.
./ask.sh "In two sentences: what is a race condition?"
The transcript, transcripts/s1e1-what-changed.txt, shows the call and the answer: a request
body with a model name, a limit on the answer's length and a list of messages; a response with
the text and the number of tokens in and out. Nothing else is needed. Cost of the run: a
fraction of a cent.
The same call from TypeScript, and the only client this repository will ever have.
Files: src/claude.ts is the client: ask(messages, options) is one fetch to the Messages
API that returns the answer's text, the token counts and the cost in dollars, computed from the
price list in one constant, with the model id in another; countTokens(text) asks the API's
count-tokens endpoint, which is free. src/ask.ts is a command line around ask, and
src/tokens.ts one around countTokens. There is no SDK: it is an HTTP API like any other, and
the whole client is shorter than most SDK quick-starts.
npx tsx --env-file=.env src/ask.ts "In one sentence: what does HTTP status 304 mean?"
npx tsx --env-file=.env src/ask.ts \
"I bought a lantern from Lantern Supply Co. How many days do I have to return it?"
npx tsx --env-file=.env src/tokens.ts "The Firefly 220R charges in 3.5 hours over USB-C."
The transcript, transcripts/s1e2-calling-a-model.txt, shows three things. A general question
is answered well, in 79 output tokens, for a twelfth of a cent. The question about Lantern
Supply Co.'s return window is not answered: the model has never seen docs/returns-policy.md
(40 days from delivery), says it does not know the company, and sends the customer to their
receipt. It cannot know. That is the problem the rest of the series solves, and notice that a
model which guessed a number instead would sound exactly as sure. Last, a 49-character sentence
is 32 tokens: a token is a few characters, a short word is one, and numbers, model names and
punctuation split into several. Everything you pay for and every limit you will meet is counted
in tokens. Cost of the three runs together: under half a cent.
Two ways to make the model's output something a program can use.
Files: src/structured.ts asks for JSON instead of prose. The shape is written twice, once as
a TypeScript type for the code and once as JSON Schema for the API, which holds the model to
it (output_config.format). The code still parses the text and checks the shape with a type
guard, because the model's output is input. src/tools.ts is tool use: one tool,
lookup_order(order_id), read from data/orders.json, and the loop that runs it. The model
answers with a tool_use block instead of text; we run the function, append the model's whole
turn and a tool_result to the conversation, and call again; it repeats until the model
answers in text. The loop is ours, and it is a plain while. src/claude.ts grows two options,
tools and format, and returns the whole content of a turn, not only its text.
The loop, as the transcript shows it:
sequenceDiagram
participant T as src/tools.ts
participant M as Claude
participant F as lookup_order, from data/orders.json
T->>M: the question, and the tool's name, description and input schema
M-->>T: a tool_use block, lookup_order("LS-2026-012077")
T->>F: run the function
F-->>T: the order's JSON
T->>M: the conversation so far, the model's turn, and a tool_result holding that JSON
M-->>T: text, delivered on October 29 by UPS Ground, with the tracking number
npx tsx --env-file=.env src/structured.ts
npx tsx --env-file=.env src/tools.ts "Has order LS-2026-012077 arrived yet?"
The transcript, transcripts/s1e3-prompts-are-code.txt, shows a customer's email turned into
an OrderSummary (the order id, the customer, two items with quantities, intent return) for
a fifth of a cent, and the tool loop: the model asks for lookup_order(LS-2026-012077), gets
the order's JSON from the file, and answers that it was delivered on October 29 by UPS Ground,
with the tracking number and the email's two Fireflies and Northstar, in two round trips for
half a cent. The prompt, the schema and the tool's description are code: they live in the
repository, they are reviewed, and from episode 7 on they are tested.
The obvious fix, and why it is a dead end.
Files: src/ask-with-doc.ts pastes a document into the prompt and asks the question after it.
Given one Markdown file it pastes that file; given the folder it pastes every Markdown file in
it (the PDF waits for episode 6). Before calling the model it counts the prompt's tokens with
the free count-tokens endpoint and prints what that much input costs on every question.
npx tsx --env-file=.env src/ask-with-doc.ts docs/returns-policy.md \
"In one sentence: how many days do I have to return a lantern?"
npx tsx --env-file=.env src/ask-with-doc.ts docs "What is the free shipping threshold?"
The transcript, transcripts/s1e4-why-it-cant-read-your-documents.txt, shows the return-window
question answered right, 40 days from delivery, once returns-policy.md is in the prompt:
3,720 tokens, three quarters of a cent of input per question. Then the whole corpus goes in,
45,331 tokens, and the model answers the free-shipping question right too, $60, and even
notices that the release notes are newer than the policy pages that still say $75. It can do
that because it sees everything. The price is nine cents of input per question, before the
answer, and it is paid again on every question because the model keeps nothing between calls.
That is the dead end. A thousand questions a day is $90 a day for 26,000 words of documentation; ten times the documentation is ninety cents a question and a long wait for each answer; thirty times does not fit at all, because this model's context window, the most it can read in one call, is a million tokens. The window is the limit; the price is the wall you hit first. The way out is to send only the few paragraphs that matter to the question, which means finding them first. Finding them is retrieval, and it is the next four episodes.
Search by meaning, with a table, a blob column and a loop.
Files: src/embed.ts turns a text into a vector, a Float32Array of 384 numbers, with a small
open embedding model, all-MiniLM-L6-v2, running on your machine through
@huggingface/transformers. Texts that mean similar things get vectors that point the same
way. The model downloads once, 97 MB into node_modules/@huggingface/transformers/.cache, and
nothing is sent anywhere: there is no second vendor and no second key, and the only paid call
in this repository is still Claude. src/cosine.ts is cosine similarity in ten lines: how alike
two vectors point, 1 for the same direction, 0 for unrelated. src/db.ts is SQLite through
better-sqlite3, one table, chunks(id, source, text, embedding), where the embedding is the
vector's bytes in a blob column like any other column. src/search.ts is the search: on its
first run it splits every Markdown file in docs/ into paragraphs (a quick split on blank
lines, for now), embeds each and inserts it; on every run it embeds the query, scores every row
with cosine in TypeScript, a full scan, and prints the top five.
npx tsx src/search.ts "what happens if the courier loses my parcel"
No --env-file: nothing here calls Claude. The transcript, transcripts/s1e5-embeddings.txt,
shows the first run indexing 414 paragraphs in three seconds, then the top five. The best
match, at 0.627, is the playbook's "A package counts as lost when it has not arrived 10
business days after the expected delivery date", and it shares not one word with the query:
courier against carrier and package, parcel against package, loses against lost. A keyword
search finds none of that. Meaning beats words, and that is the whole reason for embeddings.
Notice also that a bare heading line made the top five: paragraphs are a poor unit, which is
episode 6.
When does the full scan stop being fine? 414 rows of 384 floats is nothing; a hundred thousand
rows is still a few tens of milliseconds in a loop like this one. When you have millions of
rows, or many searches a second, you add a vector index (sqlite-vec, pgvector, or a vector
database), which is the same decision as adding an index to any other table, made for the same
reason, and not before.
Extract, transform, load. You have written this job before.
Files: src/extract.ts turns a file into text: a Markdown file as it is, a PDF page by page
through unpdf, with the page number kept, because "page 3 of the repair manual" is how a
person wants it cited. src/chunk.ts turns text into chunks, the unit we embed, store, search
and show the model: it splits at headings first, because a heading is where the author changed
the subject, then packs long sections paragraph by paragraph into pieces of about 1,200
characters (300 tokens or so), each starting with the last paragraph of the one before, the
overlap, so a fact on a boundary is whole in one of them. A PDF comes out as lines, not
paragraphs, so there a line is the unit, and it has no headings, so its chunks carry a page
instead. Every chunk keeps its source, heading and page. src/db.ts grows those columns, a
position (the chunk's place in its file, for episode 8), a files table with a SHA-256 hash
of each file, and a schema version: the store is a cache of docs/, and when its shape changes
it is thrown away and rebuilt. src/index.ts is the job: for each file in docs/, hash it;
if the hash is the one stored, skip it; otherwise delete its old chunks, extract, chunk, embed,
insert, and remember the hash; files that disappeared are removed. src/search.ts now only
searches, and shows where each hit came from.
npx tsx src/index.ts
npx tsx src/index.ts
npx tsx src/search.ts "what is the bench fee for an out-of-warranty repair"
The transcript, transcripts/s1e6-chunking-and-indexing.txt, shows the first run indexing
13 files into 287 chunks in 3.5 seconds (the 414 bare paragraphs of episode 5 are gone;
headings travel with their text now, and the FAQ's 54 question-and-answer pairs are the largest
share), the second run skipping all 13 in no time, and a search for the bench fee landing on
page 1 of the repair manual at 0.685: a fact that exists nowhere else in the corpus, found in a
PDF, with its page number. Nothing here calls Claude.
Why unpdf and not pdf-parse, the usual answer: it is plain JavaScript (pdf.js underneath),
an ES module, with no native build, and it gives the text page by page in one call; pdf-parse
is CommonJS and returns one blob for the whole file unless you hook into its page renderer.
Retrieve, prompt, answer: with a server in front and a test behind.
Files: src/retrieve.ts returns the top five chunks for a question (episode 5's search as a
function). src/prompt.ts is the system prompt and the layout: answer from the passages and
nothing else, cite every fact with the passage's number like [2], prefer the more recent
document when passages disagree, and when they do not answer, reply exactly "I don't know";
the passages are numbered and labelled with where they came from. src/pipeline.ts is
answer(question): retrieve; if the best score is under 0.25 the model is not even asked (a
question about nothing in this corpus scores 0.17 to 0.24 here; measure yours); otherwise ask,
and if the model says it does not know, cite nothing; otherwise resolve every [n] in the answer
to the passage it points at. src/server.ts is POST /ask on Node's own http module, JSON
in, answer and citations out, thirty lines. evals/questions.jsonl is ten questions, each with
the strings its answer must contain, phrased the way people ask and aimed at the traps: the
rechargeable Firefly, a $65 order, the bench fee, the torque, and the CEO's salary, whose only
right answer is "I don't know". evals/run.ts runs them through the pipeline and prints pass
or fail and the total cost.
npx tsx --env-file=.env src/server.ts &
curl -s localhost:3000/ask -d '{"question": "How many days do I have to return a lantern?"}'
curl -s localhost:3000/ask -d '{"question": "How much is the CEO paid?"}'
npx tsx --env-file=.env evals/run.ts
The transcript, transcripts/s1e7-your-first-pipeline.txt, shows the return-window question
answered right over HTTP, 40 days from delivery, citing [2] and [4], which resolve to
warranty.md > Relationship to the returns policy and returns-policy.md > Common situations,
for a third of a cent: twenty-five times cheaper than pasting the corpus in episode 4, and
faster. The question about the CEO's pay gets "I don't know." and no citations. The ten
questions score 9/10 for less than four cents. The failure is "Can staff buy a Second Light
lantern for less?": the pipeline answered "I don't know", because the handbook's employee
discount section was not among the five passages retrieved. The question says staff and Second
Light, and the nearest passage is the Second Light program in about.md; the other four, the Base
Camp Bundle, the FAQ on refurbished lanterns, Lights for Scouts and the company's beliefs, are
nearer to the question than the handbook is, which says employee, not staff. It is an honest
failure, which is the second rule working, and it is a number. Episode 8 is about that number.
Four changes that usually help, each behind an option, each measured against the ten questions.
Files: src/keyword.ts is BM25, the ranking function behind most search engines, in forty
lines and no library: it finds exact terms, the model numbers, part codes and names that
embeddings blur. src/hybrid.ts is reciprocal rank fusion: the vector list and the keyword
list merged by rank, not by score, since the scores are on different scales. src/rerank.ts
has a model read the twenty candidates and score each 0 to 10 for how well it answers the
question, in one call for all of them: the question and the instructions are paid for once,
and the model sees the candidates side by side, which is what ranking needs; a call per
candidate would be twenty prompts and twenty round trips for scores that were never compared.
The reranker is Claude Haiku 5.5, not the answering model (RERANK_MODEL; src/claude.ts now
takes a model option and prices each model it calls, and refuses one it has no price for).
src/widen.ts replaces each passage with itself and its neighbours from the same file, in
order, because a chunk boundary is arbitrary. src/cite-check.ts removes any [n] that points
at a passage the model was not shown and marks the answer, because a citation nobody can
follow looks like evidence. src/pipeline.ts switches each on with an option; the server
keeps the cite check on. evals/suite.ts runs the ten questions with a set of options,
evals/run.ts takes the options as flags, and evals/ladder.ts climbs the ladder, one
change at a time, and prints the table (it was evals/compare.ts at s1e8, and is renamed in
series 2, episode 5, which needs the name). One expectation was widened while building this: the
manual says "hand-tight plus a quarter turn, never more than 2.5 N·m", and an answer that
quotes the first half is right.
npx tsx --env-file=.env evals/run.ts --hybrid --rerank
npx tsx --env-file=.env evals/ladder.ts
The transcript, transcripts/s1e8-making-it-good.txt, is the table, and it is not the table
you would draw from the names of the techniques:
| Step | Passed | Cost of the ten | Failed |
|---|---|---|---|
| vector only | 9/10 | $0.035 | 8 |
| + hybrid | 6/10 | $0.037 | 4, 6, 7, 8 |
| + rerank | 8/10 | $0.045 | 6, 8 |
| + widen | 9/10 | $0.095 | 8 |
| + cite check | 9/10 | $0.093 | 8 |
Read it row by row. Hybrid, on its own, made things worse, and the eval is why we know: the keyword list is full of FAQ entries that say "Firefly" and "rechargeable" often, and after fusion the catalog chunk that holds the 220R's weight fell from fourth to ninth, out of the five the model sees; the chunk with the torque figure went the same way, and the model, shown the wrong half of the page, said it did not know, which is the right failure. Shipped without numbers, that change would have cost two questions. Reranking from the wider pool recovers them, for a fifth more per question than hybrid alone, because the twenty passages are read by Haiku at a twentieth of the answering model's price per token. Question 6, the bench fee, is the noise: the model is nondeterministic, so on ten questions one run is worth plus or minus one. The transcript runs the reranking step twice more, and it passes 9/10 and 8/10; the reranker puts the repair manual's bench-fee page first, and it is the answering model, reading that page, that sometimes says it does not know. Run it three times, or write fifty questions, before you believe a difference of one. Then the transcript switches the reranker to Sonnet for one run: 9/10, the same as Haiku's best, for $0.165 instead of about $0.046, three and a half times the cost of the ten. The model that writes the answer has to be good; the model that sorts passages only has to be good enough, and the eval is how you know it is. Widening costs twice as much for context these answers did not need, which makes it the most expensive change on the ladder, and the cite check found nothing to remove, which is what you want from a guard. Question 8 never passes. The handbook's employee discount section ranks 88th by vector and 38th by keyword for "Can staff buy a Second Light lantern for less?", because the question says staff and the handbook says employee; no reranker can promote a passage that never reached it. The fix for that is to rewrite the question before searching, which is a different change, and the ten questions would measure it the same way.
That is the method: a change is a flag, the flag is measured, the number decides, and the failures that remain tell you what to build next.
Series 1 ended with a pipeline and ten questions. Ten questions measured the four changes of episode 8, and they also showed their limit: one run of the same settings scored 9 and the next 8, so a difference of one question is noise, and a change that helps one kind of question and hurts another cannot be seen at all. This series builds what a team needs before it trusts a retrieval pipeline with customers: a set of cases with known answers, a pipeline that can be tested without a model, a record of every step of every run, a score that says which stage failed, a budget, a comparison between runs, and a gate in CI.
Thirty cases with known answers, each one checked against the corpus.
Files: evals/golden.jsonl is the golden set, one case per line: an id; the question,
phrased the way a customer or a new employee asks it; expect, answer or fallback; the
required facts, each with the match terms that count as stating it ("40 days", "40-day",
"forty days"); the forbidden claims, each with terms that would mean the answer made it (the
Firefly 220's 310 g given for the 220R); and the sources: the file, the page for the PDF, and
a quote copied from the extracted text that proves the answer. 22 cases expect an answer and 8
expect "I don't know": the CEO's salary, last year's unit sales, a Sentinel 2000 that does not
exist, the dental plan's deductible (the plan is named, its deductible is not), the UPS account
number (the shipping policy says it exists and does not give it), the showroom Wi-Fi, the
Firefly's price in euros, the name of the Baltimore brass spinner. A fallback case may carry the
nearest source, to show what the corpus does say. The answer cases cover eleven of the thirteen
files and every trap in the corpus: the 220 against the 220R, the $60 threshold in the release
notes against the $75 in two older pages, four facts found only in the PDF, and series 1's
unsolved staff discount. evals/golden.ts loads the set and holds the two matching helpers
every later script uses: flat (whitespace collapsed) and mentions (any of the terms,
ignoring case).
evals/check-golden.ts checks the set before anything is scored against it: ids unique, every
answer case with facts and sources, every fact and claim with terms, and every quote found in
the text that src/extract.ts produces for its file and page, the same text the index is built
from. The comparison is character for character once runs of whitespace are collapsed, because
the Markdown is wrapped at 100 columns and the PDF's text comes out a line at a time: the line
breaks are layout, the characters are the content. A quote typed from memory is the commonest
error in a golden set, and a case whose quote is not in the corpus can be scored as a retrieval
miss forever while the pipeline is right.
npx tsx evals/check-golden.ts
No key, no model. The transcript, transcripts/s2e1-the-golden-set.txt, shows the thirty cases
passing (22 answer, 8 fallback, 26 quotes in 11 files), then the bench-fee quote changed from
the manual's "$25.00" to "$25", the way a person remembers it, and the check failing on it with
exit code 1, then the quote restored and the check passing. Why thirty: enough that one case is
three points and not ten, few enough that a person can read every answer of a run, and each one
written from the document, not from memory. A golden set grows: episode 7 adds a way to turn
real questions into new cases.
Dependency injection, so the pipeline can be tested without a model.
Until now the pipeline imported its models: pipeline.ts called ask from claude.ts,
retrieve.ts called embed, and the database opened itself when any of them was imported. A
test of the pipeline's rules (cite the passage, say "I don't know" when nothing answers, do not
even ask when nothing retrieved is about the question) would have needed a key, a network, a 97
MB embedding model and money, and it would have answered differently each time. So the models
become a parameter, the way a database connection is one.
Files: src/models.ts is the Models interface, two functions, chat(messages, options) (the
Messages API) and embed(text), with realModels, the real pair, and Deps, everything the
pipeline needs from outside: its models and the chunks it searches. answer(question, options, deps) in src/pipeline.ts imports no model any more; retrieve, rerank and widen take
what they use as arguments; src/server.ts and evals/suite.ts pass realModels and the
stored chunks. src/db.ts opens lantern.db on first use instead of on import.
test/fake-models.ts is the fake pair: an embedder that hashes each word into one of the 384
dimensions, so texts that share words point the same way, and a chat model that answers from
markers in the prompt: a passage containing <<say: ...>> is the answer, cited with that
passage's number, no marker means "I don't know", and asked to score passages it gives 10 to the
one with a marker. Every behaviour a test needs is written into the test's own corpus, so a test
reads as its own explanation. test/pipeline.test.ts has five tests on Node's built-in runner,
node:test and node:assert, run through tsx: the answer cites its passage and the citation
resolves to where it came from; no marker gives "I don't know" and no citations; a question
about nothing in the corpus never reaches the model (the fake counts its calls: zero); the cite
check removes a citation to a passage that was not shown and flags the answer; and the reranker
brings an answer from seventh place into the five shown, where without it the answer is "I don't
know". npm test runs them. .github/workflows/test.yml runs the type check, the tests and the
golden-set check on every push and pull request, with no secret: none of the three calls a
model.
The same pipeline, three ways to hand it its models:
flowchart LR
answer["answer(question, options, deps)<br/>src/pipeline.ts imports no model"] --> models["deps.models, a Models<br/>chat() and embed(), src/models.ts"]
models --> real["realModels<br/>Claude over HTTP, MiniLM on your machine<br/>src/server.ts, evals/suite.ts"]
models --> fake["the fakes<br/>test/fake-models.ts: answers from markers in the prompt<br/>npm test, no key, no network"]
models --> traced["traced(models)<br/>evals/tracer.ts: the real pair, every call recorded<br/>evals/eval.ts"]
traced --> real
npm test
The transcript, transcripts/s2e2-models-as-a-parameter.txt, shows ANTHROPIC_API_KEY unset
and the five tests passing in about a tenth of a second. What these tests prove is the code
around the model: the plumbing, the rules and the guards. They say nothing about whether the
real model answers right; that is what the golden set is for, and the next three episodes build
the harness that runs it. The two are different questions and need both: a fast test that runs
on every change, and an eval that runs when the answers may have moved. The pipeline's behaviour
with the real models did not change: the ten questions of series 1 still score 9/10 for four
cents.
Every step of every case, written down, so a failure can be read instead of guessed at.
A pass or a fail says nothing about why. When the staff-discount question fails, did the search miss the handbook, did the model see it and not use it, or did it answer and the check misread the answer? Each has a different fix, and only a record of what happened tells them apart. The record is the trace: for one case, the query, what retrieval returned and with what scores, what the model was shown, the prompt, the answer, the tokens, the cost and the time.
Files: evals/tracer.ts wraps a Models (the parameter of episode 2 pays off again):
traced(models) returns models that behave exactly like the ones given and a list, calls,
that each embedding and each chat call appends to, with the text in, the reply out, the stop
reason, the tokens, the cost and the milliseconds. The pipeline does not know it is traced.
Retrieval is not a model call, so answer now reports it in its result: retrieved, the vector
search's top 20 in order with chunk id, where and score, and shown, the passages the model
saw, in the order it saw them, so [2] in an answer is shown[1]. The search always fetches 20
for this record; the vector-only pipeline still shows the model the first five, so its answers
are unchanged. src/config.ts holds SERVED, the options the server runs with (the cite
check), so the evals measure what is served unless told otherwise. evals/eval.ts <run> runs
the golden set through the traced pipeline, with the served options or the flags of series 1,
and writes evals/runs/<run>/traces/<case>.json, one file per case, refusing a run name that
exists. evals/trace.ts <run> <case> prints one trace readably: the question, the top eight
retrieved with rank, score and chunk, the passages shown, each call, the prompt's size and its
last line, the answer wrapped, the citations resolved, the total. test/tracer.test.ts checks,
with the fakes, that the tracer records the embedding and the chat call with the prompt and the
reply.
npx tsx --env-file=.env evals/eval.ts baseline
npx tsx evals/trace.ts baseline staff-discount
The first run, evals/runs/baseline/, is committed: thirty traces, $0.1001 for the thirty
questions, between one and two and a half seconds each. The transcript,
transcripts/s2e3-the-tracer.txt, lists the traces and reads one, series 1's failure, and it
answers the question episode 8 could only state: the handbook is not among the top eight
retrieved at all; the five passages shown are the Second Light program in about.md, the Base
Camp Bundle, the FAQ on refurbished lanterns, Lights for Scouts and the company's beliefs; the
model read 1,017 tokens of them, wrote 52 and said "I don't know." for a quarter of a cent. That
is a retrieval miss, read from the record in ten seconds, and the model's answer was the right
one for what it was shown. The rest of the run, read the same way, has its own surprises: the
bench-fee answer quotes the $25.00 fee and then says it does not know what an inspection alone
costs, a hedge a string match would count as a pass. Telling those apart is scoring, the next
episode.
A pass or a fail per case, and for every fail the stage that failed: retrieval, the rules, the judge.
Files: evals/checks.ts is what rules can decide. retrieval finds where the first chunk
holding one of the case's quotes came in the search's top 20, and whether one was shown; a chunk
holds a quote when it is from the same file and page and contains the quote, or either half of
it, because a chunk boundary can cut a quote in two (a half under 30 characters could match by
chance and does not count). checkAnswer finds which required facts the answer states, by their
match terms; which forbidden claims it makes, by theirs; and its citations: how many, any number
that points past the passages shown, and whether a cited passage holds a source.
evals/judge.ts is what rules cannot decide: whether every statement in the answer is backed by
the passages, and whether each [n] points at the passage that says it. A string match can find
"340" in an answer; it cannot tell "the 220R weighs 340 g, 30 g more than the 220's 310 g" from
"the 220R weighs 310 g". The judge is Claude Haiku 5.5, twenty times cheaper than the answering
model and the same model the reranker has used since series 1, episode 8 (JUDGE_MODEL, through
the model option of ask, which prices each model it calls and refuses one it has no price
for, so no call can escape the cost). It reads what the answering model read, its system prompt and its prompt, and the answer,
and returns { reason, grounded, cited } through structured output, reason first so the verdict
follows from it. One call per answered case, about two hundredths of a cent each.
evals/scoring.ts puts them in order. A case that said "I don't know" passes if it is a
fallback case and fails if not; a fallback case that answered fails, however good the answer
sounds, because the corpus does not say. An answer case is checked by the rules and then judged,
and its result names the first thing that failed: a missing fact, a forbidden claim, no
citations, a citation to a passage not shown, ungrounded, miscited. It writes
evals/runs/<run>/results.json; evals/report.ts prints the table; evals/eval.ts now scores
the run it traced, and evals/score.ts <run> scores a run again, after the cases or the checks
change, without running the pipeline. test/checks.test.ts tests the rules on hand-made traces.
The judge is a model, so it is evaluated before it is trusted. evals/judge-cases.jsonl holds
fifteen answers labelled by hand, each a real prompt from the baseline run with either the
pipeline's own answer or a wrong one made from it: a changed number, a citation to the wrong
passage, a claim no passage makes, the 220's weight given for the 220R, the wrong approver for a
$700 refund. evals/judge-eval.ts runs the judge on them and reports its agreement with the
labels. It started as twelve. Scoring the baseline found the judge wrong three ways, and each
became a labelled case: it failed the $65 free-shipping answer because it had not been shown the
system prompt that says to prefer the more recent document (now it is); and it failed the
bench-fee answer for being cautious and the torque answer for a sentence without a citation,
neither of which is what it was asked to judge, so its instructions now say to judge the facts
and not the style, and that a missing citation is not a wrong one. With those changes it agreed
with all fifteen labels on two runs.
npx tsx --env-file=.env evals/judge-eval.ts
npx tsx --env-file=.env evals/eval.ts scored
The transcript, transcripts/s2e4-scoring.txt, shows the judge's evaluation, then a scored run.
The judge agreed on 15 of 15 for grounded and 14 of 15 for cited: on this run it called the
"every 20 fills" answer correctly cited, though it said in the same breath that the manual says
10. The judge is nondeterministic too, which is why it is measured on every change to it and
never trusted on one verdict. The run, evals/runs/scored/, passed 29 of 30 for $0.1057, judge
included: all eight fallback cases said "I don't know", and the one failure is the staff
discount, with no source in the top 20, a retrieval miss. The same scorer gave the baseline run
of episode 3 (evals/score.ts baseline) 27 of 30: besides the staff discount, the two Firefly
runtime and weight answers failed on a forbidden claim, because those answers mentioned the
other model's figure as a contrast ("30 grams more than the 220's 310 g"), and this run's
answers did not. Same code, same cases, two points apart, and whether the case or the answer is
wrong there is episode 6's question. The source column says more than the score. return-window
passed with its source nowhere in the top 20: the FAQ says 40 days too, and the case lists only
the policy. holiday-return passed with its source 16th: the answer came from the FAQ and a
worked example. A passing case with a missing source is a case that lists too few sources, or a
lucky answer; either is worth knowing before it fails.
A budget the harness enforces, a record of every run, and a comparison that says which cases moved.
An eval spends money on every run, and a harness that runs in CI or in a loop can spend a great deal of it without anyone deciding to. So the harness has caps, and a run that reaches one stops: the cases it did not reach are "not run", counted and shown, never passes. And a score on its own says little; what decides a change is the same cases scored before and after, side by side.
Files: evals/ledger.ts holds the caps, RUN_CAP ($1 a run) and TOTAL_CAP ($5 across every
run in the ledger), and the ledger itself, evals/runs/ledger.json: one line per run with when
it started, the commit (marked +changes when src/, docs/ or the cases had uncommitted
edits, as a deliberate regression does), the configuration (the pipeline's options, its numbers
MIN_SCORE, SHOWN and POOL, now exported by src/pipeline.ts as SETTINGS, the model, the
judge and the budget), passed, failed, not run and the cost. evals/eval.ts now scores each
case as soon as it has run it and rewrites the run's ledger line after every case, so a run that
dies still counts what it spent; before each case it checks the run's spend against its budget
(--budget may lower it, never raise it above RUN_CAP) and the ledger's total against
TOTAL_CAP, and once either is reached every remaining case is recorded as not run. The check
is made between cases, so a run can pass its budget by at most one case. The results table and
its summary show the not-run count. evals/compare.ts <runA> <runB> lists every case whose
status changed, with the reason it failed and where its source was retrieved in each run, then
the unchanged count, the passed totals and the cost change. Series 1's ladder, which had that
name, is now evals/ladder.ts. The ledger starts with this episode: the baseline and scored
runs of episodes 3 and 4, $0.21 between them, came before it.
npx tsx --env-file=.env evals/eval.ts five
npx tsx --env-file=.env evals/eval.ts three # with SHOWN = 3 in src/pipeline.ts
npx tsx evals/compare.ts five three
npx tsx --env-file=.env evals/eval.ts capped --budget 0.02
The transcript, transcripts/s2e5-caps-ledger-compare.txt, is a change proposed and refused by
its numbers. The served pipeline scores 26 of 30 for $0.1046 (run five; scored, the same
code, scored 29 an episode earlier: the bench fee said "I don't know" this time, and the two
Firefly answers mentioned the other model's figure again). Then the change: show the model three
passages instead of five, which makes every prompt shorter. It is 31% cheaper, $0.0724, and
scores 25: one point down, inside the noise, and on the score alone it would ship. The
comparison says otherwise. Two cases went from pass to "I don't know", the generator torque and
the Sentinel 1000's runtime on low, and in both the source chunk came 4th in the search: shown
with five passages, not with three. Those are not noise; they are the change, and the trace says
why. A third case went the other way, the Firefly 220's runtime, for no reason the change
explains: its source was 4th as well, not shown, and it answered from the FAQ without the
contrast that trips the case. The change is reverted (git checkout src/pipeline.ts). Last, a
run with a two-cent budget: it ran six cases, spent $0.0224, one case past the budget as the
check allows, and recorded the other 24 as not run, so its 5 passes cannot be read as a score.
The ledger then reads back all three runs, with the setting that differed: $0.1994 of the $5.
Four ways a case fails, told apart from the record, and each fixed where it was actually wrong.
A failing case is a question for the engineer: what broke? There are four answers, and each
points somewhere different. A retrieval miss: no passage holding the answer was shown, so the
model could not have answered; look at the query and the chunks. A wrong fallback: the model
said "I don't know" with the source in front of it, or answered a question the corpus does not
answer; look at the prompt and the passage. An answer miss: the model saw the source, answered,
and got it wrong; look at the answer against the passage, and at the case's own terms. A
citation miss: the facts are right and the citations are missing or point at a passage that does
not say it; look at the [n] against the passages. And the fix is not always in the code: the
code, the corpus, the case or the harness can each be the thing that is wrong, and the trace
says which.
Files: evals/stages.ts is the labelling, in an order that matters: a fallback case that
answered is a wrong fallback; then, for an answer case, a source never shown is a retrieval
miss, whatever the answer; a source shown and "I don't know" is a wrong fallback; a wrong fact,
a forbidden claim or an ungrounded answer is an answer miss; what is left is a citation miss.
evals/diagnose.ts <run> groups a run's failures by stage, with the reason and where the source
ranked, and says where to look first. test/stages.test.ts checks each label. src/rewrite.ts
and the pipeline's rewrite option (--rewrite on the command line) are the fix for the
retrieval miss, below; src/config.ts now serves it.
The four, found in the runs of episode 5 and this one, and fixed:
Retrieval miss, fixed in the code.staff-discount in runfive : source not in the top
20, as in every run since series 1. The question says staff, the handbook says employee, and
an embedding of the question lands among the Second Light passages. The fix is query
rewriting: one call to the model turns the question into a search query in the words company
documents use ("Second Light lantern employee discount staff purchase policy"), and the
rewrite is what is searched. That query alone puts the handbook's employee discount 4th.
Searching both the question and the rewrite and fusing the lists was tried first and lost it
again: the question's list, all Second Light, outvoted the one passage the rewrite found.
Measured (runrewrite ,--cite-check --rewrite , against runcorpus , the same without the
rewrite): the staff discount passes with its source 4th and shown, no other case changed, 30
of 30, for 21% more ($0.1257 against $0.1042, the rewrite call costs about $0.0004). It is now
served. #
Answer miss, fixed in the case.firefly-220r-weight andfirefly-220-low-runtime in runfive : source 4th and shown, the right figure in the answer, and a forbidden claim matched.
The answers were right ("340 g, 30 grams more than the Firefly 220's 310 g"); the case's
forbidden term, "310", was wrong, because it matched the contrast as well as the mistake. The
terms are now the mistake itself ("220R weighs 310", "220 runs 48"). Measured by scoring runfive again under the new cases without running the pipeline (evals/runs/five-recased/ ,evals/score.ts ): both pass, nothing else moved, 26 to 28. #
Wrong fallback, fixed in the corpus.bench-fee in runfive : source 1st and shown, and
the answer was "I don't know. The passages give the paid-repair bench fee of $25.00 plus parts
[1], but none of them mentions a charge just for inspecting". The pipeline keeps only "I don't
know." of such a reply, as it should. The model was right: the manual gives the fee for a
repair, the warranty policy says a customer can decline the quote, and nothing says what
declining costs. That is a gap in the documentation, the question customers will ask, and the
eval found it.docs/warranty.md now says it: "The bench fee of $25.00 pays for the
inspection and the quote, and is charged whether or not the customer goes ahead with the
repair." The policy is the fictional company's to decide; deciding it is the fix. The case
lists the new sentence as a second source. Measured (runcorpus againstfive-recased ):
bench-fee passes, citing the new sentence, nothing else moved, 28 to 29. #
Citation miss, fixed in the harness. No run of the served pipeline produced one; runwiden (--cite-check --rewrite --widen , each passage shown with its neighbours) did:refund-approval failed as miscited. Read from the trace, the answer was right and so were
its citations: "[1][2][4]" follow "Tomas Reyes must approve it", and passage 4, the handbook,
says exactly that. The judge had attached [4] to the next sentence. The judge was wrong, so
the judge is what changed: it now returns a verdict per citation, each with the statement it
is attached to ("the sentence or clause just before it, not a later one"), and the code
decides "cited" from those. The answer joined the judge's labelled cases (now sixteen); the
judge agreed with all sixteen on two runs. Measured by scoringwiden again with the new
judge (widen-rejudged ): refund-approval passes. So didstaff-discount , whichwiden had
failed as an answer miss for saying the employee discount applies "on top of" the Second Light
price; that is a reading of the handbook, not a quote, and the judge called it ungrounded once
and grounded once. A borderline verdict that flips on a rerun is noise in the judge, and the
right response is to read it, not to tune for it.
npx tsx evals/diagnose.ts five
npx tsx --env-file=.env evals/eval.ts rewrite --cite-check --rewrite
npx tsx evals/compare.ts five rewrite
The transcript, transcripts/s2e6-diagnosing-a-failure.txt, shows run five diagnosed (two
answer misses, a wrong fallback, a retrieval miss), run widen diagnosed (an answer miss and
the citation miss), then the comparisons: five to rewrite, all three fixes together, 26 to
30 of 30, for 20% more; and widen to widen-rejudged, 28 to 30. Two lessons from the four.
Half the fixes were not in the pipeline: a case that was wrong and a document that was missing a
sentence are as common as a bad prompt, and a harness that is only ever used to change the code
will make the code worse to satisfy a bad case. And widening, measured again here, costs twice
as much ($0.2547) for nothing these thirty cases need.
The eval stops a bad change before it merges, learns from real questions, and runs every month whether anyone changed anything or not.
Files: evals/run-eval.ts is the run of episodes 3 to 6 as a function, runEval(run, options, budget), so two scripts can share it: evals/eval.ts, as before, and evals/gate.ts, the
gate. The gate runs the served pipeline on the golden set with a $0.50 budget and fails, with
exit code 1, when less than THRESHOLD (85%) of the cases pass, when any fallback case
answered, or when the budget stopped the run. The threshold sits under the noise: the served
pipeline has scored between 26 and 30 of 30 across runs, so a change that trips it has done real
harm, and a fallback case that answers fails the gate on its own because an invented answer is
the failure that costs most. .github/workflows/eval.yml runs the gate on every pull request
that touches src/, docs/, evals/ or the dependencies, on the 1st of every month, and by
hand: it installs, restores the embedding model from the cache, checks the golden set, builds
the index, runs the gate with the key from the repository secret ANTHROPIC_API_KEY, and keeps
the run's traces and results as an artifact, so a red check can be read with evals/trace.ts. A
newer push cancels the running check, so one pull request never pays twice; the ledger's caps
apply in CI as well, against the ledger committed in the branch. The
secret is set in the repository's settings (Settings, Secrets and variables, Actions), and making the
check required is a branch protection rule. A fork needs its own key there, or the gate fails for want of
one; the test workflow needs no secret at all.
src/log.ts appends every question the server answers to logs/questions.jsonl (gitignored),
with the answer and what it cited; questions can carry personal details, so that file belongs
where your other production logs live, under the same retention. evals/sample.ts [n] turns the
log into draft cases: it drops questions the golden set already has and repeats, takes a random
sample of n, and writes evals/drafts.jsonl (gitignored too) with what the pipeline said. A
draft is not a case: a person reads it against the documents, writes the facts, forbidden claims
and quotes or marks it a fallback, moves it into golden.jsonl, and runs check-golden.ts. The
first draft built that way is in the set: asked through the server while this episode was built,
"Does the Northstar come with a carry case?" got "I don't know.", and the catalog's Northstar
table says "In the box | Lantern, two mantles, funnel, carry case". It is case 31,
northstar-carry-case, and it fails: the next thing to diagnose. Its first match terms included
"comes with a carry case", which a later answer matched by saying the passages do not say
whether it "comes with a carry case"; they are now "yes" and phrases that only an affirmative
answer contains, the lesson of episode 6 again.
npx tsx --env-file=.env src/server.ts &
curl -s localhost:3000/ask -d '{"question": "Does the Northstar come with a carry case?"}'
npx tsx evals/sample.ts 5
npx tsx --env-file=.env evals/gate.ts
The transcript, transcripts/s2e7-the-harness-as-a-gate.txt, shows two questions asked through
the server, both answered "I don't know.", and the sample drafting the one the golden set does
not have yet (the dog in Annapolis; the carry case is already case 31). Then the bad change: one
line deleted from the system prompt, the rule "When the passages do not answer the question,
reply exactly: I don't know.", the kind of cleanup that looks harmless in review. The gate fails
it: every one of the eight fallback cases now answered (with the CEO's salary, the dental
deductible and the brass spinner's name, the model now explains what the passages do not say
instead of saying so in four words), two answer cases failed too, 21 of 31 against the 27
needed, exit code 1, for $0.1373. The line is restored and the gate passes: 30 of 31, every
fallback right, the carry case the one failure, exit code 0, for $0.1323.
The model behind the API is not part of this repository and it changes: a model version is
retired on a published date, and its replacement answers differently. The pipeline's code can be
identical and its answers worse. The scheduled run of eval.yml catches the bad case: a gate
that fails on the 1st of the month with no pull request behind it is drift. Every month, and
before any model change:
- Rerun:
npx tsx --env-file=.env evals/eval.ts monthly-2026-12. - Compare with the last run in the ledger at the same settings (
jq -r '.runs[] | "\(.run) \(.at) \(.passed)"' evals/runs/ledger.json):npx tsx evals/compare.ts rewrite monthly-2026-12. Read every case that moved withevals/trace.ts, and label the failures withevals/diagnose.ts. - To move to a new model, or off a deprecated one: change
MODELinsrc/claude.tsand its line inPRICES(andmodelinask.sh), rerun under a new name, compare. The judge is a model too: whenJUDGE_MODELchanges, runevals/judge-eval.tsfirst and trust the new judge only at the same agreement. So is the reranker:RERANK_MODELinsrc/rerank.tschanges the same way, measured with the reranker switched on (evals/ladder.ts). - Ship the swap when the comparison shows nothing lost that matters, and the ledger keeps the run that proved it.
Without a key, at no cost:
npm run typecheck # tsc over src/, evals/ and test/
npm test # the pipeline's rules, the tracer, the checks and the stage labels, on fakes
npx tsx evals/check-golden.ts # every quote in the golden set is in the corpus
npx tsx src/index.ts # build lantern.db from docs/ (downloads the 97 MB embedding model once)
npx tsx src/search.ts "what is the bench fee for an out-of-warranty repair"
npx tsx evals/trace.ts baseline staff-discount # read one case of a committed run
npx tsx evals/compare.ts five three # two committed runs, case by case
npx tsx evals/diagnose.ts five # a committed run's failures, by the stage that failed
With a key in .env, for cents:
npx tsx --env-file=.env src/ask.ts "In one sentence: what does HTTP status 304 mean?"
npx tsx --env-file=.env src/server.ts # then: curl -s localhost:3000/ask -d '{"question": "..."}'
npx tsx --env-file=.env evals/run.ts --hybrid --rerank # series 1's ten questions, about five cents
npx tsx --env-file=.env evals/ladder.ts # the five rows of episode 8's table, about thirty cents
npx tsx --env-file=.env evals/eval.ts my-first-run # the golden set, traced and scored, about thirteen cents
npx tsx evals/compare.ts rewrite my-first-run # against the committed run of the same settings
npx tsx --env-file=.env evals/judge-eval.ts # the judge against its labelled answers, under half a cent
npx tsx --env-file=.env evals/gate.ts # what CI runs: exit 0 or 1, about thirteen cents
A run's name must be new: evals/eval.ts refuses one that exists, so the committed runs stay as the
episodes read them. Your runs land in evals/runs/ beside them, each with its line in the ledger.
.gitignore keeps out what each machine makes for itself:
.env: your key..env.exampleis the template.lantern.db: the index.npx tsx src/index.tsbuilds it fromdocs/in a few seconds. (Ats1e5,src/search.tsfills its own store on first run; froms1e6on, a search against an empty store says to run the index.)node_modules/, and inside it the embedding model: 97 MB, downloaded by the first search or index intonode_modules/@huggingface/transformers/.cache, and never again.logs/questions.jsonl: every question the server answered, froms2e7on (src/log.ts).evals/drafts.jsonl: whatevals/sample.tsdrafts from that log.
Committed on purpose: evals/runs/, the runs the episodes read their numbers from, every trace included,
and evals/runs/ledger.json; transcripts/, the real output of each episode's commands; and
docs/repair-manual.pdf, the one PDF in the corpus.
- "ANTHROPIC_API_KEY is not set" : the command is missing
--env-file=.env, or.envis not there. Node reads the file itself from version 20.6 on (node --version). better-sqlite3will not load ("was compiled against a different Node.js version"):npm rebuild better-sqlite3. Its prebuilt binary is for one major version of Node at a time.- "Cannot find package" after checking out a tag:
npm installagain. The dependencies change ats1e5,s1e6ands2e2. - The first search or index sits for a while : the embedding model is down, once. After that, the only thing that needs the network is Claude.
- "the store is empty" : run
npx tsx src/index.tsfirst. - "exists; name the run something new" :
evals/eval.tsnever overwrites a run. - Cases "not run" : the run reached its budget (
--budget, never aboveRUN_CAP) or the ledger reachedTOTAL_CAP(evals/ledger.ts). Read the ledger, then raise the cap in the code on purpose. - Port 3000 is taken :
PORTinsrc/server.ts. - Your numbers differ from this README's by a case or two : the models are nondeterministic, and every
number here is a committed run in
evals/runs/or a transcript intranscripts/. A difference of one case in thirty is noise; series 2, episode 5 is about telling noise from a change. npm run typecheckats1e1says no inputs were found : there is no TypeScript yet at that tag, onlyask.sh. It works froms1e2on../ask.shprints nothing :jqis missing (brew install jq), or the key in.envis not set.
One commit and one tag per episode, in the order the episodes play; git log --oneline is the series.
| Tag | Episode | Video |
|---|---|---|
s1e1 |
Series 1, episode 1: What actually changed, and what didn't | https://youtu.be/55s2Yw-UBgc |
s1e2 |
Series 1, episode 2: Calling a model from code | https://youtu.be/WnyUC2Qqw-4 |
s1e3 |
Series 1, episode 3: Prompts are code | https://youtu.be/Hpfw67Trg7k |
s1e4 |
Series 1, episode 4: Why it can't read your documents | https://youtu.be/b8ZsS2FKypw |
s1e5 |
Series 1, episode 5: Embeddings, for people who know SQL | https://youtu.be/fPsBmU_FSaw |
s1e6 |
Series 1, episode 6: Chunking and indexing is ETL | https://youtu.be/_LzjKfoXAvA |
s1e7 |
Series 1, episode 7: Your first pipeline | https://youtu.be/GZPzeVkgWNs |
s1e8 |
Series 1, episode 8: Making it good, with numbers | https://youtu.be/OfuJPNoKeSU |
s2e1 |
Series 2, episode 1: The golden set | https://youtu.be/FkJjuXGa9CE |
s2e2 |
Series 2, episode 2: Models as a parameter | https://youtu.be/50XGeIVL_eA |
s2e3 |
Series 2, episode 3: The tracer | https://youtu.be/zEEFXUjOCjA |
s2e4 |
Series 2, episode 4: Scoring | https://youtu.be/cEGDHGmIix4 |
s2e5 |
Series 2, episode 5: Caps, the ledger and comparing runs | https://youtu.be/nQiUFUrc0eA |
s2e6 |
Series 2, episode 6: Diagnosing a failure | https://youtu.be/s6Zglb47T78 |
s2e7 |
Series 2, episode 7: The harness as a gate | https://youtu.be/w6aXAi353GE |
Every dependency, with its one-line reason.
@huggingface/transformers: runs the embedding model on your machine (ONNX Runtime underneath); no embedding API, no key.better-sqlite3: SQLite in-process and synchronous, the simplest database there is; npm installs a prebuilt binary.unpdf: text out of a PDF, page by page; plain JavaScript on top of pdf.js, no native build.
Development dependencies:
typescript: the type checker (npm run typecheck).tsx: runs a TypeScript file directly (npx tsx file.ts), no build step.@types/node: the types of Node's own modules (fs,http,process).@types/better-sqlite3: the types ofbetter-sqlite3.
transcripts/ holds the real output of each episode's commands, recorded when the episode was
built: the command line first, then what it printed, long lines wrapped to fit a slide. Nothing
in them is edited or invented. Each file is named for its tag.
MIT, in LICENSE: use the code, the corpus and the golden set as you like, with the notice kept.