I'm building a tool that reads every new release of React, TypeScript, Vite and friends, and tells you two things: what changed, and whether you need to touch your code before upgrading.
The summarising part is easy. Any decent model will write you a tidy paragraph about a release.
The part that decides whether the tool is worth anything is different: when it says "action required", is that true? A tracker that invents a breaking change is worse than no tracker, because you'll go looking for a migration that doesn't exist. And a tracker that misses one is worse still.
Here is everything I do about that, in the order I had to learn it.
The system prompt says it in the second line:
Use only the release notes in the user message. Do not use what you know about the project from elsewhere.
This matters more than it sounds. A model asked about "React 19.2.0" has opinions about React 19 from training. Those opinions are often right, which is exactly what makes them dangerous — they're right until the release where they aren't, and you have no way to tell which one you're looking at.
So the notes go into the user message, wrapped in a tag, and nothing else about the project does.
Two consequences follow immediately:
if (notes.trim() === "") return { ok: false, reason: "release notes are empty" };
if (notes.length > MAX_NOTES_LENGTH) {
return { ok: false, reason: `release notes are ${notes.length} characters, over the ${MAX_NOTES_LENGTH} limit` };
}
Both are failures I'd rather see than paper over.
The prompt doesn't ask the model whether you should act. It asks for a list:
"breakingChanges": every change the notes state or clearly describe as breaking, removed or incompatible, so that users must change code, configuration or runtime to upgrade.
"actionRequired": true exactly when "breakingChanges" is not empty.
And then, flatly:
Never infer a breaking change the notes do not mention, even if the version number suggests one.
That last line earns its place. Without it, a x.0.0 invites the model to reason "this is a major, majors break things" and produce something. Version numbers are a classification input for me, never evidence about content.
The output shape is constrained with a schema, so a reply that isn't that shape never reaches the database. But keep reading — schemas do less than you think.
Each key point and each breaking change links back into the official notes, to the line it came from. On the page it looks like release-notes:20, and clicking it opens the notes, scrolls to line 20 and puts a caret at the right column.
The obvious implementation is to ask the model for the line number. Don't. Models count badly, and — worse — they count badly invisibly: you get 42, it looks like a line number, and it's off by nine.
So the model returns a verbatim quote instead:
A "quote" is the stretch of the release notes the item comes from, copied character for character [...] Never adjust, shorten with an ellipsis or rewrite a quote to fit: a quote that does not appear verbatim in the notes is dropped.
And I compute the position myself, by searching my stored copy of the notes for that string:
locateCitation(notes, quote) // → { fromLine, toLine, column, length } | null
null means the quote isn't in the notes. Which means the model made it up. Which means no pointer is rendered.
This is the part I'm most pleased with, because of what it collapses. "The model must not hallucinate citations" and "references must stay inside the document" sound like two separate things to test. They're the same thing, and it's a string search. The column offset for the caret falls out for free.
Everything above is reasoning about a prompt. Reasoning about a prompt is how you end up with a prompt that feels great and is wrong.
So: an evaluation set. 14 real releases, from 8 projects — 5 of them not even in my catalogue, so I couldn't accidentally tune the product to the test. The notes are copied into the repo, not read from the database. The catalogue changes; the set must not, or two reports a month apart mean nothing.
The human judgement lives in a set.json next to them, and it's more nuanced than a pass/fail:
actionRequired, which can be true, false, or "either"
The comparison itself is a pure function with its own unit tests. The script around it only does I/O. Two prompt versions, one batch each:
| v3 | v4 | |
|---|---|---|
| cases evaluated | 13/14 | 14/14 |
| "action required" correct | 13/13 | 14/14 |
| missing breaking changes | 0 | 0 |
| invented citations | 1 | 0 |
| verbatim quotes | 66 of 72 | 77 of 81 |
Six of the fourteen are deliberate duds: patch releases whose notes say nothing but "see the changelog". All six came back "no action needed". That was the point of including them — I wanted evidence that the model shuts up when there's nothing to say, not just that it's eloquent when there is.
Two full runs cost about 50 cents through the Batch API. Roughly $0.02 a summary.
Both of these survived a fake adapter, unit tests and integration tests. Both died within minutes of the first real batch.
Line endings. Release notes fetched from GitHub contain CRLF. The model quotes them back with LF. The page searches the normalised text. So any quote spanning more than one line silently failed to locate — no crash, no log, just a pointer that quietly wasn't there. In production. locateCitation now normalises both sides.
A sixth key point. The prompt asks for 3 to 5 key points, and the schema says so too. A reply came back with six, failed validation, and threw away a summary I had already paid for.
The lesson is worth stating plainly: structured output constrains the shape of a reply, never its counts. You can guarantee "an array of objects with these fields". You cannot guarantee "at most five". So now the parser keeps the first five — they arrive in order of importance — and the prompt states the limit as a hard rule with the consequence spelled out. That change is what took the prompt to v4, and it's why the "cases evaluated" row went from 13 to 14.
A smaller thing, but it's the same instinct. When the Anthropic account runs out of credit, the affected releases go back to pending, not to failed. failed is final — those summaries would never have regenerated, even after topping up.
And the case is typed, not sniffed out of an error message: submit throws a CreditExhaustedError, collect returns credit_exhausted. "Other errors must not be mistaken for exhausted credit" is then something the compiler enforces, rather than something I promise.
If I did this again from scratch, three things carry over:
The tracker is live and collecting releases while I build the rest of it. The summaries aren't switched on in production yet — that's the next thing, and it's the first part of this project that costs money to leave running.