I maintain a small public API and my release notes used to be whatever I remembered an hour after deploying. So I wrote a deterministic OpenAPI differ: feed it a base spec and the new spec, get back a structured list of changes, no LLM in the loop to embellish anything.
The first real run compared two builds three days apart and reported over 200 modifications. My integration tests all passed. Either the tests were useless or the differ was lying.
It was the differ. I had parsed both specs into JSON and walked them recursively, comparing arrays by index. OpenAPI's paths object is nominally a map, but I was effectively diffing serialized key order — and when a developer inserted one new endpoint in the middle, every path after it lined up against the wrong neighbor. Same failure mode for the operations under each path and the parameters list. One tiny real change cascaded into a wall of false positives.
The fix was switching every comparison to identity keys: paths keyed by the path string, operations by HTTP method, parameters by (in, name), schema properties by name. Diff maps and sets, never arrays, even when the format looks ordered. After that, the same two builds produced a diff of exactly three real lines: one new endpoint, one optional request field, and a description tweak.
The second lesson came when I started classifying changes as breaking. It is not symmetric. Removing a request property or making one required breaks existing callers; the same edits on the response side mostly hurt nobody. But removing a response field breaks every client parsing it. Enum narrowing breaks request senders. I had to encode direction — request vs response — into the rules, not just slap a severity flag on each change.
I ended up packaging it as my OpenAPI changelog API: give it a base spec URL and the new spec, and it returns a machine-readable change list plus human release notes, deterministic end to end. Shipping it taught me more about the OpenAPI object model than reading the spec ever did — almost nothing in that document is semantically ordered, even when it looks like it is.