Claude Code gave a consulting team a contract total off by about $800,000 and a deadline a month late. Both looked complete. Here is a script that counts those traps in 510 real contracts, plus checks for your pipeline.
On November 8 the AI Engineer channel posted “Claude Code Looked Right but Missed $800K”, by Natalia Connolly and Kevin Madura of AlixPartners.
A healthcare company buys a smaller one: 40,000+ contract documents. Can we get better terms from a shared vendor, and how soon must we act? Using a synthetic replica of the deal, the speakers show Claude Code’s plausible answers, then check each against the data warehouse. Spend and contract counts were right. Two things were wrong.
The total contract value was off by about $800,000. The agent couldn’t find an amendment never linked to its parent contract, an “orphan”.
The act-by date was a month late. The agent said December 1; the real date was November 1, after which the contract renews by itself. It took the date from the termination clause instead of working it out from the renewal clause.
More traps in messy contract data:
Also: an access-denied error can reach the agent looking like a zero invoice.
Structure the data first. Five stages: OCR; classifying each document (agreement, amendment, NDA); extracting terms, rates and spend; mapping to spend categories; only then Claude Code, using custom tools such as “get rate card page” with business rules built in. Two weeks to build; then 40,000+ contracts ran overnight, and every answer could be checked in code.
My script shows how common three of the traps are.
CUAD (the Contract Understanding Atticus Dataset) holds 510 contracts from public SEC filings, with 41 expert-labeled fields each, including “Renewal Term” and “Notice Period To Terminate Renewal”. A contract that renews by itself unless you give notice has an act-by date before its end date.
git clone --depth 1 https://github.com/TheAtticusProject/cuad.gitcd cuad && unzip -q data.zip # gives CUADv1.json (~40 MB)
Standard library only; no model is called. The notice parser wants a period (“ninety (90) days”) next to a cue like “at least” or “prior”; months count as 30 days. The setting to change is RENEWS_ON.
Save the three blocks below, in order, as contractcheck.py.
#!/usr/bin/env python3"""contractcheck.py - four checks from the AlixPartners talk, run on CUAD.usage: python3 contractcheck.py path/to/CUADv1.json"""import json, re, statistics, sysfrom datetime import date, timedeltaRENEWS_ON = date(2027, 1, 1) # the setting to change: an end date to testWORDS = {"one hundred eighty": 180, "one hundred twenty": 120, "one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10, "twelve": 12, "fifteen": 15, "twenty": 20, "thirty": 30, "forty-five": 45, "sixty": 60, "ninety": 90}UNIT_DAYS = {"day": 1, "week": 7, "month": 30, "year": 365}NUM = r"(\d+|" + "|".join(sorted(WORDS, key=len, reverse=True)) + r")"PERIOD = re.compile(r"\b" + NUM + r"\s*(?:\(\d+\)\s*)?-?\s*" r"(?:calendar\s+|business\s+)?" r"(day|week|month|year)s?", re.I)NOTICE_CUE = re.compile( r"^\W{0,3}(prior|notice|before|in advance|advance|written)", re.I)LEAD_CUE = re.compile(r"(at least|not less than|no less than|minimum of)\s*$", re.I)def answers(doc, label): for q in doc["paragraphs"][0]["qas"]: if q["id"].endswith("__" + label): return [a["text"] for a in q["answers"]] return []def notice_match(spans): """Return (days, matched phrase) or (None, None).""" text = " ".join(spans) for m in PERIOD.finditer(text): before = text[max(0, m.start() - 25):m.start()] after = text[m.end():m.end() + 20] if LEAD_CUE.search(before) or NOTICE_CUE.search(after): n = m.group(1).lower() n = int(n) if n.isdigit() else WORDS[n] return n * UNIT_DAYS[m.group(2).lower()], m.group(0) return None, None
The deadline check:
def deadline(docs, n): """01: auto-renewing + a notice period to opt out.""" anyrenew = [d for d in docs if answers(d, "Renewal Term")] renew = [d for d in anyrenew if re.search( r"automatic", " ".join(answers(d, "Renewal Term")), re.I)] trap = [d for d in renew if answers(d, "Notice Period To Terminate Renewal")] days = [(d, notice_match(answers(d, "Notice Period To Terminate Renewal"))[0]) for d in trap] read = sorted(x for _, x in days if x) print("\n01 DEADLINE TRAP (act-by date is earlier than the end date)") print(f" any renewal or extension clause: {len(anyrenew)} of {n}") print(f" ...that says 'automatic': {len(renew)} of {n}" f" ({len(renew) / n:.0%})") print(f" ...plus a notice period to stop: {len(trap)} of {n}" f" ({len(trap) / n:.0%})") print(f" notice period parsed to days: {len(read)} of {len(trap)}") if read: print(f" median notice: {statistics.median(read):.0f} days, " f"range {read[0]}-{read[-1]} days") print(" (basis: regex on the label text; 1 month = 30 days)") bands = [(1, 30), (31, 90), (91, 180), (181, 400)] print(" " + " ".join(f"{a}-{b}d: {sum(a <= x <= b for x in read)}" for a, b in bands)) med = int(statistics.median(read)) print(f" ends {RENEWS_ON}, median {med}-day notice -> act by " f"{RENEWS_ON - timedelta(days=med)}, not {RENEWS_ON}") redacted = sum(1 for d, x in days if x is None and "*" in " ".join(answers(d, "Notice Period To Terminate Renewal"))) print(f" unreadable: {len(trap) - len(read)} ({redacted} redacted with ***)") d0 = trap[0] x0, phrase = notice_match(answers(d0, "Notice Period To Terminate Renewal")) print(f" raw: {d0['title'][:40]} -> {x0} days from {phrase!r}")
The other three checks:
def amendments(docs): """02: amendments whose parent is not in the set.""" amend = [d for d in docs if re.search(r"amend", d["title"], re.I)] names = [d["title"] for d in docs] print("\n02 AMENDMENT TRAP (amendment with no parent document)") orphans = 0 for d in amend: filer = re.split(r"[_,(]| - ", d["title"])[0].strip().upper() parent = [t for t in names if t.upper().startswith(filer) and not re.search(r"amend", t, re.I)] orphans += not parent parent.sort(key=len) tag = "ORPHAN" if not parent else "parent: " + parent[0][-24:] print(f" {d['title'][:44]:44} {tag}") print(f" amendments: {len(amend)}, orphans: {orphans} (basis: title match)")def restated(docs, n): """03: documents that replace an earlier version.""" def says(d, k): return re.search(r"amended\s+and\s+restated", d["paragraphs"][0]["context"][:k], re.I) own = [d for d in docs if says(d, 300)] near = [d for d in docs if says(d, 3000)] print("\n03 RESTATED TRAP (doc replaces an earlier version)") print(f" 'amended and restated' in first 300 chars: {len(own)} of {n}") print(f" 'amended and restated' in first 3,000 chars: {len(near)} of {n}") for d in own: print(f" {d['title'][:70]}")def coverage(docs): """04: the silent-drop check you run before trusting any total.""" sizes = sorted(len(d["paragraphs"][0]["context"]) for d in docs) empty = sum(1 for s in sizes if s == 0) print("\n04 COVERAGE (every input must come out the other end)") print(f" empty texts: {empty}; longest: {sizes[-1]:,} chars;" f" over 100k chars: {sum(s > 100_000 for s in sizes)}")def main(path): try: docs = json.load(open(path))["data"] except (OSError, ValueError, KeyError) as e: sys.exit(f"usage: contractcheck.py path/to/CUADv1.json ({e})") n = len(docs) print(f"contracts loaded: {n} (basis: CUAD v1 expert labels)") deadline(docs, n) amendments(docs) restated(docs, n) coverage(docs)if __name__ == "__main__": if len(sys.argv) != 2: sys.exit("usage: python3 contractcheck.py path/to/CUADv1.json") main(sys.argv[1])
python3 contractcheck.py CUADv1.json
contracts loaded: 510 (basis: CUAD v1 expert labels)01 DEADLINE TRAP (act-by date is earlier than the end date) any renewal or extension clause: 176 of 510 ...that says 'automatic': 115 of 510 (23%) ...plus a notice period to stop: 91 of 510 (18%) notice period parsed to days: 82 of 91 median notice: 90 days, range 15-365 days (basis: regex on the label text; 1 month = 30 days) 1-30d: 21 31-90d: 37 91-180d: 22 181-400d: 2 ends 2027-01-01, median 90-day notice -> act by 2026-10-03, not 2027-01-01 unreadable: 9 (5 redacted with ***) raw: CENTRACKINTERNATIONALINC_10_29_1999-EX-1 -> 15 days from 'fifteen (15) days'
Of 176 renewal or extension clauses, 115 say “automatic”. 91 of them, 18% of all 510, also carry a notice period. Most read “renews unless you give notice X before the end”; a few are at-will termination notices.
The median notice is 90 days: a January 1 end date means acting by October 3. The talk’s miss was 30 days; 21 of the 82 readable periods are 30 days or less. Five unreadable ones are redacted in the filing. Without the cue, the parser would grab a term length.
02 AMENDMENT TRAP (amendment with no parent document) FEDERATEDGOVERNMENTINCOMESECURITIESINC_04_28 parent: AGREE-SERVICES AGREEMENT NETGEAR,INC_04_21_2003-EX-10.16-AMENDMENT TO parent: 16-DISTRIBUTOR AGREEMENT NEONSYSTEMSINC_03_01_1999-EX-10.5-DISTRIBUTO parent: ISTRIBUTOR AGREEMENT_New Columbia Laboratories, (Bermuda) Ltd. - AMEN ORPHAN BIOAMBERINC_04_10_2013-EX-10.34-DEVELOPMENT parent: EVELOPMENT AGREEMENT (1) NETGEAR,INC_04_21_2003-EX-10.16- AMENDMENT # parent: 16-DISTRIBUTOR AGREEMENT FEDERATEDGOVERNMENTINCOMESECURITIESINC_04_28 parent: AGREE-SERVICES AGREEMENT Cerus Corporation - FIRST AMEND TO SUPPLY AN ORPHAN ADMA BioManufacturing, LLC - Amendment #3 t ORPHAN amendments: 9, orphans: 3 (basis: title match)
CUAD is a sample, like a filtered export: three of nine amendments (Columbia Laboratories, Cerus, ADMA) have no parent in the set, and those filers appear nowhere else in it. An agent asked for their terms sees only the amendment. Titles undercount, too: more amendments hide under plain agreement names.
03 RESTATED TRAP (doc replaces an earlier version) 'amended and restated' in first 300 chars: 4 of 510 'amended and restated' in first 3,000 chars: 15 of 510 PACIRA PHARMACEUTICALS, INC. - A_R STRATEGIC LICENSING, DISTRIBUTION A VARIABLESEPARATEACCOUNT_04_30_2014-EX-13.C-UNCONDITIONAL CAPITAL MAINT Reinsurance Group of America, Incorporated - A_R REMARKETING AGREEMEN Monsanto Company - SECOND A_R EXCLUSIVE AGENCY AND MARKETING AGREEMENT
4 contracts open by calling themselves “amended and restated”, replacing an earlier version. 11 more use the phrase later, mostly citing other agreements, so a keyword hit isn’t proof. If both versions are in your folder, a sum counts the deal twice.
04 COVERAGE (every input must come out the other end) empty texts: 0; longest: 338,211 chars; over 100k chars: 81
81 contracts exceed 100,000 characters. If your preprocessing has a timeout, these go first. The fix is counting: documents in must equal rows out. Save this beside contractcheck.py as mine.py and run it:
dropped: ['c2']zero spend: ['c3']notice: (60, 'sixty (60) days')
Swap in your IDs and clauses. An unexplained zero is probably an error.
In 510 real contracts, roughly 1 in 6 auto-renews with a deadline before its end date, so the setup behind the AlixPartners date miss is common. Claude Code got messy data and a hidden business rule. Before trusting an agent’s contract total, check in code: counts match, amendments link, restatements replace, and act-by equals end date minus notice.
Which deadline in your contracts would you least like to learn about a month late?
Reproduce it: contractcheck.py and mine.py print every output block above. The talk's figures ($800K, $17M, Nov 1) come from the video.
If this saves you a bill, a clap helps others find it.
Follow me for daily LLM deep-dives.
Claude Code Missed $800K in Contracts. Here’s How to Check Yours was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.