Claude Code Missed $800K in Contracts. Here’s How to Check Yours​

AlixPartners' Natalia Connolly and Kevin Madura reported on November 8 that Anthropic's Claude Code, run against a synthetic replica of a healthcare acquisition with 40,000+ contract documents, returned a total contract value off by about $800,000 and an act-by date a month late (December 1 instead of November 1). The $800,000 error came from an amendment never linked to its parent contract, and the date error came from reading the termination clause instead of deriving the date from the renewal clause. The speakers' five-stage pipeline — OCR, document classification, term extraction, spend-category mapping, then Claude Code with custom business-rule tools — was built in two weeks and ran the 40,000+ contracts overnight, and a companion script applies three of the traps to the 510 contracts in the CUAD dataset.

Claude Code gave a consulting team a contract total off by about $800,000 and a deadline a month late. Both looked complete. Here is a script that counts those traps in 510 real contracts, plus checks for your pipeline. On November 8 the AI Engineer channel posted “Claude Code Looked Right but Missed $800K” https://www.youtube.com/watch?v=u7fsyrXsPBE , by Natalia Connolly and Kevin Madura of AlixPartners. A healthcare company buys a smaller one: 40,000+ contract documents. Can we get better terms from a shared vendor, and how soon must we act? Using a synthetic replica of the deal, the speakers show Claude Code’s plausible answers, then check each against the data warehouse. Spend and contract counts were right. Two things were wrong. The total contract value was off by about $800,000. The agent couldn’t find an amendment never linked to its parent contract, an “orphan”. The act-by date was a month late. The agent said December 1; the real date was November 1, after which the contract renews by itself. It took the date from the termination clause instead of working it out from the renewal clause. More traps in messy contract data: Also: an access-denied error can reach the agent looking like a zero invoice. Structure the data first. Five stages: OCR; classifying each document agreement, amendment, NDA ; extracting terms, rates and spend; mapping to spend categories; only then Claude Code, using custom tools such as “get rate card page” with business rules built in. Two weeks to build; then 40,000+ contracts ran overnight, and every answer could be checked in code. My script shows how common three of the traps are. CUAD the Contract Understanding Atticus Dataset holds 510 contracts from public SEC filings, with 41 expert-labeled fields each, including “Renewal Term” and “Notice Period To Terminate Renewal”. A contract that renews by itself unless you give notice has an act-by date before its end date. git clone --depth 1 https://github.com/TheAtticusProject/cuad.gitcd cuad && unzip -q data.zip gives CUADv1.json ~40 MB Standard library only; no model is called. The notice parser wants a period “ninety 90 days” next to a cue like “at least” or “prior”; months count as 30 days. The setting to change is RENEWS ON. Save the three blocks below, in order, as contractcheck.py. bash /usr/bin/env python3"""contractcheck.py - four checks from the AlixPartners talk, run on CUAD.usage: python3 contractcheck.py path/to/CUADv1.json"""import json, re, statistics, sysfrom datetime import date, timedeltaRENEWS ON = date 2027, 1, 1 the setting to change: an end date to testWORDS = {"one hundred eighty": 180, "one hundred twenty": 120, "one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10, "twelve": 12, "fifteen": 15, "twenty": 20, "thirty": 30, "forty-five": 45, "sixty": 60, "ninety": 90}UNIT DAYS = {"day": 1, "week": 7, "month": 30, "year": 365}NUM = r" \d+|" + "|".join sorted WORDS, key=len, reverse=True + r" "PERIOD = re.compile r"\b" + NUM + r"\s ?:\ \d+\ \s ?-?\s " r" ?:calendar\s+|business\s+ ?" r" day|week|month|year s?", re.I NOTICE CUE = re.compile r"^\W{0,3} prior|notice|before|in advance|advance|written ", re.I LEAD CUE = re.compile r" at least|not less than|no less than|minimum of \s $", re.I def answers doc, label : for q in doc "paragraphs" 0 "qas" : if q "id" .endswith " " + label : return a "text" for a in q "answers" return def notice match spans : """Return days, matched phrase or None, None .""" text = " ".join spans for m in PERIOD.finditer text : before = text max 0, m.start - 25 :m.start after = text m.end :m.end + 20 if LEAD CUE.search before or NOTICE CUE.search after : n = m.group 1 .lower n = int n if n.isdigit else WORDS n return n UNIT DAYS m.group 2 .lower , m.group 0 return None, None The deadline check: python def deadline docs, n : """01: auto-renewing + a notice period to opt out.""" anyrenew = d for d in docs if answers d, "Renewal Term" renew = d for d in anyrenew if re.search r"automatic", " ".join answers d, "Renewal Term" , re.I trap = d for d in renew if answers d, "Notice Period To Terminate Renewal" days = d, notice match answers d, "Notice Period To Terminate Renewal" 0 for d in trap read = sorted x for , x in days if x print "\n01 DEADLINE TRAP act-by date is earlier than the end date " print f" any renewal or extension clause: {len anyrenew } of {n}" print f" ...that says 'automatic': {len renew } of {n}" f" {len renew / n:.0%} " print f" ...plus a notice period to stop: {len trap } of {n}" f" {len trap / n:.0%} " print f" notice period parsed to days: {len read } of {len trap }" if read: print f" median notice: {statistics.median read :.0f} days, " f"range {read 0 }-{read -1 } days" print " basis: regex on the label text; 1 month = 30 days " bands = 1, 30 , 31, 90 , 91, 180 , 181, 400 print " " + " ".join f"{a}-{b}d: {sum a <= x <= b for x in read }" for a, b in bands med = int statistics.median read print f" ends {RENEWS ON}, median {med}-day notice - act by " f"{RENEWS ON - timedelta days=med }, not {RENEWS ON}" redacted = sum 1 for d, x in days if x is None and " " in " ".join answers d, "Notice Period To Terminate Renewal" print f" unreadable: {len trap - len read } {redacted} redacted with " d0 = trap 0 x0, phrase = notice match answers d0, "Notice Period To Terminate Renewal" print f" raw: {d0 'title' :40 } - {x0} days from {phrase r}" The other three checks: python def amendments docs : """02: amendments whose parent is not in the set.""" amend = d for d in docs if re.search r"amend", d "title" , re.I names = d "title" for d in docs print "\n02 AMENDMENT TRAP amendment with no parent document " orphans = 0 for d in amend: filer = re.split r" , | - ", d "title" 0 .strip .upper parent = t for t in names if t.upper .startswith filer and not re.search r"amend", t, re.I orphans += not parent parent.sort key=len tag = "ORPHAN" if not parent else "parent: " + parent 0 -24: print f" {d 'title' :44 :44} {tag}" print f" amendments: {len amend }, orphans: {orphans} basis: title match " def restated docs, n : """03: documents that replace an earlier version.""" def says d, k : return re.search r"amended\s+and\s+restated", d "paragraphs" 0 "context" :k , re.I own = d for d in docs if says d, 300 near = d for d in docs if says d, 3000 print "\n03 RESTATED TRAP doc replaces an earlier version " print f" 'amended and restated' in first 300 chars: {len own } of {n}" print f" 'amended and restated' in first 3,000 chars: {len near } of {n}" for d in own: print f" {d 'title' :70 }" def coverage docs : """04: the silent-drop check you run before trusting any total.""" sizes = sorted len d "paragraphs" 0 "context" for d in docs empty = sum 1 for s in sizes if s == 0 print "\n04 COVERAGE every input must come out the other end " print f" empty texts: {empty}; longest: {sizes -1 :,} chars;" f" over 100k chars: {sum s 100 000 for s in sizes }" def main path : try: docs = json.load open path "data" except OSError, ValueError, KeyError as e: sys.exit f"usage: contractcheck.py path/to/CUADv1.json {e} " n = len docs print f"contracts loaded: {n} basis: CUAD v1 expert labels " deadline docs, n amendments docs restated docs, n coverage docs if name == " main ": if len sys.argv = 2: sys.exit "usage: python3 contractcheck.py path/to/CUADv1.json" main sys.argv 1 python3 contractcheck.py CUADv1.json contracts loaded: 510 basis: CUAD v1 expert labels 01 DEADLINE TRAP act-by date is earlier than the end date any renewal or extension clause: 176 of 510 ...that says 'automatic': 115 of 510 23% ...plus a notice period to stop: 91 of 510 18% notice period parsed to days: 82 of 91 median notice: 90 days, range 15-365 days basis: regex on the label text; 1 month = 30 days 1-30d: 21 31-90d: 37 91-180d: 22 181-400d: 2 ends 2027-01-01, median 90-day notice - act by 2026-10-03, not 2027-01-01 unreadable: 9 5 redacted with raw: CENTRACKINTERNATIONALINC 10 29 1999-EX-1 - 15 days from 'fifteen 15 days' Of 176 renewal or extension clauses, 115 say “automatic”. 91 of them, 18% of all 510, also carry a notice period. Most read “renews unless you give notice X before the end”; a few are at-will termination notices. The median notice is 90 days: a January 1 end date means acting by October 3. The talk’s miss was 30 days; 21 of the 82 readable periods are 30 days or less. Five unreadable ones are redacted in the filing. Without the cue, the parser would grab a term length. 02 AMENDMENT TRAP amendment with no parent document FEDERATEDGOVERNMENTINCOMESECURITIESINC 04 28 parent: AGREE-SERVICES AGREEMENT NETGEAR,INC 04 21 2003-EX-10.16-AMENDMENT TO parent: 16-DISTRIBUTOR AGREEMENT NEONSYSTEMSINC 03 01 1999-EX-10.5-DISTRIBUTO parent: ISTRIBUTOR AGREEMENT New Columbia Laboratories, Bermuda Ltd. - AMEN ORPHAN BIOAMBERINC 04 10 2013-EX-10.34-DEVELOPMENT parent: EVELOPMENT AGREEMENT 1 NETGEAR,INC 04 21 2003-EX-10.16- AMENDMENT parent: 16-DISTRIBUTOR AGREEMENT FEDERATEDGOVERNMENTINCOMESECURITIESINC 04 28 parent: AGREE-SERVICES AGREEMENT Cerus Corporation - FIRST AMEND TO SUPPLY AN ORPHAN ADMA BioManufacturing, LLC - Amendment 3 t ORPHAN amendments: 9, orphans: 3 basis: title match CUAD is a sample, like a filtered export: three of nine amendments Columbia Laboratories, Cerus, ADMA have no parent in the set, and those filers appear nowhere else in it. An agent asked for their terms sees only the amendment. Titles undercount, too: more amendments hide under plain agreement names. 03 RESTATED TRAP doc replaces an earlier version 'amended and restated' in first 300 chars: 4 of 510 'amended and restated' in first 3,000 chars: 15 of 510 PACIRA PHARMACEUTICALS, INC. - A R STRATEGIC LICENSING, DISTRIBUTION A VARIABLESEPARATEACCOUNT 04 30 2014-EX-13.C-UNCONDITIONAL CAPITAL MAINT Reinsurance Group of America, Incorporated - A R REMARKETING AGREEMEN Monsanto Company - SECOND A R EXCLUSIVE AGENCY AND MARKETING AGREEMENT 4 contracts open by calling themselves “amended and restated”, replacing an earlier version. 11 more use the phrase later, mostly citing other agreements, so a keyword hit isn’t proof. If both versions are in your folder, a sum counts the deal twice. 04 COVERAGE every input must come out the other end empty texts: 0; longest: 338,211 chars; over 100k chars: 81 81 contracts exceed 100,000 characters. If your preprocessing has a timeout, these go first. The fix is counting: documents in must equal rows out. Save this beside contractcheck.py as mine.py and run it: mine.py - coverage and notice checks on your own pipeline's outputfrom contractcheck import notice matchinput ids = "c1", "c2", "c3" every file you fed inrows = {"id": "c1", "spend": 120 000}, what came out the other end {"id": "c3", "spend": 0} missing = set input ids - {r "id" for r in rows}print f"dropped: {sorted missing }" print f"zero spend: { r 'id' for r in rows if r 'spend' == 0 }" clause = "This Agreement renews automatically for one 1 year terms unless " "either party gives at least sixty 60 days prior written notice." print "notice:", notice match clause dropped: 'c2' zero spend: 'c3' notice: 60, 'sixty 60 days' Swap in your IDs and clauses. An unexplained zero is probably an error. In 510 real contracts, roughly 1 in 6 auto-renews with a deadline before its end date, so the setup behind the AlixPartners date miss is common. Claude Code got messy data and a hidden business rule. Before trusting an agent’s contract total, check in code: counts match, amendments link, restatements replace, and act-by equals end date minus notice. Which deadline in your contracts would you least like to learn about a month late? Reproduce it: contractcheck.py and mine.py print every output block above. The talk's figures $800K, $17M, Nov 1 come from the video. If this saves you a bill, a clap helps others find it. Follow me for daily LLM deep-dives. Claude Code Missed $800K in Contracts. Here’s How to Check Yours https://pub.towardsai.net/claude-code-missed-800k-in-contracts-heres-how-to-check-yours-e8090336744b was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.