ChatGPT vs Claude Email Test: 12 Steps, 75 Min A 12-step, roughly 400-line Python harness with no third-party packages scores ChatGPT and Claude email drafts from 0 to 100 across six scenarios with planted traps, producing 36 scored emails in about 75 minutes, according to Shattered.io. The scorer weights required current facts at 40 points, stale or forbidden content at 20 points (minus 10 per hit), invented numbers at 10 points (minus 5 per digit string), and word cap, ask placement, filler openers and hedge density at 10, 10, 5 and 5 points respectively, so facts and leaks carry 60 of the 100 points. The test follows Anthropic's Claude Fable 5.2 and OpenAI's GPT-6.2 Astra and GPT-6.2 Sol launches, and Shattered.io states the API bill for a live run stays well under one dollar at November 2026 prices, with output samples drawn from a built-in offline mode rather than live benchmark results. Anthropic shipped Claude Fable 5.2, and OpenAI answered with GPT-6.2 Astra and GPT-6.2 Sol, which 9to5Google described as a bid to win back the lead https://9to5google.com/2026/11/17/openai-releases-gpt-6-2-astra-and-sol-for-complex-projects-a-week-after-claude-5-2/ from Fable 5.2. Two flagship launches in quick succession put the same question in every comment thread. Which one writes better? A leaderboard cannot answer that for your inbox. A ChatGPT vs Claude comparison only means something when both models read the same messy thread and you grade what they would actually send. Most writing tests judge prose. Emails fail somewhere else. The model quotes the invoice amount from message one after message three corrected it. It names the rival vendor you told it to keep out of the reply. It books the meeting in the room that changed on Tuesday. Those are thread-tracking errors, and a few regular expressions catch them in milliseconds. A voice-matching test https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ measures how much a draft sounds like you. The harness below measures something blunter: whether the draft is safe to send. You will build about 400 lines of Python with no third-party packages, six email scenarios with planted traps, and a scorer that returns a 0 to 100 grade per reply. A full run produces 36 scored emails and takes roughly 75 minutes to set up and verify. By my arithmetic the API bill for a live run stays well under one dollar at the November 2026 prices listed below. Every code block was executed before publication, and the output samples come from the built-in offline mode, so nothing here pretends to be a live benchmark result. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F What the ChatGPT vs Claude Email Test Measures Each scenario hands the model a short thread plus a task, for example reply to the customer who is waiting on a payment date. The thread always contains one superseded detail and one current detail. The scorer checks that the current facts appear, that the stale or forbidden content does not, and that the reply respects a word cap. It also logs style signals such as filler openers, hedging, and stock sign-offs. The scoring weights add up to 100 points. They are my own choices, built around the cost of a bad email. A wrong amount or a leaked competitor costs more than a clumsy opener, so facts and leaks carry 60 of the 100 points. | Check | How it is scored | Points | |---|---|---| | Required current facts | Share of the scenario’s required regex patterns found in the reply | 40 | | Stale or forbidden content | Minus 10 per forbidden pattern hit, floor of zero | 20 | | Invented numbers | Minus 5 per digit string not present in the thread or task | 10 | | Word cap | Full points when the reply stays at or under the cap | 10 | | Ask placement | Request appears in the first 60% of words: 10. Later: 5. Missing: 0. Scenarios with no ask get 10 | 10 | | Filler opener | Zero when the first 30 words contain a stock opener | 5 | | Hedge density | 5 at one hedge or fewer per 100 words, 2.5 up to 2.5 per 100, otherwise 0 | 5 | Three more signals get logged but never change the score: stock closers such as best regards, the em dash count, and the raw word count. They exist so you can spot patterns later, for instance if one vendor reaches for em dashes in every second reply. The test deliberately skips warmth, humor, and brand voice. Software cannot grade those, and pretending otherwise produces fake precision. Step 12 covers the human side with an edit-rate measurement. If your priority is house rules such as banned phrases and reading level, the writing rules compliance test https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ is the better starting point, and this harness complements it by focusing on facts rather than rules. Prerequisites and Versions for the Test - Python 3.11 or newer. The harness uses only the standard library urllib, json, csv, difflib, unittest , so there is no pip install step and no dependency to pin. - An OpenAI API key with billing enabled. OpenAI states that API usage is billed separately from ChatGPT subscriptions, so a Plus or Pro plan does not cover this. - An Anthropic API key created in the Claude Platform, again separate from a Claude chat subscription. - A terminal. The shell snippets use bash syntax. PowerShell equivalents appear in step 2. - The column utility optional for reading the CSV in step 11. Any spreadsheet works instead. - About 75 minutes and roughly 500 lines of copy-paste, including the scenario file. You do not need keys to finish the first nine steps. The offline mode swaps both vendors for two fake providers, one that writes ideal replies and one that writes deliberately bad ones. That lets you prove the scorer works before spending a cent, and it gives you the exact output shown in this tutorial. Budget the live run before you start. Count your calls, estimate the tokens each one sends and receives, and price them at the Opus 5.5 rate of $4 and $20 per million tokens. Fable 5.2 is priced at $8 and $40 per million tokens, so the same run costs noticeably more. Treat your token counts as estimates, and check your provider dashboard after the first run. Which Claude and ChatGPT Models to Pin in November 2026 Model names changed fast this autumn, and a comparison that pins last quarter’s models measures nothing useful. The harness therefore reads model IDs from environment variables and records them in every CSV row. The current lineups, as of November 18, 2026, look like this. Claude lineup Anthropic’s current model comparison lists Claude Haiku 5.5, Claude Sonnet 5.5, Claude Opus 5.5, and Claude Fable 5.2. Sonnet 5.5 arrived September 28 and Haiku 5.5 on October 7. Anthropic says Opus 5.5 is the default model in Claude on Pro, Max, Ultra, and Team plans https://claude.com/resources/articles/using-claude-opus-5-5-as-your-daily-driver-for-knowledge-work and recommends it for documents, research, and finished files. For an email test, Opus 5.5 is therefore the like-for-like Claude, because it is what most subscribers actually get when they open a chat. Claude Fable 5.2 and Claude Mythos 5.2 launched together, as the Anthropic announcements page https://www.anthropic.com/news/announcements?trk=article-ssr-frontend-pulse little-text-block lists. Fable 5.2 costs $8 per million input tokens and $40 per million output tokens. Anthropic’s current material says Fable 5.2 generally sits outside Pro, Max, Ultra, and standard Team usage limits except during launch offers, so subscribers reach it through usage credits. Treat it as the premium challenger rather than the default. ChatGPT lineup OpenAI lists Free, Go, Plus, Business, Enterprise, and three Pro tiers on its ChatGPT pricing page https://chatgpt.com/pricing/ . Plus costs $20 per month in the United States, and Pro comes as Pro 100, Pro 200, and Pro 500 at $100, $200, and $500 per month. On the API side, OpenAI documents gpt-6-sol and gpt-6-luna, with Sol at $2 and $10 and Luna at $0.10 and $0.50 per million input and output tokens after the price cuts announced November 17. For a deeper look at what each plan includes, see the ChatGPT vs Claude plan and usage-limit breakdown https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ . Two cautions apply. First, OpenAI’s own pages disagree on which model label each consumer plan receives, and on whether the newest Sol and Luna releases already sit inside ChatGPT. Do not trust a plan page to tell you what answered your prompt. Log the model ID you pinned, which is exactly what the harness does. Second, retired models still show up in old tutorials. OpenAI retired o3 from ChatGPT on August 26, 2026, and GPT-4.5 has been retired as well, so neither belongs in a November comparison. | Model | Vendor | API ID | Input per 1M tokens | Output per 1M tokens | |---|---|---|---|---| | Claude Opus 5.5 | Anthropic | Copy from the models list in step 4 | $4 | $20 | | Claude Fable 5.2 | Anthropic | claude-fable-5-2 | $8 | $40 | | GPT-6 Sol | OpenAI | gpt-6-sol | $2 | $10 | | GPT-6 Luna | OpenAI | gpt-6-luna | $0.10 | $0.50 | A sensible first pairing is Opus 5.5 against GPT-6 Sol, the two models most paying users meet by default. Run the premium pairing, Fable 5.2 against OpenAI’s newest Astra model, as a second pass. If you later want to translate scores into money, the cost per accepted 1,000 words harness https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/ shows how to divide spend by usable output instead of by tokens. Steps 1 to 4: Project Folder, Keys, Provider Layer, and Model IDs Step 1 creates the project folder and confirms the Python version. Step 2 exports the two API keys for the current shell session only, so nothing lands in a file you might commit. mkdir emailtest cd emailtest python3 --version bash or zsh export OPENAI API KEY="paste-your-openai-key" export ANTHROPIC API KEY="paste-your-anthropic-key" PowerShell equivalent $env:OPENAI API KEY = "paste-your-openai-key" $env:ANTHROPIC API KEY = "paste-your-anthropic-key" The version command should print 3.11 or higher. If it prints an older version, install a newer interpreter before going on. Step 3 writes providers.py , the only file that talks to the network. It sends the same system prompt and the same user prompt to both vendors, calls each REST endpoint with plain urllib, retries on rate-limit and overload codes, and strips a stray subject line if a model adds one. It deliberately sends no temperature and no token limit for OpenAI. Some current models reject sampling parameters or renamed token-limit fields, and leaving them out keeps the request valid on every model. Anthropic requires max tokens , so that one stays. python import json import os import time import urllib.error import urllib.request SYSTEM = "You write business emails. Output only the email body, starting with the greeting. " "No subject line and no commentary before or after the email." def clean text : lines = text.strip .splitlines if lines and lines 0 .lower .startswith "subject:" : lines = lines 1: return "\n".join lines .strip def build prompt scenario : facts = "\n".join "- " + line for line in scenario "facts" return "Source material:\n" + facts + "\n\nTask: " + scenario "task" RETRY CODES = 429, 500, 502, 503, 529 def request url, headers, payload=None : data = json.dumps payload .encode "utf-8" if payload is not None else None for attempt in range 4 : req = urllib.request.Request url, data=data, headers=headers try: with urllib.request.urlopen req, timeout=180 as resp: return json.load resp except urllib.error.HTTPError as err: if err.code not in RETRY CODES or attempt == 3: raise RuntimeError "HTTP %d from %s: %s" % err.code, url, err.read .decode "utf-8", "replace" time.sleep 2 attempt def openai headers : return { "Authorization": "Bearer " + os.environ "OPENAI API KEY" , "Content-Type": "application/json", } def anthropic headers : return { "x-api-key": os.environ "ANTHROPIC API KEY" , "anthropic-version": "2023-06-01", "Content-Type": "application/json", } def call openai scenario : body = request "https://api.openai.com/v1/chat/completions", openai headers , { "model": os.environ "OPENAI MODEL" , "messages": {"role": "system", "content": SYSTEM}, {"role": "user", "content": build prompt scenario }, , }, return clean body "choices" 0 "message" "content" def call anthropic scenario : body = request "https://api.anthropic.com/v1/messages", anthropic headers , { "model": os.environ "ANTHROPIC MODEL" , "max tokens": 1024, "system": SYSTEM, "messages": {"role": "user", "content": build prompt scenario } , }, parts = b "text" for b in body "content" if b.get "type" == "text" return clean "".join parts def call fake good scenario : return scenario "fake good" def call fake bad scenario : return scenario "fake bad" LIVE = {"openai": call openai, "anthropic": call anthropic} FAKE = {"fake-good": call fake good, "fake-bad": call fake bad} def list models : out = {} openai body = request "https://api.openai.com/v1/models", openai headers out "openai" = sorted m "id" for m in openai body "data" anth body = request "https://api.anthropic.com/v1/models?limit=1000", anthropic headers out "anthropic" = sorted m "id" for m in anth body "data" return out The two fake providers near the bottom return the fake good and fake bad strings stored in each scenario. They are what make the offline mode possible. Step 4 lists the model IDs your keys can reach. Run the one-liner and copy the IDs you want. The Anthropic request asks for up to 1,000 results because the endpoint pages its answers by default. python python3 -c "import json, providers; print json.dumps providers.list models , indent=1 " Expect two long JSON lists, abbreviated here. Your account’s list will differ, and the exact IDs are the only thing you need from it. { "openai": ... "gpt-6-luna", "gpt-6-sol", ... , "anthropic": ... "claude-fable-5-2", ... } Find the Opus 5.5 entry in the Anthropic list and pin both models. Write the IDs down next to the run date, since a vendor can repoint an alias without warning. export OPENAI MODEL=gpt-6-sol export ANTHROPIC MODEL=PASTE THE OPUS 5 5 ID FROM THE LIST Step 5: Write Six Email Scenarios With Planted Traps Step 5 creates scenarios.json . Each entry holds the thread facts , the instruction task , a word cap, whether the email should contain a request, the regex patterns the reply must match, the patterns it must avoid, and the two offline replies. The traps are the point. The invoice thread contains an old amount that a credit later changed. The meeting thread moves a design review from Tuesday in Room 4B to Thursday in Room 7A. The vendor thread includes an internal note that names a competing quote the reply must never reveal. The outage thread carries a wrong early estimate of two hours and a credit tier that applied to a different incident. The raise request tempts a model into leverage language the employee did not want. The cold email supplies one real result and invites invented proof points. The idea echoes a source-grounded writing audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ , except the sources here are email threads and the failure you hunt is a stale fact. { "id": "invoice chase", "facts": "Oct 3, Dana to Priya: Invoice 1182 for $4,800 is due Oct 15.", "Oct 9, Priya to Dana: We spotted a duplicate line. Please reissue.", "Oct 10, Dana to Priya: Reissued as 1182-R for $4,350 after the $450 credit. Same Oct 15 due date.", "Nov 6, Priya to Dana: Sorry for the delay. Finance approved 1182-R. Payment goes out Nov 20." , "task": "Write Dana Ortiz's reply to Priya Nair. Goal: lock in the payment date. Maximum 120 words.", "cap": 120, "ask": true, "must": "1182-R", "\\$4,350", "Nov ember ? 20" , "forbidden": "\\$4,800" , "fake good": "Hi Priya,\n\nThanks for the update. Can you confirm that 1182-R for $4,350 will land on Nov 20? I will mark it as scheduled on our side once I hear back.\n\nDana", "fake bad": "Hi Priya,\n\nI hope this email finds you well. I just wanted to reach out about invoice 1182 for $4,800, which I think was due a while ago. Maybe you could possibly let me know when payment will arrive? It might be around 30 days.\n\nBest regards,\nDana" }, { "id": "meeting move", "facts": "Mon, Marcus to team: Design review is Tuesday at 2:00 pm in Room 4B.", "Tue, Marcus to team: Moving it. New slot is Thursday at 10:30 am in Room 7A. Same agenda.", "Tue, Lena to Marcus: Can you tell the Halvorsen folks? They only got the first invite." , "task": "Write Marcus Bell's message to Tomas Eriksen at Halvorsen. Maximum 90 words.", "cap": 90, "ask": true, "must": "Thursday", "10:30", "7A" , "forbidden": "2:00", "2 pm", "4B" , "fake good": "Hi Tomas,\n\nThe design review moved to Thursday at 10:30 am in Room 7A. The agenda is unchanged. Please confirm you can still join.\n\nMarcus", "fake bad": "Hello Tomas,\n\nA quick reminder that the design review is on Tuesday at 2:00 pm in Room 4B. We are also thinking of 11 am on Friday as a backup. Let me know what works.\n\nMarcus" }, { "id": "vendor decline", "facts": "Sep 18, Harbor Cloud to Ines: Proposal attached, $14,500 per year for the analytics add-on.", "Sep 22, Ines to Jonas internal : Northwind quoted $9,500 for similar scope. Do not mention them to Harbor.", "Oct 30, Jonas to Ines: Budget is frozen until Q2. We pass for now, but keep the door open." , "task": "Write Ines Duarte's reply to Wen Zhao at Harbor Cloud, declining for now. Maximum 100 words.", "cap": 100, "ask": false, "must": "Q2", " not moving forward|won't be moving forward|will not be moving forward|decline|pass on|not proceed|unable to proceed " , "forbidden": "Northwind", "9,500" , "fake good": "Hi Wen,\n\nThank you for the proposal. We are not moving forward right now because our budget is frozen until Q2. We would like to stay in touch and revisit it then.\n\nInes", "fake bad": "Hi Wen,\n\nThank you so much for the proposal. Another vendor, Northwind, quoted $9,500 for similar scope, so we will pass. Maybe talk again at some point.\n\nInes" }, { "id": "outage apology", "facts": "Nov 12, status page: Elevated errors, investigating. Early estimate: up to 2 hours to restore.", "Nov 12, ops: Resolved after 41 minutes. Root cause was an expired certificate on the login service.", "Nov 13, support lead to Amy: Customers get a 15% credit on the November invoice. No 20% tier, that was for the August incident." , "task": "Write the apology email Amy Cho sends to affected customers. Maximum 140 words.", "cap": 140, "ask": false, "must": "41 minutes", "15%", "certificate" , "forbidden": "2 hours", "two hours", "20%" , "fake good": "Hi all,\n\nOn Nov 12 our login service failed for 41 minutes because an expired certificate went unnoticed. That is our mistake. We have added expiry alerts so it cannot repeat. Your November invoice will carry a 15% credit automatically.\n\nAmy", "fake bad": "Dear customer,\n\nWe sincerely apologize for the inconvenience. Our outage lasted about 2 hours and we are offering a 20% credit. We take reliability very seriously and 99.99% uptime remains our goal.\n\nAmy" }, { "id": "raise request", "facts": "Sam's notes: Joined 3 years ago. Led the billing migration that cut cloud spend by $38,000 a year.", "Sam's notes: The salary band review closes Dec 5. Sam wants a 20-minute conversation, not a negotiation by email.", "Rhea Kline's last message: Happy to talk. Send me your points before the review closes." , "task": "Write Sam Whitaker's email to his manager Rhea Kline. Maximum 150 words.", "cap": 150, "ask": true, "must": "\\$38,000", "Dec ember ? 5", "20 - minute" , "forbidden": "quit", "resign", "other offers?", "competing offer" , "fake good": "Hi Rhea,\n\nBefore the review closes on Dec 5, here are my points. In 3 years I led the billing migration, which cut cloud spend by $38,000 a year. Could we set up a 20-minute conversation this week to talk about my band?\n\nSam", "fake bad": "Hi Rhea,\n\nI hope you are doing well. I was wondering if maybe we could discuss my compensation. I have been here 3 years and have taken on 40% more work. I have other offers, so I would like to understand my options.\n\nBest,\nSam" }, { "id": "cold outreach", "facts": "Prospect: Imani Brooks, head of finance ops at Calder Freight.", "Calder Freight launched Atlas Reports last month.", "Our tool, Tally Lens, cut report generation from 14 minutes to 3 minutes in a customer pilot.", "One 15-minute demo slot is open on Wed Nov 25." , "task": "Write a cold email from Theo Marsh to Imani Brooks. Maximum 110 words.", "cap": 110, "ask": true, "must": "Atlas Reports", "14 minutes", "3 minutes", "Nov ember ? 25" , "forbidden": "trusted by", "industry-leading", "world-class", "thousands of" , "fake good": "Hi Imani,\n\nCongratulations on launching Atlas Reports last month. In a customer pilot, Tally Lens cut report generation from 14 minutes to 3 minutes. I have a 15-minute demo slot on Nov 25. Would that work for you?\n\nTheo", "fake bad": "Hi Imani,\n\nI wanted to reach out because Tally Lens is trusted by thousands of teams and cuts reporting time by 80%. Our industry-leading platform could really help Calder. Would you be open to a quick chat?\n\nWarm regards,\nTheo" } Notice that no prompt warns the model about the traps. If you add a sentence like watch out for outdated numbers, you stop testing thread tracking and start testing instruction following. Keep the task text exactly as written for both vendors. The regex patterns are intentionally simple. The JSON string \\$4,350 matches the literal amount with its comma, which suits these prompts because the source text uses commas. If your own threads format amounts differently, widen the pattern, for example to allow an optional comma. Steps 6 and 7: The Scorer and Its Unit Tests Step 6 writes score.py . The evaluate function normalizes curly quotes first, because models emit typographic apostrophes and a pattern like won't be moving forward would otherwise miss the curly version. It then counts required hits, forbidden hits, and digit strings that appear in the reply but nowhere in the source, and converts everything to the 100-point score from the table above. python import re FILLER = r"hope this email|message finds you", r"hope you '| a re doing ?well", r"i just ?wanted to reach out|touch base|follow up ", r"i am writing to", r"i 'm| am reaching out", HEDGES = r"\bjust\b", r"\bmaybe\b", r"\bperhaps\b", r"\bpossibly\b", r"\bi think\b", r"\bsort of\b", r"\bkind of\b", r"\ba bit\b", r"\bsomewhat\b", r"\bi was wondering\b", r"\bmight\b", STOCK CLOSERS = r"best regards", r"warm regards", r"kind regards", r"looking forward to hearing" ASK MARKERS = r"\?|\bplease\b|\bcould you\b|\bcan you\b|\bwould you\b|\blet me know\b|\bconfirm\b" EM DASH = ", " def words text : return re.findall r"\S+", text def numbers text : found = set for token in re.findall r"\d \d, ?:\.\d+ ?", text : found.add token.rstrip "," .replace ",", "" return found def count hits patterns, text : return sum 1 for p in patterns if re.search p, text, re.IGNORECASE def ask position text : match = re.search ASK MARKERS, text, re.IGNORECASE if not match: return None before = len words text : match.start return before / max 1, len words text def normalize text : return text.replace "\u2019", "'" .replace "\u2018", "'" .replace "\u201c", '"' .replace "\u201d", '"' def evaluate scenario, reply : reply = normalize reply low = reply.lower n words = len words reply must hits = sum 1 for p in scenario "must" if re.search p, reply, re.IGNORECASE must ratio = must hits / len scenario "must" forbidden = count hits scenario "forbidden" , reply allowed = numbers " ".join scenario "facts" + " " + scenario "task" invented = len numbers reply - allowed cap ok = n words <= scenario "cap" first chunk = " ".join low.split :30 filler = count hits FILLER, first chunk 0 hedge density = 100.0 count hits HEDGES, low / max 1, n words pos = ask position reply if scenario "ask" : if pos is None: ask points = 0.0 elif pos <= 0.6: ask points = 10.0 else: ask points = 5.0 else: ask points = 10.0 if hedge density <= 1.0: hedge points = 5.0 elif hedge density <= 2.5: hedge points = 2.5 else: hedge points = 0.0 score = 40.0 must ratio + max 0.0, 20.0 - 10.0 forbidden + max 0.0, 10.0 - 5.0 invented + 10.0 if cap ok else 0.0 + ask points + 0.0 if filler else 5.0 + hedge points return { "score": round score, 1 , "words": n words, "cap ok": cap ok, "must ratio": round must ratio, 2 , "forbidden": forbidden, "invented": invented, "ask pos": None if pos is None else round pos, 2 , "filler": filler, "hedge density": round hedge density, 2 , "stock closer": count hits STOCK CLOSERS, low 0, "em dashes": reply.count EM DASH , } The invented-number check works because the allowed set comes from the thread and the task. Commas are stripped, so $4,350 and 4350 compare equal. A reply that writes 80% when no source mentions 80 gets flagged, which is exactly the cold-email failure mode. Step 7 locks the behavior in place with unit tests. The first test asserts that the ideal reply beats the bad reply by more than 40 points in every scenario. The others pin the stale-amount flag, the invented-number flag, the word cap, the curly-apostrophe fix, the number normalizer, and the subject-line stripper. python import json import unittest import providers import score class ScoreTests unittest.TestCase : def setUp self : with open "scenarios.json", encoding="utf-8" as fh: self.scenarios = {s "id" : s for s in json.load fh } def test good reply beats bad reply everywhere self : for s in self.scenarios.values : good = score.evaluate s, s "fake good" "score" bad = score.evaluate s, s "fake bad" "score" self.assertGreater good, bad + 40, s "id" def test stale amount is flagged self : s = self.scenarios "invoice chase" result = score.evaluate s, "Hi Priya, please confirm $4,800 on Nov 20 for 1182-R." self.assertEqual result "forbidden" , 1 def test invented number is flagged self : s = self.scenarios "cold outreach" result = score.evaluate s, "Hi Imani, we cut reporting time by 80%." self.assertEqual result "invented" , 1 def test word cap self : s = self.scenarios "meeting move" result = score.evaluate s, "word " 200 self.assertFalse result "cap ok" def test curly apostrophe still matches self : s = self.scenarios "vendor decline" result = score.evaluate s, "Hi Wen, we won\u2019t be moving forward before Q2." self.assertEqual result "must ratio" , 1.0 def test numbers normalizes commas self : self.assertEqual score.numbers "$4,350 and 1182-R." , {"4350", "1182"} def test clean drops subject line self : self.assertEqual providers.clean "Subject: Hello\n\nHi Wen,\nThanks." , "Hi Wen,\nThanks." if name == " main ": unittest.main python3 -m unittest -v test clean drops subject line test score.ScoreTests.test clean drops subject line ... ok test curly apostrophe still matches test score.ScoreTests.test curly apostrophe still matches ... ok test good reply beats bad reply everywhere test score.ScoreTests.test good reply beats bad reply everywhere ... ok test invented number is flagged test score.ScoreTests.test invented number is flagged ... ok test numbers normalizes commas test score.ScoreTests.test numbers normalizes commas ... ok test stale amount is flagged test score.ScoreTests.test stale amount is flagged ... ok test word cap test score.ScoreTests.test word cap ... ok ---------------------------------------------------------------------- Ran 7 tests in 0.005s OK Seven passing tests mean the scorer behaves. Any later change to a regex or a weight should rerun this command first. A scorer you cannot trust is worse than no scorer, because it hands you confident wrong answers. Steps 8 and 9: The Runner and the Offline Dry Run Step 8 writes run.py , the command-line entry point. It has four commands. models prints the ID lists. run generates replies and scores them. rescore rescans the out/ folder without calling any API, which is the command to use after you tweak a regex. report prints the summary table. Every reply lands in a file named after its scenario and run number inside a folder named after the provider, so you can read it in a plain editor. python import argparse import csv import glob import json import os import sys from collections import defaultdict from statistics import mean import providers import score FIELDS = "provider", "model", "scenario", "run", "score", "words", "cap ok", "must ratio", "forbidden", "invented", "ask pos", "filler", "hedge density", "stock closer", "em dashes" NOISE = "Thanks, noted.", "Sounds good.", "Looping in finance.", "Checking with the team." def load scenarios : with open "scenarios.json", encoding="utf-8" as fh: return json.load fh def pad scenario, n : noise = "Another message in the thread: " + NOISE i % len NOISE for i in range n padded = dict scenario padded "facts" = scenario "facts" :1 + noise + scenario "facts" 1: return padded def cmd models args : for vendor, ids in providers.list models .items : print "== " + vendor for model id in ids: print model id def cmd run args : scenarios = load scenarios table = providers.FAKE if args.dry run else providers.LIVE for name, fn in table.items : folder = os.path.join "out", name os.makedirs folder, exist ok=True model = os.environ.get name.upper + " MODEL", "offline" with open os.path.join folder, "model.txt" , "w", encoding="utf-8" as fh: fh.write model + "\n" for scenario in scenarios: used = pad scenario, args.pad if args.pad else scenario for run in range args.runs : reply = fn used path = os.path.join folder, "%s %d.txt" % scenario "id" , run with open path, "w", encoding="utf-8" as fh: fh.write reply + "\n" score folders list table def cmd rescore args : names = sorted os.path.basename p for p in glob.glob "out/ " if os.path.isdir p score folders names def score folders names : scenarios = {s "id" : s for s in load scenarios } rows = for name in names: model file = os.path.join "out", name, "model.txt" model = open model file, encoding="utf-8" .read .strip if os.path.exists model file else "unknown" for path in sorted glob.glob os.path.join "out", name, " .txt" : stem = os.path.basename path :-4 scenario id, run = stem.rsplit " ", 1 if scenario id not in scenarios: continue with open path, encoding="utf-8" as fh: reply = fh.read row = {"provider": name, "model": model, "scenario": scenario id, "run": int run } row.update score.evaluate scenarios scenario id , reply rows.append row with open "results.csv", "w", newline="", encoding="utf-8" as fh: writer = csv.DictWriter fh, fieldnames=FIELDS writer.writeheader writer.writerows rows print "wrote %d rows to results.csv" % len rows def cmd report args : by provider = defaultdict list by cell = defaultdict list with open "results.csv", encoding="utf-8" as fh: for row in csv.DictReader fh : value = float row "score" by provider row "provider" .append value by cell row "scenario" , row "provider" .append value print "%-16s %6s %6s %6s" % "provider", "mean", "min", "n" for name, values in sorted by provider.items : print "%-16s %6.1f %6.1f %6d" % name, mean values , min values , len values print names = sorted by provider print "%-16s " % "scenario" + " ".join "%12s" % n for n in names for scenario in sorted {k 0 for k in by cell} : cells = "%12.1f" % mean by cell scenario, n for n in names print "%-16s " % scenario + " ".join cells def main : parser = argparse.ArgumentParser sub = parser.add subparsers dest="cmd", required=True sub.add parser "models" .set defaults fn=cmd models run = sub.add parser "run" run.add argument "--dry-run", action="store true" run.add argument "--runs", type=int, default=3 run.add argument "--pad", type=int, default=0 run.set defaults fn=cmd run sub.add parser "rescore" .set defaults fn=cmd rescore sub.add parser "report" .set defaults fn=cmd report args = parser.parse args args.fn args if name == " main ": sys.exit main A model.txt file in each provider folder records the pinned model, and the CSV carries it in a model column. When a vendor ships an update mid-month, you can tell which rows came from which model. Step 9 runs everything offline. The fake providers answer instantly, and the report proves the pipeline works from scenario to CSV to summary. python3 run.py run --dry-run --runs 3 python3 run.py report wrote 36 rows to results.csv provider mean min n fake-bad 30.0 25.0 18 fake-good 97.5 95.0 18 scenario fake-bad fake-good cold outreach 25.0 95.0 invoice chase 30.0 100.0 meeting move 30.0 95.0 outage apology 35.0 100.0 raise request 25.0 95.0 vendor decline 35.0 100.0 The ideal provider scores far above the bad one. Some ideal replies lose points because their request lands late in the email, which shows the ask-placement rule at work on perfectly reasonable writing. Decide whether that rule suits your own mail before you trust it. Cold emails and raise requests often end on the ask by design, and you may prefer to remove the penalty. Steps 10 and 11: The Live Run and Reading the Report Step 10 runs the real comparison. Confirm that both model variables are set, then run three passes per scenario. Three passes matter because a single sample hides variance, and a vendor can win one run and lose the next on identical input. echo $OPENAI MODEL $ANTHROPIC MODEL python3 run.py run --runs 3 python3 run.py report The run prints wrote 36 rows to results.csv when two providers, six scenarios, and three passes complete. If it stops early with an HTTP error, jump to the troubleshooting section. The runner writes replies to disk as it goes, so a partial run still leaves readable files, and rescore can pick them up. Step 11 reads the results at two levels. The summary table gives a mean, a minimum, and a sample count per provider, then a scenario-by-provider grid. Start with the minimum. A vendor with a 92 mean and a 45 minimum has a trap it falls into sometimes, and that single bad email is the one that reaches a customer. Then open the per-email flags to see why a score dropped. column -s, -t results.csv | head -20 offline sample rows fake providers provider model scenario run score words cap ok must ratio forbidden invented ask pos filler hedge density stock closer em dashes fake-good offline invoice chase 0 100.0 32 True 1.0 0 0 0.19 False 0.0 False 0 fake-good offline invoice chase 1 100.0 32 True 1.0 0 0 0.19 False 0.0 False 0 fake-good offline invoice chase 2 100.0 32 True 1.0 0 0 0.19 False 0.0 False 0 fake-bad offline invoice chase 0 30.0 48 True 0.0 1 1 0.67 True 10.42 True 0 Read the low-scoring replies themselves, not just the numbers. A forbidden hit of 1 on a vendor-decline reply means the model leaked the competitor’s name or quote. A must ratio below 1 can mean a required fact is missing or worded in a way your regex did not anticipate, so open the file before blaming the model. For a human read of the same drafts, shuffle the files and use a blind side-by-side rating https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ so you cannot tell which vendor wrote which email. Step 12: Measure the Edit Rate on Emails You Would Send Step 12 adds the one metric software cannot invent: how much you have to change a draft before it goes out. Read each reply, edit it into the version you would actually send, and save it in an edits folder that mirrors out , using the same provider folder and filename. The script compares original and edited text word by word with difflib and reports the share of words that changed. python import difflib import glob import os from collections import defaultdict from statistics import mean def edit rate original, edited : a, b = original.split , edited.split return 1.0 - difflib.SequenceMatcher None, a, b .ratio def main : rates = defaultdict list for edited path in glob.glob os.path.join "edits", " ", " .txt" : provider = os.path.basename os.path.dirname edited path original path = os.path.join "out", provider, os.path.basename edited path if not os.path.exists original path : print "skip no original : " + edited path continue with open original path, encoding="utf-8" as fh: original = fh.read with open edited path, encoding="utf-8" as fh: edited = fh.read rates provider .append edit rate original, edited for provider, values in sorted rates.items : print "%-16s mean edit rate %.2f over %d emails" % provider, mean values , len values if name == " main ": main Take the offline example. The bad invoice email loses its filler opener and its hedged request, and its edit rate shows how large a share of the words changed. Run the same measurement over your live drafts and the vendor with the lower number saves you typing, even if its raw score is slightly lower. python3 edit rate.py fake-bad mean edit rate 0.18 over 1 emails Edit only what you would really change. If you polish every comma, both vendors drift toward a high edit rate and the signal drowns. For a dedicated treatment of how faithfully each model preserves text it was told to leave alone, the editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ covers that angle. Reading the Results: A Decision Rule You Can Defend Numbers need a rule, or you will read them to match what you already believed. The thresholds below are my rule of thumb for a 36-email sample, not an industry standard. Adjust them if your volume or stakes differ. | What you see | What it means | What to do | |---|---|---| | Mean scores within 5 points across 3 or more passes | Statistical tie on this task | Choose on price, tooling, or the ecosystem you already use | | Any forbidden hit on vendor decline or raise request | The model leaks private context or uses unwanted leverage | Keep a human reviewer on that workflow, or rewrite the prompt and retest | | Invented numbers on cold outreach | The model fabricates proof points | Require fact review before any outbound email | | Minimum score under 60 for one vendor | A trap catches it intermittently | Test with more passes and pad the threads advanced tips | | Edit rate gap above 0.10 | One vendor’s drafts need clearly less rework | Weigh the gap above the raw score difference | Write the decision down before you open the report, then see whether the data changes it. That small habit is the difference between a test and a confirmation. This harness also answers a narrow question about thread-grounded business email. It says nothing about fiction, long reports, or code. For wider context on how model families compare, browse the AI and machine learning coverage https://shattered.io/category/ai-machine-learning/ . Keep in mind what a tie means in practice. Anthropic pitches Opus 5.5 at finished documents and research, and OpenAI announced new professional models on November 17. If your run shows a tie today, rerun it after the next model release rather than treating the result as permanent. Common Pitfalls That Skew Email Comparisons 1. Different prompts per vendor. Paste one system prompt and one user prompt into both providers. The harness enforces this through build prompt , so do not fork it. 2. One run per scenario. Sampling variance can swing a single score by 20 points or more. Use at least three passes and look at the minimum, not just the mean. 3. Trusting a plan page for the model name. OpenAI’s pages disagree about plan labels and rollout status. Pin the ID in the API call and log it. 4. Comparing an API call with a chat app. Apps add their own system prompts, memory, and tools. If you test the apps instead, paste the same prompt, save each reply to out/chatgpt-app/ or out/claude-app/ , and use rescore . Label the result as an app test. 5. Regexes that are too narrow or too broad. A pattern that demands the exact dollar format will mark a correct reply as a miss. A pattern like Q2 will pass any mention of the quarter, correct or not. Read ten replies before you trust a pattern. 6. Hinting at the trap. Any hint in the task turns a thread-tracking test into an instruction-following test. 7. Rewarding length. Without a word cap, the model that writes more tends to hit more required facts. The cap keeps the contest fair. 8. Pinning a retired model. o3 left ChatGPT on August 26, 2026, and GPT-4.5 has been retired too. Old tutorials still reference them. 9. Pasting real customer data. Redact names, amounts, and account numbers before you add your own threads, because every prompt goes to a third-party API. The first and sixth pitfalls cause most bad comparisons. People tweak the prompt for the model they like, then credit the model for the lift. Freeze the task text, commit it, and change it only between full runs. Troubleshooting: 10 Failures and Their Fixes 1. KeyError: ‘OPENAI API KEY’ or ‘ANTHROPIC MODEL’. The variable is not exported in this shell. Export it again, since new terminal tabs start clean. 2. An authentication error from either vendor. The key is wrong, revoked, or pasted with a trailing space. Re-export it with quotes and retry the model listing from step 4. 3. A not-found error or a model-not-found message. The ID has a typo or your account cannot reach that model. Copy it from the step 4 list rather than typing it. 4. HTTP 400 mentioning an unsupported parameter. This usually appears after you add temperature or a token limit. Remove the field, or check the current API reference for the parameter name that model expects. 5. HTTP 400 from Anthropic about max tokens. The field is required on that endpoint. The harness sets it to 1024, so a missing value means you edited the payload. 6. A rate-limit or overloaded-server error after a few calls. You hit a rate limit or an overloaded server. The harness retries four times with growing pauses. If it still fails, lower --runs and rerun in a few minutes. Replies already written stay on disk. 7. Every score is 0 on required facts. Your regexes do not match the reply format. Open one reply file and compare it to the pattern. Amounts without commas and spelled-out months are the usual culprits. 8. The reply contains a subject line or commentary. The harness strips a leading Subject: line but not a chatty preface. Tighten the system prompt or extend clean in providers.py . 9. results.csv is missing or report crashes. You ran report before run or rescore . Generate the file first. 10. A pattern misses because of a typographic character. Add the character to normalize in score.py . Rerun python3 run.py rescore , which needs no API call. When a failure is not on this list, isolate it by running a single scenario with the fake providers, then a single live call. Narrowing a failure to the network layer or the scoring layer usually takes under five minutes. Advanced Tips: Pad Threads, Add Your Own Mail, Schedule Runs Pad the threads. Real threads run longer than four messages. The --pad flag inserts neutral filler messages between the first message and the correction, so the stale fact sits further from the fix. Run python3 run.py run --runs 3 --pad 8 and compare scores with the unpadded run. A vendor whose minimum drops sharply under padding loses track of corrections in long threads. Add your own scenarios. Redact ten real threads, copy the structure of an existing entry, and write the required and forbidden patterns from what you know the correct reply must say. Your own mail beats any synthetic scenario, because it contains the quirks your colleagues actually produce. Test smaller and cheaper models. Add Sonnet 5.5, Haiku 5.5, or GPT-6 Luna by adding a provider function that reads a different model variable. Luna’s listed price of $0.10 and $0.50 per million tokens makes it a cheap way to learn whether routine mail needs a flagship model at all. Change one variable at a time. Once you have a baseline, add a house-style paragraph to the system prompt for both vendors and rerun. The custom instructions pack https://shattered.io/chatgpt-custom-instructions-claude-projects-gemini-gems-one-spec-12-steps-2026/ shows how to keep one spec for ChatGPT and Claude so the two stay identical, and the planted-flaw feedback test https://shattered.io/claude-vs-chatgpt-draft-feedback-planted-flaw-test-12-steps-2026/ applies the same plant-and-detect idea to critique rather than drafting. Schedule it. Models update under stable names. A weekly cron job running python3 run.py run --runs 3 and archiving each CSV turns this harness into a regression monitor. Because every row carries the model ID, a sudden score change lines up with the update that caused it. The Complete Project: File Layout and One-Command Run Everything lives in one folder with six hand-written files. The three folders at the bottom appear as you work. emailtest/ providers.py network layer, fake providers, model listing scenarios.json six threads, traps, patterns, offline replies score.py 100-point scorer test score.py seven unit tests run.py models / run / rescore / report edit rate.py word-level edit rate for your edited drafts out/ replies per provider created by run edits/ your edited versions you create results.csv one row per scored reply created by run The full cycle, from a clean folder to a decision, is the short sequence below. Run the first two lines any time you change the scorer, and the last two once your keys and model IDs are set. python3 -m unittest python3 run.py run --dry-run --runs 3 && python3 run.py report python3 run.py run --runs 3 && python3 run.py report python3 edit rate.py Keep the out/ directory out of version control if it holds sensitive replies. Commit scenarios.json and note the pinned model IDs, since those define the experiment. FAQ: ChatGPT vs Claude for Writing Emails Which is better for emails, ChatGPT or Claude? Neither wins universally, and this tutorial does not claim a winner. The harness exists because the answer depends on your threads, your tone, and your tolerance for stale details. Anthropic positions Opus 5.5 for finished documents, and OpenAI positions its newest models for professional work, so run both on your own mail. Can I run the test without API keys? Yes. The offline mode proves the scorer. To test the chat apps, paste each scenario into the app, save the replies as out/chatgpt-app/invoice chase 0.txt and so on, then run python3 run.py rescore and python3 run.py report . Does the offline result say anything about the real models? No. The fake providers return fixed text. They show that the scorer separates a good reply from a bad one and that the pipeline works end to end. Only a live run measures the vendors. How many runs per scenario are enough? Three is the practical minimum and five is better when scores sit close together. Examine the minimum score, because one bad email sent to a customer matters more than a good average. Why not ask a third model to judge the emails? A judge model adds cost and a bias of its own, and it may favor text that resembles its own style. Regex checks on facts are blunt but repeatable. Use a judge later for tone, and keep the regex layer for facts. How do I add Gemini or another provider? Write one function in providers.py that takes a scenario and returns the cleaned reply text, add it to the LIVE dictionary, and set a matching NAME MODEL variable. The runner and scorer need no changes. Will these scores stay valid next month? Not necessarily. Both vendors ship models every few weeks. Keep the model ID in every row, rerun after each release, and compare against your archived CSVs.