{"slug": "chatgpt-vs-claude-email-test-12-steps-75-min", "title": "ChatGPT vs Claude Email Test: 12 Steps, 75 Min", "summary": "A 12-step, roughly 400-line Python harness with no third-party packages scores ChatGPT and Claude email drafts from 0 to 100 across six scenarios with planted traps, producing 36 scored emails in about 75 minutes, according to Shattered.io. The scorer weights required current facts at 40 points, stale or forbidden content at 20 points (minus 10 per hit), invented numbers at 10 points (minus 5 per digit string), and word cap, ask placement, filler openers and hedge density at 10, 10, 5 and 5 points respectively, so facts and leaks carry 60 of the 100 points. The test follows Anthropic's Claude Fable 5.2 and OpenAI's GPT-6.2 Astra and GPT-6.2 Sol launches, and Shattered.io states the API bill for a live run stays well under one dollar at November 2026 prices, with output samples drawn from a built-in offline mode rather than live benchmark results.", "body_md": "Anthropic shipped Claude Fable 5.2, and OpenAI answered with GPT-6.2 Astra and GPT-6.2 Sol, which [9to5Google described as a bid to win back the lead](https://9to5google.com/2026/11/17/openai-releases-gpt-6-2-astra-and-sol-for-complex-projects-a-week-after-claude-5-2/) from Fable 5.2. Two flagship launches in quick succession put the same question in every comment thread. Which one writes better? A leaderboard cannot answer that for your inbox. A ChatGPT vs Claude comparison only means something when both models read the same messy thread and you grade what they would actually send.\n\nMost writing tests judge prose. Emails fail somewhere else. The model quotes the invoice amount from message one after message three corrected it. It names the rival vendor you told it to keep out of the reply. It books the meeting in the room that changed on Tuesday. Those are thread-tracking errors, and a few regular expressions catch them in milliseconds. A [voice-matching test](https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/) measures how much a draft sounds like you. The harness below measures something blunter: whether the draft is safe to send.\n\nYou will build about 400 lines of Python with no third-party packages, six email scenarios with planted traps, and a scorer that returns a 0 to 100 grade per reply. A full run produces 36 scored emails and takes roughly 75 minutes to set up and verify. By my arithmetic the API bill for a live run stays well under one dollar at the November 2026 prices listed below. Every code block was executed before publication, and the output samples come from the built-in offline mode, so nothing here pretends to be a live benchmark result.\n\nDon't miss new tech stories on Google\n\nAdd Shattered once in the Google app and our stories show up in your news suggestions.\n\n[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)\n\n## What the ChatGPT vs Claude Email Test Measures\n\nEach scenario hands the model a short thread plus a task, for example reply to the customer who is waiting on a payment date. The thread always contains one superseded detail and one current detail. The scorer checks that the current facts appear, that the stale or forbidden content does not, and that the reply respects a word cap. It also logs style signals such as filler openers, hedging, and stock sign-offs.\n\nThe scoring weights add up to 100 points. They are my own choices, built around the cost of a bad email. A wrong amount or a leaked competitor costs more than a clumsy opener, so facts and leaks carry 60 of the 100 points.\n\n| Check | How it is scored | Points | \n|---|---|---|\n| Required current facts | Share of the scenario’s required regex patterns found in the reply | 40 | \n| Stale or forbidden content | Minus 10 per forbidden pattern hit, floor of zero | 20 | \n| Invented numbers | Minus 5 per digit string not present in the thread or task | 10 | \n| Word cap | Full points when the reply stays at or under the cap | 10 | \n| Ask placement | Request appears in the first 60% of words: 10. Later: 5. Missing: 0. Scenarios with no ask get 10 | 10 | \n| Filler opener | Zero when the first 30 words contain a stock opener | 5 | \n| Hedge density | 5 at one hedge or fewer per 100 words, 2.5 up to 2.5 per 100, otherwise 0 | 5 | \n\nThree more signals get logged but never change the score: stock closers such as best regards, the em dash count, and the raw word count. They exist so you can spot patterns later, for instance if one vendor reaches for em dashes in every second reply.\n\nThe test deliberately skips warmth, humor, and brand voice. Software cannot grade those, and pretending otherwise produces fake precision. Step 12 covers the human side with an edit-rate measurement. If your priority is house rules such as banned phrases and reading level, the [writing rules compliance test](https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/) is the better starting point, and this harness complements it by focusing on facts rather than rules.\n\n## Prerequisites and Versions for the Test\n\n- **Python 3.11 or newer.** The harness uses only the standard library (urllib, json, csv, difflib, unittest), so there is no pip install step and no dependency to pin.\n- **An OpenAI API key with billing enabled.** OpenAI states that API usage is billed separately from ChatGPT subscriptions, so a Plus or Pro plan does not cover this.\n- **An Anthropic API key** created in the Claude Platform, again separate from a Claude chat subscription.\n- **A terminal.** The shell snippets use bash syntax. PowerShell equivalents appear in step 2.\n- **The column utility** (optional) for reading the CSV in step 11. Any spreadsheet works instead.\n- **About 75 minutes** and roughly 500 lines of copy-paste, including the scenario file.\n\nYou do not need keys to finish the first nine steps. The offline mode swaps both vendors for two fake providers, one that writes ideal replies and one that writes deliberately bad ones. That lets you prove the scorer works before spending a cent, and it gives you the exact output shown in this tutorial.\n\nBudget the live run before you start. Count your calls, estimate the tokens each one sends and receives, and price them at the Opus 5.5 rate of $4 and $20 per million tokens. Fable 5.2 is priced at $8 and $40 per million tokens, so the same run costs noticeably more. Treat your token counts as estimates, and check your provider dashboard after the first run.\n\n## Which Claude and ChatGPT Models to Pin in November 2026\n\nModel names changed fast this autumn, and a comparison that pins last quarter’s models measures nothing useful. The harness therefore reads model IDs from environment variables and records them in every CSV row. The current lineups, as of November 18, 2026, look like this.\n\n### Claude lineup\n\nAnthropic’s current model comparison lists Claude Haiku 5.5, Claude Sonnet 5.5, Claude Opus 5.5, and Claude Fable 5.2. Sonnet 5.5 arrived September 28 and Haiku 5.5 on October 7. Anthropic says [Opus 5.5 is the default model in Claude on Pro, Max, Ultra, and Team plans](https://claude.com/resources/articles/using-claude-opus-5-5-as-your-daily-driver-for-knowledge-work) and recommends it for documents, research, and finished files. For an email test, Opus 5.5 is therefore the like-for-like Claude, because it is what most subscribers actually get when they open a chat.\n\nClaude Fable 5.2 and Claude Mythos 5.2 launched together, as the [Anthropic announcements page](https://www.anthropic.com/news/announcements?trk=article-ssr-frontend-pulse_little-text-block) lists. Fable 5.2 costs $8 per million input tokens and $40 per million output tokens. Anthropic’s current material says Fable 5.2 generally sits outside Pro, Max, Ultra, and standard Team usage limits except during launch offers, so subscribers reach it through usage credits. Treat it as the premium challenger rather than the default.\n\n### ChatGPT lineup\n\nOpenAI lists Free, Go, Plus, Business, Enterprise, and three Pro tiers on its [ChatGPT pricing page](https://chatgpt.com/pricing/). Plus costs $20 per month in the United States, and Pro comes as Pro 100, Pro 200, and Pro 500 at $100, $200, and $500 per month. On the API side, OpenAI documents gpt-6-sol and gpt-6-luna, with Sol at $2 and $10 and Luna at $0.10 and $0.50 per million input and output tokens after the price cuts announced November 17. For a deeper look at what each plan includes, see the [ChatGPT vs Claude plan and usage-limit breakdown](https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/).\n\nTwo cautions apply. First, OpenAI’s own pages disagree on which model label each consumer plan receives, and on whether the newest Sol and Luna releases already sit inside ChatGPT. Do not trust a plan page to tell you what answered your prompt. Log the model ID you pinned, which is exactly what the harness does. Second, retired models still show up in old tutorials. OpenAI retired o3 from ChatGPT on August 26, 2026, and GPT-4.5 has been retired as well, so neither belongs in a November comparison.\n\n| Model | Vendor | API ID | Input per 1M tokens | Output per 1M tokens | \n|---|---|---|---|---|\n| Claude Opus 5.5 | Anthropic | Copy from the models list in step 4 | $4 | $20 | \n| Claude Fable 5.2 | Anthropic | claude-fable-5-2 | $8 | $40 | \n| GPT-6 Sol | OpenAI | gpt-6-sol | $2 | $10 | \n| GPT-6 Luna | OpenAI | gpt-6-luna | $0.10 | $0.50 | \n\nA sensible first pairing is Opus 5.5 against GPT-6 Sol, the two models most paying users meet by default. Run the premium pairing, Fable 5.2 against OpenAI’s newest Astra model, as a second pass. If you later want to translate scores into money, the [cost per accepted 1,000 words harness](https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/) shows how to divide spend by usable output instead of by tokens.\n\n## Steps 1 to 4: Project Folder, Keys, Provider Layer, and Model IDs\n\n**Step 1** creates the project folder and confirms the Python version. **Step 2** exports the two API keys for the current shell session only, so nothing lands in a file you might commit.\n\n```\nmkdir emailtest\ncd emailtest\npython3 --version\n\n# bash or zsh\nexport OPENAI_API_KEY=\"paste-your-openai-key\"\nexport ANTHROPIC_API_KEY=\"paste-your-anthropic-key\"\n\n# PowerShell equivalent\n# $env:OPENAI_API_KEY = \"paste-your-openai-key\"\n# $env:ANTHROPIC_API_KEY = \"paste-your-anthropic-key\"\n```\n\nThe version command should print 3.11 or higher. If it prints an older version, install a newer interpreter before going on.\n\n**Step 3** writes `providers.py`, the only file that talks to the network. It sends the same system prompt and the same user prompt to both vendors, calls each REST endpoint with plain urllib, retries on rate-limit and overload codes, and strips a stray subject line if a model adds one. It deliberately sends no temperature and no token limit for OpenAI. Some current models reject sampling parameters or renamed token-limit fields, and leaving them out keeps the request valid on every model. Anthropic requires `max_tokens`, so that one stays.\n\n``` python\nimport json\nimport os\nimport time\nimport urllib.error\nimport urllib.request\n\nSYSTEM = (\n    \"You write business emails. Output only the email body, starting with the greeting. \"\n    \"No subject line and no commentary before or after the email.\"\n)\n\ndef clean(text):\n    lines = text.strip().splitlines()\n    if lines and lines[0].lower().startswith(\"subject:\"):\n        lines = lines[1:]\n    return \"\\n\".join(lines).strip()\n\ndef build_prompt(scenario):\n    facts = \"\\n\".join(\"- \" + line for line in scenario[\"facts\"])\n    return \"Source material:\\n\" + facts + \"\\n\\nTask: \" + scenario[\"task\"]\n\nRETRY_CODES = (429, 500, 502, 503, 529)\n\ndef _request(url, headers, payload=None):\n    data = json.dumps(payload).encode(\"utf-8\") if payload is not None else None\n    for attempt in range(4):\n        req = urllib.request.Request(url, data=data, headers=headers)\n        try:\n            with urllib.request.urlopen(req, timeout=180) as resp:\n                return json.load(resp)\n        except urllib.error.HTTPError as err:\n            if err.code not in RETRY_CODES or attempt == 3:\n                raise RuntimeError(\"HTTP %d from %s: %s\" % (err.code, url, err.read().decode(\"utf-8\", \"replace\")))\n            time.sleep(2 ** attempt)\n\ndef _openai_headers():\n    return {\n        \"Authorization\": \"Bearer \" + os.environ[\"OPENAI_API_KEY\"],\n        \"Content-Type\": \"application/json\",\n    }\n\ndef _anthropic_headers():\n    return {\n        \"x-api-key\": os.environ[\"ANTHROPIC_API_KEY\"],\n        \"anthropic-version\": \"2023-06-01\",\n        \"Content-Type\": \"application/json\",\n    }\n\ndef call_openai(scenario):\n    body = _request(\n        \"https://api.openai.com/v1/chat/completions\",\n        _openai_headers(),\n        {\n            \"model\": os.environ[\"OPENAI_MODEL\"],\n            \"messages\": [\n                {\"role\": \"system\", \"content\": SYSTEM},\n                {\"role\": \"user\", \"content\": build_prompt(scenario)},\n            ],\n        },\n    )\n    return clean(body[\"choices\"][0][\"message\"][\"content\"])\n\ndef call_anthropic(scenario):\n    body = _request(\n        \"https://api.anthropic.com/v1/messages\",\n        _anthropic_headers(),\n        {\n            \"model\": os.environ[\"ANTHROPIC_MODEL\"],\n            \"max_tokens\": 1024,\n            \"system\": SYSTEM,\n            \"messages\": [{\"role\": \"user\", \"content\": build_prompt(scenario)}],\n        },\n    )\n    parts = [b[\"text\"] for b in body[\"content\"] if b.get(\"type\") == \"text\"]\n    return clean(\"\".join(parts))\n\ndef call_fake_good(scenario):\n    return scenario[\"fake_good\"]\n\ndef call_fake_bad(scenario):\n    return scenario[\"fake_bad\"]\n\nLIVE = {\"openai\": call_openai, \"anthropic\": call_anthropic}\nFAKE = {\"fake-good\": call_fake_good, \"fake-bad\": call_fake_bad}\n\ndef list_models():\n    out = {}\n    openai_body = _request(\"https://api.openai.com/v1/models\", _openai_headers())\n    out[\"openai\"] = sorted(m[\"id\"] for m in openai_body[\"data\"])\n    anth_body = _request(\"https://api.anthropic.com/v1/models?limit=1000\", _anthropic_headers())\n    out[\"anthropic\"] = sorted(m[\"id\"] for m in anth_body[\"data\"])\n    return out\n```\n\nThe two fake providers near the bottom return the `fake_good` and `fake_bad` strings stored in each scenario. They are what make the offline mode possible.\n\n**Step 4** lists the model IDs your keys can reach. Run the one-liner and copy the IDs you want. The Anthropic request asks for up to 1,000 results because the endpoint pages its answers by default.\n\n``` python\npython3 -c \"import json, providers; print(json.dumps(providers.list_models(), indent=1))\"\n```\n\nExpect two long JSON lists, abbreviated here. Your account’s list will differ, and the exact IDs are the only thing you need from it.\n\n```\n{\n \"openai\": [\n  ...\n  \"gpt-6-luna\",\n  \"gpt-6-sol\",\n  ...\n ],\n \"anthropic\": [\n  ...\n  \"claude-fable-5-2\",\n  ...\n ]\n}\n```\n\nFind the Opus 5.5 entry in the Anthropic list and pin both models. Write the IDs down next to the run date, since a vendor can repoint an alias without warning.\n\n```\nexport OPENAI_MODEL=gpt-6-sol\nexport ANTHROPIC_MODEL=PASTE_THE_OPUS_5_5_ID_FROM_THE_LIST\n```\n\n## Step 5: Write Six Email Scenarios With Planted Traps\n\n**Step 5** creates `scenarios.json`. Each entry holds the thread (` facts`), the instruction (` task`), a word cap, whether the email should contain a request, the regex patterns the reply must match, the patterns it must avoid, and the two offline replies.\n\nThe traps are the point. The invoice thread contains an old amount that a credit later changed. The meeting thread moves a design review from Tuesday in Room 4B to Thursday in Room 7A. The vendor thread includes an internal note that names a competing quote the reply must never reveal. The outage thread carries a wrong early estimate of two hours and a credit tier that applied to a different incident. The raise request tempts a model into leverage language the employee did not want. The cold email supplies one real result and invites invented proof points. The idea echoes a [source-grounded writing audit](https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/), except the sources here are email threads and the failure you hunt is a stale fact.\n\n```\n[\n  {\n    \"id\": \"invoice_chase\",\n    \"facts\": [\n      \"Oct 3, Dana to Priya: Invoice 1182 for $4,800 is due Oct 15.\",\n      \"Oct 9, Priya to Dana: We spotted a duplicate line. Please reissue.\",\n      \"Oct 10, Dana to Priya: Reissued as 1182-R for $4,350 after the $450 credit. Same Oct 15 due date.\",\n      \"Nov 6, Priya to Dana: Sorry for the delay. Finance approved 1182-R. Payment goes out Nov 20.\"\n    ],\n    \"task\": \"Write Dana Ortiz's reply to Priya Nair. Goal: lock in the payment date. Maximum 120 words.\",\n    \"cap\": 120,\n    \"ask\": true,\n    \"must\": [\"1182-R\", \"\\\\$4,350\", \"Nov(ember)? 20\"],\n    \"forbidden\": [\"\\\\$4,800\"],\n    \"fake_good\": \"Hi Priya,\\n\\nThanks for the update. Can you confirm that 1182-R for $4,350 will land on Nov 20? I will mark it as scheduled on our side once I hear back.\\n\\nDana\",\n    \"fake_bad\": \"Hi Priya,\\n\\nI hope this email finds you well. I just wanted to reach out about invoice 1182 for $4,800, which I think was due a while ago. Maybe you could possibly let me know when payment will arrive? It might be around 30 days.\\n\\nBest regards,\\nDana\"\n  },\n  {\n    \"id\": \"meeting_move\",\n    \"facts\": [\n      \"Mon, Marcus to team: Design review is Tuesday at 2:00 pm in Room 4B.\",\n      \"Tue, Marcus to team: Moving it. New slot is Thursday at 10:30 am in Room 7A. Same agenda.\",\n      \"Tue, Lena to Marcus: Can you tell the Halvorsen folks? They only got the first invite.\"\n    ],\n    \"task\": \"Write Marcus Bell's message to Tomas Eriksen at Halvorsen. Maximum 90 words.\",\n    \"cap\": 90,\n    \"ask\": true,\n    \"must\": [\"Thursday\", \"10:30\", \"7A\"],\n    \"forbidden\": [\"2:00\", \"2 pm\", \"4B\"],\n    \"fake_good\": \"Hi Tomas,\\n\\nThe design review moved to Thursday at 10:30 am in Room 7A. The agenda is unchanged. Please confirm you can still join.\\n\\nMarcus\",\n    \"fake_bad\": \"Hello Tomas,\\n\\nA quick reminder that the design review is on Tuesday at 2:00 pm in Room 4B. We are also thinking of 11 am on Friday as a backup. Let me know what works.\\n\\nMarcus\"\n  },\n  {\n    \"id\": \"vendor_decline\",\n    \"facts\": [\n      \"Sep 18, Harbor Cloud to Ines: Proposal attached, $14,500 per year for the analytics add-on.\",\n      \"Sep 22, Ines to Jonas (internal): Northwind quoted $9,500 for similar scope. Do not mention them to Harbor.\",\n      \"Oct 30, Jonas to Ines: Budget is frozen until Q2. We pass for now, but keep the door open.\"\n    ],\n    \"task\": \"Write Ines Duarte's reply to Wen Zhao at Harbor Cloud, declining for now. Maximum 100 words.\",\n    \"cap\": 100,\n    \"ask\": false,\n    \"must\": [\"Q2\", \"(not moving forward|won't be moving forward|will not be moving forward|decline|pass on|not proceed|unable to proceed)\"],\n    \"forbidden\": [\"Northwind\", \"9,500\"],\n    \"fake_good\": \"Hi Wen,\\n\\nThank you for the proposal. We are not moving forward right now because our budget is frozen until Q2. We would like to stay in touch and revisit it then.\\n\\nInes\",\n    \"fake_bad\": \"Hi Wen,\\n\\nThank you so much for the proposal. Another vendor, Northwind, quoted $9,500 for similar scope, so we will pass. Maybe talk again at some point.\\n\\nInes\"\n  },\n  {\n    \"id\": \"outage_apology\",\n    \"facts\": [\n      \"Nov 12, status page: Elevated errors, investigating. Early estimate: up to 2 hours to restore.\",\n      \"Nov 12, ops: Resolved after 41 minutes. Root cause was an expired certificate on the login service.\",\n      \"Nov 13, support lead to Amy: Customers get a 15% credit on the November invoice. No 20% tier, that was for the August incident.\"\n    ],\n    \"task\": \"Write the apology email Amy Cho sends to affected customers. Maximum 140 words.\",\n    \"cap\": 140,\n    \"ask\": false,\n    \"must\": [\"41 minutes\", \"15%\", \"certificate\"],\n    \"forbidden\": [\"2 hours\", \"two hours\", \"20%\"],\n    \"fake_good\": \"Hi all,\\n\\nOn Nov 12 our login service failed for 41 minutes because an expired certificate went unnoticed. That is our mistake. We have added expiry alerts so it cannot repeat. Your November invoice will carry a 15% credit automatically.\\n\\nAmy\",\n    \"fake_bad\": \"Dear customer,\\n\\nWe sincerely apologize for the inconvenience. Our outage lasted about 2 hours and we are offering a 20% credit. We take reliability very seriously and 99.99% uptime remains our goal.\\n\\nAmy\"\n  },\n  {\n    \"id\": \"raise_request\",\n    \"facts\": [\n      \"Sam's notes: Joined 3 years ago. Led the billing migration that cut cloud spend by $38,000 a year.\",\n      \"Sam's notes: The salary band review closes Dec 5. Sam wants a 20-minute conversation, not a negotiation by email.\",\n      \"Rhea Kline's last message: Happy to talk. Send me your points before the review closes.\"\n    ],\n    \"task\": \"Write Sam Whitaker's email to his manager Rhea Kline. Maximum 150 words.\",\n    \"cap\": 150,\n    \"ask\": true,\n    \"must\": [\"\\\\$38,000\", \"Dec(ember)? 5\", \"20[- ]minute\"],\n    \"forbidden\": [\"quit\", \"resign\", \"other offers?\", \"competing offer\"],\n    \"fake_good\": \"Hi Rhea,\\n\\nBefore the review closes on Dec 5, here are my points. In 3 years I led the billing migration, which cut cloud spend by $38,000 a year. Could we set up a 20-minute conversation this week to talk about my band?\\n\\nSam\",\n    \"fake_bad\": \"Hi Rhea,\\n\\nI hope you are doing well. I was wondering if maybe we could discuss my compensation. I have been here 3 years and have taken on 40% more work. I have other offers, so I would like to understand my options.\\n\\nBest,\\nSam\"\n  },\n  {\n    \"id\": \"cold_outreach\",\n    \"facts\": [\n      \"Prospect: Imani Brooks, head of finance ops at Calder Freight.\",\n      \"Calder Freight launched Atlas Reports last month.\",\n      \"Our tool, Tally Lens, cut report generation from 14 minutes to 3 minutes in a customer pilot.\",\n      \"One 15-minute demo slot is open on Wed Nov 25.\"\n    ],\n    \"task\": \"Write a cold email from Theo Marsh to Imani Brooks. Maximum 110 words.\",\n    \"cap\": 110,\n    \"ask\": true,\n    \"must\": [\"Atlas Reports\", \"14 minutes\", \"3 minutes\", \"Nov(ember)? 25\"],\n    \"forbidden\": [\"trusted by\", \"industry-leading\", \"world-class\", \"thousands of\"],\n    \"fake_good\": \"Hi Imani,\\n\\nCongratulations on launching Atlas Reports last month. In a customer pilot, Tally Lens cut report generation from 14 minutes to 3 minutes. I have a 15-minute demo slot on Nov 25. Would that work for you?\\n\\nTheo\",\n    \"fake_bad\": \"Hi Imani,\\n\\nI wanted to reach out because Tally Lens is trusted by thousands of teams and cuts reporting time by 80%. Our industry-leading platform could really help Calder. Would you be open to a quick chat?\\n\\nWarm regards,\\nTheo\"\n  }\n]\n```\n\nNotice that no prompt warns the model about the traps. If you add a sentence like watch out for outdated numbers, you stop testing thread tracking and start testing instruction following. Keep the task text exactly as written for both vendors.\n\nThe regex patterns are intentionally simple. The JSON string `\\\\$4,350` matches the literal amount with its comma, which suits these prompts because the source text uses commas. If your own threads format amounts differently, widen the pattern, for example to allow an optional comma.\n\n## Steps 6 and 7: The Scorer and Its Unit Tests\n\n**Step 6** writes `score.py`. The `evaluate` function normalizes curly quotes first, because models emit typographic apostrophes and a pattern like `won't be moving forward` would otherwise miss the curly version. It then counts required hits, forbidden hits, and digit strings that appear in the reply but nowhere in the source, and converts everything to the 100-point score from the table above.\n\n``` python\nimport re\n\nFILLER = [\n    r\"hope this (email|message) finds you\",\n    r\"hope you('| a)re (doing )?well\",\n    r\"i (just )?wanted to (reach out|touch base|follow up)\",\n    r\"i am writing to\",\n    r\"i('m| am) reaching out\",\n]\nHEDGES = [\n    r\"\\bjust\\b\", r\"\\bmaybe\\b\", r\"\\bperhaps\\b\", r\"\\bpossibly\\b\", r\"\\bi think\\b\",\n    r\"\\bsort of\\b\", r\"\\bkind of\\b\", r\"\\ba bit\\b\", r\"\\bsomewhat\\b\", r\"\\bi was wondering\\b\",\n    r\"\\bmight\\b\",\n]\nSTOCK_CLOSERS = [r\"best regards\", r\"warm regards\", r\"kind regards\", r\"looking forward to hearing\"]\nASK_MARKERS = r\"\\?|\\bplease\\b|\\bcould you\\b|\\bcan you\\b|\\bwould you\\b|\\blet me know\\b|\\bconfirm\\b\"\nEM_DASH = \", \"\n\ndef words(text):\n    return re.findall(r\"\\S+\", text)\n\ndef numbers(text):\n    found = set()\n    for token in re.findall(r\"\\d[\\d,]*(?:\\.\\d+)?\", text):\n        found.add(token.rstrip(\",\").replace(\",\", \"\"))\n    return found\n\ndef count_hits(patterns, text):\n    return sum(1 for p in patterns if re.search(p, text, re.IGNORECASE))\n\ndef ask_position(text):\n    match = re.search(ASK_MARKERS, text, re.IGNORECASE)\n    if not match:\n        return None\n    before = len(words(text[: match.start()]))\n    return before / max(1, len(words(text)))\n\ndef normalize(text):\n    return text.replace(\"\\u2019\", \"'\").replace(\"\\u2018\", \"'\").replace(\"\\u201c\", '\"').replace(\"\\u201d\", '\"')\n\ndef evaluate(scenario, reply):\n    reply = normalize(reply)\n    low = reply.lower()\n    n_words = len(words(reply))\n    must_hits = sum(1 for p in scenario[\"must\"] if re.search(p, reply, re.IGNORECASE))\n    must_ratio = must_hits / len(scenario[\"must\"])\n    forbidden = count_hits(scenario[\"forbidden\"], reply)\n    allowed = numbers(\" \".join(scenario[\"facts\"]) + \" \" + scenario[\"task\"])\n    invented = len(numbers(reply) - allowed)\n    cap_ok = n_words <= scenario[\"cap\"]\n    first_chunk = \" \".join(low.split()[:30])\n    filler = count_hits(FILLER, first_chunk) > 0\n    hedge_density = 100.0 * count_hits(HEDGES, low) / max(1, n_words)\n    pos = ask_position(reply)\n\n    if scenario[\"ask\"]:\n        if pos is None:\n            ask_points = 0.0\n        elif pos <= 0.6:\n            ask_points = 10.0\n        else:\n            ask_points = 5.0\n    else:\n        ask_points = 10.0\n\n    if hedge_density <= 1.0:\n        hedge_points = 5.0\n    elif hedge_density <= 2.5:\n        hedge_points = 2.5\n    else:\n        hedge_points = 0.0\n\n    score = (\n        40.0 * must_ratio\n        + max(0.0, 20.0 - 10.0 * forbidden)\n        + max(0.0, 10.0 - 5.0 * invented)\n        + (10.0 if cap_ok else 0.0)\n        + ask_points\n        + (0.0 if filler else 5.0)\n        + hedge_points\n    )\n    return {\n        \"score\": round(score, 1),\n        \"words\": n_words,\n        \"cap_ok\": cap_ok,\n        \"must_ratio\": round(must_ratio, 2),\n        \"forbidden\": forbidden,\n        \"invented\": invented,\n        \"ask_pos\": None if pos is None else round(pos, 2),\n        \"filler\": filler,\n        \"hedge_density\": round(hedge_density, 2),\n        \"stock_closer\": count_hits(STOCK_CLOSERS, low) > 0,\n        \"em_dashes\": reply.count(EM_DASH),\n    }\n```\n\nThe invented-number check works because the allowed set comes from the thread and the task. Commas are stripped, so `$4,350` and `4350` compare equal. A reply that writes 80% when no source mentions 80 gets flagged, which is exactly the cold-email failure mode.\n\n**Step 7** locks the behavior in place with unit tests. The first test asserts that the ideal reply beats the bad reply by more than 40 points in every scenario. The others pin the stale-amount flag, the invented-number flag, the word cap, the curly-apostrophe fix, the number normalizer, and the subject-line stripper.\n\n``` python\nimport json\nimport unittest\n\nimport providers\nimport score\n\nclass ScoreTests(unittest.TestCase):\n    def setUp(self):\n        with open(\"scenarios.json\", encoding=\"utf-8\") as fh:\n            self.scenarios = {s[\"id\"]: s for s in json.load(fh)}\n\n    def test_good_reply_beats_bad_reply_everywhere(self):\n        for s in self.scenarios.values():\n            good = score.evaluate(s, s[\"fake_good\"])[\"score\"]\n            bad = score.evaluate(s, s[\"fake_bad\"])[\"score\"]\n            self.assertGreater(good, bad + 40, s[\"id\"])\n\n    def test_stale_amount_is_flagged(self):\n        s = self.scenarios[\"invoice_chase\"]\n        result = score.evaluate(s, \"Hi Priya, please confirm $4,800 on Nov 20 for 1182-R.\")\n        self.assertEqual(result[\"forbidden\"], 1)\n\n    def test_invented_number_is_flagged(self):\n        s = self.scenarios[\"cold_outreach\"]\n        result = score.evaluate(s, \"Hi Imani, we cut reporting time by 80%.\")\n        self.assertEqual(result[\"invented\"], 1)\n\n    def test_word_cap(self):\n        s = self.scenarios[\"meeting_move\"]\n        result = score.evaluate(s, \"word \" * 200)\n        self.assertFalse(result[\"cap_ok\"])\n\n    def test_curly_apostrophe_still_matches(self):\n        s = self.scenarios[\"vendor_decline\"]\n        result = score.evaluate(s, \"Hi Wen, we won\\u2019t be moving forward before Q2.\")\n        self.assertEqual(result[\"must_ratio\"], 1.0)\n\n    def test_numbers_normalizes_commas(self):\n        self.assertEqual(score.numbers(\"$4,350 and 1182-R.\"), {\"4350\", \"1182\"})\n\n    def test_clean_drops_subject_line(self):\n        self.assertEqual(providers.clean(\"Subject: Hello\\n\\nHi Wen,\\nThanks.\"), \"Hi Wen,\\nThanks.\")\n\nif __name__ == \"__main__\":\n    unittest.main()\npython3 -m unittest -v\n\ntest_clean_drops_subject_line (test_score.ScoreTests.test_clean_drops_subject_line) ... ok\ntest_curly_apostrophe_still_matches (test_score.ScoreTests.test_curly_apostrophe_still_matches) ... ok\ntest_good_reply_beats_bad_reply_everywhere (test_score.ScoreTests.test_good_reply_beats_bad_reply_everywhere) ... ok\ntest_invented_number_is_flagged (test_score.ScoreTests.test_invented_number_is_flagged) ... ok\ntest_numbers_normalizes_commas (test_score.ScoreTests.test_numbers_normalizes_commas) ... ok\ntest_stale_amount_is_flagged (test_score.ScoreTests.test_stale_amount_is_flagged) ... ok\ntest_word_cap (test_score.ScoreTests.test_word_cap) ... ok\n\n----------------------------------------------------------------------\nRan 7 tests in 0.005s\n\nOK\n```\n\nSeven passing tests mean the scorer behaves. Any later change to a regex or a weight should rerun this command first. A scorer you cannot trust is worse than no scorer, because it hands you confident wrong answers.\n\n## Steps 8 and 9: The Runner and the Offline Dry Run\n\n**Step 8** writes `run.py`, the command-line entry point. It has four commands. `models` prints the ID lists. `run` generates replies and scores them. `rescore` rescans the `out/` folder without calling any API, which is the command to use after you tweak a regex. `report` prints the summary table. Every reply lands in a file named after its scenario and run number inside a folder named after the provider, so you can read it in a plain editor.\n\n``` python\nimport argparse\nimport csv\nimport glob\nimport json\nimport os\nimport sys\nfrom collections import defaultdict\nfrom statistics import mean\n\nimport providers\nimport score\n\nFIELDS = [\"provider\", \"model\", \"scenario\", \"run\", \"score\", \"words\", \"cap_ok\",\n          \"must_ratio\", \"forbidden\", \"invented\", \"ask_pos\", \"filler\",\n          \"hedge_density\", \"stock_closer\", \"em_dashes\"]\nNOISE = [\"Thanks, noted.\", \"Sounds good.\", \"Looping in finance.\", \"Checking with the team.\"]\n\ndef load_scenarios():\n    with open(\"scenarios.json\", encoding=\"utf-8\") as fh:\n        return json.load(fh)\n\ndef pad(scenario, n):\n    noise = [\"Another message in the thread: \" + NOISE[i % len(NOISE)] for i in range(n)]\n    padded = dict(scenario)\n    padded[\"facts\"] = scenario[\"facts\"][:1] + noise + scenario[\"facts\"][1:]\n    return padded\n\ndef cmd_models(_args):\n    for vendor, ids in providers.list_models().items():\n        print(\"== \" + vendor)\n        for model_id in ids:\n            print(model_id)\n\ndef cmd_run(args):\n    scenarios = load_scenarios()\n    table = providers.FAKE if args.dry_run else providers.LIVE\n    for name, fn in table.items():\n        folder = os.path.join(\"out\", name)\n        os.makedirs(folder, exist_ok=True)\n        model = os.environ.get(name.upper() + \"_MODEL\", \"offline\")\n        with open(os.path.join(folder, \"model.txt\"), \"w\", encoding=\"utf-8\") as fh:\n            fh.write(model + \"\\n\")\n        for scenario in scenarios:\n            used = pad(scenario, args.pad) if args.pad else scenario\n            for run in range(args.runs):\n                reply = fn(used)\n                path = os.path.join(folder, \"%s__%d.txt\" % (scenario[\"id\"], run))\n                with open(path, \"w\", encoding=\"utf-8\") as fh:\n                    fh.write(reply + \"\\n\")\n    score_folders(list(table))\n\ndef cmd_rescore(_args):\n    names = sorted(os.path.basename(p) for p in glob.glob(\"out/*\") if os.path.isdir(p))\n    score_folders(names)\n\ndef score_folders(names):\n    scenarios = {s[\"id\"]: s for s in load_scenarios()}\n    rows = []\n    for name in names:\n        model_file = os.path.join(\"out\", name, \"model.txt\")\n        model = open(model_file, encoding=\"utf-8\").read().strip() if os.path.exists(model_file) else \"unknown\"\n        for path in sorted(glob.glob(os.path.join(\"out\", name, \"*__*.txt\"))):\n            stem = os.path.basename(path)[:-4]\n            scenario_id, run = stem.rsplit(\"__\", 1)\n            if scenario_id not in scenarios:\n                continue\n            with open(path, encoding=\"utf-8\") as fh:\n                reply = fh.read()\n            row = {\"provider\": name, \"model\": model, \"scenario\": scenario_id, \"run\": int(run)}\n            row.update(score.evaluate(scenarios[scenario_id], reply))\n            rows.append(row)\n    with open(\"results.csv\", \"w\", newline=\"\", encoding=\"utf-8\") as fh:\n        writer = csv.DictWriter(fh, fieldnames=FIELDS)\n        writer.writeheader()\n        writer.writerows(rows)\n    print(\"wrote %d rows to results.csv\" % len(rows))\n\ndef cmd_report(_args):\n    by_provider = defaultdict(list)\n    by_cell = defaultdict(list)\n    with open(\"results.csv\", encoding=\"utf-8\") as fh:\n        for row in csv.DictReader(fh):\n            value = float(row[\"score\"])\n            by_provider[row[\"provider\"]].append(value)\n            by_cell[(row[\"scenario\"], row[\"provider\"])].append(value)\n    print(\"%-16s %6s %6s %6s\" % (\"provider\", \"mean\", \"min\", \"n\"))\n    for name, values in sorted(by_provider.items()):\n        print(\"%-16s %6.1f %6.1f %6d\" % (name, mean(values), min(values), len(values)))\n    print()\n    names = sorted(by_provider)\n    print(\"%-16s \" % \"scenario\" + \" \".join(\"%12s\" % n for n in names))\n    for scenario in sorted({k[0] for k in by_cell}):\n        cells = [\"%12.1f\" % mean(by_cell[(scenario, n)]) for n in names]\n        print(\"%-16s \" % scenario + \" \".join(cells))\n\ndef main():\n    parser = argparse.ArgumentParser()\n    sub = parser.add_subparsers(dest=\"cmd\", required=True)\n    sub.add_parser(\"models\").set_defaults(fn=cmd_models)\n    run = sub.add_parser(\"run\")\n    run.add_argument(\"--dry-run\", action=\"store_true\")\n    run.add_argument(\"--runs\", type=int, default=3)\n    run.add_argument(\"--pad\", type=int, default=0)\n    run.set_defaults(fn=cmd_run)\n    sub.add_parser(\"rescore\").set_defaults(fn=cmd_rescore)\n    sub.add_parser(\"report\").set_defaults(fn=cmd_report)\n    args = parser.parse_args()\n    args.fn(args)\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\nA `model.txt` file in each provider folder records the pinned model, and the CSV carries it in a `model` column. When a vendor ships an update mid-month, you can tell which rows came from which model.\n\n**Step 9** runs everything offline. The fake providers answer instantly, and the report proves the pipeline works from scenario to CSV to summary.\n\n```\npython3 run.py run --dry-run --runs 3\npython3 run.py report\n\nwrote 36 rows to results.csv\nprovider           mean    min      n\nfake-bad           30.0   25.0     18\nfake-good          97.5   95.0     18\n\nscenario             fake-bad    fake-good\ncold_outreach            25.0         95.0\ninvoice_chase            30.0        100.0\nmeeting_move             30.0         95.0\noutage_apology           35.0        100.0\nraise_request            25.0         95.0\nvendor_decline           35.0        100.0\n```\n\nThe ideal provider scores far above the bad one. Some ideal replies lose points because their request lands late in the email, which shows the ask-placement rule at work on perfectly reasonable writing. Decide whether that rule suits your own mail before you trust it. Cold emails and raise requests often end on the ask by design, and you may prefer to remove the penalty.\n\n## Steps 10 and 11: The Live Run and Reading the Report\n\n**Step 10** runs the real comparison. Confirm that both model variables are set, then run three passes per scenario. Three passes matter because a single sample hides variance, and a vendor can win one run and lose the next on identical input.\n\n```\necho $OPENAI_MODEL $ANTHROPIC_MODEL\npython3 run.py run --runs 3\npython3 run.py report\n```\n\nThe run prints `wrote 36 rows to results.csv` when two providers, six scenarios, and three passes complete. If it stops early with an HTTP error, jump to the troubleshooting section. The runner writes replies to disk as it goes, so a partial run still leaves readable files, and `rescore` can pick them up.\n\n**Step 11** reads the results at two levels. The summary table gives a mean, a minimum, and a sample count per provider, then a scenario-by-provider grid. Start with the minimum. A vendor with a 92 mean and a 45 minimum has a trap it falls into sometimes, and that single bad email is the one that reaches a customer. Then open the per-email flags to see why a score dropped.\n\n```\ncolumn -s, -t results.csv | head -20\n\n# offline sample rows (fake providers)\nprovider   model    scenario        run  score  words  cap_ok  must_ratio  forbidden  invented  ask_pos  filler  hedge_density  stock_closer  em_dashes\nfake-good  offline  invoice_chase   0    100.0  32     True    1.0         0          0         0.19     False   0.0            False         0\nfake-good  offline  invoice_chase   1    100.0  32     True    1.0         0          0         0.19     False   0.0            False         0\nfake-good  offline  invoice_chase   2    100.0  32     True    1.0         0          0         0.19     False   0.0            False         0\nfake-bad   offline  invoice_chase   0    30.0   48     True    0.0         1          1         0.67     True    10.42          True          0\n```\n\nRead the low-scoring replies themselves, not just the numbers. A forbidden hit of 1 on a vendor-decline reply means the model leaked the competitor’s name or quote. A `must_ratio` below 1 can mean a required fact is missing or worded in a way your regex did not anticipate, so open the file before blaming the model. For a human read of the same drafts, shuffle the files and use a [blind side-by-side rating](https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/) so you cannot tell which vendor wrote which email.\n\n## Step 12: Measure the Edit Rate on Emails You Would Send\n\n**Step 12** adds the one metric software cannot invent: how much you have to change a draft before it goes out. Read each reply, edit it into the version you would actually send, and save it in an `edits` folder that mirrors `out`, using the same provider folder and filename. The script compares original and edited text word by word with `difflib` and reports the share of words that changed.\n\n``` python\nimport difflib\nimport glob\nimport os\nfrom collections import defaultdict\nfrom statistics import mean\n\ndef edit_rate(original, edited):\n    a, b = original.split(), edited.split()\n    return 1.0 - difflib.SequenceMatcher(None, a, b).ratio()\n\ndef main():\n    rates = defaultdict(list)\n    for edited_path in glob.glob(os.path.join(\"edits\", \"*\", \"*.txt\")):\n        provider = os.path.basename(os.path.dirname(edited_path))\n        original_path = os.path.join(\"out\", provider, os.path.basename(edited_path))\n        if not os.path.exists(original_path):\n            print(\"skip (no original): \" + edited_path)\n            continue\n        with open(original_path, encoding=\"utf-8\") as fh:\n            original = fh.read()\n        with open(edited_path, encoding=\"utf-8\") as fh:\n            edited = fh.read()\n        rates[provider].append(edit_rate(original, edited))\n    for provider, values in sorted(rates.items()):\n        print(\"%-16s mean edit rate %.2f over %d emails\" % (provider, mean(values), len(values)))\n\nif __name__ == \"__main__\":\n    main()\n```\n\nTake the offline example. The bad invoice email loses its filler opener and its hedged request, and its edit rate shows how large a share of the words changed. Run the same measurement over your live drafts and the vendor with the lower number saves you typing, even if its raw score is slightly lower.\n\n```\npython3 edit_rate.py\n\nfake-bad         mean edit rate 0.18 over 1 emails\n```\n\nEdit only what you would really change. If you polish every comma, both vendors drift toward a high edit rate and the signal drowns. For a dedicated treatment of how faithfully each model preserves text it was told to leave alone, the [editing fidelity test](https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/) covers that angle.\n\n## Reading the Results: A Decision Rule You Can Defend\n\nNumbers need a rule, or you will read them to match what you already believed. The thresholds below are my rule of thumb for a 36-email sample, not an industry standard. Adjust them if your volume or stakes differ.\n\n| What you see | What it means | What to do | \n|---|---|---|\n| Mean scores within 5 points across 3 or more passes | Statistical tie on this task | Choose on price, tooling, or the ecosystem you already use | \n| Any forbidden hit on vendor_decline or raise_request | The model leaks private context or uses unwanted leverage | Keep a human reviewer on that workflow, or rewrite the prompt and retest | \n| Invented numbers on cold_outreach | The model fabricates proof points | Require fact review before any outbound email | \n| Minimum score under 60 for one vendor | A trap catches it intermittently | Test with more passes and pad the threads (advanced tips) | \n| Edit rate gap above 0.10 | One vendor’s drafts need clearly less rework | Weigh the gap above the raw score difference | \n\nWrite the decision down before you open the report, then see whether the data changes it. That small habit is the difference between a test and a confirmation. This harness also answers a narrow question about thread-grounded business email. It says nothing about fiction, long reports, or code. For wider context on how model families compare, browse the [AI and machine learning coverage](https://shattered.io/category/ai-machine-learning/).\n\nKeep in mind what a tie means in practice. Anthropic pitches Opus 5.5 at finished documents and research, and OpenAI announced new professional models on November 17. If your run shows a tie today, rerun it after the next model release rather than treating the result as permanent.\n\n## Common Pitfalls That Skew Email Comparisons\n\n1. **Different prompts per vendor.** Paste one system prompt and one user prompt into both providers. The harness enforces this through`build_prompt` , so do not fork it.\n2. **One run per scenario.** Sampling variance can swing a single score by 20 points or more. Use at least three passes and look at the minimum, not just the mean.\n3. **Trusting a plan page for the model name.** OpenAI’s pages disagree about plan labels and rollout status. Pin the ID in the API call and log it.\n4. **Comparing an API call with a chat app.** Apps add their own system prompts, memory, and tools. If you test the apps instead, paste the same prompt, save each reply to`out/chatgpt-app/` or`out/claude-app/` , and use`rescore` . Label the result as an app test.\n5. **Regexes that are too narrow or too broad.** A pattern that demands the exact dollar format will mark a correct reply as a miss. A pattern like`Q2` will pass any mention of the quarter, correct or not. Read ten replies before you trust a pattern.\n6. **Hinting at the trap.** Any hint in the task turns a thread-tracking test into an instruction-following test.\n7. **Rewarding length.** Without a word cap, the model that writes more tends to hit more required facts. The cap keeps the contest fair.\n8. **Pinning a retired model.** o3 left ChatGPT on August 26, 2026, and GPT-4.5 has been retired too. Old tutorials still reference them.\n9. **Pasting real customer data.** Redact names, amounts, and account numbers before you add your own threads, because every prompt goes to a third-party API.\n\nThe first and sixth pitfalls cause most bad comparisons. People tweak the prompt for the model they like, then credit the model for the lift. Freeze the task text, commit it, and change it only between full runs.\n\n## Troubleshooting: 10 Failures and Their Fixes\n\n1. **KeyError: ‘OPENAI_API_KEY’ or ‘ANTHROPIC_MODEL’.** The variable is not exported in this shell. Export it again, since new terminal tabs start clean.\n2. **An authentication error from either vendor.** The key is wrong, revoked, or pasted with a trailing space. Re-export it with quotes and retry the model listing from step 4.\n3. **A not-found error or a model-not-found message.** The ID has a typo or your account cannot reach that model. Copy it from the step 4 list rather than typing it.\n4. **HTTP 400 mentioning an unsupported parameter.** This usually appears after you add temperature or a token limit. Remove the field, or check the current API reference for the parameter name that model expects.\n5. **HTTP 400 from Anthropic about max_tokens.** The field is required on that endpoint. The harness sets it to 1024, so a missing value means you edited the payload.\n6. **A rate-limit or overloaded-server error after a few calls.** You hit a rate limit or an overloaded server. The harness retries four times with growing pauses. If it still fails, lower`--runs` and rerun in a few minutes. Replies already written stay on disk.\n7. **Every score is 0 on required facts.** Your regexes do not match the reply format. Open one reply file and compare it to the pattern. Amounts without commas and spelled-out months are the usual culprits.\n8. **The reply contains a subject line or commentary.** The harness strips a leading`Subject:` line but not a chatty preface. Tighten the system prompt or extend`clean` in`providers.py` .\n9. **results.csv is missing or report crashes.** You ran`report` before`run` or`rescore` . Generate the file first.\n10. **A pattern misses because of a typographic character.** Add the character to`normalize` in`score.py` . Rerun`python3 run.py rescore` , which needs no API call.\n\nWhen a failure is not on this list, isolate it by running a single scenario with the fake providers, then a single live call. Narrowing a failure to the network layer or the scoring layer usually takes under five minutes.\n\n## Advanced Tips: Pad Threads, Add Your Own Mail, Schedule Runs\n\n**Pad the threads.** Real threads run longer than four messages. The `--pad` flag inserts neutral filler messages between the first message and the correction, so the stale fact sits further from the fix. Run `python3 run.py run --runs 3 --pad 8` and compare scores with the unpadded run. A vendor whose minimum drops sharply under padding loses track of corrections in long threads.\n\n**Add your own scenarios.** Redact ten real threads, copy the structure of an existing entry, and write the required and forbidden patterns from what you know the correct reply must say. Your own mail beats any synthetic scenario, because it contains the quirks your colleagues actually produce.\n\n**Test smaller and cheaper models.** Add Sonnet 5.5, Haiku 5.5, or GPT-6 Luna by adding a provider function that reads a different model variable. Luna’s listed price of $0.10 and $0.50 per million tokens makes it a cheap way to learn whether routine mail needs a flagship model at all.\n\n**Change one variable at a time.** Once you have a baseline, add a house-style paragraph to the system prompt for both vendors and rerun. The [custom instructions pack](https://shattered.io/chatgpt-custom-instructions-claude-projects-gemini-gems-one-spec-12-steps-2026/) shows how to keep one spec for ChatGPT and Claude so the two stay identical, and the [planted-flaw feedback test](https://shattered.io/claude-vs-chatgpt-draft-feedback-planted-flaw-test-12-steps-2026/) applies the same plant-and-detect idea to critique rather than drafting.\n\n**Schedule it.** Models update under stable names. A weekly cron job running `python3 run.py run --runs 3` and archiving each CSV turns this harness into a regression monitor. Because every row carries the model ID, a sudden score change lines up with the update that caused it.\n\n## The Complete Project: File Layout and One-Command Run\n\nEverything lives in one folder with six hand-written files. The three folders at the bottom appear as you work.\n\n```\nemailtest/\n  providers.py     network layer, fake providers, model listing\n  scenarios.json   six threads, traps, patterns, offline replies\n  score.py         100-point scorer\n  test_score.py    seven unit tests\n  run.py           models / run / rescore / report\n  edit_rate.py     word-level edit rate for your edited drafts\n  out/             replies per provider (created by run)\n  edits/           your edited versions (you create)\n  results.csv      one row per scored reply (created by run)\n```\n\nThe full cycle, from a clean folder to a decision, is the short sequence below. Run the first two lines any time you change the scorer, and the last two once your keys and model IDs are set.\n\n```\npython3 -m unittest\npython3 run.py run --dry-run --runs 3 && python3 run.py report\npython3 run.py run --runs 3 && python3 run.py report\npython3 edit_rate.py\n```\n\nKeep the `out/` directory out of version control if it holds sensitive replies. Commit `scenarios.json` and note the pinned model IDs, since those define the experiment.\n\n## FAQ: ChatGPT vs Claude for Writing Emails\n\n**Which is better for emails, ChatGPT or Claude?**\n\nNeither wins universally, and this tutorial does not claim a winner. The harness exists because the answer depends on your threads, your tone, and your tolerance for stale details. Anthropic positions Opus 5.5 for finished documents, and OpenAI positions its newest models for professional work, so run both on your own mail.\n\n**Can I run the test without API keys?**\n\nYes. The offline mode proves the scorer. To test the chat apps, paste each scenario into the app, save the replies as `out/chatgpt-app/invoice_chase__0.txt` and so on, then run `python3 run.py rescore` and `python3 run.py report`.\n\n**Does the offline result say anything about the real models?**\n\nNo. The fake providers return fixed text. They show that the scorer separates a good reply from a bad one and that the pipeline works end to end. Only a live run measures the vendors.\n\n**How many runs per scenario are enough?**\n\nThree is the practical minimum and five is better when scores sit close together. Examine the minimum score, because one bad email sent to a customer matters more than a good average.\n\n**Why not ask a third model to judge the emails?**\n\nA judge model adds cost and a bias of its own, and it may favor text that resembles its own style. Regex checks on facts are blunt but repeatable. Use a judge later for tone, and keep the regex layer for facts.\n\n**How do I add Gemini or another provider?**\n\nWrite one function in `providers.py` that takes a scenario and returns the cleaned reply text, add it to the `LIVE` dictionary, and set a matching `NAME_MODEL` variable. The runner and scorer need no changes.\n\n**Will these scores stay valid next month?**\n\nNot necessarily. Both vendors ship models every few weeks. Keep the model ID in every row, rerun after each release, and compare against your archived CSVs.", "url": "https://wpnews.pro/news/chatgpt-vs-claude-email-test-12-steps-75-min", "canonical_source": "https://shattered.io/chatgpt-vs-claude-email-reply-test-12-steps-2026/", "published_at": "2026-11-18 17:39:57+00:00", "updated_at": "2026-11-18 17:41:33.089251+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "generative-ai", "ai-products"], "entities": ["Anthropic", "Claude Fable 5.2", "OpenAI", "GPT-6.2 Astra", "GPT-6.2 Sol", "Shattered.io", "9to5Google"], "also_reported_by": [], "takeaway": "A regex-based scorer that weights current facts at 40 points and penalizes stale or forbidden content at 10 points per hit catches thread-tracking errors in email drafts that prose-quality leaderboards miss.", "recommendation": {"advice": "Score email drafts with regex checks on required current facts and forbidden stale details before judging prose; the harness grades 36 replies in about 75 minutes for under one dollar in API calls.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/chatgpt-vs-claude-email-test-12-steps-75-min", "markdown": "https://wpnews.pro/news/chatgpt-vs-claude-email-test-12-steps-75-min.md", "text": "https://wpnews.pro/news/chatgpt-vs-claude-email-test-12-steps-75-min.txt", "jsonld": "https://wpnews.pro/news/chatgpt-vs-claude-email-test-12-steps-75-min.jsonld"}}