Claude vs ChatGPT Summary Test: 12 Steps, 80 Min A 12-step, roughly 80-minute test harness published on Shattered scores Claude and ChatGPT summaries with plain Python and a hand-written fact manifest instead of a judge model, targeting four failure types: dropped numbers, invented numbers, lost negations and lost attribution. The harness caps summaries at 70 words, runs two prompts (bare and guarded) with five runs per cell, and calibrates the deterministic scorer against three control summaries. The test arrives after OpenAI introduced GPT-6.2 Astra and GPT-6.2 Sol on November 17, 2026, and Anthropic shipped Claude Fable 5.2 and Mythos 5.2 earlier in November, with Claude Opus 5.5 remaining the default on Pro, Max, Ultra and Team plans. OpenAI introduced GPT-6.2 Astra and GPT-6.2 Sol https://learn.chatgpt.com/docs/whats-new/gpt-62-astra-sol on November 17, 2026, and its own ChatGPT documentation now lists them as the newest models. Anthropic shipped Claude Fable 5.2 and Mythos 5.2 earlier in November, while Claude Opus 5.5 stays the default model on the Pro, Max, Ultra and Team plans. Any Claude vs ChatGPT https://shattered.io/claude-vs-chatgpt-draft-feedback-planted-flaw-test-12-steps-2026/ verdict written a month ago describes a matchup that no longer exists. Opinions about which assistant writes better age fast. A measurement ages slowly, provided you pin the model names and rerun it when either vendor ships. Most comparisons judge style, and style is easy to fake. Summaries fail in a quieter way: a number drifts, a name disappears, or did not approve turns into approved inside a perfectly fluent sentence. The harness below scores exactly those failures with plain Python and no judge model, so the same summary always earns the same score. It pairs naturally with the writing rules compliance test https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ , which checks house style while this one checks whether the facts survive compression. Plan on about 80 minutes. The API runs cost cents, and manual mode needs no keys at all. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Claude vs ChatGPT Summaries Need Their Own Test A summary earns trust because it is short, and nobody rereads the source afterward. That makes four failure types expensive. The first is a dropped number, such as a vote count or a deadline. The second is an invented number, where the model adds a plausible figure the source never contained. The third is a lost negation, the worst of the four, because the summary asserts the opposite of the source. The fourth is a lost attribution, where an estimate from a draft audit becomes a flat statement of fact. The water board source below shows the pattern. It says the board did not approve a $12 monthly surcharge. A careless summary writes that the board approved the surcharge, keeps the names and the dates, and reads smoothly. A human skimming for tone will not catch it. A script that finds the word surcharge and then looks for a negation cue within 90 characters will. Editing tasks measure something different. The editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ asks how much of a draft a model changes when told to leave most of it alone. Summarization forces loss, because a long text must shrink to a short one, so the useful question becomes which facts survive the cut. The design has five parts, each small enough to read in one sitting: - Short closed-source documents, written about fictional organizations so neither model can recall the facts from training. - A 70-word cap that creates real pressure to drop something. - Two prompts, a bare one and a guarded one that names the traps, so you can measure how much instruction helps each model. - Five runs per cell, because a single sample hides variance. - A deterministic scorer driven by a hand-written fact manifest, calibrated with three control summaries perfect, lazy and sabotaged . The scorer deliberately avoids a judge model. A judge has its own blind spots, may favor text that resembles its own output, and returns different scores on reruns. A regular expression returns the same number every time, and you can read the rule behind it. The tradeoff is a hand-written manifest, which takes about 15 minutes per source. A separate source-grounded writing audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ looks at citations in longer drafts, while this harness stays on short documents where every fact can be enumerated. Prerequisites and Pinned Versions Everything below was run on Python 3.12.3 on Linux. Python 3.11 or newer works because the code uses only the standard library plus two vendor SDKs. You need a terminal, a text editor, and one of two access paths. API mode needs an Anthropic API key and an OpenAI API key. Manual mode needs only a Claude account and a ChatGPT account, since you paste the prompts into the apps yourself. - Python: 3.11 or newer tested on 3.12.3 . - pip packages: the latest anthropic and openai releases. The harness calls the OpenAI Responses API, which every current SDK release includes. Model names move quickly, so pin them before you start. The table lists the state of both product lines on November 19, 2026, taken from the vendors’ own pages and release notes. | Item | State on November 19, 2026 | What to record in your run log | |---|---|---| | Claude default model Pro, Max, Ultra, Team | Claude Opus 5.5 | Exact API model ID from Anthropic’s model list | | Newest Claude releases | Claude Fable 5.2 and Claude Mythos 5.2, announced in November 2026 | Whether you test a newest-tier model or the default | | Claude small model | Claude Haiku 5.5, aimed at high-volume work such as summarization | Use as a cheap second Claude row | | Opus 5.5 API price | $4 per million input tokens, $20 per million output tokens | Price on your run date | | Haiku 5.5 API price | $0.10 input, $0.50 output per million tokens for prompts up to 100,000 tokens | Price on your run date | | Fable 5.2 API price | $8 input, $40 output per million tokens | Price on your run date | | Newest OpenAI models | GPT-6.2 Astra and GPT-6.2 Sol, introduced November 17, 2026 | The label your picker or API shows | | ChatGPT Plus | $20 per month | Plan used for manual mode | | ChatGPT Pro | Three tiers: $100, $200 and $500 per month Pro 500 added November 19 | Tier used, if any | Anthropic’s own knowledge work guide for Opus 5.5 https://claude.com/de/resources/articles/using-claude-opus-5-5-as-your-daily-driver-for-knowledge-work describes it as the default model in Claude on those four plans, so Opus 5.5 is the fair counterpart when you test the app rather than the API. OpenAI’s release notes present GPT-6.2 Astra and Sol as introduced for ChatGPT Work and Codex, which means the ordinary chat model picker may show a different name than the API does. Write down whatever label you actually see. GitHub’s changelog entry from November 17 gives a second confirmation that both names are live for developers. Plan prices and usage caps shift often enough that the ChatGPT vs Claude pricing and usage limits breakdown https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ is worth checking before you decide which tiers to test. For the API runs, the arithmetic is small. Each provider receives 30 calls 3 sources, 2 prompts, 5 runs . At roughly 250 input tokens and 100 output tokens per call, that is about 7,500 input tokens and 3,000 output tokens, which comes to under ten cents at Opus 5.5’s listed rates and a fraction of a cent on Haiku 5.5. Models that spend hidden reasoning tokens will cost more, but the order of magnitude holds. Set Up the Workspace Steps 1 and 2 Step 1: Create the project folder and virtual environment A virtual environment keeps the two SDKs away from your system Python and makes the setup reproducible on another machine. Run these commands from the directory where you keep projects. mkdir summary-faithfulness cd summary-faithfulness python3 -m venv .venv source .venv/bin/activate python --version pip install --upgrade pip anthropic openai On Windows, activate with .venv\Scripts\activate instead of the source line. The version check should print 3.11 or higher. Install a newer interpreter first if it does not. Step 2: Export your keys and pin the model IDs Both SDKs read their API keys from environment variables, so no key ever touches a source file. The two model variables are the part that matters for the experiment. Copy each exact model ID from the provider’s current model list rather than typing a name from memory, because an alias such as a family name can silently point at a newer release next month. export ANTHROPIC API KEY="paste-your-anthropic-key" export OPENAI API KEY="paste-your-openai-key" export CLAUDE MODEL="paste-exact-model-id-from-anthropic-docs" export OPENAI MODEL="paste-exact-model-id-from-openai-docs" The harness copies the value of each model variable into outputs/meta.jsonl beside every saved summary. When a result looks odd three weeks from now, that file tells you which model produced it. For a first pass, test the flagship you would actually use for writing on each side. Anthropic positions Opus 5.5 for knowledge work, and OpenAI lists GPT-6.2 Astra and Sol as its newest pair. A second pass with Haiku 5.5 on the Claude side is cheap and answers a practical question, since Anthropic aims that model at summarization volume. Plant the Facts: Sources and Fact Manifests Steps 3 and 4 Step 3: Choose sources that contain checkable traps A good test source is closed, short and booby-trapped. Closed means the summary can only use what is on the page, so any extra number is an invention you can detect. Short means 100 to 140 words, which keeps the manifest writable by hand. Booby-trapped means each document contains the specific facts that summarizers like to lose. Every source in this project carries at least five hard numbers, at least two statements of what did not happen or is not yet known, and at least one claim attributed to a secondary source such as a draft audit or a preliminary review. The organizations are fictional, so neither model can lean on training data. The three domains differ on purpose: a local utility vote, a clinic software outage, and a quarterly earnings note. A model that handles civic prose well may stumble on earnings language, and you want to see that gap. Voice and tone are a separate question that deserves a separate test, and the stylometry-based voice match https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ covers it. Nothing here asks which summary sounds nicer. The only question is whether the facts you planted survived. Each source gets a manifest with three parts. The numbers list holds one entry per critical figure, and each entry is a list of acceptable surface forms, because 6.5 percent and 6.5% are both correct. The entities list does the same for names. The guards list handles the dangerous facts. A guard has an anchor the words that show the fact appears in the summary and cues the words that show the qualifier survived . A guard passes only when a cue sits within 90 characters of an anchor. Step 4: Write cases.json Save the following as cases.json . Each case also stores a hand-written gold summary and a sabotaged one, which the calibration step uses. The gold summaries must fit the 70-word cap and must score exactly 1.0, and a unit test enforces both. { "id": "water-board", "max words": 70, "text": "The Harlow County Water Board voted 5 to 2 on Tuesday to raise residential rates by 6.5 percent starting January 1, 2027. The increase adds about $4.80 to the average monthly bill of $73.90. Board chair Marisol Vega said the money will fund the replacement of 11 miles of cast-iron pipe built in the 1950s. The board did not approve a separate proposal to add a $12 monthly infrastructure surcharge. According to a draft audit from the county comptroller, the utility lost an estimated 18 percent of treated water to leaks last year. Vega said the board will not raise rates again before 2029 unless costs rise sharply. Residents can comment at a public hearing on December 3 at the Lakeview Library, and low-income customers may apply for a bill credit through the county's Neighbor Aid program.", "gold": "Harlow County Water Board voted 5-2 to raise residential rates 6.5 percent from January 1, 2027, adding about $4.80 to the $73.90 average bill. Chair Marisol Vega said it funds 11 miles of pipe replacement. The board rejected a $12 surcharge and will not raise rates before 2029. A draft comptroller audit estimates 18 percent of water leaks. Hearing: December 3, Lakeview Library. Neighbor Aid offers credits.", "planted": "The Harlow County Water Board voted 5-2 to raise residential rates 6.5 percent and approved a $12 monthly surcharge. The county says 18 percent of water leaks. Rates may rise again in 2028. Chair Marisol Vega called the plan overdue.", "manifest": { "numbers": "5 to 2", "5-2", "five to two", "5:2" , "6.5 percent", "6.5%" , "$4.80", "4.80", "$4.8" , "11 miles", "11-mile", "11 mi" , "18 percent", "18%" , "2029" , "december 3", "dec. 3", "dec 3" , "entities": "harlow county water board", "harlow county board", "harlow water board", "water board", "harlow" , "marisol vega", "vega" , "comptroller" , "lakeview library", "lakeview" , "neighbor aid" , "guards": {"label": "surcharge rejected", "anchor": "surcharge" , "cues": "rejected", "not", "no", "declined", "did not", "didn't", "turned down", "voted down", "denied", "blocked", "scrapped", "dropped" }, {"label": "no rise before 2029", "anchor": "2029" , "cues": "not", "no", "until", "unless", "won't", "freeze", "hold", "before", "through", "pledged" }, {"label": "leak figure is a draft estimate", "anchor": "18 percent", "18%" , "cues": "draft", "audit", "comptroller", "estimate", "estimated", "estimates" } } }, { "id": "clinic-outage", "max words": 70, "text": "Brightpath Clinics reported that its scheduling system was unavailable for 3 hours and 40 minutes on October 14, from 8:20 a.m. to 12:00 p.m. Eastern. A failed database migration caused the outage, according to a preliminary review by engineer Dana Okafor. About 2,300 appointment reminders were not sent, and 41 patients were double-booked. No patient records were lost or exposed, the company said, and there is no evidence of unauthorized access. Brightpath said the vendor, Meridian Cloud, has not yet confirmed the root cause. The company will credit 15 percent of the October fee to affected clinics and plans to publish a full report by November 4.", "gold": "Brightpath Clinics' scheduling system was down 3 hours and 40 minutes on October 14 after a failed database migration, according to a preliminary review by engineer Dana Okafor. About 2,300 reminders went unsent and 41 patients were double-booked. No records were lost or exposed. Vendor Meridian Cloud has not confirmed the root cause. Clinics get a 15 percent fee credit, and a full report is due November 4.", "planted": "Brightpath's scheduling system failed for about 4 hours on October 14 because of Meridian Cloud's faulty migration. 2,300 reminders went out late and 41 patients lost records. Brightpath will refund 25 percent of fees.", "manifest": { "numbers": "3 hours and 40 minutes", "3 hours 40 minutes", "3 hours, 40 minutes", "3h40", "3h 40m" , "october 14", "oct. 14", "oct 14" , "2,300" , "41 patients", "41 double-booked", "41 appointments", "41" , "15 percent", "15%" , "november 4", "nov. 4", "nov 4" , "entities": "brightpath clinics", "brightpath" , "dana okafor", "okafor" , "meridian cloud", "meridian" , "guards": {"label": "records not lost", "anchor": "records" , "cues": "no", "not", "none", "without", "neither", "nor", "wasn't", "weren't", "never" }, {"label": "root cause unconfirmed", "anchor": "root cause" , "cues": "not", "hasn't", "has not", "yet", "unconfirmed", "unclear", "pending", "unknown" }, {"label": "cause is preliminary", "anchor": "migration" , "cues": "preliminary", "according to", "initial", "early", "review" } } }, { "id": "quarterly-results", "max words": 70, "text": "Northwind Gadgets said third-quarter revenue rose 12 percent to $48.6 million, below the $50 million analysts had expected, according to a survey by the research firm Calloway Partners. Subscription revenue grew 31 percent and now makes up 22 percent of sales. Hardware revenue fell 4 percent after the company discontinued the Orbit speaker in July. Chief financial officer Priya Raman said the company expects fourth-quarter revenue between $52 million and $55 million, but she cautioned that the forecast assumes no new tariffs on imported components. The company has not decided whether to replace the Orbit. Northwind cut 30 jobs, about 3 percent of its workforce, in September.", "gold": "Northwind Gadgets' third-quarter revenue rose 12 percent to $48.6 million, below the $50 million analysts expected, per Calloway Partners. Subscriptions grew 31 percent while hardware fell 4 percent after the Orbit speaker was discontinued. CFO Priya Raman guided to $52 million to $55 million, assuming no new tariffs. Northwind has not decided on an Orbit replacement and cut 30 jobs.", "planted": "Northwind Gadgets beat expectations with revenue up 12 percent to $48.6 million. Subscriptions grew 31 percent and hardware rose 4 percent. The company will replace the Orbit speaker and expects $60 million next quarter, even with new tariffs.", "manifest": { "numbers": "12 percent", "12%" , "$48.6 million", "48.6 million" , "$50 million", "50 million" , "31 percent", "31%" , "4 percent", "4%" , "$52 million", "52 million" , "$55 million", "55 million" , "30 jobs", "30 positions", "30 employees", "30 roles" , "entities": "northwind gadgets", "northwind" , "priya raman", "raman" , "calloway partners", "calloway" , "orbit" , "guards": {"label": "revenue missed expectations", "anchor": "$50 million", "50 million" , "cues": "below", "miss", "missed", "short", "under", "lower", "less" }, {"label": "forecast assumes no new tariffs", "anchor": "tariff", "tariffs" , "cues": "no new", "assuming", "assumes", "assumption", "if", "unless", "provided", "without", "absent" }, {"label": "Orbit replacement undecided", "anchor": "replacement", "replace" , "cues": "not decided", "has not", "hasn't", "undecided", "yet", "unsure", "no decision", "no plan" }, {"label": "hardware fell", "anchor": "hardware" , "cues": "fell", "fall", "dropped", "declined", "down", "decline", "slid", "shrank", "decrease", "lower" } } } Write the manifest before you see any model output. If you write it afterward, you will unconsciously widen the cue lists until your favorite model passes. Change rules later only for reasons you can defend in a sentence. Build the Deterministic Scorer Step 5 Step 5: Write scorer.py The scorer turns one summary and one case into a dictionary of numbers. It has four jobs. It normalizes typography so an en dash or a curly apostrophe cannot fool the matcher. It measures recall for numbers and entities. It runs each guard and labels the outcome pass , omitted or unmarked . And it hunts for invented numbers by comparing every digit sequence in the summary against the digits in the source. python import re from decimal import Decimal, InvalidOperation NUMBER = re.compile r"\d \d, ?:\.\d+ ?" GUARD WINDOW = 90 TYPOGRAPHY = {"\u2018": "'", "\u2019": "'", "\u2013": "-", ", ": "-", "\u00a0": " "} def clean text : low = text.lower for fancy, plain in TYPOGRAPHY.items : low = low.replace fancy, plain return low def norm number token : token = token.rstrip "," .replace ",", "" try: return format Decimal token .normalize , "f" except InvalidOperation: return token def numbers in text : return {norm number t for t in NUMBER.findall text } def pattern form : return r" ?< \w. " + re.escape form.lower + r" ? \w " def positions text, forms : low = clean text found = for form in forms: found += m.start for m in re.finditer pattern form , low return found def has any text, forms : return bool positions text, forms def guard status text, guard : anchors = positions text, guard "anchor" if not anchors: return "omitted" cues = positions text, guard "cues" if any abs a - c <= GUARD WINDOW for a in anchors for c in cues : return "pass" return "unmarked" def score summary, case : manifest = case "manifest" numbers = has any summary, forms for forms in manifest "numbers" entities = has any summary, forms for forms in manifest "entities" guards = {g "label" : guard status summary, g for g in manifest "guards" } invented = sorted numbers in summary - numbers in case "text" number recall = sum numbers / len numbers entity recall = sum entities / len entities guard rate = sum 1 for s in guards.values if s == "pass" / len guards base = number recall + entity recall + guard rate / 3 faithfulness = max 0.0, base - 0.1 len invented words = len summary.split return { "number recall": round number recall, 3 , "entity recall": round entity recall, 3 , "guard rate": round guard rate, 3 , "guards": guards, "invented": invented, "words": words, "within limit": words <= case "max words" , "faithfulness": round faithfulness, 3 , } The guard statuses carry more information than a boolean. omitted means the anchor never appeared, so the model dropped the fact entirely. unmarked means the anchor appeared without any cue nearby, which is the signature of a flipped negation or a stripped attribution. Those two outcomes call for different fixes in your prompt, so the report keeps them apart. The final score averages three recall figures and subtracts 0.1 for each invented number, with a floor at zero. The table below explains each column and what it cannot see. | Metric | What it counts | Failure it catches | Known blind spot | |---|---|---|---| | number recall | Share of critical figures present in any accepted form | Dropped votes, deadlines, amounts | Cannot tell if a kept figure now sits beside the wrong noun | | entity recall | Share of people and organizations still named | Vanished names and sources | Accepts a last-name-only mention | | guard rate | Share of negations, pending items and attributions kept | Flipped outcomes, estimates turned into facts | Cue lists can miss a valid paraphrase | | invented | Digit sequences in the summary absent from the source | Made-up figures | Misses an invented figure that reuses a digit from the source | | within limit | Whether word count stays at or under the cap | Ignored length instructions | Counts words by whitespace only | | faithfulness | Mean of the three recalls minus 0.1 per invented number | Overall fact survival | Equal weights hide that one flipped negation outweighs one lost name | That last blind spot matters. A summary that drops a name and a summary that flips a verdict can earn similar scores, even though the second does far more damage. Keep the guard rate visible next to the headline number, and treat any guard below 1.0 as the first thing to read. Build the Harness and Calibrate It Steps 6 to 8 Step 6: Write harness.py The harness has three subcommands. generate calls the APIs and saves one text file per summary. controls writes the three calibration summaries. report scores everything on disk and prints a table. The folder layout does the bookkeeping: outputs/