cd /news/large-language-models/claude-vs-chatgpt-summary-test-12-st… · home › topics › large-language-models › article
[ARTICLE · art-172901] src=shattered.io ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Claude vs ChatGPT Summary Test: 12 Steps, 80 Min

A 12-step, roughly 80-minute test harness published on Shattered scores Claude and ChatGPT summaries with plain Python and a hand-written fact manifest instead of a judge model, targeting four failure types: dropped numbers, invented numbers, lost negations and lost attribution. The harness caps summaries at 70 words, runs two prompts (bare and guarded) with five runs per cell, and calibrates the deterministic scorer against three control summaries. The test arrives after OpenAI introduced GPT-6.2 Astra and GPT-6.2 Sol on November 17, 2026, and Anthropic shipped Claude Fable 5.2 and Mythos 5.2 earlier in November, with Claude Opus 5.5 remaining the default on Pro, Max, Ultra and Team plans.

recommendation

Replace judge-model scoring with a hand-written fact manifest and regex rules, run five samples per prompt, and calibrate the scorer against perfect, lazy and sabotaged control summaries before trusting any Claude vs ChatGPT comparison.

more in playbook: Evals and testing for LLM apps
by read38 min views1 publishedNov 19, 2026
Claude vs ChatGPT Summary Test: 12 Steps, 80 Min
Image: Shattered (auto-discovered)

OpenAI introduced GPT-6.2 Astra and GPT-6.2 Sol on November 17, 2026, and its own ChatGPT documentation now lists them as the newest models. Anthropic shipped Claude Fable 5.2 and Mythos 5.2 earlier in November, while Claude Opus 5.5 stays the default model on the Pro, Max, Ultra and Team plans. Any Claude vs ChatGPT verdict written a month ago describes a matchup that no longer exists.

Opinions about which assistant writes better age fast. A measurement ages slowly, provided you pin the model names and rerun it when either vendor ships. Most comparisons judge style, and style is easy to fake. Summaries fail in a quieter way: a number drifts, a name disappears, or did not approve turns into approved inside a perfectly fluent sentence. The harness below scores exactly those failures with plain Python and no judge model, so the same summary always earns the same score. It pairs naturally with the writing rules compliance test, which checks house style while this one checks whether the facts survive compression.

Plan on about 80 minutes. The API runs cost cents, and manual mode needs no keys at all.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why Claude vs ChatGPT Summaries Need Their Own Test #

A summary earns trust because it is short, and nobody rereads the source afterward. That makes four failure types expensive. The first is a dropped number, such as a vote count or a deadline. The second is an invented number, where the model adds a plausible figure the source never contained. The third is a lost negation, the worst of the four, because the summary asserts the opposite of the source. The fourth is a lost attribution, where an estimate from a draft audit becomes a flat statement of fact.

The water board source below shows the pattern. It says the board did not approve a $12 monthly surcharge. A careless summary writes that the board approved the surcharge, keeps the names and the dates, and reads smoothly. A human skimming for tone will not catch it. A script that finds the word surcharge and then looks for a negation cue within 90 characters will.

Editing tasks measure something different. The editing fidelity test asks how much of a draft a model changes when told to leave most of it alone. Summarization forces loss, because a long text must shrink to a short one, so the useful question becomes which facts survive the cut.

The design has five parts, each small enough to read in one sitting:

  • Short closed-source documents, written about fictional organizations so neither model can recall the facts from training.
  • A 70-word cap that creates real pressure to drop something.
  • Two prompts, a bare one and a guarded one that names the traps, so you can measure how much instruction helps each model.
  • Five runs per cell, because a single sample hides variance.
  • A deterministic scorer driven by a hand-written fact manifest, calibrated with three control summaries (perfect, lazy and sabotaged).

The scorer deliberately avoids a judge model. A judge has its own blind spots, may favor text that resembles its own output, and returns different scores on reruns. A regular expression returns the same number every time, and you can read the rule behind it. The tradeoff is a hand-written manifest, which takes about 15 minutes per source. A separate source-grounded writing audit looks at citations in longer drafts, while this harness stays on short documents where every fact can be enumerated.

Prerequisites and Pinned Versions #

Everything below was run on Python 3.12.3 on Linux. Python 3.11 or newer works because the code uses only the standard library plus two vendor SDKs. You need a terminal, a text editor, and one of two access paths. API mode needs an Anthropic API key and an OpenAI API key. Manual mode needs only a Claude account and a ChatGPT account, since you paste the prompts into the apps yourself.

  • Python: 3.11 or newer (tested on 3.12.3).
  • pip packages: the latestanthropic andopenai releases. The harness calls the OpenAI Responses API, which every current SDK release includes.

Model names move quickly, so pin them before you start. The table lists the state of both product lines on November 19, 2026, taken from the vendors’ own pages and release notes.

Item State on November 19, 2026 What to record in your run log
Claude default model (Pro, Max, Ultra, Team) Claude Opus 5.5 Exact API model ID from Anthropic’s model list
Newest Claude releases Claude Fable 5.2 and Claude Mythos 5.2, announced in November 2026 Whether you test a newest-tier model or the default
Claude small model Claude Haiku 5.5, aimed at high-volume work such as summarization Use as a cheap second Claude row
Opus 5.5 API price $4 per million input tokens, $20 per million output tokens Price on your run date
Haiku 5.5 API price $0.10 input, $0.50 output per million tokens for prompts up to 100,000 tokens Price on your run date
Fable 5.2 API price $8 input, $40 output per million tokens Price on your run date
Newest OpenAI models GPT-6.2 Astra and GPT-6.2 Sol, introduced November 17, 2026 The label your picker or API shows
ChatGPT Plus $20 per month Plan used for manual mode
ChatGPT Pro Three tiers: $100, $200 and $500 per month (Pro 500 added November 19) Tier used, if any

Anthropic’s own knowledge work guide for Opus 5.5 describes it as the default model in Claude on those four plans, so Opus 5.5 is the fair counterpart when you test the app rather than the API. OpenAI’s release notes present GPT-6.2 Astra and Sol as introduced for ChatGPT Work and Codex, which means the ordinary chat model picker may show a different name than the API does. Write down whatever label you actually see. GitHub’s changelog entry from November 17 gives a second confirmation that both names are live for developers.

Plan prices and usage caps shift often enough that the ChatGPT vs Claude pricing and usage limits breakdown is worth checking before you decide which tiers to test. For the API runs, the arithmetic is small. Each provider receives 30 calls (3 sources, 2 prompts, 5 runs). At roughly 250 input tokens and 100 output tokens per call, that is about 7,500 input tokens and 3,000 output tokens, which comes to under ten cents at Opus 5.5’s listed rates and a fraction of a cent on Haiku 5.5. Models that spend hidden reasoning tokens will cost more, but the order of magnitude holds.

Set Up the Workspace (Steps 1 and 2) #

Step 1: Create the project folder and virtual environment

A virtual environment keeps the two SDKs away from your system Python and makes the setup reproducible on another machine. Run these commands from the directory where you keep projects.

mkdir summary-faithfulness
cd summary-faithfulness
python3 -m venv .venv
source .venv/bin/activate
python --version
pip install --upgrade pip anthropic openai

On Windows, activate with .venv\Scripts\activate instead of the source line. The version check should print 3.11 or higher. Install a newer interpreter first if it does not.

Step 2: Export your keys and pin the model IDs

Both SDKs read their API keys from environment variables, so no key ever touches a source file. The two model variables are the part that matters for the experiment. Copy each exact model ID from the provider’s current model list rather than typing a name from memory, because an alias such as a family name can silently point at a newer release next month.

export ANTHROPIC_API_KEY="paste-your-anthropic-key"
export OPENAI_API_KEY="paste-your-openai-key"
export CLAUDE_MODEL="paste-exact-model-id-from-anthropic-docs"
export OPENAI_MODEL="paste-exact-model-id-from-openai-docs"

The harness copies the value of each model variable into outputs/meta.jsonl beside every saved summary. When a result looks odd three weeks from now, that file tells you which model produced it. For a first pass, test the flagship you would actually use for writing on each side. Anthropic positions Opus 5.5 for knowledge work, and OpenAI lists GPT-6.2 Astra and Sol as its newest pair. A second pass with Haiku 5.5 on the Claude side is cheap and answers a practical question, since Anthropic aims that model at summarization volume.

Plant the Facts: Sources and Fact Manifests (Steps 3 and 4) #

Step 3: Choose sources that contain checkable traps

A good test source is closed, short and booby-trapped. Closed means the summary can only use what is on the page, so any extra number is an invention you can detect. Short means 100 to 140 words, which keeps the manifest writable by hand. Booby-trapped means each document contains the specific facts that summarizers like to lose.

Every source in this project carries at least five hard numbers, at least two statements of what did not happen or is not yet known, and at least one claim attributed to a secondary source such as a draft audit or a preliminary review. The organizations are fictional, so neither model can lean on training data. The three domains differ on purpose: a local utility vote, a clinic software outage, and a quarterly earnings note. A model that handles civic prose well may stumble on earnings language, and you want to see that gap.

Voice and tone are a separate question that deserves a separate test, and the stylometry-based voice match covers it. Nothing here asks which summary sounds nicer. The only question is whether the facts you planted survived.

Each source gets a manifest with three parts. The numbers list holds one entry per critical figure, and each entry is a list of acceptable surface forms, because 6.5 percent and 6.5% are both correct. The entities list does the same for names. The guards list handles the dangerous facts. A guard has an anchor (the words that show the fact appears in the summary) and cues (the words that show the qualifier survived). A guard passes only when a cue sits within 90 characters of an anchor.

Step 4: Write cases.json

Save the following as cases.json. Each case also stores a hand-written gold summary and a sabotaged one, which the calibration step uses. The gold summaries must fit the 70-word cap and must score exactly 1.0, and a unit test enforces both.

[
  {
    "id": "water-board",
    "max_words": 70,
    "text": "The Harlow County Water Board voted 5 to 2 on Tuesday to raise residential rates by 6.5 percent starting January 1, 2027. The increase adds about $4.80 to the average monthly bill of $73.90. Board chair Marisol Vega said the money will fund the replacement of 11 miles of cast-iron pipe built in the 1950s. The board did not approve a separate proposal to add a $12 monthly infrastructure surcharge. According to a draft audit from the county comptroller, the utility lost an estimated 18 percent of treated water to leaks last year. Vega said the board will not raise rates again before 2029 unless costs rise sharply. Residents can comment at a public hearing on December 3 at the Lakeview Library, and low-income customers may apply for a bill credit through the county's Neighbor Aid program.",
    "gold": "Harlow County Water Board voted 5-2 to raise residential rates 6.5 percent from January 1, 2027, adding about $4.80 to the $73.90 average bill. Chair Marisol Vega said it funds 11 miles of pipe replacement. The board rejected a $12 surcharge and will not raise rates before 2029. A draft comptroller audit estimates 18 percent of water leaks. Hearing: December 3, Lakeview Library. Neighbor Aid offers credits.",
    "planted": "The Harlow County Water Board voted 5-2 to raise residential rates 6.5 percent and approved a $12 monthly surcharge. The county says 18 percent of water leaks. Rates may rise again in 2028. Chair Marisol Vega called the plan overdue.",
    "manifest": {
      "numbers": [
        ["5 to 2", "5-2", "five to two", "5:2"],
        ["6.5 percent", "6.5%"],
        ["$4.80", "4.80", "$4.8"],
        ["11 miles", "11-mile", "11 mi"],
        ["18 percent", "18%"],
        ["2029"],
        ["december 3", "dec. 3", "dec 3"]
      ],
      "entities": [
        ["harlow county water board", "harlow county board", "harlow water board", "water board", "harlow"],
        ["marisol vega", "vega"],
        ["comptroller"],
        ["lakeview library", "lakeview"],
        ["neighbor aid"]
      ],
      "guards": [
        {"label": "surcharge rejected", "anchor": ["surcharge"], "cues": ["rejected", "not", "no", "declined", "did not", "didn't", "turned down", "voted down", "denied", "blocked", "scrapped", "dropped"]},
        {"label": "no rise before 2029", "anchor": ["2029"], "cues": ["not", "no", "until", "unless", "won't", "freeze", "hold", "before", "through", "pledged"]},
        {"label": "leak figure is a draft estimate", "anchor": ["18 percent", "18%"], "cues": ["draft", "audit", "comptroller", "estimate", "estimated", "estimates"]}
      ]
    }
  },
  {
    "id": "clinic-outage",
    "max_words": 70,
    "text": "Brightpath Clinics reported that its scheduling system was unavailable for 3 hours and 40 minutes on October 14, from 8:20 a.m. to 12:00 p.m. Eastern. A failed database migration caused the outage, according to a preliminary review by engineer Dana Okafor. About 2,300 appointment reminders were not sent, and 41 patients were double-booked. No patient records were lost or exposed, the company said, and there is no evidence of unauthorized access. Brightpath said the vendor, Meridian Cloud, has not yet confirmed the root cause. The company will credit 15 percent of the October fee to affected clinics and plans to publish a full report by November 4.",
    "gold": "Brightpath Clinics' scheduling system was down 3 hours and 40 minutes on October 14 after a failed database migration, according to a preliminary review by engineer Dana Okafor. About 2,300 reminders went unsent and 41 patients were double-booked. No records were lost or exposed. Vendor Meridian Cloud has not confirmed the root cause. Clinics get a 15 percent fee credit, and a full report is due November 4.",
    "planted": "Brightpath's scheduling system failed for about 4 hours on October 14 because of Meridian Cloud's faulty migration. 2,300 reminders went out late and 41 patients lost records. Brightpath will refund 25 percent of fees.",
    "manifest": {
      "numbers": [
        ["3 hours and 40 minutes", "3 hours 40 minutes", "3 hours, 40 minutes", "3h40", "3h 40m"],
        ["october 14", "oct. 14", "oct 14"],
        ["2,300"],
        ["41 patients", "41 double-booked", "41 appointments", "41"],
        ["15 percent", "15%"],
        ["november 4", "nov. 4", "nov 4"]
      ],
      "entities": [
        ["brightpath clinics", "brightpath"],
        ["dana okafor", "okafor"],
        ["meridian cloud", "meridian"]
      ],
      "guards": [
        {"label": "records not lost", "anchor": ["records"], "cues": ["no", "not", "none", "without", "neither", "nor", "wasn't", "weren't", "never"]},
        {"label": "root cause unconfirmed", "anchor": ["root cause"], "cues": ["not", "hasn't", "has not", "yet", "unconfirmed", "unclear", "pending", "unknown"]},
        {"label": "cause is preliminary", "anchor": ["migration"], "cues": ["preliminary", "according to", "initial", "early", "review"]}
      ]
    }
  },
  {
    "id": "quarterly-results",
    "max_words": 70,
    "text": "Northwind Gadgets said third-quarter revenue rose 12 percent to $48.6 million, below the $50 million analysts had expected, according to a survey by the research firm Calloway Partners. Subscription revenue grew 31 percent and now makes up 22 percent of sales. Hardware revenue fell 4 percent after the company discontinued the Orbit speaker in July. Chief financial officer Priya Raman said the company expects fourth-quarter revenue between $52 million and $55 million, but she cautioned that the forecast assumes no new tariffs on imported components. The company has not decided whether to replace the Orbit. Northwind cut 30 jobs, about 3 percent of its workforce, in September.",
    "gold": "Northwind Gadgets' third-quarter revenue rose 12 percent to $48.6 million, below the $50 million analysts expected, per Calloway Partners. Subscriptions grew 31 percent while hardware fell 4 percent after the Orbit speaker was discontinued. CFO Priya Raman guided to $52 million to $55 million, assuming no new tariffs. Northwind has not decided on an Orbit replacement and cut 30 jobs.",
    "planted": "Northwind Gadgets beat expectations with revenue up 12 percent to $48.6 million. Subscriptions grew 31 percent and hardware rose 4 percent. The company will replace the Orbit speaker and expects $60 million next quarter, even with new tariffs.",
    "manifest": {
      "numbers": [
        ["12 percent", "12%"],
        ["$48.6 million", "48.6 million"],
        ["$50 million", "50 million"],
        ["31 percent", "31%"],
        ["4 percent", "4%"],
        ["$52 million", "52 million"],
        ["$55 million", "55 million"],
        ["30 jobs", "30 positions", "30 employees", "30 roles"]
      ],
      "entities": [
        ["northwind gadgets", "northwind"],
        ["priya raman", "raman"],
        ["calloway partners", "calloway"],
        ["orbit"]
      ],
      "guards": [
        {"label": "revenue missed expectations", "anchor": ["$50 million", "50 million"], "cues": ["below", "miss", "missed", "short", "under", "lower", "less"]},
        {"label": "forecast assumes no new tariffs", "anchor": ["tariff", "tariffs"], "cues": ["no new", "assuming", "assumes", "assumption", "if", "unless", "provided", "without", "absent"]},
        {"label": "Orbit replacement undecided", "anchor": ["replacement", "replace"], "cues": ["not decided", "has not", "hasn't", "undecided", "yet", "unsure", "no decision", "no plan"]},
        {"label": "hardware fell", "anchor": ["hardware"], "cues": ["fell", "fall", "dropped", "declined", "down", "decline", "slid", "shrank", "decrease", "lower"]}
      ]
    }
  }
]

Write the manifest before you see any model output. If you write it afterward, you will unconsciously widen the cue lists until your favorite model passes. Change rules later only for reasons you can defend in a sentence.

Build the Deterministic Scorer (Step 5) #

Step 5: Write scorer.py

The scorer turns one summary and one case into a dictionary of numbers. It has four jobs. It normalizes typography so an en dash or a curly apostrophe cannot fool the matcher. It measures recall for numbers and entities. It runs each guard and labels the outcome pass, omitted or unmarked. And it hunts for invented numbers by comparing every digit sequence in the summary against the digits in the source.

import re
from decimal import Decimal, InvalidOperation

NUMBER = re.compile(r"\d[\d,]*(?:\.\d+)?")
GUARD_WINDOW = 90

TYPOGRAPHY = {"\u2018": "'", "\u2019": "'", "\u2013": "-", ", ": "-", "\u00a0": " "}

def clean(text):
    low = text.lower()
    for fancy, plain in TYPOGRAPHY.items():
        low = low.replace(fancy, plain)
    return low

def norm_number(token):
    token = token.rstrip(",").replace(",", "")
    try:
        return format(Decimal(token).normalize(), "f")
    except InvalidOperation:
        return token

def numbers_in(text):
    return {norm_number(t) for t in NUMBER.findall(text)}

def pattern(form):
    return r"(?<![\w.])" + re.escape(form.lower()) + r"(?!\w)"

def positions(text, forms):
    low = clean(text)
    found = []
    for form in forms:
        found += [m.start() for m in re.finditer(pattern(form), low)]
    return found

def has_any(text, forms):
    return bool(positions(text, forms))

def guard_status(text, guard):
    anchors = positions(text, guard["anchor"])
    if not anchors:
        return "omitted"
    cues = positions(text, guard["cues"])
    if any(abs(a - c) <= GUARD_WINDOW for a in anchors for c in cues):
        return "pass"
    return "unmarked"

def score(summary, case):
    manifest = case["manifest"]
    numbers = [has_any(summary, forms) for forms in manifest["numbers"]]
    entities = [has_any(summary, forms) for forms in manifest["entities"]]
    guards = {g["label"]: guard_status(summary, g) for g in manifest["guards"]}
    invented = sorted(numbers_in(summary) - numbers_in(case["text"]))

    number_recall = sum(numbers) / len(numbers)
    entity_recall = sum(entities) / len(entities)
    guard_rate = sum(1 for s in guards.values() if s == "pass") / len(guards)
    base = (number_recall + entity_recall + guard_rate) / 3
    faithfulness = max(0.0, base - 0.1 * len(invented))
    words = len(summary.split())

    return {
        "number_recall": round(number_recall, 3),
        "entity_recall": round(entity_recall, 3),
        "guard_rate": round(guard_rate, 3),
        "guards": guards,
        "invented": invented,
        "words": words,
        "within_limit": words <= case["max_words"],
        "faithfulness": round(faithfulness, 3),
    }

The guard statuses carry more information than a boolean. omitted means the anchor never appeared, so the model dropped the fact entirely. unmarked means the anchor appeared without any cue nearby, which is the signature of a flipped negation or a stripped attribution. Those two outcomes call for different fixes in your prompt, so the report keeps them apart.

The final score averages three recall figures and subtracts 0.1 for each invented number, with a floor at zero. The table below explains each column and what it cannot see.

Metric What it counts Failure it catches Known blind spot
number_recall Share of critical figures present in any accepted form Dropped votes, deadlines, amounts Cannot tell if a kept figure now sits beside the wrong noun
entity_recall Share of people and organizations still named Vanished names and sources Accepts a last-name-only mention
guard_rate Share of negations, pending items and attributions kept Flipped outcomes, estimates turned into facts Cue lists can miss a valid paraphrase
invented Digit sequences in the summary absent from the source Made-up figures Misses an invented figure that reuses a digit from the source
within_limit Whether word count stays at or under the cap Ignored length instructions Counts words by whitespace only
faithfulness Mean of the three recalls minus 0.1 per invented number Overall fact survival Equal weights hide that one flipped negation outweighs one lost name

That last blind spot matters. A summary that drops a name and a summary that flips a verdict can earn similar scores, even though the second does far more damage. Keep the guard rate visible next to the headline number, and treat any guard below 1.0 as the first thing to read.

Build the Harness and Calibrate It (Steps 6 to 8) #

Step 6: Write harness.py

The harness has three subcommands. generate calls the APIs and saves one text file per summary. controls writes the three calibration summaries. report scores everything on disk and prints a table. The folder layout does the bookkeeping: outputs/<provider>/<prompt>/<case>__<run>.txt. Any folder name works as a provider label, which is what makes manual mode possible later.

import argparse
import csv
import json
import os
import pathlib
import statistics
import time

from scorer import score

ROOT = pathlib.Path(__file__).parent
OUT = ROOT / "outputs"
CASES = json.loads((ROOT / "cases.json").read_text(encoding="utf-8"))

PROMPTS = {
    "bare": "Summarize the text below in at most {n} words.\n\n{text}",
    "guarded": (
        "Summarize the text below in at most {n} words. Keep every number exactly as written. "
        "Keep the names of people and organizations. Keep any statement about what did not happen, "
        "was rejected, or is not yet known. Mark estimates and secondhand claims as such. "
        "Add no facts that are not in the text.\n\n{text}"
    ),
}

def call_claude(prompt):
    import anthropic

    client = anthropic.Anthropic()
    reply = client.messages.create(
        model=os.environ["CLAUDE_MODEL"],
        max_tokens=2000,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(block.text for block in reply.content if block.type == "text")

def call_chatgpt(prompt):
    from openai import OpenAI

    client = OpenAI()
    reply = client.responses.create(model=os.environ["OPENAI_MODEL"], input=prompt)
    return reply.output_text

PROVIDERS = {"claude": call_claude, "chatgpt": call_chatgpt}
MODEL_ENV = {"claude": "CLAUDE_MODEL", "chatgpt": "OPENAI_MODEL"}

def generate(args):
    for provider in args.providers.split(","):
        for prompt_name, template in PROMPTS.items():
            folder = OUT / provider / prompt_name
            folder.mkdir(parents=True, exist_ok=True)
            for case in CASES:
                for run in range(1, args.runs + 1):
                    target = folder / f"{case['id']}__{run}.txt"
                    if target.exists():
                        continue
                    prompt = template.format(n=case["max_words"], text=case["text"])
                    started = time.time()
                    for attempt in range(3):
                        try:
                            text = PROVIDERS[provider](prompt).strip()
                            break
                        except Exception as error:
                            print(f"{provider} {case['id']} run {run} attempt {attempt + 1}: {error}")
                            time.sleep(2 ** attempt * 3)
                    else:
                        continue
                    target.write_text(text, encoding="utf-8")
                    meta = {
                        "provider": provider,
                        "model": os.environ.get(MODEL_ENV[provider], ""),
                        "prompt": prompt_name,
                        "case": case["id"],
                        "run": run,
                        "seconds": round(time.time() - started, 1),
                        "stamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
                    }
                    with open(OUT / "meta.jsonl", "a", encoding="utf-8") as handle:
                        handle.write(json.dumps(meta) + "\n")
                    print("saved", target.relative_to(OUT))

def controls(_args):
    for case in CASES:
        lead = " ".join(case["text"].split()[: case["max_words"]])
        for kind, text in (("gold", case["gold"]), ("planted", case["planted"]), ("lead", lead)):
            folder = OUT / kind / "control"
            folder.mkdir(parents=True, exist_ok=True)
            (folder / f"{case['id']}__1.txt").write_text(text, encoding="utf-8")
    print("controls written")

def collect():
    by_id = {c["id"]: c for c in CASES}
    rows = []
    for path in sorted(OUT.glob("*/*/*.txt")):
        case_id, run = path.stem.rsplit("__", 1)
        if case_id not in by_id:
            continue
        result = score(path.read_text(encoding="utf-8"), by_id[case_id])
        rows.append({"provider": path.parts[-3], "prompt": path.parts[-2], "case": case_id, "run": run, **result})
    return rows

def report(args):
    rows = collect()
    if not rows:
        raise SystemExit("no outputs found: run generate or controls first")
    with open(ROOT / "results.csv", "w", newline="", encoding="utf-8") as handle:
        fields = ["provider", "prompt", "case", "run", "faithfulness", "number_recall", "entity_recall",
                  "guard_rate", "invented", "words", "within_limit", "guards"]
        writer = csv.DictWriter(handle, fieldnames=fields, extrasaction="ignore")
        writer.writeheader()
        for row in rows:
            writer.writerow({**row, "invented": " ".join(row["invented"]), "guards": json.dumps(row["guards"])})

    groups = {}
    for row in rows:
        groups.setdefault((row["provider"], row["prompt"]), []).append(row)

    lines = ["| provider | prompt | n | mean | min | numbers | guards | invented | in limit |",
             "|---|---|---|---|---|---|---|---|---|"]
    for (provider, prompt), items in groups.items():
        mean = lambda key: statistics.mean(r[key] for r in items)
        lines.append(
            f"| {provider} | {prompt} | {len(items)} | {mean('faithfulness'):.2f} | "
            f"{min(r['faithfulness'] for r in items):.2f} | {mean('number_recall'):.2f} | "
            f"{mean('guard_rate'):.2f} | {sum(len(r['invented']) for r in items)} | "
            f"{sum(r['within_limit'] for r in items)}/{len(items)} |"
        )
    table = "\n".join(lines)
    (ROOT / "report.md").write_text(table + "\n", encoding="utf-8")
    print(table)

    if args.show_fails:
        print()
        for row in rows:
            bad = {k: v for k, v in row["guards"].items() if v != "pass"}
            if bad or row["invented"]:
                print(f"{row['provider']}/{row['prompt']} {row['case']} run {row['run']}: {bad} invented={row['invented']}")

def main():
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="command", required=True)
    gen = sub.add_parser("generate")
    gen.add_argument("--providers", default="claude,chatgpt")
    gen.add_argument("--runs", type=int, default=5)
    gen.set_defaults(func=generate)
    sub.add_parser("controls").set_defaults(func=controls)
    rep = sub.add_parser("report")
    rep.add_argument("--show-fails", action="store_true")
    rep.set_defaults(func=report)
    args = parser.parse_args()
    args.func(args)

if __name__ == "__main__":
    main()

Four design choices in that file matter. Generation is resumable, because generate skips any file that already exists. Failed calls retry three times with waits of 3, 6 and 12 seconds. The token ceiling is 2,000, far above what a 70-word summary needs, because models that think before answering can spend a small ceiling on reasoning and return nothing. And no sampling parameters are set, which keeps each call valid on every model and mirrors the apps’ defaults. Every saved file also gets a line in meta.jsonl with the model value you exported.

The cost per accepted 1,000 words method shows how to turn token counts into a price per usable result. At this length the bill is too small to matter, so the harness leaves it out.

Step 7: Add the unit tests

The tests encode the scorer’s contract in five assertions. Gold summaries must score exactly 1.0 and fit the cap. Sabotaged summaries must score below 0.6. Decimal forms must match. An en dash and a curly apostrophe must not break matching. And an invented number must be flagged. Save this as test_scorer.py.

import json
import pathlib
import unittest

from scorer import numbers_in, score

CASES = json.loads((pathlib.Path(__file__).parent / "cases.json").read_text(encoding="utf-8"))

class ScorerCalibration(unittest.TestCase):
    def test_gold_summaries_score_one(self):
        for case in CASES:
            result = score(case["gold"], case)
            self.assertEqual(result["faithfulness"], 1.0, case["id"])
            self.assertTrue(result["within_limit"], case["id"])

    def test_planted_summaries_score_low(self):
        for case in CASES:
            self.assertLess(score(case["planted"], case)["faithfulness"], 0.6, case["id"])

    def test_decimal_forms_match(self):
        self.assertEqual(numbers_in("$4.80 and 4.8"), {"4.8"})

    def test_en_dash_and_curly_apostrophe_match(self):
        case = CASES[0]
        text = case["gold"].replace("5-2", "5\u20132").replace("will not", "won\u2019t")
        self.assertEqual(score(text, case)["faithfulness"], 1.0)

    def test_invented_number_is_flagged(self):
        case = CASES[0]
        result = score(case["gold"] + " It affects 9,999 homes.", case)
        self.assertEqual(result["invented"], ["9999"])

if __name__ == "__main__":
    unittest.main()

Run the suite from the project folder:

$ python -m unittest -v
test_decimal_forms_match (test_scorer.ScorerCalibration.test_decimal_forms_match) ... ok
test_en_dash_and_curly_apostrophe_match (test_scorer.ScorerCalibration.test_en_dash_and_curly_apostrophe_match) ... ok
test_gold_summaries_score_one (test_scorer.ScorerCalibration.test_gold_summaries_score_one) ... ok
test_invented_number_is_flagged (test_scorer.ScorerCalibration.test_invented_number_is_flagged) ... ok
test_planted_summaries_score_low (test_scorer.ScorerCalibration.test_planted_summaries_score_low) ... ok

----------------------------------------------------------------------
Ran 5 tests in 0.013s

OK

If any test fails here, stop. A scorer that fails its own calibration will produce numbers you cannot defend, and the cheapest moment to find that out is before spending a single API call.

Step 8: Run the three controls and read the ladder

The controls are summaries you wrote or generated without a model. The gold summary is perfect by construction. The sabotaged summary flips the surcharge, invents a figure and drops most names. The lead baseline simply copies the first 70 words of the source, which is what a lazy truncation would produce. Generate them and print the report:

$ python harness.py controls
controls written

$ python harness.py report
| provider | prompt | n | mean | min | numbers | guards | invented | in limit |
|---|---|---|---|---|---|---|---|---|
| gold | control | 3 | 1.00 | 1.00 | 1.00 | 1.00 | 0 | 3/3 |
| lead | control | 3 | 0.62 | 0.43 | 0.66 | 0.50 | 0 | 3/3 |
| planted | control | 3 | 0.23 | 0.18 | 0.48 | 0.00 | 3 | 3/3 |

The failure details show how the guards behave. Run the same report with the failure flag:

$ python harness.py report --show-fails
lead/control clinic-outage run 1: {'root cause unconfirmed': 'omitted'} invented=[]
lead/control quarterly-results run 1: {'forecast assumes no new tariffs': 'omitted', 'Orbit replacement undecided': 'omitted'} invented=[]
lead/control water-board run 1: {'no rise before 2029': 'omitted', 'leak figure is a draft estimate': 'omitted'} invented=[]
planted/control clinic-outage run 1: {'records not lost': 'unmarked', 'root cause unconfirmed': 'omitted', 'cause is preliminary': 'unmarked'} invented=['25']
planted/control quarterly-results run 1: {'revenue missed expectations': 'omitted', 'forecast assumes no new tariffs': 'unmarked', 'Orbit replacement undecided': 'unmarked', 'hardware fell': 'unmarked'} invented=['60']
planted/control water-board run 1: {'surcharge rejected': 'unmarked', 'no rise before 2029': 'omitted', 'leak figure is a draft estimate': 'unmarked'} invented=['2028']

The lead baseline mostly fails by omission, because 70 words from the top of a document never reach the qualifiers near the bottom. The sabotaged summaries fail by unmarked, since they mention the surcharge, the tariffs and the hardware line without the qualifier that made each statement true. Real model outputs will fall somewhere between those two patterns, and which pattern dominates tells you what to fix.

Choose Prompts and Run Counts (Step 9) #

Step 9: Lock the two prompt variants and the sample size

The harness ships two prompts. The bare prompt asks for a summary in at most 70 words and nothing else, which approximates what a busy person types. The guarded prompt adds five instructions: keep every number exactly as written, keep the names of people and organizations, keep statements about what did not happen or is not yet known, mark estimates and secondhand claims, and add no facts. Comparing the two answers a question that matters more than the model choice itself, namely how much a careful instruction buys on each side.

The guarded prompt names categories of trap but never a specific fact. It does not mention surcharges, tariffs or the year 2029. If it did, you would be testing whether a model can follow a pointer, and the scores would reflect your manifest rather than the model. This is the same reason the manifest is written before any run.

Sample size follows from variance. Each cell in the report holds 15 scored summaries per provider and prompt (3 sources times 5 runs). That is enough to see a large gap and too few to rank two close models. Treat any difference under 0.05 as a tie until you add runs, and use the bootstrap script from the advanced section to see how wide the uncertainty is. To raise the count, change one flag:

python harness.py generate --providers claude,chatgpt --runs 10

Because generation skips files that exist, raising the count only pays for the new runs. Keep every setting identical between providers. If you give one model a system prompt, give the other the same one, and record any mismatch as a limitation.

Run Both Models (Step 10) #

Step 10: Generate the summaries

With the four variables exported, one command produces every output file. The loop runs providers first, then prompts, then cases, then runs, and prints one line per saved file.

python harness.py generate --providers claude,chatgpt --runs 5

Expect 60 lines of the form saved claude/bare/water-board__1.txt, and expect the run to take a few minutes. A retry message such as attempt 1: ... is normal during rate-limit spikes. A file that never appears after three failed attempts is skipped, so rerun the same command and it fills the gaps. Each saved file also adds a line to the metadata log, shaped like this:

{"provider": "claude", "model": "<value of CLAUDE_MODEL>", "prompt": "bare", "case": "water-board", "run": 1, "seconds": 2.1, "stamp": "<ISO timestamp>"}

Manual mode for app users. If you only have Claude and ChatGPT accounts, skip generate. For each source and each prompt, open a fresh conversation, paste the full prompt text, copy the reply, and save it as outputs/claude-app/bare/water-board__1.txt (or outputs/chatgpt-app/guarded/clinic-outage__3.txt and so on). The report command picks up any provider folder with the same three-level structure, so the same scoring applies. Three habits keep manual runs honest. Use a new conversation for every sample. Switch off memory and personalization features on the test account if your plan has them. And delete any lead-in such as a sentence announcing the summary, because the word counter would charge it against the cap. Add a notes.txt recording the plan name, the date, and the model label the app displayed.

App behavior shifts under you between releases, so the dated notes file is how you will know, next quarter, exactly what you tested.

Score and Read the Report (Step 11) #

Step 11: Run the report and open the failures

python harness.py report --show-fails

The report prints one row per provider and prompt. The mean column is the headline, but read min next, because the worst summary is the one that would reach a reader. Then read guards, the share of negations, pending items and attributions that survived, because that column tracks the failures that mislead. Then check invented, which counts every figure that appeared from nowhere. Any nonzero value deserves a look at the exact file named in the failure list.

Open the failing files yourself. The scorer finds candidates, and you decide whether each is a true failure or a valid paraphrase the cue list missed. The second case happens more than you would expect. Take this hand-written summary of the water board source:

Harlow County's water board approved a 6.5% rate hike (5-2) from January 1, 2027, adding
about $4.80 to a $73.90 average bill, to replace 11 miles of 1950s pipe, said chair Marisol
Vega. It shelved a $12 surcharge and promised a freeze until 2029. A draft county comptroller
audit put leaks at about 18%. Public hearing December 3, Lakeview Library. Neighbor Aid
credits available.

It is faithful on every fact, yet the scorer returns a lower guard rate and a lower faithfulness score, with surcharge rejected marked unmarked. The word shelved was not in the cue list. The fix is to append it to the cues in cases.json, after which the same summary scores 1.0. Make the change, then rescore every provider’s outputs, not just the model you were debugging. Otherwise one model benefits from a fix the other never got.

A single short source also cannot tell you whether quality holds up over a long session. The long-form drift test covers that separate question, and the two together give a fuller picture than either alone.

Compare cells in a fixed order. Start with the two guarded rows, which show each model at its most careful. Then compare each provider’s bare row against its guarded row, because the gap measures how much the model needs to be told what matters. Finish with the two bare rows, which come closest to everyday use.

Turn the Numbers Into a Decision (Step 12) #

Step 12: Apply decision rules and archive the run

Numbers become a decision only when you attach actions to patterns in advance. The table below gives starting rules. Adjust the thresholds to your tolerance, but set them before the run, not after.

Pattern in the report What it means Action
Guarded mean 0.90 or higher for both models, bare mean well below it Instruction matters more than model choice Standardize on the guarded prompt and ship it in your team template
One model has invented above 0 in any cell It adds figures the source never contained Keep a human check on numbers for that model, or avoid it for financial text
Guard rate below 0.80 for one model on the bare prompt It drops qualifiers when not warned Reserve it for drafts a human will read against the source
Min score far below mean Occasional bad runs hide behind a good average Add runs and set a rule that every summary gets spot-checked
within_limit below 100 percent The model ignores length caps Add a length check to your workflow or tighten the prompt
Both models near 1.00 everywhere The sources are too easy Lower the word cap or add harder sources and rerun
Gap under 0.05 with overlapping bootstrap ranges No evidence of a real difference Call it a tie and choose on price or interface

Then archive the run. Copy outputs/, results.csv, report.md, cases.json and your notes file into a folder named with the date. The pinned model IDs inside meta.jsonl make the run reproducible in spirit, even though the vendors may retire or update those models later. Both vendors moved fast in November 2026, with Anthropic announcing Fable 5.2, Mythos 5.2 and Haiku 5.5 and OpenAI introducing GPT-6.2 Astra and Sol, so a rerun after the next release costs minutes and keeps the conclusion current.

Treat vendor benchmark claims as claims about other tasks. A score on a coding or workflow benchmark says little about whether a model keeps a negation in a 70-word summary. Claude vs ChatGPT for writing is really several tests, and summary fidelity is one of them. More head-to-head measurements sit in the AI and machine learning coverage.

Common Pitfalls #

Most bad conclusions from this kind of test trace back to a short list of mistakes. Check each before trusting a result.

  1. Comparing different surfaces. A Claude API call on Opus 5.5 against a ChatGPT Plus chat is not a like-for-like test. Apps add system prompts, memory and tool routing. Compare API with API or app with app, and record the plan name and displayed model label.
  2. Running each cell once. One sample per cell turns luck into a verdict. Five runs is the floor, and the min column exists to expose the bad draw that a single run would hide.
  3. Leaking the manifest into the prompt. If the prompt names the surcharge or the 2029 pledge, the model gets a pointer and the score measures obedience, not judgment. Name categories of trap, never specific facts.
  4. Narrow cue lists. The shelved example above is typical. A valid paraphrase fails because nobody listed the verb. Review failures by hand, widen cues with a defensible reason, and rescore both providers.
  5. Letting hidden reasoning eat the token ceiling. With a small ceiling, a thinking model can exhaust the budget before writing a word, and you save an empty file that scores zero. Keep the ceiling generous and delete zero-byte files before rerunning.
  6. Editing the manifest after seeing results. Rule changes made while staring at one model’s failures bias the test toward that model. Freeze the manifest, run, and log every later change with a reason.
  7. Pasting chat replies with extras. Introductory sentences, markdown bullets and trailing offers to revise change the word count and can introduce stray digits, which inflates the invented count. Save only the summary text.

Troubleshooting #

These are the failures most likely to appear, in roughly the order you will meet them.

  1. KeyError: 'CLAUDE_MODEL' or OPENAI_MODEL. The variable is not set in the current shell. Run echo $CLAUDE_MODEL in the same terminal that launches the harness, and re-export if it prints nothing.
  2. ModuleNotFoundError: No module named 'anthropic'. The virtual environment is not active. Run source .venv/bin/activate again, thenpip install --upgrade anthropic openai .
  3. Authentication error. The key is wrong, expired, or carries a trailing space or newline from copy-paste. Re-export it inside straight quotes. For OpenAI, also confirm the account has available credit.
  4. A model not found error. The model ID has a typo, or your key lacks access to that model. Copy the identifier from the provider’s model list rather than from a blog post.
  5. 429 rate limit. The built-in retry waits 3, 6 and 12 seconds. If a call still fails, rerun the same command, since existing files are skipped and only the gaps are refilled.
  6. Blank summaries that score near zero. Zero-byte files usually mean the reply contained no text blocks, often because reasoning consumed the ceiling. Raisemax_tokens in the Claude call, delete empty files withfind outputs -size 0 -delete , and rerungenerate .
  7. The gold test fails after editing cases.json. The failing case id is in the assertion message. Usually the gold summary exceeds 70 words or a manifest form no longer matches the gold text. Score the gold summary in a Python shell and read the guard statuses.
  8. ValueError: not enough values to unpack in collect. A file name lacks the double underscore. Rename it to case-id__run.txt , for examplewater-board__2.txt .
  9. The report says no outputs found. You are probably outside the project folder, or the files sit at the wrong depth. The glob expectsoutputs/<provider>/<prompt>/<file>.txt , three levels down.
  10. Every model scores 1.00. The sources are saturated, or a provider folder accidentally holds gold text. Open two files by hand, then tighten the word cap or add a harder source.

Advanced Tips #

Once the baseline works, five upgrades make the test sharper without changing its logic.

Add uncertainty estimates. A mean from 15 summaries hides how wide the doubt is. Save this as bootstrap.py and run it after the report. It resamples each cell 2,000 times and prints a 95 percent range.

import collections
import csv
import random
import statistics

cells = collections.defaultdict(list)
with open("results.csv", encoding="utf-8") as handle:
    for row in csv.DictReader(handle):
        cells[(row["provider"], row["prompt"])].append(float(row["faithfulness"]))

random.seed(7)
for (provider, prompt), scores in cells.items():
    means = sorted(statistics.mean(random.choices(scores, k=len(scores))) for _ in range(2000))
    print(f"{provider}/{prompt}: mean {statistics.mean(scores):.3f}, 95% range {means[50]:.3f} to {means[1949]:.3f}")

Here is the output on the three control sets:

gold/control: mean 1.000, 95% range 1.000 to 1.000
lead/control: mean 0.617, 95% range 0.435 to 0.750
planted/control: mean 0.233, 95% range 0.176 to 0.289

When the ranges for two models overlap heavily, you do not have a winner yet, so add runs. When they separate cleanly, you can say so with more confidence than a bare average allows.

Test the other routes to the same model. Microsoft’s community blog reports that GPT-6.2 Astra and GPT-6.2 Sol are generally available in Microsoft Foundry, and Anthropic lists its newest Claude models on AWS, Google Cloud and Azure as well. Running the same prompts through a cloud route checks whether the hosting surface changes behavior. Add a provider function and a folder label, and the report handles the rest.

Use batch pricing for big runs. Anthropic’s pricing page advertises a 50 percent saving with batch processing, which suits a 10-run sweep that needs no instant replies.

Extend the checks. Add a direction guard for every claim that rose or fell, the way the hardware line works in the earnings case. Add unit checks so that a figure in millions cannot silently become a figure in thousands. Add a second language by translating the cue lists, because number and entity matching already work across languages. If readers of your summaries also need a plain-language version, the reading-level rewrite test scores that axis separately.

Add a judge model only as a second opinion. If you want a model to rate coherence or tone, record its scores in a separate column and track how often it disagrees with the deterministic score. Never let it override a failed guard. Schedule the whole pipeline with cron or a CI job after each major release, commit the dated archive, and you have a time series of how both assistants change on the task you actually care about.

Frequently Asked Questions #

Who wins the Claude vs ChatGPT summary test?

No honest answer exists without your own sources. Both vendors shipped new models in November 2026, and published benchmarks mostly measure coding and workflow tasks rather than fact retention in short summaries. The harness gives you a measured answer for your material, with the guard rate and invented count as the numbers to weigh most.

Can I run this Claude vs ChatGPT test without API keys?

Yes. Use manual mode: paste each prompt into a fresh conversation in the Claude app and the ChatGPT app, save each reply as a text file in the right folder, and run the report command. Record the plan and the displayed model label in a notes file, since app model names can differ from API IDs.

Why not ask a model to grade the summaries?

A judge model adds its own errors, may prefer text that resembles its own style, and can return different scores on reruns. A deterministic scorer returns identical numbers every time, and you can inspect the rule behind each one. Use a judge model, if at all, as a labeled second column.

How many runs per cell do I need?

Start with five per source, which gives 15 summaries per provider and prompt with three sources. Add runs until the bootstrap ranges for the models you care about either separate or clearly overlap. Treat gaps under 0.05 as ties until they do.

Does a high faithfulness score mean the summary is good?

No. It means the planted facts survived. A summary can keep every number and still read badly, bury the main point, or ignore the audience. Pair this test with readability and voice checks before choosing a tool for a writing workflow.

Do I need to retest when a model updates?

Yes, and the cost is low. Change the model environment variables, rerun generation into a dated folder, and compare reports. Vendors retire, rename and replace models often, so a conclusion without a date and a model ID loses value quickly.

── more in #large-language-models 4 stories · sorted by recency
── more on @anthropic 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/claude-vs-chatgpt-su…] indexed:0 read:38min 2026-11-19 · —