Claude vs ChatGPT Headline Test: 12 Steps, 80 Min A 12-step, roughly 80-minute Python harness lets editorial teams compare Claude Fable 5.2 and OpenAI's GPT-6.2 Astra and Sol on headline writing by pushing 10 article briefs through each model, applying seven automatic rules to every headline, checking meta descriptions and sending anonymized lead headlines to blind human raters, at a cost of about $2.40 in API fees for a premium-versus-premium run. The test targets rule violations such as invented statistics, padded phrasing and repeated sentence skeletons, with number grounding flagged as the most revealing check because a single fabricated figure in a published headline becomes a correction. Anthropic released Claude Fable 5.2 on November 9 and OpenAI put GPT-6.2 Astra and GPT-6.2 Sol into preview on November 17, both leaning on coding, science and professional agent work. Anthropic released Claude Fable 5.2 on November 9, and OpenAI put GPT-6.2 Astra and GPT-6.2 Sol into preview on November 17, according to a weekly AI recap https://amdatalakehouse.substack.com/p/ai-weekly-gpt-62-gemini-4-hydrogen . Both launches lean on coding, science and professional agent work. Neither tells a copywriter which model produces the better 58-character headline. That gap is the reason to run your own Claude vs ChatGPT test, and headlines are the cheapest place to start. A headline is short enough to score by machine and tight enough to expose sloppy habits: invented statistics, padded phrasing, buzzwords, the same sentence skeleton repeated eight times. The Python harness below pushes 10 article briefs through each model, applies seven automatic rules to every headline, checks the meta description, and then hands anonymized lead headlines to human raters. It takes 12 steps and about 80 minutes. A premium-versus-premium run costs about $2.40 in API fees. Long-form tests such as a voice matching test https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ tell you whether a model can hold a style across 2,000 words. A headline test answers a different question: does the model obey hard constraints when every character counts? Most editorial teams need both answers, and the second one costs a fraction of the first. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Headlines Make a Sharper Claude vs ChatGPT for Writing Test Full-article comparisons mix too many variables. Length, structure, topic knowledge, formatting habits and rater fatigue all leak into the verdict, and by the time a reader finishes the third 1,500-word sample, the first one is a blur. A headline strips that noise away. The model gets a brief, a handful of facts and a few hard rules. It either follows them or it does not. The rules are binary, and binary rules scale. Ten briefs, three runs each and eight headlines per call produce 240 headlines per model. A script scores all 240 in well under a second, so you can rerun the whole test after every prompt tweak instead of rating a sample by hand once and hoping it generalizes. That speed changes how you work. You stop arguing about which model feels better and start looking at a table of failure rates. The most revealing rule is number grounding. Each brief carries a short list of facts, and the prompt tells the model to use only numbers from that list. Models love a punchy statistic, so some headlines arrive with a claim like 47% faster that appears nowhere in the brief. A single invented number in a published headline is a correction waiting to happen, which makes this the rule an editor should care about most. If you have already run a writing rules compliance test https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ , the checks will feel familiar. The difference is scale: on short text, one miss costs you 1 of 8 options rather than one buried sentence in a long draft. Machines cannot judge which headline a reader will click, so the harness adds a blind human pass. Each rater sees one lead headline per model for every brief, with the model names hidden and the order shuffled. A rater finishes 10 briefs in about 10 minutes. That is short enough that you can recruit three colleagues without a speech. Be clear about what the result proves. A headline score measures headline obedience on your briefs, your rules and your prompt. It does not crown an overall winner for writing, and a model that loses here may still win on dialogue, long-form structure or tone. Treat the output as one data point in a larger evaluation, and keep the raw run files so you can rescore them when you change the rules. What the November 2026 Releases Change for Claude vs ChatGPT Start with what Anthropic shipped. Claude Fable 5.2 and Claude Mythos 5.2 are its newest models, announced in November 2026, and Anthropic’s own page frames them as models for “daunting technical work.” Fable 5.2 generates output up to 30% faster than Fable 5.1 and costs $8 per million input tokens and $40 per million output tokens. An optional fast mode runs up to 2.5 times faster at $16 and $80. Subscribers on Pro, Max, Ultra, Team and Enterprise plans can use Fable 5.2, and developers reach it through the Claude Platform, AWS, Google Cloud and Microsoft’s cloud. Mythos 5.2 is a different animal. Anthropic limits it to vetted organizations in its verification programs, so a normal team cannot test it and this tutorial skips it. Anthropic’s lineup also lists Opus 5.5, Sonnet 5.5 and Haiku 5.5. The company says Opus 5.5 costs 40% less to run than Opus 5 on typical workloads, and that Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. It calls Haiku 5.5 its “fastest, most capable small model,” built for high-volume work such as summarization. Fable 5.2 and GPT-6 Sol sit at very different prices, so a fair test also pairs a small Claude tier against OpenAI’s cheap model. The config in Step 2 leaves room for that third arm. Read the current prices for those tiers off Anthropic’s pricing page before you add them, because this tutorial only prints figures it could verify. On the OpenAI side, GPT-6.2 Astra targets advanced reasoning, coding and complex professional work, while GPT-6.2 Sol targets faster, cheaper multistep tasks. Both were released on November 17 in the API, ChatGPT Work and Codex, and on November 20 they entered preview in ChatGPT Work and Codex on desktop. OpenAI also announced GPT-6 Sol and GPT-6 Luna, with API identifiers gpt-6-sol and gpt-6-luna , as covered in a November 20 recap https://deyronlabs.com/en/news/ai-weekly-recap-2026-11-20/ . GPT-6 Sol costs $2 per million input tokens and $10 per million output tokens. GPT-6 Luna costs $0.10 and $0.50. Because the 6.2 pair is also in the API, the harness can test it there, but this run tests the GPT-6 pair. The harness accepts any model ID, so you can add the 6.2 models whenever you like. Consumer plans matter if you test inside the apps instead of the API. OpenAI’s pricing page https://chatgpt.com/pricing lists ChatGPT Free at $0, Go at $8 per month in the US, Plus at $20 and Pro in several tiers. Claude’s plan names are Pro, Max, Ultra, Team and Enterprise, and Anthropic’s pricing page https://claude.com/pricing holds the current monthly numbers. OpenAI’s release notes describe Free and Go users on Luna and Plus and above on Sol, yet the original announcement said Sol and Luna were not yet in regular chat and would roll out gradually. The mapping has shifted from day to day, so the model label in your own picker is the only record you can trust. | Model | Status on Nov 21, 2026 | API price per 1M tokens input / output | Role in this test | |---|---|---|---| | Claude Fable 5.2 | Released Nov 9 | $8 / $40 | Premium Claude arm fable-5.2 | | Claude Fable 5.2 fast mode | Optional, up to 2.5x faster | $16 / $80 | Optional latency arm | | GPT-6 Sol | Announced the week of Nov 16 | $2 / $10 | Premium OpenAI arm gpt-6-sol | | GPT-6 Luna | Announced alongside Sol | $0.10 / $0.50 | Budget OpenAI arm gpt-6-luna | | Claude Mythos 5.2 | Vetted organizations only | $8 / $40 | Excluded, no general access | | GPT-6.2 Astra and Sol | Released Nov 17 in the API, Work and Codex; preview in ChatGPT Work and Codex on desktop Nov 20 | Not tested | Optional, add by model ID | Prerequisites and Versions for the Headline Harness The project needs little beyond a terminal. Gather these before you start, because a missing key or a guessed model ID burns the most time. - Python 3.11 or newer. The offline run that produced the sample output below used Python 3.12.3. Nothing in the code needs a newer feature than 3.11. - The anthropic and openai Python packages, latest release of each. Install them in a virtual environment. I name no version numbers because both SDKs ship often. Run pip freeze requirements.lock after your first successful live run so the test stays reproducible. - An Anthropic API key exported as ANTHROPIC API KEY , with access to Claude Fable 5.2 on the Claude Platform. - An OpenAI API key exported as OPENAI API KEY , with access to gpt-6-sol and gpt-6-luna . - The exact Claude model ID , copied from Anthropic’s model documentation into the CLAUDE MODEL ID environment variable. Do not guess it. A model name you remember from a blog post and a model ID the API accepts are different strings. - Three human raters for the blind pass, each with about 10 minutes free. Pick people who did not write the briefs. - A budget of roughly $2.50 and 80 minutes for the premium pair. Under the assumptions stated later, one full run of that pair costs about $2.39. One more disclosure about what was run. The offline path, which uses a mock model, ran end to end on Python 3.12.3, and every number in the sample-output section comes from that run. The two live adapters follow each SDK’s documented call shape, but only your first live call can confirm that a vendor has not changed a parameter since this was written. Start with --limit 1 so a mistake costs pennies. Why do versions matter so much in a Claude vs ChatGPT comparison? Both vendors changed their lineups within the same fortnight, and the Latest option in ChatGPT’s model menu now points at GPT-6, while an explicit GPT-5.6 Sol selection stays on GPT-5.6. A result that names only ChatGPT or only Claude is unreproducible. Record the model ID, the plan or API tier, the date and the prompt for every run, and the harness does that automatically by saving a models.json file beside the raw responses. If your priority is spend per usable output rather than spend per call, pair this test with a cost per accepted 1,000 words https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/ calculation. Steps 1 to 3: Project Setup, Model Pinning and Briefs Step 1: Create the project and install the SDKs Make a clean directory, a virtual environment and the two SDKs. Keep the keys in environment variables rather than files, so a stray git add cannot publish them. mkdir headline bench && cd headline bench python3 -m venv .venv source .venv/bin/activate python --version pip install anthropic openai mkdir tests runs echo "runs/" .gitignore export ANTHROPIC API KEY="paste-your-key-here" export OPENAI API KEY="paste-your-key-here" export CLAUDE MODEL ID="paste-the-id-from-anthropic-docs" The python --version line is a checkpoint. If it prints anything older than 3.11, recreate the environment with a newer interpreter before you go on. The runs/ entry in .gitignore matters more than it looks, because each run folder holds raw model responses and you do not want 30 megabytes of JSON in your repository history. Step 2: Pin the models and prices in config.py Every constant that can change lives in one file. The MODELS dictionary maps a short label to a provider, an exact model ID and the price per million tokens. The labels appear in every report column, so choose names you will recognize in six months. The prices below match the figures in the table above, and the comment reminds you to re-check them. The Claude ID reads from the environment on purpose, because this tutorial does not invent identifiers. python import os from pathlib import Path ROOT = Path file .parent RUNS DIR = ROOT / "runs" RUNS PER BRIEF = int os.getenv "RUNS PER BRIEF", "3" HEADLINES PER CALL = 8 MAX OUTPUT TOKENS = int os.getenv "MAX OUTPUT TOKENS", "4000" Prices are USD per million tokens, copied from each vendor's pricing page on 2026-11-21. Re-check them before you trust the cost column. MODELS = { "fable-5.2": { "provider": "anthropic", "model id": os.getenv "CLAUDE MODEL ID", "" , "price in": 8.0, "price out": 40.0, }, "gpt-6-sol": { "provider": "openai", "model id": os.getenv "OPENAI SOL ID", "gpt-6-sol" , "price in": 2.0, "price out": 10.0, }, "gpt-6-luna": { "provider": "openai", "model id": os.getenv "OPENAI LUNA ID", "gpt-6-luna" , "price in": 0.10, "price out": 0.50, }, } BANNED = "game-changer", "game changer", "revolutionary", "unlock", "groundbreaking", "mind-blowing", "next-level", "cutting-edge", "supercharge", "skyrocket", "you won't believe", HEADLINE MAX CHARS = 60 META MIN CHARS = 120 META MAX CHARS = 155 NEAR DUPLICATE JACCARD = 0.6 measured on words outside the keyword, digits ignored The BANNED list is the part you will edit most. It holds phrases that mark a headline as machine-flavored in your house style. Swap in your own list. The NEAR DUPLICATE JACCARD threshold is explained in Step 7, and RUNS PER BRIEF controls how many times each model sees each brief. Three runs is the minimum that exposes variance. One run per brief tells you about luck. Step 3: Write 10 briefs with facts the models must respect Each brief has an ID, a primary keyword, an audience and three facts. The facts are the source of truth for the number-grounding rule, so write them like an editor handing material to a writer: plain sentences with the real figures in them. Every brief in this file includes a step count and a time in minutes, because the offline mock model uses them. Your real briefs can carry any facts you like. {"id": "ssh-keys", "keyword": "rotate SSH keys", "audience": "Linux admins", "facts": "The procedure has 6 steps.", "It takes about 25 minutes per server.", "Ed25519 keys replace 2048-bit RSA keys." }, {"id": "pg-backup", "keyword": "Postgres backups", "audience": "backend developers", "facts": "The setup has 8 steps.", "A full restore test takes 40 minutes.", "Nightly dumps are kept for 14 days." }, {"id": "dns-audit", "keyword": "DNS audit", "audience": "IT managers", "facts": "The audit has 5 steps.", "It takes 30 minutes for a single domain.", "Stale CNAME records were found on 3 of 12 test domains." }, {"id": "docker-slim", "keyword": "Docker image size", "audience": "DevOps engineers", "facts": "The guide has 7 steps.", "The full run takes 35 minutes.", "It cuts a test image from 1,240 MB to 180 MB." }, {"id": "mfa-rollout", "keyword": "MFA rollout", "audience": "security leads", "facts": "The rollout plan has 9 steps.", "Setup takes 60 minutes per team.", "A pilot group of 50 users goes first for 2 weeks." }, {"id": "log-retention", "keyword": "log retention", "audience": "compliance teams", "facts": "The policy template has 4 steps.", "Setup takes 45 minutes.", "Retention ranges from 30 to 365 days across 4 log classes." }, {"id": "k8s-limits", "keyword": "Kubernetes resource limits", "audience": "platform engineers", "facts": "The walkthrough has 10 steps.", "It takes 50 minutes.", "Right-sizing cut test pod memory by 38% on a 3-node cluster." }, {"id": "git-hooks", "keyword": "Git pre-commit hooks", "audience": "team leads", "facts": "The setup has 6 steps.", "It takes 20 minutes to install across a repo.", "Secret scanning runs in under 3 seconds." }, {"id": "wifi-guest", "keyword": "guest Wi-Fi isolation", "audience": "small business owners", "facts": "The router change has 5 steps.", "It takes 15 minutes.", "Guests get their own VLAN 30." }, {"id": "cert-renewal", "keyword": "TLS certificate renewal", "audience": "site reliability engineers", "facts": "The automation has 7 steps.", "The setup takes 40 minutes.", "Certificates renew 30 days before expiry and the alert fires after 2 failed attempts." } Keep the briefs realistic for your own site. A tutorial publisher should test tutorial briefs, a news desk should test briefs with names, dates and dollar amounts. Vary the keyword length, too. A three-word keyword like Kubernetes resource limits leaves little room in a 60-character headline, and that pressure is exactly what separates models that count characters from models that guess. The same discipline applies to the source material in a source-grounded writing audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ : the less ambiguity in the brief, the more any failure belongs to the model. Steps 4 to 6: Prompt, Provider Layer and Generation Run Step 4: Write one prompt that both models receive Fairness starts with an identical prompt. The system message sets the role. The user message states the rules, asks for JSON in one exact shape and appends the brief. It names the number rule twice, once as a requirement and once as a prohibition, because models skim. SYSTEM = "You are a headline editor for a technology news site. " "Follow the style rules exactly and reply with JSON only." USER TEMPLATE = """Write {n} distinct headlines and one meta description for the article brief below. Rules: - Every headline is 60 characters or fewer. - Every headline contains the primary keyword and one specific number taken from the facts. - Use only numbers that appear in the facts. Never invent a statistic. - No em dashes. Do not use these words: {banned}. - Order the headlines best first. - The meta description is 120 to 155 characters and contains the primary keyword. Return exactly this JSON shape: {{"headlines": "..." , "meta": "..."}} BRIEF JSON: {brief}""" Three design choices deserve a note. First, the prompt sends no temperature. Some current models reject or ignore sampling parameters, and a parameter that works for one vendor and errors for the other would break the symmetry. Second, asking for best-first order gives the blind pass a fair target, since the lead headline is the one the model itself would publish. Third, the brief travels as JSON after a fixed marker, which lets the mock model parse it and lets you diff prompts between runs. This prompt is only the first stage of a real editorial flow. If you want to see how drafting, editing and headline passes chain together, the writing pipeline test https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ covers that setup. Step 5: Build the provider layer One function, call model , hides the vendor differences. It takes a config entry, a system message and a user message, and returns the text, the token counts and the wall-clock seconds. The Anthropic adapter uses the Messages API with a system parameter. The OpenAI adapter uses the Responses API with instructions and input . Both read token usage off the response so the report can price each run. The third branch, mock , is the offline stand-in: it produces headlines from templates, some good and some deliberately flawed, so you can exercise every later step without spending money. python import json import random import re import time from config import MAX OUTPUT TOKENS def anthropic cfg, system, user : from anthropic import Anthropic resp = Anthropic .messages.create model=cfg "model id" , max tokens=MAX OUTPUT TOKENS, system=system, messages= {"role": "user", "content": user} , text = "".join b.text for b in resp.content if getattr b, "type", "" == "text" return text, resp.usage.input tokens, resp.usage.output tokens def openai cfg, system, user : from openai import OpenAI resp = OpenAI .responses.create model=cfg "model id" , instructions=system, input=user, max output tokens=MAX OUTPUT TOKENS, return resp.output text, resp.usage.input tokens, resp.usage.output tokens Offline stand-in so you can test the whole pipeline without spending a cent. GOOD = "{kw}: {a} Steps, {b} Min", "{kw} in {a} Steps", "How to Handle {kw} in {b} Minutes", "{kw} Done Right: {a} Steps", "A {b}-Minute Fix for {kw}", "Fix Your {kw} in {b} Minutes", "{a} Steps to Better {kw}", "Tackle {kw} in Under {b} Minutes", BAD = "Unlock {kw} With This One Trick", "{kw} Explained, Fast", "{kw}: 87% Faster Setups", "How to Get {kw} Right Without the Headaches That Slow Teams Down", "Why {kw} Matters", def mock cfg, system, user, seed : brief = json.loads user.split "BRIEF JSON:", 1 1 facts = " ".join brief "facts" a = re.search r" \d+ steps", facts .group 1 b = re.search r" \d+ minutes", facts .group 1 rng = random.Random seed bad rate = 0.12 if "steady" in cfg "model id" else 0.40 good, bad = GOOD : , BAD : rng.shuffle good rng.shuffle bad lines = for in range 8 : pool = bad if rng.random < bad rate and bad or not good else good lines.append pool.pop .format kw=brief "keyword" , a=a, b=b meta = f"Learn how to handle {brief 'keyword' } with {a} clear steps and a {b}-minute plan that {brief 'audience' } can follow today." if "steady" in cfg "model id" : meta += " Includes a rollback checklist." return json.dumps {"headlines": lines, "meta": meta} , 400, 220 def call model cfg, system, user, seed="" : start = time.perf counter kind = cfg "provider" if kind == "anthropic": text, tin, tout = anthropic cfg, system, user elif kind == "openai": text, tin, tout = openai cfg, system, user elif kind == "mock": text, tin, tout = mock cfg, system, user, seed else: raise ValueError f"unknown provider {kind}" return { "text": text, "tokens in": tin, "tokens out": tout, "seconds": round time.perf counter - start, 3 , } Notice that the mock draws a different mix of flawed headlines for the steady and sloppy labels, and seeds its random generator from the model, brief and run number. Seeding makes the dry run reproducible, which means the sample report below is the same on your machine as on mine. The live adapters import their SDKs inside the function. That keeps the offline path working on a machine where neither SDK is installed yet. Step 6: Generate headlines and save every raw response The generator loops over models, briefs and runs, calls the provider, parses the JSON and writes one file per call. It stores the raw text next to the parsed result. That habit pays off the first time a rule changes, because you can rescore old runs without paying for new ones. A parse failure does not stop the run. The record gets parsed: null , and the report counts it as a failure, which is the honest treatment for a model that cannot follow a reply-with-JSON-only instruction. python import argparse import json import re from datetime import datetime from config import BANNED, HEADLINES PER CALL, MODELS, ROOT, RUNS DIR, RUNS PER BRIEF from providers import call model SYSTEM = "You are a headline editor for a technology news site. " "Follow the style rules exactly and reply with JSON only." USER TEMPLATE = """Write {n} distinct headlines and one meta description for the article brief below. Rules: - Every headline is 60 characters or fewer. - Every headline contains the primary keyword and one specific number taken from the facts. - Use only numbers that appear in the facts. Never invent a statistic. - No em dashes. Do not use these words: {banned}. - Order the headlines best first. - The meta description is 120 to 155 characters and contains the primary keyword. Return exactly this JSON shape: {{"headlines": "..." , "meta": "..."}} BRIEF JSON: {brief}""" def build prompt brief : return USER TEMPLATE.format n=HEADLINES PER CALL, banned=", ".join BANNED , brief=json.dumps brief def parse json text : text = re.sub r" ?:json ?", "", text start, end = text.find "{" , text.rfind "}" if start == -1 or end == -1: return None try: return json.loads text start : end + 1 except json.JSONDecodeError: return None def main : ap = argparse.ArgumentParser ap.add argument "--models", default="fable-5.2,gpt-6-sol" ap.add argument "--mock", action="store true", help="offline stand-in models" ap.add argument "--limit", type=int, default=0, help="only the first N briefs" args = ap.parse args briefs = json.loads ROOT / "briefs.json" .read text if args.limit: briefs = briefs : args.limit if args.mock: models = { "mock-steady": {"provider": "mock", "model id": "mock-steady", "price in": 0, "price out": 0}, "mock-sloppy": {"provider": "mock", "model id": "mock-sloppy", "price in": 0, "price out": 0}, } else: models = {k: MODELS k for k in args.models.split "," } for label, cfg in models.items : if not cfg "model id" : raise SystemExit f"Set the model ID for {label} see config.py before running." run dir = RUNS DIR / datetime.now .strftime "%Y%m%d-%H%M%S" for label in models: run dir / label .mkdir parents=True run dir / "models.json" .write text json.dumps models, indent=2 for label, cfg in models.items : for brief in briefs: for run in range 1, RUNS PER BRIEF + 1 : result = call model cfg, SYSTEM, build prompt brief , seed=f"{label}-{brief 'id' }-{run}" result "parsed" = parse json result "text" result "brief id" , result "run" , result "model" = brief "id" , run, label path = run dir / label / f"{brief 'id' }-{run}.json" path.write text json.dumps result, indent=2 status = "ok" if result "parsed" else "PARSE FAIL" print f"{label:12} {brief 'id' :14} run {run} {result 'tokens out' : 5} tok {result 'seconds' : 5}s {status}" print f"\nRun saved to {run dir}" if name == " main ": main Run the offline path first. python generate.py --mock --limit 1 You should see one line per call with the model label, brief ID, run number, output tokens, seconds and an ok flag, then a Run saved to path. When that works, switch to a live smoke test with a single brief: python generate.py --models fable-5.2,gpt-6-sol --limit 1 Check the saved JSON files by eye. The text field should hold a JSON object with eight headlines and a meta description. If either model wraps the JSON in commentary or code fences, the parser in parse json strips the fences and trims to the outer braces. That covers the common cases without hiding real failures. Steps 7 to 9: Scoring Rules, Meta Checks and the Report Step 7: Score every headline against seven rules The scoring module turns taste into arithmetic. Each headline gets seven true-or-false results, and a headline counts as clean only when all seven pass. The table lists what each rule catches and where the threshold sits. | Rule | What it catches | Threshold | |---|---|---| | length ok | Headlines that risk truncation in search results | 60 characters or fewer | | has number | Vague headlines with no specific figure | At least one digit | | numbers grounded | Invented statistics | Every number appears in the brief’s facts | | no banned | House-style buzzwords | No phrase from the BANNED list | | no em dash | Em and en dashes | None present | | has keyword | Missing primary keyword | Every keyword word present, any order | | not duplicate | Near-copies inside one call | Word overlap below 0.6 | Two details in the code need explaining. The first is how numbers are compared. The regular expression pulls out digit groups, strips trailing punctuation and removes thousands separators, so 1,240 in a brief matches 1240 in a headline. Without that normalization a model that copies a figure exactly would still fail the grounding rule. The second detail is the duplicate check. Every headline must contain the primary keyword and a number, so two perfectly different headlines will share a lot of words. The jaccard function therefore ignores the keyword’s words and all digits before it measures overlap. What remains is phrasing: Fix Your X in 25 Minutes and Tackle X in Under 25 Minutes score as different, while X in 6 Steps for Admins next to X in 6 Steps for Admins Today scores as a near-copy. python import re from config import BANNED, HEADLINE MAX CHARS, META MAX CHARS, META MIN CHARS, NEAR DUPLICATE JACCARD, NUM RE = re.compile r"\d \d, ?:\.\d+ ?" def numbers in text : return {n.rstrip ",." .replace ",", "" for n in NUM RE.findall text } def words text : return set re.findall r" a-z0-9 +", text.lower def jaccard a, b, ignore=frozenset : wa = {w for w in words a if w not in ignore and not w.isdigit } wb = {w for w in words b if w not in ignore and not w.isdigit } return len wa & wb / len wa | wb if wa | wb else 0.0 def has keyword text, keyword : return words keyword <= words text def banned hit text : low = text.lower return any b in low for b in BANNED def check headline text, brief, others : fact numbers = numbers in " ".join brief "facts" nums = numbers in text kw words = words brief "keyword" return { "length ok": len text <= HEADLINE MAX CHARS, "has number": bool nums , "numbers grounded": nums <= fact numbers, "no banned": not banned hit text , "no em dash": ", " not in text and "\u2013" not in text, "has keyword": has keyword text, brief "keyword" , "not duplicate": all jaccard text, o, kw words < NEAR DUPLICATE JACCARD for o in others , } def check meta text, brief : fact numbers = numbers in " ".join brief "facts" return { "length ok": META MIN CHARS <= len text <= META MAX CHARS, "has keyword": has keyword text, brief "keyword" , "numbers grounded": numbers in text <= fact numbers, "no banned": not banned hit text , "no em dash": ", " not in text and "\u2013" not in text, } def score call parsed, brief : heads = h for h in parsed.get "headlines", if isinstance h, str head results = for i, h in enumerate heads : others = heads :i + heads i + 1 : head results.append {"text": h, "checks": check headline h, brief, others } meta = parsed.get "meta", "" if isinstance parsed.get "meta", "" , str else "" return {"headlines": head results, "meta": {"text": meta, "checks": check meta meta, brief }} The rules are deliberately dumb. A smarter rule, such as whether a headline sounds natural belongs to the human pass. Dumb rules have one huge advantage: you can write a unit test for each of them in two lines. The test file below covers number normalization, invented numbers, a clean headline, an over-long headline, banned words with dashes, and near-duplicates. It runs with plain Python, so you do not need a test runner. python import sys from pathlib import Path sys.path.insert 0, str Path file .resolve .parent.parent from checks import check headline, jaccard, numbers in BRIEF = {"keyword": "rotate SSH keys", "facts": "The procedure has 6 steps.", "It takes about 25 minutes." } def test numbers in strips commas and punctuation : assert numbers in "Cut 1,240 MB to 180 MB." == {"1240", "180"} def test invented number is flagged : r = check headline "Rotate SSH Keys: 87% Faster", BRIEF, assert r "numbers grounded" is False def test clean headline passes every rule : r = check headline "Rotate SSH Keys in 6 Steps, 25 Min", BRIEF, assert all r.values def test long headline fails length : assert check headline "Rotate SSH Keys in 6 Steps " + "x" 40, BRIEF, "length ok" is False def test banned word and em dash : r = check headline "Unlock Rotate SSH Keys, 6 Steps", BRIEF, assert r "no banned" is False and r "no em dash" is False def test near duplicates flagged : other = "Rotate SSH Keys in 6 Steps for Admins" kw = {"rotate", "ssh", "keys"} assert jaccard other, "Rotate SSH Keys in 6 Steps for Admins Today", kw 0.6 assert check headline "Rotate SSH Keys in 6 Steps for Admins Today", BRIEF, other "not duplicate" is False if name == " main ": tests = v for k, v in dict globals .items if k.startswith "test " for fn in tests: fn print "PASS", fn. name print f"{len tests } tests passed" Run it before every change to the rules: python tests/test checks.py PASS test numbers in strips commas and punctuation PASS test invented number is flagged PASS test clean headline passes every rule PASS test long headline fails length PASS test banned word and em dash PASS test near duplicates flagged 6 tests passed Step 8: Check the meta description the same way The same module scores the meta description with five rules: length between 120 and 155 characters, primary keyword present, numbers grounded in the facts, no banned phrases and no dashes. The function is check meta , and it sits in the same module. Meta descriptions fail in a different pattern from headlines. Models overshoot the length limit more often than they invent numbers, because a 155-character window feels tiny while writing and huge while counting. Keep the meta scoring separate from the headline scoring, as the report does, so a model that writes excellent headlines but sloppy descriptions shows up as exactly that. The same split works when you test revisions instead of fresh copy, as an editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ does with seeded errors. One design choice is worth defending. The meta check requires the keyword but does not require a number, because a description that crams in a figure just to satisfy a rule reads like a headline wearing a trench coat. Adjust it if your site’s house style says otherwise. Step 9: Build the report table The report loads the saved runs, scores each parsed response and prints a Markdown table with one column per model. It also writes the table to report.md inside the run folder, so every run carries its own scorecard. Cost comes from the token counts the providers returned, multiplied by the prices in config.py . A parse failure increments a counter and skips scoring, so a model that returns broken JSON cannot hide behind a high pass rate on the calls that worked. python import json import sys from collections import defaultdict from checks import score call from config import ROOT, RUNS DIR RULES = "length ok", "has number", "numbers grounded", "no banned", "no em dash", "has keyword", "not duplicate" def latest run : return sorted p for p in RUNS DIR.iterdir if p.is dir -1 def main run dir : briefs = {b "id" : b for b in json.loads ROOT / "briefs.json" .read text } models = json.loads run dir / "models.json" .read text stats = {} for label, cfg in models.items : s = defaultdict int for f in sorted run dir / label .glob " .json" : rec = json.loads f.read text s "calls" += 1 s "tok out" += rec "tokens out" s "tok in" += rec "tokens in" s "seconds" += rec "seconds" if not rec "parsed" : s "parse fail" += 1 continue scored = score call rec "parsed" , briefs rec "brief id" for h in scored "headlines" : s "headlines" += 1 for r in RULES: s r += h "checks" r s "all pass" += all h "checks" .values s "meta n" += 1 s "meta pass" += all scored "meta" "checks" .values s "cost" = s "tok in" / 1e6 cfg "price in" + s "tok out" / 1e6 cfg "price out" stats label = s pct = lambda a, b: f"{100 a / b:.1f}%" if b else "n/a" lines = "| Metric | " + " | ".join stats + " |", "|---|" + "---|" len stats for r in RULES: lines.append f"| {r} | " + " | ".join pct s r , s "headlines" for s in stats.values + " |" lines.append "| all 7 rules | " + " | ".join pct s "all pass" , s "headlines" for s in stats.values + " |" lines.append "| meta passes all | " + " | ".join pct s "meta pass" , s "meta n" for s in stats.values + " |" lines.append "| parse failures | " + " | ".join str s "parse fail" for s in stats.values + " |" lines.append "| mean output tokens/call | " + " | ".join f"{s 'tok out' / s 'calls' :.0f}" for s in stats.values + " |" lines.append "| mean seconds/call | " + " | ".join f"{s 'seconds' / s 'calls' :.1f}" for s in stats.values + " |" lines.append "| est. cost USD | " + " | ".join f"{s 'cost' :.4f}" for s in stats.values + " |" out = "\n".join lines run dir / "report.md" .write text out + "\n" print out if name == " main ": main import "pathlib" .Path sys.argv 1 if len sys.argv 1 else latest run Run it against the newest run folder with python report.py , or pass a folder path to rescore an older run. Rescoring is free, which is the whole point of saving raw responses in Step 6. Steps 10 to 12: Blind Vote, Dry Run and Live Run Step 10: Build the blind sheet for human raters Automatic rules say which headlines are legal. They cannot say which one a reader would rather click. The blind script takes the lead headline from run 1 of every brief and every model, shuffles the order with a fixed seed and writes a CSV that shows only the options, labeled A and B. The answer key goes into a separate JSON file that raters never see. The score command reads one or more completed rater CSVs, translates letters back to model labels and prints vote shares. python import csv import json import random import sys from collections import Counter from pathlib import Path from report import latest run LETTERS = "ABCDEFG" def make run dir, seed=2026 : models = json.loads run dir / "models.json" .read text rng = random.Random seed rows, key = , {} brief ids = sorted {f.stem.rsplit "-", 1 0 for f in run dir / next iter models .glob " .json" } for bid in brief ids: options = for label in models: rec = json.loads run dir / label / f"{bid}-1.json" .read text if rec "parsed" and rec "parsed" .get "headlines" : options.append label, rec "parsed" "headlines" 0 rng.shuffle options key bid = {LETTERS i : label for i, label, in enumerate options } row = {"brief id": bid, "pick": ""} for i, , text in enumerate options : row f"option {LETTERS i }" = text rows.append row cols = "brief id" + f"option {LETTERS i }" for i in range len models + "pick" with open run dir / "blind sheet.csv", "w", newline="" as fh: w = csv.DictWriter fh, fieldnames=cols w.writeheader w.writerows rows run dir / "blind key.json" .write text json.dumps key, indent=2 print f"Wrote {run dir / 'blind sheet.csv'} {len rows } rows . Give raters the CSV only, never the key." def score run dir, rater files : key = json.loads run dir / "blind key.json" .read text wins, total = Counter , 0 for path in rater files: for row in csv.DictReader open path : pick = row "pick" .strip .upper if pick in key row "brief id" : wins key row "brief id" pick += 1 total += 1 for label, n in wins.most common : print f"{label:12} {n: 3} of {total} votes {100 n / total:.0f}% " if name == " main ": cmd = sys.argv 1 run = latest run if cmd == "make": make run elif cmd == "score": score run, Path p for p in sys.argv 2: Brief your raters in three sentences. Tell them to pick the headline they would click, tell them to leave the cell blank if the two feel equal, and tell them not to guess which model wrote which. Send the CSV only. If you want to know how raters behave on longer text, the blind test of full drafts https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ uses the same logic with 1,000-word samples. Step 11: Run the full offline dry run Before you spend a cent, run the complete chain on the mock models. Each command should finish in about a second. python tests/test checks.py python generate.py --mock python report.py python blind.py make If all four succeed, the pipeline is sound and any later failure comes from credentials, model access or the vendors’ APIs. That separation saves hours of guessing. Step 12: Run live, then read the results with discipline Export your keys and the Claude model ID, run a one-brief smoke test, then run the full set. python generate.py --models fable-5.2,gpt-6-sol --limit 1 python generate.py --models fable-5.2,gpt-6-sol python report.py python blind.py make Send blind sheet.csv to three raters and collect their files. Then run python blind.py score rater1.csv rater2.csv rater3.csv . Before you read any numbers, write down your decision rule. A sensible one: a model wins a rule only when it leads by at least 10 percentage points on that rule and the lead holds in each of the three runs. Three raters voting on 10 briefs give you 30 votes, and a 17 to 13 split sits well inside the noise of 30 coin flips, so treat anything closer than about 20 to 10 as a tie. When the automatic rules and the human votes disagree, trust the rules for obedience and the humans for appeal, and say so in your write-up. Sample Output From the Offline Run Every number in this section comes from the mock models, not from Claude or ChatGPT. The mock steady model produces few flawed headlines and the mock sloppy model many, so the table shows the pipeline catching a known difference. Your live numbers will look different. The report for a full 10-brief offline run: | Metric | mock-steady | mock-sloppy | |---|---|---| | length ok | 98.8% | 91.7% | | has number | 90.0% | 66.7% | | numbers grounded | 97.9% | 91.2% | | no banned | 96.2% | 90.4% | | no em dash | 100.0% | 100.0% | | has keyword | 100.0% | 100.0% | | not duplicate | 100.0% | 100.0% | | all 7 rules | 87.9% | 57.9% | | meta passes all | 70.0% | 30.0% | | parse failures | 0 | 0 | | mean output tokens/call | 220 | 220 | | mean seconds/call | 0.0 | 0.0 | | est. cost USD | 0.0000 | 0.0000 | Read it from the bottom up. Cost and seconds are zero because the mock costs nothing and answers instantly. Parse failures are zero because the mock always emits valid JSON, which a live model will not always do. The two rows that matter most are numbers grounded and all 7 rules . The sloppy model fails grounding on far more headlines than the steady one and lands clean headlines much less often. To see how a single bad call looks, score one saved response headline by headline: How to Get rotate SSH keys Right Without the Headaches That Slow Teams Down | 'length ok', 'has number' rotate SSH keys Done Right: 6 Steps | PASS Fix Your rotate SSH keys in 25 Minutes | PASS Why rotate SSH keys Matters | 'has number' 6 Steps to Better rotate SSH keys | PASS rotate SSH keys in 6 Steps | PASS Unlock rotate SSH keys With This One Trick | 'has number', 'no banned' rotate SSH keys: 87% Faster Setups | 'numbers grounded' The last line is the case this whole test exists to catch. The headline has a number, so has number passes. But 87 appears nowhere in the brief’s facts, so numbers grounded fails. A model that only passes the contains-a-digit rule would look excellent on a shallow check and still ship a fabricated statistic. The blind sheet that python blind.py make writes looks like this, with the answer key stored separately: brief id,option A,option B,pick cert-renewal,A 40-Minute Fix for TLS certificate renewal,TLS certificate renewal Done Right: 7 Steps, dns-audit,"DNS audit: 5 Steps, 30 Min",Unlock DNS audit With This One Trick, docker-slim,A 35-Minute Fix for Docker image size,Tackle Docker image size in Under 35 Minutes, Finally, the scoring step. I filled a sheet with coin-flip picks to test the code path, and this is what python blind.py score rater1.csv printed: mock-steady 7 of 10 votes 70% mock-sloppy 3 of 10 votes 30% Those picks were random. One label still collected 70 percent of the votes. Ten votes from one rater prove almost nothing, which is why the decision rule in Step 12 asks for three raters and a wide margin. Cost and Time Budget for a Live Run A live run is cheap, but the cost scales with output tokens, and output tokens include any hidden reasoning a model does before it answers. Assume 30 calls per model 10 briefs times 3 runs , a short input prompt per call and about 1,500 output tokens per call once reasoning is counted. That makes 45,000 output tokens per model. At the per-million prices listed earlier, the totals come out as follows. | Model | Input cost | Output cost | Total for one full run | |---|---|---|---| | Claude Fable 5.2 | Not listed | $1.800 | Not listed | | Claude Fable 5.2 fast mode | Not listed | $3.600 | Not listed | | GPT-6 Sol | $0.027 | Not listed | $0.48 | | GPT-6 Luna | $0.001 | $0.023 | $0.02 | The premium pair, Fable 5.2 and GPT-6 Sol, is priced per million tokens: $8 input and $40 output for Fable 5.2, $2 input and $10 output for GPT-6 Sol. The assumption that matters is output tokens. If a model burns 6,000 tokens of hidden reasoning on every headline call, the Fable 5.2 line grows quickly, and you want to know that before the invoice arrives. The harness prints real token counts per call, so after the smoke test, multiply the observed output tokens by 30 and by the model’s output price to forecast the full run. Teams on ChatGPT Business or Enterprise meter some models by credits rather than tokens, as OpenAI documents on its rate card for Business and Enterprise https://help.openai.com/en/articles/11481834-chatgpt-rate-card-business-enterpriseedu-credit-based-pricing . If your company routes through those plans, your cost per run follows that card, not the table above. The time budget is easier. Setup and keys take about 10 minutes. Writing 10 briefs takes 15 if you borrow real articles from your backlog. Typing or pasting the code and running the dry run takes 20. The live generation itself finishes in a few minutes. Raters need about 15 minutes in parallel, and reading the results takes the last 10 or so. That sums to the 80 minutes in the title, and most of it is human time rather than compute. Common Pitfalls in a Claude vs ChatGPT Headline Test Comparing unequal tiers. Fable 5.2 costs four times what GPT-6 Sol costs on both input and output, and Sol costs 20 times what Luna costs. A flagship against a budget model is a price test dressed up as a quality test. Pair models by price band, or label the comparison as premium against value and report cost next to quality. Trusting an alias. ChatGPT’s Latest selection moved to GPT-6 in the model menu, while an explicit GPT-5.6 Sol selection stayed where it was. If your log says latest instead of an ID, nobody can reproduce the run in two weeks. Always record the exact ID or the exact picker label, plus the plan and the date. Sending a temperature to every model. Some current models reject sampling parameters or ignore them. If you add temperature for one vendor and not the other, you break the symmetry, and if you add it for both, one call may fail. Keep the default behavior unless you have a documented reason. Letting raters see the labels. A rater who knows which headline came from the model they prefer will pick it. Shuffle the order, strip the names and keep the key in a separate file. Do not run the vote in a shared chat where people can discuss their picks. Judging from one run. A single call per brief measures luck. Three runs expose variance, and the spread between those runs tells you how large a gap must be before it means anything. If a model’s all-rules pass rate swings from 70 percent to 90 percent across runs, a 5-point lead over its rival is noise. Capping output tokens too low. Reasoning models spend tokens before they write the answer. If MAX OUTPUT TOKENS is small, the response can end mid-JSON, and your parse-failure count blames the model for a limit you set. Start at 4,000 as the config does and raise it if parse failures appear. Writing rules that reward gaming. A rule that only checks for a digit invites invented numbers. Pairing has number with numbers grounded closes that gap. When you add a rule, ask how a lazy model could satisfy it without doing the real work. The same lesson applies to long drafts in an essay writing comparison https://shattered.io/best-ai-for-writing-essays-claude-vs-chatgpt-test-12-steps/ , where length targets invite padding. Troubleshooting: 10 Failures and Their Fixes - Set the model ID for fable-5.2 see config.py before running. The CLAUDE MODEL ID variable is empty in your current shell. Export it again in the terminal where you run the script, because a new terminal tab starts without it. - Anthropic rejects the model ID. The ID you pasted does not match what the API accepts. Copy it from Anthropic’s model documentation instead of typing it from memory, and check for stray spaces or quotes. - ModuleNotFoundError: No module named 'anthropic' or openai . The virtual environment is not active. Run source .venv/bin/activate , confirm with which python , then reinstall with pip install anthropic openai . - OpenAI rejects the request for gpt-6-sol . Your key or project may lack access to the model, or the rollout has not reached your account. Check the project permissions in the OpenAI dashboard, then try gpt-6-luna to separate a key problem from a model-access problem. - Parse failures climb above zero. Open one saved JSON file and read the text field. If the JSON ends abruptly, raise MAX OUTPUT TOKENS . If the model added a paragraph before the JSON, the parser trims it, so look instead for invalid commas or unescaped quotes inside a headline. - HTTP 429 or rate-limit errors during generation. The loop calls models back to back. Add a short time.sleep 1 between calls, rerun one model at a time, or lower RUNS PER BRIEF to 2 while you wait for your limits to reset. - report.py crashes with an index error or prints an empty table. The newest folder in runs/ is probably an aborted run with no response files. List the folder with ls runs , delete the empty one, or pass the right path: python report.py runs/20261121-173027 . - has keyword fails on every headline for one brief. The check compares whole words, so a keyword of Postgres backups fails when the headline says Postgres backup. Rewrite the keyword in the brief to match the form you want, or add stemming if your keywords vary. - Both blind options look identical. The two models produced the same lead headline, which happens with short keywords and tight rules. Tell raters to leave those rows blank, since a tie is a legitimate result. - The cost row shows 0.0000 on a live run. Either the prices in config.py are zero for that label or the provider returned no usage data. Print result "tokens in" for one call, and if it reads 0, check that you are reading the usage object from the response and not a wrapper. Advanced Tips for a Sharper Comparison Add a cost-matched arm. Fable 5.2 against GPT-6 Sol is premium versus premium. Add Claude Haiku 5.5 against GPT-6 Luna for the value tier. Copy the config entry, set the model ID and the price you verified on Anthropic’s pricing page, and rerun. The report adds a column automatically. Test fast mode as its own arm. If your Claude account exposes Fable 5.2’s fast mode, add it under a new label with the $16 and $80 prices. Compare mean seconds per call and the cost column, then ask whether a speed-up of up to 2.5 times is worth paying double for headline work, where latency rarely matters. Rescore instead of regenerating. Because every raw response is saved, you can change a rule, a threshold or the banned list and rerun python report.py runs/