Anthropic released Claude Fable 5.2 on November 9, and OpenAI put GPT-6.2 Astra and GPT-6.2 Sol into preview on November 17, according to a weekly AI recap. Both launches lean on coding, science and professional agent work. Neither tells a copywriter which model produces the better 58-character headline. That gap is the reason to run your own Claude vs ChatGPT test, and headlines are the cheapest place to start.
A headline is short enough to score by machine and tight enough to expose sloppy habits: invented statistics, padded phrasing, buzzwords, the same sentence skeleton repeated eight times. The Python harness below pushes 10 article briefs through each model, applies seven automatic rules to every headline, checks the meta description, and then hands anonymized lead headlines to human raters. It takes 12 steps and about 80 minutes. A premium-versus-premium run costs about $2.40 in API fees.
Long-form tests such as a voice matching test tell you whether a model can hold a style across 2,000 words. A headline test answers a different question: does the model obey hard constraints when every character counts? Most editorial teams need both answers, and the second one costs a fraction of the first.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
Why Headlines Make a Sharper Claude vs ChatGPT for Writing Test #
Full-article comparisons mix too many variables. Length, structure, topic knowledge, formatting habits and rater fatigue all leak into the verdict, and by the time a reader finishes the third 1,500-word sample, the first one is a blur. A headline strips that noise away. The model gets a brief, a handful of facts and a few hard rules. It either follows them or it does not.
The rules are binary, and binary rules scale. Ten briefs, three runs each and eight headlines per call produce 240 headlines per model. A script scores all 240 in well under a second, so you can rerun the whole test after every prompt tweak instead of rating a sample by hand once and hoping it generalizes. That speed changes how you work. You stop arguing about which model feels better and start looking at a table of failure rates.
The most revealing rule is number grounding. Each brief carries a short list of facts, and the prompt tells the model to use only numbers from that list. Models love a punchy statistic, so some headlines arrive with a claim like 47% faster that appears nowhere in the brief. A single invented number in a published headline is a correction waiting to happen, which makes this the rule an editor should care about most. If you have already run a writing rules compliance test, the checks will feel familiar. The difference is scale: on short text, one miss costs you 1 of 8 options rather than one buried sentence in a long draft.
Machines cannot judge which headline a reader will click, so the harness adds a blind human pass. Each rater sees one lead headline per model for every brief, with the model names hidden and the order shuffled. A rater finishes 10 briefs in about 10 minutes. That is short enough that you can recruit three colleagues without a speech.
Be clear about what the result proves. A headline score measures headline obedience on your briefs, your rules and your prompt. It does not crown an overall winner for writing, and a model that loses here may still win on dialogue, long-form structure or tone. Treat the output as one data point in a larger evaluation, and keep the raw run files so you can rescore them when you change the rules.
What the November 2026 Releases Change for Claude vs ChatGPT #
Start with what Anthropic shipped. Claude Fable 5.2 and Claude Mythos 5.2 are its newest models, announced in November 2026, and Anthropic’s own page frames them as models for “daunting technical work.” Fable 5.2 generates output up to 30% faster than Fable 5.1 and costs $8 per million input tokens and $40 per million output tokens. An optional fast mode runs up to 2.5 times faster at $16 and $80. Subscribers on Pro, Max, Ultra, Team and Enterprise plans can use Fable 5.2, and developers reach it through the Claude Platform, AWS, Google Cloud and Microsoft’s cloud. Mythos 5.2 is a different animal. Anthropic limits it to vetted organizations in its verification programs, so a normal team cannot test it and this tutorial skips it.
Anthropic’s lineup also lists Opus 5.5, Sonnet 5.5 and Haiku 5.5. The company says Opus 5.5 costs 40% less to run than Opus 5 on typical workloads, and that Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. It calls Haiku 5.5 its “fastest, most capable small model,” built for high-volume work such as summarization. Fable 5.2 and GPT-6 Sol sit at very different prices, so a fair test also pairs a small Claude tier against OpenAI’s cheap model. The config in Step 2 leaves room for that third arm. Read the current prices for those tiers off Anthropic’s pricing page before you add them, because this tutorial only prints figures it could verify.
On the OpenAI side, GPT-6.2 Astra targets advanced reasoning, coding and complex professional work, while GPT-6.2 Sol targets faster, cheaper multistep tasks. Both were released on November 17 in the API, ChatGPT Work and Codex, and on November 20 they entered preview in ChatGPT Work and Codex on desktop. OpenAI also announced GPT-6 Sol and GPT-6 Luna, with API identifiers gpt-6-sol and gpt-6-luna, as covered in a November 20 recap. GPT-6 Sol costs $2 per million input tokens and $10 per million output tokens. GPT-6 Luna costs $0.10 and $0.50. Because the 6.2 pair is also in the API, the harness can test it there, but this run tests the GPT-6 pair. The harness accepts any model ID, so you can add the 6.2 models whenever you like.
Consumer plans matter if you test inside the apps instead of the API. OpenAI’s pricing page lists ChatGPT Free at $0, Go at $8 per month in the US, Plus at $20 and Pro in several tiers. Claude’s plan names are Pro, Max, Ultra, Team and Enterprise, and Anthropic’s pricing page holds the current monthly numbers. OpenAI’s release notes describe Free and Go users on Luna and Plus and above on Sol, yet the original announcement said Sol and Luna were not yet in regular chat and would roll out gradually. The mapping has shifted from day to day, so the model label in your own picker is the only record you can trust.
| Model | Status on Nov 21, 2026 | API price per 1M tokens (input / output) | Role in this test |
|---|---|---|---|
| Claude Fable 5.2 | Released Nov 9 | $8 / $40 | Premium Claude arm ( fable-5.2 ) |
| Claude Fable 5.2 fast mode | Optional, up to 2.5x faster | $16 / $80 | Optional latency arm |
| GPT-6 Sol | Announced the week of Nov 16 | $2 / $10 | Premium OpenAI arm ( gpt-6-sol ) |
| GPT-6 Luna | Announced alongside Sol | $0.10 / $0.50 | Budget OpenAI arm ( gpt-6-luna ) |
| Claude Mythos 5.2 | Vetted organizations only | $8 / $40 | Excluded, no general access |
| GPT-6.2 Astra and Sol | Released Nov 17 in the API, Work and Codex; preview in ChatGPT Work and Codex on desktop Nov 20 | Not tested | Optional, add by model ID |
Prerequisites and Versions for the Headline Harness #
The project needs little beyond a terminal. Gather these before you start, because a missing key or a guessed model ID burns the most time.
- Python 3.11 or newer. The offline run that produced the sample output below used Python 3.12.3. Nothing in the code needs a newer feature than 3.11.
- The
anthropicandopenaiPython packages, latest release of each. Install them in a virtual environment. I name no version numbers because both SDKs ship often. Runpip freeze > requirements.lockafter your first successful live run so the test stays reproducible. - An Anthropic API key exported as
ANTHROPIC_API_KEY, with access to Claude Fable 5.2 on the Claude Platform. - An OpenAI API key exported as
OPENAI_API_KEY, with access togpt-6-solandgpt-6-luna. - The exact Claude model ID , copied from Anthropic’s model documentation into the
CLAUDE_MODEL_IDenvironment variable. Do not guess it. A model name you remember from a blog post and a model ID the API accepts are different strings. - Three human raters for the blind pass, each with about 10 minutes free. Pick people who did not write the briefs.
- A budget of roughly $2.50 and 80 minutes for the premium pair. Under the assumptions stated later, one full run of that pair costs about $2.39.
One more disclosure about what was run. The offline path, which uses a mock model, ran end to end on Python 3.12.3, and every number in the sample-output section comes from that run. The two live adapters follow each SDK’s documented call shape, but only your first live call can confirm that a vendor has not changed a parameter since this was written. Start with --limit 1 so a mistake costs pennies.
Why do versions matter so much in a Claude vs ChatGPT comparison? Both vendors changed their lineups within the same fortnight, and the Latest option in ChatGPT’s model menu now points at GPT-6, while an explicit GPT-5.6 Sol selection stays on GPT-5.6. A result that names only ChatGPT or only Claude is unreproducible. Record the model ID, the plan or API tier, the date and the prompt for every run, and the harness does that automatically by saving a models.json file beside the raw responses. If your priority is spend per usable output rather than spend per call, pair this test with a cost per accepted 1,000 words calculation.
Steps 1 to 3: Project Setup, Model Pinning and Briefs #
Step 1: Create the project and install the SDKs
Make a clean directory, a virtual environment and the two SDKs. Keep the keys in environment variables rather than files, so a stray git add cannot publish them.
mkdir headline_bench && cd headline_bench
python3 -m venv .venv
source .venv/bin/activate
python --version
pip install anthropic openai
mkdir tests runs
echo "runs/" >> .gitignore
export ANTHROPIC_API_KEY="paste-your-key-here"
export OPENAI_API_KEY="paste-your-key-here"
export CLAUDE_MODEL_ID="paste-the-id-from-anthropic-docs"
The python --version line is a checkpoint. If it prints anything older than 3.11, recreate the environment with a newer interpreter before you go on. The runs/ entry in .gitignore matters more than it looks, because each run folder holds raw model responses and you do not want 30 megabytes of JSON in your repository history.
Step 2: Pin the models and prices in config.py
Every constant that can change lives in one file. The MODELS dictionary maps a short label to a provider, an exact model ID and the price per million tokens. The labels appear in every report column, so choose names you will recognize in six months. The prices below match the figures in the table above, and the comment reminds you to re-check them. The Claude ID reads from the environment on purpose, because this tutorial does not invent identifiers.
import os
from pathlib import Path
ROOT = Path(__file__).parent
RUNS_DIR = ROOT / "runs"
RUNS_PER_BRIEF = int(os.getenv("RUNS_PER_BRIEF", "3"))
HEADLINES_PER_CALL = 8
MAX_OUTPUT_TOKENS = int(os.getenv("MAX_OUTPUT_TOKENS", "4000"))
MODELS = {
"fable-5.2": {
"provider": "anthropic",
"model_id": os.getenv("CLAUDE_MODEL_ID", ""),
"price_in": 8.0,
"price_out": 40.0,
},
"gpt-6-sol": {
"provider": "openai",
"model_id": os.getenv("OPENAI_SOL_ID", "gpt-6-sol"),
"price_in": 2.0,
"price_out": 10.0,
},
"gpt-6-luna": {
"provider": "openai",
"model_id": os.getenv("OPENAI_LUNA_ID", "gpt-6-luna"),
"price_in": 0.10,
"price_out": 0.50,
},
}
BANNED = [
"game-changer", "game changer", "revolutionary", "unlock", "groundbreaking",
"mind-blowing", "next-level", "cutting-edge", "supercharge", "skyrocket",
"you won't believe",
]
HEADLINE_MAX_CHARS = 60
META_MIN_CHARS = 120
META_MAX_CHARS = 155
NEAR_DUPLICATE_JACCARD = 0.6 # measured on words outside the keyword, digits ignored
The BANNED list is the part you will edit most. It holds phrases that mark a headline as machine-flavored in your house style. Swap in your own list. The NEAR_DUPLICATE_JACCARD threshold is explained in Step 7, and RUNS_PER_BRIEF controls how many times each model sees each brief. Three runs is the minimum that exposes variance. One run per brief tells you about luck.
Step 3: Write 10 briefs with facts the models must respect
Each brief has an ID, a primary keyword, an audience and three facts. The facts are the source of truth for the number-grounding rule, so write them like an editor handing material to a writer: plain sentences with the real figures in them. Every brief in this file includes a step count and a time in minutes, because the offline mock model uses them. Your real briefs can carry any facts you like.
[
{"id": "ssh-keys", "keyword": "rotate SSH keys", "audience": "Linux admins",
"facts": ["The procedure has 6 steps.", "It takes about 25 minutes per server.", "Ed25519 keys replace 2048-bit RSA keys."]},
{"id": "pg-backup", "keyword": "Postgres backups", "audience": "backend developers",
"facts": ["The setup has 8 steps.", "A full restore test takes 40 minutes.", "Nightly dumps are kept for 14 days."]},
{"id": "dns-audit", "keyword": "DNS audit", "audience": "IT managers",
"facts": ["The audit has 5 steps.", "It takes 30 minutes for a single domain.", "Stale CNAME records were found on 3 of 12 test domains."]},
{"id": "docker-slim", "keyword": "Docker image size", "audience": "DevOps engineers",
"facts": ["The guide has 7 steps.", "The full run takes 35 minutes.", "It cuts a test image from 1,240 MB to 180 MB."]},
{"id": "mfa-rollout", "keyword": "MFA rollout", "audience": "security leads",
"facts": ["The rollout plan has 9 steps.", "Setup takes 60 minutes per team.", "A pilot group of 50 users goes first for 2 weeks."]},
{"id": "log-retention", "keyword": "log retention", "audience": "compliance teams",
"facts": ["The policy template has 4 steps.", "Setup takes 45 minutes.", "Retention ranges from 30 to 365 days across 4 log classes."]},
{"id": "k8s-limits", "keyword": "Kubernetes resource limits", "audience": "platform engineers",
"facts": ["The walkthrough has 10 steps.", "It takes 50 minutes.", "Right-sizing cut test pod memory by 38% on a 3-node cluster."]},
{"id": "git-hooks", "keyword": "Git pre-commit hooks", "audience": "team leads",
"facts": ["The setup has 6 steps.", "It takes 20 minutes to install across a repo.", "Secret scanning runs in under 3 seconds."]},
{"id": "wifi-guest", "keyword": "guest Wi-Fi isolation", "audience": "small business owners",
"facts": ["The router change has 5 steps.", "It takes 15 minutes.", "Guests get their own VLAN 30."]},
{"id": "cert-renewal", "keyword": "TLS certificate renewal", "audience": "site reliability engineers",
"facts": ["The automation has 7 steps.", "The setup takes 40 minutes.", "Certificates renew 30 days before expiry and the alert fires after 2 failed attempts."]}
]
Keep the briefs realistic for your own site. A tutorial publisher should test tutorial briefs, a news desk should test briefs with names, dates and dollar amounts. Vary the keyword length, too. A three-word keyword like Kubernetes resource limits leaves little room in a 60-character headline, and that pressure is exactly what separates models that count characters from models that guess. The same discipline applies to the source material in a source-grounded writing audit: the less ambiguity in the brief, the more any failure belongs to the model.
Steps 4 to 6: Prompt, Provider Layer and Generation Run #
Step 4: Write one prompt that both models receive
Fairness starts with an identical prompt. The system message sets the role. The user message states the rules, asks for JSON in one exact shape and appends the brief. It names the number rule twice, once as a requirement and once as a prohibition, because models skim.
SYSTEM = (
"You are a headline editor for a technology news site. "
"Follow the style rules exactly and reply with JSON only."
)
USER_TEMPLATE = """Write {n} distinct headlines and one meta description for the article brief below.
Rules:
- Every headline is 60 characters or fewer.
- Every headline contains the primary keyword and one specific number taken from the facts.
- Use only numbers that appear in the facts. Never invent a statistic.
- No em dashes. Do not use these words: {banned}.
- Order the headlines best first.
- The meta description is 120 to 155 characters and contains the primary keyword.
Return exactly this JSON shape: {{"headlines": ["..."], "meta": "..."}}
BRIEF_JSON: {brief}"""
Three design choices deserve a note. First, the prompt sends no temperature. Some current models reject or ignore sampling parameters, and a parameter that works for one vendor and errors for the other would break the symmetry. Second, asking for best-first order gives the blind pass a fair target, since the lead headline is the one the model itself would publish. Third, the brief travels as JSON after a fixed marker, which lets the mock model parse it and lets you diff prompts between runs. This prompt is only the first stage of a real editorial flow. If you want to see how drafting, editing and headline passes chain together, the writing pipeline test covers that setup.
Step 5: Build the provider layer
One function, call_model, hides the vendor differences. It takes a config entry, a system message and a user message, and returns the text, the token counts and the wall-clock seconds. The Anthropic adapter uses the Messages API with a system parameter. The OpenAI adapter uses the Responses API with instructions and input. Both read token usage off the response so the report can price each run. The third branch, mock, is the offline stand-in: it produces headlines from templates, some good and some deliberately flawed, so you can exercise every later step without spending money.
import json
import random
import re
import time
from config import MAX_OUTPUT_TOKENS
def _anthropic(cfg, system, user):
from anthropic import Anthropic
resp = Anthropic().messages.create(
model=cfg["model_id"],
max_tokens=MAX_OUTPUT_TOKENS,
system=system,
messages=[{"role": "user", "content": user}],
)
text = "".join(b.text for b in resp.content if getattr(b, "type", "") == "text")
return text, resp.usage.input_tokens, resp.usage.output_tokens
def _openai(cfg, system, user):
from openai import OpenAI
resp = OpenAI().responses.create(
model=cfg["model_id"],
instructions=system,
input=user,
max_output_tokens=MAX_OUTPUT_TOKENS,
)
return resp.output_text, resp.usage.input_tokens, resp.usage.output_tokens
GOOD = [
"{kw}: {a} Steps, {b} Min",
"{kw} in {a} Steps",
"How to Handle {kw} in {b} Minutes",
"{kw} Done Right: {a} Steps",
"A {b}-Minute Fix for {kw}",
"Fix Your {kw} in {b} Minutes",
"{a} Steps to Better {kw}",
"Tackle {kw} in Under {b} Minutes",
]
BAD = [
"Unlock {kw} With This One Trick",
"{kw} Explained, Fast",
"{kw}: 87% Faster Setups",
"How to Get {kw} Right Without the Headaches That Slow Teams Down",
"Why {kw} Matters",
]
def _mock(cfg, system, user, seed):
brief = json.loads(user.split("BRIEF_JSON:", 1)[1])
facts = " ".join(brief["facts"])
a = re.search(r"(\d+) steps", facts).group(1)
b = re.search(r"(\d+) minutes", facts).group(1)
rng = random.Random(seed)
bad_rate = 0.12 if "steady" in cfg["model_id"] else 0.40
good, bad = GOOD[:], BAD[:]
rng.shuffle(good)
rng.shuffle(bad)
lines = []
for _ in range(8):
pool = bad if (rng.random() < bad_rate and bad) or not good else good
lines.append(pool.pop().format(kw=brief["keyword"], a=a, b=b))
meta = f"Learn how to handle {brief['keyword']} with {a} clear steps and a {b}-minute plan that {brief['audience']} can follow today."
if "steady" in cfg["model_id"]:
meta += " Includes a rollback checklist."
return json.dumps({"headlines": lines, "meta": meta}), 400, 220
def call_model(cfg, system, user, seed=""):
start = time.perf_counter()
kind = cfg["provider"]
if kind == "anthropic":
text, tin, tout = _anthropic(cfg, system, user)
elif kind == "openai":
text, tin, tout = _openai(cfg, system, user)
elif kind == "mock":
text, tin, tout = _mock(cfg, system, user, seed)
else:
raise ValueError(f"unknown provider {kind}")
return {
"text": text,
"tokens_in": tin,
"tokens_out": tout,
"seconds": round(time.perf_counter() - start, 3),
}
Notice that the mock draws a different mix of flawed headlines for the steady and sloppy labels, and seeds its random generator from the model, brief and run number. Seeding makes the dry run reproducible, which means the sample report below is the same on your machine as on mine. The live adapters import their SDKs inside the function. That keeps the offline path working on a machine where neither SDK is installed yet.
Step 6: Generate headlines and save every raw response
The generator loops over models, briefs and runs, calls the provider, parses the JSON and writes one file per call. It stores the raw text next to the parsed result. That habit pays off the first time a rule changes, because you can rescore old runs without paying for new ones. A parse failure does not stop the run. The record gets parsed: null, and the report counts it as a failure, which is the honest treatment for a model that cannot follow a reply-with-JSON-only instruction.
import argparse
import json
import re
from datetime import datetime
from config import BANNED, HEADLINES_PER_CALL, MODELS, ROOT, RUNS_DIR, RUNS_PER_BRIEF
from providers import call_model
SYSTEM = (
"You are a headline editor for a technology news site. "
"Follow the style rules exactly and reply with JSON only."
)
USER_TEMPLATE = """Write {n} distinct headlines and one meta description for the article brief below.
Rules:
- Every headline is 60 characters or fewer.
- Every headline contains the primary keyword and one specific number taken from the facts.
- Use only numbers that appear in the facts. Never invent a statistic.
- No em dashes. Do not use these words: {banned}.
- Order the headlines best first.
- The meta description is 120 to 155 characters and contains the primary keyword.
Return exactly this JSON shape: {{"headlines": ["..."], "meta": "..."}}
BRIEF_JSON: {brief}"""
def build_prompt(brief):
return USER_TEMPLATE.format(
n=HEADLINES_PER_CALL, banned=", ".join(BANNED), brief=json.dumps(brief)
)
def parse_json(text):
text = re.sub(r"```(?:json)?", "", text)
start, end = text.find("{"), text.rfind("}")
if start == -1 or end == -1:
return None
try:
return json.loads(text[start : end + 1])
except json.JSONDecodeError:
return None
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--models", default="fable-5.2,gpt-6-sol")
ap.add_argument("--mock", action="store_true", help="offline stand-in models")
ap.add_argument("--limit", type=int, default=0, help="only the first N briefs")
args = ap.parse_args()
briefs = json.loads((ROOT / "briefs.json").read_text())
if args.limit:
briefs = briefs[: args.limit]
if args.mock:
models = {
"mock-steady": {"provider": "mock", "model_id": "mock-steady", "price_in": 0, "price_out": 0},
"mock-sloppy": {"provider": "mock", "model_id": "mock-sloppy", "price_in": 0, "price_out": 0},
}
else:
models = {k: MODELS[k] for k in args.models.split(",")}
for label, cfg in models.items():
if not cfg["model_id"]:
raise SystemExit(f"Set the model ID for {label} (see config.py) before running.")
run_dir = RUNS_DIR / datetime.now().strftime("%Y%m%d-%H%M%S")
for label in models:
(run_dir / label).mkdir(parents=True)
(run_dir / "models.json").write_text(json.dumps(models, indent=2))
for label, cfg in models.items():
for brief in briefs:
for run in range(1, RUNS_PER_BRIEF + 1):
result = call_model(cfg, SYSTEM, build_prompt(brief), seed=f"{label}-{brief['id']}-{run}")
result["parsed"] = parse_json(result["text"])
result["brief_id"], result["run"], result["model"] = brief["id"], run, label
path = run_dir / label / f"{brief['id']}-{run}.json"
path.write_text(json.dumps(result, indent=2))
status = "ok" if result["parsed"] else "PARSE FAIL"
print(f"{label:12} {brief['id']:14} run {run} {result['tokens_out']:>5} tok {result['seconds']:>5}s {status}")
print(f"\nRun saved to {run_dir}")
if __name__ == "__main__":
main()
Run the offline path first.
python generate.py --mock --limit 1
You should see one line per call with the model label, brief ID, run number, output tokens, seconds and an ok flag, then a Run saved to path. When that works, switch to a live smoke test with a single brief:
python generate.py --models fable-5.2,gpt-6-sol --limit 1
Check the saved JSON files by eye. The text field should hold a JSON object with eight headlines and a meta description. If either model wraps the JSON in commentary or code fences, the parser in parse_json strips the fences and trims to the outer braces. That covers the common cases without hiding real failures.
Steps 7 to 9: Scoring Rules, Meta Checks and the Report #
Step 7: Score every headline against seven rules
The scoring module turns taste into arithmetic. Each headline gets seven true-or-false results, and a headline counts as clean only when all seven pass. The table lists what each rule catches and where the threshold sits.
| Rule | What it catches | Threshold |
|---|---|---|
length_ok |
Headlines that risk truncation in search results | 60 characters or fewer |
has_number |
Vague headlines with no specific figure | At least one digit |
numbers_grounded |
Invented statistics | Every number appears in the brief’s facts |
no_banned |
House-style buzzwords | No phrase from the BANNED list |
no_em_dash |
Em and en dashes | None present |
has_keyword |
Missing primary keyword | Every keyword word present, any order |
not_duplicate |
Near-copies inside one call | Word overlap below 0.6 |
Two details in the code need explaining. The first is how numbers are compared. The regular expression pulls out digit groups, strips trailing punctuation and removes thousands separators, so 1,240 in a brief matches 1240 in a headline. Without that normalization a model that copies a figure exactly would still fail the grounding rule. The second detail is the duplicate check. Every headline must contain the primary keyword and a number, so two perfectly different headlines will share a lot of words. The jaccard function therefore ignores the keyword’s words and all digits before it measures overlap. What remains is phrasing: Fix Your X in 25 Minutes and Tackle X in Under 25 Minutes score as different, while X in 6 Steps for Admins next to X in 6 Steps for Admins Today scores as a near-copy.
import re
from config import (
BANNED, HEADLINE_MAX_CHARS, META_MAX_CHARS, META_MIN_CHARS, NEAR_DUPLICATE_JACCARD,
)
NUM_RE = re.compile(r"\d[\d,]*(?:\.\d+)?")
def numbers_in(text):
return {n.rstrip(",.").replace(",", "") for n in NUM_RE.findall(text)}
def words(text):
return set(re.findall(r"[a-z0-9]+", text.lower()))
def jaccard(a, b, ignore=frozenset()):
wa = {w for w in words(a) if w not in ignore and not w.isdigit()}
wb = {w for w in words(b) if w not in ignore and not w.isdigit()}
return len(wa & wb) / len(wa | wb) if wa | wb else 0.0
def has_keyword(text, keyword):
return words(keyword) <= words(text)
def banned_hit(text):
low = text.lower()
return any(b in low for b in BANNED)
def check_headline(text, brief, others):
fact_numbers = numbers_in(" ".join(brief["facts"]))
nums = numbers_in(text)
kw_words = words(brief["keyword"])
return {
"length_ok": len(text) <= HEADLINE_MAX_CHARS,
"has_number": bool(nums),
"numbers_grounded": nums <= fact_numbers,
"no_banned": not banned_hit(text),
"no_em_dash": ", " not in text and "\u2013" not in text,
"has_keyword": has_keyword(text, brief["keyword"]),
"not_duplicate": all(jaccard(text, o, kw_words) < NEAR_DUPLICATE_JACCARD for o in others),
}
def check_meta(text, brief):
fact_numbers = numbers_in(" ".join(brief["facts"]))
return {
"length_ok": META_MIN_CHARS <= len(text) <= META_MAX_CHARS,
"has_keyword": has_keyword(text, brief["keyword"]),
"numbers_grounded": numbers_in(text) <= fact_numbers,
"no_banned": not banned_hit(text),
"no_em_dash": ", " not in text and "\u2013" not in text,
}
def score_call(parsed, brief):
heads = [h for h in parsed.get("headlines", []) if isinstance(h, str)]
head_results = []
for i, h in enumerate(heads):
others = heads[:i] + heads[i + 1 :]
head_results.append({"text": h, "checks": check_headline(h, brief, others)})
meta = parsed.get("meta", "") if isinstance(parsed.get("meta", ""), str) else ""
return {"headlines": head_results, "meta": {"text": meta, "checks": check_meta(meta, brief)}}
The rules are deliberately dumb. A smarter rule, such as whether a headline sounds natural belongs to the human pass. Dumb rules have one huge advantage: you can write a unit test for each of them in two lines. The test file below covers number normalization, invented numbers, a clean headline, an over-long headline, banned words with dashes, and near-duplicates. It runs with plain Python, so you do not need a test runner.
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from checks import check_headline, jaccard, numbers_in
BRIEF = {"keyword": "rotate SSH keys", "facts": ["The procedure has 6 steps.", "It takes about 25 minutes."]}
def test_numbers_in_strips_commas_and_punctuation():
assert numbers_in("Cut 1,240 MB to 180 MB.") == {"1240", "180"}
def test_invented_number_is_flagged():
r = check_headline("Rotate SSH Keys: 87% Faster", BRIEF, [])
assert r["numbers_grounded"] is False
def test_clean_headline_passes_every_rule():
r = check_headline("Rotate SSH Keys in 6 Steps, 25 Min", BRIEF, [])
assert all(r.values())
def test_long_headline_fails_length():
assert check_headline("Rotate SSH Keys in 6 Steps " + "x" * 40, BRIEF, [])["length_ok"] is False
def test_banned_word_and_em_dash():
r = check_headline("Unlock Rotate SSH Keys, 6 Steps", BRIEF, [])
assert r["no_banned"] is False and r["no_em_dash"] is False
def test_near_duplicates_flagged():
other = "Rotate SSH Keys in 6 Steps for Admins"
kw = {"rotate", "ssh", "keys"}
assert jaccard(other, "Rotate SSH Keys in 6 Steps for Admins Today", kw) > 0.6
assert check_headline("Rotate SSH Keys in 6 Steps for Admins Today", BRIEF, [other])["not_duplicate"] is False
if __name__ == "__main__":
tests = [v for k, v in dict(globals()).items() if k.startswith("test_")]
for fn in tests:
fn()
print("PASS", fn.__name__)
print(f"{len(tests)} tests passed")
Run it before every change to the rules:
python tests/test_checks.py
PASS test_numbers_in_strips_commas_and_punctuation
PASS test_invented_number_is_flagged
PASS test_clean_headline_passes_every_rule
PASS test_long_headline_fails_length
PASS test_banned_word_and_em_dash
PASS test_near_duplicates_flagged
6 tests passed
Step 8: Check the meta description the same way
The same module scores the meta description with five rules: length between 120 and 155 characters, primary keyword present, numbers grounded in the facts, no banned phrases and no dashes. The function is check_meta, and it sits in the same module. Meta descriptions fail in a different pattern from headlines. Models overshoot the length limit more often than they invent numbers, because a 155-character window feels tiny while writing and huge while counting. Keep the meta scoring separate from the headline scoring, as the report does, so a model that writes excellent headlines but sloppy descriptions shows up as exactly that. The same split works when you test revisions instead of fresh copy, as an editing fidelity test does with seeded errors.
One design choice is worth defending. The meta check requires the keyword but does not require a number, because a description that crams in a figure just to satisfy a rule reads like a headline wearing a trench coat. Adjust it if your site’s house style says otherwise.
Step 9: Build the report table
The report loads the saved runs, scores each parsed response and prints a Markdown table with one column per model. It also writes the table to report.md inside the run folder, so every run carries its own scorecard. Cost comes from the token counts the providers returned, multiplied by the prices in config.py. A parse failure increments a counter and skips scoring, so a model that returns broken JSON cannot hide behind a high pass rate on the calls that worked.
import json
import sys
from collections import defaultdict
from checks import score_call
from config import ROOT, RUNS_DIR
RULES = ["length_ok", "has_number", "numbers_grounded", "no_banned", "no_em_dash", "has_keyword", "not_duplicate"]
def latest_run():
return sorted(p for p in RUNS_DIR.iterdir() if p.is_dir())[-1]
def main(run_dir):
briefs = {b["id"]: b for b in json.loads((ROOT / "briefs.json").read_text())}
models = json.loads((run_dir / "models.json").read_text())
stats = {}
for label, cfg in models.items():
s = defaultdict(int)
for f in sorted((run_dir / label).glob("*.json")):
rec = json.loads(f.read_text())
s["calls"] += 1
s["tok_out"] += rec["tokens_out"]
s["tok_in"] += rec["tokens_in"]
s["seconds"] += rec["seconds"]
if not rec["parsed"]:
s["parse_fail"] += 1
continue
scored = score_call(rec["parsed"], briefs[rec["brief_id"]])
for h in scored["headlines"]:
s["headlines"] += 1
for r in RULES:
s[r] += h["checks"][r]
s["all_pass"] += all(h["checks"].values())
s["meta_n"] += 1
s["meta_pass"] += all(scored["meta"]["checks"].values())
s["cost"] = s["tok_in"] / 1e6 * cfg["price_in"] + s["tok_out"] / 1e6 * cfg["price_out"]
stats[label] = s
pct = lambda a, b: f"{100 * a / b:.1f}%" if b else "n/a"
lines = ["| Metric | " + " | ".join(stats) + " |", "|---|" + "---|" * len(stats)]
for r in RULES:
lines.append(f"| {r} | " + " | ".join(pct(s[r], s["headlines"]) for s in stats.values()) + " |")
lines.append("| all 7 rules | " + " | ".join(pct(s["all_pass"], s["headlines"]) for s in stats.values()) + " |")
lines.append("| meta passes all | " + " | ".join(pct(s["meta_pass"], s["meta_n"]) for s in stats.values()) + " |")
lines.append("| parse failures | " + " | ".join(str(s["parse_fail"]) for s in stats.values()) + " |")
lines.append("| mean output tokens/call | " + " | ".join(f"{s['tok_out'] / s['calls']:.0f}" for s in stats.values()) + " |")
lines.append("| mean seconds/call | " + " | ".join(f"{s['seconds'] / s['calls']:.1f}" for s in stats.values()) + " |")
lines.append("| est. cost (USD) | " + " | ".join(f"{s['cost']:.4f}" for s in stats.values()) + " |")
out = "\n".join(lines)
(run_dir / "report.md").write_text(out + "\n")
print(out)
if __name__ == "__main__":
main(__import__("pathlib").Path(sys.argv[1]) if len(sys.argv) > 1 else latest_run())
Run it against the newest run folder with python report.py, or pass a folder path to rescore an older run. Rescoring is free, which is the whole point of saving raw responses in Step 6.
Steps 10 to 12: Blind Vote, Dry Run and Live Run #
Step 10: Build the blind sheet for human raters
Automatic rules say which headlines are legal. They cannot say which one a reader would rather click. The blind script takes the lead headline from run 1 of every brief and every model, shuffles the order with a fixed seed and writes a CSV that shows only the options, labeled A and B. The answer key goes into a separate JSON file that raters never see. The score command reads one or more completed rater CSVs, translates letters back to model labels and prints vote shares.
import csv
import json
import random
import sys
from collections import Counter
from pathlib import Path
from report import latest_run
LETTERS = "ABCDEFG"
def make(run_dir, seed=2026):
models = json.loads((run_dir / "models.json").read_text())
rng = random.Random(seed)
rows, key = [], {}
brief_ids = sorted({f.stem.rsplit("-", 1)[0] for f in (run_dir / next(iter(models))).glob("*.json")})
for bid in brief_ids:
options = []
for label in models:
rec = json.loads((run_dir / label / f"{bid}-1.json").read_text())
if rec["parsed"] and rec["parsed"].get("headlines"):
options.append((label, rec["parsed"]["headlines"][0]))
rng.shuffle(options)
key[bid] = {LETTERS[i]: label for i, (label, _) in enumerate(options)}
row = {"brief_id": bid, "pick": ""}
for i, (_, text) in enumerate(options):
row[f"option_{LETTERS[i]}"] = text
rows.append(row)
cols = ["brief_id"] + [f"option_{LETTERS[i]}" for i in range(len(models))] + ["pick"]
with open(run_dir / "blind_sheet.csv", "w", newline="") as fh:
w = csv.DictWriter(fh, fieldnames=cols)
w.writeheader()
w.writerows(rows)
(run_dir / "blind_key.json").write_text(json.dumps(key, indent=2))
print(f"Wrote {run_dir / 'blind_sheet.csv'} ({len(rows)} rows). Give raters the CSV only, never the key.")
def score(run_dir, rater_files):
key = json.loads((run_dir / "blind_key.json").read_text())
wins, total = Counter(), 0
for path in rater_files:
for row in csv.DictReader(open(path)):
pick = row["pick"].strip().upper()
if pick in key[row["brief_id"]]:
wins[key[row["brief_id"]][pick]] += 1
total += 1
for label, n in wins.most_common():
print(f"{label:12} {n:>3} of {total} votes ({100 * n / total:.0f}%)")
if __name__ == "__main__":
cmd = sys.argv[1]
run = latest_run()
if cmd == "make":
make(run)
elif cmd == "score":
score(run, [Path(p) for p in sys.argv[2:]])
Brief your raters in three sentences. Tell them to pick the headline they would click, tell them to leave the cell blank if the two feel equal, and tell them not to guess which model wrote which. Send the CSV only. If you want to know how raters behave on longer text, the blind test of full drafts uses the same logic with 1,000-word samples.
Step 11: Run the full offline dry run
Before you spend a cent, run the complete chain on the mock models. Each command should finish in about a second.
python tests/test_checks.py
python generate.py --mock
python report.py
python blind.py make
If all four succeed, the pipeline is sound and any later failure comes from credentials, model access or the vendors’ APIs. That separation saves hours of guessing.
Step 12: Run live, then read the results with discipline
Export your keys and the Claude model ID, run a one-brief smoke test, then run the full set.
python generate.py --models fable-5.2,gpt-6-sol --limit 1
python generate.py --models fable-5.2,gpt-6-sol
python report.py
python blind.py make
Send blind_sheet.csv to three raters and collect their files. Then run python blind.py score rater1.csv rater2.csv rater3.csv. Before you read any numbers, write down your decision rule. A sensible one: a model wins a rule only when it leads by at least 10 percentage points on that rule and the lead holds in each of the three runs. Three raters voting on 10 briefs give you 30 votes, and a 17 to 13 split sits well inside the noise of 30 coin flips, so treat anything closer than about 20 to 10 as a tie. When the automatic rules and the human votes disagree, trust the rules for obedience and the humans for appeal, and say so in your write-up.
Sample Output From the Offline Run #
Every number in this section comes from the mock models, not from Claude or ChatGPT. The mock steady model produces few flawed headlines and the mock sloppy model many, so the table shows the pipeline catching a known difference. Your live numbers will look different. The report for a full 10-brief offline run:
| Metric | mock-steady | mock-sloppy |
|---|---|---|
| length_ok | 98.8% | 91.7% |
| has_number | 90.0% | 66.7% |
| numbers_grounded | 97.9% | 91.2% |
| no_banned | 96.2% | 90.4% |
| no_em_dash | 100.0% | 100.0% |
| has_keyword | 100.0% | 100.0% |
| not_duplicate | 100.0% | 100.0% |
| all 7 rules | 87.9% | 57.9% |
| meta passes all | 70.0% | 30.0% |
| parse failures | 0 | 0 |
| mean output tokens/call | 220 | 220 |
| mean seconds/call | 0.0 | 0.0 |
| est. cost (USD) | 0.0000 | 0.0000 |
Read it from the bottom up. Cost and seconds are zero because the mock costs nothing and answers instantly. Parse failures are zero because the mock always emits valid JSON, which a live model will not always do. The two rows that matter most are numbers_grounded and all 7 rules. The sloppy model fails grounding on far more headlines than the steady one and lands clean headlines much less often. To see how a single bad call looks, score one saved response headline by headline:
How to Get rotate SSH keys Right Without the Headaches That Slow Teams Down | ['length_ok', 'has_number']
rotate SSH keys Done Right: 6 Steps | PASS
Fix Your rotate SSH keys in 25 Minutes | PASS
Why rotate SSH keys Matters | ['has_number']
6 Steps to Better rotate SSH keys | PASS
rotate SSH keys in 6 Steps | PASS
Unlock rotate SSH keys With This One Trick | ['has_number', 'no_banned']
rotate SSH keys: 87% Faster Setups | ['numbers_grounded']
The last line is the case this whole test exists to catch. The headline has a number, so has_number passes. But 87 appears nowhere in the brief’s facts, so numbers_grounded fails. A model that only passes the contains-a-digit rule would look excellent on a shallow check and still ship a fabricated statistic. The blind sheet that python blind.py make writes looks like this, with the answer key stored separately:
brief_id,option_A,option_B,pick
cert-renewal,A 40-Minute Fix for TLS certificate renewal,TLS certificate renewal Done Right: 7 Steps,
dns-audit,"DNS audit: 5 Steps, 30 Min",Unlock DNS audit With This One Trick,
docker-slim,A 35-Minute Fix for Docker image size,Tackle Docker image size in Under 35 Minutes,
Finally, the scoring step. I filled a sheet with coin-flip picks to test the code path, and this is what python blind.py score rater1.csv printed:
mock-steady 7 of 10 votes (70%)
mock-sloppy 3 of 10 votes (30%)
Those picks were random. One label still collected 70 percent of the votes. Ten votes from one rater prove almost nothing, which is why the decision rule in Step 12 asks for three raters and a wide margin.
Cost and Time Budget for a Live Run #
A live run is cheap, but the cost scales with output tokens, and output tokens include any hidden reasoning a model does before it answers. Assume 30 calls per model (10 briefs times 3 runs), a short input prompt per call and about 1,500 output tokens per call once reasoning is counted. That makes 45,000 output tokens per model. At the per-million prices listed earlier, the totals come out as follows.
| Model | Input cost | Output cost | Total for one full run |
|---|---|---|---|
| Claude Fable 5.2 | Not listed | $1.800 | Not listed |
| Claude Fable 5.2 fast mode | Not listed | $3.600 | Not listed |
| GPT-6 Sol | $0.027 | Not listed | $0.48 |
| GPT-6 Luna | $0.001 | $0.023 | $0.02 |
The premium pair, Fable 5.2 and GPT-6 Sol, is priced per million tokens: $8 input and $40 output for Fable 5.2, $2 input and $10 output for GPT-6 Sol. The assumption that matters is output tokens. If a model burns 6,000 tokens of hidden reasoning on every headline call, the Fable 5.2 line grows quickly, and you want to know that before the invoice arrives. The harness prints real token counts per call, so after the smoke test, multiply the observed output tokens by 30 and by the model’s output price to forecast the full run.
Teams on ChatGPT Business or Enterprise meter some models by credits rather than tokens, as OpenAI documents on its rate card for Business and Enterprise. If your company routes through those plans, your cost per run follows that card, not the table above.
The time budget is easier. Setup and keys take about 10 minutes. Writing 10 briefs takes 15 if you borrow real articles from your backlog. Typing or pasting the code and running the dry run takes 20. The live generation itself finishes in a few minutes. Raters need about 15 minutes in parallel, and reading the results takes the last 10 or so. That sums to the 80 minutes in the title, and most of it is human time rather than compute.
Common Pitfalls in a Claude vs ChatGPT Headline Test #
Comparing unequal tiers. Fable 5.2 costs four times what GPT-6 Sol costs on both input and output, and Sol costs 20 times what Luna costs. A flagship against a budget model is a price test dressed up as a quality test. Pair models by price band, or label the comparison as premium against value and report cost next to quality.
Trusting an alias. ChatGPT’s Latest selection moved to GPT-6 in the model menu, while an explicit GPT-5.6 Sol selection stayed where it was. If your log says latest instead of an ID, nobody can reproduce the run in two weeks. Always record the exact ID or the exact picker label, plus the plan and the date.
Sending a temperature to every model. Some current models reject sampling parameters or ignore them. If you add temperature for one vendor and not the other, you break the symmetry, and if you add it for both, one call may fail. Keep the default behavior unless you have a documented reason.
Letting raters see the labels. A rater who knows which headline came from the model they prefer will pick it. Shuffle the order, strip the names and keep the key in a separate file. Do not run the vote in a shared chat where people can discuss their picks.
Judging from one run. A single call per brief measures luck. Three runs expose variance, and the spread between those runs tells you how large a gap must be before it means anything. If a model’s all-rules pass rate swings from 70 percent to 90 percent across runs, a 5-point lead over its rival is noise.
Capping output tokens too low. Reasoning models spend tokens before they write the answer. If MAX_OUTPUT_TOKENS is small, the response can end mid-JSON, and your parse-failure count blames the model for a limit you set. Start at 4,000 as the config does and raise it if parse failures appear.
Writing rules that reward gaming. A rule that only checks for a digit invites invented numbers. Pairing has_number with numbers_grounded closes that gap. When you add a rule, ask how a lazy model could satisfy it without doing the real work. The same lesson applies to long drafts in an essay writing comparison, where length targets invite padding.
Troubleshooting: 10 Failures and Their Fixes #
Set the model ID for fable-5.2 (see config.py) before running.TheCLAUDE_MODEL_IDvariable is empty in your current shell. Export it again in the terminal where you run the script, because a new terminal tab starts without it.- Anthropic rejects the model ID. The ID you pasted does not match what the API accepts. Copy it from Anthropic’s model documentation instead of typing it from memory, and check for stray spaces or quotes.
ModuleNotFoundError: No module named 'anthropic'(oropenai). The virtual environment is not active. Runsource .venv/bin/activate, confirm withwhich python, then reinstall withpip install anthropic openai.- OpenAI rejects the request for
gpt-6-sol. Your key or project may lack access to the model, or the rollout has not reached your account. Check the project permissions in the OpenAI dashboard, then trygpt-6-lunato separate a key problem from a model-access problem. - Parse failures climb above zero. Open one saved JSON file and read the
textfield. If the JSON ends abruptly, raiseMAX_OUTPUT_TOKENS. If the model added a paragraph before the JSON, the parser trims it, so look instead for invalid commas or unescaped quotes inside a headline. - HTTP 429 or rate-limit errors during generation. The loop calls models back to back. Add a short
time.sleep(1)between calls, rerun one model at a time, or lowerRUNS_PER_BRIEFto 2 while you wait for your limits to reset. report.pycrashes with an index error or prints an empty table. The newest folder inruns/is probably an aborted run with no response files. List the folder withls runs, delete the empty one, or pass the right path:python report.py runs/20261121-173027.has_keywordfails on every headline for one brief. The check compares whole words, so a keyword of Postgres backups fails when the headline says Postgres backup. Rewrite the keyword in the brief to match the form you want, or add stemming if your keywords vary.- Both blind options look identical. The two models produced the same lead headline, which happens with short keywords and tight rules. Tell raters to leave those rows blank, since a tie is a legitimate result.
- The cost row shows 0.0000 on a live run. Either the prices in
config.pyare zero for that label or the provider returned no usage data. Printresult["tokens_in"]for one call, and if it reads 0, check that you are reading theusageobject from the response and not a wrapper.
Advanced Tips for a Sharper Comparison #
Add a cost-matched arm. Fable 5.2 against GPT-6 Sol is premium versus premium. Add Claude Haiku 5.5 against GPT-6 Luna for the value tier. Copy the config entry, set the model ID and the price you verified on Anthropic’s pricing page, and rerun. The report adds a column automatically.
Test fast mode as its own arm. If your Claude account exposes Fable 5.2’s fast mode, add it under a new label with the $16 and $80 prices. Compare mean seconds per call and the cost column, then ask whether a speed-up of up to 2.5 times is worth paying double for headline work, where latency rarely matters.
Rescore instead of regenerating. Because every raw response is saved, you can change a rule, a threshold or the banned list and rerun python report.py runs/<folder> at no cost. Keep a changelog of rule edits so a later reader can see why numbers moved.
Add your published headlines as a control. Pull 10 headlines your editors wrote for similar briefs and treat them as a third arm in the blind sheet. If both models lose to your team on appeal while beating them on rule compliance, you have learned how to split the work: let the model enforce style rules and let editors choose.
Run a prompt ablation. Rerun the same briefs with a looser prompt that drops the number rule. A model whose grounding failures jump when the rule disappears depends heavily on instructions. A model whose failures barely change has habits you may not want. For a longer-horizon view of habits across many turns, the long-form drift test applies the same logic to 3,000-word drafts.
Wire it into CI. Run the unit tests and the offline dry run on every change to checks.py or the prompt. The mock path needs no keys, so it works in any pipeline, and it catches a broken rule before it corrupts a live comparison.
Pre-register your decision rule. Write the winning margin, the number of raters and the tie handling into a file before the first live run. People who set the rule afterward tend to pick the one that favors their favorite. For more model comparisons across vendors, browse the AI and machine learning section.
Complete Project: File Tree and Run Order #
The finished project has six Python modules, one test file, a JSON file of briefs and a two-line requirements file. The code stays compact. This is the full tree after a dry run:
headline_bench/
requirements.txt
config.py
briefs.json
providers.py
generate.py
checks.py
report.py
blind.py
tests/
test_checks.py
runs/
20261121-173027/
models.json
report.md
blind_sheet.csv
blind_key.json
mock-steady/ssh-keys-1.json ... cert-renewal-3.json
mock-sloppy/ssh-keys-1.json ... cert-renewal-3.json
The requirements.txt file holds two lines, anthropic and openai, and you should replace them with pinned versions from pip freeze after your first live run. The run order is always the same. Start with the unit tests, then generate, then report, then build the blind sheet, then collect votes and score them. Because each stage reads files the previous stage wrote, you can stop after any step and resume later. If a run goes wrong halfway, delete that run folder and start again. The folders are timestamped, so nothing overwrites an earlier result.
FAQ: Claude vs ChatGPT for Headline Writing #
Which is better for headlines, Claude or ChatGPT?
The sources checked for this tutorial do not include an independent, dated benchmark of headline quality for Claude Fable 5.2 or GPT-6 Sol. Anthropic frames Fable 5.2 around technical work, and OpenAI frames its new models around reasoning, coding and professional tasks. Run the harness on your own briefs and decide from your failure rates.
Do I need paid chat subscriptions to run this test?
No. The harness uses API keys and bills per token, so the cost depends on which models you pick and how many tokens they use. Paid plans only matter if you want to test the chat apps by hand. In that case ChatGPT Plus is $20 per month, and Claude’s current plan prices sit on Anthropic’s pricing page.
Can I test inside the chat apps instead of the API?
Yes, but the result is less controlled. Paste the same prompt into each app, copy the JSON into a file shaped like the saved run files and record the model label shown in the picker, your plan and the date. Because OpenAI’s rollout of the GPT-6 models has been gradual, two users on the same plan may see different defaults.
How many briefs do I need?
Ten is a minimum for a first pass. Thirty gives steadier results. With three runs and eight headlines per call, 10 briefs yield 240 headlines per model, enough to see a gap of 10 to 15 percentage points on a rule. Smaller differences need more briefs.
Why not let a model judge the headlines?
A model judge can run as a third signal, but it carries a known weakness: models tend to prefer text that resembles their own output. Rules and blind human votes stay neutral between vendors. If you add a model judge, keep it out of the winner decision and use it only to flag rows for a human to review.
Does the same harness work for meta descriptions or social posts?
Yes. The meta description already has its own checks. For social posts, change the length limit in config.py, replace the banned list and edit the rules in check_headline. Keep the structure of facts, rules and blind votes.
Which model should a small newsroom choose?
Choose by cost per clean headline, not by brand. Fable 5.2 costs four times what GPT-6 Sol costs per token, and Luna costs one twentieth of Sol. If a cheaper model clears your rules at a similar rate, its price advantage compounds across thousands of headlines. If it fails the grounding rule, the savings disappear the first time you publish a bad number.