Claude vs ChatGPT Story Variety: 12 Steps, 85 Min [2026]

Anthropic recently released Claude Fable 5.2 and says testers found it a clearer communicator than earlier versions. Any writing comparison built on earlier models is already out of date.

Most Claude vs ChatGPT for writing posts grade a single answer per prompt, then declare a winner on vibes. That misses a failure readers notice fast: the same hero name, the same opening line, the same coffee-shop slogan on every run. Voice fidelity is one axis, and the voice-match stylometry test covers it. Variety is a separate axis, and it decides whether a model is useful for brainstorming, fiction drafts and copy ideation.

The harness below sends four creative prompts to both models ten times each, in two conditions, then scores the pools locally with five metrics. No LLM judge sits in the loop. You get distinct n-gram ratios, pairwise overlap, opener repetition, name reuse and cliche rate, plus a shuffled blind sheet for your own eyes. Budget about 85 minutes and a couple of dollars at list prices.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why Claude vs ChatGPT Variety Is the Missing Writing Test #

Ask any chatbot for a short story about a lighthouse keeper ten times in separate chats. You will often see the same skeleton: a weathered keeper, a storm, a mysterious light, a last-minute choice. Individual drafts read fine. The pool reads like one draft with the nouns shuffled. Writers call this sameness, and it costs real time because you cannot brainstorm with a tool that keeps handing back its first idea.

Variety matters most in four jobs: brainstorming premises, naming characters and products, drafting ad-copy variants, and generating fiction beats you plan to rewrite. In each of them you ask the same tool the same open question many times and expect different answers. Claude vs ChatGPT for writing is usually framed as a quality contest, but a tool that scores a point higher on polish and returns the same three ideas is worse for these jobs than a slightly rougher tool that surprises you.

The effect has two layers. The first is lexical, which means repeated phrases, openers and character names. You can measure that with plain string math. The second is conceptual, which means the same plot beat or joke wrapped in new words. Local metrics only catch the first layer, so the blind step at the end of the tutorial exists to catch the second.

Variety also interacts with every other writing property. A model that obeys an instruction perfectly may do it in the same three ways each time. A model that preserves your text during edits, a property the editing fidelity test measures, may still produce a tired first draft. Treat the metrics as a profile, not a ranking. One model can win on variety and lose on length control, and that is a usable result.

Two design decisions keep the test honest. First, both models see the identical prompt text in every run, with no system prompt and no custom instructions, so product-level memory features cannot leak into the comparison. Second, the harness runs a plain condition and a seeded condition. The seeded condition adds a random constraint card (a setting and an object) to each prompt. If a model repeats itself in the plain condition but diversifies when handed a card, the sameness comes from the prompt’s open space, not from an inability to vary. That distinction changes how you should prompt it.

Sampling temperature stays untouched on purpose. Newer reasoning-style models in both families may restrict or ignore sampling controls, and a test that sets temperature on one vendor but not the other compares settings instead of models. Running ten trials per prompt gives you the spread that a single dial would only hint at.

Prerequisites and Versions #

The harness is plain Python with two vendor SDKs. Nothing else is required, and the scoring code uses only the standard library, so you can score a pool offline.

• A current Python release.
• The anthropic SDK, latest release from pip. It provides the Messages API client for the Claude side.
• The openai SDK, latest release from pip. It provides the Chat Completions client for the ChatGPT side.
• An Anthropic API key from a console account with billing enabled. Consumer Claude plans do not include API access.
• An OpenAI API key from a platform account with billing enabled. A ChatGPT subscription does not include API access.
• Under 50 MB of disk. Outputs are short text files.
• About 85 minutes. Budget 15 for setup, 25 for coding, 25 for generation and 20 for scoring and the blind review.

Keep API access and subscriptions separate in your head. The apps (Claude and ChatGPT) apply their own system prompts, memory and model routing. The API gives you a named model and nothing else, which is what a fair test needs. If you only have the apps, you can still run the prompts by hand and drop the outputs into the same folder layout, but expect the manual route to take far longer than ten trials times eight cells.

If you have run a README accuracy test or any other multi-vendor script before, reuse the same virtual environment. Both SDKs read their keys from environment variables, which keeps secrets out of the repository.

Do not paste keys into the scripts. A leaked key in a public repo gets scraped within minutes, and a creative-writing sweep is exactly the sort of job you might share on a gist later.

Which Models to Test on November 14, 2026 #

Model names matter more than usual this week. Both vendors changed their lineups inside the last few weeks, and the public changelog tracker shows how fast the cadence runs. Test the newest model each account can actually reach, and write the exact ID into your results file. The table lists what the research sheet confirmed as of today.

Model	Vendor	Status	API price per 1M tokens (input / output)
Claude Fable 5.2	Anthropic	Newest general-purpose model	$8 / $40 (fast mode $16 / $80)
Claude Opus 5.5	Anthropic	Announced September 22, 2026, 40% cheaper to run than Opus 5 on typical workloads	$4 / $20
Claude Sonnet 5.5	Anthropic	Announced September 28, 2026, 30%+ faster than Sonnet 5	Check the pricing page
Claude Haiku 5.5	Anthropic	Listed as Anthropic’s fastest, most capable small model	Check the pricing page
GPT-6 Sol	OpenAI	Announced November 13, 2026	Check the rate card
GPT-6.1 Terra	OpenAI	Began rolling out October 26, 2026 (API ID gpt-6.1-terra)	$1 / $5
GPT-6 Luna	OpenAI	Announced November 13, 2026	$0.10 / $0.50

The Claude side

Fable 5.2 is the model to feature. It reached Pro, Max, Ultra, Team and Enterprise users on November 9 and is also available through the developer platform and the major cloud marketplaces, according to Anthropic’s announcement as summarized by an AI news archive. Anthropic also released Mythos 5.2 that day, but it is limited to vetted cyberdefenders and life-sciences researchers (see Anthropic’s resources page), so it is not a writing candidate you can reach. Fable 5.1 is superseded by 5.2, and the older $10 and $50 Fable price no longer applies.

For budget runs, Opus 5.5 at $4 and $20 per million tokens is the mid-tier pick, and Haiku 5.5 suits a cheap smoke test. Anthropic’s model list reportedly uses claude-fable-5-2 as the API identifier for Fable 5.2, but confirm the string in your own console before you export it. A single wrong character produces a 404 from the API.

The ChatGPT side

OpenAI’s release notes describe a fast rollout of the GPT-6 family. GPT-6.1 Terra started rolling out on October 26 in the API and in ChatGPT Work and Codex. OpenAI’s Work and Codex documentation lists Sol and Luna as not available in ordinary ChatGPT conversations, which is one more reason to test through the API, where the model ID is explicit.

Pairing is the hard part. Fable 5.2 costs eight times as much per input token as Terra, and the AI model coverage on this site tracks how often the lineups shift. Run three pairings and label each one: frontier against frontier (Fable 5.2 against GPT-6 Sol), mid-tier (Opus 5.5 against Terra), and small (Haiku 5.5 against Luna). Keep each pairing in its own output folder so the scores never mix.

Retired models belong in the history section of your notes, not in the test. OpenAI has retired GPT-4o, GPT-4.1 and o4-mini from ChatGPT, and GPT-5 Instant and Thinking went with them. Any creative-writing verdict you read that rests on those models describes software you can no longer pick in the app.

Plan names also changed. ChatGPT’s consumer plans are Free, Go, Plus and Pro, per the official ChatGPT pricing page, and OpenAI’s help pages list Pro tiers at $100, $200 and $500 a month. Claude’s consumer plans are Pro, Max and Ultra above the free tier. Subscription tiers do not matter for the API harness, but note them if you also run the apps by hand.

Steps 1 and 2: Create the Folder, Environment and Dependencies #

Start with an isolated environment so the SDK versions you test with are the versions you can reproduce next month. Vendors ship SDK updates often, and a result that depends on an unpinned library is hard to defend.

mkdir varietybench
cd varietybench
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install --upgrade anthropic openai
pip freeze | grep -E "anthropic|openai" > requirements.lock
mkdir out blind

Add a .gitignore with .venv, out and blind before you commit anything. Generated outputs bloat a repository fast, and the blind folder holds your answer key, which you do not want a collaborator to stumble on before they rate the sheet. Commit the prompts, the cliche list and the notes file instead, since those define the experiment and are what someone else needs to reproduce it.

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead of the source line. The pip freeze line records the exact SDK versions in requirements.lock, so you can paste them into your notes. Everything else in the project is five small files that you will create in the next steps.

Check the install before moving on. A broken import now saves you from a confusing stack trace after you have spent money on API calls.

python -c "import anthropic, openai; print('anthropic', anthropic.__version__, 'openai', openai.__version__)"

Expected output is a single line with two version strings. If either import fails, the virtual environment is not active. Run the activation command again and repeat the check.

Steps 3 and 4: Write the Prompts, the Cliche List and the Model IDs #

The prompt file decides what you measure. Four prompts cover different creative shapes: a descriptive opening, a dialogue scene, a monologue and a list of short copy lines. Each carries a length rule so the harness can flag runs that ignore it.

[
  {
    "id": "lighthouse",
    "text": "Write the opening 120 to 160 words of a short story about a lighthouse keeper. Third person only. No title.",
    "unit": "words", "min": 120, "max": 160
  },
  {
    "id": "getaway",
    "text": "Write a 120 to 160 word scene in which two characters argue about who will drive the getaway car. Use dialogue. No title.",
    "unit": "words", "min": 120, "max": 160
  },
  {
    "id": "villain",
    "text": "Write a 100 to 140 word monologue from a villain who believes he is the hero. No title.",
    "unit": "words", "min": 100, "max": 140
  },
  {
    "id": "taglines",
    "text": "Give me 10 taglines for a coffee shop that opens at 5 a.m. One per line, no numbering, no commentary.",
    "unit": "lines", "min": 10, "max": 10
  }
]

Notice the constraints inside each prompt. Third person only, no title, one per line. These rules give the scorer something objective to check and stop models from padding with a heading that would skew the n-gram counts. If you add prompts, keep the same fields: id, text, unit, min and max.

The cliche list is the second input. It holds stock phrases that creative-writing tools overuse. The list below is a starting point for fiction and coffee copy, and you should edit it freely, because a cliche is only a cliche relative to your genre. Keep entries lowercase. The scorer lowercases the output before it counts.

a shiver ran down
little did
heart pounded
heart raced
the air was thick
couldn't help but
eyes sparkled
weathered face
salt-sprayed
whispered secrets
for the first time in
the world seemed to
steal your morning
wake up and smell
rise and grind
brewed to perfection
more than just coffee

Steer clear of padding the list with generic filler words. A list that matches half of ordinary English will flag every run and tell you nothing. Aim for phrases that a human editor would circle in red. To put dollars on usable drafts instead of raw output, see the writing cost per accepted words test.

Now set the keys and model IDs. Export them in the shell instead of writing them into any file, and use one terminal session per pairing so the labels stay straight.

export ANTHROPIC_API_KEY="your-anthropic-key"
export OPENAI_API_KEY="your-openai-key"
export CLAUDE_MODEL="claude-fable-5-2"   # confirm the exact ID in your console
export OPENAI_MODEL="gpt-6.1-terra"      # or the Sol or Luna ID from the models page
export OUT_DIR="out"                     # use a new folder per pairing, e.g. out_mid

The scripts read CLAUDE_MODEL and OPENAI_MODEL at call time and fail loudly with a KeyError if either is missing. That is deliberate. A default model hidden in the code would let a stale name slip into a published result. The OUT_DIR variable lets one project folder hold several pairings without overwriting anything.

Write down the date, the two model IDs and the SDK versions in a notes file now. Vendors patch models behind stable names, so a number without a date is a weak claim.

Step 5: Build the Variety Metrics Module #

All scoring lives in one standard-library file. Each function takes a list of texts, which is one pool of ten runs for one model on one prompt, and returns a single number. Keeping the metrics separate from file handling makes them easy to unit test and easy to swap.

import re
from collections import Counter
from itertools import combinations

WORD = re.compile(r"[a-z0-9]+(?:['’-][a-z0-9]+)*")
CAPS = re.compile(r"\b[A-Z][a-z]{2,}\b")
LOWER = re.compile(r"\b[a-z]{3,}\b")

def tokens(text):
    return WORD.findall(text.lower().replace("’", "'"))

def ngrams(toks, n):
    return [tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)]

def distinct_n(texts, n):
    grams = [g for t in texts for g in ngrams(tokens(t), n)]
    return len(set(grams)) / len(grams) if grams else 0.0

def mean_jaccard(texts, n=3):
    sets = [set(ngrams(tokens(t), n)) for t in texts]
    pairs = [(a, b) for a, b in combinations(sets, 2) if a or b]
    if not pairs:
        return 0.0
    return sum(len(a & b) / len(a | b) for a, b in pairs) / len(pairs)

def cross_jaccard(texts_a, texts_b, n=3):
    sa = [set(ngrams(tokens(t), n)) for t in texts_a]
    sb = [set(ngrams(tokens(t), n)) for t in texts_b]
    pairs = [(a, b) for a in sa for b in sb if a or b]
    if not pairs:
        return 0.0
    return sum(len(a & b) / len(a | b) for a, b in pairs) / len(pairs)

def top_opener_share(texts, k=3):
    heads = [" ".join(tokens(t)[:k]) for t in texts if tokens(t)]
    if not heads:
        return 0.0
    return Counter(heads).most_common(1)[0][1] / len(heads)

def top_name_share(texts):
    lower_seen = {w for t in texts for w in LOWER.findall(t)}
    seen = Counter()
    for t in texts:
        seen.update({w for w in CAPS.findall(t) if w.lower() not in lower_seen})
    if not texts or not seen:
        return 0.0
    return seen.most_common(1)[0][1] / len(texts)

def cliche_rate(texts, cliches):
    words = sum(len(tokens(t)) for t in texts)
    low = [t.lower().replace("’", "'") for t in texts]
    hits = sum(t.count(c) for t in low for c in cliches)
    return 1000.0 * hits / max(words, 1)

def size_of(text, unit):
    if unit == "lines":
        return len([ln for ln in text.splitlines() if ln.strip()])
    return len(tokens(text))

Each metric answers a different question, and none of them is sufficient alone. The red-flag thresholds in the table are rules of thumb, not published standards, so tune them to your genre after you see a few real pools.

Metric	What it measures	Better is	Red flag (rule of thumb)
distinct2 / distinct3	Share of unique word bigrams or trigrams across the whole pool	Higher	Below 0.2 on trigrams means the pool recycles whole phrases
jaccard3	Mean 3-gram overlap between every pair of runs	Lower	Above 0.25 means any two runs share a quarter of their phrasing
opener_share	Share of runs that start with the most common first three words	Lower	Above 0.5 means half the pool opens identically
name_share	Share of runs containing the most common proper-noun candidate	Lower	Above 0.6 means one name dominates the pool
cliche_per_1k	Listed stock phrases per 1,000 words	Lower	Any steady nonzero rate on a short prompt
length_ok	Share of runs inside the requested size	Higher	Below 0.9 means the model ignores length rules
cross_jaccard	3-gram overlap between the two models’ pools	Lower	High values mean both models converge on one answer

Two implementation details deserve a second look. The name metric treats a capitalized word as a name candidate only if its lowercase form never appears anywhere in the pool. That filters out sentence-starting words like the and she without a part-of-speech tagger, though it can misfire on a capitalized slogan word in the taglines prompt. Treat name_share as meaningful for fiction prompts and ignore it for taglines.

The opener metric uses the first three tokens after lowercasing. Three is a compromise. Two catches too many harmless matches such as the word the followed by a noun, and five almost never matches even when the openings feel identical to a human. Change the k argument if your prompts invite longer set phrases.

The metrics assume at least ten runs per pool. With fewer, distinct-n ratios inflate and Jaccard values swing wildly. A pool of three tells you almost nothing.

Step 6: Write the Generator for Both Vendors #

The generator loops over prompts, runs and models, calls each API with the same text and writes one file per output. It skips files that already exist, so an interrupted sweep resumes where it stopped. A thread pool with three workers keeps the sweep fast without tripping rate limits on most accounts. For writing that must stay tied to real sources, the source-grounded writing audit is the companion test.

import argparse
import json
import os
import random
import sys
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

OUT = Path(os.environ.get("OUT_DIR", "out"))

SETTINGS = ["a night ferry", "a closed ski resort", "a desert radio station", "a hospital laundry",
            "a high-school reunion", "a container port", "a village wedding", "an orbital hotel",
            "a night market", "a flooded library", "a mountain post office", "a traveling circus"]
OBJECTS = ["a cracked compass", "a borrowed umbrella", "a locked suitcase", "a broken metronome",
           "a jar of buttons", "a paper crane", "an old ticket stub", "a dented trumpet",
           "a pair of binoculars", "a handwritten menu", "a rusted key", "a tin whistle"]

def seed_card(run):
    rng = random.Random(1000 + run)
    return f"Use this card for inspiration: setting = {rng.choice(SETTINGS)}, object = {rng.choice(OBJECTS)}."

def call_claude(prompt):
    import anthropic

    client = anthropic.Anthropic()
    r = client.messages.create(
        model=os.environ["CLAUDE_MODEL"],
        max_tokens=4000,
        messages=[{"role": "user", "content": prompt}],
    )
    text = "".join(b.text for b in r.content if b.type == "text")
    return text, r.usage.input_tokens, r.usage.output_tokens

def call_chatgpt(prompt):
    from openai import OpenAI

    client = OpenAI()
    r = client.chat.completions.create(
        model=os.environ["OPENAI_MODEL"],
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content, r.usage.prompt_tokens, r.usage.completion_tokens

MODELS = {"claude": call_claude, "chatgpt": call_chatgpt}

def with_retry(fn, prompt):
    for attempt in range(4):
        try:
            return fn(prompt)
        except Exception as exc:
            wait = 2 ** attempt * 3
            print(f"  retry in {wait}s: {type(exc).__name__}: {exc}", file=sys.stderr)
            time.sleep(wait)
    raise RuntimeError("giving up after 4 attempts")

def job(args):
    cond, label, pid, run, prompt = args
    dest = OUT / cond / label / f"{pid}-{run:02d}.txt"
    if dest.exists():
        return None
    dest.parent.mkdir(parents=True, exist_ok=True)
    start = time.time()
    text, tin, tout = with_retry(MODELS[label], prompt)
    dest.write_text(text.strip() + "\n", encoding="utf-8")
    rec = {"cond": cond, "label": label, "prompt": pid, "run": run,
           "in": tin, "out": tout, "secs": round(time.time() - start, 1)}
    print(f"{cond:7} {label:8} {pid:11} run {run:02d} ok ({tout} out tokens)")
    return rec

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--cond", choices=["plain", "seeded"], required=True)
    ap.add_argument("--runs", type=int, default=10)
    ap.add_argument("--workers", type=int, default=3)
    a = ap.parse_args()

    prompts = json.loads(Path("prompts.json").read_text(encoding="utf-8"))
    jobs = []
    for p in prompts:
        for run in range(1, a.runs + 1):
            text = p["text"] if a.cond == "plain" else f"{p['text']}\n\n{seed_card(run)}"
            for label in MODELS:
                jobs.append((a.cond, label, p["id"], run, text))

    OUT.mkdir(exist_ok=True)
    with ThreadPoolExecutor(max_workers=a.workers) as pool, (OUT / "runs.jsonl").open("a", encoding="utf-8") as log:
        for rec in pool.map(job, jobs):
            if rec:
                log.write(json.dumps(rec) + "\n")
                log.flush()

if __name__ == "__main__":
    main()

Three choices in this file are worth explaining. First, the seeded condition derives its card from a fixed random seed per run number, so run 4 uses the same setting and object for Claude and for ChatGPT. The two models face identical cards, and your results are reproducible tomorrow. Second, the Claude call joins only the blocks whose type is text, so any thinking blocks a reasoning model returns do not pollute the output file. Third, the OpenAI call sets no token cap, because newer reasoning models may expect a different parameter name for caps, and the wrong one raises an error.

The retry wrapper waits three, six, twelve and twenty-four seconds. That covers brief rate-limit spikes and transient 5xx errors. It does not hide a wrong model ID, because a 404 fails four times in a row and then stops the run with a clear message.

Expect hidden reasoning tokens on some models. Output-token counts in the log include them on most vendors, which means a short story can bill for far more than its visible length. The log file records input and output tokens per call so you can reconcile the invoice later.

Steps 7 to 9: Smoke Test, Plain Run and Seeded Run #

Never launch the full sweep first. A one-run smoke test costs a few cents and proves both keys, both model IDs and the file layout work.

python generate.py --cond plain --runs 1
ls out/plain/claude out/plain/chatgpt

You should see eight files in total, four per model. The console prints one line per call. The token count in each line will vary by model and by how much hidden reasoning it used.

plain   claude   lighthouse  run 01 ok (<n> out tokens)
plain   chatgpt  lighthouse  run 01 ok (<n> out tokens)
plain   claude   getaway     run 01 ok (<n> out tokens)
...

Open two of the files and read them. If one is empty, the model spent its whole budget thinking, so raise max_tokens in the Claude call or switch to a smaller reasoning setting. If one contains a title line or commentary, note it, because that is a data point about instruction following. Then run the full plain condition.

python generate.py --cond plain --runs 10

That is 40 calls per model. At roughly 250 output tokens per call and no hidden reasoning, the arithmetic at list prices comes to about 10,000 output tokens per model per condition. For Fable 5.2 at $40 per million output tokens that is about $0.40, and for Terra at $5 it is about $0.05. Hidden reasoning can multiply those figures several times, so check your usage dashboard after the first condition and decide whether to continue.

Run the seeded condition next. The command is identical except for the flag.

python generate.py --cond seeded --runs 10

Wall-clock time depends on model speed and your rate limits. Anthropic describes Fable 5.2 as up to 30% faster than the 5.1 version it replaced, but speed claims vary by load, so measure it yourself from the secs field in runs.jsonl. While the sweep runs, resist reading outputs one by one. Early impressions bias the blind step later.

If the sweep dies halfway, rerun the same command. Existing files are skipped, and only the missing cells hit the API. Never delete a partial folder to start over unless you intend to pay for the full set again.

Step 10: Score the Pools #

The scorer groups outputs by condition, prompt and model, runs every metric and prints a table. It also writes scores.csv for spreadsheets, and it computes cross-model overlap for each prompt. Instruction compliance is a separate axis, and the writing rules compliance test covers it.

import csv
import os
import json
from collections import defaultdict
from pathlib import Path

import variety as v

OUT = Path(os.environ.get("OUT_DIR", "out"))

def main():
    prompts = {p["id"]: p for p in json.loads(Path("prompts.json").read_text(encoding="utf-8"))}
    cliches = [c.strip().lower() for c in Path("cliches.txt").read_text(encoding="utf-8").splitlines() if c.strip()]

    pools = defaultdict(list)
    for f in sorted(OUT.glob("*/*/*.txt")):
        cond, label = f.parent.parent.name, f.parent.name
        pid = f.stem.rsplit("-", 1)[0]
        if pid in prompts:
            pools[(cond, pid, label)].append(f.read_text(encoding="utf-8"))
    if not pools:
        raise SystemExit("no outputs found in out/")

    rows = []
    for (cond, pid, label), texts in sorted(pools.items()):
        p = prompts[pid]
        ok = sum(1 for t in texts if p["min"] <= v.size_of(t, p["unit"]) <= p["max"])
        rows.append({
            "cond": cond, "prompt": pid, "model": label, "n": len(texts),
            "distinct2": round(v.distinct_n(texts, 2), 3),
            "distinct3": round(v.distinct_n(texts, 3), 3),
            "jaccard3": round(v.mean_jaccard(texts, 3), 3),
            "opener_share": round(v.top_opener_share(texts), 2),
            "name_share": round(v.top_name_share(texts), 2),
            "cliche_per_1k": round(v.cliche_rate(texts, cliches), 1),
            "length_ok": round(ok / len(texts), 2),
        })

    with open("scores.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)

    head = f"{'cond':7} {'prompt':11} {'model':8} {'n':>3} {'dist2':>6} {'dist3':>6} {'jac3':>6} {'open':>5} {'name':>5} {'clich':>6} {'len':>5}"
    print(head)
    for r in rows:
        print(f"{r['cond']:7} {r['prompt']:11} {r['model']:8} {r['n']:>3} {r['distinct2']:>6} {r['distinct3']:>6} "
              f"{r['jaccard3']:>6} {r['opener_share']:>5} {r['name_share']:>5} {r['cliche_per_1k']:>6} {r['length_ok']:>5}")

    print("\ncross-model overlap (mean 3-gram Jaccard, claude vs chatgpt):")
    labels = sorted({k[2] for k in pools})
    if len(labels) == 2:
        a, b = labels
        for (cond, pid, label), texts in sorted(pools.items()):
            if label == a and (cond, pid, b) in pools:
                print(f"  {cond:7} {pid:11} {v.cross_jaccard(texts, pools[(cond, pid, b)]):.3f}")

if __name__ == "__main__":
    main()

The fixture run below proves the pipeline end to end. It uses synthetic text built from random word pools, one pool built to repeat itself and one built to vary, under the labels fixture-low and fixture-high. These are not Claude or ChatGPT outputs, and the numbers carry no information about either vendor. They show what a repetitive pool and a varied pool look like in the table.

cond    prompt      model      n  dist2  dist3   jac3  open  name  clich   len
plain   lighthouse  fixture-high  10  0.216  0.415  0.138   0.2   0.8    0.0   0.0
plain   lighthouse  fixture-low  10  0.127   0.29  0.271   1.0   1.0    8.5   0.0
plain   taglines    fixture-high  10  0.205  0.358  0.195   0.1   0.9    0.0   1.0
plain   taglines    fixture-low  10   0.01   0.01    1.0   1.0   1.0  142.9   1.0

Read the fixture rows like this. The repetitive lighthouse pool opens identically in all ten runs (open 1.0), shares a large part of its trigrams between an average pair and carries a high cliche rate. The varied pool opens identically in only 20% of runs. The taglines fixture shows the extreme case, where ten identical lines give a distinct-trigram ratio of 0.01 and Jaccard of 1.0. The len column reads 0.0 for the lighthouse fixtures because the synthetic text runs under 120 words, which also shows the length check working.

Open scores.csv in a spreadsheet and build a pivot with model across the top and prompt down the side. Color-scale each metric column separately. A pivot shows at a glance whether a model’s weakness is concentrated on one prompt, which is common, or spread across all four, which is a stronger signal about the model itself. Keep the raw outputs next to the pivot, so a surprising cell can be traced back to the actual text in under a minute.

With real data, read each prompt separately before you average anything. Averaging across prompts hides the interesting cases, such as a model that varies story openings well but repeats taglines.

Steps 11 and 12: Read the Results and Run the Blind Check #

Step 11: Interpret the table.

Compare plain against seeded first, within each model. The gap between those two rows is the most useful number in the whole exercise. A large gap means the model needs help escaping its defaults, and a seed card or a random constraint in your prompt will pay off. A small gap in a pool that already scores well means the model varies by itself. A small gap in a pool that scores badly is the worst case, because no prompt trick moved it.

Pattern in the table	Likely meaning	What to do next
High opener_share in plain, low in seeded	Model locks onto one opening unless given a constraint	Add a random constraint card to creative prompts
High name_share in both conditions	A stock character name sits in the model’s defaults	Ban the name in the prompt or supply a name list
Low distinct3 and high jaccard3 in both	Pool recycles whole phrases	Rewrite the prompt with more specific, unusual constraints
High cross_jaccard between vendors	Both models converge on the same idea	Rely on the blind check, since lexical metrics cannot separate them
length_ok under 0.9	Model ignores size rules	Restate limits as a hard maximum and test again
Cliche rate near zero but blind scores say stale	Staleness lives in plot, not phrases	Extend the cliche list with plot beats

A hypothetical reading shows the logic. Suppose one model posts an opener_share of 0.8 in the plain condition and 0.1 in the seeded condition, while the other posts 0.3 and 0.2. The first model repeats itself when the prompt is open and varies when constrained, so you would add a constraint to every brainstorming prompt. The second varies on its own, so the constraint adds little. Neither number says which model writes better prose. That judgment belongs to the blind step and to the other tests in your toolkit.

Then compare models within the same condition. Prefer differences that hold on three or more prompts. A lead on one prompt is noise at this sample size. Resist computing a single winner score, because the weights are a value judgment and you can pick weights that crown either vendor.

Step 12: Run the blind check.

Lexical metrics miss conceptual sameness, so finish with a blind read. The script below samples four outputs per model for one condition and prompt, shuffles them under neutral tags and writes a rating sheet. You rate surprise and polish from 1 to 5 without knowing the vendor, then reveal the averages. The blind writing test covers a wider blind comparison, and this version trims the idea to what a variety check needs.

import csv
import json
import os
import random
import statistics as st
import sys
from collections import defaultdict
from pathlib import Path

OUT = Path(os.environ.get("OUT_DIR", "out"))
BLIND = Path("blind")

def make(cond, pid, per_model=4, seed=7):
    rng = random.Random(seed)
    picks = []
    for model_dir in sorted((OUT / cond).iterdir()):
        files = sorted(model_dir.glob(f"{pid}-*.txt"))
        picks += rng.sample(files, min(per_model, len(files)))
    if not picks:
        raise SystemExit("no outputs for that condition and prompt")
    rng.shuffle(picks)
    BLIND.mkdir(exist_ok=True)
    stem = f"{cond}-{pid}"
    key, sheet = {}, [f"# Blind sheet: {stem}\n", "Rate each text 1-5 for surprise and 1-5 for polish.\n"]
    for i, f in enumerate(picks, 1):
        tag = f"T{i:02d}"
        key[tag] = f.parent.name
        sheet.append(f"\n## {tag}\n\n{f.read_text(encoding='utf-8').strip()}\n")
    (BLIND / f"{stem}-sheet.md").write_text("\n".join(sheet), encoding="utf-8")
    (BLIND / f"{stem}-key.json").write_text(json.dumps(key, indent=2), encoding="utf-8")
    with open(BLIND / f"{stem}-ratings.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["tag", "surprise", "polish"])
        for tag in key:
            w.writerow([tag, "", ""])
    print(f"wrote {len(key)} shuffled texts to {BLIND}/{stem}-sheet.md")

def reveal(cond, pid):
    stem = f"{cond}-{pid}"
    key = json.loads((BLIND / f"{stem}-key.json").read_text(encoding="utf-8"))
    sur, pol = defaultdict(list), defaultdict(list)
    with open(BLIND / f"{stem}-ratings.csv", encoding="utf-8") as fh:
        for row in csv.DictReader(fh):
            if row["surprise"] and row["polish"]:
                sur[key[row["tag"]]].append(int(row["surprise"]))
                pol[key[row["tag"]]].append(int(row["polish"]))
    for label in sorted(sur):
        print(f"{label:8} surprise {st.mean(sur[label]):.2f}  polish {st.mean(pol[label]):.2f}  n={len(sur[label])}")

if __name__ == "__main__":
    cmd, cond, pid = sys.argv[1:4]
    {"make": make, "reveal": reveal}[cmd](cond, pid)
python blind.py make plain lighthouse
python blind.py reveal plain lighthouse

Do the ratings in one sitting and do not open the key file first. Eight texts take about ten minutes. If you have a colleague, have them rate a second copy, and compare. Large disagreement between raters tells you the preference is personal, not a model property.

The reveal prints one line per vendor with mean surprise, mean polish and the number of ratings. Surprise rewards novelty and polish rewards craft, and they often disagree. A model that scores high on both is rare, and a model that scores high on one and low on the other points to a clear use case, such as ideation versus final copy.

Common Pitfalls #

These six mistakes are the easiest to make, because the harness still produces numbers when you commit them.

1. Testing a retired or superseded model. Fable 5.1 gave way to 5.2 on November 9, and the GPT-4o family left ChatGPT in February. Check the vendor model list on the day you run, and write the ID and date next to every table.
2. Mixing app outputs with API outputs. The consumer apps add system prompts, memory and routing. OpenAI also announced on November 10 that it would retire automatic switching from Instant to Thinking for Plus and Pro users, so a manual app test depends on which model you picked. Keep the two sources in separate folders.
3. Running too few trials. Three runs per prompt cannot distinguish a repetitive model from a lucky one. Ten is the practical minimum, and twenty tightens the picture for prompts you care about.
4. Letting early reading bias the blind step. If you skim outputs during the sweep, you will recognize vendor styles later and your ratings stop being blind. Shuffle first, read second.
5. Treating one prompt as the verdict. Dialogue, description and copy lines stress different habits. A model can lead on one and trail on another, so report per-prompt results and check long outputs with thelong-form drift test .
6. Over-trusting name_share on short-copy prompts. Slogans are capitalized by habit, so the name heuristic flags them. Ignore the metric for the taglines prompt and use opener_share and distinct3 instead.

A seventh trap hides in the cost line. Hidden reasoning tokens bill as output on most vendors, so two models with the same visible length can differ several times over in price. Always read the usage numbers from the log rather than estimating from word counts.

Troubleshooting #

Eleven problems come up often enough to list. Each has a quick check and a fix.

1. KeyError: CLAUDE_MODEL or OPENAI_MODEL. The variable is not exported in this shell. Run export again, or check that you did not open a new terminal tab.
2. 404 or model not found. The ID is wrong, retired, or not enabled for your account. Copy the string from the vendor’s model list instead of typing it. Note that Anthropic’s Fable 5.2 ID is reportedly claude-fable-5-2, so a missing hyphen is the usual cause.
3. 401 authentication error. The key is missing, revoked or from the wrong organization. Print the first four characters with echo to confirm which key the shell holds, never the whole key.
4. 429 rate limit. Lower –workers to 1 or 2. The retry wrapper handles brief spikes, but sustained limits need fewer parallel calls.
5. Empty output files. The model used its token budget on hidden reasoning. Raise max_tokens in the Claude call, or choose a smaller reasoning setting where the vendor offers one.
6. OpenAI error about an unsupported parameter. You added max_tokens or temperature to the call. Remove it, or use the parameter name the current documentation lists for your model.
7. score.py prints no outputs found. OUT_DIR points at the wrong folder, or the generator wrote nothing. List out/plain/claude and confirm .txt files exist.
8. Every pool scores as perfectly varied. You have too few runs, or outputs contain different titles and headings that inflate uniqueness. Strip headings and rerun with ten trials.
9. length_ok is zero for every prompt. The models returned markdown lists or extra commentary. Read three files, then tighten the prompt wording or adjust min and max.
10. Unicode characters break the cliche match. Curly apostrophes can defeat a plain string match. The scorer normalizes the right single quote, but other characters may need the same replace call.
11. ModuleNotFoundError for anthropic or openai. The virtual environment is not active. Run the source line again and reinstall if needed.

When a problem persists, rerun the smoke test with one run and one prompt. A minimal failing case is easier to diagnose than a half-finished sweep.

Advanced Tips #

Once the base harness works, five extensions pay for themselves. Each one reuses the same files with a small change.

• Sweep more constraint cards. Grow the SETTINGS and OBJECTS lists to 40 entries each and run 20 trials. A larger card space separates models that follow constraints creatively from models that fold the card into the same plot every time.
• Add a name-ban condition. After the first scoring pass, take the most common name from name_share and ban it in the prompt. If the model switches to a different stock name, you have found a family of defaults rather than one.
• Score a third axis with embeddings. Lexical metrics miss conceptual repetition. Embed each output with any embedding model, then compute mean pairwise cosine similarity per pool. Keep this optional, because it adds a third vendor to the bill.
• Build a two-stage pipeline. Use one model to brainstorm ten premises and another to draft from the best three. Thewriting pipeline test shows how to split roles across a Terra and Opus 5.5 pairing, and this harness can score the brainstorm stage directly.
• Track drift over time. Run the same sweep monthly with the same prompts and seeds, then plot distinct3 per model. Vendors update models behind stable IDs, and a rising jaccard3 is an early sign that a model got more predictable.

Prompt design deserves its own experiment. Compare an open prompt, a prompt with a constraint card and a prompt that asks for ten different premises in one response. Many users assume the third is best, but single-response lists often share structure inside the response, which distinct3 across runs will not catch. Score within-response overlap separately if you try it.

Finally, keep a frozen copy of every output folder. Models change, and a frozen pool lets you rescore with improved metrics later without paying for generation again. Compress the folder, name it with the date and model IDs, and store it with your notes.

The Complete Project and Run Order #

The finished project is six files and two output folders. Every file appeared above in full, so copying them in order gives you a working harness.

varietybench/
  prompts.json      four prompts with size rules
  cliches.txt       stock phrases, one per line
  variety.py        metric functions (standard library only)
  generate.py       calls both vendors, writes out/<cond>/<model>/<prompt>-NN.txt
  score.py          prints the table and writes scores.csv
  blind.py          builds and reveals the blind rating sheet
  out/              generated texts and runs.jsonl
  blind/            sheets, keys and ratings

Run order for a fresh pairing, from an activated environment with keys and model IDs exported:

python generate.py --cond plain --runs 10
python generate.py --cond seeded --runs 10
python score.py
python blind.py make plain lighthouse
python blind.py reveal plain lighthouse

Testing status matters here. The metrics, scorer and blind tool ran end to end on synthetic fixtures before publication, and all five scripts compile cleanly. The generator is the one file that needs live keys, and its vendor calls follow the standard Messages and Chat Completions patterns. Run the one-run smoke test before you spend on a full sweep, as Steps 7 to 9 describe.

To compare a second pairing, change the model variables, set OUT_DIR to a new folder, and repeat. The scorer reads whatever folder OUT_DIR names, so pairings never collide. Keep the notes file beside each folder with the date, IDs and SDK versions.

When you report results, publish the model IDs, the run date, the SDK versions from requirements.lock, the number of trials and the prompt file. Add the caveat that metrics describe your account and a single day. Any Claude vs ChatGPT for writing claim that omits those details is hard to check, and vendors patch models behind stable names often enough that a month-old number can mislead. Link the raw output folder if you can, so readers can rescore it with their own metrics.

Last, decide in advance what result would change your mind. Write it down before you run anything, such as: if one model’s opener_share stays under 0.3 on all four prompts in both conditions, it becomes the default for brainstorming. Pre-registering the threshold stops you from rationalizing whatever the numbers show.

FAQ #

Who wins in Claude vs ChatGPT for writing?

Neither wins everywhere, and a verdict depends on the task and the model version. This harness measures one dimension, variety, and gives you numbers for your own account and date. Voice fidelity, editing discipline and long-form consistency are separate axes with separate tests.

Which models should I compare right now?

Test the newest model each account reaches. On November 14, 2026 that means Claude Fable 5.2 and the GPT-6 family, where GPT-6.1 Terra began rolling out on October 26. Check the exact IDs in each vendor’s console before you start.

How much does the full sweep cost?

At list prices and roughly 250 output tokens per call, one model’s 80 calls come to about 20,000 output tokens. That is about $0.80 on Fable 5.2 at $40 per million output tokens and about $0.10 on GPT-6.1 Terra at $5. Hidden reasoning tokens can multiply the figure, so check the usage dashboard after the first condition.

Why not set the temperature to the same value for both?

Newer reasoning-style models may restrict or ignore sampling controls, and the parameter names differ between vendors. Setting a value on one side but not the other compares settings instead of models. Ten trials per prompt capture the natural spread without touching the dial.

Can I run this with the Claude and ChatGPT apps instead of the APIs?

You can, but results are less clean. The apps add system prompts and memory, and OpenAI announced on November 10 that it would retire automatic Instant to Thinking switching for Plus and Pro users, so the model depends on your picker choice. If you go manual, use fresh chats for every run and save outputs in the same folder layout.

What does a high opener share actually mean?

It means a large fraction of runs begin with the same first three words. A share above 0.5 means half your pool opens identically. Compare the plain and seeded conditions to see whether a constraint card fixes it.

Do these metrics replace human judgment?

No. They catch lexical sameness only. A pool can score well on distinct n-grams and still retell the same plot in new words, which is why the blind rating step stays in the workflow.

source & further reading

shattered.io — original article Why US Chip Curbs Failed: 678K Chips, 34% Share LoRA Fine-Tuning Rank Sweep: 12 Steps, 90 Min [2026] Cloudflare D1 Adds AI Search Source, Caps Results at 50 MB