Claude vs ChatGPT Story Variety: 12 Steps, 85 Min [2026]

A new 12-step, roughly 85-minute harness from Shattered tests Claude versus ChatGPT for writing variety by sending four creative prompts to both models ten times each in plain and seeded conditions, then scoring the output pools locally with five metrics — distinct n-gram ratios, pairwise overlap, opener repetition, name reuse and cliche rate — with no LLM judge in the loop. The test targets a failure the outlet says single-answer comparisons miss: the same hero name, opening line and coffee-shop slogan recurring across runs, which Shattered argues makes a model less useful for brainstorming, fiction drafts and copy ideation. Shattered notes the comparison follows Anthropic's release of Claude Fable 5.2, which the company says testers found a clearer communicator than earlier versions, and that sampling temperature is left untouched because newer reasoning-style models in both families may restrict or ignore sampling controls.

Anthropic recently released Claude Fable 5.2 and says testers found it a clearer communicator than earlier versions https://ohmyai.ai/archive?tag=health . Any writing comparison built on earlier models is already out of date. Most Claude vs ChatGPT for writing posts grade a single answer per prompt, then declare a winner on vibes. That misses a failure readers notice fast: the same hero name, the same opening line, the same coffee-shop slogan on every run. Voice fidelity is one axis, and the voice-match stylometry test https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ covers it. Variety is a separate axis, and it decides whether a model is useful for brainstorming, fiction drafts and copy ideation. The harness below sends four creative prompts to both models ten times each, in two conditions, then scores the pools locally with five metrics. No LLM judge sits in the loop. You get distinct n-gram ratios, pairwise overlap, opener repetition, name reuse and cliche rate, plus a shuffled blind sheet for your own eyes. Budget about 85 minutes and a couple of dollars at list prices. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Claude vs ChatGPT Variety Is the Missing Writing Test Ask any chatbot for a short story about a lighthouse keeper ten times in separate chats. You will often see the same skeleton: a weathered keeper, a storm, a mysterious light, a last-minute choice. Individual drafts read fine. The pool reads like one draft with the nouns shuffled. Writers call this sameness, and it costs real time because you cannot brainstorm with a tool that keeps handing back its first idea. Variety matters most in four jobs: brainstorming premises, naming characters and products, drafting ad-copy variants, and generating fiction beats you plan to rewrite. In each of them you ask the same tool the same open question many times and expect different answers. Claude vs ChatGPT for writing is usually framed as a quality contest, but a tool that scores a point higher on polish and returns the same three ideas is worse for these jobs than a slightly rougher tool that surprises you. The effect has two layers. The first is lexical, which means repeated phrases, openers and character names. You can measure that with plain string math. The second is conceptual, which means the same plot beat or joke wrapped in new words. Local metrics only catch the first layer, so the blind step at the end of the tutorial exists to catch the second. Variety also interacts with every other writing property. A model that obeys an instruction perfectly may do it in the same three ways each time. A model that preserves your text during edits, a property the editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ measures, may still produce a tired first draft. Treat the metrics as a profile, not a ranking. One model can win on variety and lose on length control, and that is a usable result. Two design decisions keep the test honest. First, both models see the identical prompt text in every run, with no system prompt and no custom instructions, so product-level memory features cannot leak into the comparison. Second, the harness runs a plain condition and a seeded condition. The seeded condition adds a random constraint card a setting and an object to each prompt. If a model repeats itself in the plain condition but diversifies when handed a card, the sameness comes from the prompt’s open space, not from an inability to vary. That distinction changes how you should prompt it. Sampling temperature stays untouched on purpose. Newer reasoning-style models in both families may restrict or ignore sampling controls, and a test that sets temperature on one vendor but not the other compares settings instead of models. Running ten trials per prompt gives you the spread that a single dial would only hint at. Prerequisites and Versions The harness is plain Python with two vendor SDKs. Nothing else is required, and the scoring code uses only the standard library, so you can score a pool offline. - A current Python release. - The anthropic SDK, latest release from pip. It provides the Messages API client for the Claude side. - The openai SDK, latest release from pip. It provides the Chat Completions client for the ChatGPT side. - An Anthropic API key from a console account with billing enabled. Consumer Claude plans do not include API access. - An OpenAI API key from a platform account with billing enabled. A ChatGPT subscription does not include API access. - Under 50 MB of disk. Outputs are short text files. - About 85 minutes. Budget 15 for setup, 25 for coding, 25 for generation and 20 for scoring and the blind review. Keep API access and subscriptions separate in your head. The apps Claude and ChatGPT apply their own system prompts, memory and model routing. The API gives you a named model and nothing else, which is what a fair test needs. If you only have the apps, you can still run the prompts by hand and drop the outputs into the same folder layout, but expect the manual route to take far longer than ten trials times eight cells. If you have run a README accuracy test https://shattered.io/chatgpt-vs-claude-readme-accuracy-test-12-steps-2026/ or any other multi-vendor script before, reuse the same virtual environment. Both SDKs read their keys from environment variables, which keeps secrets out of the repository. Do not paste keys into the scripts. A leaked key in a public repo gets scraped within minutes, and a creative-writing sweep is exactly the sort of job you might share on a gist later. Which Models to Test on November 14, 2026 Model names matter more than usual this week. Both vendors changed their lineups inside the last few weeks, and the public changelog tracker https://reconn-ai.com/llm-changelog.php?trk=public post comment-text shows how fast the cadence runs. Test the newest model each account can actually reach, and write the exact ID into your results file. The table lists what the research sheet confirmed as of today. | Model | Vendor | Status | API price per 1M tokens input / output | |---|---|---|---| | Claude Fable 5.2 | Anthropic | Newest general-purpose model | $8 / $40 fast mode $16 / $80 | | Claude Opus 5.5 | Anthropic | Announced September 22, 2026, 40% cheaper to run than Opus 5 on typical workloads | $4 / $20 | | Claude Sonnet 5.5 | Anthropic | Announced September 28, 2026, 30%+ faster than Sonnet 5 | Check the pricing page | | Claude Haiku 5.5 | Anthropic | Listed as Anthropic’s fastest, most capable small model | Check the pricing page | | GPT-6 Sol | OpenAI | Announced November 13, 2026 | Check the rate card | | GPT-6.1 Terra | OpenAI | Began rolling out October 26, 2026 API ID gpt-6.1-terra | $1 / $5 | | GPT-6 Luna | OpenAI | Announced November 13, 2026 | $0.10 / $0.50 | The Claude side Fable 5.2 is the model to feature. It reached Pro, Max, Ultra, Team and Enterprise users on November 9 and is also available through the developer platform and the major cloud marketplaces, according to Anthropic’s announcement as summarized by an AI news archive. Anthropic also released Mythos 5.2 that day, but it is limited to vetted cyberdefenders and life-sciences researchers see Anthropic’s resources page https://claude.com/resources , so it is not a writing candidate you can reach. Fable 5.1 is superseded by 5.2, and the older $10 and $50 Fable price no longer applies. For budget runs, Opus 5.5 at $4 and $20 per million tokens is the mid-tier pick, and Haiku 5.5 suits a cheap smoke test. Anthropic’s model list reportedly uses claude-fable-5-2 as the API identifier for Fable 5.2, but confirm the string in your own console before you export it. A single wrong character produces a 404 from the API. The ChatGPT side OpenAI’s release notes describe a fast rollout of the GPT-6 family. GPT-6.1 Terra started rolling out on October 26 in the API and in ChatGPT Work and Codex. OpenAI’s Work and Codex documentation lists Sol and Luna as not available in ordinary ChatGPT conversations, which is one more reason to test through the API, where the model ID is explicit. Pairing is the hard part. Fable 5.2 costs eight times as much per input token as Terra, and the AI model coverage on this site https://shattered.io/category/ai-machine-learning/ tracks how often the lineups shift. Run three pairings and label each one: frontier against frontier Fable 5.2 against GPT-6 Sol , mid-tier Opus 5.5 against Terra , and small Haiku 5.5 against Luna . Keep each pairing in its own output folder so the scores never mix. Retired models belong in the history section of your notes, not in the test. OpenAI has retired GPT-4o, GPT-4.1 and o4-mini from ChatGPT, and GPT-5 Instant and Thinking went with them. Any creative-writing verdict you read that rests on those models describes software you can no longer pick in the app. Plan names also changed. ChatGPT’s consumer plans are Free, Go, Plus and Pro, per the official ChatGPT pricing page https://chatgpt.com/pricing , and OpenAI’s help pages list Pro tiers at $100, $200 and $500 a month. Claude’s consumer plans are Pro, Max and Ultra above the free tier. Subscription tiers do not matter for the API harness, but note them if you also run the apps by hand. Steps 1 and 2: Create the Folder, Environment and Dependencies Start with an isolated environment so the SDK versions you test with are the versions you can reproduce next month. Vendors ship SDK updates often, and a result that depends on an unpinned library is hard to defend. mkdir varietybench cd varietybench python3 -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install --upgrade anthropic openai pip freeze | grep -E "anthropic|openai" requirements.lock mkdir out blind Add a .gitignore with .venv, out and blind before you commit anything. Generated outputs bloat a repository fast, and the blind folder holds your answer key, which you do not want a collaborator to stumble on before they rate the sheet. Commit the prompts, the cliche list and the notes file instead, since those define the experiment and are what someone else needs to reproduce it. On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead of the source line. The pip freeze line records the exact SDK versions in requirements.lock, so you can paste them into your notes. Everything else in the project is five small files that you will create in the next steps. Check the install before moving on. A broken import now saves you from a confusing stack trace after you have spent money on API calls. python python -c "import anthropic, openai; print 'anthropic', anthropic. version , 'openai', openai. version " Expected output is a single line with two version strings. If either import fails, the virtual environment is not active. Run the activation command again and repeat the check. Steps 3 and 4: Write the Prompts, the Cliche List and the Model IDs The prompt file decides what you measure. Four prompts cover different creative shapes: a descriptive opening, a dialogue scene, a monologue and a list of short copy lines. Each carries a length rule so the harness can flag runs that ignore it. { "id": "lighthouse", "text": "Write the opening 120 to 160 words of a short story about a lighthouse keeper. Third person only. No title.", "unit": "words", "min": 120, "max": 160 }, { "id": "getaway", "text": "Write a 120 to 160 word scene in which two characters argue about who will drive the getaway car. Use dialogue. No title.", "unit": "words", "min": 120, "max": 160 }, { "id": "villain", "text": "Write a 100 to 140 word monologue from a villain who believes he is the hero. No title.", "unit": "words", "min": 100, "max": 140 }, { "id": "taglines", "text": "Give me 10 taglines for a coffee shop that opens at 5 a.m. One per line, no numbering, no commentary.", "unit": "lines", "min": 10, "max": 10 } Notice the constraints inside each prompt. Third person only, no title, one per line. These rules give the scorer something objective to check and stop models from padding with a heading that would skew the n-gram counts. If you add prompts, keep the same fields: id, text, unit, min and max. The cliche list is the second input. It holds stock phrases that creative-writing tools overuse. The list below is a starting point for fiction and coffee copy, and you should edit it freely, because a cliche is only a cliche relative to your genre. Keep entries lowercase. The scorer lowercases the output before it counts. a shiver ran down little did heart pounded heart raced the air was thick couldn't help but eyes sparkled weathered face salt-sprayed whispered secrets for the first time in the world seemed to steal your morning wake up and smell rise and grind brewed to perfection more than just coffee Steer clear of padding the list with generic filler words. A list that matches half of ordinary English will flag every run and tell you nothing. Aim for phrases that a human editor would circle in red. To put dollars on usable drafts instead of raw output, see the writing cost per accepted words test https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/ . Now set the keys and model IDs. Export them in the shell instead of writing them into any file, and use one terminal session per pairing so the labels stay straight. export ANTHROPIC API KEY="your-anthropic-key" export OPENAI API KEY="your-openai-key" export CLAUDE MODEL="claude-fable-5-2" confirm the exact ID in your console export OPENAI MODEL="gpt-6.1-terra" or the Sol or Luna ID from the models page export OUT DIR="out" use a new folder per pairing, e.g. out mid The scripts read CLAUDE MODEL and OPENAI MODEL at call time and fail loudly with a KeyError if either is missing. That is deliberate. A default model hidden in the code would let a stale name slip into a published result. The OUT DIR variable lets one project folder hold several pairings without overwriting anything. Write down the date, the two model IDs and the SDK versions in a notes file now. Vendors patch models behind stable names, so a number without a date is a weak claim. Step 5: Build the Variety Metrics Module All scoring lives in one standard-library file. Each function takes a list of texts, which is one pool of ten runs for one model on one prompt, and returns a single number. Keeping the metrics separate from file handling makes them easy to unit test and easy to swap. python import re from collections import Counter from itertools import combinations WORD = re.compile r" a-z0-9 + ?: '’- a-z0-9 + " CAPS = re.compile r"\b A-Z a-z {2,}\b" LOWER = re.compile r"\b a-z {3,}\b" def tokens text : return WORD.findall text.lower .replace "’", "'" def ngrams toks, n : return tuple toks i:i + n for i in range len toks - n + 1 def distinct n texts, n : grams = g for t in texts for g in ngrams tokens t , n return len set grams / len grams if grams else 0.0 def mean jaccard texts, n=3 : sets = set ngrams tokens t , n for t in texts pairs = a, b for a, b in combinations sets, 2 if a or b if not pairs: return 0.0 return sum len a & b / len a | b for a, b in pairs / len pairs def cross jaccard texts a, texts b, n=3 : sa = set ngrams tokens t , n for t in texts a sb = set ngrams tokens t , n for t in texts b pairs = a, b for a in sa for b in sb if a or b if not pairs: return 0.0 return sum len a & b / len a | b for a, b in pairs / len pairs def top opener share texts, k=3 : heads = " ".join tokens t :k for t in texts if tokens t if not heads: return 0.0 return Counter heads .most common 1 0 1 / len heads def top name share texts : lower seen = {w for t in texts for w in LOWER.findall t } seen = Counter for t in texts: seen.update {w for w in CAPS.findall t if w.lower not in lower seen} if not texts or not seen: return 0.0 return seen.most common 1 0 1 / len texts def cliche rate texts, cliches : words = sum len tokens t for t in texts low = t.lower .replace "’", "'" for t in texts hits = sum t.count c for t in low for c in cliches return 1000.0 hits / max words, 1 def size of text, unit : if unit == "lines": return len ln for ln in text.splitlines if ln.strip return len tokens text Each metric answers a different question, and none of them is sufficient alone. The red-flag thresholds in the table are rules of thumb, not published standards, so tune them to your genre after you see a few real pools. | Metric | What it measures | Better is | Red flag rule of thumb | |---|---|---|---| | distinct2 / distinct3 | Share of unique word bigrams or trigrams across the whole pool | Higher | Below 0.2 on trigrams means the pool recycles whole phrases | | jaccard3 | Mean 3-gram overlap between every pair of runs | Lower | Above 0.25 means any two runs share a quarter of their phrasing | | opener share | Share of runs that start with the most common first three words | Lower | Above 0.5 means half the pool opens identically | | name share | Share of runs containing the most common proper-noun candidate | Lower | Above 0.6 means one name dominates the pool | | cliche per 1k | Listed stock phrases per 1,000 words | Lower | Any steady nonzero rate on a short prompt | | length ok | Share of runs inside the requested size | Higher | Below 0.9 means the model ignores length rules | | cross jaccard | 3-gram overlap between the two models’ pools | Lower | High values mean both models converge on one answer | Two implementation details deserve a second look. The name metric treats a capitalized word as a name candidate only if its lowercase form never appears anywhere in the pool. That filters out sentence-starting words like the and she without a part-of-speech tagger, though it can misfire on a capitalized slogan word in the taglines prompt. Treat name share as meaningful for fiction prompts and ignore it for taglines. The opener metric uses the first three tokens after lowercasing. Three is a compromise. Two catches too many harmless matches such as the word the followed by a noun, and five almost never matches even when the openings feel identical to a human. Change the k argument if your prompts invite longer set phrases. The metrics assume at least ten runs per pool. With fewer, distinct-n ratios inflate and Jaccard values swing wildly. A pool of three tells you almost nothing. Step 6: Write the Generator for Both Vendors The generator loops over prompts, runs and models, calls each API with the same text and writes one file per output. It skips files that already exist, so an interrupted sweep resumes where it stopped. A thread pool with three workers keeps the sweep fast without tripping rate limits on most accounts. For writing that must stay tied to real sources, the source-grounded writing audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ is the companion test. python import argparse import json import os import random import sys import time from concurrent.futures import ThreadPoolExecutor from pathlib import Path OUT = Path os.environ.get "OUT DIR", "out" SETTINGS = "a night ferry", "a closed ski resort", "a desert radio station", "a hospital laundry", "a high-school reunion", "a container port", "a village wedding", "an orbital hotel", "a night market", "a flooded library", "a mountain post office", "a traveling circus" OBJECTS = "a cracked compass", "a borrowed umbrella", "a locked suitcase", "a broken metronome", "a jar of buttons", "a paper crane", "an old ticket stub", "a dented trumpet", "a pair of binoculars", "a handwritten menu", "a rusted key", "a tin whistle" def seed card run : rng = random.Random 1000 + run return f"Use this card for inspiration: setting = {rng.choice SETTINGS }, object = {rng.choice OBJECTS }." def call claude prompt : import anthropic client = anthropic.Anthropic r = client.messages.create model=os.environ "CLAUDE MODEL" , max tokens=4000, messages= {"role": "user", "content": prompt} , text = "".join b.text for b in r.content if b.type == "text" return text, r.usage.input tokens, r.usage.output tokens def call chatgpt prompt : from openai import OpenAI client = OpenAI r = client.chat.completions.create model=os.environ "OPENAI MODEL" , messages= {"role": "user", "content": prompt} , return r.choices 0 .message.content, r.usage.prompt tokens, r.usage.completion tokens MODELS = {"claude": call claude, "chatgpt": call chatgpt} def with retry fn, prompt : for attempt in range 4 : try: return fn prompt except Exception as exc: wait = 2 attempt 3 print f" retry in {wait}s: {type exc . name }: {exc}", file=sys.stderr time.sleep wait raise RuntimeError "giving up after 4 attempts" def job args : cond, label, pid, run, prompt = args dest = OUT / cond / label / f"{pid}-{run:02d}.txt" if dest.exists : return None dest.parent.mkdir parents=True, exist ok=True start = time.time text, tin, tout = with retry MODELS label , prompt dest.write text text.strip + "\n", encoding="utf-8" rec = {"cond": cond, "label": label, "prompt": pid, "run": run, "in": tin, "out": tout, "secs": round time.time - start, 1 } print f"{cond:7} {label:8} {pid:11} run {run:02d} ok {tout} out tokens " return rec def main : ap = argparse.ArgumentParser ap.add argument "--cond", choices= "plain", "seeded" , required=True ap.add argument "--runs", type=int, default=10 ap.add argument "--workers", type=int, default=3 a = ap.parse args prompts = json.loads Path "prompts.json" .read text encoding="utf-8" jobs = for p in prompts: for run in range 1, a.runs + 1 : text = p "text" if a.cond == "plain" else f"{p 'text' }\n\n{seed card run }" for label in MODELS: jobs.append a.cond, label, p "id" , run, text OUT.mkdir exist ok=True with ThreadPoolExecutor max workers=a.workers as pool, OUT / "runs.jsonl" .open "a", encoding="utf-8" as log: for rec in pool.map job, jobs : if rec: log.write json.dumps rec + "\n" log.flush if name == " main ": main Three choices in this file are worth explaining. First, the seeded condition derives its card from a fixed random seed per run number, so run 4 uses the same setting and object for Claude and for ChatGPT. The two models face identical cards, and your results are reproducible tomorrow. Second, the Claude call joins only the blocks whose type is text, so any thinking blocks a reasoning model returns do not pollute the output file. Third, the OpenAI call sets no token cap, because newer reasoning models may expect a different parameter name for caps, and the wrong one raises an error. The retry wrapper waits three, six, twelve and twenty-four seconds. That covers brief rate-limit spikes and transient 5xx errors. It does not hide a wrong model ID, because a 404 fails four times in a row and then stops the run with a clear message. Expect hidden reasoning tokens on some models. Output-token counts in the log include them on most vendors, which means a short story can bill for far more than its visible length. The log file records input and output tokens per call so you can reconcile the invoice later. Steps 7 to 9: Smoke Test, Plain Run and Seeded Run Never launch the full sweep first. A one-run smoke test costs a few cents and proves both keys, both model IDs and the file layout work. python generate.py --cond plain --runs 1 ls out/plain/claude out/plain/chatgpt You should see eight files in total, four per model. The console prints one line per call. The token count in each line will vary by model and by how much hidden reasoning it used. plain claude lighthouse run 01 ok <n out tokens plain chatgpt lighthouse run 01 ok <n out tokens plain claude getaway run 01 ok <n out tokens ... Open two of the files and read them. If one is empty, the model spent its whole budget thinking, so raise max tokens in the Claude call or switch to a smaller reasoning setting. If one contains a title line or commentary, note it, because that is a data point about instruction following. Then run the full plain condition. python generate.py --cond plain --runs 10 That is 40 calls per model. At roughly 250 output tokens per call and no hidden reasoning, the arithmetic at list prices comes to about 10,000 output tokens per model per condition. For Fable 5.2 at $40 per million output tokens that is about $0.40, and for Terra at $5 it is about $0.05. Hidden reasoning can multiply those figures several times, so check your usage dashboard after the first condition and decide whether to continue. Run the seeded condition next. The command is identical except for the flag. python generate.py --cond seeded --runs 10 Wall-clock time depends on model speed and your rate limits. Anthropic describes Fable 5.2 as up to 30% faster than the 5.1 version it replaced, but speed claims vary by load, so measure it yourself from the secs field in runs.jsonl. While the sweep runs, resist reading outputs one by one. Early impressions bias the blind step later. If the sweep dies halfway, rerun the same command. Existing files are skipped, and only the missing cells hit the API. Never delete a partial folder to start over unless you intend to pay for the full set again. Step 10: Score the Pools The scorer groups outputs by condition, prompt and model, runs every metric and prints a table. It also writes scores.csv for spreadsheets, and it computes cross-model overlap for each prompt. Instruction compliance is a separate axis, and the writing rules compliance test https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ covers it. python import csv import os import json from collections import defaultdict from pathlib import Path import variety as v OUT = Path os.environ.get "OUT DIR", "out" def main : prompts = {p "id" : p for p in json.loads Path "prompts.json" .read text encoding="utf-8" } cliches = c.strip .lower for c in Path "cliches.txt" .read text encoding="utf-8" .splitlines if c.strip pools = defaultdict list for f in sorted OUT.glob " / / .txt" : cond, label = f.parent.parent.name, f.parent.name pid = f.stem.rsplit "-", 1 0 if pid in prompts: pools cond, pid, label .append f.read text encoding="utf-8" if not pools: raise SystemExit "no outputs found in out/" rows = for cond, pid, label , texts in sorted pools.items : p = prompts pid ok = sum 1 for t in texts if p "min" <= v.size of t, p "unit" <= p "max" rows.append { "cond": cond, "prompt": pid, "model": label, "n": len texts , "distinct2": round v.distinct n texts, 2 , 3 , "distinct3": round v.distinct n texts, 3 , 3 , "jaccard3": round v.mean jaccard texts, 3 , 3 , "opener share": round v.top opener share texts , 2 , "name share": round v.top name share texts , 2 , "cliche per 1k": round v.cliche rate texts, cliches , 1 , "length ok": round ok / len texts , 2 , } with open "scores.csv", "w", newline="", encoding="utf-8" as fh: w = csv.DictWriter fh, fieldnames=list rows 0 .keys w.writeheader w.writerows rows head = f"{'cond':7} {'prompt':11} {'model':8} {'n': 3} {'dist2': 6} {'dist3': 6} {'jac3': 6} {'open': 5} {'name': 5} {'clich': 6} {'len': 5}" print head for r in rows: print f"{r 'cond' :7} {r 'prompt' :11} {r 'model' :8} {r 'n' : 3} {r 'distinct2' : 6} {r 'distinct3' : 6} " f"{r 'jaccard3' : 6} {r 'opener share' : 5} {r 'name share' : 5} {r 'cliche per 1k' : 6} {r 'length ok' : 5}" print "\ncross-model overlap mean 3-gram Jaccard, claude vs chatgpt :" labels = sorted {k 2 for k in pools} if len labels == 2: a, b = labels for cond, pid, label , texts in sorted pools.items : if label == a and cond, pid, b in pools: print f" {cond:7} {pid:11} {v.cross jaccard texts, pools cond, pid, b :.3f}" if name == " main ": main The fixture run below proves the pipeline end to end. It uses synthetic text built from random word pools, one pool built to repeat itself and one built to vary, under the labels fixture-low and fixture-high. These are not Claude or ChatGPT outputs, and the numbers carry no information about either vendor. They show what a repetitive pool and a varied pool look like in the table. cond prompt model n dist2 dist3 jac3 open name clich len plain lighthouse fixture-high 10 0.216 0.415 0.138 0.2 0.8 0.0 0.0 plain lighthouse fixture-low 10 0.127 0.29 0.271 1.0 1.0 8.5 0.0 plain taglines fixture-high 10 0.205 0.358 0.195 0.1 0.9 0.0 1.0 plain taglines fixture-low 10 0.01 0.01 1.0 1.0 1.0 142.9 1.0 Read the fixture rows like this. The repetitive lighthouse pool opens identically in all ten runs open 1.0 , shares a large part of its trigrams between an average pair and carries a high cliche rate. The varied pool opens identically in only 20% of runs. The taglines fixture shows the extreme case, where ten identical lines give a distinct-trigram ratio of 0.01 and Jaccard of 1.0. The len column reads 0.0 for the lighthouse fixtures because the synthetic text runs under 120 words, which also shows the length check working. Open scores.csv in a spreadsheet and build a pivot with model across the top and prompt down the side. Color-scale each metric column separately. A pivot shows at a glance whether a model’s weakness is concentrated on one prompt, which is common, or spread across all four, which is a stronger signal about the model itself. Keep the raw outputs next to the pivot, so a surprising cell can be traced back to the actual text in under a minute. With real data, read each prompt separately before you average anything. Averaging across prompts hides the interesting cases, such as a model that varies story openings well but repeats taglines. Steps 11 and 12: Read the Results and Run the Blind Check Step 11: Interpret the table. Compare plain against seeded first, within each model. The gap between those two rows is the most useful number in the whole exercise. A large gap means the model needs help escaping its defaults, and a seed card or a random constraint in your prompt will pay off. A small gap in a pool that already scores well means the model varies by itself. A small gap in a pool that scores badly is the worst case, because no prompt trick moved it. | Pattern in the table | Likely meaning | What to do next | |---|---|---| | High opener share in plain, low in seeded | Model locks onto one opening unless given a constraint | Add a random constraint card to creative prompts | | High name share in both conditions | A stock character name sits in the model’s defaults | Ban the name in the prompt or supply a name list | | Low distinct3 and high jaccard3 in both | Pool recycles whole phrases | Rewrite the prompt with more specific, unusual constraints | | High cross jaccard between vendors | Both models converge on the same idea | Rely on the blind check, since lexical metrics cannot separate them | | length ok under 0.9 | Model ignores size rules | Restate limits as a hard maximum and test again | | Cliche rate near zero but blind scores say stale | Staleness lives in plot, not phrases | Extend the cliche list with plot beats | A hypothetical reading shows the logic. Suppose one model posts an opener share of 0.8 in the plain condition and 0.1 in the seeded condition, while the other posts 0.3 and 0.2. The first model repeats itself when the prompt is open and varies when constrained, so you would add a constraint to every brainstorming prompt. The second varies on its own, so the constraint adds little. Neither number says which model writes better prose. That judgment belongs to the blind step and to the other tests in your toolkit. Then compare models within the same condition. Prefer differences that hold on three or more prompts. A lead on one prompt is noise at this sample size. Resist computing a single winner score, because the weights are a value judgment and you can pick weights that crown either vendor. Step 12: Run the blind check. Lexical metrics miss conceptual sameness, so finish with a blind read. The script below samples four outputs per model for one condition and prompt, shuffles them under neutral tags and writes a rating sheet. You rate surprise and polish from 1 to 5 without knowing the vendor, then reveal the averages. The blind writing test https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ covers a wider blind comparison, and this version trims the idea to what a variety check needs. python import csv import json import os import random import statistics as st import sys from collections import defaultdict from pathlib import Path OUT = Path os.environ.get "OUT DIR", "out" BLIND = Path "blind" def make cond, pid, per model=4, seed=7 : rng = random.Random seed picks = for model dir in sorted OUT / cond .iterdir : files = sorted model dir.glob f"{pid}- .txt" picks += rng.sample files, min per model, len files if not picks: raise SystemExit "no outputs for that condition and prompt" rng.shuffle picks BLIND.mkdir exist ok=True stem = f"{cond}-{pid}" key, sheet = {}, f" Blind sheet: {stem}\n", "Rate each text 1-5 for surprise and 1-5 for polish.\n" for i, f in enumerate picks, 1 : tag = f"T{i:02d}" key tag = f.parent.name sheet.append f"\n {tag}\n\n{f.read text encoding='utf-8' .strip }\n" BLIND / f"{stem}-sheet.md" .write text "\n".join sheet , encoding="utf-8" BLIND / f"{stem}-key.json" .write text json.dumps key, indent=2 , encoding="utf-8" with open BLIND / f"{stem}-ratings.csv", "w", newline="", encoding="utf-8" as fh: w = csv.writer fh w.writerow "tag", "surprise", "polish" for tag in key: w.writerow tag, "", "" print f"wrote {len key } shuffled texts to {BLIND}/{stem}-sheet.md" def reveal cond, pid : stem = f"{cond}-{pid}" key = json.loads BLIND / f"{stem}-key.json" .read text encoding="utf-8" sur, pol = defaultdict list , defaultdict list with open BLIND / f"{stem}-ratings.csv", encoding="utf-8" as fh: for row in csv.DictReader fh : if row "surprise" and row "polish" : sur key row "tag" .append int row "surprise" pol key row "tag" .append int row "polish" for label in sorted sur : print f"{label:8} surprise {st.mean sur label :.2f} polish {st.mean pol label :.2f} n={len sur label }" if name == " main ": cmd, cond, pid = sys.argv 1:4 {"make": make, "reveal": reveal} cmd cond, pid python blind.py make plain lighthouse open blind/plain-lighthouse-sheet.md, fill in blind/plain-lighthouse-ratings.csv python blind.py reveal plain lighthouse Do the ratings in one sitting and do not open the key file first. Eight texts take about ten minutes. If you have a colleague, have them rate a second copy, and compare. Large disagreement between raters tells you the preference is personal, not a model property. The reveal prints one line per vendor with mean surprise, mean polish and the number of ratings. Surprise rewards novelty and polish rewards craft, and they often disagree. A model that scores high on both is rare, and a model that scores high on one and low on the other points to a clear use case, such as ideation versus final copy. Common Pitfalls These six mistakes are the easiest to make, because the harness still produces numbers when you commit them. 1. Testing a retired or superseded model. Fable 5.1 gave way to 5.2 on November 9, and the GPT-4o family left ChatGPT in February. Check the vendor model list on the day you run, and write the ID and date next to every table. 2. Mixing app outputs with API outputs. The consumer apps add system prompts, memory and routing. OpenAI also announced on November 10 that it would retire automatic switching from Instant to Thinking for Plus and Pro users, so a manual app test depends on which model you picked. Keep the two sources in separate folders. 3. Running too few trials. Three runs per prompt cannot distinguish a repetitive model from a lucky one. Ten is the practical minimum, and twenty tightens the picture for prompts you care about. 4. Letting early reading bias the blind step. If you skim outputs during the sweep, you will recognize vendor styles later and your ratings stop being blind. Shuffle first, read second. 5. Treating one prompt as the verdict. Dialogue, description and copy lines stress different habits. A model can lead on one and trail on another, so report per-prompt results and check long outputs with the long-form drift test https://shattered.io/claude-vs-chatgpt-long-form-drift-test-12-steps-2026/ . 6. Over-trusting name share on short-copy prompts. Slogans are capitalized by habit, so the name heuristic flags them. Ignore the metric for the taglines prompt and use opener share and distinct3 instead. A seventh trap hides in the cost line. Hidden reasoning tokens bill as output on most vendors, so two models with the same visible length can differ several times over in price. Always read the usage numbers from the log rather than estimating from word counts. Troubleshooting Eleven problems come up often enough to list. Each has a quick check and a fix. 1. KeyError: CLAUDE MODEL or OPENAI MODEL. The variable is not exported in this shell. Run export again, or check that you did not open a new terminal tab. 2. 404 or model not found. The ID is wrong, retired, or not enabled for your account. Copy the string from the vendor’s model list instead of typing it. Note that Anthropic’s Fable 5.2 ID is reportedly claude-fable-5-2, so a missing hyphen is the usual cause. 3. 401 authentication error. The key is missing, revoked or from the wrong organization. Print the first four characters with echo to confirm which key the shell holds, never the whole key. 4. 429 rate limit. Lower –workers to 1 or 2. The retry wrapper handles brief spikes, but sustained limits need fewer parallel calls. 5. Empty output files. The model used its token budget on hidden reasoning. Raise max tokens in the Claude call, or choose a smaller reasoning setting where the vendor offers one. 6. OpenAI error about an unsupported parameter. You added max tokens or temperature to the call. Remove it, or use the parameter name the current documentation lists for your model. 7. score.py prints no outputs found. OUT DIR points at the wrong folder, or the generator wrote nothing. List out/plain/claude and confirm .txt files exist. 8. Every pool scores as perfectly varied. You have too few runs, or outputs contain different titles and headings that inflate uniqueness. Strip headings and rerun with ten trials. 9. length ok is zero for every prompt. The models returned markdown lists or extra commentary. Read three files, then tighten the prompt wording or adjust min and max. 10. Unicode characters break the cliche match. Curly apostrophes can defeat a plain string match. The scorer normalizes the right single quote, but other characters may need the same replace call. 11. ModuleNotFoundError for anthropic or openai. The virtual environment is not active. Run the source line again and reinstall if needed. When a problem persists, rerun the smoke test with one run and one prompt. A minimal failing case is easier to diagnose than a half-finished sweep. Advanced Tips Once the base harness works, five extensions pay for themselves. Each one reuses the same files with a small change. - Sweep more constraint cards. Grow the SETTINGS and OBJECTS lists to 40 entries each and run 20 trials. A larger card space separates models that follow constraints creatively from models that fold the card into the same plot every time. - Add a name-ban condition. After the first scoring pass, take the most common name from name share and ban it in the prompt. If the model switches to a different stock name, you have found a family of defaults rather than one. - Score a third axis with embeddings. Lexical metrics miss conceptual repetition. Embed each output with any embedding model, then compute mean pairwise cosine similarity per pool. Keep this optional, because it adds a third vendor to the bill. - Build a two-stage pipeline. Use one model to brainstorm ten premises and another to draft from the best three. The writing pipeline test https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ shows how to split roles across a Terra and Opus 5.5 pairing, and this harness can score the brainstorm stage directly. - Track drift over time. Run the same sweep monthly with the same prompts and seeds, then plot distinct3 per model. Vendors update models behind stable IDs, and a rising jaccard3 is an early sign that a model got more predictable. Prompt design deserves its own experiment. Compare an open prompt, a prompt with a constraint card and a prompt that asks for ten different premises in one response. Many users assume the third is best, but single-response lists often share structure inside the response, which distinct3 across runs will not catch. Score within-response overlap separately if you try it. Finally, keep a frozen copy of every output folder. Models change, and a frozen pool lets you rescore with improved metrics later without paying for generation again. Compress the folder, name it with the date and model IDs, and store it with your notes. The Complete Project and Run Order The finished project is six files and two output folders. Every file appeared above in full, so copying them in order gives you a working harness. varietybench/ prompts.json four prompts with size rules cliches.txt stock phrases, one per line variety.py metric functions standard library only generate.py calls both vendors, writes out/<cond /<model /<prompt -NN.txt score.py prints the table and writes scores.csv blind.py builds and reveals the blind rating sheet out/ generated texts and runs.jsonl blind/ sheets, keys and ratings Run order for a fresh pairing, from an activated environment with keys and model IDs exported: python generate.py --cond plain --runs 10 python generate.py --cond seeded --runs 10 python score.py python blind.py make plain lighthouse python blind.py reveal plain lighthouse Testing status matters here. The metrics, scorer and blind tool ran end to end on synthetic fixtures before publication, and all five scripts compile cleanly. The generator is the one file that needs live keys, and its vendor calls follow the standard Messages and Chat Completions patterns. Run the one-run smoke test before you spend on a full sweep, as Steps 7 to 9 describe. To compare a second pairing, change the model variables, set OUT DIR to a new folder, and repeat. The scorer reads whatever folder OUT DIR names, so pairings never collide. Keep the notes file beside each folder with the date, IDs and SDK versions. When you report results, publish the model IDs, the run date, the SDK versions from requirements.lock, the number of trials and the prompt file. Add the caveat that metrics describe your account and a single day. Any Claude vs ChatGPT for writing claim that omits those details is hard to check, and vendors patch models behind stable names often enough that a month-old number can mislead. Link the raw output folder if you can, so readers can rescore it with their own metrics. Last, decide in advance what result would change your mind. Write it down before you run anything, such as: if one model’s opener share stays under 0.3 on all four prompts in both conditions, it becomes the default for brainstorming. Pre-registering the threshold stops you from rationalizing whatever the numbers show. FAQ Who wins in Claude vs ChatGPT for writing? Neither wins everywhere, and a verdict depends on the task and the model version. This harness measures one dimension, variety, and gives you numbers for your own account and date. Voice fidelity, editing discipline and long-form consistency are separate axes with separate tests. Which models should I compare right now? Test the newest model each account reaches. On November 14, 2026 that means Claude Fable 5.2 and the GPT-6 family, where GPT-6.1 Terra began rolling out on October 26. Check the exact IDs in each vendor’s console before you start. How much does the full sweep cost? At list prices and roughly 250 output tokens per call, one model’s 80 calls come to about 20,000 output tokens. That is about $0.80 on Fable 5.2 at $40 per million output tokens and about $0.10 on GPT-6.1 Terra at $5. Hidden reasoning tokens can multiply the figure, so check the usage dashboard after the first condition. Why not set the temperature to the same value for both? Newer reasoning-style models may restrict or ignore sampling controls, and the parameter names differ between vendors. Setting a value on one side but not the other compares settings instead of models. Ten trials per prompt capture the natural spread without touching the dial. Can I run this with the Claude and ChatGPT apps instead of the APIs? You can, but results are less clean. The apps add system prompts and memory, and OpenAI announced on November 10 that it would retire automatic Instant to Thinking switching for Plus and Pro users, so the model depends on your picker choice. If you go manual, use fresh chats for every run and save outputs in the same folder layout. What does a high opener share actually mean? It means a large fraction of runs begin with the same first three words. A share above 0.5 means half your pool opens identically. Compare the plain and seeded conditions to see whether a constraint card fixes it. Do these metrics replace human judgment? No. They catch lexical sameness only. A pool can score well on distinct n-grams and still retell the same plot in new words, which is why the blind rating step stays in the workflow.