![Claude vs ChatGPT Drift Test: 12 Steps, 90 Min [2026]](https://shattered.io/wp-content/uploads/2026/11/claude-vs-chatgpt-long-form-drift-test-12-steps-2026-gen.webp)
Claude vs ChatGPT for writing looks like a settled argument until you ask either assistant for 4,000 words in six pieces. Anthropic now sells a Claude 5.5 family of Opus 5.5, Sonnet 5.5 and Haiku 5.5, and OpenAI is rolling ChatGPT over to the GPT-6 generation. The useful question for 2026 is no longer who writes the prettier paragraph. It is who keeps the facts straight once a draft outgrows a single reply.
A small Python harness answers it. You feed both assistants the same six-section handbook brief, then count how many names, prices, dates and versions wander between section one and section six. A blind pairwise test tells you which prose readers prefer. The drift test tells you which assistant you can trust with a long document, and what each clean section costs.
Plan on about 90 minutes. The offline dry run needs no API keys and no money. A live run stays small because each condition is six short sections, but read the cost line the harness prints before you scale anything up.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
Why Long-Form Drift Decides Claude vs ChatGPT for Writing #
Short prompts hide the failure that hurts in real work. Ask for a 300-word product blurb and both assistants return clean copy with the right name. Ask for a 4,000-word handbook written one section at a time and small things start to slip. The product turns into “NorthWind Sync” in section three. A $49 plan becomes $59 in section five. A launch date shifts by four days. Nobody notices until a customer does.
The harness tracks four kinds of drift:
- Name drift: the product name written any way other than the canonical spelling.
- Fact drift: a price, date, version, storage size, retention window or uptime figure that contradicts the ledger.
- Repetition drift: four-word phrases and sentence openers recycled from earlier sections.
- Style drift: stock phrases such as game-changer and sentence lengths that flatten into a monotone as the draft grows.
Each is cheap to measure with plain Python, which matters. If you ask a third model to grade the first two, you add a new source of error and a new bill. Deterministic checks give the same answer every time you run them, so a changed score means the model changed, not the judge.
The test also splits the memory strategy from the model. Real writers use three patterns. They paste the whole draft back in (full history), they paste a compressed summary (summary), or they paste a fact sheet plus the last few words (ledger). A model that holds facts with a ledger but loses them with a summary is telling you about your workflow, not about its writing. A model that fails under all three is telling you about itself.
The harness complements, rather than replaces, other checks. A seeded-error editing test checks whether an assistant fixes what you point at, and a source-grounded audit checks claims against documents. Drift sits between them: nobody points at the error, and no document is there to catch it.
Popularity makes the question urgent. A Bloomberg-reported study, as summarized by Techmeme, put Claude’s share of UK consumer AI subscriptions at 29% against ChatGPT’s 52% as of July 2026. Both are widely paid for, so both are plausible choices for the same job, and your own measurements beat anyone’s market share.
The Lineup on November 6, 2026 #
A fair Claude vs ChatGPT run starts by pinning what you test, because both vendors changed their lineups within the last few weeks. On the Claude side, the current family is Opus 5.5, Sonnet 5.5 and Haiku 5.5. Anthropic describes Opus 5.5 as leading in agentic coding and knowledge work at 40% lower typical cost than Opus 5. It says Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. Anthropic’s own page calls Haiku 5.5 its fastest, most capable small model, built for high-volume work like summarization, subagents and browser use.
On the OpenAI side, GPT-6 is rolling out to eligible Pro, Plus, Business and Enterprise users, and GPT-6 Pro runs on GPT-6 Astra for the Pro $100, Pro $200, Business and Enterprise plans. GPT-6.1 Terra began rolling out in ChatGPT Work and Codex on October 26, according to The Agentic Weekly. The API lists it as gpt-6.1-terra, per vanemmerik.ai’s release roundup. OpenAI documents GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Sol and GPT-6 Luna for Work and Codex rather than ordinary chat, so a result from the chat window and a result from the API are not the same experiment.
The API rate card is what the cost column of the harness runs on. These are the per-million-token prices on the sheets dated November 5 and 6, 2026:
| Model | Input per 1M tokens | Output per 1M tokens | Note |
|---|---|---|---|
| Claude Sonnet 5.5 | $2.00 | $10.00 | Cache reads $0.10 per 1M |
| Claude Haiku 5.5 | $0.10 | $0.50 | Prompts up to 100,000 tokens |
| Claude Opus 5.5 | Read from Anthropic pricing | Read from Anthropic pricing | Not in our sources, so no number here |
| GPT-6.1 Sol | $2.00 | $10.00 | OpenAI rate card, Nov. 5 |
| GPT-6.1 Terra | $1.00 | $5.00 | API ID gpt-6.1-terra |
| GPT-6 Sol | $2.00 | $10.00 | OpenAI rate card, Nov. 5 |
| GPT-6 Astra | $10.00 | $50.00 | Powers GPT-6 Pro |
Sonnet 5.5 and GPT-6.1 Sol share a $2 and $10 price, so they make the fair default pair in config.json. Swap in Opus 5.5 against GPT-6 Astra for a flagship run, or Haiku 5.5 against Terra for the budget tier.
For plans, ChatGPT’s pricing page lists Free, Go, Plus at $20 a month, and three Pro tiers at $100, $200 and $500. Business and Enterprise round out the line. Claude’s plans are Free, Pro, Max, Ultra, Team and Enterprise. Chat plans matter only if you repeat the test by hand in the apps. The scripted run bills your API account, and the usage limits and plan prices of each subscription never enter the numbers.
One more date matters. OpenAI retired GPT-5 Instant and Thinking, GPT-4o, GPT-4.1 and o4-mini from ChatGPT on February 13, 2026, and GPT-5.1 on March 11, 2026. Any guide that tells you to test those is out of date.
Prerequisites and Versions #
You need a terminal, a text editor and two accounts if you want the live run. Here is the exact setup the code below was tested on:
- Python 3.11 or newer. The tests below ran on Python 3.12.3. The code uses only the standard library plus the two vendor SDKs.
- The
anthropicandopenaiPython packages, both installed at their latest release from PyPI. Pin whatever pip resolves withpip freeze, as Step 1 shows. - An Anthropic API key exported as
ANTHROPIC_API_KEYand anOpenAI API key exported asOPENAI_API_KEY. Both SDKs read these automatically. - Two exact model IDs exported as
CLAUDE_MODELandOPENAI_MODEL. Copy them from each vendor’s model list instead of guessing. The only ID stated here isgpt-6.1-terra. - About 20 MB of disk for the virtual environment and run files.
Skip the keys entirely for the first hour. The dry run uses an offline fake provider that injects errors on purpose, so you can prove the checks work before any real model touches them. If you want a wider field after this, the three-vendor API harness adds Gemini to the same pattern.
Steps 1 and 2: Project Skeleton and Config #
Step 1 creates the folder, the virtual environment and the dependency lock. Run it from any empty directory.
mkdir driftlab && cd driftlab
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
mkdir driftlab tests runs
touch driftlab/__init__.py
printf "anthropic\nopenai\n" > requirements.txt
pip install -r requirements.txt
pip freeze > requirements.lock
The lock file is not decoration. SDK releases change response objects, and a rerun in three months should use the same packages that produced your baseline.
Step 2 is config.json, the one place that names models, prices and conditions. Prices live here with an as_of date, because a comparison with undated prices is useless the day a rate card changes.
{
"label": "baseline",
"words_per_section": 600,
"max_output_tokens": 1800,
"modes": ["full", "summary", "ledger"],
"models": [
{
"key": "claude",
"provider": "anthropic",
"model_env": "CLAUDE_MODEL",
"label": "Claude Sonnet 5.5",
"price": {"input": 2.00, "output": 10.00, "as_of": "2026-11-06"}
},
{
"key": "chatgpt",
"provider": "openai",
"model_env": "OPENAI_MODEL",
"label": "GPT-6.1 Sol",
"price": {"input": 2.00, "output": 10.00, "as_of": "2026-11-06"}
}
]
}
Three design choices deserve a sentence each. The config stores the environment variable name, not the model ID, so the file is safe to commit and the same file runs a different model tomorrow. Sampling settings such as temperature are absent on purpose. Several current models restrict or ignore them, and a fair test leaves both vendors on defaults and records that fact. And max_output_tokens sits well above the 600-word target, because models that reason before answering spend part of that budget on thinking, which the troubleshooting section returns to.
Also log what the config cannot hold. For every run, write down the exact model ID, the plan or API tier, the date and the reasoning mode. That record is what makes a result reproducible, and it is the first thing to check when two people get different answers.
| Field to record | Example | Why it matters |
|---|---|---|
| Exact model ID | The string you exported in CLAUDE_MODEL | Vendors ship new snapshots under familiar names |
| Surface | API, ChatGPT chat, Work or Codex | Some models are documented only for Work and Codex |
| Plan or tier | Plus, Pro 100, Team, API pay-as-you-go | Plan limits change which models you can reach |
| Run date | 2026-11-06 | Models, prices and limits move within weeks |
| Reasoning mode | Default, or the effort setting you chose | Thinking time changes cost, speed and output length |
| Run label | nov06-a | Keeps repeat runs separate on disk |
Steps 3 and 4: Canon Ledger and Section Briefs #
The canon ledger is the ground truth the checker holds both assistants to. The product is fictional on purpose, so neither model can lean on training data. Step 3 writes canon.json:
{
"name": "Northwind Sync",
"name_pattern": "north[\\s-]?wind[\\s-]?sync(?:er|s|ing)?|nw[\\s-]?sync",
"slots": [
{"id": "team_price", "anchor": "team plan", "value": "\\$\\d+(?:\\.\\d+)?", "expected": "$49"},
{"id": "launch_date", "anchor": "launch|debut|first shipped", "value": "(?:January|February|March|April|May|June|July|August|September|October|November|December)\\s+\\d{1,2}", "expected": "March 14"},
{"id": "version", "anchor": "version", "value": "\\b\\d+\\.\\d+(?:\\.\\d+)?\\b", "expected": "3.2"},
{"id": "free_storage", "anchor": "free tier|free plan", "value": "\\d+\\s?GB", "expected": "15 GB"},
{"id": "retention", "anchor": "retention|retained|file history", "value": "\\d+\\s?days", "expected": "30 days"},
{"id": "uptime", "anchor": "uptime", "value": "\\d+\\.\\d+\\s?%", "expected": "99.95%"}
],
"ledger": [
"Product name: Northwind Sync (always written exactly like this)",
"Team plan price: $49 per seat per month",
"First shipped: March 14",
"Current version: 3.2",
"Free tier storage: 15 GB",
"File history retention: 30 days",
"Measured uptime: 99.95%"
]
}
Each slot is a tiny rule: find a sentence that contains the anchor phrase, pull every value of the right shape out of that sentence, and flag any value that differs from the expected one. The name pattern catches the variants that models actually produce, such as a capital W in the middle, an added “er” suffix, a space inside the name, or an abbreviation.
The ledger list is what the model sees. The slots are what the checker sees. Keeping them separate lets you change a fact in one place and update the other deliberately, which also prevents the model from receiving the regexes.
Step 4 is briefs.json, six section prompts. Notice that facts return on purpose: the version appears in sections one and five, and the launch date and Team plan price come back in section six. A fact stated once cannot drift. A fact the draft must restate several times can.
[
{"title": "Overview", "brief": "Introduce the product, say when it first shipped and which version is current."},
{"title": "Pricing", "brief": "Explain the Team plan price and what the free tier includes."},
{"title": "Storage and history", "brief": "Describe free tier storage and how long file history is retained."},
{"title": "Reliability", "brief": "Discuss measured uptime and how the team reports incidents."},
{"title": "Security", "brief": "Cover encryption and access controls, and name the current version again."},
{"title": "Migration and outlook", "brief": "Explain moving from other tools, restate the launch date and the Team plan price, and close with the next milestones."}
]
Swap this fixture for your own domain once the pipeline works. A legal brief, a technical manual or a product catalog each produce different drift. Keep six sections and restate each fact at least twice.
Steps 5 and 6: Providers and a Resumable Runner #
Step 5 is driftlab/providers.py. It wraps both SDKs behind one complete() method and adds the offline fake. The real adapters return the same four fields, so the rest of the harness never knows which vendor it is talking to.
import os, random, re, time
from dataclasses import dataclass
@dataclass
class Reply:
text: str
input_tokens: int
output_tokens: int
seconds: float
class AnthropicProvider:
def __init__(self, model):
import anthropic
self.client, self.model = anthropic.Anthropic(), model
def complete(self, system, prompt, max_tokens, meta=None):
start = time.time()
r = self.client.messages.create(
model=self.model, max_tokens=max_tokens, system=system,
messages=[{"role": "user", "content": prompt}])
text = "".join(b.text for b in r.content if b.type == "text")
return Reply(text, r.usage.input_tokens, r.usage.output_tokens, time.time() - start)
class OpenAIProvider:
def __init__(self, model):
import openai
self.client, self.model = openai.OpenAI(), model
def complete(self, system, prompt, max_tokens, meta=None):
start = time.time()
r = self.client.responses.create(
model=self.model, instructions=system, input=prompt,
max_output_tokens=max_tokens)
return Reply(r.output_text, r.usage.input_tokens, r.usage.output_tokens, time.time() - start)
Both adapters send one user message with no tools, no web search and no extra settings. The prompt is identical across vendors, which is the whole point. The next block finishes the file with the fake provider and the factory. The fake drifts on purpose: wrong values and name variants appear more often in later sections and in summary mode, so the checks have real errors to find.
WRONG = {
"$49": ["$59", "$45"], "March 14": ["March 18", "April 2"], "3.2": ["3.1", "3.0"],
"15 GB": ["10 GB", "20 GB"], "30 days": ["14 days", "60 days"], "99.95%": ["99.9%", "99.5%"],
}
VARIANTS = ["NorthWind Sync", "Northwind Syncer", "North Wind Sync", "NW Sync"]
OPENERS = ["In practice,", "For most teams,", "At scale,", "Put simply,"]
FILLER = [
"Admins review the dashboard each Monday before planning the week.",
"Support answers most tickets within a few hours.",
"Large folders sync in the background without blocking the desktop client.",
"Teams that mix Windows, macOS and Linux laptops see the same file tree everywhere.",
"The changelog lists every fix.",
]
class FakeProvider:
"""Offline stand-in that injects drift on purpose so the checks have something to find."""
BASE = {"full": 0.03, "summary": 0.12, "ledger": 0.02}
def __init__(self, key):
self.key = key
def complete(self, system, prompt, max_tokens, meta=None):
meta = meta or {}
if meta.get("kind") == "summary":
return Reply("Northwind Sync, Team plan $49, shipped March 14, version 3.2.", 900, 40, 0.4)
idx, mode = meta["index"], meta["mode"]
rng = random.Random(f"{self.key}-{mode}-{idx}")
extra = 0.015 if self.key == "chatgpt" else 0.0
p = self.BASE[mode] + extra + 0.03 * idx
name = lambda: rng.choice(VARIANTS) if rng.random() < p else "Northwind Sync"
val = lambda v: rng.choice(WRONG[v]) if rng.random() < p else v
opener = lambda: rng.choice(OPENERS[:2]) if mode == "summary" and idx > 2 else rng.choice(OPENERS)
sents = [
f"{opener()} {name()} first shipped on {val('March 14')} and now runs version {val('3.2')}.",
f"The Team plan costs {val('$49')} per seat each month.",
f"{rng.choice(FILLER)}",
f"The free tier gives every account {val('15 GB')} of space.",
f"File history is retained for {val('30 days')} on every paid plan.",
f"{opener()} {name()} reports measured uptime of {val('99.95%')} over the past year.",
f"{rng.choice(FILLER)}",
"Teams that outgrow the free tier usually upgrade within a quarter.",
]
text = " ".join(sents)
return Reply(text, 700 + 450 * idx if mode == "full" else 700, len(text.split()) * 2, 1.0 + 0.1 * idx)
def make_provider(spec, dry):
if dry:
return FakeProvider(spec["key"])
model = os.environ.get(spec["model_env"])
if not model:
raise SystemExit(f"Set {spec['model_env']} to the exact model ID for {spec['label']}")
return AnthropicProvider(model) if spec["provider"] == "anthropic" else OpenAIProvider(model)
The fake gives the ChatGPT key a slightly higher base error rate than the Claude key. That is a fixture setting and carries no claim about either real assistant. Step 6 is driftlab/runner.py, where the three memory conditions live. Section one always receives the full ledger. After that, each mode feeds the model something different.
import json
from pathlib import Path
SYSTEM = (
"You are drafting one section of a long internal handbook for a software product. "
"Write about {words} words of plain prose in three paragraphs. No headings, no bullet lists. "
"Write the product name exactly as it appears in the ledger or earlier text. "
"State only facts that appear there. Do not invent numbers."
)
def build_prompt(mode, canon, brief, idx, texts, summary):
ask = f"Section {idx + 1}: {brief['title']}. {brief['brief']}"
if idx == 0:
ledger = "\n".join(canon["ledger"])
return f"Fact ledger:\n{ledger}\n\n{ask}"
if mode == "full":
history = "\n\n".join(texts)
return f"Earlier sections:\n{history}\n\n{ask}"
if mode == "summary":
return f"Running summary of earlier sections:\n{summary}\n\n{ask}"
ledger = "\n".join(canon["ledger"])
tail = " ".join(texts[-1].split()[-40:])
return f"Fact ledger:\n{ledger}\n\nEnd of previous section: ...{tail}\n\n{ask}"
def run_one(spec, mode, provider, cfg, canon, briefs, out_dir):
path = Path(out_dir) / f"{spec['key']}-{mode}.json"
if path.exists():
return json.loads(path.read_text())
system = SYSTEM.format(words=cfg["words_per_section"])
texts, summary, records = [], "", []
for idx, brief in enumerate(briefs):
prompt = build_prompt(mode, canon, brief, idx, texts, summary)
meta = {"index": idx, "mode": mode, "kind": "section"}
reply = provider.complete(system, prompt, cfg["max_output_tokens"], meta)
texts.append(reply.text)
records.append({"index": idx, "title": brief["title"], "kind": "section", "text": reply.text,
"in": reply.input_tokens, "out": reply.output_tokens, "sec": reply.seconds})
if mode == "summary" and idx < len(briefs) - 1:
ask = ("Summarize everything stated so far in under 120 words. Keep every name, "
"price, date, version and number exactly as written.\n\n" + "\n\n".join(texts))
s = provider.complete("You compress drafts without changing facts.", ask, 400,
{"index": idx, "mode": mode, "kind": "summary"})
summary = s.text
records.append({"index": idx, "title": "summary", "kind": "summary", "text": s.text,
"in": s.input_tokens, "out": s.output_tokens, "sec": s.seconds})
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(records, indent=1))
return records
Two details protect your budget. The runner writes one file per model and mode only after all six sections succeed, and it returns the saved file if one exists. A crashed run therefore never leaves half a result, and a rerun never pays twice for a finished condition. Summary mode also bills the extra summarizer calls, which the cost column counts, since a cheaper summary strategy that drifts costs more in the end than it saves.
If you want to see how a staged workflow changes the outcome, the multi-stage writing pipeline tutorial chains drafting, editing and fact checks across different models.
Steps 7 and 8: Continuity Checks and Drift Metrics #
Step 7 starts driftlab/checks.py with the two checks that read the canon. Both are pure functions of text, so you can test them without a network.
import re
from statistics import mean, pstdev
SENT = re.compile(r"(?<=[.!?])\s+")
CLICHES = ["game-changer", "unlock the power", "dive into", "cutting-edge", "revolutionize", "at its core"]
def sentences(text):
return [s for s in SENT.split(text.strip()) if s]
def words(text):
return re.findall(r"[A-Za-z0-9$%.'-]+", text)
def name_violations(text, canon):
pattern = re.compile(canon["name_pattern"], re.I)
return [m.group(0) for m in pattern.finditer(text) if m.group(0) != canon["name"]]
def fact_violations(text, canon):
bad = []
for slot in canon["slots"]:
anchor = re.compile(slot["anchor"], re.I)
value = re.compile(slot["value"], re.I)
want = slot["expected"].replace(" ", "").lower()
for sent in sentences(text):
if anchor.search(sent):
for m in value.finditer(sent):
if m.group(0).replace(" ", "").lower() != want:
bad.append((slot["id"], m.group(0)))
return bad
The name check compares the matched text to the canonical string case-sensitively, so “northwind sync” in lowercase counts as drift. That is deliberate. A brand name that changes case is a brand error. The fact check compares values after stripping spaces, so “15GB” and “15 GB” both pass.
Step 8 appends the repetition and style metrics plus the combined section score:
def ngrams(tokens, n=4):
return {tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)}
def repeat_rate(text, earlier):
now = ngrams([w.lower() for w in words(text)])
before = set()
for e in earlier:
before |= ngrams([w.lower() for w in words(e)])
return len(now & before) / len(now) if now else 0.0
def opener_repeat(text, earlier):
key = lambda s: " ".join(s.lower().split()[:2])
seen = {key(s) for e in earlier for s in sentences(e)}
sents = sentences(text)
return sum(key(s) in seen for s in sents) / len(sents) if sents else 0.0
def score_section(text, canon, earlier):
n = len(words(text)) or 1
names = name_violations(text, canon)
facts = fact_violations(text, canon)
cliches = sum(text.lower().count(c) for c in CLICHES)
lengths = [len(words(s)) for s in sentences(text)] or [0]
weighted = 2 * len(names) + 3 * len(facts) + cliches
return {
"words": n, "names": len(names), "facts": len(facts), "cliches": cliches,
"repeat": repeat_rate(text, earlier), "openers": opener_repeat(text, earlier),
"sent_mean": mean(lengths), "sent_sd": pstdev(lengths),
"drift_per_1k": weighted * 1000 / n, "clean": weighted == 0,
}
The weighting is a judgment call, and you should tune it. A wrong price costs a reader more than a stray cliche, so facts weigh 3, names weigh 2 and cliches weigh 1. Dividing by length gives drift per 1,000 words, which stops a wordy model from looking better simply because it spreads the same errors over more text. A section is “clean” only when the weighted total is zero.
Repetition is scored against everything written earlier in the same run, never against the other vendor. Sentence-length standard deviation shows flattening, since a draft that slides into uniform sentences loses its rhythm. For a deeper style comparison against a named author, the voice-match stylometry test goes further than these coarse counters.
Steps 9 and 10: Report and Command Line #
Step 9 is driftlab/report.py. It converts raw records into one row per model and mode, computes cost from the dated prices, and adds the drift curve, which is the table you will read first.
import json
from pathlib import Path
from statistics import mean
from .checks import score_section
def cost_of(records, price):
tin = sum(r["in"] for r in records)
tout = sum(r["out"] for r in records)
return (tin * price["input"] + tout * price["output"]) / 1_000_000
def summarize(spec, mode, records, canon):
sections = [r for r in records if r["kind"] == "section"]
scored, earlier = [], []
for r in sections:
scored.append(score_section(r["text"], canon, earlier))
earlier.append(r["text"])
half = len(scored) // 2
first = mean(s["drift_per_1k"] for s in scored[:half])
last = mean(s["drift_per_1k"] for s in scored[half:])
cost = cost_of(records, spec["price"])
clean = sum(s["clean"] for s in scored)
later = scored[1:]
return {
"model": spec["label"], "mode": mode, "sections": len(scored), "clean": clean,
"names": sum(s["names"] for s in scored), "facts": sum(s["facts"] for s in scored),
"drift_per_1k": mean(s["drift_per_1k"] for s in scored), "slope": last - first,
"repeat": mean(s["repeat"] for s in later), "openers": mean(s["openers"] for s in later),
"sent_sd": mean(s["sent_sd"] for s in scored), "cost": cost,
"cost_per_clean": cost / clean if clean else None,
"seconds": sum(r["sec"] for r in records),
}
def render(rows, label):
head = ("| Model | Mode | Clean | Names | Facts | Drift/1k | Slope | Repeat | Cost | $/clean |\n"
"|---|---|---|---|---|---|---|---|---|---|\n")
lines = []
for r in rows:
pc = f"{r['cost_per_clean']:.4f}" if r["cost_per_clean"] else "n/a"
lines.append(f"| {r['model']} | {r['mode']} | {r['clean']}/{r['sections']} | {r['names']} | "
f"{r['facts']} | {r['drift_per_1k']:.1f} | {r['slope']:+.1f} | "
f"{r['repeat']:.2f} | ${r['cost']:.4f} | {pc} |")
return f"# Drift report: {label}\n\n" + head + "\n".join(lines) + "\n"
def curve(spec, mode, records, canon):
earlier, cells = [], []
for r in records:
if r["kind"] != "section":
continue
cells.append(f"{score_section(r['text'], canon, earlier)['drift_per_1k']:.0f}")
earlier.append(r["text"])
return f"| {spec['label']} | {mode} | " + " | ".join(cells) + " |"
def build(cfg, canon, out_dir):
rows, curves = [], []
for spec in cfg["models"]:
for mode in cfg["modes"]:
path = Path(out_dir) / f"{spec['key']}-{mode}.json"
if path.exists():
records = json.loads(path.read_text())
rows.append(summarize(spec, mode, records, canon))
curves.append(curve(spec, mode, records, canon))
rows.sort(key=lambda r: (r["drift_per_1k"], r["cost"]))
n = curves[0].count("|") - 3 if curves else 0
head = "| Model | Mode | " + " | ".join(f"S{i + 1}" for i in range(n)) + " |\n"
head += "|---|---|" + "---|" * n + "\n"
md = render(rows, cfg["label"]) + "\n## Drift per 1,000 words by section\n\n" + head + "\n".join(curves) + "\n"
(Path(out_dir) / "report.md").write_text(md)
(Path(out_dir) / "report.json").write_text(json.dumps(rows, indent=1))
return md
The metric that earns its place is dollars per clean section. A cheap model that ships two clean sections out of six costs more per usable section than a pricier one that ships five. The slope column subtracts the first-half average from the second-half average, so a positive number means the draft got worse as it grew.
Step 10 is driftlab/cli.py, the entry point:
import argparse, json
from pathlib import Path
from .providers import make_provider
from .report import build
from .runner import run_one
ROOT = Path(__file__).resolve().parent.parent
def load(name):
return json.loads((ROOT / name).read_text())
def main():
ap = argparse.ArgumentParser()
ap.add_argument("command", choices=["run", "report"])
ap.add_argument("--dry", action="store_true", help="use the offline fake provider")
ap.add_argument("--only", help="run a single model key, for example claude")
ap.add_argument("--label", help="override the run label")
args = ap.parse_args()
cfg, canon, briefs = load("config.json"), load("canon.json"), load("briefs.json")
if args.label:
cfg["label"] = args.label
out_dir = ROOT / "runs" / (cfg["label"] + ("-dry" if args.dry else ""))
if args.command == "run":
for spec in cfg["models"]:
if args.only and spec["key"] != args.only:
continue
provider = make_provider(spec, args.dry)
for mode in cfg["modes"]:
print(f"running {spec['key']} / {mode}")
run_one(spec, mode, provider, cfg, canon, briefs, out_dir)
print(build(cfg, canon, out_dir))
if __name__ == "__main__":
main()
Dry runs write to a folder ending in -dry, so fake results can never mix with real ones. The --label flag gives every live run its own folder, which you need for repeat runs.
The command line stays thin on purpose. It loads three JSON files, builds one provider per model, loops over the modes and prints the report. Everything that could go wrong sits in a file you can test alone. If you later add a third vendor, you add one entry to config.json and one adapter class, and the runner, the checks and the report need no edits. That separation also makes the harness easy to hand to a colleague, because they can run the dry mode first, see the report shape and trust the machinery before they spend a cent.
Steps 11 and 12: Tests, Dry Run and Live Run #
Step 11 locks the checks with unit tests before a model is involved. If a regex breaks later, these fail first. Save this as tests/test_checks.py:
import json, unittest
from pathlib import Path
from driftlab.checks import fact_violations, name_violations, repeat_rate, score_section
CANON = json.loads((Path(__file__).resolve().parent.parent / "canon.json").read_text())
class ChecksTest(unittest.TestCase):
def test_exact_name_passes(self):
self.assertEqual(name_violations("Northwind Sync is fast.", CANON), [])
def test_name_variants_fail(self):
bad = name_violations("NorthWind Sync and Northwind Syncer differ.", CANON)
self.assertEqual(len(bad), 2)
def test_wrong_price_fails(self):
bad = fact_violations("The Team plan costs $59 per seat.", CANON)
self.assertEqual(bad, [("team_price", "$59")])
def test_right_facts_pass(self):
text = "The Team plan costs $49. The free tier has 15 GB. Uptime is 99.95%."
self.assertEqual(fact_violations(text, CANON), [])
def test_repeat_rate_detects_copy(self):
a = "the quick brown fox jumps over the lazy dog today"
self.assertGreater(repeat_rate(a, [a]), 0.9)
def test_clean_section_flag(self):
s = score_section("Northwind Sync shipped version 3.2 on March 14.", CANON, [])
self.assertTrue(s["clean"])
if __name__ == "__main__":
unittest.main()
Run the tests, then the dry run:
python -m unittest discover -s tests -v
python -m driftlab.cli run --dry
All six tests pass in a fraction of a second. The dry run prints the report below. These numbers come from the offline fake, which injects errors on a schedule, so read them for shape and never as a verdict on either vendor:
| Model | Mode | Clean | Names | Facts | Drift/1k | Slope | Repeat | Cost | $/clean |
|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | full | 4/6 | 0 | 2 | 11.0 | +22.0 | 0.83 | $0.0326 | 0.0081 |
| GPT-6.1 Sol | ledger | 3/6 | 0 | 3 | 17.1 | +11.2 | 0.84 | $0.0190 | 0.0063 |
| Claude Sonnet 5.5 | ledger | 3/6 | 2 | 3 | 25.2 | +26.3 | 0.80 | $0.0186 | 0.0062 |
| GPT-6.1 Sol | full | 1/6 | 3 | 5 | 42.4 | +18.9 | 0.80 | $0.0320 | 0.0320 |
| GPT-6.1 Sol | summary | 1/6 | 3 | 8 | 55.7 | +52.2 | 0.78 | $0.0301 | 0.0301 |
| Claude Sonnet 5.5 | summary | 0/6 | 1 | 14 | 84.0 | +7.9 | 0.73 | $0.0299 | n/a |
## Drift per 1,000 words by section
| Model | Mode | S1 | S2 | S3 | S4 | S5 | S6 |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | full | 0 | 0 | 0 | 0 | 34 | 32 |
| Claude Sonnet 5.5 | summary | 94 | 72 | 74 | 22 | 103 | 138 |
| Claude Sonnet 5.5 | ledger | 0 | 0 | 36 | 0 | 58 | 57 |
| GPT-6.1 Sol | full | 0 | 63 | 36 | 37 | 60 | 60 |
| GPT-6.1 Sol | summary | 55 | 34 | 0 | 57 | 100 | 88 |
| GPT-6.1 Sol | ledger | 0 | 34 | 0 | 0 | 35 | 33 |
The shape is what you should expect from a real run: drift climbs toward the last sections, and summary mode is the leakiest memory strategy. The dollar figures are fake-token costs and carry no information about real pricing behavior.
Step 12 is the live run. Export the keys and exact IDs, run each vendor separately so one failure cannot block the other, and label the run:
export ANTHROPIC_API_KEY=... # your key
export OPENAI_API_KEY=... # your key
export CLAUDE_MODEL=... # exact ID from Anthropic's model list
export OPENAI_MODEL=... # exact ID from OpenAI's model list
python -m driftlab.cli run --only claude --label nov06-a
python -m driftlab.cli run --only chatgpt --label nov06-a
python -m driftlab.cli report --label nov06-a
A live run makes 18 section calls per vendor plus 5 summarizer calls in summary mode. Run the same label scheme three times (nov06-a, nov06-b, nov06-c) and compare the median. One run measures luck, and three runs begin to measure the model.
Before you read any numbers, open one saved run file and read a full section with your own eyes. Automated scores catch what you wrote rules for and nothing else. If a model answers every brief with a refusal, a heading list or a section that ignores the ledger, the scores will look odd in ways that only a human glance explains. Spend five minutes on this check. It has saved more runs than any regex.
Reading the Report: A Decision Rule #
Read the report in this order. First look at the drift curve, because it shows where the damage starts. Then read the Clean column, then dollars per clean section, and last the repetition and slope columns.
| Pattern in your report | What it means | Action |
|---|---|---|
| Drift rises in every mode for one model | The model loses facts as drafts lengthen | Keep drafts short or rotate in a ledger every section |
| Drift is low in ledger mode and high in summary mode | Your memory method is the weak link | Stop summarizing, or force the summarizer to copy facts verbatim |
| Name drift only, no fact drift | Casing and suffix habits, not memory | Add a spelling line to the system prompt and a post-edit replace |
| Repeat above 0.30 after section three | The model recycles its own phrasing | Vary the briefs or ask for a different opening per section |
| Low drift but a high price per clean section | Accurate but expensive | Try a cheaper model with a ledger in every call |
| Both vendors clean in ledger mode | Quality is a tie on this fixture | Decide on price, tone and workflow, then test your own domain |
Two examples show how to apply the table. Suppose Claude holds every fact in ledger mode but shows name variants in summary mode. The fix lives in your workflow: make the summarizer copy names verbatim, and the gap should close. Now suppose ChatGPT shows rising fact drift in every mode, including ledger. That points at the model on this fixture, and the remedy is a shorter section length or a ledger check after every section.
Treat the thresholds as starting points. The 0.30 repeat line is a starting guess, not a published benchmark, so calibrate it against drafts you consider acceptable. The honest result of a good run is often a split decision, where one assistant holds names better and the other holds numbers better. That split is useful, because you can route work by task.
Do not stretch a Claude vs ChatGPT verdict past what you measured. A fictional file-sync handbook says little about a medical protocol or a legal contract. When you change domains, rebuild the canon and rerun all three modes.
Common Pitfalls That Skew a Drift Test #
- Comparing different surfaces. OpenAI documents several newer models for ChatGPT Work and Codex rather than ordinary chat, and plan tiers change which models you reach. An API run and a chat-window run are different experiments. Record the surface on every result.
- Letting sampling settings differ. If one adapter sets a temperature and the other cannot, you are measuring settings. Leave both vendors on defaults, as this harness does.
- Trusting one run. Language models vary between calls. A single run can crown the wrong winner, so repeat under three labels and compare medians.
- Multi-fact sentences. The fact check reads one sentence at a time. If a model writes “The Team plan costs $49, while the Business plan costs $99,” the $99 trips the price slot. Keep slot anchors specific, and read flagged items before you believe a score.
- Prompts that favor one vendor. Any tweak you make for one assistant, such as extra bullet rules, must go to both. Change the shared prompt or change nothing.
- Grading with a model. A third model scoring the first two adds cost, adds its own bias, and returns different scores on a rerun. Keep model judges for taste and use code for facts.
- Hiding untrusted text in the draft. If you later feed real customer text into these prompts, instructions inside it can steer the model. Aprompt injection test lab shows how to probe for that before it costs you.
- Forgetting to date the prices. A cost column built on last month’s rate card misleads. The
as_offield exists so you notice when it is stale.
Troubleshooting: 11 Errors and Fixes #
1. “Set CLAUDE_MODEL to the exact model ID” and the run stops. The environment variable is empty in the current shell. Export it again and confirm with echo $CLAUDE_MODEL. A new terminal tab does not inherit exports from the old one.
2. A 404 or “model not found” error. The ID is wrong or your account cannot reach that model. Copy the ID from the vendor’s model list, not from a blog post. The one ID stated here is gpt-6.1-terra, and even that needs API access on your account.
3. A 401 authentication error. The key is missing, revoked or pasted with a trailing space. Re-export it and make sure the right key goes to the right vendor.
4. A 429 rate-limit error mid-run. You exceeded a requests or tokens limit. The runner saves nothing for a half-finished condition, so rerun the same command later. It skips finished conditions and repeats only the failed one.
5. ModuleNotFoundError: No module named 'driftlab'. You ran a script directly. Run python -m driftlab.cli from the project root, where the driftlab folder sits.
6. Empty text from the OpenAI adapter. Reasoning models can spend the whole output budget on thinking and return no visible text. Raise max_output_tokens in config.json to 4000 or more, delete the empty run file, and rerun.
7. Sections cut off mid-sentence. The output hit its token ceiling. Raise the limit the same way. A truncated section also loses its later facts, which makes drift look better than it is.
8. A parameter such as temperature is rejected. Some current models restrict sampling settings. Remove the setting rather than working around it, since the harness uses defaults anyway.
9. json.decoder.JSONDecodeError on a run file. You interrupted a write. Delete the broken file in runs/<label>/ and rerun that condition.
10. Cost shows $0.0000 on a live run. The adapter returned zero usage, usually because the response object changed. Print r.usage once, fix the field names in the adapter, and rerun. The pip freeze lock helps you spot an SDK upgrade as the cause.
11. Facts flagged that look correct. Curly quotes, a non-breaking space or a form like “March 14th” can slip past the expected strings. Print the flagged tuple, then widen the slot regex or add a normalization step.
Advanced Tips for Sharper Results #
Seed decoys to test detection, not just generation
Add a deliberate contradiction to the earlier-sections text before section four, such as a Team plan price of $59 in section two. Then ask the assistant to flag any inconsistency before it writes. A model that spots the planted error does a different job from one that merely avoids making errors. It is a second experiment on the same harness, and it needs one extra field in the runner.
Cache the ledger and cheapen the summarizer
The ledger repeats in every ledger-mode call, which makes it a good candidate for prompt caching. Anthropic lists Sonnet 5.5 cache reads at $0.10 per million tokens after cutting them by 50% on October 7, 2026, so a repeated ledger costs little on that side. Check OpenAI’s current rate card for its own caching terms before assuming parity. For the summarizer, try the small tier. Haiku 5.5 lists at $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens, and Terra lists at $1 and $5. A cheap summarizer that preserves facts beats an expensive one that does not.
Gate drafts in CI
Treat the checks as a linter for any generated document. Keep canon.json in your repository, run fact_violations and name_violations in a CI job against generated copy, and fail the build when either returns anything. The same two functions that grade models also protect your docs from them.
Scale the fixture, not the model count
Resist adding vendors before you add domains. Three domains on two models say more than one domain on six. When you do widen, the AI model comparison for Terra against Opus 5.5 gives a sense of the flagship tier, and the wider AI and machine learning coverage tracks new releases as they land.
The Complete Working Project #
The whole project is roughly 360 lines of Python plus four small JSON files. Here is the final layout:
driftlab/
config.json models, prices with as_of dates, modes
canon.json name pattern, fact slots, ledger text
briefs.json six section prompts
requirements.txt anthropic, openai
requirements.lock pip freeze output
driftlab/
__init__.py
providers.py Anthropic, OpenAI and offline fake adapters
runner.py three memory modes, resumable per condition
checks.py name, fact, repetition and style metrics
report.py cost, summary table, drift curve
cli.py run and report commands
tests/
test_checks.py six unit tests
runs/
baseline-dry/ fake-provider records and report.md
nov06-a/ your first live run
Run it in this order: install, unit tests, dry run, one live label per vendor, report, then repeat under two more labels. The order matters because each stage proves something the next one relies on. Tests prove the checks work, the dry run proves the plumbing works, and only then do live calls prove anything about a model. Commit config.json, canon.json, briefs.json, the code and the report files. Never commit API keys or the .venv folder.
When the harness fits your needs, extend it in the direction your work demands. Writers of long reports add more sections. Marketing teams add a tone check. Engineering teams tied to code can compare how the same models behave in Claude vs ChatGPT for coding tasks, where a wrong constant breaks a build rather than a paragraph.
FAQ #
In Claude vs ChatGPT for writing, which one wins?
Neither wins everywhere, and a fair answer depends on the task. Both vendors shipped new generations within weeks of each other, so older reviews describe retired models. Run this harness on your own domain and decide from drift, cost per clean section and your own taste.
What does the drift test measure that a normal comparison misses?
It measures consistency across a long document. A standard comparison grades one reply on style and polish. The drift test checks whether names, prices, dates and versions survive across six sections, and whether phrasing repeats.
Do I need API keys to try this?
No. The offline dry run uses a fake provider and costs nothing. You need keys from Anthropic and OpenAI only for the live run, plus the exact model ID for each.
Which models should I put in config.json?
Use the pair that matches your real workload. Sonnet 5.5 and GPT-6.1 Sol share a $2 and $10 per-million-token price, which makes them a fair mid-tier pair. For a flagship run, test Opus 5.5 against GPT-6 Astra, and expect Astra at $10 input and $50 output per million tokens to cost far more.
Why does the harness avoid using a model as the judge?
A judge model adds cost and its own errors, and it can return different scores for the same text. Regex checks give identical results on every run. Use human review or a model judge for tone, and use code for facts.
How many runs do I need before I trust the result?
Three runs under separate labels is the practical minimum. Compare the median drift per 1,000 words and the count of clean sections. If the vendors swap places between runs, call it a tie on that fixture.
Can I test the ChatGPT and Claude apps instead of the API?
You can, by pasting the same prompts into each app and saving the replies into the run files by hand. Record the plan and the model each app reports, because OpenAI documents some models for Work and Codex rather than ordinary chat. App behavior also changes with plan limits, so label those runs clearly.
Will the same harness work for other languages or topics?
Yes. Replace the ledger, the slots and the briefs, and rebuild the name pattern for your product. Rerun all three memory modes, since a new domain can shift where drift appears.