Claude vs ChatGPT Drift Test: 12 Steps, 90 Min [2026] A Python-based drift test comparing Anthropic's Claude 5.5 family (Opus 5.5, Sonnet 5.5, Haiku 5.5) against OpenAI's GPT-6-generation ChatGPT measures how many names, prices, dates and versions change across a six-section, 4,000-word handbook draft, tracking name drift, fact drift, repetition drift and style drift with deterministic checks rather than a third model as judge. The roughly 90-minute harness, published by Shattered, runs offline without API keys and separates memory strategy — full history, summary, or a fact ledger — from the model itself. The test arrives as a Bloomberg-reported study summarized by Techmeme put Claude's share of UK consumer AI subscriptions at 29% versus ChatGPT's 52% as of July 2026. Claude vs ChatGPT for writing looks like a settled argument until you ask either assistant for 4,000 words in six pieces. Anthropic now sells a Claude 5.5 family https://support.claude.com/en/articles/12138966-release-notes?change=new-security-and-admin-controls-for-enterprise-plans of Opus 5.5, Sonnet 5.5 and Haiku 5.5, and OpenAI is rolling ChatGPT over to the GPT-6 generation. The useful question for 2026 is no longer who writes the prettier paragraph. It is who keeps the facts straight once a draft outgrows a single reply. A small Python harness answers it. You feed both assistants the same six-section handbook brief, then count how many names, prices, dates and versions wander between section one and section six. A blind pairwise test https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ tells you which prose readers prefer. The drift test tells you which assistant you can trust with a long document, and what each clean section costs. Plan on about 90 minutes. The offline dry run needs no API keys and no money. A live run stays small because each condition is six short sections, but read the cost line the harness prints before you scale anything up. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Long-Form Drift Decides Claude vs ChatGPT for Writing Short prompts hide the failure that hurts in real work. Ask for a 300-word product blurb and both assistants return clean copy with the right name. Ask for a 4,000-word handbook written one section at a time and small things start to slip. The product turns into “NorthWind Sync” in section three. A $49 plan becomes $59 in section five. A launch date shifts by four days. Nobody notices until a customer does. The harness tracks four kinds of drift: - Name drift: the product name written any way other than the canonical spelling. - Fact drift: a price, date, version, storage size, retention window or uptime figure that contradicts the ledger. - Repetition drift: four-word phrases and sentence openers recycled from earlier sections. - Style drift: stock phrases such as game-changer and sentence lengths that flatten into a monotone as the draft grows. Each is cheap to measure with plain Python, which matters. If you ask a third model to grade the first two, you add a new source of error and a new bill. Deterministic checks give the same answer every time you run them, so a changed score means the model changed, not the judge. The test also splits the memory strategy from the model. Real writers use three patterns. They paste the whole draft back in full history , they paste a compressed summary summary , or they paste a fact sheet plus the last few words ledger . A model that holds facts with a ledger but loses them with a summary is telling you about your workflow, not about its writing. A model that fails under all three is telling you about itself. The harness complements, rather than replaces, other checks. A seeded-error editing test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ checks whether an assistant fixes what you point at, and a source-grounded audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ checks claims against documents. Drift sits between them: nobody points at the error, and no document is there to catch it. Popularity makes the question urgent. A Bloomberg-reported study, as summarized by Techmeme https://www.techmeme.com/261105/p18 , put Claude’s share of UK consumer AI subscriptions at 29% against ChatGPT’s 52% as of July 2026. Both are widely paid for, so both are plausible choices for the same job, and your own measurements beat anyone’s market share. The Lineup on November 6, 2026 A fair Claude vs ChatGPT run starts by pinning what you test, because both vendors changed their lineups within the last few weeks. On the Claude side, the current family is Opus 5.5, Sonnet 5.5 and Haiku 5.5. Anthropic describes Opus 5.5 as leading in agentic coding and knowledge work at 40% lower typical cost than Opus 5. It says Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. Anthropic’s own page calls Haiku 5.5 its fastest, most capable small model, built for high-volume work like summarization, subagents and browser use. On the OpenAI side, GPT-6 is rolling out to eligible Pro, Plus, Business and Enterprise users, and GPT-6 Pro runs on GPT-6 Astra for the Pro $100, Pro $200, Business and Enterprise plans. GPT-6.1 Terra began rolling out in ChatGPT Work and Codex on October 26, according to The Agentic Weekly https://theagenticweekly.com/news . The API lists it as gpt-6.1-terra , per vanemmerik.ai’s release roundup https://vanemmerik.ai/all/ . OpenAI documents GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Sol and GPT-6 Luna for Work and Codex rather than ordinary chat, so a result from the chat window and a result from the API are not the same experiment. The API rate card is what the cost column of the harness runs on. These are the per-million-token prices on the sheets dated November 5 and 6, 2026: | Model | Input per 1M tokens | Output per 1M tokens | Note | |---|---|---|---| | Claude Sonnet 5.5 | $2.00 | $10.00 | Cache reads $0.10 per 1M | | Claude Haiku 5.5 | $0.10 | $0.50 | Prompts up to 100,000 tokens | | Claude Opus 5.5 | Read from Anthropic pricing | Read from Anthropic pricing | Not in our sources, so no number here | | GPT-6.1 Sol | $2.00 | $10.00 | OpenAI rate card, Nov. 5 | | GPT-6.1 Terra | $1.00 | $5.00 | API ID gpt-6.1-terra | | GPT-6 Sol | $2.00 | $10.00 | OpenAI rate card, Nov. 5 | | GPT-6 Astra | $10.00 | $50.00 | Powers GPT-6 Pro | Sonnet 5.5 and GPT-6.1 Sol share a $2 and $10 price, so they make the fair default pair in config.json . Swap in Opus 5.5 against GPT-6 Astra for a flagship run, or Haiku 5.5 against Terra for the budget tier. For plans, ChatGPT’s pricing page https://chatgpt.com/pricing/ lists Free, Go, Plus at $20 a month, and three Pro tiers at $100, $200 and $500. Business and Enterprise round out the line. Claude’s plans are Free, Pro, Max, Ultra, Team and Enterprise. Chat plans matter only if you repeat the test by hand in the apps. The scripted run bills your API account, and the usage limits and plan prices https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ of each subscription never enter the numbers. One more date matters. OpenAI retired GPT-5 Instant and Thinking, GPT-4o, GPT-4.1 and o4-mini from ChatGPT on February 13, 2026, and GPT-5.1 on March 11, 2026. Any guide that tells you to test those is out of date. Prerequisites and Versions You need a terminal, a text editor and two accounts if you want the live run. Here is the exact setup the code below was tested on: - Python 3.11 or newer. The tests below ran on Python 3.12.3. The code uses only the standard library plus the two vendor SDKs. - The anthropic and openai Python packages, both installed at their latest release from PyPI. Pin whatever pip resolves with pip freeze , as Step 1 shows. - An Anthropic API key exported as ANTHROPIC API KEY and an OpenAI API key exported as OPENAI API KEY . Both SDKs read these automatically. - Two exact model IDs exported as CLAUDE MODEL and OPENAI MODEL . Copy them from each vendor’s model list instead of guessing. The only ID stated here is gpt-6.1-terra . - About 20 MB of disk for the virtual environment and run files. Skip the keys entirely for the first hour. The dry run uses an offline fake provider that injects errors on purpose, so you can prove the checks work before any real model touches them. If you want a wider field after this, the three-vendor API harness https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/ adds Gemini to the same pattern. Steps 1 and 2: Project Skeleton and Config Step 1 creates the folder, the virtual environment and the dependency lock. Run it from any empty directory. mkdir driftlab && cd driftlab python3 -m venv .venv source .venv/bin/activate Windows: .venv\Scripts\activate mkdir driftlab tests runs touch driftlab/ init .py printf "anthropic\nopenai\n" requirements.txt pip install -r requirements.txt pip freeze requirements.lock The lock file is not decoration. SDK releases change response objects, and a rerun in three months should use the same packages that produced your baseline. Step 2 is config.json , the one place that names models, prices and conditions. Prices live here with an as of date, because a comparison with undated prices is useless the day a rate card changes. { "label": "baseline", "words per section": 600, "max output tokens": 1800, "modes": "full", "summary", "ledger" , "models": { "key": "claude", "provider": "anthropic", "model env": "CLAUDE MODEL", "label": "Claude Sonnet 5.5", "price": {"input": 2.00, "output": 10.00, "as of": "2026-11-06"} }, { "key": "chatgpt", "provider": "openai", "model env": "OPENAI MODEL", "label": "GPT-6.1 Sol", "price": {"input": 2.00, "output": 10.00, "as of": "2026-11-06"} } } Three design choices deserve a sentence each. The config stores the environment variable name, not the model ID, so the file is safe to commit and the same file runs a different model tomorrow. Sampling settings such as temperature are absent on purpose. Several current models restrict or ignore them, and a fair test leaves both vendors on defaults and records that fact. And max output tokens sits well above the 600-word target, because models that reason before answering spend part of that budget on thinking, which the troubleshooting section returns to. Also log what the config cannot hold. For every run, write down the exact model ID, the plan or API tier, the date and the reasoning mode. That record is what makes a result reproducible, and it is the first thing to check when two people get different answers. | Field to record | Example | Why it matters | |---|---|---| | Exact model ID | The string you exported in CLAUDE MODEL | Vendors ship new snapshots under familiar names | | Surface | API, ChatGPT chat, Work or Codex | Some models are documented only for Work and Codex | | Plan or tier | Plus, Pro 100, Team, API pay-as-you-go | Plan limits change which models you can reach | | Run date | 2026-11-06 | Models, prices and limits move within weeks | | Reasoning mode | Default, or the effort setting you chose | Thinking time changes cost, speed and output length | | Run label | nov06-a | Keeps repeat runs separate on disk | Steps 3 and 4: Canon Ledger and Section Briefs The canon ledger is the ground truth the checker holds both assistants to. The product is fictional on purpose, so neither model can lean on training data. Step 3 writes canon.json : { "name": "Northwind Sync", "name pattern": "north \\s- ?wind \\s- ?sync ?:er|s|ing ?|nw \\s- ?sync", "slots": {"id": "team price", "anchor": "team plan", "value": "\\$\\d+ ?:\\.\\d+ ?", "expected": "$49"}, {"id": "launch date", "anchor": "launch|debut|first shipped", "value": " ?:January|February|March|April|May|June|July|August|September|October|November|December \\s+\\d{1,2}", "expected": "March 14"}, {"id": "version", "anchor": "version", "value": "\\b\\d+\\.\\d+ ?:\\.\\d+ ?\\b", "expected": "3.2"}, {"id": "free storage", "anchor": "free tier|free plan", "value": "\\d+\\s?GB", "expected": "15 GB"}, {"id": "retention", "anchor": "retention|retained|file history", "value": "\\d+\\s?days", "expected": "30 days"}, {"id": "uptime", "anchor": "uptime", "value": "\\d+\\.\\d+\\s?%", "expected": "99.95%"} , "ledger": "Product name: Northwind Sync always written exactly like this ", "Team plan price: $49 per seat per month", "First shipped: March 14", "Current version: 3.2", "Free tier storage: 15 GB", "File history retention: 30 days", "Measured uptime: 99.95%" } Each slot is a tiny rule: find a sentence that contains the anchor phrase, pull every value of the right shape out of that sentence, and flag any value that differs from the expected one. The name pattern catches the variants that models actually produce, such as a capital W in the middle, an added “er” suffix, a space inside the name, or an abbreviation. The ledger list is what the model sees. The slots are what the checker sees. Keeping them separate lets you change a fact in one place and update the other deliberately, which also prevents the model from receiving the regexes. Step 4 is briefs.json , six section prompts. Notice that facts return on purpose: the version appears in sections one and five, and the launch date and Team plan price come back in section six. A fact stated once cannot drift. A fact the draft must restate several times can. {"title": "Overview", "brief": "Introduce the product, say when it first shipped and which version is current."}, {"title": "Pricing", "brief": "Explain the Team plan price and what the free tier includes."}, {"title": "Storage and history", "brief": "Describe free tier storage and how long file history is retained."}, {"title": "Reliability", "brief": "Discuss measured uptime and how the team reports incidents."}, {"title": "Security", "brief": "Cover encryption and access controls, and name the current version again."}, {"title": "Migration and outlook", "brief": "Explain moving from other tools, restate the launch date and the Team plan price, and close with the next milestones."} Swap this fixture for your own domain once the pipeline works. A legal brief, a technical manual or a product catalog each produce different drift. Keep six sections and restate each fact at least twice. Steps 5 and 6: Providers and a Resumable Runner Step 5 is driftlab/providers.py . It wraps both SDKs behind one complete method and adds the offline fake. The real adapters return the same four fields, so the rest of the harness never knows which vendor it is talking to. python import os, random, re, time from dataclasses import dataclass @dataclass class Reply: text: str input tokens: int output tokens: int seconds: float class AnthropicProvider: def init self, model : import anthropic self.client, self.model = anthropic.Anthropic , model def complete self, system, prompt, max tokens, meta=None : start = time.time r = self.client.messages.create model=self.model, max tokens=max tokens, system=system, messages= {"role": "user", "content": prompt} text = "".join b.text for b in r.content if b.type == "text" return Reply text, r.usage.input tokens, r.usage.output tokens, time.time - start class OpenAIProvider: def init self, model : import openai self.client, self.model = openai.OpenAI , model def complete self, system, prompt, max tokens, meta=None : start = time.time r = self.client.responses.create model=self.model, instructions=system, input=prompt, max output tokens=max tokens return Reply r.output text, r.usage.input tokens, r.usage.output tokens, time.time - start Both adapters send one user message with no tools, no web search and no extra settings. The prompt is identical across vendors, which is the whole point. The next block finishes the file with the fake provider and the factory. The fake drifts on purpose: wrong values and name variants appear more often in later sections and in summary mode, so the checks have real errors to find. WRONG = { "$49": "$59", "$45" , "March 14": "March 18", "April 2" , "3.2": "3.1", "3.0" , "15 GB": "10 GB", "20 GB" , "30 days": "14 days", "60 days" , "99.95%": "99.9%", "99.5%" , } VARIANTS = "NorthWind Sync", "Northwind Syncer", "North Wind Sync", "NW Sync" OPENERS = "In practice,", "For most teams,", "At scale,", "Put simply," FILLER = "Admins review the dashboard each Monday before planning the week.", "Support answers most tickets within a few hours.", "Large folders sync in the background without blocking the desktop client.", "Teams that mix Windows, macOS and Linux laptops see the same file tree everywhere.", "The changelog lists every fix.", class FakeProvider: """Offline stand-in that injects drift on purpose so the checks have something to find.""" BASE = {"full": 0.03, "summary": 0.12, "ledger": 0.02} def init self, key : self.key = key def complete self, system, prompt, max tokens, meta=None : meta = meta or {} if meta.get "kind" == "summary": return Reply "Northwind Sync, Team plan $49, shipped March 14, version 3.2.", 900, 40, 0.4 idx, mode = meta "index" , meta "mode" rng = random.Random f"{self.key}-{mode}-{idx}" extra = 0.015 if self.key == "chatgpt" else 0.0 p = self.BASE mode + extra + 0.03 idx name = lambda: rng.choice VARIANTS if rng.random < p else "Northwind Sync" val = lambda v: rng.choice WRONG v if rng.random < p else v opener = lambda: rng.choice OPENERS :2 if mode == "summary" and idx 2 else rng.choice OPENERS sents = f"{opener } {name } first shipped on {val 'March 14' } and now runs version {val '3.2' }.", f"The Team plan costs {val '$49' } per seat each month.", f"{rng.choice FILLER }", f"The free tier gives every account {val '15 GB' } of space.", f"File history is retained for {val '30 days' } on every paid plan.", f"{opener } {name } reports measured uptime of {val '99.95%' } over the past year.", f"{rng.choice FILLER }", "Teams that outgrow the free tier usually upgrade within a quarter.", text = " ".join sents return Reply text, 700 + 450 idx if mode == "full" else 700, len text.split 2, 1.0 + 0.1 idx def make provider spec, dry : if dry: return FakeProvider spec "key" model = os.environ.get spec "model env" if not model: raise SystemExit f"Set {spec 'model env' } to the exact model ID for {spec 'label' }" return AnthropicProvider model if spec "provider" == "anthropic" else OpenAIProvider model The fake gives the ChatGPT key a slightly higher base error rate than the Claude key. That is a fixture setting and carries no claim about either real assistant. Step 6 is driftlab/runner.py , where the three memory conditions live. Section one always receives the full ledger. After that, each mode feeds the model something different. python import json from pathlib import Path SYSTEM = "You are drafting one section of a long internal handbook for a software product. " "Write about {words} words of plain prose in three paragraphs. No headings, no bullet lists. " "Write the product name exactly as it appears in the ledger or earlier text. " "State only facts that appear there. Do not invent numbers." def build prompt mode, canon, brief, idx, texts, summary : ask = f"Section {idx + 1}: {brief 'title' }. {brief 'brief' }" if idx == 0: ledger = "\n".join canon "ledger" return f"Fact ledger:\n{ledger}\n\n{ask}" if mode == "full": history = "\n\n".join texts return f"Earlier sections:\n{history}\n\n{ask}" if mode == "summary": return f"Running summary of earlier sections:\n{summary}\n\n{ask}" ledger = "\n".join canon "ledger" tail = " ".join texts -1 .split -40: return f"Fact ledger:\n{ledger}\n\nEnd of previous section: ...{tail}\n\n{ask}" def run one spec, mode, provider, cfg, canon, briefs, out dir : path = Path out dir / f"{spec 'key' }-{mode}.json" if path.exists : return json.loads path.read text system = SYSTEM.format words=cfg "words per section" texts, summary, records = , "", for idx, brief in enumerate briefs : prompt = build prompt mode, canon, brief, idx, texts, summary meta = {"index": idx, "mode": mode, "kind": "section"} reply = provider.complete system, prompt, cfg "max output tokens" , meta texts.append reply.text records.append {"index": idx, "title": brief "title" , "kind": "section", "text": reply.text, "in": reply.input tokens, "out": reply.output tokens, "sec": reply.seconds} if mode == "summary" and idx < len briefs - 1: ask = "Summarize everything stated so far in under 120 words. Keep every name, " "price, date, version and number exactly as written.\n\n" + "\n\n".join texts s = provider.complete "You compress drafts without changing facts.", ask, 400, {"index": idx, "mode": mode, "kind": "summary"} summary = s.text records.append {"index": idx, "title": "summary", "kind": "summary", "text": s.text, "in": s.input tokens, "out": s.output tokens, "sec": s.seconds} path.parent.mkdir parents=True, exist ok=True path.write text json.dumps records, indent=1 return records Two details protect your budget. The runner writes one file per model and mode only after all six sections succeed, and it returns the saved file if one exists. A crashed run therefore never leaves half a result, and a rerun never pays twice for a finished condition. Summary mode also bills the extra summarizer calls, which the cost column counts, since a cheaper summary strategy that drifts costs more in the end than it saves. If you want to see how a staged workflow changes the outcome, the multi-stage writing pipeline https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ tutorial chains drafting, editing and fact checks across different models. Steps 7 and 8: Continuity Checks and Drift Metrics Step 7 starts driftlab/checks.py with the two checks that read the canon. Both are pure functions of text, so you can test them without a network. python import re from statistics import mean, pstdev SENT = re.compile r" ?<= . ? \s+" CLICHES = "game-changer", "unlock the power", "dive into", "cutting-edge", "revolutionize", "at its core" def sentences text : return s for s in SENT.split text.strip if s def words text : return re.findall r" A-Za-z0-9$%.'- +", text def name violations text, canon : pattern = re.compile canon "name pattern" , re.I return m.group 0 for m in pattern.finditer text if m.group 0 = canon "name" def fact violations text, canon : bad = for slot in canon "slots" : anchor = re.compile slot "anchor" , re.I value = re.compile slot "value" , re.I want = slot "expected" .replace " ", "" .lower for sent in sentences text : if anchor.search sent : for m in value.finditer sent : if m.group 0 .replace " ", "" .lower = want: bad.append slot "id" , m.group 0 return bad The name check compares the matched text to the canonical string case-sensitively, so “northwind sync” in lowercase counts as drift. That is deliberate. A brand name that changes case is a brand error. The fact check compares values after stripping spaces, so “15GB” and “15 GB” both pass. Step 8 appends the repetition and style metrics plus the combined section score: python def ngrams tokens, n=4 : return {tuple tokens i:i + n for i in range len tokens - n + 1 } def repeat rate text, earlier : now = ngrams w.lower for w in words text before = set for e in earlier: before |= ngrams w.lower for w in words e return len now & before / len now if now else 0.0 def opener repeat text, earlier : key = lambda s: " ".join s.lower .split :2 seen = {key s for e in earlier for s in sentences e } sents = sentences text return sum key s in seen for s in sents / len sents if sents else 0.0 def score section text, canon, earlier : n = len words text or 1 names = name violations text, canon facts = fact violations text, canon cliches = sum text.lower .count c for c in CLICHES lengths = len words s for s in sentences text or 0 weighted = 2 len names + 3 len facts + cliches return { "words": n, "names": len names , "facts": len facts , "cliches": cliches, "repeat": repeat rate text, earlier , "openers": opener repeat text, earlier , "sent mean": mean lengths , "sent sd": pstdev lengths , "drift per 1k": weighted 1000 / n, "clean": weighted == 0, } The weighting is a judgment call, and you should tune it. A wrong price costs a reader more than a stray cliche, so facts weigh 3, names weigh 2 and cliches weigh 1. Dividing by length gives drift per 1,000 words, which stops a wordy model from looking better simply because it spreads the same errors over more text. A section is “clean” only when the weighted total is zero. Repetition is scored against everything written earlier in the same run, never against the other vendor. Sentence-length standard deviation shows flattening, since a draft that slides into uniform sentences loses its rhythm. For a deeper style comparison against a named author, the voice-match stylometry test https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ goes further than these coarse counters. Steps 9 and 10: Report and Command Line Step 9 is driftlab/report.py . It converts raw records into one row per model and mode, computes cost from the dated prices, and adds the drift curve, which is the table you will read first. python import json from pathlib import Path from statistics import mean from .checks import score section def cost of records, price : tin = sum r "in" for r in records tout = sum r "out" for r in records return tin price "input" + tout price "output" / 1 000 000 def summarize spec, mode, records, canon : sections = r for r in records if r "kind" == "section" scored, earlier = , for r in sections: scored.append score section r "text" , canon, earlier earlier.append r "text" half = len scored // 2 first = mean s "drift per 1k" for s in scored :half last = mean s "drift per 1k" for s in scored half: cost = cost of records, spec "price" clean = sum s "clean" for s in scored later = scored 1: return { "model": spec "label" , "mode": mode, "sections": len scored , "clean": clean, "names": sum s "names" for s in scored , "facts": sum s "facts" for s in scored , "drift per 1k": mean s "drift per 1k" for s in scored , "slope": last - first, "repeat": mean s "repeat" for s in later , "openers": mean s "openers" for s in later , "sent sd": mean s "sent sd" for s in scored , "cost": cost, "cost per clean": cost / clean if clean else None, "seconds": sum r "sec" for r in records , } def render rows, label : head = "| Model | Mode | Clean | Names | Facts | Drift/1k | Slope | Repeat | Cost | $/clean |\n" "|---|---|---|---|---|---|---|---|---|---|\n" lines = for r in rows: pc = f"{r 'cost per clean' :.4f}" if r "cost per clean" else "n/a" lines.append f"| {r 'model' } | {r 'mode' } | {r 'clean' }/{r 'sections' } | {r 'names' } | " f"{r 'facts' } | {r 'drift per 1k' :.1f} | {r 'slope' :+.1f} | " f"{r 'repeat' :.2f} | ${r 'cost' :.4f} | {pc} |" return f" Drift report: {label}\n\n" + head + "\n".join lines + "\n" def curve spec, mode, records, canon : earlier, cells = , for r in records: if r "kind" = "section": continue cells.append f"{score section r 'text' , canon, earlier 'drift per 1k' :.0f}" earlier.append r "text" return f"| {spec 'label' } | {mode} | " + " | ".join cells + " |" def build cfg, canon, out dir : rows, curves = , for spec in cfg "models" : for mode in cfg "modes" : path = Path out dir / f"{spec 'key' }-{mode}.json" if path.exists : records = json.loads path.read text rows.append summarize spec, mode, records, canon curves.append curve spec, mode, records, canon rows.sort key=lambda r: r "drift per 1k" , r "cost" n = curves 0 .count "|" - 3 if curves else 0 head = "| Model | Mode | " + " | ".join f"S{i + 1}" for i in range n + " |\n" head += "|---|---|" + "---|" n + "\n" md = render rows, cfg "label" + "\n Drift per 1,000 words by section\n\n" + head + "\n".join curves + "\n" Path out dir / "report.md" .write text md Path out dir / "report.json" .write text json.dumps rows, indent=1 return md The metric that earns its place is dollars per clean section. A cheap model that ships two clean sections out of six costs more per usable section than a pricier one that ships five. The slope column subtracts the first-half average from the second-half average, so a positive number means the draft got worse as it grew. Step 10 is driftlab/cli.py , the entry point: python import argparse, json from pathlib import Path from .providers import make provider from .report import build from .runner import run one ROOT = Path file .resolve .parent.parent def load name : return json.loads ROOT / name .read text def main : ap = argparse.ArgumentParser ap.add argument "command", choices= "run", "report" ap.add argument "--dry", action="store true", help="use the offline fake provider" ap.add argument "--only", help="run a single model key, for example claude" ap.add argument "--label", help="override the run label" args = ap.parse args cfg, canon, briefs = load "config.json" , load "canon.json" , load "briefs.json" if args.label: cfg "label" = args.label out dir = ROOT / "runs" / cfg "label" + "-dry" if args.dry else "" if args.command == "run": for spec in cfg "models" : if args.only and spec "key" = args.only: continue provider = make provider spec, args.dry for mode in cfg "modes" : print f"running {spec 'key' } / {mode}" run one spec, mode, provider, cfg, canon, briefs, out dir print build cfg, canon, out dir if name == " main ": main Dry runs write to a folder ending in -dry , so fake results can never mix with real ones. The --label flag gives every live run its own folder, which you need for repeat runs. The command line stays thin on purpose. It loads three JSON files, builds one provider per model, loops over the modes and prints the report. Everything that could go wrong sits in a file you can test alone. If you later add a third vendor, you add one entry to config.json and one adapter class, and the runner, the checks and the report need no edits. That separation also makes the harness easy to hand to a colleague, because they can run the dry mode first, see the report shape and trust the machinery before they spend a cent. Steps 11 and 12: Tests, Dry Run and Live Run Step 11 locks the checks with unit tests before a model is involved. If a regex breaks later, these fail first. Save this as tests/test checks.py : python import json, unittest from pathlib import Path from driftlab.checks import fact violations, name violations, repeat rate, score section CANON = json.loads Path file .resolve .parent.parent / "canon.json" .read text class ChecksTest unittest.TestCase : def test exact name passes self : self.assertEqual name violations "Northwind Sync is fast.", CANON , def test name variants fail self : bad = name violations "NorthWind Sync and Northwind Syncer differ.", CANON self.assertEqual len bad , 2 def test wrong price fails self : bad = fact violations "The Team plan costs $59 per seat.", CANON self.assertEqual bad, "team price", "$59" def test right facts pass self : text = "The Team plan costs $49. The free tier has 15 GB. Uptime is 99.95%." self.assertEqual fact violations text, CANON , def test repeat rate detects copy self : a = "the quick brown fox jumps over the lazy dog today" self.assertGreater repeat rate a, a , 0.9 def test clean section flag self : s = score section "Northwind Sync shipped version 3.2 on March 14.", CANON, self.assertTrue s "clean" if name == " main ": unittest.main Run the tests, then the dry run: python -m unittest discover -s tests -v python -m driftlab.cli run --dry All six tests pass in a fraction of a second. The dry run prints the report below. These numbers come from the offline fake, which injects errors on a schedule, so read them for shape and never as a verdict on either vendor: Drift report: baseline | Model | Mode | Clean | Names | Facts | Drift/1k | Slope | Repeat | Cost | $/clean | |---|---|---|---|---|---|---|---|---|---| | Claude Sonnet 5.5 | full | 4/6 | 0 | 2 | 11.0 | +22.0 | 0.83 | $0.0326 | 0.0081 | | GPT-6.1 Sol | ledger | 3/6 | 0 | 3 | 17.1 | +11.2 | 0.84 | $0.0190 | 0.0063 | | Claude Sonnet 5.5 | ledger | 3/6 | 2 | 3 | 25.2 | +26.3 | 0.80 | $0.0186 | 0.0062 | | GPT-6.1 Sol | full | 1/6 | 3 | 5 | 42.4 | +18.9 | 0.80 | $0.0320 | 0.0320 | | GPT-6.1 Sol | summary | 1/6 | 3 | 8 | 55.7 | +52.2 | 0.78 | $0.0301 | 0.0301 | | Claude Sonnet 5.5 | summary | 0/6 | 1 | 14 | 84.0 | +7.9 | 0.73 | $0.0299 | n/a | Drift per 1,000 words by section | Model | Mode | S1 | S2 | S3 | S4 | S5 | S6 | |---|---|---|---|---|---|---|---| | Claude Sonnet 5.5 | full | 0 | 0 | 0 | 0 | 34 | 32 | | Claude Sonnet 5.5 | summary | 94 | 72 | 74 | 22 | 103 | 138 | | Claude Sonnet 5.5 | ledger | 0 | 0 | 36 | 0 | 58 | 57 | | GPT-6.1 Sol | full | 0 | 63 | 36 | 37 | 60 | 60 | | GPT-6.1 Sol | summary | 55 | 34 | 0 | 57 | 100 | 88 | | GPT-6.1 Sol | ledger | 0 | 34 | 0 | 0 | 35 | 33 | The shape is what you should expect from a real run: drift climbs toward the last sections, and summary mode is the leakiest memory strategy. The dollar figures are fake-token costs and carry no information about real pricing behavior. Step 12 is the live run. Export the keys and exact IDs, run each vendor separately so one failure cannot block the other, and label the run: export ANTHROPIC API KEY=... your key export OPENAI API KEY=... your key export CLAUDE MODEL=... exact ID from Anthropic's model list export OPENAI MODEL=... exact ID from OpenAI's model list python -m driftlab.cli run --only claude --label nov06-a python -m driftlab.cli run --only chatgpt --label nov06-a python -m driftlab.cli report --label nov06-a A live run makes 18 section calls per vendor plus 5 summarizer calls in summary mode. Run the same label scheme three times nov06-a, nov06-b, nov06-c and compare the median. One run measures luck, and three runs begin to measure the model. Before you read any numbers, open one saved run file and read a full section with your own eyes. Automated scores catch what you wrote rules for and nothing else. If a model answers every brief with a refusal, a heading list or a section that ignores the ledger, the scores will look odd in ways that only a human glance explains. Spend five minutes on this check. It has saved more runs than any regex. Reading the Report: A Decision Rule Read the report in this order. First look at the drift curve, because it shows where the damage starts. Then read the Clean column, then dollars per clean section, and last the repetition and slope columns. | Pattern in your report | What it means | Action | |---|---|---| | Drift rises in every mode for one model | The model loses facts as drafts lengthen | Keep drafts short or rotate in a ledger every section | | Drift is low in ledger mode and high in summary mode | Your memory method is the weak link | Stop summarizing, or force the summarizer to copy facts verbatim | | Name drift only, no fact drift | Casing and suffix habits, not memory | Add a spelling line to the system prompt and a post-edit replace | | Repeat above 0.30 after section three | The model recycles its own phrasing | Vary the briefs or ask for a different opening per section | | Low drift but a high price per clean section | Accurate but expensive | Try a cheaper model with a ledger in every call | | Both vendors clean in ledger mode | Quality is a tie on this fixture | Decide on price, tone and workflow, then test your own domain | Two examples show how to apply the table. Suppose Claude holds every fact in ledger mode but shows name variants in summary mode. The fix lives in your workflow: make the summarizer copy names verbatim, and the gap should close. Now suppose ChatGPT shows rising fact drift in every mode, including ledger. That points at the model on this fixture, and the remedy is a shorter section length or a ledger check after every section. Treat the thresholds as starting points. The 0.30 repeat line is a starting guess, not a published benchmark, so calibrate it against drafts you consider acceptable. The honest result of a good run is often a split decision, where one assistant holds names better and the other holds numbers better. That split is useful, because you can route work by task. Do not stretch a Claude vs ChatGPT verdict past what you measured. A fictional file-sync handbook says little about a medical protocol or a legal contract. When you change domains, rebuild the canon and rerun all three modes. Common Pitfalls That Skew a Drift Test 1. Comparing different surfaces. OpenAI documents several newer models for ChatGPT Work and Codex rather than ordinary chat, and plan tiers change which models you reach. An API run and a chat-window run are different experiments. Record the surface on every result. 2. Letting sampling settings differ. If one adapter sets a temperature and the other cannot, you are measuring settings. Leave both vendors on defaults, as this harness does. 3. Trusting one run. Language models vary between calls. A single run can crown the wrong winner, so repeat under three labels and compare medians. 4. Multi-fact sentences. The fact check reads one sentence at a time. If a model writes “The Team plan costs $49, while the Business plan costs $99,” the $99 trips the price slot. Keep slot anchors specific, and read flagged items before you believe a score. 5. Prompts that favor one vendor. Any tweak you make for one assistant, such as extra bullet rules, must go to both. Change the shared prompt or change nothing. 6. Grading with a model. A third model scoring the first two adds cost, adds its own bias, and returns different scores on a rerun. Keep model judges for taste and use code for facts. 7. Hiding untrusted text in the draft. If you later feed real customer text into these prompts, instructions inside it can steer the model. A prompt injection test lab https://shattered.io/prompt-injection-test-lab-llm-harness-2026/ shows how to probe for that before it costs you. 8. Forgetting to date the prices. A cost column built on last month’s rate card misleads. The as of field exists so you notice when it is stale. Troubleshooting: 11 Errors and Fixes 1. “Set CLAUDE MODEL to the exact model ID” and the run stops. The environment variable is empty in the current shell. Export it again and confirm with echo $CLAUDE MODEL . A new terminal tab does not inherit exports from the old one. 2. A 404 or “model not found” error. The ID is wrong or your account cannot reach that model. Copy the ID from the vendor’s model list, not from a blog post. The one ID stated here is gpt-6.1-terra , and even that needs API access on your account. 3. A 401 authentication error. The key is missing, revoked or pasted with a trailing space. Re-export it and make sure the right key goes to the right vendor. 4. A 429 rate-limit error mid-run. You exceeded a requests or tokens limit. The runner saves nothing for a half-finished condition, so rerun the same command later. It skips finished conditions and repeats only the failed one. 5. ModuleNotFoundError: No module named 'driftlab' . You ran a script directly. Run python -m driftlab.cli from the project root, where the driftlab folder sits. 6. Empty text from the OpenAI adapter. Reasoning models can spend the whole output budget on thinking and return no visible text. Raise max output tokens in config.json to 4000 or more, delete the empty run file, and rerun. 7. Sections cut off mid-sentence. The output hit its token ceiling. Raise the limit the same way. A truncated section also loses its later facts, which makes drift look better than it is. 8. A parameter such as temperature is rejected. Some current models restrict sampling settings. Remove the setting rather than working around it, since the harness uses defaults anyway. 9. json.decoder.JSONDecodeError on a run file. You interrupted a write. Delete the broken file in runs/