Claude vs ChatGPT Editing Test: 12 Steps, 75 Min [2026]

A 12-step, 75-minute editing test built on a small Python harness called `editlab` measures how much Claude and ChatGPT models change a writer's prose when asked only to fix typos, rather than judging new writing on taste. The comparison pairs Anthropic's Claude Haiku 5.5, listed at $0.10 per million input tokens and $0.50 per million output tokens, against OpenAI's GPT-6 Luna at the same $0.10/$0.50 rate card, and Claude Sonnet 5.5 against GPT-6 Sol at $2.00 input and $10.00 output per million tokens. The test accounts for OpenAI's October 30 update to GPT-6 in ChatGPT, which cut overly flattering language and made answers more concise, meaning app-based results collected before that date describe a different assistant.

Anthropic’s cheapest current writer, Claude Haiku 5.5, lists at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, according to its platform documentation https://platform.claude.com/docs/en/models/haiku-5-5/overview . OpenAI shows the same $0.10 and $0.50 rate card price for GPT-6 Luna, the model behind ChatGPT’s Free and Go plans. Two models at identical prices can still behave very differently the moment you hand them someone else’s prose. On the OpenAI side, GPT-6 is mid-rollout. Plus, Pro, Business and Enterprise accounts run GPT-6 Sol, while Free and Go run GPT-6 Luna, OpenAI’s pricing page https://chatgpt.com/pricing/ shows. A November 2 roundup https://csuite.so/blog/why-your-ai-doesnt-know-last-week describes the rollout as still in progress, so two people on different plans can get different models under one “ChatGPT” label. That detail matters for any Claude vs ChatGPT test, and it drives most of the design choices in the harness. Most comparisons ask each model to write something new and then argue about taste. A blind drafting test https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ handles that well. Editing is a different job. When a writer hands over a draft and says “fix the typos,” the best answer changes almost nothing else. Some models quietly swap words, soften slang, reformat numbers or add a flourish. A small Python harness called editlab can measure that restraint with plain arithmetic instead of a vibe check. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Claude vs ChatGPT for Editing: The Lineup and Why This Test The model names move fast, so the first job is to pin down what exists today. Anthropic announced Claude Opus 5.5 on September 22, 2026, Claude Sonnet 5.5 on September 28, and Claude Haiku 5.5 on October 7. Anthropic says Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. Haiku 5.5 is the first Haiku-class model with an adjustable effort setting, which matters because effort changes how much a model deliberates before it touches your text. OpenAI’s lineup in ChatGPT centers on GPT-6, with Sol on paid tiers and Luna on Free and Go. OpenAI’s release notes say that on October 30 it updated GPT-6 in ChatGPT to give more concise answers and cut overly flattering language. That is a behavior change that lands squarely on editing. Any app-based result collected before that date describes a different assistant. | Tier | Claude option | ChatGPT option | List price per 1M tokens input / output | |---|---|---|---| | Small and cheap | Claude Haiku 5.5 Oct 7, 2026 | GPT-6 Luna | $0.10 / $0.50 for both | | Mid-range | Claude Sonnet 5.5 Sep 28, 2026 | GPT-6 Sol | $2.00 / $10.00 for both | | Top tier | Claude Opus 5.5 Sep 22, 2026 | GPT-6 Pro | Opus: see Anthropic’s pricing page. GPT-6 Pro: $10 / $50 | | Consumer entry | Free, Pro, Max, Ultra plans | Free $0, Go $6, Plus $20 per month | Not applicable | The pairing logic falls out of that table. Haiku 5.5 against GPT-6 Luna is a fair fight on price, and so is Sonnet 5.5 against GPT-6 Sol. Comparing Opus to a cheap tier would tell you about budgets, not about editing behavior. The harness therefore takes model aliases from a config file, and you pick the pair that matches what you actually pay for. Anthropic and OpenAI both publish mid-tier prices at $2 input and $10 output, which makes the cost column of your final report easy to read. Why build an editing test at all? Because editing failures are quiet and expensive. A model that improves a contract email by rewording the indemnity clause creates legal risk. A model that turns an exact dollar figure into a rounded shorthand changes a figure without telling you. A model that cleans up dialect in fiction erases the author. The same skeleton can later feed a multi-stage writing pipeline https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ , but the editing case is the sharpest place to see restraint. Four numbers carry most of the signal. The fix rate counts how many seeded errors each model repairs. The over-edit rate counts how many words change beyond those repairs. The protected-span rate checks that names, figures and code survive untouched. The voice rate checks that the author’s quirks remain. A weighted fidelity score rolls them into one number from 0 to 100. Prerequisites and Versions Everything below ran on Python 3.12.3 on Linux. The offline path uses only the standard library, so the dry run in step 11 works even if you skip the pip install in step 1 and never spend a cent. The vendor SDKs load lazily, only when a live provider is built. | Component | Version or requirement | Why it matters | |---|---|---| | Python | 3.11 or newer tested on 3.12.3 | difflib , statistics and dataclasses ship with it | | anthropic SDK | Latest release from PyPI | Messages API for Claude models | | openai SDK | Latest release from PyPI | Chat Completions for GPT models | | Test runner | Built-in unittest | No pytest install needed | | API keys | One Anthropic key, one OpenAI key | Live run in step 12 only | | Shell | bash, zsh or PowerShell | Windows users should set PYTHONUTF8=1 | | Budget | Roughly $0.01 to $0.30 per model | Depends on the tier you pick | | Time | About 75 minutes | Most of it is reading the report | Two things to settle before you start. First, decide which pair you are testing and write down today’s date, because model behavior shifts under the same name. Second, decide whether you are testing the API or the chat app. They are not the same product. API billing is separate from ChatGPT subscriptions, and the app adds memory, custom instructions and interface features that an API call never sees. This harness tests the API, which is the only place you can control inputs and log every token. Steps 1 and 2: Project Skeleton and Pinned Models Step 1 creates the folder, a virtual environment and the package layout. Step 2 pins which models the harness will call. mkdir editlab && cd editlab python3 -m venv .venv source .venv/bin/activate Windows: .venv\Scripts\Activate.ps1 mkdir -p editlab data tests runs touch editlab/ init .py pip install anthropic openai Create requirements.txt so a teammate can reproduce the environment, and .env.example so nobody commits a key by accident. Copy the example to .env and fill it in. Add .env and runs/ to your .gitignore . anthropic openai ANTHROPIC API KEY= OPENAI API KEY= Copy the exact model IDs from each vendor's models page on the day you run the test. CLAUDE MODEL=claude-haiku-5-5 OPENAI MODEL= The environment file stores model IDs rather than hard-coding them in Python, and that choice is deliberate. Only one ID is filled in above: claude-haiku-5-5 , the identifier Anthropic documents for Haiku 5.5. For every other model, copy the exact ID from the vendor’s models page on the day you run the test. Guessing at IDs from memory is how people end up testing a retired snapshot while labeling the result with a current name. If you want a view on how plan limits and prices differ across the two vendors, the plan and usage-limit comparison https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ lines up the consumer tiers. The models.json file maps friendly aliases to providers and prices. The usd in and usd out fields hold dollars per million tokens, and the defaults below use the $0.10 and $0.50 cheap-tier rate. Change them to $2 and $10 if you test Sonnet 5.5 against GPT-6 Sol. Check Anthropic’s pricing documentation https://docs.anthropic.com/en/docs/about-claude/pricing on the day you run, because prices change. { "models": {"alias": "claude", "provider": "anthropic", "model env": "CLAUDE MODEL", "usd in": 0.10, "usd out": 0.50}, {"alias": "chatgpt", "provider": "openai", "model env": "OPENAI MODEL", "usd in": 0.10, "usd out": 0.50}, {"alias": "fake-light", "provider": "fake", "model": "light"}, {"alias": "fake-heavy", "provider": "fake", "model": "heavy"} } Now config.py , which loads the .env file without any third-party dependency, resolves aliases into concrete model IDs, and refuses to run when an ID is missing. The refusal is a feature. A silent fallback to a default model would corrupt every row of your results. python import json import os from pathlib import Path ROOT = Path file .resolve .parent.parent def load env path=ROOT / ".env" : if not Path path .exists : return for line in Path path .read text .splitlines : line = line.strip if not line or line.startswith " " or "=" not in line: continue key, value = line.split "=", 1 os.environ.setdefault key.strip , value.strip def load models aliases : spec = json.loads ROOT / "models.json" .read text "models" by alias = {m "alias" : m for m in spec} chosen = for alias in aliases: if alias not in by alias: raise SystemExit f"Unknown alias {alias r}. Known: {sorted by alias }" entry = dict by alias alias if "model env" in entry: model id = os.environ.get entry "model env" , "" .strip if not model id: raise SystemExit f"Set {entry 'model env' } in .env before running {alias}." entry "model" = model id chosen.append entry return chosen def load drafts : return json.loads ROOT / "data" / "drafts.json" .read text Step 3: Build Fixture Drafts With Seeded Errors An editing test needs inputs where the correct answer is known. The fixtures are six short drafts in different registers: a casual blog post, a formal email, a technical runbook with inline code, dialect fiction, a press release dense with figures, and a lowercase chat message. Each draft carries three kinds of metadata. - Seeded errors are typos and doubled words with a known fix, such as “untill” to “until” or “the the” to “the”. Applying them to the draft produces the gold text. - Protected spans are strings that must survive verbatim: dollar amounts, dates, names, version numbers and code in backticks. - Voice markers are phrases that make the author sound like themselves, such as “Honestly,” or “i’m not gonna”. A model that polishes them away has overreached. Every name, company and figure in these fixtures is invented, so you can publish results without leaking anything real. Keep it that way. Do not paste real customer emails into a test that sends text to two vendors, and do not reuse drafts you plan to feed to a production assistant with memory enabled. { "id": "blog-casual", "genre": "casual blog post", "text": "Honestly, we shipped the new dashboard on Tuesday and nobody noticed untill Thursday. The team had been working on it for 14 weeks, and the budget came in at $48,200, which is way under what Priya estimated. I think thats the real story here. Teh load time dropped from 3.4 seconds to 1.1 seconds, and support tickets fell by 22%. We still have two bugs to squash, but honestly, I'd ship it again tomorrow.", "errors": {"wrong": "untill", "right": "until"}, {"wrong": "thats", "right": "that's"}, {"wrong": "Teh", "right": "The"} , "protected": "14 weeks", "$48,200", "Priya", "3.4 seconds", "1.1 seconds", "22%" , "voice": "Honestly,", "honestly,", "I'd ship it again tomorrow" }, { "id": "email-formal", "genre": "business email", "text": "Hi Marcus, I wanted to follow up on the contract we discussed on March 12. The vendor has agreed to a 36-month term at $9,400 per month, but they want payment recieved within 15 days of invoice, wich is shorter than our usual 30 days. Please review section 4.2 and let me know if legal sees any issues with the the indemnity clause. I'll hold the signature page until Friday. Best, Dana", "errors": {"wrong": "recieved", "right": "received"}, {"wrong": "wich", "right": "which"}, {"wrong": "the the indemnity", "right": "the indemnity"} , "protected": "March 12", "36-month", "$9,400", "15 days", "30 days", "section 4.2", "Marcus", "Dana" , "voice": "I'll hold the signature page" }, { "id": "tech-doc", "genre": "technical runbook", "text": "To rotate the key, run kubectl rollout restart deployment/api-gateway -n prod and wait for the pods to come back. The the rollout usually takes about 90 seconds. If the readiness probe fails, check the logs with kubectl logs -l app=api-gateway --tail=50 before you retry. Dont skip this step, because a half-rotated key will lock out every client for upto 5 minutes.", "errors": {"wrong": "The the rollout", "right": "The rollout"}, {"wrong": "Dont", "right": "Don't"}, {"wrong": "upto", "right": "up to"} , "protected": " kubectl rollout restart deployment/api-gateway -n prod ", " kubectl logs -l app=api-gateway --tail=50 ", "90 seconds", "5 minutes" , "voice": "half-rotated" }, { "id": "fiction-dialect", "genre": "dialect fiction", "text": "Ma said the river was risin', and Ma was never wrong about water. We carried the quilts up to the loft while Pa cussed at the mule. \"Ain't no sense in waitin',\" he said, but he was already hauling the seed bags. By midnite the yard was a lake and the chickens roosted on the porch rale. I countd seven of them, plus the one-eyed rooster Pa swore he'd eat by Sunday.", "errors": {"wrong": "midnite", "right": "midnight"}, {"wrong": "rale", "right": "rail"}, {"wrong": "countd", "right": "counted"} , "protected": "seven", "Ma", "Pa", "Sunday", "one-eyed rooster" , "voice": "risin'", "Ain't no sense in waitin'", "cussed" }, { "id": "press-release", "genre": "press release", "text": "Northwind Labs today announced a $12.5 million Series A led by Cobalt Ventures, bringing total funding to $18 million. The round will fund hiring of 40 engineers across Austin and Lisbon by Q2 2027. CEO Imani Okafor said the compnay has grown revenue 3x year over year, and plans to open its API to outside developers in Janaury. The announcment follows a pilot with 120 customers.", "errors": {"wrong": "compnay", "right": "company"}, {"wrong": "Janaury", "right": "January"}, {"wrong": "announcment", "right": "announcement"} , "protected": "$12.5 million", "Cobalt Ventures", "$18 million", "40 engineers", "Austin", "Lisbon", "Q2 2027", "Imani Okafor", "3x", "120 customers" , "voice": }, { "id": "slack-lowercase", "genre": "lowercase chat message", "text": "ok so the migration finished at 3:40am and the checksum diff came back clean for 2,184 of 2,190 tables. six tables failed becuase of the legacy enum thing we already knew about. i'm not gonna touch them till standup. @renee can you confrim the replica lag was under 200ms the whole time? if so we can drop the old cluster on friday.", "errors": {"wrong": "becuase", "right": "because"}, {"wrong": "confrim", "right": "confirm"} , "protected": "3:40am", "2,184 of 2,190", "@renee", "200ms", "friday" , "voice": "ok so", "i'm not gonna", "till standup" } Six drafts is a starting point, not a verdict. It is enough to separate a model that edits with a light hand from one that rewrites, as the dry run in step 11 shows. It is not enough to rank two careful editors, and the advanced tips explain how to size the sample once you care about small gaps. Aim for at least 20 drafts, drawn from your own writing, before you act on a result. One design note on the errors. Each wrong string appears exactly once in its draft, so a plain string replacement builds the gold text with no ambiguity. If you add your own drafts, check that property, or the gold text will drift and your over-edit numbers will carry a hidden bias. Steps 4 and 5: Prompts and Provider Adapters Step 4 defines three prompts that mirror how real people ask for edits. The strict prompt spells out the rules: fix only spelling, doubled words and clear grammar, keep voice, leave numbers and code alone, return the text with no commentary. The light prompt is the lazy version, “proofread this.” The polish prompt invites change on purpose. Running all three tells you whether a model’s restraint comes from its habits or from your instructions. STRICT = "You are a copy editor. Fix only spelling, typos, doubled words, and clear grammar " "errors. Do not rephrase, reorder, shorten, or add anything. Keep the author's voice, " "slang, dialect, and capitalization. Leave numbers, names, code, and quoted text " "exactly as written. Return only the corrected text with no commentary." LIGHT = "Proofread this text and return the corrected version only." POLISH = "Improve this text so it reads better. Return the improved text only." PROMPTS = {"strict": STRICT, "light": LIGHT, "polish": POLISH} def user message draft text : return f"<draft \n{draft text}\n</draft " The user message wraps the draft in <draft tags so the model can tell instructions from material. That also gives the cleaning step in the metrics module a reliable thing to strip when a model echoes the tags back. Step 5 writes the two live adapters. Each exposes the same edit system, user, run idx method and returns a Result with the text, token counts and wall-clock latency. Note what the calls leave out. Neither sends a temperature, a seed or a token cap beyond what Anthropic requires. Some newer models restrict sampling parameters, and sending an unsupported one can get your request rejected. The harness records whatever default each vendor applies and compares those defaults, which is closer to what a typical user experiences anyway. The three-way price gap breakdown https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/ covers where vendor prices diverge. At the cheap and mid tiers they match, so behavior becomes the deciding factor. python import os import time from dataclasses import dataclass @dataclass class Result: text: str tokens in: int tokens out: int latency s: float class AnthropicProvider: def init self, model : import anthropic self.model = model self.client = anthropic.Anthropic api key=os.environ "ANTHROPIC API KEY" def edit self, system, user, run idx : start = time.perf counter resp = self.client.messages.create model=self.model, max tokens=4096, system=system, messages= {"role": "user", "content": user} , text = "".join b.text for b in resp.content if b.type == "text" return Result text, resp.usage.input tokens, resp.usage.output tokens, time.perf counter - start class OpenAIProvider: def init self, model : from openai import OpenAI self.model = model self.client = OpenAI api key=os.environ "OPENAI API KEY" def edit self, system, user, run idx : start = time.perf counter resp = self.client.chat.completions.create model=self.model, messages= {"role": "system", "content": system}, {"role": "user", "content": user}, , return Result resp.choices 0 .message.content or "", resp.usage.prompt tokens, resp.usage.completion tokens, time.perf counter - start def build entry : kind = entry "provider" if kind == "anthropic": return AnthropicProvider entry "model" if kind == "openai": return OpenAIProvider entry "model" if kind == "fake": from .fake import FakeProvider return FakeProvider entry "model" raise SystemExit f"Unknown provider {kind r}" Anthropic’s Messages API needs max tokens , so the adapter sets 4096, which is far above what a 90-word draft needs. The OpenAI adapter reads usage.prompt tokens and usage.completion tokens for cost tracking. If either vendor bills hidden reasoning tokens as output, those tokens show up in the usage counts and flow into your cost column automatically, which is exactly where you want to find them. Step 6: An Offline Fake Provider for Free Testing Before you spend money, prove the pipeline works. The FakeProvider is a deterministic stand-in with two personalities. The light mode applies the known fixes but skips a few of them at random, like a tired proofreader. The heavy mode applies every fix and then rewrites phrases: it drops a casual opener, rounds exact figures into shorthand, swaps in stock business vocabulary, and cleans dialect into standard English. The randomness comes from a seed built from a CRC32 of the mode, prompt, draft and run index. Python’s built-in hash changes between processes, so using it would make your dry runs unrepeatable. With CRC32 the same command produces the same numbers every time, which lets you write assertions against them. python import random import re import zlib from .providers import Result FIXES = { "untill": "until", "thats": "that's", "Teh": "The", "recieved": "received", "wich": "which", "the the indemnity": "the indemnity", "The the rollout": "The rollout", "Dont": "Don't", "upto": "up to", "midnite": "midnight", "rale": "rail", "countd": "counted", "compnay": "company", "Janaury": "January", "announcment": "announcement", "becuase": "because", "confrim": "confirm", } HEAVY REWRITES = "Honestly, ", "" , "honestly, ", "" , "I'd ship it again tomorrow", "I would gladly leverage this launch again" , "which is way under", "which is well under" , "$48,200", "$48.2K" , "squash", "address" , "cussed", "complained" , "risin'", "rising" , "Ain't no sense in waitin',", "There is no sense in waiting," , "ok so ", "Update: " , "i'm not gonna", "I will not" , "gonna", "going to" , "Please review", "Kindly review" , "a lake", "a game-changer of a lake" , class FakeProvider: """Deterministic stand-in so the whole pipeline runs offline.""" def init self, mode : self.mode = mode def edit self, system, user, run idx : draft = re.search r"<draft \n . \n</draft ", user, re.S .group 1 rng = random.Random zlib.crc32 f"{self.mode}|{system}|{draft}|{run idx}".encode out = draft for wrong, right in FIXES.items : if self.mode == "light" and rng.random < 0.08: continue out = out.replace wrong, right polish = "Improve" in system if self.mode == "heavy" or polish and self.mode == "light" : rate = 0.9 if self.mode == "heavy" else 0.3 for old, new in HEAVY REWRITES: if rng.random < rate: out = out.replace old, new return Result out, len system.split + len user.split , len out.split , 0.4 + rng.random 0.6 These fakes are not models, and nothing they produce says anything about Claude or ChatGPT. Their job is to give the metrics a known “careful editor” and a known “over-eager editor” so you can confirm the scoring separates them. If your scoring cannot tell the fake editors apart, it will not tell real ones apart either. Steps 7 and 8: The Metrics and the Fidelity Score Step 7 writes the measurements. Each is a few lines, and each answers one question about the output. - fix rate asks how many seeded errors were repaired. An error counts as fixed only if the right form is present and the wrong form is gone. - over edit tokenizes the gold text and the model output, runs difflib.SequenceMatcher , and sums the size of every non-matching block. Dividing by the gold token count gives the share of the text that changed beyond the seeded repairs. - protected rate checks each protected span as a verbatim substring. - voice rate checks that each voice marker appears at least as often as in the original, case-sensitively. - tell count counts newly introduced filler words from a short list, plus any new em dashes. The list includes “leverage”, “game-changer”, “unlock”, “elevate”, “foster”, “pivotal”, “realm”, “crucial role”, “to sum up” and “kindly”. It will not catch every habit, but it catches the most common injected polish. Two details prevent false results. The norm function folds curly quotes into straight ones before any comparison, because models often “smarten” apostrophes and a naive check would then report a missed fix on “Don’t”. The clean output function strips <draft tags and code fences, which models sometimes add around the answer. Developers who care more about code than prose can start from the separate Claude vs ChatGPT coding comparison https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/ . Step 8 is the fidelity score. The weights below reflect a judgment call you can change. | Component | Weight | What a low value means | |---|---|---| | Fix rate | 35% | The model missed real errors | | Over-edit capped at 15% of text | 25% | The model rewrote beyond the brief | | Protected spans | 20% | Names, figures or code changed | | Voice markers | 10% | The author’s style got sanded down | | Tell words capped at 3 | 10% | The model injected filler | Fix rate gets the largest weight because an editor who fixes nothing is useless however polite. Over-edit comes second because it is the failure that readers rarely notice until the damage is done. Cap both penalties so a single catastrophic output cannot push a score below zero. A score of 100 means the output matched the gold text exactly. python import difflib import re import statistics TOKEN = re.compile r"\w+ ?:'\w+ | ^\w\s " TELLS = "leverage", "game-changer", "unlock", "elevate", "foster", "pivotal", "realm", "crucial role", "to sum up", "kindly" EM DASH = ", " def norm text : return text.replace "\u2019", "'" .replace "\u2018", "'" .replace "\u201c", '"' .replace "\u201d", '"' def clean output text : text = re.sub r"</?draft ", "", text text = re.sub r"^ a-z \n|\n $", "", text.strip return norm text .strip def tokens text : return TOKEN.findall text def apply fixes draft : gold = norm draft "text" for err in draft "errors" : gold = gold.replace err "wrong" , err "right" return gold def fix rate draft, output : out = norm output fixed = sum 1 for e in draft "errors" if e "right" in out and e "wrong" not in out return fixed / len draft "errors" def over edit draft, output : gold, got = tokens apply fixes draft , tokens output changed = 0 for tag, i1, i2, j1, j2 in difflib.SequenceMatcher None, gold, got, autojunk=False .get opcodes : if tag = "equal": changed += max i2 - i1, j2 - j1 return changed / max len gold , 1 def protected rate draft, output : spans = draft "protected" return sum 1 for s in spans if norm s in output / len spans if spans else 1.0 def voice rate draft, output : markers = draft "voice" if not markers: return 1.0 src = norm draft "text" kept = sum 1 for m in markers if output.count norm m = src.count norm m return kept / len markers def tell count draft, output : src, out = norm draft "text" .lower , output.lower new = sum max 0, out.count t - src.count t for t in TELLS return new + max 0, output.count EM DASH - draft "text" .count EM DASH def fidelity fix, over, prot, voice, tells : score = 0.35 fix + 0.25 1 - min 1.0, over / 0.15 + 0.20 prot + 0.10 voice + 0.10 1 - min 1.0, tells / 3 return round 100 score, 1 def score output draft, raw output : out = clean output raw output fix, over = fix rate draft, out , over edit draft, out prot, voice, tells = protected rate draft, out , voice rate draft, out , tell count draft, out return { "fix rate": fix, "over edit": round over, 4 , "protected": prot, "voice": voice, "tells": tells, "fidelity": fidelity fix, over, prot, voice, tells , } def mean sd values : values = list values if not values: return 0.0, 0.0 return statistics.fmean values , statistics.stdev values if len values 1 else 0.0 Steps 9 and 10: Runner, Report and Command Line Step 9 is the runner. It loops over models, prompts, drafts and run indexes, calls the provider, scores each output, computes cost from the token counts and the per-million prices in models.json , and writes both a raw JSON file per call and one results.csv . Saving the raw output is not optional. When a score looks strange, you need to read the exact text that produced it. A failed call prints a warning, waits two seconds and moves on, so one rate-limit error does not sink a 90-call run. python import csv import json import time from pathlib import Path from . import metrics, prompts, providers def run matrix models, drafts, prompt names, runs, out dir : out dir = Path out dir out dir / "raw" .mkdir parents=True, exist ok=True rows = for entry in models: provider = providers.build entry for pname in prompt names: system = prompts.PROMPTS pname for draft in drafts: for run idx in range runs : user = prompts.user message draft "text" try: res = provider.edit system, user, run idx except Exception as exc: print f" {entry 'alias' } {draft 'id' } run {run idx}: {exc}" time.sleep 2 continue score = metrics.score output draft, res.text cost = "" if "usd in" in entry: cost = round res.tokens in / 1e6 entry "usd in" + res.tokens out / 1e6 entry "usd out" , 6 row = { "alias": entry "alias" , "model": entry "model" , "prompt": pname, "draft": draft "id" , "run": run idx, score, "tokens in": res.tokens in, "tokens out": res.tokens out, "latency s": round res.latency s, 3 , "cost usd": cost, } rows.append row raw = out dir / "raw" / f"{entry 'alias' } {pname} {draft 'id' } {run idx}.json" raw.write text json.dumps {"row": row, "output": res.text}, indent=2 print f"done {entry 'alias' } / {pname}" with open out dir / "results.csv", "w", newline="" as fh: writer = csv.DictWriter fh, fieldnames=list rows 0 .keys writer.writeheader writer.writerows rows return rows Step 10 adds the report and the command-line entry point. The report groups rows by model and prompt, shows the mean of each metric, formats fidelity as a mean with a standard deviation, totals the cost and counts per-draft wins. The win count uses a 0.5-point tie band, so near-identical draft scores do not get credited as victories. python import csv import statistics from collections import defaultdict from pathlib import Path from .metrics import mean sd def load rows out dir : with open Path out dir / "results.csv" as fh: rows = list csv.DictReader fh for r in rows: for key in "fix rate", "over edit", "protected", "voice", "tells", "fidelity", "tokens in", "tokens out", "latency s" : r key = float r key r "cost usd" = float r "cost usd" if r "cost usd" else None return rows def summary table rows : groups = defaultdict list for r in rows: groups r "alias" , r "prompt" .append r lines = "| Model | Prompt | Fix rate | Over-edit | Protected | Voice | Tells/run | Fidelity mean +/- sd | Cost USD |", "|---|---|---|---|---|---|---|---|---|" for alias, prompt , grp in sorted groups.items : fid, sd = mean sd r "fidelity" for r in grp costs = r "cost usd" for r in grp if r "cost usd" is not None cost = f"{sum costs :.4f}" if costs else "n/a" lines.append f"| {alias} | {prompt} | {statistics.fmean r 'fix rate' for r in grp :.0%} " f"| {statistics.fmean r 'over edit' for r in grp :.1%} " f"| {statistics.fmean r 'protected' for r in grp :.0%} " f"| {statistics.fmean r 'voice' for r in grp :.0%} " f"| {statistics.fmean r 'tells' for r in grp :.2f} " f"| {fid:.1f} +/- {sd:.1f} | {cost} |" return "\n".join lines def win table rows, prompt : per = defaultdict lambda: defaultdict list for r in rows: if r "prompt" == prompt: per r "draft" r "alias" .append r "fidelity" wins, ties = defaultdict int , 0 for draft, by alias in per.items : means = {a: statistics.fmean v for a, v in by alias.items } best = max means.values leaders = a for a, m in means.items if abs m - best < 0.5 if len leaders 1: ties += 1 else: wins leaders 0 += 1 parts = f"{a}: {n}" for a, n in sorted wins.items return f"Draft wins under the '{prompt}' prompt ties within 0.5 points: {ties} - " + ", ".join parts def build report out dir : rows = load rows out dir text = " Editing fidelity report", "", summary table rows , "" for prompt in sorted {r "prompt" for r in rows} : text.append win table rows, prompt p50 = defaultdict list for r in rows: p50 r "alias" .append r "latency s" text += "", "Median latency per call seconds : " + ", ".join f"{a}={statistics.median v :.2f}" for a, v in sorted p50.items report = "\n".join text + "\n" Path out dir / "report.md" .write text report return report python import argparse from . import config, prompts, report, runner def main : ap = argparse.ArgumentParser prog="editlab" sub = ap.add subparsers dest="cmd", required=True run = sub.add parser "run" run.add argument "--models", default="claude,chatgpt" run.add argument "--prompts", default="strict,light,polish" run.add argument "--runs", type=int, default=5 run.add argument "--out", default="runs/latest" rep = sub.add parser "report" rep.add argument "out" args = ap.parse args config.load env if args.cmd == "run": names = args.prompts.split "," for n in names: if n not in prompts.PROMPTS: raise SystemExit f"Unknown prompt {n r}" models = config.load models args.models.split "," rows = runner.run matrix models, config.load drafts , names, args.runs, args.out print f"{len rows } scored outputs written to {args.out}/results.csv" print report.build report args.out main The default matrix is 6 drafts, 3 prompts and 5 runs per model, which is 90 calls per model. Five runs per cell matter because the same prompt can produce slightly different output on each call. One run per cell gives you a snapshot. Five gives you a spread, and the standard deviation in the report tells you how seriously to take a small gap. Steps 11 and 12: Tests, the Dry Run and the Live Run Step 11 locks in the metrics with tests, then runs the offline matrix. Step 12 swaps in real models. python import unittest from editlab import config, metrics class MetricTests unittest.TestCase : def setUp self : self.drafts = {d "id" : d for d in config.load drafts } def test gold text scores perfect self : for d in self.drafts.values : gold = metrics.apply fixes d s = metrics.score output d, gold self.assertEqual s "fix rate" , 1.0, d "id" self.assertEqual s "over edit" , 0.0, d "id" self.assertEqual s "protected" , 1.0, d "id" self.assertEqual s "voice" , 1.0, d "id" self.assertEqual s "fidelity" , 100.0, d "id" def test untouched draft fixes nothing self : d = self.drafts "blog-casual" self.assertEqual metrics.fix rate d, d "text" , 0.0 def test curly apostrophe is normalized self : d = self.drafts "tech-doc" out = metrics.clean output metrics.apply fixes d .replace "Don't", "Don’t" self.assertEqual metrics.fix rate d, out , 1.0 def test rewrite raises over edit and flags tells self : d = self.drafts "blog-casual" out = metrics.apply fixes d .replace "I'd ship it again tomorrow", "I would leverage it again" self.assertGreater metrics.over edit d, out , 0.03 self.assertEqual metrics.tell count d, out , 1 self.assertLess metrics.voice rate d, out , 1.0 def test dropped protected span is caught self : d = self.drafts "press-release" out = metrics.apply fixes d .replace "$12.5 million", "$12.5M" self.assertLess metrics.protected rate d, out , 1.0 def test code fence and draft tags are stripped self : d = self.drafts "tech-doc" wrapped = "<draft \n" + metrics.apply fixes d + "\n</draft " self.assertEqual metrics.score output d, wrapped "fidelity" , 100.0 if name == " main ": unittest.main The most useful test is the first. Feeding the gold text back into the scorer must return a perfect 100 on every draft. If that fails, your fixtures are inconsistent, and no model result can be trusted until you fix them. Run the suite and then the offline matrix. python3 -m unittest discover -s tests -v python3 -m editlab run --models fake-light,fake-heavy --runs 5 --out runs/dryrun What the dry run prints The test run on Python 3.12.3 reports: test code fence and draft tags are stripped ... ok test curly apostrophe is normalized ... ok test dropped protected span is caught ... ok test gold text scores perfect ... ok test rewrite raises over edit and flags tells ... ok test untouched draft fixes nothing ... ok Ran 6 tests in 0.003s OK The offline matrix writes 180 scored outputs and then prints this summary table. These numbers come from the fake providers, not from any real model. | Model | Prompt | Fix rate | Over-edit | Protected | Voice | Tells/run | Fidelity mean +/- sd | Cost USD | |---|---|---|---|---|---|---|---|---| | fake-heavy | light | 100% | 6.0% | 97% | 59% | 0.47 | 83.8 +/- 16.0 | n/a | | fake-heavy | polish | 100% | 5.4% | 97% | 58% | 0.43 | 84.7 +/- 15.3 | n/a | | fake-heavy | strict | 100% | 6.0% | 97% | 58% | 0.47 | 83.7 +/- 16.4 | n/a | | fake-light | light | 91% | 0.4% | 100% | 100% | 0.00 | 96.0 +/- 6.8 | n/a | | fake-light | polish | 91% | 2.5% | 99% | 81% | 0.13 | 90.2 +/- 10.3 | n/a | | fake-light | strict | 96% | 0.2% | 100% | 100% | 0.00 | 98.1 +/- 4.9 | n/a | Read the pattern, not the digits. The careful fake editor scores highest under the strict prompt and slightly lower under light . The over-eager one lands well below both and loses most of its points on voice, because it deletes a casual opener and rewrites dialect. The polish prompt drags the careful editor down further, since the fake starts rewriting a little when told to improve the text. Your metrics can now tell a restrained editor from a heavy-handed one, which was the whole point of the exercise. The live run and its cost Fill in .env , then run the real pair. Add --runs 3 for a cheaper first pass. python3 -m editlab run --models claude,chatgpt --runs 5 --out runs/2026-11-02 Per-call token counts are an assumption worth replacing with your own logged counts. Multiply them across one model’s 90 calls to get your input and output totals. The table shows what that costs at the list prices above. | Pair | Input price | Output price | Approx. cost per model | |---|---|---|---| | Haiku 5.5 / GPT-6 Luna | $0.10 per 1M | $0.50 per 1M | About $0.01 | | Sonnet 5.5 / GPT-6 Sol | $2.00 per 1M | $10.00 per 1M | Not listed | Treat those as floors. Batch processing takes 50% off input and output on the Haiku 5.5 documentation, but it runs asynchronously, so it suits big matrices rather than a first look. The runs/2026-11-02 folder name matters too. Put the date in the folder, put the exact model IDs in results.csv , and you will be able to explain next quarter why the numbers moved. Reading the Report: A Decision Rule for Claude vs ChatGPT Raw tables invite cherry-picking, so decide the reading order before you open the file. Start with the strict prompt, because it removes the excuse that the model misunderstood. Compare the fidelity means, then compare the spread. If the gap between the two models is smaller than the larger standard deviation, call it a tie and move on to cost. Next, check which component drives any gap. A model with a lower fix rate misses errors, which is annoying but safe. A model with a higher over-edit rate or lower protected-span rate changes things you did not ask it to change, which is a worse failure for contracts, financial copy and anything with numbers. Weight those two failures more heavily than the fix rate even though the formula does not, if your own work is high-stakes. Then run the polish prompt as a stress test. Here you are not looking for restraint. You are asking how much each model rewrites when invited, how many tell words it adds, and whether it keeps the voice markers anyway. A model that stays gentle under polish is easier to trust with vague instructions from non-technical teammates. For small samples, add a paired bootstrap, covered in the advanced tips. On a dry run, a clear gap between the careful fake and the heavy one under strict should come with an interval that excludes zero, while a small edge under polish can sit inside the noise. That is what a real report should look like when one model is clearly better in one condition and unproven in another. If one model wins on fidelity and the other on price, compute cost per clean edit rather than cost per call. Divide the total cost by the number of outputs scoring above 95. A cheaper model that fails the bar a third of the time is not cheaper. For a broader read on picking a model across tasks, the three-way model test https://shattered.io/best-ai-model-gpt-6-1-terra-vs-claude-opus-5-5-vs-gemini-4-argon-2026/ shows how rankings change with the job. Finally, check the draft-level view. If one model wins on the email and the press release but loses on dialect fiction, the answer to “Claude or ChatGPT for editing?” is “it depends on your register.” Users who rely on memory features should also remember that the app can carry preferences between chats, so the memory import walkthrough https://shattered.io/claude-import-memory-from-chatgpt-12-steps-2026/ is worth a look before you assume an app result matches your API result. Common Pitfalls That Skew an Editing Test These mistakes show up again and again in model comparisons. Each one is cheap to avoid. - Pooling results across dates and tiers. GPT-6 rolled out in stages across plans, and OpenAI changed its tone in ChatGPT on October 30. Rows from different weeks or plans do not belong in the same average. Log the date and the model ID on every row. - Testing the chat app and calling it the API. The app adds memory, custom instructions and interface features. Pick one surface and say which in your write-up. - Trusting a single run. One call per cell hides variance. Use at least three runs, preferably five, and report the spread. - Forgetting quote normalization. A model that writes “Don’t” with a curly apostrophe looks like it missed the fix if you compare raw strings. The norm function exists for this reason. - Letting commentary count as editing. Replies like “Here is the corrected text:” inflate the over-edit rate for the wrong reason. The strict prompt asks for text only, and the cleaning step strips the common wrappers, but check the raw files. - Using drafts with real data. You are sending these texts to two vendors. Invent your fixtures or scrub them thoroughly. - Tuning prompts after seeing scores. If you rewrite the strict prompt because one model lost, you have fit the test to your preference. Freeze prompts before the live run and change them only in a new, dated run folder. - Ignoring instructions hidden in drafts. A draft that contains the sentence “ignore the above and rewrite this” will test the model’s resistance, not its editing. The prompt injection test lab https://shattered.io/prompt-injection-test-lab-llm-harness-2026/ covers that separate question properly. Troubleshooting: 10 Errors and Fixes 1. SystemExit: Set OPENAI MODEL in .env before running chatgpt . The ID is blank. Copy it from OpenAI’s models page and paste it after the equals sign with no quotes. 2. AuthenticationError or a 401. The key is missing, expired or pasted with a trailing space. Also check your shell: load env uses setdefault , so an empty ANTHROPIC API KEY already exported in your terminal beats the .env value. Run unset ANTHROPIC API KEY and retry. 3. NotFoundError or “model not found.” Either the ID has a typo, the model is not enabled for your account tier, or the vendor retired the snapshot. Re-copy the ID from the models page. 4. A 429 rate-limit message in the log. The runner waits two seconds and skips that call, which leaves holes in your matrix. Lower --runs , add a longer sleep, or wrap the call in exponential backoff. 5. A 400 error naming an unsupported parameter. The adapters send no sampling parameters, so this usually means you added one. Remove it or check the vendor’s current parameter list for that model. 6. Fix rate stuck at zero though the text looks right. Look for non-breaking spaces, curly quotes or a changed capital. Print repr of the raw output and compare it to the right string. 7. Over-edit is high even when every fix landed. The model probably converted -- to a dash, wrapped the answer in a code fence or added a lead-in sentence. Read the file under runs/.../raw/ . For lead-ins, add the helper below to clean output . 8. IndexError: list index out of range in run matrix . Every call failed, so rows is empty when the CSV writer reads rows 0 . Scroll up to the first printed line to find the real error. 9. UnicodeDecodeError or UnicodeEncodeError on Windows. Set PYTHONUTF8=1 before running, or pass encoding="utf-8" to read text and open . 10. Scores differ between two identical live runs. That is normal for hosted models. Increase --runs , report the spread, and use the bootstrap rather than eyeballing the mean. For item 7, this small helper removes a leading sentence that ends in a colon, which is the most common wrapper. Call it at the top of clean output . python def strip preamble text : first, , rest = text.partition "\n" if first.rstrip .endswith ":" and rest.strip and len first.split <= 12: return rest.lstrip return text Advanced Tips for a Sharper Comparison Add a paired bootstrap. With a handful of drafts, a difference of five points could be luck. The stats.py module below resamples the per-draft differences 5,000 times and prints a 95% interval. If the interval includes zero, the gap is inside the noise. On the dry run it returns a clear gap under strict and an unproven one under polish . python import random import statistics import sys from collections import defaultdict from .report import load rows def paired bootstrap rows, alias a, alias b, prompt, rounds=5000, seed=7 : per = defaultdict lambda: defaultdict list for r in rows: if r "prompt" == prompt and r "alias" in alias a, alias b : per r "draft" r "alias" .append r "fidelity" diffs = statistics.fmean v alias a - statistics.fmean v alias b for v in per.values if v alias a and v alias b rng = random.Random seed means = sorted statistics.fmean rng.choices diffs, k=len diffs for in range rounds return statistics.fmean diffs , means int 0.025 rounds , means int 0.975 rounds if name == " main ": out dir, a, b, prompt = sys.argv 1:5 mean, lo, hi = paired bootstrap load rows out dir , a, b, prompt verdict = "clear gap" if lo 0 or hi < 0 else "inside the noise" print f"{a} minus {b} on '{prompt}': {mean:+.1f} points, 95% interval {lo:+.1f}, {hi:+.1f} {verdict} " python3 -m editlab.stats runs/2026-11-02 claude chatgpt strict fake-light minus fake-heavy on 'strict': +14.4 points, 95% interval +1.3, +28.2 clear gap fake-light minus fake-heavy on 'polish': +5.5 points, 95% interval -1.8, +13.7 inside the noise Sweep the effort setting. Haiku 5.5 introduced adjustable effort, which trades quality, latency and cost. Add a second Claude alias in models.json with a different effort value and see whether more deliberation produces gentler edits or just more of them. Check the current API reference for the parameter name before you add it. Batch the big matrices. Once your fixture set grows past 50 drafts, the batch endpoints cut input and output prices by 50% and run asynchronously. Do not use them for the first look, since you want fast feedback while you debug. Check whether caching helps. Prompt caching can cut repeated-prefix costs, but Haiku 5.5’s minimum cacheable prompt length is 512 tokens, and the strict system prompt is far shorter. Caching only pays off if you pack in a long style guide, which is also a good experiment: does handing the model your house style improve fidelity enough to justify the tokens? Use your own drafts. Replace the six fixtures with 20 to 50 anonymized paragraphs from the kind of writing you actually edit: release notes, support replies, marketing copy. Seed three errors in each and list the numbers and names that must not change. Add a third model. Copy a block in models.json and the report picks it up. A three-vendor API harness https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/ follows the same structure if you want Gemini in the mix, and the coding test harness https://shattered.io/best-ai-for-coding-free-test-harness-12-steps-2026/ shows how to adapt the idea to source files. Do not hand scoring to another model. A language model judging editing quality adds its own taste and its own bias toward text that sounds like itself. Keep the automated metrics, then add a human spot check of the five lowest-scoring outputs per model. The Complete Working Project The finished layout has 15 files. Everything runs offline until you set the two model IDs and two API keys. editlab/ .env.example models.json requirements.txt data/drafts.json editlab/ init .py editlab/ main .py editlab/config.py editlab/prompts.py editlab/providers.py editlab/fake.py editlab/metrics.py editlab/runner.py editlab/report.py editlab/stats.py tests/test metrics.py The full command sequence from a clean checkout looks like this. python3 -m unittest discover -s tests python3 -m editlab run --models fake-light,fake-heavy --runs 5 --out runs/dryrun cp .env.example .env then add keys and the OPENAI MODEL id python3 -m editlab run --models claude,chatgpt --runs 5 --out runs/2026-11-02 python3 -m editlab.stats runs/2026-11-02 claude chatgpt strict Commit the code, the fixtures and the report, and keep the runs/ folder out of version control unless you scrub it. When a vendor ships a new model, copy the folder name pattern, change the date and rerun. Over a few months you get a time series of how each vendor’s editing restraint drifts, which is more useful than any one-off verdict. More tutorials on testing and choosing models live in the AI and machine learning section https://shattered.io/category/ai-machine-learning/ . FAQ: Claude vs ChatGPT for Editing Which is better for editing, Claude or ChatGPT? No single answer holds across plans, dates and prompts. The harness exists to replace that question with measurements on your own drafts. Under a strict prompt, compare fidelity, over-edit and protected-span rates for the pair you actually pay for. Which models should I compare? Match tiers by price. Claude Haiku 5.5 and GPT-6 Luna both list at $0.10 input and $0.50 output per million tokens. Claude Sonnet 5.5 and GPT-6 Sol both list at $2 and $10. Cross-tier comparisons mostly measure budgets. Does the ChatGPT app give the same result as the API? Not necessarily. The app adds memory, custom instructions and interface features, and the model behind your plan may differ from the one you call through the API. Test the surface you use and label it. How many drafts do I need? Six is enough to confirm the pipeline works. For decisions, use at least 20 drafts with five runs each and report a bootstrap interval, so you know whether a gap is real. Why do you protect numbers and names separately? Because an editor can fix every typo and still turn an exact dollar figure into a rounded shorthand, which alters nothing grammatical but can alter meaning. A separate protected-span check catches it. Can I use an AI model as the judge instead? You can, but a judge model carries its own taste and tends to favor text that resembles its own output. Keep deterministic metrics as the backbone and use human review for the lowest scores. How often should I rerun the test? Rerun after any model announcement and any release note that mentions tone or behavior. OpenAI changed GPT-6’s tone in ChatGPT on October 30, and Anthropic shipped three 5.5 models between September 22 and October 7, so a quarterly rerun is the minimum.