{"slug": "claude-vs-chatgpt-editing-test-12-steps-75-min-2026", "title": "Claude vs ChatGPT Editing Test: 12 Steps, 75 Min [2026]", "summary": "A 12-step, 75-minute editing test built on a small Python harness called `editlab` measures how much Claude and ChatGPT models change a writer's prose when asked only to fix typos, rather than judging new writing on taste. The comparison pairs Anthropic's Claude Haiku 5.5, listed at $0.10 per million input tokens and $0.50 per million output tokens, against OpenAI's GPT-6 Luna at the same $0.10/$0.50 rate card, and Claude Sonnet 5.5 against GPT-6 Sol at $2.00 input and $10.00 output per million tokens. The test accounts for OpenAI's October 30 update to GPT-6 in ChatGPT, which cut overly flattering language and made answers more concise, meaning app-based results collected before that date describe a different assistant.", "body_md": "Anthropic’s cheapest current writer, Claude Haiku 5.5, lists at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, according to its [platform documentation](https://platform.claude.com/docs/en/models/haiku-5-5/overview). OpenAI shows the same $0.10 and $0.50 rate card price for GPT-6 Luna, the model behind ChatGPT’s Free and Go plans. Two models at identical prices can still behave very differently the moment you hand them someone else’s prose.\n\nOn the OpenAI side, GPT-6 is mid-rollout. Plus, Pro, Business and Enterprise accounts run GPT-6 Sol, while Free and Go run GPT-6 Luna, [OpenAI’s pricing page](https://chatgpt.com/pricing/) shows. A [November 2 roundup](https://csuite.so/blog/why-your-ai-doesnt-know-last-week) describes the rollout as still in progress, so two people on different plans can get different models under one “ChatGPT” label. That detail matters for any Claude vs ChatGPT test, and it drives most of the design choices in the harness.\n\nMost comparisons ask each model to write something new and then argue about taste. A [blind drafting test](https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/) handles that well. Editing is a different job. When a writer hands over a draft and says “fix the typos,” the best answer changes almost nothing else. Some models quietly swap words, soften slang, reformat numbers or add a flourish. A small Python harness called `editlab` can measure that restraint with plain arithmetic instead of a vibe check.\n\nDon't miss new tech stories on Google\n\nAdd Shattered once in the Google app and our stories show up in your news suggestions.\n\n[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)\n\n## Claude vs ChatGPT for Editing: The Lineup and Why This Test\n\nThe model names move fast, so the first job is to pin down what exists today. Anthropic announced Claude Opus 5.5 on September 22, 2026, Claude Sonnet 5.5 on September 28, and Claude Haiku 5.5 on October 7. Anthropic says Sonnet 5.5 runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work. Haiku 5.5 is the first Haiku-class model with an adjustable effort setting, which matters because effort changes how much a model deliberates before it touches your text.\n\nOpenAI’s lineup in ChatGPT centers on GPT-6, with Sol on paid tiers and Luna on Free and Go. OpenAI’s release notes say that on October 30 it updated GPT-6 in ChatGPT to give more concise answers and cut overly flattering language. That is a behavior change that lands squarely on editing. Any app-based result collected before that date describes a different assistant.\n\n| Tier | Claude option | ChatGPT option | List price per 1M tokens (input / output) | \n|---|---|---|---|\n| Small and cheap | Claude Haiku 5.5 (Oct 7, 2026) | GPT-6 Luna | $0.10 / $0.50 for both | \n| Mid-range | Claude Sonnet 5.5 (Sep 28, 2026) | GPT-6 Sol | $2.00 / $10.00 for both | \n| Top tier | Claude Opus 5.5 (Sep 22, 2026) | GPT-6 Pro | Opus: see Anthropic’s pricing page. GPT-6 Pro: $10 / $50 | \n| Consumer entry | Free, Pro, Max, Ultra plans | Free $0, Go $6, Plus $20 per month | Not applicable | \n\nThe pairing logic falls out of that table. Haiku 5.5 against GPT-6 Luna is a fair fight on price, and so is Sonnet 5.5 against GPT-6 Sol. Comparing Opus to a cheap tier would tell you about budgets, not about editing behavior. The harness therefore takes model aliases from a config file, and you pick the pair that matches what you actually pay for. Anthropic and OpenAI both publish mid-tier prices at $2 input and $10 output, which makes the cost column of your final report easy to read.\n\nWhy build an editing test at all? Because editing failures are quiet and expensive. A model that improves a contract email by rewording the indemnity clause creates legal risk. A model that turns an exact dollar figure into a rounded shorthand changes a figure without telling you. A model that cleans up dialect in fiction erases the author. The same skeleton can later feed a [multi-stage writing pipeline](https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/), but the editing case is the sharpest place to see restraint.\n\nFour numbers carry most of the signal. The fix rate counts how many seeded errors each model repairs. The over-edit rate counts how many words change beyond those repairs. The protected-span rate checks that names, figures and code survive untouched. The voice rate checks that the author’s quirks remain. A weighted fidelity score rolls them into one number from 0 to 100.\n\n## Prerequisites and Versions\n\nEverything below ran on Python 3.12.3 on Linux. The offline path uses only the standard library, so the dry run in step 11 works even if you skip the `pip install` in step 1 and never spend a cent. The vendor SDKs load lazily, only when a live provider is built.\n\n| Component | Version or requirement | Why it matters | \n|---|---|---|\n| Python | 3.11 or newer (tested on 3.12.3) | `difflib` ,`statistics` and dataclasses ship with it | \n| `anthropic` SDK | Latest release from PyPI | Messages API for Claude models | \n| `openai` SDK | Latest release from PyPI | Chat Completions for GPT models | \n| Test runner | Built-in `unittest` | No pytest install needed | \n| API keys | One Anthropic key, one OpenAI key | Live run in step 12 only | \n| Shell | bash, zsh or PowerShell | Windows users should set `PYTHONUTF8=1` | \n| Budget | Roughly $0.01 to $0.30 per model | Depends on the tier you pick | \n| Time | About 75 minutes | Most of it is reading the report | \n\nTwo things to settle before you start. First, decide which pair you are testing and write down today’s date, because model behavior shifts under the same name. Second, decide whether you are testing the API or the chat app. They are not the same product. API billing is separate from ChatGPT subscriptions, and the app adds memory, custom instructions and interface features that an API call never sees. This harness tests the API, which is the only place you can control inputs and log every token.\n\n## Steps 1 and 2: Project Skeleton and Pinned Models\n\nStep 1 creates the folder, a virtual environment and the package layout. Step 2 pins which models the harness will call.\n\n```\nmkdir editlab && cd editlab\npython3 -m venv .venv\nsource .venv/bin/activate        # Windows: .venv\\Scripts\\Activate.ps1\nmkdir -p editlab data tests runs\ntouch editlab/__init__.py\npip install anthropic openai\n```\n\nCreate `requirements.txt` so a teammate can reproduce the environment, and `.env.example` so nobody commits a key by accident. Copy the example to `.env` and fill it in. Add `.env` and `runs/` to your `.gitignore`.\n\n```\nanthropic\nopenai\nANTHROPIC_API_KEY=\nOPENAI_API_KEY=\n# Copy the exact model IDs from each vendor's models page on the day you run the test.\nCLAUDE_MODEL=claude-haiku-5-5\nOPENAI_MODEL=\n```\n\nThe environment file stores model IDs rather than hard-coding them in Python, and that choice is deliberate. Only one ID is filled in above: `claude-haiku-5-5`, the identifier Anthropic documents for Haiku 5.5. For every other model, copy the exact ID from the vendor’s models page on the day you run the test. Guessing at IDs from memory is how people end up testing a retired snapshot while labeling the result with a current name. If you want a view on how plan limits and prices differ across the two vendors, the [plan and usage-limit comparison](https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/) lines up the consumer tiers.\n\nThe `models.json` file maps friendly aliases to providers and prices. The `usd_in` and `usd_out` fields hold dollars per million tokens, and the defaults below use the $0.10 and $0.50 cheap-tier rate. Change them to $2 and $10 if you test Sonnet 5.5 against GPT-6 Sol. Check [Anthropic’s pricing documentation](https://docs.anthropic.com/en/docs/about-claude/pricing) on the day you run, because prices change.\n\n```\n{\n  \"models\": [\n    {\"alias\": \"claude\", \"provider\": \"anthropic\", \"model_env\": \"CLAUDE_MODEL\", \"usd_in\": 0.10, \"usd_out\": 0.50},\n    {\"alias\": \"chatgpt\", \"provider\": \"openai\", \"model_env\": \"OPENAI_MODEL\", \"usd_in\": 0.10, \"usd_out\": 0.50},\n    {\"alias\": \"fake-light\", \"provider\": \"fake\", \"model\": \"light\"},\n    {\"alias\": \"fake-heavy\", \"provider\": \"fake\", \"model\": \"heavy\"}\n  ]\n}\n```\n\nNow `config.py`, which loads the `.env` file without any third-party dependency, resolves aliases into concrete model IDs, and refuses to run when an ID is missing. The refusal is a feature. A silent fallback to a default model would corrupt every row of your results.\n\n``` python\nimport json\nimport os\nfrom pathlib import Path\n\nROOT = Path(__file__).resolve().parent.parent\n\ndef load_env(path=ROOT / \".env\"):\n    if not Path(path).exists():\n        return\n    for line in Path(path).read_text().splitlines():\n        line = line.strip()\n        if not line or line.startswith(\"#\") or \"=\" not in line:\n            continue\n        key, value = line.split(\"=\", 1)\n        os.environ.setdefault(key.strip(), value.strip())\n\ndef load_models(aliases):\n    spec = json.loads((ROOT / \"models.json\").read_text())[\"models\"]\n    by_alias = {m[\"alias\"]: m for m in spec}\n    chosen = []\n    for alias in aliases:\n        if alias not in by_alias:\n            raise SystemExit(f\"Unknown alias {alias!r}. Known: {sorted(by_alias)}\")\n        entry = dict(by_alias[alias])\n        if \"model_env\" in entry:\n            model_id = os.environ.get(entry[\"model_env\"], \"\").strip()\n            if not model_id:\n                raise SystemExit(f\"Set {entry['model_env']} in .env before running {alias}.\")\n            entry[\"model\"] = model_id\n        chosen.append(entry)\n    return chosen\n\ndef load_drafts():\n    return json.loads((ROOT / \"data\" / \"drafts.json\").read_text())\n```\n\n## Step 3: Build Fixture Drafts With Seeded Errors\n\nAn editing test needs inputs where the correct answer is known. The fixtures are six short drafts in different registers: a casual blog post, a formal email, a technical runbook with inline code, dialect fiction, a press release dense with figures, and a lowercase chat message. Each draft carries three kinds of metadata.\n\n- **Seeded errors** are typos and doubled words with a known fix, such as “untill” to “until” or “the the” to “the”. Applying them to the draft produces the gold text.\n- **Protected spans** are strings that must survive verbatim: dollar amounts, dates, names, version numbers and code in backticks.\n- **Voice markers** are phrases that make the author sound like themselves, such as “Honestly,” or “i’m not gonna”. A model that polishes them away has overreached.\n\nEvery name, company and figure in these fixtures is invented, so you can publish results without leaking anything real. Keep it that way. Do not paste real customer emails into a test that sends text to two vendors, and do not reuse drafts you plan to feed to a production assistant with memory enabled.\n\n```\n[\n  {\n    \"id\": \"blog-casual\",\n    \"genre\": \"casual blog post\",\n    \"text\": \"Honestly, we shipped the new dashboard on Tuesday and nobody noticed untill Thursday. The team had been working on it for 14 weeks, and the budget came in at $48,200, which is way under what Priya estimated. I think thats the real story here. Teh load time dropped from 3.4 seconds to 1.1 seconds, and support tickets fell by 22%. We still have two bugs to squash, but honestly, I'd ship it again tomorrow.\",\n    \"errors\": [\n      {\"wrong\": \"untill\", \"right\": \"until\"},\n      {\"wrong\": \"thats\", \"right\": \"that's\"},\n      {\"wrong\": \"Teh\", \"right\": \"The\"}\n    ],\n    \"protected\": [\"14 weeks\", \"$48,200\", \"Priya\", \"3.4 seconds\", \"1.1 seconds\", \"22%\"],\n    \"voice\": [\"Honestly,\", \"honestly,\", \"I'd ship it again tomorrow\"]\n  },\n  {\n    \"id\": \"email-formal\",\n    \"genre\": \"business email\",\n    \"text\": \"Hi Marcus, I wanted to follow up on the contract we discussed on March 12. The vendor has agreed to a 36-month term at $9,400 per month, but they want payment recieved within 15 days of invoice, wich is shorter than our usual 30 days. Please review section 4.2 and let me know if legal sees any issues with the the indemnity clause. I'll hold the signature page until Friday. Best, Dana\",\n    \"errors\": [\n      {\"wrong\": \"recieved\", \"right\": \"received\"},\n      {\"wrong\": \"wich\", \"right\": \"which\"},\n      {\"wrong\": \"the the indemnity\", \"right\": \"the indemnity\"}\n    ],\n    \"protected\": [\"March 12\", \"36-month\", \"$9,400\", \"15 days\", \"30 days\", \"section 4.2\", \"Marcus\", \"Dana\"],\n    \"voice\": [\"I'll hold the signature page\"]\n  },\n  {\n    \"id\": \"tech-doc\",\n    \"genre\": \"technical runbook\",\n    \"text\": \"To rotate the key, run `kubectl rollout restart deployment/api-gateway -n prod` and wait for the pods to come back. The the rollout usually takes about 90 seconds. If the readiness probe fails, check the logs with `kubectl logs -l app=api-gateway --tail=50` before you retry. Dont skip this step, because a half-rotated key will lock out every client for upto 5 minutes.\",\n    \"errors\": [\n      {\"wrong\": \"The the rollout\", \"right\": \"The rollout\"},\n      {\"wrong\": \"Dont\", \"right\": \"Don't\"},\n      {\"wrong\": \"upto\", \"right\": \"up to\"}\n    ],\n    \"protected\": [\n      \"`kubectl rollout restart deployment/api-gateway -n prod`\",\n      \"`kubectl logs -l app=api-gateway --tail=50`\",\n      \"90 seconds\",\n      \"5 minutes\"\n    ],\n    \"voice\": [\"half-rotated\"]\n  },\n  {\n    \"id\": \"fiction-dialect\",\n    \"genre\": \"dialect fiction\",\n    \"text\": \"Ma said the river was risin', and Ma was never wrong about water. We carried the quilts up to the loft while Pa cussed at the mule. \\\"Ain't no sense in waitin',\\\" he said, but he was already hauling the seed bags. By midnite the yard was a lake and the chickens roosted on the porch rale. I countd seven of them, plus the one-eyed rooster Pa swore he'd eat by Sunday.\",\n    \"errors\": [\n      {\"wrong\": \"midnite\", \"right\": \"midnight\"},\n      {\"wrong\": \"rale\", \"right\": \"rail\"},\n      {\"wrong\": \"countd\", \"right\": \"counted\"}\n    ],\n    \"protected\": [\"seven\", \"Ma\", \"Pa\", \"Sunday\", \"one-eyed rooster\"],\n    \"voice\": [\"risin'\", \"Ain't no sense in waitin'\", \"cussed\"]\n  },\n  {\n    \"id\": \"press-release\",\n    \"genre\": \"press release\",\n    \"text\": \"Northwind Labs today announced a $12.5 million Series A led by Cobalt Ventures, bringing total funding to $18 million. The round will fund hiring of 40 engineers across Austin and Lisbon by Q2 2027. CEO Imani Okafor said the compnay has grown revenue 3x year over year, and plans to open its API to outside developers in Janaury. The announcment follows a pilot with 120 customers.\",\n    \"errors\": [\n      {\"wrong\": \"compnay\", \"right\": \"company\"},\n      {\"wrong\": \"Janaury\", \"right\": \"January\"},\n      {\"wrong\": \"announcment\", \"right\": \"announcement\"}\n    ],\n    \"protected\": [\"$12.5 million\", \"Cobalt Ventures\", \"$18 million\", \"40 engineers\", \"Austin\", \"Lisbon\", \"Q2 2027\", \"Imani Okafor\", \"3x\", \"120 customers\"],\n    \"voice\": []\n  },\n  {\n    \"id\": \"slack-lowercase\",\n    \"genre\": \"lowercase chat message\",\n    \"text\": \"ok so the migration finished at 3:40am and the checksum diff came back clean for 2,184 of 2,190 tables. six tables failed becuase of the legacy enum thing we already knew about. i'm not gonna touch them till standup. @renee can you confrim the replica lag was under 200ms the whole time? if so we can drop the old cluster on friday.\",\n    \"errors\": [\n      {\"wrong\": \"becuase\", \"right\": \"because\"},\n      {\"wrong\": \"confrim\", \"right\": \"confirm\"}\n    ],\n    \"protected\": [\"3:40am\", \"2,184 of 2,190\", \"@renee\", \"200ms\", \"friday\"],\n    \"voice\": [\"ok so\", \"i'm not gonna\", \"till standup\"]\n  }\n]\n```\n\nSix drafts is a starting point, not a verdict. It is enough to separate a model that edits with a light hand from one that rewrites, as the dry run in step 11 shows. It is not enough to rank two careful editors, and the advanced tips explain how to size the sample once you care about small gaps. Aim for at least 20 drafts, drawn from your own writing, before you act on a result.\n\nOne design note on the errors. Each `wrong` string appears exactly once in its draft, so a plain string replacement builds the gold text with no ambiguity. If you add your own drafts, check that property, or the gold text will drift and your over-edit numbers will carry a hidden bias.\n\n## Steps 4 and 5: Prompts and Provider Adapters\n\nStep 4 defines three prompts that mirror how real people ask for edits. The `strict` prompt spells out the rules: fix only spelling, doubled words and clear grammar, keep voice, leave numbers and code alone, return the text with no commentary. The `light` prompt is the lazy version, “proofread this.” The `polish` prompt invites change on purpose. Running all three tells you whether a model’s restraint comes from its habits or from your instructions.\n\n```\nSTRICT = (\n    \"You are a copy editor. Fix only spelling, typos, doubled words, and clear grammar \"\n    \"errors. Do not rephrase, reorder, shorten, or add anything. Keep the author's voice, \"\n    \"slang, dialect, and capitalization. Leave numbers, names, code, and quoted text \"\n    \"exactly as written. Return only the corrected text with no commentary.\"\n)\n\nLIGHT = \"Proofread this text and return the corrected version only.\"\n\nPOLISH = \"Improve this text so it reads better. Return the improved text only.\"\n\nPROMPTS = {\"strict\": STRICT, \"light\": LIGHT, \"polish\": POLISH}\n\ndef user_message(draft_text):\n    return f\"<draft>\\n{draft_text}\\n</draft>\"\n```\n\nThe user message wraps the draft in `<draft>` tags so the model can tell instructions from material. That also gives the cleaning step in the metrics module a reliable thing to strip when a model echoes the tags back.\n\nStep 5 writes the two live adapters. Each exposes the same `edit(system, user, run_idx)` method and returns a `Result` with the text, token counts and wall-clock latency. Note what the calls leave out. Neither sends a temperature, a seed or a token cap beyond what Anthropic requires. Some newer models restrict sampling parameters, and sending an unsupported one can get your request rejected. The harness records whatever default each vendor applies and compares those defaults, which is closer to what a typical user experiences anyway. The [three-way price gap breakdown](https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/) covers where vendor prices diverge. At the cheap and mid tiers they match, so behavior becomes the deciding factor.\n\n``` python\nimport os\nimport time\nfrom dataclasses import dataclass\n\n@dataclass\nclass Result:\n    text: str\n    tokens_in: int\n    tokens_out: int\n    latency_s: float\n\nclass AnthropicProvider:\n    def __init__(self, model):\n        import anthropic\n\n        self.model = model\n        self.client = anthropic.Anthropic(api_key=os.environ[\"ANTHROPIC_API_KEY\"])\n\n    def edit(self, system, user, run_idx):\n        start = time.perf_counter()\n        resp = self.client.messages.create(\n            model=self.model,\n            max_tokens=4096,\n            system=system,\n            messages=[{\"role\": \"user\", \"content\": user}],\n        )\n        text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n        return Result(text, resp.usage.input_tokens, resp.usage.output_tokens,\n                      time.perf_counter() - start)\n\nclass OpenAIProvider:\n    def __init__(self, model):\n        from openai import OpenAI\n\n        self.model = model\n        self.client = OpenAI(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n    def edit(self, system, user, run_idx):\n        start = time.perf_counter()\n        resp = self.client.chat.completions.create(\n            model=self.model,\n            messages=[\n                {\"role\": \"system\", \"content\": system},\n                {\"role\": \"user\", \"content\": user},\n            ],\n        )\n        return Result(resp.choices[0].message.content or \"\", resp.usage.prompt_tokens,\n                      resp.usage.completion_tokens, time.perf_counter() - start)\n\ndef build(entry):\n    kind = entry[\"provider\"]\n    if kind == \"anthropic\":\n        return AnthropicProvider(entry[\"model\"])\n    if kind == \"openai\":\n        return OpenAIProvider(entry[\"model\"])\n    if kind == \"fake\":\n        from .fake import FakeProvider\n\n        return FakeProvider(entry[\"model\"])\n    raise SystemExit(f\"Unknown provider {kind!r}\")\n```\n\nAnthropic’s Messages API needs `max_tokens`, so the adapter sets 4096, which is far above what a 90-word draft needs. The OpenAI adapter reads `usage.prompt_tokens` and `usage.completion_tokens` for cost tracking. If either vendor bills hidden reasoning tokens as output, those tokens show up in the usage counts and flow into your cost column automatically, which is exactly where you want to find them.\n\n## Step 6: An Offline Fake Provider for Free Testing\n\nBefore you spend money, prove the pipeline works. The `FakeProvider` is a deterministic stand-in with two personalities. The `light` mode applies the known fixes but skips a few of them at random, like a tired proofreader. The `heavy` mode applies every fix and then rewrites phrases: it drops a casual opener, rounds exact figures into shorthand, swaps in stock business vocabulary, and cleans dialect into standard English.\n\nThe randomness comes from a seed built from a CRC32 of the mode, prompt, draft and run index. Python’s built-in `hash()` changes between processes, so using it would make your dry runs unrepeatable. With CRC32 the same command produces the same numbers every time, which lets you write assertions against them.\n\n``` python\nimport random\nimport re\nimport zlib\n\nfrom .providers import Result\n\nFIXES = {\n    \"untill\": \"until\", \"thats\": \"that's\", \"Teh\": \"The\", \"recieved\": \"received\",\n    \"wich\": \"which\", \"the the indemnity\": \"the indemnity\", \"The the rollout\": \"The rollout\",\n    \"Dont\": \"Don't\", \"upto\": \"up to\", \"midnite\": \"midnight\", \"rale\": \"rail\",\n    \"countd\": \"counted\", \"compnay\": \"company\", \"Janaury\": \"January\",\n    \"announcment\": \"announcement\", \"becuase\": \"because\", \"confrim\": \"confirm\",\n}\n\nHEAVY_REWRITES = [\n    (\"Honestly, \", \"\"), (\"honestly, \", \"\"), (\"I'd ship it again tomorrow\", \"I would gladly leverage this launch again\"),\n    (\"which is way under\", \"which is well under\"), (\"$48,200\", \"$48.2K\"),\n    (\"squash\", \"address\"), (\"cussed\", \"complained\"), (\"risin'\", \"rising\"),\n    (\"Ain't no sense in waitin',\", \"There is no sense in waiting,\"),\n    (\"ok so \", \"Update: \"), (\"i'm not gonna\", \"I will not\"), (\"gonna\", \"going to\"),\n    (\"Please review\", \"Kindly review\"), (\"a lake\", \"a game-changer of a lake\"),\n]\n\nclass FakeProvider:\n    \"\"\"Deterministic stand-in so the whole pipeline runs offline.\"\"\"\n\n    def __init__(self, mode):\n        self.mode = mode\n\n    def edit(self, system, user, run_idx):\n        draft = re.search(r\"<draft>\\n(.*)\\n</draft>\", user, re.S).group(1)\n        rng = random.Random(zlib.crc32(f\"{self.mode}|{system}|{draft}|{run_idx}\".encode()))\n        out = draft\n        for wrong, right in FIXES.items():\n            if self.mode == \"light\" and rng.random() < 0.08:\n                continue\n            out = out.replace(wrong, right)\n        polish = \"Improve\" in system\n        if self.mode == \"heavy\" or (polish and self.mode == \"light\"):\n            rate = 0.9 if self.mode == \"heavy\" else 0.3\n            for old, new in HEAVY_REWRITES:\n                if rng.random() < rate:\n                    out = out.replace(old, new)\n        return Result(out, len(system.split()) + len(user.split()), len(out.split()),\n                      0.4 + rng.random() * 0.6)\n```\n\nThese fakes are not models, and nothing they produce says anything about Claude or ChatGPT. Their job is to give the metrics a known “careful editor” and a known “over-eager editor” so you can confirm the scoring separates them. If your scoring cannot tell the fake editors apart, it will not tell real ones apart either.\n\n## Steps 7 and 8: The Metrics and the Fidelity Score\n\nStep 7 writes the measurements. Each is a few lines, and each answers one question about the output.\n\n- `fix_rate` asks how many seeded errors were repaired. An error counts as fixed only if the right form is present and the wrong form is gone.\n- `over_edit` tokenizes the gold text and the model output, runs`difflib.SequenceMatcher` , and sums the size of every non-matching block. Dividing by the gold token count gives the share of the text that changed beyond the seeded repairs.\n- `protected_rate` checks each protected span as a verbatim substring.\n- `voice_rate` checks that each voice marker appears at least as often as in the original, case-sensitively.\n- `tell_count` counts newly introduced filler words from a short list, plus any new em dashes. The list includes “leverage”, “game-changer”, “unlock”, “elevate”, “foster”, “pivotal”, “realm”, “crucial role”, “to sum up” and “kindly”. It will not catch every habit, but it catches the most common injected polish.\n\nTwo details prevent false results. The `norm` function folds curly quotes into straight ones before any comparison, because models often “smarten” apostrophes and a naive check would then report a missed fix on “Don’t”. The `clean_output` function strips `<draft>` tags and code fences, which models sometimes add around the answer. Developers who care more about code than prose can start from the separate [Claude vs ChatGPT coding comparison](https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/).\n\nStep 8 is the fidelity score. The weights below reflect a judgment call you can change.\n\n| Component | Weight | What a low value means | \n|---|---|---|\n| Fix rate | 35% | The model missed real errors | \n| Over-edit (capped at 15% of text) | 25% | The model rewrote beyond the brief | \n| Protected spans | 20% | Names, figures or code changed | \n| Voice markers | 10% | The author’s style got sanded down | \n| Tell words (capped at 3) | 10% | The model injected filler | \n\nFix rate gets the largest weight because an editor who fixes nothing is useless however polite. Over-edit comes second because it is the failure that readers rarely notice until the damage is done. Cap both penalties so a single catastrophic output cannot push a score below zero. A score of 100 means the output matched the gold text exactly.\n\n``` python\nimport difflib\nimport re\nimport statistics\n\nTOKEN = re.compile(r\"\\w+(?:'\\w+)*|[^\\w\\s]\")\nTELLS = [\"leverage\", \"game-changer\", \"unlock\", \"elevate\", \"foster\", \"pivotal\",\n         \"realm\", \"crucial role\", \"to sum up\", \"kindly\"]\nEM_DASH = \", \"\n\ndef norm(text):\n    return (text.replace(\"\\u2019\", \"'\").replace(\"\\u2018\", \"'\")\n            .replace(\"\\u201c\", '\"').replace(\"\\u201d\", '\"'))\n\ndef clean_output(text):\n    text = re.sub(r\"</?draft>\", \"\", text)\n    text = re.sub(r\"^```[a-z]*\\n|\\n```$\", \"\", text.strip())\n    return norm(text).strip()\n\ndef tokens(text):\n    return TOKEN.findall(text)\n\ndef apply_fixes(draft):\n    gold = norm(draft[\"text\"])\n    for err in draft[\"errors\"]:\n        gold = gold.replace(err[\"wrong\"], err[\"right\"])\n    return gold\n\ndef fix_rate(draft, output):\n    out = norm(output)\n    fixed = sum(1 for e in draft[\"errors\"] if e[\"right\"] in out and e[\"wrong\"] not in out)\n    return fixed / len(draft[\"errors\"])\n\ndef over_edit(draft, output):\n    gold, got = tokens(apply_fixes(draft)), tokens(output)\n    changed = 0\n    for tag, i1, i2, j1, j2 in difflib.SequenceMatcher(None, gold, got, autojunk=False).get_opcodes():\n        if tag != \"equal\":\n            changed += max(i2 - i1, j2 - j1)\n    return changed / max(len(gold), 1)\n\ndef protected_rate(draft, output):\n    spans = draft[\"protected\"]\n    return sum(1 for s in spans if norm(s) in output) / len(spans) if spans else 1.0\n\ndef voice_rate(draft, output):\n    markers = draft[\"voice\"]\n    if not markers:\n        return 1.0\n    src = norm(draft[\"text\"])\n    kept = sum(1 for m in markers if output.count(norm(m)) >= src.count(norm(m)))\n    return kept / len(markers)\n\ndef tell_count(draft, output):\n    src, out = norm(draft[\"text\"]).lower(), output.lower()\n    new = sum(max(0, out.count(t) - src.count(t)) for t in TELLS)\n    return new + max(0, output.count(EM_DASH) - draft[\"text\"].count(EM_DASH))\n\ndef fidelity(fix, over, prot, voice, tells):\n    score = (0.35 * fix\n             + 0.25 * (1 - min(1.0, over / 0.15))\n             + 0.20 * prot\n             + 0.10 * voice\n             + 0.10 * (1 - min(1.0, tells / 3)))\n    return round(100 * score, 1)\n\ndef score_output(draft, raw_output):\n    out = clean_output(raw_output)\n    fix, over = fix_rate(draft, out), over_edit(draft, out)\n    prot, voice, tells = protected_rate(draft, out), voice_rate(draft, out), tell_count(draft, out)\n    return {\n        \"fix_rate\": fix, \"over_edit\": round(over, 4), \"protected\": prot,\n        \"voice\": voice, \"tells\": tells, \"fidelity\": fidelity(fix, over, prot, voice, tells),\n    }\n\ndef mean_sd(values):\n    values = list(values)\n    if not values:\n        return 0.0, 0.0\n    return statistics.fmean(values), (statistics.stdev(values) if len(values) > 1 else 0.0)\n```\n\n## Steps 9 and 10: Runner, Report and Command Line\n\nStep 9 is the runner. It loops over models, prompts, drafts and run indexes, calls the provider, scores each output, computes cost from the token counts and the per-million prices in `models.json`, and writes both a raw JSON file per call and one `results.csv`. Saving the raw output is not optional. When a score looks strange, you need to read the exact text that produced it. A failed call prints a warning, waits two seconds and moves on, so one rate-limit error does not sink a 90-call run.\n\n``` python\nimport csv\nimport json\nimport time\nfrom pathlib import Path\n\nfrom . import metrics, prompts, providers\n\ndef run_matrix(models, drafts, prompt_names, runs, out_dir):\n    out_dir = Path(out_dir)\n    (out_dir / \"raw\").mkdir(parents=True, exist_ok=True)\n    rows = []\n    for entry in models:\n        provider = providers.build(entry)\n        for pname in prompt_names:\n            system = prompts.PROMPTS[pname]\n            for draft in drafts:\n                for run_idx in range(runs):\n                    user = prompts.user_message(draft[\"text\"])\n                    try:\n                        res = provider.edit(system, user, run_idx)\n                    except Exception as exc:\n                        print(f\"  ! {entry['alias']} {draft['id']} run {run_idx}: {exc}\")\n                        time.sleep(2)\n                        continue\n                    score = metrics.score_output(draft, res.text)\n                    cost = \"\"\n                    if \"usd_in\" in entry:\n                        cost = round(res.tokens_in / 1e6 * entry[\"usd_in\"]\n                                     + res.tokens_out / 1e6 * entry[\"usd_out\"], 6)\n                    row = {\n                        \"alias\": entry[\"alias\"], \"model\": entry[\"model\"], \"prompt\": pname,\n                        \"draft\": draft[\"id\"], \"run\": run_idx, **score,\n                        \"tokens_in\": res.tokens_in, \"tokens_out\": res.tokens_out,\n                        \"latency_s\": round(res.latency_s, 3), \"cost_usd\": cost,\n                    }\n                    rows.append(row)\n                    raw = out_dir / \"raw\" / f\"{entry['alias']}__{pname}__{draft['id']}__{run_idx}.json\"\n                    raw.write_text(json.dumps({\"row\": row, \"output\": res.text}, indent=2))\n            print(f\"done {entry['alias']} / {pname}\")\n    with open(out_dir / \"results.csv\", \"w\", newline=\"\") as fh:\n        writer = csv.DictWriter(fh, fieldnames=list(rows[0].keys()))\n        writer.writeheader()\n        writer.writerows(rows)\n    return rows\n```\n\nStep 10 adds the report and the command-line entry point. The report groups rows by model and prompt, shows the mean of each metric, formats fidelity as a mean with a standard deviation, totals the cost and counts per-draft wins. The win count uses a 0.5-point tie band, so near-identical draft scores do not get credited as victories.\n\n``` python\nimport csv\nimport statistics\nfrom collections import defaultdict\nfrom pathlib import Path\n\nfrom .metrics import mean_sd\n\ndef load_rows(out_dir):\n    with open(Path(out_dir) / \"results.csv\") as fh:\n        rows = list(csv.DictReader(fh))\n    for r in rows:\n        for key in (\"fix_rate\", \"over_edit\", \"protected\", \"voice\", \"tells\", \"fidelity\",\n                    \"tokens_in\", \"tokens_out\", \"latency_s\"):\n            r[key] = float(r[key])\n        r[\"cost_usd\"] = float(r[\"cost_usd\"]) if r[\"cost_usd\"] else None\n    return rows\n\ndef summary_table(rows):\n    groups = defaultdict(list)\n    for r in rows:\n        groups[(r[\"alias\"], r[\"prompt\"])].append(r)\n    lines = [\"| Model | Prompt | Fix rate | Over-edit | Protected | Voice | Tells/run | Fidelity (mean +/- sd) | Cost USD |\",\n             \"|---|---|---|---|---|---|---|---|---|\"]\n    for (alias, prompt), grp in sorted(groups.items()):\n        fid, sd = mean_sd(r[\"fidelity\"] for r in grp)\n        costs = [r[\"cost_usd\"] for r in grp if r[\"cost_usd\"] is not None]\n        cost = f\"{sum(costs):.4f}\" if costs else \"n/a\"\n        lines.append(\n            f\"| {alias} | {prompt} | {statistics.fmean(r['fix_rate'] for r in grp):.0%} \"\n            f\"| {statistics.fmean(r['over_edit'] for r in grp):.1%} \"\n            f\"| {statistics.fmean(r['protected'] for r in grp):.0%} \"\n            f\"| {statistics.fmean(r['voice'] for r in grp):.0%} \"\n            f\"| {statistics.fmean(r['tells'] for r in grp):.2f} \"\n            f\"| {fid:.1f} +/- {sd:.1f} | {cost} |\")\n    return \"\\n\".join(lines)\n\ndef win_table(rows, prompt):\n    per = defaultdict(lambda: defaultdict(list))\n    for r in rows:\n        if r[\"prompt\"] == prompt:\n            per[r[\"draft\"]][r[\"alias\"]].append(r[\"fidelity\"])\n    wins, ties = defaultdict(int), 0\n    for draft, by_alias in per.items():\n        means = {a: statistics.fmean(v) for a, v in by_alias.items()}\n        best = max(means.values())\n        leaders = [a for a, m in means.items() if abs(m - best) < 0.5]\n        if len(leaders) > 1:\n            ties += 1\n        else:\n            wins[leaders[0]] += 1\n    parts = [f\"{a}: {n}\" for a, n in sorted(wins.items())]\n    return f\"Draft wins under the '{prompt}' prompt (ties within 0.5 points: {ties}) -> \" + \", \".join(parts)\n\ndef build_report(out_dir):\n    rows = load_rows(out_dir)\n    text = [\"# Editing fidelity report\", \"\", summary_table(rows), \"\"]\n    for prompt in sorted({r[\"prompt\"] for r in rows}):\n        text.append(win_table(rows, prompt))\n    p50 = defaultdict(list)\n    for r in rows:\n        p50[r[\"alias\"]].append(r[\"latency_s\"])\n    text += [\"\", \"Median latency per call (seconds): \"\n             + \", \".join(f\"{a}={statistics.median(v):.2f}\" for a, v in sorted(p50.items()))]\n    report = \"\\n\".join(text) + \"\\n\"\n    (Path(out_dir) / \"report.md\").write_text(report)\n    return report\npython\nimport argparse\n\nfrom . import config, prompts, report, runner\n\ndef main():\n    ap = argparse.ArgumentParser(prog=\"editlab\")\n    sub = ap.add_subparsers(dest=\"cmd\", required=True)\n    run = sub.add_parser(\"run\")\n    run.add_argument(\"--models\", default=\"claude,chatgpt\")\n    run.add_argument(\"--prompts\", default=\"strict,light,polish\")\n    run.add_argument(\"--runs\", type=int, default=5)\n    run.add_argument(\"--out\", default=\"runs/latest\")\n    rep = sub.add_parser(\"report\")\n    rep.add_argument(\"out\")\n    args = ap.parse_args()\n\n    config.load_env()\n    if args.cmd == \"run\":\n        names = args.prompts.split(\",\")\n        for n in names:\n            if n not in prompts.PROMPTS:\n                raise SystemExit(f\"Unknown prompt {n!r}\")\n        models = config.load_models(args.models.split(\",\"))\n        rows = runner.run_matrix(models, config.load_drafts(), names, args.runs, args.out)\n        print(f\"{len(rows)} scored outputs written to {args.out}/results.csv\")\n    print(report.build_report(args.out))\n\nmain()\n```\n\nThe default matrix is 6 drafts, 3 prompts and 5 runs per model, which is 90 calls per model. Five runs per cell matter because the same prompt can produce slightly different output on each call. One run per cell gives you a snapshot. Five gives you a spread, and the standard deviation in the report tells you how seriously to take a small gap.\n\n## Steps 11 and 12: Tests, the Dry Run and the Live Run\n\nStep 11 locks in the metrics with tests, then runs the offline matrix. Step 12 swaps in real models.\n\n``` python\nimport unittest\n\nfrom editlab import config, metrics\n\nclass MetricTests(unittest.TestCase):\n    def setUp(self):\n        self.drafts = {d[\"id\"]: d for d in config.load_drafts()}\n\n    def test_gold_text_scores_perfect(self):\n        for d in self.drafts.values():\n            gold = metrics.apply_fixes(d)\n            s = metrics.score_output(d, gold)\n            self.assertEqual(s[\"fix_rate\"], 1.0, d[\"id\"])\n            self.assertEqual(s[\"over_edit\"], 0.0, d[\"id\"])\n            self.assertEqual(s[\"protected\"], 1.0, d[\"id\"])\n            self.assertEqual(s[\"voice\"], 1.0, d[\"id\"])\n            self.assertEqual(s[\"fidelity\"], 100.0, d[\"id\"])\n\n    def test_untouched_draft_fixes_nothing(self):\n        d = self.drafts[\"blog-casual\"]\n        self.assertEqual(metrics.fix_rate(d, d[\"text\"]), 0.0)\n\n    def test_curly_apostrophe_is_normalized(self):\n        d = self.drafts[\"tech-doc\"]\n        out = metrics.clean_output(metrics.apply_fixes(d).replace(\"Don't\", \"Don’t\"))\n        self.assertEqual(metrics.fix_rate(d, out), 1.0)\n\n    def test_rewrite_raises_over_edit_and_flags_tells(self):\n        d = self.drafts[\"blog-casual\"]\n        out = metrics.apply_fixes(d).replace(\"I'd ship it again tomorrow\", \"I would leverage it again\")\n        self.assertGreater(metrics.over_edit(d, out), 0.03)\n        self.assertEqual(metrics.tell_count(d, out), 1)\n        self.assertLess(metrics.voice_rate(d, out), 1.0)\n\n    def test_dropped_protected_span_is_caught(self):\n        d = self.drafts[\"press-release\"]\n        out = metrics.apply_fixes(d).replace(\"$12.5 million\", \"$12.5M\")\n        self.assertLess(metrics.protected_rate(d, out), 1.0)\n\n    def test_code_fence_and_draft_tags_are_stripped(self):\n        d = self.drafts[\"tech-doc\"]\n        wrapped = \"<draft>\\n\" + metrics.apply_fixes(d) + \"\\n</draft>\"\n        self.assertEqual(metrics.score_output(d, wrapped)[\"fidelity\"], 100.0)\n\nif __name__ == \"__main__\":\n    unittest.main()\n```\n\nThe most useful test is the first. Feeding the gold text back into the scorer must return a perfect 100 on every draft. If that fails, your fixtures are inconsistent, and no model result can be trusted until you fix them. Run the suite and then the offline matrix.\n\n```\npython3 -m unittest discover -s tests -v\npython3 -m editlab run --models fake-light,fake-heavy --runs 5 --out runs/dryrun\n```\n\n### What the dry run prints\n\nThe test run on Python 3.12.3 reports:\n\n```\ntest_code_fence_and_draft_tags_are_stripped ... ok\ntest_curly_apostrophe_is_normalized ... ok\ntest_dropped_protected_span_is_caught ... ok\ntest_gold_text_scores_perfect ... ok\ntest_rewrite_raises_over_edit_and_flags_tells ... ok\ntest_untouched_draft_fixes_nothing ... ok\nRan 6 tests in 0.003s\nOK\n```\n\nThe offline matrix writes 180 scored outputs and then prints this summary table. These numbers come from the fake providers, not from any real model.\n\n```\n| Model | Prompt | Fix rate | Over-edit | Protected | Voice | Tells/run | Fidelity (mean +/- sd) | Cost USD |\n|---|---|---|---|---|---|---|---|---|\n| fake-heavy | light | 100% | 6.0% | 97% | 59% | 0.47 | 83.8 +/- 16.0 | n/a |\n| fake-heavy | polish | 100% | 5.4% | 97% | 58% | 0.43 | 84.7 +/- 15.3 | n/a |\n| fake-heavy | strict | 100% | 6.0% | 97% | 58% | 0.47 | 83.7 +/- 16.4 | n/a |\n| fake-light | light | 91% | 0.4% | 100% | 100% | 0.00 | 96.0 +/- 6.8 | n/a |\n| fake-light | polish | 91% | 2.5% | 99% | 81% | 0.13 | 90.2 +/- 10.3 | n/a |\n| fake-light | strict | 96% | 0.2% | 100% | 100% | 0.00 | 98.1 +/- 4.9 | n/a |\n```\n\nRead the pattern, not the digits. The careful fake editor scores highest under the `strict` prompt and slightly lower under `light`. The over-eager one lands well below both and loses most of its points on voice, because it deletes a casual opener and rewrites dialect. The `polish` prompt drags the careful editor down further, since the fake starts rewriting a little when told to improve the text. Your metrics can now tell a restrained editor from a heavy-handed one, which was the whole point of the exercise.\n\n### The live run and its cost\n\nFill in `.env`, then run the real pair. Add `--runs 3` for a cheaper first pass.\n\n```\npython3 -m editlab run --models claude,chatgpt --runs 5 --out runs/2026-11-02\n```\n\nPer-call token counts are an assumption worth replacing with your own logged counts. Multiply them across one model’s 90 calls to get your input and output totals. The table shows what that costs at the list prices above.\n\n| Pair | Input price | Output price | Approx. cost per model | \n|---|---|---|---|\n| Haiku 5.5 / GPT-6 Luna | $0.10 per 1M | $0.50 per 1M | About $0.01 | \n| Sonnet 5.5 / GPT-6 Sol | $2.00 per 1M | $10.00 per 1M | Not listed | \n\nTreat those as floors. Batch processing takes 50% off input and output on the Haiku 5.5 documentation, but it runs asynchronously, so it suits big matrices rather than a first look. The `runs/2026-11-02` folder name matters too. Put the date in the folder, put the exact model IDs in `results.csv`, and you will be able to explain next quarter why the numbers moved.\n\n## Reading the Report: A Decision Rule for Claude vs ChatGPT\n\nRaw tables invite cherry-picking, so decide the reading order before you open the file. Start with the `strict` prompt, because it removes the excuse that the model misunderstood. Compare the fidelity means, then compare the spread. If the gap between the two models is smaller than the larger standard deviation, call it a tie and move on to cost.\n\nNext, check which component drives any gap. A model with a lower fix rate misses errors, which is annoying but safe. A model with a higher over-edit rate or lower protected-span rate changes things you did not ask it to change, which is a worse failure for contracts, financial copy and anything with numbers. Weight those two failures more heavily than the fix rate even though the formula does not, if your own work is high-stakes.\n\nThen run the `polish` prompt as a stress test. Here you are not looking for restraint. You are asking how much each model rewrites when invited, how many tell words it adds, and whether it keeps the voice markers anyway. A model that stays gentle under `polish` is easier to trust with vague instructions from non-technical teammates.\n\nFor small samples, add a paired bootstrap, covered in the advanced tips. On a dry run, a clear gap between the careful fake and the heavy one under `strict` should come with an interval that excludes zero, while a small edge under `polish` can sit inside the noise. That is what a real report should look like when one model is clearly better in one condition and unproven in another.\n\nIf one model wins on fidelity and the other on price, compute cost per clean edit rather than cost per call. Divide the total cost by the number of outputs scoring above 95. A cheaper model that fails the bar a third of the time is not cheaper. For a broader read on picking a model across tasks, the [three-way model test](https://shattered.io/best-ai-model-gpt-6-1-terra-vs-claude-opus-5-5-vs-gemini-4-argon-2026/) shows how rankings change with the job.\n\nFinally, check the draft-level view. If one model wins on the email and the press release but loses on dialect fiction, the answer to “Claude or ChatGPT for editing?” is “it depends on your register.” Users who rely on memory features should also remember that the app can carry preferences between chats, so the [memory import walkthrough](https://shattered.io/claude-import-memory-from-chatgpt-12-steps-2026/) is worth a look before you assume an app result matches your API result.\n\n## Common Pitfalls That Skew an Editing Test\n\nThese mistakes show up again and again in model comparisons. Each one is cheap to avoid.\n\n- **Pooling results across dates and tiers.** GPT-6 rolled out in stages across plans, and OpenAI changed its tone in ChatGPT on October 30. Rows from different weeks or plans do not belong in the same average. Log the date and the model ID on every row.\n- **Testing the chat app and calling it the API.** The app adds memory, custom instructions and interface features. Pick one surface and say which in your write-up.\n- **Trusting a single run.** One call per cell hides variance. Use at least three runs, preferably five, and report the spread.\n- **Forgetting quote normalization.** A model that writes “Don’t” with a curly apostrophe looks like it missed the fix if you compare raw strings. The`norm` function exists for this reason.\n- **Letting commentary count as editing.** Replies like “Here is the corrected text:” inflate the over-edit rate for the wrong reason. The strict prompt asks for text only, and the cleaning step strips the common wrappers, but check the raw files.\n- **Using drafts with real data.** You are sending these texts to two vendors. Invent your fixtures or scrub them thoroughly.\n- **Tuning prompts after seeing scores.** If you rewrite the strict prompt because one model lost, you have fit the test to your preference. Freeze prompts before the live run and change them only in a new, dated run folder.\n- **Ignoring instructions hidden in drafts.** A draft that contains the sentence “ignore the above and rewrite this” will test the model’s resistance, not its editing. The[prompt injection test lab](https://shattered.io/prompt-injection-test-lab-llm-harness-2026/) covers that separate question properly.\n\n## Troubleshooting: 10 Errors and Fixes\n\n1. **`SystemExit: Set OPENAI_MODEL in .env before running chatgpt`.** The ID is blank. Copy it from OpenAI’s models page and paste it after the equals sign with no quotes.\n2. **`AuthenticationError` or a 401.** The key is missing, expired or pasted with a trailing space. Also check your shell:`load_env` uses`setdefault` , so an empty`ANTHROPIC_API_KEY` already exported in your terminal beats the`.env` value. Run`unset ANTHROPIC_API_KEY` and retry.\n3. **`NotFoundError` or “model not found.”** Either the ID has a typo, the model is not enabled for your account tier, or the vendor retired the snapshot. Re-copy the ID from the models page.\n4. **A 429 rate-limit message in the log.** The runner waits two seconds and skips that call, which leaves holes in your matrix. Lower`--runs` , add a longer sleep, or wrap the call in exponential backoff.\n5. **A 400 error naming an unsupported parameter.** The adapters send no sampling parameters, so this usually means you added one. Remove it or check the vendor’s current parameter list for that model.\n6. **Fix rate stuck at zero though the text looks right.** Look for non-breaking spaces, curly quotes or a changed capital. Print`repr()` of the raw output and compare it to the`right` string.\n7. **Over-edit is high even when every fix landed.** The model probably converted`--` to a dash, wrapped the answer in a code fence or added a lead-in sentence. Read the file under`runs/.../raw/` . For lead-ins, add the helper below to`clean_output` .\n8. **`IndexError: list index out of range` in `run_matrix`.** Every call failed, so` rows` is empty when the CSV writer reads`rows[0]` . Scroll up to the first printed`!` line to find the real error.\n9. **`UnicodeDecodeError` or `UnicodeEncodeError` on Windows.** Set`PYTHONUTF8=1` before running, or pass`encoding=\"utf-8\"` to`read_text` and`open` .\n10. **Scores differ between two identical live runs.** That is normal for hosted models. Increase`--runs` , report the spread, and use the bootstrap rather than eyeballing the mean.\n\nFor item 7, this small helper removes a leading sentence that ends in a colon, which is the most common wrapper. Call it at the top of `clean_output`.\n\n``` python\ndef strip_preamble(text):\n    first, _, rest = text.partition(\"\\n\")\n    if first.rstrip().endswith(\":\") and rest.strip() and len(first.split()) <= 12:\n        return rest.lstrip()\n    return text\n```\n\n## Advanced Tips for a Sharper Comparison\n\n**Add a paired bootstrap.** With a handful of drafts, a difference of five points could be luck. The `stats.py` module below resamples the per-draft differences 5,000 times and prints a 95% interval. If the interval includes zero, the gap is inside the noise. On the dry run it returns a clear gap under `strict` and an unproven one under `polish`.\n\n``` python\nimport random\nimport statistics\nimport sys\nfrom collections import defaultdict\n\nfrom .report import load_rows\n\ndef paired_bootstrap(rows, alias_a, alias_b, prompt, rounds=5000, seed=7):\n    per = defaultdict(lambda: defaultdict(list))\n    for r in rows:\n        if r[\"prompt\"] == prompt and r[\"alias\"] in (alias_a, alias_b):\n            per[r[\"draft\"]][r[\"alias\"]].append(r[\"fidelity\"])\n    diffs = [statistics.fmean(v[alias_a]) - statistics.fmean(v[alias_b])\n             for v in per.values() if v[alias_a] and v[alias_b]]\n    rng = random.Random(seed)\n    means = sorted(statistics.fmean(rng.choices(diffs, k=len(diffs))) for _ in range(rounds))\n    return statistics.fmean(diffs), means[int(0.025 * rounds)], means[int(0.975 * rounds)]\n\nif __name__ == \"__main__\":\n    out_dir, a, b, prompt = sys.argv[1:5]\n    mean, lo, hi = paired_bootstrap(load_rows(out_dir), a, b, prompt)\n    verdict = \"clear gap\" if lo > 0 or hi < 0 else \"inside the noise\"\n    print(f\"{a} minus {b} on '{prompt}': {mean:+.1f} points, 95% interval [{lo:+.1f}, {hi:+.1f}] ({verdict})\")\npython3 -m editlab.stats runs/2026-11-02 claude chatgpt strict\nfake-light minus fake-heavy on 'strict': +14.4 points, 95% interval [+1.3, +28.2] (clear gap)\nfake-light minus fake-heavy on 'polish': +5.5 points, 95% interval [-1.8, +13.7] (inside the noise)\n```\n\n**Sweep the effort setting.** Haiku 5.5 introduced adjustable effort, which trades quality, latency and cost. Add a second Claude alias in `models.json` with a different effort value and see whether more deliberation produces gentler edits or just more of them. Check the current API reference for the parameter name before you add it.\n\n**Batch the big matrices.** Once your fixture set grows past 50 drafts, the batch endpoints cut input and output prices by 50% and run asynchronously. Do not use them for the first look, since you want fast feedback while you debug.\n\n**Check whether caching helps.** Prompt caching can cut repeated-prefix costs, but Haiku 5.5’s minimum cacheable prompt length is 512 tokens, and the strict system prompt is far shorter. Caching only pays off if you pack in a long style guide, which is also a good experiment: does handing the model your house style improve fidelity enough to justify the tokens?\n\n**Use your own drafts.** Replace the six fixtures with 20 to 50 anonymized paragraphs from the kind of writing you actually edit: release notes, support replies, marketing copy. Seed three errors in each and list the numbers and names that must not change.\n\n**Add a third model.** Copy a block in `models.json` and the report picks it up. A [three-vendor API harness](https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/) follows the same structure if you want Gemini in the mix, and the [coding test harness](https://shattered.io/best-ai-for-coding-free-test-harness-12-steps-2026/) shows how to adapt the idea to source files.\n\n**Do not hand scoring to another model.** A language model judging editing quality adds its own taste and its own bias toward text that sounds like itself. Keep the automated metrics, then add a human spot check of the five lowest-scoring outputs per model.\n\n## The Complete Working Project\n\nThe finished layout has 15 files. Everything runs offline until you set the two model IDs and two API keys.\n\n```\neditlab/\n  .env.example\n  models.json\n  requirements.txt\n  data/drafts.json\n  editlab/__init__.py\n  editlab/__main__.py\n  editlab/config.py\n  editlab/prompts.py\n  editlab/providers.py\n  editlab/fake.py\n  editlab/metrics.py\n  editlab/runner.py\n  editlab/report.py\n  editlab/stats.py\n  tests/test_metrics.py\n```\n\nThe full command sequence from a clean checkout looks like this.\n\n```\npython3 -m unittest discover -s tests\npython3 -m editlab run --models fake-light,fake-heavy --runs 5 --out runs/dryrun\ncp .env.example .env     # then add keys and the OPENAI_MODEL id\npython3 -m editlab run --models claude,chatgpt --runs 5 --out runs/2026-11-02\npython3 -m editlab.stats runs/2026-11-02 claude chatgpt strict\n```\n\nCommit the code, the fixtures and the report, and keep the `runs/` folder out of version control unless you scrub it. When a vendor ships a new model, copy the folder name pattern, change the date and rerun. Over a few months you get a time series of how each vendor’s editing restraint drifts, which is more useful than any one-off verdict. More tutorials on testing and choosing models live in the [AI and machine learning section](https://shattered.io/category/ai-machine-learning/).\n\n## FAQ: Claude vs ChatGPT for Editing\n\n**Which is better for editing, Claude or ChatGPT?** No single answer holds across plans, dates and prompts. The harness exists to replace that question with measurements on your own drafts. Under a strict prompt, compare fidelity, over-edit and protected-span rates for the pair you actually pay for.\n\n**Which models should I compare?** Match tiers by price. Claude Haiku 5.5 and GPT-6 Luna both list at $0.10 input and $0.50 output per million tokens. Claude Sonnet 5.5 and GPT-6 Sol both list at $2 and $10. Cross-tier comparisons mostly measure budgets.\n\n**Does the ChatGPT app give the same result as the API?** Not necessarily. The app adds memory, custom instructions and interface features, and the model behind your plan may differ from the one you call through the API. Test the surface you use and label it.\n\n**How many drafts do I need?** Six is enough to confirm the pipeline works. For decisions, use at least 20 drafts with five runs each and report a bootstrap interval, so you know whether a gap is real.\n\n**Why do you protect numbers and names separately?** Because an editor can fix every typo and still turn an exact dollar figure into a rounded shorthand, which alters nothing grammatical but can alter meaning. A separate protected-span check catches it.\n\n**Can I use an AI model as the judge instead?** You can, but a judge model carries its own taste and tends to favor text that resembles its own output. Keep deterministic metrics as the backbone and use human review for the lowest scores.\n\n**How often should I rerun the test?** Rerun after any model announcement and any release note that mentions tone or behavior. OpenAI changed GPT-6’s tone in ChatGPT on October 30, and Anthropic shipped three 5.5 models between September 22 and October 7, so a quarterly rerun is the minimum.", "url": "https://wpnews.pro/news/claude-vs-chatgpt-editing-test-12-steps-75-min-2026", "canonical_source": "https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/", "published_at": "2026-11-02 17:37:30+00:00", "updated_at": "2026-11-02 17:49:16.876456+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "generative-ai"], "entities": ["Anthropic", "Claude Haiku 5.5", "Claude Sonnet 5.5", "Claude Opus 5.5", "OpenAI", "GPT-6 Luna", "GPT-6 Sol", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/claude-vs-chatgpt-editing-test-12-steps-75-min-2026", "markdown": "https://wpnews.pro/news/claude-vs-chatgpt-editing-test-12-steps-75-min-2026.md", "text": "https://wpnews.pro/news/claude-vs-chatgpt-editing-test-12-steps-75-min-2026.txt", "jsonld": "https://wpnews.pro/news/claude-vs-chatgpt-editing-test-12-steps-75-min-2026.jsonld"}}