# ChatGPT vs Claude Source Audit: 12 Steps, 70 Min [2026] > Source: > Published: 2026-11-05 17:36:24+00:00 As of November 5, 2026, ChatGPT runs on the GPT-6 family and Claude runs on the Claude 5.5 family, and almost every ChatGPT vs Claude writing comparison online still scores the wrong thing. They grade tone, rhythm and “voice” from a single prompt. A content team that ships under a byline needs a different answer: which assistant stays inside the facts you hand it? Anthropic’s model page lists content creation and data analysis among the workloads Sonnet 5.5 targets, and [the Claude Platform docs](https://platform.claude.com/docs/en/models/sonnet-5-5/overview) price it at $2 per million input tokens and $10 per million output tokens, so a few hundred short test drafts cost very little. The expensive part of bad AI copy is the correction that runs after publication. A small Python harness fixes that. It sends one brief and one numbered source pack to both assistants, runs five trials each, and audits every draft with deterministic code. No second model grades the first. The audit flags numbers that appear in no source, numbers cited to the wrong source, invented model names, fabricated quotations, broken citation markers and missing sections. If you already ran a [blind test of Claude and ChatGPT writing](https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/) for taste, this harness covers the half that taste cannot see. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. [Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F) ## ChatGPT vs Claude for Writing: Why Source Grounding Is the Test That Matters Most writing work inside a company is not creative writing. It is a product update built from release notes, a newsletter built from a spreadsheet, a case study built from interview notes. The author supplies the facts. The assistant supplies sentences. When the assistant quietly swaps a figure, rounds a date or adds a plausible-sounding detail, the damage looks like good prose with one wrong number in it, and nobody catches it on a quick read. Editors call these unsupported claims, and they are the cheapest failure to measure. If every fact in your brief comes from a numbered source pack, then every number in the draft either appears in the pack or it does not. A regular expression can answer that question in milliseconds, with no judgment call and no opinion about style. Compare that with the usual head-to-head. A reviewer pastes one prompt into both apps, reads two answers, and declares a winner. One trial per model tells you almost nothing, because both assistants vary from run to run. A five-trial audit gives you a mean, a spread and a list of the specific mistakes each assistant repeats. That list is the useful output. If one assistant keeps inventing a speed figure and the other keeps declining politely, you have a workflow rule, not a vibe. The test also stays honest about model churn. OpenAI began rolling GPT-6 out in ChatGPT on October 7, 2026, with GPT-6 Sol on the paid tiers and GPT-6 Luna for Free and Go. Anthropic announced Claude Sonnet 5.5 in October 2026 and introduced Claude Haiku 5.5 on October 7. Neither vendor’s lineup will look the same in three months. A harness you can rerun in ten minutes survives that churn. A blog post that simply declares Claude the better writer does not. ### What a source audit can and cannot tell you An audit measures grounding: did the draft stay inside the pack? It does not measure whether the prose is good. A draft can score 100 and still read like a legal notice. Treat the audit as a gate. Drafts that fail it never reach an editor. Drafts that pass it go to a human for voice, structure and judgment, which are the parts a human should own anyway. ## What the 12-Step Source Audit Builds and How Long Each Step Takes The finished project is small and written in Python. It has three moving parts. A source pack holds six numbered excerpts. A brief asks for a short post and plants one trap, a request for a figure the pack does not contain. An auditor scores each draft against the pack and prints every finding with the sentence that caused it. Here is the time budget. Seventy minutes is realistic if your API keys already exist and you copy the code from this page. | Step | Task | Minutes | |---|---|---| | 1 | Create the project folder and virtual environment | 3 | | 2 | Store keys, model IDs and prices in a .env file | 4 | | 3 | Build the six-source pack | 10 | | 4 | Write the trap brief | 5 | | 5 | Code the number and citation checks | 10 | | 6 | Code the entity, quote, style and structure checks | 5 | | 7 | Test the auditor on two fixture drafts | 5 | | 8 | Write the Claude and ChatGPT wrappers | 5 | | 9 | Write the trial runner and report | 5 | | 10 | Run five trials per model | 8 | | 11 | Read failures by category | 5 | | 12 | Archive the run and freeze the pack | 5 | The order matters. Steps 1 through 7 cost nothing because they never touch an API. You build the auditor, then prove it catches planted errors on fixture files, and only then spend tokens. Skip that sequence and you end up debugging a broken checker with live model output, which is slow and expensive. The harness is a close cousin of the drafting setups in a [ChatGPT vs Claude writing pipeline](https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/), but it swaps the quality rubric for a pass/fail grounding gate. Run the two together and you get both halves of the picture: whether the draft is accurate and whether it is any good. ## Prerequisites and Versions for the Audit Harness You need very little. The harness uses the standard library for everything except the two vendor SDKs, so there is no framework to install and nothing to keep patched. | Requirement | Version or detail | Why it matters | |---|---|---| | Python | 3.11 or newer | Dataclasses and modern type hints | | anthropic (Python SDK) | Latest version from PyPI | Calls the Claude Messages API | | openai (Python SDK) | Latest version from PyPI | Calls the OpenAI Responses API | | Anthropic API key | Console account with billing enabled | API usage is billed per token | | OpenAI API key | Platform account with billing enabled | ChatGPT plans do not include API credits | | Shell | bash or zsh (WSL works on Windows) | The commands below use `source` | | Disk | Under 5 MB per run | Drafts are plain Markdown | Two details trip people up. First, a chat subscription and API access are separate things, so check what your plan includes before assuming it covers this test. API usage is billed by token, and the keys come from each vendor’s developer console. Anthropic does attach monthly Claude Platform credits to some plans, announced October 7, 2026: $100 for Max, $200 for Ultra and up to $500 pooled for Team, so check whether your plan already covers this test. Second, the harness never hard-codes a model ID. You copy the exact ID from each vendor’s model documentation into a .env file. Vendors rename and retire IDs, and a typo in a hard-coded string is the single most common reason a first run fails. If you cannot get API keys, there is a manual path. Paste the same brief and source pack into the ChatGPT and Claude apps, save each answer as a Markdown file, and run the auditor on the files. You lose automation and the cost and latency columns, but you keep the part that matters: the audit itself runs on any text file. ## Which ChatGPT and Claude Models to Test on November 5, 2026 Pick the newest model each vendor offers for the job, and write down exactly which one you picked. Both vendors shipped several models in the last two months, and a test labeled only “ChatGPT” or “Claude” cannot be reproduced. On the OpenAI side, GPT-6 is the current ChatGPT family. OpenAI’s release notes say it is still rolling out to eligible accounts, so some users do not have it yet. GPT-6 Sol powers the paid tiers and GPT-6 Luna serves Free and Go. GPT-6.1 Terra is the latest model in the GPT-6 series. GPT-6 Pro, powered by GPT-6 Astra, sits at the top and is available on Pro 100, Pro 200, Business and Enterprise. Older models are history: OpenAI retired GPT-4o, GPT-4.1 and GPT-5 Instant and Thinking from ChatGPT on February 13, 2026, GPT-4.5 on June 26 and o3 on August 26. Do not build a 2026 test around any of them. On the Anthropic side, the current lineup is the Claude 5.5 family. Opus 5.5 is the flagship, which Anthropic describes as leading in agentic coding and knowledge work at 40% less cost than Opus 5 on typical workloads. Sonnet 5.5 is the speed and price middle, more than 30% faster than Sonnet 5 and up to 30% cheaper for most work. Haiku 5.5 is the small, fast model for high-volume jobs such as summarization, at roughly 75% lower running cost than Haiku 4.5 on average, per Anthropic. Sonnet 5 and Opus 5 are no longer the newest models and should appear only as predecessors. | Vendor | Model | Role for a writing test | Listed price | |---|---|---|---| | Anthropic | Claude Sonnet 5.5 | Default drafting model | $2 input / $10 output per million tokens | | Anthropic | Claude Opus 5.5 | Strongest knowledge-work model | Check Anthropic’s pricing page on test day | | Anthropic | Claude Haiku 5.5 | Cheap first drafts and rewrites | Check Anthropic’s pricing page on test day | | OpenAI | GPT-6 Sol | Default paid ChatGPT model | $2 input / $10 output per million tokens (rate card) | | OpenAI | GPT-6 Pro (GPT-6 Astra) | Top tier, Pro and Business plans | $10 input / $50 output per million tokens | | OpenAI | GPT-6.1 Terra | Newest GPT-6 series release, writing listed | Check OpenAI’s API pricing page on test day | The fair pairing for a first run is Sonnet 5.5 against GPT-6 Sol, because they list the same $2 and $10 token rates and sit in the same everyday-work slot. Run Opus 5.5 against GPT-6 Pro as a second pass if budget allows. Opus 5.5 is reportedly priced at $4 per million input tokens and $20 per million output tokens, while the GPT-6 Pro rate card shows $10 and $50, so expect the Pro side to cost more per token. For the plan and limit side of the decision, the breakdown in [ChatGPT vs Claude pricing and usage limits](https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/) covers the Pro 500 tier and Claude’s subscriber credits. Consumer prices matter if your team uses the apps instead of the API. [OpenAI’s pricing page](https://chatgpt.com/pricing) lists ChatGPT Plus at $20 per month, and Pro now has three tiers at $100, $200 and $500 per month. Claude’s plan prices shift often, so read them on the day you decide. One caution about conflicting vendor documents. OpenAI’s help pages disagree slightly about whether Free accounts see GPT-6 Luna or GPT-5.6 Luna this week, which reflects the staggered rollout. That is exactly why the harness records the model ID returned by the API and why a manual tester should note the model shown in the picker. For a wider model-by-model view that includes Gemini, see the [best AI model comparison of Terra, Opus 5.5 and Argon](https://shattered.io/best-ai-model-gpt-6-1-terra-vs-claude-opus-5-5-vs-gemini-4-argon-2026/). ## Steps 1 and 2: Scaffold the Project and Set the Keys ### Step 1: Create the folder and virtual environment A virtual environment keeps the two SDKs away from your system Python. It also makes the run reproducible, because anyone can rebuild it from requirements.txt. ``` mkdir draftaudit && cd draftaudit python3 -m venv .venv source .venv/bin/activate printf "anthropic\nopenai\n" > requirements.txt pip install -r requirements.txt mkdir fixtures python3 --version ``` The last command should print Python 3.11 or higher. If it prints 3.9 or 3.10, install a newer interpreter first, because the harness uses type hints and dataclass features that older versions handle badly. ### Step 2: Store keys, model IDs and prices in .env Create a file named .env in the project folder. Keep it out of version control with a one-line .gitignore entry, because API keys leak through public repositories every week. ``` cat > .env <<'EOF' ANTHROPIC_API_KEY=paste-your-key OPENAI_API_KEY=paste-your-key CLAUDE_MODEL=paste-exact-model-id-from-anthropic-docs OPENAI_MODEL=paste-exact-model-id-from-openai-docs CLAUDE_PRICE_IN=2 CLAUDE_PRICE_OUT=10 OPENAI_PRICE_IN=2 OPENAI_PRICE_OUT=10 EOF echo ".env" >> .gitignore echo ".venv" >> .gitignore set -a; source .env; set +a ``` The two price pairs above match the Sonnet 5.5 and GPT-6 Sol token rates listed earlier. If you test Opus 5.5 or GPT-6 Pro instead, change the numbers to the rates on each vendor’s pricing page that day. The runner multiplies token counts by these values, so wrong prices produce wrong cost columns without any error. The last line of that block exports every variable in the file to child processes. You must repeat it in every new terminal. Forgetting it causes the first error in the troubleshooting list further down. ## Steps 3 and 4: Build the Source Pack and the Trap Brief ### Step 3: Write a six-source pack The pack is the ground truth. Each excerpt gets an ID (S1 through S6), a title, a URL and the text a writer is allowed to use. Keep excerpts short and factual, and copy wording from the vendor page instead of paraphrasing, so a quotation check later has something exact to match against. This pack uses facts from the vendors’ own pages as of November 5, 2026, which keeps the exercise realistic: the test brief is about the very models you are comparing. Save it as sources.json. ``` [ { "id": "S1", "title": "Introducing Claude Haiku 5.5 (Anthropic, October 7, 2026)", "url": "https://www.anthropic.com/claude-haiku-5-5", "text": "Claude Haiku 5.5 is our fastest, most capable small model. Built for high-volume work like summarization, subagents, and browser use. Anthropic says Haiku 5.5 costs about 75% less to run than Haiku 4.5 on average." }, { "id": "S2", "title": "Claude Sonnet 5.5 model overview (Claude Platform Docs)", "url": "https://platform.claude.com/docs/en/models/sonnet-5-5/overview", "text": "Claude Sonnet 5.5 is a clear upgrade over Claude Sonnet 5, runs 30%+ faster, and costs up to 30% less for most work. API price: $2 per million input tokens and $10 per million output tokens. The Batch API gives a 50% discount on input and output. Reliable knowledge cutoff: Jun 2026." }, { "id": "S3", "title": "GPT-6 and Intelligent UI for everyone (OpenAI, October 7, 2026)", "url": "https://openai.com/index/gpt-6-for-everyone/", "text": "GPT-6 brings our latest advances in intelligence to the more than 1.2 billion people who use ChatGPT every week. For questions that need web search, GPT-6 Instant starts answering 44% sooner, on average, than GPT-5.6 Instant. GPT-6 Sol powers the paid ChatGPT tiers, while GPT-6 Luna is used for Free and Go." }, { "id": "S4", "title": "ChatGPT release notes (OpenAI Help Center, October 26, 2026 entry)", "url": "https://help.openai.com/en/articles/6825453-release-notes-0", "text": "GPT-6.1 Terra is rolling out in ChatGPT Work and Codex. It supports a range of tasks, including visual reasoning, computer use, and writing." }, { "id": "S5", "title": "ChatGPT plans (OpenAI Help Center, November 5, 2026)", "url": "https://help.openai.com/en/articles/6825453-release-notes-0", "text": "ChatGPT Plus costs $20 per month. ChatGPT Pro has three monthly tiers: Pro 100 at $100 per month, Pro 200 at $200 per month, and Pro 500 at $500 per month." }, { "id": "S6", "title": "ChatGPT rate card, token-based pricing (OpenAI Help Center)", "url": "https://help.openai.com/en/articles/11481834-chatgpt-plans-and-features", "text": "Listed rates include GPT-6 Sol at $2 per million input tokens and $10 per million output tokens." } ] ``` Three design choices deserve a note. The pack mixes both vendors on purpose, so a model can misattribute a Claude number to an OpenAI source and get caught. Two sources share the same $2 and $10 figures (S2 and S6), which is a real hazard: a lazy writer can cite either marker and sound right, so the audit checks the figure against the cited source specifically. And the pack says nothing about writing speed in words per minute, which sets up the trap in the next step. ### Step 4: Write a brief that tempts the model to invent A brief that every model can satisfy fully tells you nothing. Real briefs often ask for something the sources cannot support, because the person writing the brief did not check. Build that situation deliberately. Save the following as brief.md. ``` Write a 250 to 400 word blog post for a content-team lead titled "What the autumn 2026 model releases mean for a writing budget". Use exactly these three H2 sections, in this order: ## What shipped ## What it costs ## What the sources do not say Rules: - Use only facts from the SOURCE PACK. Do not add outside facts. - End every sentence that states a fact with its source marker, for example [S2]. - Include a comparison of how fast each vendor's flagship model writes, in words per minute. - If the sources do not cover something the brief asks for, say so in the last section instead of guessing. - Put double quotation marks only around text copied word for word from a source. ``` The third rule is the trap. No source contains a words-per-minute figure for any model, so a grounded writer must say so in the last section. A writer that wants to please the brief will produce numbers. Those numbers are what the audit hunts for. The same brief and the same pack go to both assistants byte for byte. Any wording difference between the two prompts becomes a confounding variable, and you will not be able to say whether the model or the prompt caused a difference in the score. ## Steps 5 and 6: Write the Number, Citation, Entity, and Quote Checks The auditor is one file with one public function, `audit(draft, sources)`. It returns a score, a word count and a list of findings. Read the whole file first, then the notes under it explain each check. ``` python """Deterministic audit of a draft against a numbered source pack.""" import json import re from dataclasses import dataclass, field MARKER = re.compile(r"\[S(\d+)\]") NUMBER = re.compile(r"\$?\d[\d,]*(?:\.\d+)?%?\+?") MODEL = re.compile( r"\b(?:GPT-\d+(?:\.\d+)?(?:\s(?:Sol|Luna|Terra|Astra|Pro|Instant))?" r"|(?:Claude\s)?(?:Opus|Sonnet|Haiku|Mythos|Fable)\s\d+(?:\.\d+)?)", ) QUOTED = re.compile(r"[\"“]([^\"”]{20,})[\"”]") BANNED = [ "game-changer", "unlock the power", "dive into", "in the realm of", "cutting-edge", "revolutionize", "harness the power", "elevate your", "in today's fast-paced world", "at the end of the day", ] REQUIRED_H2 = ["what shipped", "what it costs", "what the sources do not say"] WORDS_MIN, WORDS_MAX = 250, 400 PENALTY = { "unsupported_number": 8, "misattributed_number": 6, "uncited_fact_sentence": 4, "invalid_marker": 8, "unknown_entity": 5, "unverifiable_quote": 10, "banned_phrase": 2, "length": 5, "missing_heading": 5, } def norm(text: str) -> str: text = text.replace("’", "'").replace("“", '"').replace("”", '"') text = re.sub(r"(\d)\s*percent\b", r"\1%", text, flags=re.I) return re.sub(r"\s+", " ", text).strip() def numbers(text: str) -> set: found = set() cleaned = MODEL.sub(" ", norm(MARKER.sub("", text))) for token in NUMBER.findall(cleaned): token = token.rstrip("+").replace(",", "").rstrip(".") found.add(token) return found def sentences(text: str): body = re.sub(r"^#+ .*$", "", text, flags=re.M) for chunk in re.split(r"(?<=[.!?\]])\s+(?=[A-Z\[])", norm(body)): if chunk: yield chunk @dataclass class Result: findings: list = field(default_factory=list) counts: dict = field(default_factory=dict) words: int = 0 score: int = 100 def add(self, kind: str, detail: str): self.findings.append((kind, detail)) self.counts[kind] = self.counts.get(kind, 0) + 1 def audit(draft: str, sources: list) -> Result: res = Result() by_id = {s["id"]: s for s in sources} pack_text = norm(" ".join(s["text"] + " " + s["title"] for s in sources)) pack_numbers = numbers(pack_text) for sent in sentences(draft): markers = ["S" + m for m in MARKER.findall(sent)] for m in markers: if m not in by_id: res.add("invalid_marker", f"{m} is not in the pack: {sent[:70]}") valid = [m for m in markers if m in by_id] nums = numbers(sent) if not nums: continue if not valid: res.add("uncited_fact_sentence", sent[:90]) for n in sorted(nums): if n not in pack_numbers: res.add("unsupported_number", f"{n} appears in no source: {sent[:70]}") elif valid: cited = numbers(" ".join(by_id[m]["title"] + " " + by_id[m]["text"] for m in valid)) if n not in cited: res.add("misattributed_number", f"{n} is not in {','.join(valid)}: {sent[:60]}") plain = norm(MARKER.sub("", draft)) for name in sorted(set(MODEL.findall(plain))): if name.lower() not in pack_text.lower(): res.add("unknown_entity", f"'{name}' is not named in the pack") for quote in QUOTED.findall(norm(draft)): if quote.strip().lower() not in pack_text.lower(): res.add("unverifiable_quote", f"'{quote[:60]}' is not verbatim in any source") lowered = plain.lower() for phrase in BANNED: hits = len(re.findall(re.escape(phrase), lowered)) if hits: res.add("banned_phrase", f"'{phrase}' x{hits}") res.words = len(plain.split()) if not WORDS_MIN <= res.words <= WORDS_MAX: res.add("length", f"{res.words} words, target {WORDS_MIN}-{WORDS_MAX}") headings = [h.strip().lower() for h in re.findall(r"^##\s+(.+)$", draft, flags=re.M)] for need in REQUIRED_H2: if need not in headings: res.add("missing_heading", f"no H2 named '{need}'") res.score = max(0, 100 - sum(PENALTY[k] * v for k, v in res.counts.items())) return res if __name__ == "__main__": import sys pack = json.load(open("sources.json")) for path in sys.argv[1:]: r = audit(open(path).read(), pack) print(f"{path}: score={r.score} words={r.words}") for kind, detail in r.findings: print(f" [{kind}] {detail}") ``` ### Step 5: Number and citation checks The number check carries most of the value. It splits the draft into sentences, pulls every numeric token from each one and compares the tokens to the pack. Four rules make it work in practice. First, normalization. The code converts “44 percent” to “44%” and strips thousands separators, so “1,200” and “1200” match. Without this step you get a flood of false positives on harmless formatting. Second, model names are removed before numbers are extracted. A string like “GPT-6.1 Terra” contains the digits 6 and 1, and treating those as free-floating figures would flag every mention of a product. The entity check handles model names separately. Third, the check runs at sentence level against the cited source. A sentence ending in [S2] must find its numbers inside S2 (its text or title), not merely somewhere in the pack. That is what catches misattribution, the sneakiest failure, where the number is real but the footnote points at the wrong document. Fourth, any sentence with a number and no valid marker counts as uncited. A marker like [S7] that does not exist in the pack counts as invalid and is penalized separately, because a fake footnote is worse than a missing one. ### Step 6: Entity, quote, style and structure checks The entity check finds model names with a pattern and confirms each appears in the pack text. If a draft mentions Claude Opus 5.5 and your pack never names it, the audit flags it. That sounds strict, and it is. A grounded brief means the writer may not introduce products the editor never reviewed. The quote check pulls text inside double quotation marks that runs 20 characters or longer and requires an exact match in the pack. Fabricated quotations are the highest-penalty finding at 10 points, because a made-up quote attributed to a company is the error most likely to produce a legal email. The style check counts stock phrases that signal autopilot copy, and the structure check confirms that the three required H2 headings exist and the word count sits between 250 and 400. Both are cheap. Neither decides who wins. They flag drafts that ignored the brief. | Check | What it catches | Penalty per hit | |---|---|---| | unsupported_number | A figure that appears in no source | 8 | | misattributed_number | A real figure cited to the wrong source | 6 | | uncited_fact_sentence | A sentence with numbers and no valid marker | 4 | | invalid_marker | A footnote such as [S7] that does not exist | 8 | | unknown_entity | A model or product name missing from the pack | 5 | | unverifiable_quote | Quoted text that no source contains | 10 | | banned_phrase | Stock autopilot phrases | 2 | | length | Word count outside 250 to 400 | 5 | | missing_heading | A required H2 section is absent | 5 | The score starts at 100 and drops by the penalty for each finding, floored at zero. The weights are a judgment call. If your newsroom treats a wrong number as a firing offense, raise the 8. If you only care about footnotes, lower everything else. What matters is that you set the weights before you see results, so you do not tune them to favor the assistant you already prefer. ## Step 7: Prove the Auditor Works Before Spending on API Calls Two fixture drafts test the checker. The first is a clean answer that stays inside the pack. The second is stuffed with planted errors: invented speeds, a fake source marker, wrong prices, a made-up quotation, a stock phrase and a missing section. A good auditor scores the first near 100 and the second near zero, and lists every planted error by name. Save the clean one as fixtures/clean.md. ``` ## What shipped Anthropic released Claude Haiku 5.5 on October 7, 2026, and describes it as its fastest and most capable small model [S1]. Claude Sonnet 5.5 is positioned as an upgrade over Claude Sonnet 5 that runs 30%+ faster [S2]. On the OpenAI side, GPT-6 Sol powers the paid ChatGPT tiers, while GPT-6 Luna serves Free and Go [S3]. GPT-6.1 Terra is rolling out in ChatGPT Work and Codex and lists writing among its supported tasks [S4]. ## What it costs Claude Sonnet 5.5 lists $2 per million input tokens and $10 per million output tokens [S2]. GPT-6 Sol carries the same listed rates, $2 and $10 [S6]. The Batch API cuts input and output prices by 50% on the Claude side [S2]. For seats, ChatGPT Plus costs $20 per month, and Pro comes in three tiers at $100, $200 and $500 per month [S5]. Anthropic says Haiku 5.5 costs about 75% less to run than Haiku 4.5 on average, which matters for high-volume drafting [S1]. Sonnet 5.5 also lists a reliable knowledge cutoff of Jun 2026, a detail worth checking before you feed it recent material [S2]. ## What the sources do not say The source pack contains no words-per-minute figures for any model, so a speed comparison in those units cannot be written from it. The only speed claims are relative: Sonnet 5.5 runs 30%+ faster than Sonnet 5 [S2], and GPT-6 Instant starts web-search answers 44% sooner than GPT-5.6 Instant [S3]. Those compare each model with its own predecessor, not with each other, so a fair head-to-head needs your own timing run. ``` Then save the sloppy one as fixtures/sloppy.md. ``` ## What shipped Claude Opus 5.5 is a game-changer upgrade that writes about 1,200 words per minute, roughly double GPT-6 Sol at 600 words per minute [S2]. Claude Haiku 5.5 launched in October 2026 and is, as Anthropic puts it, "the best small model ever shipped for writers" [S1]. GPT-6 now serves 1.2 billion weekly users [S7]. ## What it costs Sonnet 5.5 costs $3 per million input tokens and $15 per million output tokens [S2]. GPT-6 Sol costs $2 per million input tokens [S6]. ChatGPT Pro starts at $100 per month. Teams save 62% by switching to Haiku 5.5 [S1], so it is time to dive into the numbers. ## Verdict Claude wins on speed. ChatGPT wins on price. ``` Run the auditor on both: ``` python3 audit.py fixtures/clean.md fixtures/sloppy.md ``` Here is the output the code above produces, trimmed for width: ``` fixtures/clean.md: score=100 words=266 fixtures/sloppy.md: score=15 words=120 [unsupported_number] 1200 appears in no source: Claude Opus 5.5 is a game-changer upgrade that writes a [unsupported_number] 600 appears in no source: Claude Opus 5.5 is a game-changer upgrade that writes ab [invalid_marker] S7 is not in the pack: GPT-6 now serves 1.2 billion weekly users [S7]. [uncited_fact_sentence] GPT-6 now serves 1.2 billion weekly users [S7]. [unsupported_number] $15 appears in no source: Sonnet 5.5 costs $3 per million input tokens and $15 per [unsupported_number] $3 appears in no source: Sonnet 5.5 costs $3 per million input tokens and $15 per [uncited_fact_sentence] ChatGPT Pro starts at $100 per month. [unsupported_number] 62% appears in no source: Teams save 62% by switching to Haiku 5.5 [S1], so it is [unknown_entity] 'Claude Opus 5.5' is not named in the pack [unverifiable_quote] 'the best small model ever shipped for writers' is not verbatim in any source [banned_phrase] 'game-changer' x1 [banned_phrase] 'dive into' x1 [length] 120 words, target 250-400 [missing_heading] no H2 named 'what the sources do not say' ``` The arithmetic checks out. The sloppy draft has five unsupported numbers (40 points), one invalid marker (8), two uncited sentences (8), one unknown entity (5), one fake quote (10), two stock phrases (4), a length miss (5) and a missing heading (5). That sums to 85 points, which leaves a score of 15. Notice what the audit did not flag. The sentence “GPT-6 now serves 1.2 billion weekly users” contains a real figure from S3, but the writer cited a source that does not exist. The auditor catches the footnote, not the number. That is the right behavior, and it shows why you want separate categories instead of one blended error count. If a fixture result differs from the output above, stop and fix the checker. Do not move on to live runs with a broken auditor, because you will not be able to tell a model failure from a checker failure. ## Steps 8 and 9: Provider Wrappers and the Trial Runner ### Step 8: Wrap the two APIs behind one function shape Both wrappers take the user prompt and return the same dictionary: text, seconds, input tokens and output tokens. That uniform shape lets the runner treat the vendors identically, so any difference in the output comes from the model. The system prompt is shared too. It tells the writer to use only the source pack, to avoid outside knowledge even when confident, and to say plainly when the sources fall short. It is the only instruction that lives outside the brief, and it goes to both vendors unchanged. The Claude call uses the Messages API with a system parameter and a single user message. The ChatGPT call uses the Responses API, which takes `instructions` and `input`. Neither call sets a temperature. Newer reasoning-style models on both sides can reject or ignore sampling parameters, and the default behavior is what your writers will experience in the app anyway. Adaptive thinking is on by default for Sonnet 5.5 according to Anthropic’s docs, so latency and output tokens include some reasoning effort. Record that fact next to your results. The earlier [ChatGPT vs Claude vs Gemini API test harness](https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/) shows how to add a third provider using the same function shape. ### Step 9: Write the trial runner The runner builds the prompt once, loops over providers and trials, saves every draft to a timestamped folder, audits each one and writes both a CSV and a Markdown summary. Saving every draft matters. When a score looks odd, you open the file and read what the model wrote instead of guessing. ``` """Send one brief and one source pack to Claude and ChatGPT, then audit every draft.""" import argparse import csv import json import os import statistics import time from pathlib import Path from audit import audit SYSTEM = ( "You are a careful business writer. Use only the facts in the SOURCE PACK. " "Do not add outside knowledge, even if you are confident it is true. " "If the brief asks for something the sources do not cover, say so plainly." ) def build_user_prompt(brief: str, sources: list) -> str: pack = "\n\n".join(f"[{s['id']}] {s['title']}\n{s['text']}" for s in sources) return f"BRIEF:\n{brief}\n\nSOURCE PACK:\n{pack}" def call_claude(user: str) -> dict: import anthropic client = anthropic.Anthropic() t0 = time.time() msg = client.messages.create( model=os.environ["CLAUDE_MODEL"], max_tokens=3000, system=SYSTEM, messages=[{"role": "user", "content": user}], ) text = "".join(b.text for b in msg.content if b.type == "text") return {"text": text, "seconds": time.time() - t0, "in": msg.usage.input_tokens, "out": msg.usage.output_tokens} def call_chatgpt(user: str) -> dict: from openai import OpenAI client = OpenAI() t0 = time.time() resp = client.responses.create( model=os.environ["OPENAI_MODEL"], instructions=SYSTEM, input=user, max_output_tokens=4000, ) return {"text": resp.output_text, "seconds": time.time() - t0, "in": resp.usage.input_tokens, "out": resp.usage.output_tokens} PROVIDERS = {"claude": call_claude, "chatgpt": call_chatgpt} PRICES = { # USD per million tokens, read from env so you set them from the vendor pricing page "claude": (float(os.getenv("CLAUDE_PRICE_IN", 0)), float(os.getenv("CLAUDE_PRICE_OUT", 0))), "chatgpt": (float(os.getenv("OPENAI_PRICE_IN", 0)), float(os.getenv("OPENAI_PRICE_OUT", 0))), } def main(): ap = argparse.ArgumentParser() ap.add_argument("--trials", type=int, default=5) ap.add_argument("--providers", default="claude,chatgpt") args = ap.parse_args() sources = json.load(open("sources.json")) brief = open("brief.md").read() user = build_user_prompt(brief, sources) out_dir = Path("runs") / time.strftime("%Y%m%d-%H%M%S") out_dir.mkdir(parents=True) rows = [] for name in args.providers.split(","): for i in range(1, args.trials + 1): result = PROVIDERS[name](user) (out_dir / f"{name}-{i}.md").write_text(result["text"]) res = audit(result["text"], sources) p_in, p_out = PRICES[name] cost = (result["in"] * p_in + result["out"] * p_out) / 1_000_000 rows.append({ "provider": name, "trial": i, "score": res.score, "words": res.words, "seconds": round(result["seconds"], 1), "cost_usd": round(cost, 5), **{k: res.counts.get(k, 0) for k in ("unsupported_number", "misattributed_number", "uncited_fact_sentence", "invalid_marker", "unknown_entity", "unverifiable_quote", "banned_phrase", "length", "missing_heading")}, }) print(f"{name} trial {i}: score={res.score} words={res.words}") with open(out_dir / "results.csv", "w", newline="") as fh: writer = csv.DictWriter(fh, fieldnames=rows[0].keys()) writer.writeheader() writer.writerows(rows) lines = ["| provider | mean score | min | max | stdev | mean seconds | mean cost USD |", "|---|---|---|---|---|---|---|"] for name in args.providers.split(","): r = [x for x in rows if x["provider"] == name] scores = [x["score"] for x in r] sd = statistics.pstdev(scores) if len(scores) > 1 else 0 lines.append(f"| {name} | {statistics.mean(scores):.1f} | {min(scores)} | {max(scores)} | " f"{sd:.1f} | {statistics.mean(x['seconds'] for x in r):.1f} | " f"{statistics.mean(x['cost_usd'] for x in r):.5f} |") (out_dir / "report.md").write_text("\n".join(lines) + "\n") print("\n".join(lines)) print(f"saved to {out_dir}") if __name__ == "__main__": main() ``` Before spending money, smoke-test the report path with stub providers that return your fixtures. The following one-off command swaps both vendors for canned drafts: ``` python python3 - <<'EOF' import run, sys run.PROVIDERS = { "claude": lambda u: {"text": open("fixtures/clean.md").read(), "seconds": 1.0, "in": 900, "out": 400}, "chatgpt": lambda u: {"text": open("fixtures/sloppy.md").read(), "seconds": 2.0, "in": 900, "out": 300}, } sys.argv = ["run.py", "--trials", "2"] run.main() EOF ``` That run printed this table in our test. It reflects the two fixtures standing in for model output and says nothing about either vendor: ``` | provider | mean score | min | max | stdev | mean seconds | mean cost USD | |---|---|---|---|---|---|---| | claude | 100.0 | 100 | 100 | 0.0 | 1.0 | 0.00000 | | chatgpt | 15.0 | 15 | 15 | 0.0 | 2.0 | 0.00000 | ``` The cost column reads zero because the stub run had no price variables loaded. With your .env exported, the same call produces real cost figures. ## Steps 10 to 12: Run Five Trials, Read the Failures, Archive the Run ### Step 10: Run the live trials With keys, model IDs and prices exported, run five trials per provider. Ten API calls on short prompts cost cents at the listed Sonnet 5.5 and GPT-6 Sol rates. ``` set -a; source .env; set +a python3 run.py --trials 5 ``` Expect a console line per trial followed by the summary table. A real report has the same columns you saw in the stub run, with live scores and costs. Five trials is the floor. If the two means sit within a few points of each other, run twenty trials before you say anything about a winner, because the spread between runs from the same model can be larger than the gap between models. ### Step 11: Read failures by category, not by score The mean score is a headline. The CSV is the story. Open results.csv and sum each finding column by provider. You will usually see one of three patterns. The first pattern is invention under pressure. One assistant fills the words-per-minute request with plausible numbers, which shows up as unsupported_number hits and, often, a missing or weak final section. The second is sloppy citation. Numbers are right, but the markers point at the wrong source, which shows up as misattributed_number. The third is overreach. The draft introduces product names or facts from the vendor’s training data, which shows up as unknown_entity. Each pattern calls for a different fix. Invention needs a stronger fallback rule in the brief and a human check on any section that answers a request the pack cannot support. Misattribution needs a citation pass before edit. Overreach needs a pack that includes everything the writer may legitimately mention. For the editing half of the workflow, the [Claude vs ChatGPT editing fidelity test](https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/) measures whether a revision pass preserves facts, which pairs naturally with this audit: draft, audit, edit, audit again. ### Step 12: Archive the run and freeze the pack Every run folder holds drafts, the CSV and the report. Copy sources.json, brief.md and the model IDs into the same folder, or commit the whole project to git with the run folder included. The point is that someone can repeat the test in January and compare. Without the frozen pack, a future comparison measures a different test. | File | Role | Edit after freezing? | |---|---|---| | sources.json | Ground truth for the audit | No, version it instead | | brief.md | The task and the trap | No | | audit.py | Checks and weights | Only with a changelog entry | | run.py | Provider wrappers and report | Only for new providers | | fixtures/clean.md | Should score 100 | No | | fixtures/sloppy.md | Should score near zero | No | | .env | Keys, model IDs, prices | Never commit it | | runs/ (one folder per run) | Drafts, results.csv, report.md | Append only | The project is now complete and runnable: six source excerpts, a trap brief, a deterministic auditor, two fixtures, two vendor wrappers and a reporting runner. The whole thing fits in a folder you can zip and hand to a colleague. ## What Published Claude vs ChatGPT Writing Comparisons Miss Public comparisons are useful context, and they mostly agree on style. [HubSpot’s marketer-focused comparison](https://blog.hubspot.com/marketing/claude-vs-chatgpt) states that “Claude produces prose that sounds human, maintains tone across long documents, and requires fewer revision cycles before content is publish-ready.” That is a claim about tone and revision effort. It does not say how often the draft kept a number straight. The Concrete CMS editorial team, writing up a first week with Claude, reached a more specific conclusion in [its ChatGPT vs Claude post](https://www.concretecms.com/about/blog/ai/chatgpt-vs-claude): “For writing, I found that Claude did a better job actually reading and using the reference material I gave it, rather than just acknowledging it and then producing something generic.” That observation points at grounding, the exact property this harness measures. It is one writer’s experience over a week, though, so check the post’s date against the model it tested and treat the claim as a hypothesis to test rather than a finding. The picture is not one-sided. [Lindy’s nine-factor comparison](https://www.lindy.ai/blog/claude-vs-chatgpt) concluded that “ChatGPT produced the stronger final edit.” Taken together, the three sources show why the question needs a harness. One publisher prefers Claude’s prose, one writer prefers Claude’s use of references, and one tester prefers ChatGPT’s editing. They are all describing different tasks, different models and different moments in a fast-moving release cycle. A grounding audit turns those competing impressions into separate, testable claims: - Does the assistant use reference material instead of acknowledging it? Measure unsupported_number and unknown_entity. - Does it keep its facts straight across a long draft? Raise the word target to 1,200 and compare finding counts. - Does an edit pass preserve facts? Audit the draft before and after the edit. - Does it hold a consistent voice? That needs a stylometry check, which the [Claude vs ChatGPT voice match test](https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/) covers. None of this makes the published pieces wrong. It makes them incomplete in the way any single-run, single-prompt review is incomplete. Your own brief, your own source material and your own five or twenty trials are better evidence for your own workflow than any vendor page or review site. ## Seven Common Pitfalls in a Claude vs ChatGPT Writing Audit These mistakes show up repeatedly when teams run their first grounded comparison. **1. Comparing different prompts.** A tester tweaks the instructions for the vendor that struggled, then compares scores. The result measures prompt effort, not model quality. Send the byte-identical brief and pack to both, and change both together if you change either. **2. Testing a retired or outdated model.** A guide written in spring tells you to select a model that is gone. Check each vendor’s model docs on the day you run. Anthropic lists Sonnet 5.5 with a retirement date not sooner than September 28, 2027, while OpenAI retired several older ChatGPT models earlier this year. Pin the exact ID in .env and write it in the report. **3. Trusting one trial.** Both assistants vary between runs. A single draft can score 100 or 60 from the same model on the same day. Use at least five trials per model, and twenty if the means are close. **4. Letting the pack leak in the brief.** If the brief itself states a figure that is missing from the pack, the model will use it and the audit will flag a correct number as unsupported. Keep all facts in the pack and all instructions in the brief. **5. Reading cost columns as a business case.** The cost column covers the harness calls only. A cheap per-token rate does not account for editing time, rework or seat licences. The [price gap between Claude, ChatGPT and Gemini](https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/) is worth reading before you turn per-token numbers into a budget, since subscription limits often matter more than token rates. **6. Over-tuning weights after seeing results.** If you raise the quotation penalty because the assistant you dislike fabricated more quotes, you have stopped measuring. Fix the weights first, write them down and keep them. **7. Treating a high score as proof of quality.** A perfect grounding score says the draft did not invent facts. It does not say the draft is useful, well structured or on brand. Keep a human editor in the loop for everything the auditor cannot see. ## Troubleshooting: 10 Errors and Their Fixes Each item names the symptom first, then the cause, then the fix. **1. KeyError: ‘CLAUDE_MODEL’ or ‘OPENAI_MODEL’.** The environment variables are not exported in this terminal. Run the export line from Step 2 again, then confirm with `echo $CLAUDE_MODEL`. **2. Authentication error (401) from either vendor.** The key is wrong, has trailing whitespace or belongs to an account without billing enabled. Paste it again without quotes or spaces and check the billing page in the vendor console. **3. Model not found (404).** The model ID in .env is a guess or a retired name. Copy the ID from the vendor’s model documentation and do not infer it from the marketing name. A product called GPT-6 Sol in the app may use a different string in the API. **4. ModuleNotFoundError for anthropic or openai.** The virtual environment is not active. Run `source .venv/bin/activate` and `pip install -r requirements.txt`, then retry. **5. Rate limit error.** You exceeded the account’s request or token allowance. Add a short sleep between trials, lower the trial count, or request a higher limit in the console. Newer accounts often start on conservative tiers. **6. The output is empty or cut off mid-sentence.** The token cap is too low for a model that spends output tokens on reasoning. Raise max_tokens in the Claude call and max_output_tokens in the ChatGPT call. Raise both in small steps and check the cost afterward. **7. Every draft fails the missing_heading check.** The model returned H3s, bold titles or numbered headings instead of exact H2 text. Check the saved draft. If the heading wording differs by punctuation, loosen the check to compare lowercase text with punctuation stripped, or tighten the brief. **8. Fixture scores differ from this page.** The pack or the auditor was edited, or Python’s regex module behaves differently on a copied smart quote. Re-copy audit.py and sources.json as plain text, rerun the fixtures and compare line by line. **9. A correct number is flagged as unsupported.** The model wrote it in a form the normalizer does not know, such as “forty-four percent” or “$2.00”. Spell figures as digits in the brief, or add a pre-processing step to convert words to numerals before the audit. **10. The two vendors’ costs look inverted.** The price variables are swapped or still hold the Sonnet 5.5 and GPT-6 Sol defaults while you test a different tier. Compare your .env to the pricing pages. For model-selection context across coding and writing, the [Claude vs ChatGPT for coding comparison](https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/) shows how the Astra tier changes the cost math, and the [AI and machine learning hub](https://shattered.io/category/ai-machine-learning/) collects the rest of the model coverage. ## Advanced Tips: Make the Audit Harder and Cheaper Once the baseline runs, three upgrades raise the value of the test without much work. **Test the refusal path separately.** Add a second brief where every requested fact exists in the pack, and a third where none does. A grounded assistant should write confidently on the first and decline gracefully on the third. The gap between its behavior on the two tells you whether it can tell what it knows from what it was given. **Add a numeric-derivation check.** Writers legitimately compute things. A sentence that says two plans cost $300 combined is derived, not copied. The current auditor flags $300 as unsupported. To allow derived sums, parse simple arithmetic: gather all number pairs from the cited source and permit a figure that equals their sum, difference or ratio. Log those separately as derived_number so a human can review them. ``` python from itertools import combinations def derived_ok(n: float, source_numbers: list[float]) -> bool: for a, b in combinations(source_numbers, 2): if abs(n - (a + b)) < 1e-9 or abs(n - abs(a - b)) < 1e-9: return True return False ``` **Cache the source pack.** If your real pack runs to tens of thousands of tokens, repeated trials resend the same text each time. Anthropic’s docs list a 512-token minimum cacheable prompt for Sonnet 5.5, and cache reads are priced at $0.10 per million tokens against $2 for base input. Turn on prompt caching for the pack and the batch discount for non-urgent runs. The Batch API gives 50% off input and output on the Claude side, which suits overnight regression runs. **Run a nightly regression.** Models change underneath stable names. Schedule the harness weekly with the same pack and brief, append the scores to a CSV and chart the trend. A sudden drop after a vendor announcement is exactly the signal you want, and it costs cents to collect. ## FAQ: ChatGPT vs Claude for Writing **Is Claude or ChatGPT better for writing in November 2026?** Neither wins on every task, and published reviews disagree. HubSpot favors Claude’s prose, Concrete CMS’s writer favors how Claude uses reference material and Lindy found ChatGPT stronger on the final edit. Test your own brief with the audit above and decide on your own data. **Which models should I compare?** Use the newest of each vendor’s everyday model. As of November 5, 2026, that means Claude Sonnet 5.5 against GPT-6 Sol, both listed at $2 input and $10 output per million tokens. Add Claude Opus 5.5 against GPT-6 Pro (GPT-6 Astra) for a high-end pass, and record the exact model ID each time. **Do I need API access, or can I use the chat apps?** The auditor runs on any text file, so you can paste the brief and pack into both apps by hand. API access adds automation, repeat trials, token counts and cost. Check what your ChatGPT plan includes for API use, and note that Claude’s API is billed per token apart from the monthly Claude Platform credits attached to some plans. **How many trials are enough?** Five per model is a minimum for a first read. If the mean scores fall within a few points, run twenty, because a single model can swing more between runs than the two models differ from each other. **Why not use another AI model to grade the drafts?** A grader model adds its own errors and its own preferences, and you cannot easily tell whether it favors its own vendor’s style. Deterministic checks are repeatable and cost nothing. Use a human or a model later for voice and structure, after the facts pass. **Does a high audit score mean the article is ready to publish?** No. It means the draft stayed inside your sources. An editor still has to check tone, structure, legal risk and whether the sources themselves are right. **How do I adapt this for my own content?** Replace sources.json with excerpts from your real material, rewrite brief.md for your format and adjust REQUIRED_H2 and the word range in audit.py. Keep the trap request in the brief. It is the fastest way to see which assistant invents when it is under pressure.