Claude vs ChatGPT Voice Match: 12 Steps, 80 Min [2026]

A 12-step, roughly 80-minute stylometry harness using Burrows' Delta can measure how closely Claude or ChatGPT drafts match a specific author's voice, expressed as a percentage of gap closure between each model's generic output and the author's own noise floor. The test pins the November 3, 2026 lineups: OpenAI's paid ChatGPT plans run GPT-6 Sol while Free and Go run GPT-6 Luna, and Anthropic's writing-relevant family is Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, with Anthropic cutting the Sonnet 5.5 cache-read price to $0.10 per million tokens alongside the Haiku 5.5 release. The author notes neither vendor's official pages offer a dated, controlled writing comparison between GPT-6 and Claude, and that stylometry measures texture rather than quality or model intelligence.

Ask ten writers whether Claude or ChatGPT sounds more like them and you get ten confident answers and zero data. The Claude vs ChatGPT for writing debate gets simpler once you stop asking which model writes better and start asking which one lands closer to you. That distance is measurable, and the setup you give each model nothing, a style guide, or a few of your own samples moves it. The lineups also changed under everyone’s feet. As of November 3, 2026, OpenAI’s pricing page https://chatgpt.com/pricing has paid ChatGPT plans running GPT-6 Sol and the Free and Go plans running GPT-6 Luna, while Anthropic’s writing-relevant family is Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5. Pin those names, the plan and the date in every run, or your numbers will mean nothing next month. Two sibling tutorials on this site cover neighboring questions. A blind rating test for Claude vs ChatGPT for writing https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ asks humans to pick winners, and a ChatGPT vs Claude writing pipeline https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ chains the models across drafting stages. Neither measures whether a draft sounds like one specific author. That gap is the target here, and the answer comes out as one percentage per model and per setup. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why Voice Match Needs Its Own Test Quality rubrics reward the same things in every writer. Clear structure, correct facts, no filler. A draft can ace all of that and still read like a stranger wrote it, which is a problem when the byline is yours. Voice lives in small habits: how long your sentences run, how often you reach for a parenthesis, whether you ask a question mid-paragraph, which little words you lean on without thinking. Those little words are the key. Stylometry, the statistical study of writing style, has long used the most frequent words of a language the, of, and, but, I to tell authors apart, because writers pick them unconsciously and topic barely moves them. Burrows’ Delta turns that idea into one number. You z-score each frequent word against how much your own documents vary, then average the absolute distances. Lower means closer to you. A raw Delta is hard to read alone, so the harness adds three anchors. The first is a noise floor, the Delta your own held-out documents score against each other in a leave-one-out loop. No model should be expected to beat you at being you. The second is each model’s generic output, produced with no style information, which marks where that model starts. The third is gap closure, the share of the distance between generic and floor that a setup closes. Zero percent means your style guide or samples did nothing. One hundred percent means the draft sits as close to you as you sit to yourself. Two boundaries apply before you start. Stylometry measures texture, not quality, so a 90 percent closure on a muddled essay still gives you a muddled essay. And the numbers say nothing about which model is smarter. Neither vendor’s current official pages offer a dated, controlled writing comparison between GPT-6 and Claude, which is exactly why a local test beats a leaderboard. If you also want to grade factual fidelity and structure, pair this harness with the editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ from the same series. Pin the November 2026 Lineups Before You Run Anything Anthropic’s current family for writing work has three tiers. Claude Opus 5.5 is the flagship for knowledge work, Sonnet 5.5 is the general-purpose workhorse, and Haiku 5.5 is the fast small model. Anthropic also cut the Sonnet 5.5 cache-read price to $0.10 per million tokens alongside the Haiku 5.5 release, which matters later if you reuse a long few-shot prompt across many briefs. List prices for every model in this test sit in the first table. OpenAI’s side moved just as fast. GPT-6 reached ChatGPT on October 7, 2026, with Plus, Pro, Business and Enterprise on GPT-6 Sol and Free and Go on GPT-6 Luna. GPT-6 Pro, powered by GPT-6 Astra, is available on Pro 100, Pro 200, Business and Enterprise. According to OpenAI’s release notes, an October 29 update made GPT-6 in ChatGPT answer more concisely and flatter the user less. That is a style change, so any voice test you ran before that date measured a different assistant. | Model | Vendor | Role in this test | API input per 1M tokens | API output per 1M tokens | |---|---|---|---|---| | Claude Opus 5.5 | Anthropic | Quality ceiling for hard drafts | $4 | $20 | | Claude Sonnet 5.5 | Anthropic | Main drafting baseline | $2 | $10 | | Claude Haiku 5.5 | Anthropic | Cheap high-volume tier | $0.10 prompts up to 100,000 tokens | $0.50 prompts up to 100,000 tokens | | GPT-6 Sol | OpenAI | Paid-plan drafting baseline | $2 | $10 | | GPT-6 Luna | OpenAI | Cheap high-volume tier | $0.10 | $0.50 | | GPT-6 Astra | OpenAI | Top tier behind GPT-6 Pro | $10 | $50 | Pricing moved a lot this summer, so treat any older number you remember as stale. Anthropic made the Sonnet $2 and $10 rate permanent on August 10, 2026 and canceled a planned September increase, as sakuttoAI’s price-cut write-up https://sakutto.ai/en/articles/openai-anthropic-llm-price-cuts explains. On the app side, Claude’s pricing page https://claude.ai/pricing lists Claude Pro at $20 a month when billed monthly, with an annual option billed at $200 up front. Anthropic also announced on October 7 that Max subscribers get $100 in monthly Claude Platform API credits and Ultra subscribers get $200. A weekly roundup from an AI newsletter on note.com https://note.com/ kihonushi/n/n82421d293670?hl=en reported that Max 5x and Max 20x were renamed Max and Ultra on October 30, so check plan names on your own account before you cite them. | ChatGPT plan | Monthly price | What matters for a writing test | |---|---|---| | Free | $0 | GPT-6 Luna, with limits on uploads, memory and context | | Plus | $20 | GPT-6 Sol, plus memory, projects, scheduled tasks and custom GPTs | | Pro 100 | $100 | GPT-6 Pro GPT-6 Astra access | | Pro 200 | $200 | More usage than Pro 100, open to new subscribers again | | Pro 500 | $500 | Highest Pro allowance, adds GPT-6 Astra Ultrafast in Work and Codex | | Business | $25 per user monthly, $20 per user billed annually | Standard seat, billed per user | One scope note keeps the results honest. The harness below calls the APIs, so it tests the models and not the app layer. Memory, projects and custom GPTs on the ChatGPT side, and projects on the Claude side, change the experience a paid subscriber gets. The Advanced Tips section shows how to add a manual arm for those. If plan costs are your real question, the breakdown of ChatGPT vs Claude pricing and usage limits https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ goes deeper than this tutorial needs to. Prerequisites and Versions Gather these before you touch a terminal. None of it is exotic, but a missing piece costs you an evening. - Python 3.11 or newer. The config loader uses tomllib, which ships with the standard library from 3.11. This tutorial was run on Python 3.12.3. - A POSIX shell. bash or zsh on Linux or macOS. On Windows, use WSL or translate the activation command. - Two API keys. Export them as ANTHROPIC API KEY and OPENAI API KEY. Nothing else reads credentials. - Latest releases of the anthropic and openai packages, plus pytest. No pins are listed because the code only touches the Messages API on one side and the Responses API on the other. Run pip freeze after installing and save the output with your results. - Exact model IDs. Copy them from each vendor’s models page. The config ships with REPLACE WITH placeholders on purpose, and the CLI refuses to run until you fill them in. - Your own writing. At least 8 plain-text files of 250 words or more, ideally 12 to 20. Blog posts, newsletters and long emails work. Strip quoted text from other people, code blocks and boilerplate signatures. - A small budget. A full run is 72 generations: 2 models, 3 setups, 6 briefs, 2 repeats. With the prices in the first table, that lands at a few dollars on the flagship tiers and well under a dollar on Sonnet 5.5 or GPT-6 Sol. Reasoning tokens can push that up, so set a spending cap on both consoles first. The corpus is the one input that cannot be faked. Eight documents gives the noise floor only five or six held-out samples, which is thin. Twelve gives nine. If you write in several registers casual newsletter and formal report, say , run the harness once per register, because mixing them inflates your own variance and makes every model look closer than it is. Steps 1 and 2: Project Layout, Config and Corpus Loader Step 1: scaffold the project. Create the folders, a virtual environment and the two environment variables. The empty init .py makes voicematch an importable package, and pytest relies on that. mkdir voicematch && cd voicematch mkdir -p voicematch tests scripts corpus out touch voicematch/ init .py python3 -m venv .venv && source .venv/bin/activate pip install anthropic openai pytest export ANTHROPIC API KEY="paste-your-key-here" export OPENAI API KEY="paste-your-key-here" Drop your writing into corpus/ as .txt files. If you only want to check the plumbing first, skip ahead to Step 12 and generate a synthetic demo corpus instead. Step 2: write the config and the corpus loader. All tunable values live in one TOML file so a rerun a month from now uses identical settings. Three choices deserve a word. The profile docs value of 3 means three of your documents are shown to the models as examples and as the source of the derived style guide , and every remaining document is held out for scoring only. The seed makes that split reproducible. The tells list holds stock phrases you personally never write, which the report counts as a side signal. corpus dir = "corpus" min words = 250 min docs = 8 profile docs = 3 excerpt words = 220 seed = 7 stylometry mfw = 100 copy ngram = 6 run target words = 350 repeats = 2 max tokens = 4000 conditions = "generic", "guide", "fewshot" tells = "dive into", "game-changer", "unlock", "leverage", "elevate", "realm of", "journey", "at its core", "stands as", "cutting-edge", models.claude provider = "anthropic" model = "REPLACE WITH CLAUDE MODEL ID" models.chatgpt provider = "openai" model = "REPLACE WITH OPENAI MODEL ID" models.fake provider = "fake" model = "fake-1" The loader below does three jobs. It tokenizes into lowercase words with apostrophes kept, it splits text into sentences, and it enforces the corpus rules. The split is deterministic, so Claude and ChatGPT always see the same three profile documents. The excerpt helper trims each profile document to whole sentences under a word limit, which keeps few-shot prompts a predictable size. python import random import re from pathlib import Path WORD RE = re.compile r" a-z + ?:' a-z + ?" SENT RE = re.compile r" ?<= . ? \s+" def normalize text: str - str: return text.replace "\u2019", "'" .replace "\u2018", "'" def words text: str - list str : return WORD RE.findall normalize text .lower def sentences text: str - list str : parts = SENT RE.split normalize text .strip return p.strip for p in parts if len words p = 2 def load corpus directory: str, min words: int, min docs: int - dict str, str : docs = {} for path in sorted Path directory .glob " .txt" : text = path.read text encoding="utf-8" .strip if len words text = min words: docs path.name = text if len docs < min docs: raise SystemExit f"Need at least {min docs} documents of {min words}+ words in " f"{directory}, found {len docs }." return docs def split corpus docs: dict str, str , profile n: int, seed: int : names = sorted docs random.Random seed .shuffle names profile = {n: docs n for n in names :profile n } holdout = {n: docs n for n in names profile n: } return profile, holdout def excerpt text: str, max words: int - str: out, count = , 0 for sent in sentences text : n = len words sent if count + n max words and out: break out.append sent count += n return " ".join out Notice what the loader refuses to do. It will not quietly accept short files or a tiny corpus, because a silent fallback there would produce a noise floor that looks fine and means nothing. Fail loudly, fix the input, rerun. Steps 3 to 5: Stylometry, Noise Floor and Gap Closure Step 3: extract features. Two kinds of signal come out of every text. The frequent-word profile feeds Burrows’ Delta. The surface profile captures mean sentence length, its coefficient of variation a burstiness proxy , and punctuation rates per 1,000 words for commas, semicolons, long dashes, opening parentheses, question marks and colons. The long dash is written as a Unicode escape in the source so the file stays plain ASCII. Step 4: build the reference and score with Delta. The reference takes the 100 most frequent words across your held-out documents, then records each word’s mean and standard deviation across those documents. Scoring a generated text means computing its frequency for the same words, z-scoring against your variation, and averaging the absolute values. Step 5: add the controls. The noise floor, the AI-tell density, the copy rate and the closure formula all live in the same module. Read the whole file once, then the notes after it. The discipline here frozen inputs, a written metric, controls before conclusions mirrors what a good retrieval eval does, and the RAG pipeline with evals build https://shattered.io/build-rag-pipeline-hybrid-search-evals-12-steps-2026/ shows the same habits on a different problem. python from collections import Counter from dataclasses import dataclass from statistics import mean, median, pstdev from .textutils import normalize, sentences, words PUNCT = {"comma": ",", "semicolon": ";", "dash": ", ", "paren": " ", "question": "?", "colon": ":"} def rel freq tokens: list str , vocab: list str - list float : counts, n = Counter tokens , max len tokens , 1 return counts w / n for w in vocab def surface text: str - dict str, float : toks, sents = words text , sentences text lengths = len words s for s in sents or 0 n = max len toks , 1 feats = { "sent mean": mean lengths , "sent cv": pstdev lengths / max mean lengths , 1e-9 , } norm = normalize text for name, char in PUNCT.items : feats name = 1000 norm.count char / n return feats @dataclass class Reference: vocab: list str mu: list float sigma: list float surf mu: dict str, float surf sigma: dict str, float def build reference texts: list str , k: int - Reference: token lists = words t for t in texts pooled = Counter w for toks in token lists for w in toks vocab = w for w, in pooled.most common k cols = list zip rel freq t, vocab for t in token lists mu = mean c for c in cols sigma = max pstdev c , 0.1 m, 1e-4 for c, m in zip cols, mu feats = surface t for t in texts s mu = {f: mean d f for d in feats for f in feats 0 } s sd = {f: max pstdev d f for d in feats , 0.05 abs s mu f , 1e-3 for f in feats 0 } return Reference vocab, mu, sigma, s mu, s sd def burrows delta text: str, ref: Reference - float: freq = rel freq words text , ref.vocab return mean abs x - m / s for x, m, s in zip freq, ref.mu, ref.sigma def surface gap text: str, ref: Reference - float: feats = surface text return mean abs feats f - ref.surf mu f / ref.surf sigma f for f in feats def noise floor texts: list str , k: int - float: scores = for i, held in enumerate texts : rest = texts :i + texts i + 1: scores.append burrows delta held, build reference rest, k return median scores def tell density text: str, tells: list str - float: low, n = normalize text .lower , max len words text , 1 return 1000 sum low.count t for t in tells / n def copy rate text: str, source texts: list str , n: int - float: def grams toks : return {tuple toks i:i + n for i in range len toks - n + 1 } src = set .union grams words t for t in source texts gen = grams words text return len gen & src / len gen if gen else 0.0 def closure generic: float, condition: float, floor: float : gap = generic - floor if gap < 0.05: return None return generic - condition / gap Why the sigma floor exists A frequent word that barely varies across your documents has a tiny standard deviation. Dividing by it turns a single extra occurrence in a 350-word draft into a z-score of 6 or 8, and one word dominates the average. The code puts a floor under each sigma so that cannot happen. The same idea protects the surface features. Without the floor, short corpora produce wild Deltas, and you will see the symptom in the troubleshooting list below. Gap closure and its guard Closure is generic minus condition divided by generic minus floor . If a model’s generic output already sits within 0.05 Delta of your noise floor, the denominator collapses and the ratio explodes, so the function returns None and the report prints n/a. That is the correct answer in that case. It means the model’s default voice is already indistinguishable from yours at this resolution, and there is no gap left to close. The copy rate counts how many 6-word sequences in a generated text also appear in your profile documents. Few-shot prompts invite copying. A model that parrots your sentences scores a great Delta and a terrible rating as a writing assistant, so the report prints the copy rate beside every closure. Steps 6 and 7: Three Setups, One Brief Step 6: define the briefs and the derived guide. Six short briefs cover different registers: advice, narrative, argument, customer communication, reflection and opinion. They are deliberately unrelated to any specific corpus, because topic words leak into the frequent-word list. Edit them to suit your field, but keep them away from what you usually write about. If your corpus is all about databases, a brief about databases hands both models free vocabulary. The style guide is generated from your profile documents by measuring them, not by asking a person to describe themselves. That choice is about fairness. A hand-written guide reflects how well you can articulate your own habits, and a clever phrase in it might happen to suit one model’s training. A measured guide is the same for both vendors, and it reruns identically next quarter. Step 7: build the three conditions. The generic condition sends the brief with a one-line system prompt and nothing else. The guide condition adds the measured style notes to the system prompt. The fewshot condition adds three excerpts of your writing and asks for the voice but not the content. All three share the same user message, so the only variable is the system prompt. python from statistics import mean from .stylometry import surface from .textutils import excerpt BRIEFS = {"id": "b1", "topic": "why small teams should write down how they make decisions"}, {"id": "b2", "topic": "what a first week at a new job actually feels like"}, {"id": "b3", "topic": "the case for boring technology on a side project"}, {"id": "b4", "topic": "how to explain a delayed launch to customers"}, {"id": "b5", "topic": "lessons from a spreadsheet that grew out of control"}, {"id": "b6", "topic": "why the best meeting is the one you skip"}, def derive guide profile texts: list str - str: feats = surface t for t in profile texts avg = {k: mean f k for f in feats for k in feats 0 } return "\n".join f"- Average sentence length is about {avg 'sent mean' :.0f} words, " f"with a spread of {avg 'sent cv' :.2f} std divided by mean . Mix short and long.", f"- Use roughly {avg 'comma' :.0f} commas, {avg 'paren' :.0f} opening parentheses " f"and {avg 'question' :.0f} question marks per 1,000 words.", f"- Use roughly {avg 'semicolon' :.0f} semicolons, {avg 'colon' :.0f} colons " f"and {avg 'dash' :.0f} long dashes per 1,000 words.", "- Write in plain words. No marketing phrases, no stock openings, no closing summary.", def build prompt condition: str, brief: dict, profile: dict str, str , target words: int, excerpt words: int : task = f"Write an essay of about {target words} words on this topic: " f"{brief 'topic' }. Return only the essay text, with no title and no preamble." base = "You are a writing assistant." if condition == "generic": return base, task if condition == "guide": guide = derive guide list profile.values return f"{base} Match this author's measured style:\n{guide}", task if condition == "fewshot": shots = "\n\n".join f"EXAMPLE {i}:\n{excerpt t, excerpt words }" for i, t in enumerate profile.values , 1 system = f"{base} Below are samples by one author. Match the voice, rhythm " f"and punctuation habits. Do not reuse their sentences or subjects.\n\n{shots}" return system, task raise ValueError f"unknown condition: {condition}" If you already keep style notes inside a Claude project or a ChatGPT memory, you can test those too. Moving that context between assistants is its own task, and the guide to importing memory from ChatGPT into Claude https://shattered.io/claude-import-memory-from-chatgpt-12-steps-2026/ covers it. For this harness, keep the system prompts above so both sides receive byte-identical instructions. Steps 8 and 9: Provider Adapters, a Cache and an Offline Fake Step 8: wrap both SDKs behind one method. The Anthropic adapter calls the Messages API and joins the text blocks of the reply. The OpenAI adapter calls the Responses API with the system text as instructions and reads output text. Neither call sets temperature. Some current reasoning-style models reject sampling parameters outright, and sending a value to one vendor but not the other would break the comparison. If you want to test temperature, do it as a separate experiment per model. Both adapters take the same max tokens budget, set to 4,000 in the config. That looks generous for a 350-word essay, and it should. Models that reason before answering spend part of the output budget on thinking, and a tight cap returns an empty string with no error. The empty-completion guard in the cache function exists because of exactly that failure. Step 9: add a cache, retries and a fake provider. Every generation is stored under a hash of the provider, model, both prompts and the repeat index. Rerun the harness after changing only the report code and it costs nothing. The retry loop makes four attempts with exponential backoff, which absorbs ordinary rate-limit blips without hiding a bad key. The fake provider is the piece that makes the whole project testable offline. For prompts without examples, it returns a deliberately generic paragraph stuffed with stock phrases. When the system prompt contains examples, it recycles your own sentences at random. It is not a model and its scores say nothing about Claude or ChatGPT. It proves the pipeline end to end, and it gives the unit tests something deterministic to chew on. python import hashlib import json import random import re import time from pathlib import Path from .textutils import sentences GENERIC FAKE = "Many teams struggle with this topic. It is a journey that " "requires careful thought. Let us dive into the main points. First, clarity " "matters because it helps everyone move in the same direction. Second, " "consistency builds trust over time. Third, leverage the tools you already " "have to unlock better results. Overall, the benefits are clear and the " "approach is practical for almost any group of people." class AnthropicProvider: def init self, model: str : import anthropic self.client, self.model = anthropic.Anthropic , model def generate self, system: str, user: str, max tokens: int - str: r = self.client.messages.create model=self.model, max tokens=max tokens, system=system, messages= {"role": "user", "content": user} , return "".join b.text for b in r.content if b.type == "text" .strip class OpenAIProvider: def init self, model: str : import openai self.client, self.model = openai.OpenAI , model def generate self, system: str, user: str, max tokens: int - str: r = self.client.responses.create model=self.model, instructions=system, input=user, max output tokens=max tokens, return r.output text or "" .strip class FakeProvider: """Offline stand-in: generic prose, or recycled sentences when examples are present.""" def init self, model: str : self.model = model def generate self, system: str, user: str, max tokens: int - str: if "EXAMPLE 1:" not in system: return GENERIC FAKE 3 pool = for block in re.split r"EXAMPLE \d+:\n", system 1: : pool += sentences block.split "\n\n" 0 rng = random.Random hashlib.sha256 user.encode .hexdigest return " ".join rng.choice pool for in range 30 def make provider spec: dict : kind = spec "provider" if kind == "anthropic": return AnthropicProvider spec "model" if kind == "openai": return OpenAIProvider spec "model" if kind == "fake": return FakeProvider spec "model" raise ValueError f"unknown provider: {kind}" def cached generate provider, spec: dict, system: str, user: str, repeat: int, max tokens: int, cache dir: Path - str: key = hashlib.sha256 json.dumps spec "provider" , spec "model" , system, user, repeat .encode .hexdigest path = cache dir / f"{key}.txt" if path.exists : return path.read text encoding="utf-8" for attempt in range 4 : try: text = provider.generate system, user, max tokens break except Exception: if attempt == 3: raise time.sleep 2 attempt if not text: raise RuntimeError "empty completion: raise max tokens or check the model id" cache dir.mkdir parents=True, exist ok=True path.write text text, encoding="utf-8" return text The adapter layer is deliberately thin so you can add a third vendor in ten lines. If you later want to route requests between models by cost or task, the same shape underpins the Claude vs GPT router build https://shattered.io/build-llm-router-claude-gpt-2026/ . Steps 10 and 11: The Runner and the Report Step 10: write the runner. It loads the config and the corpus, splits off the profile documents, builds the reference from the held-out texts, and computes the leave-one-out noise floor once. Then it loops over models, conditions, briefs and repeats. For every generation it records word count, length ratio, Delta, surface gap, tell density and copy rate. Everything lands in out/results.json, along with the controls, so a later script can re-aggregate without calling any API. python import json import tomllib from pathlib import Path from .prompts import BRIEFS, build prompt from .providers import cached generate, make provider from .stylometry import build reference, burrows delta, copy rate, noise floor, surface gap, tell density from .textutils import load corpus, split corpus, words def load config path: str = "config.toml" - dict: with open path, "rb" as fh: return tomllib.load fh def run cfg: dict, model keys: list str , out dir: str = "out" - dict: c, s, r = cfg "corpus" , cfg "stylometry" , cfg "run" docs = load corpus c "dir" , c "min words" , c "min docs" profile, holdout = split corpus docs, c "profile docs" , c "seed" hold texts, prof texts = list holdout.values , list profile.values ref = build reference hold texts, s "mfw" floor = noise floor hold texts, s "mfw" out = Path out dir rows = for key in model keys: spec = cfg "models" key provider = make provider spec for cond in r "conditions" : for brief in BRIEFS: system, user = build prompt cond, brief, profile, r "target words" , c "excerpt words" for rep in range r "repeats" : text = cached generate provider, spec, system, user, rep, r "max tokens" , out / "cache" n = len words text rows.append { "model": key, "condition": cond, "brief": brief "id" , "repeat": rep, "words": n, "length ratio": n / r "target words" , "delta": burrows delta text, ref , "surface gap": surface gap text, ref , "tells": tell density text, r "tells" , "copy": copy rate text, prof texts, s "copy ngram" , } controls = { "floor": floor, "holdout docs": len hold texts , "profile docs": sorted profile , } out.mkdir exist ok=True out / "results.json" .write text json.dumps {"controls": controls, "rows": rows}, indent=2 , encoding="utf-8" return {"controls": controls, "rows": rows} One detail trips people up. The reference is built from held-out documents only, and the noise floor uses the same documents in a leave-one-out loop. Models never see a held-out document, so they cannot copy their way to a good score. That separation is the whole reason the project asks for at least eight files. Step 11: write the report. The summary groups rows by model and condition, averages each metric, and computes closure against that model’s own generic mean and the shared noise floor. The Markdown table is what you paste into your notes. Keep the columns in this order, because the decision rule later in the tutorial reads left to right. python from collections import defaultdict from statistics import mean from .stylometry import closure def summarize result: dict - list dict : groups = defaultdict list for row in result "rows" : groups row "model" , row "condition" .append row floor = result "controls" "floor" summary = for model, cond , rows in sorted groups.items : generic = groups.get model, "generic" g = mean x "delta" for x in generic if generic else None d = mean x "delta" for x in rows summary.append { "model": model, "condition": cond, "n": len rows , "delta": d, "surface gap": mean x "surface gap" for x in rows , "tells": mean x "tells" for x in rows , "copy": mean x "copy" for x in rows , "length ratio": mean x "length ratio" for x in rows , "closure": closure g, d, floor if g is not None else None, } return summary def to markdown result: dict - str: floor = result "controls" "floor" lines = f"Noise floor your own held-out docs, leave-one-out : delta {floor:.2f}", "", "| model | condition | n | delta | surface | tells/1k | copy | len | closure |", "|---|---|---|---|---|---|---|---|---|", for s in summarize result : c = "n/a" if s "closure" is None else f"{s 'closure' :.0%}" lines.append f"| {s 'model' } | {s 'condition' } | {s 'n' } | {s 'delta' :.2f} | " f"{s 'surface gap' :.2f} | {s 'tells' :.1f} | {s 'copy' :.1%} | " f"{s 'length ratio' :.2f} | {c} |" return "\n".join lines Step 12: Tests, the Dry Run and Your First Live Run Step 12: prove it works, then spend money. The test file covers tokenization, sentence splitting, the direction of Delta, tell counting, copy detection, the closure math and its guard, and an end-to-end dry run through the CLI. The end-to-end test builds a synthetic corpus in a temporary folder and never touches your real corpus directory. The demo corpus script produces twelve documents of templated sentences in one fixed voice, which is enough to exercise every code path. python import subprocess import sys from pathlib import Path from voicematch.stylometry import build reference, burrows delta, closure, copy rate, tell density from voicematch.textutils import sentences, words A = "I wrote the plan twice . It was short, and it worked. Why wait? " 12 B = "Teams should leverage tools to unlock results and dive into the journey. " 12 def test words handles curly apostrophes : assert words "It’s fine" == "it's", "fine" def test sentences split : assert len sentences "One two. Three four Five six?" == 3 def test delta prefers same voice : ref = build reference A, A + " Extra words here.", A.replace "plan", "note" , 20 assert burrows delta A, ref < burrows delta B, ref def test tell density counts : assert tell density B, "leverage", "unlock" 0 assert tell density A, "leverage", "unlock" == 0 def test copy rate detects verbatim : assert copy rate A, A , 6 0.9 assert copy rate B, A , 6 == 0 def test closure math and guard : assert closure 3.0, 2.0, 1.0 == 0.5 assert closure 1.02, 1.0, 1.0 is None def test end to end dry run tmp path : demo = tmp path / "demo corpus" subprocess.run sys.executable, "scripts/make demo corpus.py", str demo , check=True cfg = Path "config.toml" .read text .replace 'dir = "corpus"', f'dir = "{demo}"' tmp path / "config.toml" .write text cfg done = subprocess.run sys.executable, "cli.py", "--config", str tmp path / "config.toml" , "--models", "fake", "--out", str tmp path / "out" , check=True, capture output=True, text=True assert "closure" in done.stdout assert tmp path / "out" / "results.json" .exists python import random import sys from pathlib import Path THINGS = "spreadsheet", "standup", "backlog", "release note", "coffee machine", "pull request", "roadmap", "calendar invite", "wiki page", "build script", "dashboard", "retro" TEMPLATES = "I tested the {t} twice once on purpose .", "The {t} is slower than it looks, but it is honest.", "Nobody asked about the {t}, so I wrote it down anyway.", "Why does the {t} always break on a Friday?", "My rule for the {t} is simple: fix it before it grows.", "A {t} is only useful if somebody reads it and somebody rarely does .", "We argued about the {t} for an hour, then shipped it.", "The {t} taught me more than any course did.", def main target: str = "corpus" : rng = random.Random 11 pool = tpl.format t=t for tpl in TEMPLATES for t in THINGS Path target .mkdir parents=True, exist ok=True for i in range 1, 13 : body = " ".join rng.sample pool, 40 Path target, f"post{i:02d}.txt" .write text body + "\n", encoding="utf-8" print f"wrote 12 demo documents to {target}/" if name == " main ": main sys.argv 1:2 The command line ties everything together. It validates model keys, refuses to run while a model ID still starts with REPLACE WITH, writes a Markdown report next to the JSON, and prints it. python import argparse import sys from voicematch.report import to markdown from voicematch.runner import load config, run def main - int: ap = argparse.ArgumentParser description="Claude vs ChatGPT voice-match test" ap.add argument "--config", default="config.toml" ap.add argument "--models", nargs="+", default= "fake" , help="keys from models. in the config, e.g. claude chatgpt" ap.add argument "--out", default="out" args = ap.parse args cfg = load config args.config for key in args.models: if key not in cfg "models" : sys.exit f"unknown model key: {key}" if cfg "models" key "model" .startswith "REPLACE WITH" : sys.exit f"set a real model id for {key} in {args.config}" result = run cfg, args.models, args.out report = to markdown result with open f"{args.out}/report.md", "w", encoding="utf-8" as fh: fh.write report + "\n" print report return 0 if name == " main ": raise SystemExit main Run the tests from the project root, because the end-to-end test calls scripts and the CLI by relative path. python -m pytest -q python scripts/make demo corpus.py python cli.py --models fake Here is the real output from those commands on the demo corpus, captured on Python 3.12.3. The test run prints seven passing tests, and the dry run prints the report. ....... 100% 7 passed in 0.41s Noise floor your own held-out docs, leave-one-out : delta 1.06 | model | condition | n | delta | surface | tells/1k | copy | len | closure | |---|---|---|---|---|---|---|---|---| | fake | fewshot | 12 | 1.64 | 0.84 | 0.0 | 72.3% | 0.93 | 78% | | fake | generic | 12 | 3.66 | 7.84 | 60.6 | 0.0% | 0.57 | 0% | | fake | guide | 12 | 3.66 | 7.84 | 60.6 | 0.0% | 0.57 | 0% | Read that table the way you will read a live one. The generic row sits well above the floor, so there is a clear gap to close. The fewshot row lands much nearer the floor, which closes most of that gap. But the copy column reads very high, because the fake provider literally recycles your sentences. On a live run, a copy rate that high would tell you the model is quoting your samples, and you would discard the closure number. The guide row matches generic exactly, since the fake ignores the guide. That identical pair is a useful sanity check in its own right: when a real model’s guide row equals its generic row to two decimals, suspect that your system prompt never reached it. The JSON file keeps every row. The first generic row from that run looks like this, trimmed to the relevant fields: { "model": "fake", "condition": "generic", "brief": "b1", "repeat": 0, "words": 198, "length ratio": 0.566, "delta": 3.658, "surface gap": 7.840, "tells": 60.606, "copy": 0.0 } When the dry run passes, open config.toml, paste the exact model IDs from each vendor’s models page, and run the live comparison. Pick the same tier on each side, such as Sonnet 5.5 against GPT-6 Sol, then repeat with Opus 5.5 against GPT-6 Astra if the budget allows. python cli.py --models claude chatgpt cat out/report.md A live run prints the same table with two models and nine rows. Expect the generic rows to differ from each other, because each assistant has its own house style. The tells column may separate them as well, because stock phrases differ by vendor and by release. I have not published live numbers here on purpose. Your corpus decides them, and a table from someone else’s writing would tell you nothing about yours. Reading the Results: A Decision Rule You Can Defend Start with the controls, then the ratios. The noise floor tells you the best realistic Delta. Each model’s generic row tells you where it starts. Closure tells you how much of that distance a setup recovers. Only after that do you look at the cost per run from the first table. The wider AI and machine learning coverage https://shattered.io/category/ai-machine-learning/ on this site tracks the model releases that can invalidate a result, so check it before you trust a run older than a few weeks. What each column means Delta is the frequent-word distance, and lower is better. Surface is the average z-distance across sentence length, burstiness and punctuation rates. Tells counts stock phrases per 1,000 words. Copy is the share of 6-word sequences lifted from your profile documents. Len is generated words divided by the 350-word target. Closure is the share of the generic-to-floor gap that the setup closed. Turning the table into a choice | Pattern in your report | What it suggests | What to do next | |---|---|---| | Few-shot closure well above guide closure on both models | Examples carry your voice better than measured rules do | Ship the few-shot prompt and rotate the sample documents | | Guide closure within a few points of few-shot closure | Measured rules are enough | Use the guide, because it costs fewer prompt tokens | | High closure with copy above 5% | The model is quoting your samples | Discard the number, shorten the excerpts, retest | | Generic Delta already near the floor | The default voice is close to yours and closure prints n/a | Compare the surface and tells columns instead | | Closure swings widely between briefs | Unstable style adherence | Raise repeats and read the per-brief rows in the JSON | | Closure below zero | The setup pushed the model away from you | Check length ratio and whether the guide over-constrains | | Len far from 1.00 | The model ignored the length target | Fix length before comparing Delta, since it shifts word frequencies | The rule I would defend in a review is simple. Among the setups whose copy rate stays under 5 percent and whose length ratio stays between 0.8 and 1.2, pick the one with the highest closure. If two are within a few points of each other, pick the cheaper one. Those two thresholds are my own working numbers, not published standards, so tune them to your tolerance and write the values down before you see the results. Deciding the rule after the data is how people talk themselves into the model they already liked. When the winner differs by tier, that is a real finding. A cheap tier that closes most of the gap with a few-shot prompt can beat a flagship that needs more prompt engineering, and the price columns in the first table make the trade explicit. The same logic shows up at a larger scale in the Claude vs ChatGPT vs Gemini price gap analysis https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/ . Common Pitfalls That Skew a Voice Test Most bad stylometry results come from the setup, not the math. These are the traps I see most often. 1. Leaking held-out documents. If any held-out file ends up in a prompt, the model can echo it and Delta collapses toward the floor. The split is seeded and deterministic for this reason. Never hand-pick which files go into the prompts. 2. Mixing registers. A corpus of casual newsletters and formal reports has huge internal variance, so the sigma values balloon and every model looks closer to you than it is. Run one corpus per register. 3. Comparing different tiers. Setting Claude Opus 5.5 against GPT-6 Luna compares a flagship to a budget model, and the price table shows an order-of-magnitude gap. Match tiers first, then run cross-tier tests as a separate study. 4. Ignoring release dates. Both vendors shipped in October. GPT-6 reached ChatGPT on October 7, and according to OpenAI’s release notes the October 29 update changed its tone and retired GPT-5.5 from ChatGPT. Haiku 5.5 landed on October 7. Log the date and exact model ID with every run. 5. Reading Delta across corpora. A Delta of 1.4 on your writing and a Delta of 1.4 on a colleague’s are different distances. Compare closures and floors within one corpus, never raw Deltas between people. 6. Accepting short drafts. A 200-word reply to a 350-word request gives noisy frequencies and breaks fair comparison. Watch the len column and rerun any cell below 0.8. 7. Treating stylometry as AI detection. Delta measures distance to your habits. A low score does not prove a human wrote the text, and a high score does not prove a machine did. Never use this harness to accuse a person. 8. Trusting one run. Language models vary between calls. Two repeats on six briefs is a floor for exploration, not a verdict. Raise the repeat count before you publish a conclusion. Troubleshooting: 12 Errors and Fixes 1. SystemExit: Need at least 8 documents. Fewer than eight files cleared the length filter. Add more writing or lower min words in the config, but expect a noisier floor if you do. 2. set a real model id for claude . The config still holds a REPLACE WITH placeholder. Paste the exact ID from the vendor’s models page and rerun. 3. 401 or authentication errors. The key is missing, expired or exported in a different shell. Run echo with the variable name in the same terminal that runs the CLI. 4. 404 model not found. The ID is misspelled, not enabled for your account, or retired. OpenAI’s release notes show older models leaving ChatGPT, and API IDs follow their own schedule. 5. RuntimeError: empty completion. A reasoning model spent the whole output budget thinking. Raise max tokens in the config and delete the empty cache entry if one exists. 6. 429 rate limit errors that survive the retries. Lower repeats, run one model at a time with –models, or wait. Cached generations are not repeated, so progress is kept. 7. ModuleNotFoundError: No module named voicematch. You ran pytest or the CLI from the wrong directory, or init .py is missing. Run from the project root and confirm the empty file exists. 8. ModuleNotFoundError: tomllib. You are on Python 3.10 or older. Upgrade to 3.11 or newer. 9. Closure prints n/a for every row. The generic Delta sits within 0.05 of the floor. Either the model already writes like you at this resolution or your corpus is too uniform. Check the surface column. 10. Closure above 100 percent. The draft landed closer than your own leave-one-out floor. Usually that means copying, so check the copy column, or the held-out set is small and the floor is pessimistic. 11. Delta jumps wildly between reruns of the same prompt. Short outputs or a tiny corpus. Raise target words, add documents, and confirm the sigma floor is still in stylometry.py. 12. The guide row equals the generic row to two decimals. The system prompt did not reach the model, or the adapter dropped it. Print the prompt for one cell and read it. Advanced Tips for a Sharper Comparison Add confidence intervals. Averages hide spread. A bootstrap over the per-row deltas tells you whether a three-point closure gap is real or noise. This helper drops straight into report.py. python import random from statistics import mean def bootstrap ci values, n=2000, alpha=0.05, seed=7 : rng = random.Random seed means = sorted mean rng.choices values, k=len values for in range n return means int n alpha / 2 , means int n 1 - alpha / 2 - 1 Call it on the delta values of each model and condition, and print the interval beside the mean. When two intervals overlap heavily, call the result a tie. Test a combined condition. Add a fourth branch to build prompt that includes both the measured guide and the three excerpts. If it beats few-shot alone, you have evidence that rules and examples stack. Keep it as a separate condition so the original three stay comparable. Cache the long prefix. The few-shot system prompt is identical across briefs, which makes it a good fit for provider-side prompt caching. Anthropic lists a $0.10 per million cache-read price for Sonnet 5.5, and OpenAI lists cached-input rates for GPT-6 Sol and Luna. Caching changes cost, not output, so you can turn it on without invalidating earlier runs. Add a manual app arm. The API harness cannot see memory, projects or custom GPTs. Create one project or memory on each side with the same style notes, generate the six briefs by hand with the same wording, save each reply as a text file, and score them with a small script that calls burrows delta directly. Label that arm as app-layer results, never mix it into the API table. Layer a human check on the finalists. Stylometry is a filter. Once two setups look close, run a blind read of both. The blind test tutorial linked in the introduction already has the blinding code, so reuse it instead of rebuilding it. Add a third vendor. The adapter shape makes a Gemini arm a ten-line change. The three-vendor API test harness https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/ shows one way to wire it, and the same noise floor and closure math apply unchanged. The Complete Working Project The finished project is small enough to hold in your head. Every file below appeared in full above, so copy them in order and the tree will match. voicematch/ config.toml cli.py requirements.txt voicematch/ init .py textutils.py stylometry.py prompts.py providers.py runner.py report.py scripts/ make demo corpus.py tests/ test stylometry.py corpus/ your .txt files, 8 or more out/ results.json, report.md, cache/ The run order is the same every time. Install the packages, drop your writing into corpus/, run the tests, run the offline dry run, fill in the two model IDs, then run the live comparison. Save out/results.json, out/report.md, the pip freeze output and the date alongside each other. That bundle is your evidence, and it is the thing to attach if someone challenges the result. pip install anthropic openai pytest python -m pytest -q python cli.py --models fake python cli.py --models claude chatgpt pip freeze out/environment.txt FAQ: Claude vs ChatGPT for Writing in Your Voice Which is better for writing like me, Claude or ChatGPT? Neither vendor’s current official pages publish a dated, controlled comparison, so the honest answer depends on your corpus. Run the harness. The result can differ by setup, because one model may respond better to examples and the other to rules. How many documents do I need? Eight is the enforced minimum and twelve to twenty is the practical target. More documents make the noise floor and the sigma values more stable. Length matters too, so favor documents above 250 words. Does a low Delta mean the text was written by a human? No. Delta measures distance to your habits and nothing more. It is not an AI detector, and it should never be used to judge whether a person cheated. Can I test the ChatGPT and Claude apps instead of the APIs? You can, by hand. Generate the six briefs in each app with identical instructions, save the outputs as text files, and score them with burrows delta. Report those results separately, because app memory, projects and release-note changes sit outside the API arm. How often should I rerun the test? After every model release or announced behavior change, and at least monthly. GPT-6 reached ChatGPT on October 7, 2026, Haiku 5.5 shipped the same day, and OpenAI’s release notes describe an October 29 tone update, so a result from September is already history. Does this method work for judging code? No. Function-word statistics suit prose. For programming tasks, see the Claude vs ChatGPT for coding comparison https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/ , which looks at cost and capability instead. Is it safe to send my writing to an API? Read each vendor’s API data-handling terms before you start. The harness only sends the three profile documents, trimmed to excerpts, and never your held-out files. Leave out anything confidential.