Claude vs ChatGPT Voice Match: 12 Steps, 80 Min [2026]

Ask ten writers whether Claude or ChatGPT sounds more like them and you get ten confident answers and zero data. The Claude vs ChatGPT for writing debate gets simpler once you stop asking which model writes better and start asking which one lands closer to you. That distance is measurable, and the setup you give each model (nothing, a style guide, or a few of your own samples) moves it.

The lineups also changed under everyone’s feet. As of November 3, 2026, OpenAI’s pricing page has paid ChatGPT plans running GPT-6 Sol and the Free and Go plans running GPT-6 Luna, while Anthropic’s writing-relevant family is Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5. Pin those names, the plan and the date in every run, or your numbers will mean nothing next month.

Two sibling tutorials on this site cover neighboring questions. A blind rating test for Claude vs ChatGPT for writing asks humans to pick winners, and a ChatGPT vs Claude writing pipeline chains the models across drafting stages. Neither measures whether a draft sounds like one specific author. That gap is the target here, and the answer comes out as one percentage per model and per setup.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why Voice Match Needs Its Own Test #

Quality rubrics reward the same things in every writer. Clear structure, correct facts, no filler. A draft can ace all of that and still read like a stranger wrote it, which is a problem when the byline is yours. Voice lives in small habits: how long your sentences run, how often you reach for a parenthesis, whether you ask a question mid-paragraph, which little words you lean on without thinking.

Those little words are the key. Stylometry, the statistical study of writing style, has long used the most frequent words of a language (the, of, and, but, I) to tell authors apart, because writers pick them unconsciously and topic barely moves them. Burrows’ Delta turns that idea into one number. You z-score each frequent word against how much your own documents vary, then average the absolute distances. Lower means closer to you.

A raw Delta is hard to read alone, so the harness adds three anchors. The first is a noise floor, the Delta your own held-out documents score against each other in a leave-one-out loop. No model should be expected to beat you at being you. The second is each model’s generic output, produced with no style information, which marks where that model starts. The third is gap closure, the share of the distance between generic and floor that a setup closes. Zero percent means your style guide or samples did nothing. One hundred percent means the draft sits as close to you as you sit to yourself.

Two boundaries apply before you start. Stylometry measures texture, not quality, so a 90 percent closure on a muddled essay still gives you a muddled essay. And the numbers say nothing about which model is smarter. Neither vendor’s current official pages offer a dated, controlled writing comparison between GPT-6 and Claude, which is exactly why a local test beats a leaderboard. If you also want to grade factual fidelity and structure, pair this harness with the editing fidelity test from the same series.

Pin the November 2026 Lineups Before You Run Anything #

Anthropic’s current family for writing work has three tiers. Claude Opus 5.5 is the flagship for knowledge work, Sonnet 5.5 is the general-purpose workhorse, and Haiku 5.5 is the fast small model. Anthropic also cut the Sonnet 5.5 cache-read price to $0.10 per million tokens alongside the Haiku 5.5 release, which matters later if you reuse a long few-shot prompt across many briefs. List prices for every model in this test sit in the first table.

OpenAI’s side moved just as fast. GPT-6 reached ChatGPT on October 7, 2026, with Plus, Pro, Business and Enterprise on GPT-6 Sol and Free and Go on GPT-6 Luna. GPT-6 Pro, powered by GPT-6 Astra, is available on Pro 100, Pro 200, Business and Enterprise. According to OpenAI’s release notes, an October 29 update made GPT-6 in ChatGPT answer more concisely and flatter the user less. That is a style change, so any voice test you ran before that date measured a different assistant.

Model	Vendor	Role in this test	API input per 1M tokens	API output per 1M tokens
Claude Opus 5.5	Anthropic	Quality ceiling for hard drafts	$4	$20
Claude Sonnet 5.5	Anthropic	Main drafting baseline	$2	$10
Claude Haiku 5.5	Anthropic	Cheap high-volume tier	$0.10 (prompts up to 100,000 tokens)	$0.50 (prompts up to 100,000 tokens)
GPT-6 Sol	OpenAI	Paid-plan drafting baseline	$2	$10
GPT-6 Luna	OpenAI	Cheap high-volume tier	$0.10	$0.50
GPT-6 Astra	OpenAI	Top tier behind GPT-6 Pro	$10	$50

Pricing moved a lot this summer, so treat any older number you remember as stale. Anthropic made the Sonnet $2 and $10 rate permanent on August 10, 2026 and canceled a planned September increase, as sakuttoAI’s price-cut write-up explains. On the app side, Claude’s pricing page lists Claude Pro at $20 a month when billed monthly, with an annual option billed at $200 up front. Anthropic also announced on October 7 that Max subscribers get $100 in monthly Claude Platform API credits and Ultra subscribers get $200. A weekly roundup from an AI newsletter on note.com reported that Max 5x and Max 20x were renamed Max and Ultra on October 30, so check plan names on your own account before you cite them.

ChatGPT plan	Monthly price	What matters for a writing test
Free	$0	GPT-6 Luna, with limits on uploads, memory and context
Plus	$20	GPT-6 Sol, plus memory, projects, scheduled tasks and custom GPTs
Pro 100	$100	GPT-6 Pro (GPT-6 Astra) access
Pro 200	$200	More usage than Pro 100, open to new subscribers again
Pro 500	$500	Highest Pro allowance, adds GPT-6 Astra Ultrafast in Work and Codex
Business	$25 per user monthly, $20 per user billed annually	Standard seat, billed per user

One scope note keeps the results honest. The harness below calls the APIs, so it tests the models and not the app layer. Memory, projects and custom GPTs on the ChatGPT side, and projects on the Claude side, change the experience a paid subscriber gets. The Advanced Tips section shows how to add a manual arm for those. If plan costs are your real question, the breakdown of ChatGPT vs Claude pricing and usage limits goes deeper than this tutorial needs to.

Prerequisites and Versions #

Gather these before you touch a terminal. None of it is exotic, but a missing piece costs you an evening.

• Python 3.11 or newer. The config uses tomllib, which ships with the standard library from 3.11. This tutorial was run on Python 3.12.3.
• A POSIX shell. bash or zsh on Linux or macOS. On Windows, use WSL or translate the activation command.
• Two API keys. Export them as ANTHROPIC_API_KEY and OPENAI_API_KEY. Nothing else reads credentials.
• Latest releases of the anthropic and openai packages, plus pytest. No pins are listed because the code only touches the Messages API on one side and the Responses API on the other. Run pip freeze after installing and save the output with your results.
• Exact model IDs. Copy them from each vendor’s models page. The config ships with REPLACE_WITH placeholders on purpose, and the CLI refuses to run until you fill them in.
• Your own writing. At least 8 plain-text files of 250 words or more, ideally 12 to 20. Blog posts, newsletters and long emails work. Strip quoted text from other people, code blocks and boilerplate signatures.
• A small budget. A full run is 72 generations: 2 models, 3 setups, 6 briefs, 2 repeats. With the prices in the first table, that lands at a few dollars on the flagship tiers and well under a dollar on Sonnet 5.5 or GPT-6 Sol. Reasoning tokens can push that up, so set a spending cap on both consoles first.

The corpus is the one input that cannot be faked. Eight documents gives the noise floor only five or six held-out samples, which is thin. Twelve gives nine. If you write in several registers (casual newsletter and formal report, say), run the harness once per register, because mixing them inflates your own variance and makes every model look closer than it is.

Steps 1 and 2: Project Layout, Config and Corpus #

Step 1: scaffold the project. Create the folders, a virtual environment and the two environment variables. The empty init.py makes voicematch an importable package, and pytest relies on that.

mkdir voicematch && cd voicematch
mkdir -p voicematch tests scripts corpus out
touch voicematch/__init__.py
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic openai pytest
export ANTHROPIC_API_KEY="paste-your-key-here"
export OPENAI_API_KEY="paste-your-key-here"

Drop your writing into corpus/ as .txt files. If you only want to check the plumbing first, skip ahead to Step 12 and generate a synthetic demo corpus instead.

Step 2: write the config and the corpus . All tunable values live in one TOML file so a rerun a month from now uses identical settings. Three choices deserve a word. The profile_docs value of 3 means three of your documents are shown to the models (as examples and as the source of the derived style guide), and every remaining document is held out for scoring only. The seed makes that split reproducible. The tells list holds stock phrases you personally never write, which the report counts as a side signal.

[corpus]
dir = "corpus"
min_words = 250
min_docs = 8
profile_docs = 3
excerpt_words = 220
seed = 7

[stylometry]
mfw = 100
copy_ngram = 6

[run]
target_words = 350
repeats = 2
max_tokens = 4000
conditions = ["generic", "guide", "fewshot"]
tells = [
  "dive into", "game-changer", "unlock", "leverage", "elevate",
  "realm of", "journey", "at its core", "stands as", "cutting-edge",
]

[models.claude]
provider = "anthropic"
model = "REPLACE_WITH_CLAUDE_MODEL_ID"

[models.chatgpt]
provider = "openai"
model = "REPLACE_WITH_OPENAI_MODEL_ID"

[models.fake]
provider = "fake"
model = "fake-1"

The below does three jobs. It tokenizes into lowercase words with apostrophes kept, it splits text into sentences, and it enforces the corpus rules. The split is deterministic, so Claude and ChatGPT always see the same three profile documents. The excerpt helper trims each profile document to whole sentences under a word limit, which keeps few-shot prompts a predictable size.

import random
import re
from pathlib import Path

WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
SENT_RE = re.compile(r"(?<=[.!?])\s+")

def normalize(text: str) -> str:
    return text.replace("\u2019", "'").replace("\u2018", "'")

def words(text: str) -> list[str]:
    return WORD_RE.findall(normalize(text).lower())

def sentences(text: str) -> list[str]:
    parts = SENT_RE.split(normalize(text).strip())
    return [p.strip() for p in parts if len(words(p)) >= 2]

def load_corpus(directory: str, min_words: int, min_docs: int) -> dict[str, str]:
    docs = {}
    for path in sorted(Path(directory).glob("*.txt")):
        text = path.read_text(encoding="utf-8").strip()
        if len(words(text)) >= min_words:
            docs[path.name] = text
    if len(docs) < min_docs:
        raise SystemExit(
            f"Need at least {min_docs} documents of {min_words}+ words in "
            f"{directory}, found {len(docs)}."
        )
    return docs

def split_corpus(docs: dict[str, str], profile_n: int, seed: int):
    names = sorted(docs)
    random.Random(seed).shuffle(names)
    profile = {n: docs[n] for n in names[:profile_n]}
    holdout = {n: docs[n] for n in names[profile_n:]}
    return profile, holdout

def excerpt(text: str, max_words: int) -> str:
    out, count = [], 0
    for sent in sentences(text):
        n = len(words(sent))
        if count + n > max_words and out:
            break
        out.append(sent)
        count += n
    return " ".join(out)

Notice what the refuses to do. It will not quietly accept short files or a tiny corpus, because a silent fallback there would produce a noise floor that looks fine and means nothing. Fail loudly, fix the input, rerun.

Steps 3 to 5: Stylometry, Noise Floor and Gap Closure #

Step 3: extract features. Two kinds of signal come out of every text. The frequent-word profile feeds Burrows’ Delta. The surface profile captures mean sentence length, its coefficient of variation (a burstiness proxy), and punctuation rates per 1,000 words for commas, semicolons, long dashes, opening parentheses, question marks and colons. The long dash is written as a Unicode escape in the source so the file stays plain ASCII.

Step 4: build the reference and score with Delta. The reference takes the 100 most frequent words across your held-out documents, then records each word’s mean and standard deviation across those documents. Scoring a generated text means computing its frequency for the same words, z-scoring against your variation, and averaging the absolute values.

Step 5: add the controls. The noise floor, the AI-tell density, the copy rate and the closure formula all live in the same module. Read the whole file once, then the notes after it. The discipline here (frozen inputs, a written metric, controls before conclusions) mirrors what a good retrieval eval does, and the RAG pipeline with evals build shows the same habits on a different problem.

from collections import Counter
from dataclasses import dataclass
from statistics import mean, median, pstdev

from .textutils import normalize, sentences, words

PUNCT = {"comma": ",", "semicolon": ";", "dash": ", ", "paren": "(", "question": "?", "colon": ":"}

def rel_freq(tokens: list[str], vocab: list[str]) -> list[float]:
    counts, n = Counter(tokens), max(len(tokens), 1)
    return [counts[w] / n for w in vocab]

def surface(text: str) -> dict[str, float]:
    toks, sents = words(text), sentences(text)
    lengths = [len(words(s)) for s in sents] or [0]
    n = max(len(toks), 1)
    feats = {
        "sent_mean": mean(lengths),
        "sent_cv": pstdev(lengths) / max(mean(lengths), 1e-9),
    }
    norm = normalize(text)
    for name, char in PUNCT.items():
        feats[name] = 1000 * norm.count(char) / n
    return feats

@dataclass
class Reference:
    vocab: list[str]
    mu: list[float]
    sigma: list[float]
    surf_mu: dict[str, float]
    surf_sigma: dict[str, float]

def build_reference(texts: list[str], k: int) -> Reference:
    token_lists = [words(t) for t in texts]
    pooled = Counter(w for toks in token_lists for w in toks)
    vocab = [w for w, _ in pooled.most_common(k)]
    cols = list(zip(*[rel_freq(t, vocab) for t in token_lists]))
    mu = [mean(c) for c in cols]
    sigma = [max(pstdev(c), 0.1 * m, 1e-4) for c, m in zip(cols, mu)]
    feats = [surface(t) for t in texts]
    s_mu = {f: mean(d[f] for d in feats) for f in feats[0]}
    s_sd = {f: max(pstdev([d[f] for d in feats]), 0.05 * abs(s_mu[f]), 1e-3) for f in feats[0]}
    return Reference(vocab, mu, sigma, s_mu, s_sd)

def burrows_delta(text: str, ref: Reference) -> float:
    freq = rel_freq(words(text), ref.vocab)
    return mean(abs((x - m) / s) for x, m, s in zip(freq, ref.mu, ref.sigma))

def surface_gap(text: str, ref: Reference) -> float:
    feats = surface(text)
    return mean(abs((feats[f] - ref.surf_mu[f]) / ref.surf_sigma[f]) for f in feats)

def noise_floor(texts: list[str], k: int) -> float:
    scores = []
    for i, held in enumerate(texts):
        rest = texts[:i] + texts[i + 1:]
        scores.append(burrows_delta(held, build_reference(rest, k)))
    return median(scores)

def tell_density(text: str, tells: list[str]) -> float:
    low, n = normalize(text).lower(), max(len(words(text)), 1)
    return 1000 * sum(low.count(t) for t in tells) / n

def copy_rate(text: str, source_texts: list[str], n: int) -> float:
    def grams(toks):
        return {tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)}
    src = set().union(*[grams(words(t)) for t in source_texts])
    gen = grams(words(text))
    return len(gen & src) / len(gen) if gen else 0.0

def closure(generic: float, condition: float, floor: float):
    gap = generic - floor
    if gap < 0.05:
        return None
    return (generic - condition) / gap

Why the sigma floor exists

A frequent word that barely varies across your documents has a tiny standard deviation. Dividing by it turns a single extra occurrence in a 350-word draft into a z-score of 6 or 8, and one word dominates the average. The code puts a floor under each sigma so that cannot happen. The same idea protects the surface features. Without the floor, short corpora produce wild Deltas, and you will see the symptom in the troubleshooting list below.

Gap closure and its guard

Closure is (generic minus condition) divided by (generic minus floor). If a model’s generic output already sits within 0.05 Delta of your noise floor, the denominator collapses and the ratio explodes, so the function returns None and the report prints n/a. That is the correct answer in that case. It means the model’s default voice is already indistinguishable from yours at this resolution, and there is no gap left to close.

The copy rate counts how many 6-word sequences in a generated text also appear in your profile documents. Few-shot prompts invite copying. A model that parrots your sentences scores a great Delta and a terrible rating as a writing assistant, so the report prints the copy rate beside every closure.

Steps 6 and 7: Three Setups, One Brief #

Step 6: define the briefs and the derived guide. Six short briefs cover different registers: advice, narrative, argument, customer communication, reflection and opinion. They are deliberately unrelated to any specific corpus, because topic words leak into the frequent-word list. Edit them to suit your field, but keep them away from what you usually write about. If your corpus is all about databases, a brief about databases hands both models free vocabulary.

The style guide is generated from your profile documents by measuring them, not by asking a person to describe themselves. That choice is about fairness. A hand-written guide reflects how well you can articulate your own habits, and a clever phrase in it might happen to suit one model’s training. A measured guide is the same for both vendors, and it reruns identically next quarter.

Step 7: build the three conditions. The generic condition sends the brief with a one-line system prompt and nothing else. The guide condition adds the measured style notes to the system prompt. The fewshot condition adds three excerpts of your writing and asks for the voice but not the content. All three share the same user message, so the only variable is the system prompt.

from statistics import mean

from .stylometry import surface
from .textutils import excerpt

BRIEFS = [
    {"id": "b1", "topic": "why small teams should write down how they make decisions"},
    {"id": "b2", "topic": "what a first week at a new job actually feels like"},
    {"id": "b3", "topic": "the case for boring technology on a side project"},
    {"id": "b4", "topic": "how to explain a delayed launch to customers"},
    {"id": "b5", "topic": "lessons from a spreadsheet that grew out of control"},
    {"id": "b6", "topic": "why the best meeting is the one you skip"},
]

def derive_guide(profile_texts: list[str]) -> str:
    feats = [surface(t) for t in profile_texts]
    avg = {k: mean(f[k] for f in feats) for k in feats[0]}
    return "\n".join([
        f"- Average sentence length is about {avg['sent_mean']:.0f} words, "
        f"with a spread of {avg['sent_cv']:.2f} (std divided by mean). Mix short and long.",
        f"- Use roughly {avg['comma']:.0f} commas, {avg['paren']:.0f} opening parentheses "
        f"and {avg['question']:.0f} question marks per 1,000 words.",
        f"- Use roughly {avg['semicolon']:.0f} semicolons, {avg['colon']:.0f} colons "
        f"and {avg['dash']:.0f} long dashes per 1,000 words.",
        "- Write in plain words. No marketing phrases, no stock openings, no closing summary.",
    ])

def build_prompt(condition: str, brief: dict, profile: dict[str, str],
                 target_words: int, excerpt_words: int):
    task = (f"Write an essay of about {target_words} words on this topic: "
            f"{brief['topic']}. Return only the essay text, with no title and no preamble.")
    base = "You are a writing assistant."
    if condition == "generic":
        return base, task
    if condition == "guide":
        guide = derive_guide(list(profile.values()))
        return f"{base} Match this author's measured style:\n{guide}", task
    if condition == "fewshot":
        shots = "\n\n".join(
            f"EXAMPLE {i}:\n{excerpt(t, excerpt_words)}"
            for i, t in enumerate(profile.values(), 1)
        )
        system = (f"{base} Below are samples by one author. Match the voice, rhythm "
                  f"and punctuation habits. Do not reuse their sentences or subjects.\n\n{shots}")
        return system, task
    raise ValueError(f"unknown condition: {condition}")

If you already keep style notes inside a Claude project or a ChatGPT memory, you can test those too. Moving that context between assistants is its own task, and the guide to importing memory from ChatGPT into Claude covers it. For this harness, keep the system prompts above so both sides receive byte-identical instructions.

Steps 8 and 9: Provider Adapters, a Cache and an Offline Fake #

Step 8: wrap both SDKs behind one method. The Anthropic adapter calls the Messages API and joins the text blocks of the reply. The OpenAI adapter calls the Responses API with the system text as instructions and reads output_text. Neither call sets temperature. Some current reasoning-style models reject sampling parameters outright, and sending a value to one vendor but not the other would break the comparison. If you want to test temperature, do it as a separate experiment per model.

Both adapters take the same max_tokens budget, set to 4,000 in the config. That looks generous for a 350-word essay, and it should. Models that reason before answering spend part of the output budget on thinking, and a tight cap returns an empty string with no error. The empty-completion guard in the cache function exists because of exactly that failure.

Step 9: add a cache, retries and a fake provider. Every generation is stored under a hash of the provider, model, both prompts and the repeat index. Rerun the harness after changing only the report code and it costs nothing. The retry loop makes four attempts with exponential backoff, which absorbs ordinary rate-limit blips without hiding a bad key.

The fake provider is the piece that makes the whole project testable offline. For prompts without examples, it returns a deliberately generic paragraph stuffed with stock phrases. When the system prompt contains examples, it recycles your own sentences at random. It is not a model and its scores say nothing about Claude or ChatGPT. It proves the pipeline end to end, and it gives the unit tests something deterministic to chew on.

import hashlib
import json
import random
import re
import time
from pathlib import Path

from .textutils import sentences

GENERIC_FAKE = (
    "Many teams struggle with this topic. It is a journey that "
    "requires careful thought. Let us dive into the main points. First, clarity "
    "matters because it helps everyone move in the same direction. Second, "
    "consistency builds trust over time. Third, leverage the tools you already "
    "have to unlock better results. Overall, the benefits are clear and the "
    "approach is practical for almost any group of people."
)

class AnthropicProvider:
    def __init__(self, model: str):
        import anthropic
        self.client, self.model = anthropic.Anthropic(), model

    def generate(self, system: str, user: str, max_tokens: int) -> str:
        r = self.client.messages.create(
            model=self.model, max_tokens=max_tokens, system=system,
            messages=[{"role": "user", "content": user}],
        )
        return "".join(b.text for b in r.content if b.type == "text").strip()

class OpenAIProvider:
    def __init__(self, model: str):
        import openai
        self.client, self.model = openai.OpenAI(), model

    def generate(self, system: str, user: str, max_tokens: int) -> str:
        r = self.client.responses.create(
            model=self.model, instructions=system, input=user,
            max_output_tokens=max_tokens,
        )
        return (r.output_text or "").strip()

class FakeProvider:
    """Offline stand-in: generic prose, or recycled sentences when examples are present."""

    def __init__(self, model: str):
        self.model = model

    def generate(self, system: str, user: str, max_tokens: int) -> str:
        if "EXAMPLE 1:" not in system:
            return GENERIC_FAKE * 3
        pool = []
        for block in re.split(r"EXAMPLE \d+:\n", system)[1:]:
            pool += sentences(block.split("\n\n")[0])
        rng = random.Random(hashlib.sha256(user.encode()).hexdigest())
        return " ".join(rng.choice(pool) for _ in range(30))

def make_provider(spec: dict):
    kind = spec["provider"]
    if kind == "anthropic":
        return AnthropicProvider(spec["model"])
    if kind == "openai":
        return OpenAIProvider(spec["model"])
    if kind == "fake":
        return FakeProvider(spec["model"])
    raise ValueError(f"unknown provider: {kind}")

def cached_generate(provider, spec: dict, system: str, user: str, repeat: int,
                    max_tokens: int, cache_dir: Path) -> str:
    key = hashlib.sha256(json.dumps(
        [spec["provider"], spec["model"], system, user, repeat]).encode()).hexdigest()
    path = cache_dir / f"{key}.txt"
    if path.exists():
        return path.read_text(encoding="utf-8")
    for attempt in range(4):
        try:
            text = provider.generate(system, user, max_tokens)
            break
        except Exception:
            if attempt == 3:
                raise
            time.sleep(2 ** attempt)
    if not text:
        raise RuntimeError("empty completion: raise max_tokens or check the model id")
    cache_dir.mkdir(parents=True, exist_ok=True)
    path.write_text(text, encoding="utf-8")
    return text

The adapter layer is deliberately thin so you can add a third vendor in ten lines. If you later want to route requests between models by cost or task, the same shape underpins the Claude vs GPT router build.

Steps 10 and 11: The Runner and the Report #

Step 10: write the runner. It loads the config and the corpus, splits off the profile documents, builds the reference from the held-out texts, and computes the leave-one-out noise floor once. Then it loops over models, conditions, briefs and repeats. For every generation it records word count, length ratio, Delta, surface gap, tell density and copy rate. Everything lands in out/results.json, along with the controls, so a later script can re-aggregate without calling any API.

import json
import tomllib
from pathlib import Path

from .prompts import BRIEFS, build_prompt
from .providers import cached_generate, make_provider
from .stylometry import (build_reference, burrows_delta, copy_rate,
                         noise_floor, surface_gap, tell_density)
from .textutils import load_corpus, split_corpus, words

def load_config(path: str = "config.toml") -> dict:
    with open(path, "rb") as fh:
        return tomllib.load(fh)

def run(cfg: dict, model_keys: list[str], out_dir: str = "out") -> dict:
    c, s, r = cfg["corpus"], cfg["stylometry"], cfg["run"]
    docs = load_corpus(c["dir"], c["min_words"], c["min_docs"])
    profile, holdout = split_corpus(docs, c["profile_docs"], c["seed"])
    hold_texts, prof_texts = list(holdout.values()), list(profile.values())
    ref = build_reference(hold_texts, s["mfw"])
    floor = noise_floor(hold_texts, s["mfw"])
    out = Path(out_dir)
    rows = []
    for key in model_keys:
        spec = cfg["models"][key]
        provider = make_provider(spec)
        for cond in r["conditions"]:
            for brief in BRIEFS:
                system, user = build_prompt(cond, brief, profile, r["target_words"],
                                            c["excerpt_words"])
                for rep in range(r["repeats"]):
                    text = cached_generate(provider, spec, system, user, rep,
                                           r["max_tokens"], out / "cache")
                    n = len(words(text))
                    rows.append({
                        "model": key, "condition": cond, "brief": brief["id"],
                        "repeat": rep, "words": n,
                        "length_ratio": n / r["target_words"],
                        "delta": burrows_delta(text, ref),
                        "surface_gap": surface_gap(text, ref),
                        "tells": tell_density(text, r["tells"]),
                        "copy": copy_rate(text, prof_texts, s["copy_ngram"]),
                    })
    controls = {
        "floor": floor,
        "holdout_docs": len(hold_texts),
        "profile_docs": sorted(profile),
    }
    out.mkdir(exist_ok=True)
    (out / "results.json").write_text(
        json.dumps({"controls": controls, "rows": rows}, indent=2), encoding="utf-8")
    return {"controls": controls, "rows": rows}

One detail trips people up. The reference is built from held-out documents only, and the noise floor uses the same documents in a leave-one-out loop. Models never see a held-out document, so they cannot copy their way to a good score. That separation is the whole reason the project asks for at least eight files.

Step 11: write the report. The summary groups rows by model and condition, averages each metric, and computes closure against that model’s own generic mean and the shared noise floor. The Markdown table is what you paste into your notes. Keep the columns in this order, because the decision rule later in the tutorial reads left to right.

from collections import defaultdict
from statistics import mean

from .stylometry import closure

def summarize(result: dict) -> list[dict]:
    groups = defaultdict(list)
    for row in result["rows"]:
        groups[(row["model"], row["condition"])].append(row)
    floor = result["controls"]["floor"]
    summary = []
    for (model, cond), rows in sorted(groups.items()):
        generic = groups.get((model, "generic"))
        g = mean(x["delta"] for x in generic) if generic else None
        d = mean(x["delta"] for x in rows)
        summary.append({
            "model": model, "condition": cond, "n": len(rows),
            "delta": d,
            "surface_gap": mean(x["surface_gap"] for x in rows),
            "tells": mean(x["tells"] for x in rows),
            "copy": mean(x["copy"] for x in rows),
            "length_ratio": mean(x["length_ratio"] for x in rows),
            "closure": closure(g, d, floor) if g is not None else None,
        })
    return summary

def to_markdown(result: dict) -> str:
    floor = result["controls"]["floor"]
    lines = [
        f"Noise floor (your own held-out docs, leave-one-out): delta {floor:.2f}",
        "",
        "| model | condition | n | delta | surface | tells/1k | copy | len | closure |",
        "|---|---|---|---|---|---|---|---|---|",
    ]
    for s in summarize(result):
        c = "n/a" if s["closure"] is None else f"{s['closure']:.0%}"
        lines.append(
            f"| {s['model']} | {s['condition']} | {s['n']} | {s['delta']:.2f} | "
            f"{s['surface_gap']:.2f} | {s['tells']:.1f} | {s['copy']:.1%} | "
            f"{s['length_ratio']:.2f} | {c} |")
    return "\n".join(lines)

Step 12: Tests, the Dry Run and Your First Live Run #

Step 12: prove it works, then spend money. The test file covers tokenization, sentence splitting, the direction of Delta, tell counting, copy detection, the closure math and its guard, and an end-to-end dry run through the CLI. The end-to-end test builds a synthetic corpus in a temporary folder and never touches your real corpus directory. The demo corpus script produces twelve documents of templated sentences in one fixed voice, which is enough to exercise every code path.

import subprocess
import sys
from pathlib import Path

from voicematch.stylometry import (build_reference, burrows_delta, closure,
                                   copy_rate, tell_density)
from voicematch.textutils import sentences, words

A = "I wrote the plan (twice). It was short, and it worked. Why wait? " * 12
B = "Teams should leverage tools to unlock results and dive into the journey. " * 12

def test_words_handles_curly_apostrophes():
    assert words("It’s fine") == ["it's", "fine"]

def test_sentences_split():
    assert len(sentences("One two. Three four! Five six?")) == 3

def test_delta_prefers_same_voice():
    ref = build_reference([A, A + " Extra words here.", A.replace("plan", "note")], 20)
    assert burrows_delta(A, ref) < burrows_delta(B, ref)

def test_tell_density_counts():
    assert tell_density(B, ["leverage", "unlock"]) > 0
    assert tell_density(A, ["leverage", "unlock"]) == 0

def test_copy_rate_detects_verbatim():
    assert copy_rate(A, [A], 6) > 0.9
    assert copy_rate(B, [A], 6) == 0

def test_closure_math_and_guard():
    assert closure(3.0, 2.0, 1.0) == 0.5
    assert closure(1.02, 1.0, 1.0) is None

def test_end_to_end_dry_run(tmp_path):
    demo = tmp_path / "demo_corpus"
    subprocess.run([sys.executable, "scripts/make_demo_corpus.py", str(demo)], check=True)
    cfg = Path("config.toml").read_text().replace('dir = "corpus"', f'dir = "{demo}"')
    (tmp_path / "config.toml").write_text(cfg)
    done = subprocess.run(
        [sys.executable, "cli.py", "--config", str(tmp_path / "config.toml"),
         "--models", "fake", "--out", str(tmp_path / "out")],
        check=True, capture_output=True, text=True)
    assert "closure" in done.stdout
    assert (tmp_path / "out" / "results.json").exists()
python
import random
import sys
from pathlib import Path

THINGS = ["spreadsheet", "standup", "backlog", "release note", "coffee machine",
          "pull request", "roadmap", "calendar invite", "wiki page", "build script",
          "dashboard", "retro"]
TEMPLATES = [
    "I tested the {t} twice (once on purpose).",
    "The {t} is slower than it looks, but it is honest.",
    "Nobody asked about the {t}, so I wrote it down anyway.",
    "Why does the {t} always break on a Friday?",
    "My rule for the {t} is simple: fix it before it grows.",
    "A {t} is only useful if somebody reads it (and somebody rarely does).",
    "We argued about the {t} for an hour, then shipped it.",
    "The {t} taught me more than any course did.",
]

def main(target: str = "corpus"):
    rng = random.Random(11)
    pool = [tpl.format(t=t) for tpl in TEMPLATES for t in THINGS]
    Path(target).mkdir(parents=True, exist_ok=True)
    for i in range(1, 13):
        body = " ".join(rng.sample(pool, 40))
        Path(target, f"post{i:02d}.txt").write_text(body + "\n", encoding="utf-8")
    print(f"wrote 12 demo documents to {target}/")

if __name__ == "__main__":
    main(*sys.argv[1:2])

The command line ties everything together. It validates model keys, refuses to run while a model ID still starts with REPLACE_WITH, writes a Markdown report next to the JSON, and prints it.

import argparse
import sys

from voicematch.report import to_markdown
from voicematch.runner import load_config, run

def main() -> int:
    ap = argparse.ArgumentParser(description="Claude vs ChatGPT voice-match test")
    ap.add_argument("--config", default="config.toml")
    ap.add_argument("--models", nargs="+", default=["fake"],
                    help="keys from [models.*] in the config, e.g. claude chatgpt")
    ap.add_argument("--out", default="out")
    args = ap.parse_args()
    cfg = load_config(args.config)
    for key in args.models:
        if key not in cfg["models"]:
            sys.exit(f"unknown model key: {key}")
        if cfg["models"][key]["model"].startswith("REPLACE_WITH"):
            sys.exit(f"set a real model id for [{key}] in {args.config}")
    result = run(cfg, args.models, args.out)
    report = to_markdown(result)
    with open(f"{args.out}/report.md", "w", encoding="utf-8") as fh:
        fh.write(report + "\n")
    print(report)
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Run the tests from the project root, because the end-to-end test calls scripts and the CLI by relative path.

python -m pytest -q
python scripts/make_demo_corpus.py
python cli.py --models fake

Here is the real output from those commands on the demo corpus, captured on Python 3.12.3. The test run prints seven passing tests, and the dry run prints the report.

.......                                                                  [100%]
7 passed in 0.41s

Noise floor (your own held-out docs, leave-one-out): delta 1.06

| model | condition | n | delta | surface | tells/1k | copy | len | closure |
|---|---|---|---|---|---|---|---|---|
| fake | fewshot | 12 | 1.64 | 0.84 | 0.0 | 72.3% | 0.93 | 78% |
| fake | generic | 12 | 3.66 | 7.84 | 60.6 | 0.0% | 0.57 | 0% |
| fake | guide | 12 | 3.66 | 7.84 | 60.6 | 0.0% | 0.57 | 0% |

Read that table the way you will read a live one. The generic row sits well above the floor, so there is a clear gap to close. The fewshot row lands much nearer the floor, which closes most of that gap. But the copy column reads very high, because the fake provider literally recycles your sentences. On a live run, a copy rate that high would tell you the model is quoting your samples, and you would discard the closure number. The guide row matches generic exactly, since the fake ignores the guide. That identical pair is a useful sanity check in its own right: when a real model’s guide row equals its generic row to two decimals, suspect that your system prompt never reached it.

The JSON file keeps every row. The first generic row from that run looks like this, trimmed to the relevant fields:

{
  "model": "fake",
  "condition": "generic",
  "brief": "b1",
  "repeat": 0,
  "words": 198,
  "length_ratio": 0.566,
  "delta": 3.658,
  "surface_gap": 7.840,
  "tells": 60.606,
  "copy": 0.0
}

When the dry run passes, open config.toml, paste the exact model IDs from each vendor’s models page, and run the live comparison. Pick the same tier on each side, such as Sonnet 5.5 against GPT-6 Sol, then repeat with Opus 5.5 against GPT-6 Astra if the budget allows.

python cli.py --models claude chatgpt
cat out/report.md

A live run prints the same table with two models and nine rows. Expect the generic rows to differ from each other, because each assistant has its own house style. The tells column may separate them as well, because stock phrases differ by vendor and by release. I have not published live numbers here on purpose. Your corpus decides them, and a table from someone else’s writing would tell you nothing about yours.

Reading the Results: A Decision Rule You Can Defend #

Start with the controls, then the ratios. The noise floor tells you the best realistic Delta. Each model’s generic row tells you where it starts. Closure tells you how much of that distance a setup recovers. Only after that do you look at the cost per run from the first table. The wider AI and machine learning coverage on this site tracks the model releases that can invalidate a result, so check it before you trust a run older than a few weeks.

What each column means

Delta is the frequent-word distance, and lower is better. Surface is the average z-distance across sentence length, burstiness and punctuation rates. Tells counts stock phrases per 1,000 words. Copy is the share of 6-word sequences lifted from your profile documents. Len is generated words divided by the 350-word target. Closure is the share of the generic-to-floor gap that the setup closed.

Turning the table into a choice

Pattern in your report	What it suggests	What to do next
Few-shot closure well above guide closure on both models	Examples carry your voice better than measured rules do	Ship the few-shot prompt and rotate the sample documents
Guide closure within a few points of few-shot closure	Measured rules are enough	Use the guide, because it costs fewer prompt tokens
High closure with copy above 5%	The model is quoting your samples	Discard the number, shorten the excerpts, retest
Generic Delta already near the floor	The default voice is close to yours and closure prints n/a	Compare the surface and tells columns instead
Closure swings widely between briefs	Unstable style adherence	Raise repeats and read the per-brief rows in the JSON
Closure below zero	The setup pushed the model away from you	Check length ratio and whether the guide over-constrains
Len far from 1.00	The model ignored the length target	Fix length before comparing Delta, since it shifts word frequencies

The rule I would defend in a review is simple. Among the setups whose copy rate stays under 5 percent and whose length ratio stays between 0.8 and 1.2, pick the one with the highest closure. If two are within a few points of each other, pick the cheaper one. Those two thresholds are my own working numbers, not published standards, so tune them to your tolerance and write the values down before you see the results. Deciding the rule after the data is how people talk themselves into the model they already liked.

When the winner differs by tier, that is a real finding. A cheap tier that closes most of the gap with a few-shot prompt can beat a flagship that needs more prompt engineering, and the price columns in the first table make the trade explicit. The same logic shows up at a larger scale in the Claude vs ChatGPT vs Gemini price gap analysis.

Common Pitfalls That Skew a Voice Test #

Most bad stylometry results come from the setup, not the math. These are the traps I see most often.

1. Leaking held-out documents. If any held-out file ends up in a prompt, the model can echo it and Delta collapses toward the floor. The split is seeded and deterministic for this reason. Never hand-pick which files go into the prompts.
2. Mixing registers. A corpus of casual newsletters and formal reports has huge internal variance, so the sigma values balloon and every model looks closer to you than it is. Run one corpus per register.
3. Comparing different tiers. Setting Claude Opus 5.5 against GPT-6 Luna compares a flagship to a budget model, and the price table shows an order-of-magnitude gap. Match tiers first, then run cross-tier tests as a separate study.
4. Ignoring release dates. Both vendors shipped in October. GPT-6 reached ChatGPT on October 7, and according to OpenAI’s release notes the October 29 update changed its tone and retired GPT-5.5 from ChatGPT. Haiku 5.5 landed on October 7. Log the date and exact model ID with every run.
5. Reading Delta across corpora. A Delta of 1.4 on your writing and a Delta of 1.4 on a colleague’s are different distances. Compare closures and floors within one corpus, never raw Deltas between people.
6. Accepting short drafts. A 200-word reply to a 350-word request gives noisy frequencies and breaks fair comparison. Watch the len column and rerun any cell below 0.8.
7. Treating stylometry as AI detection. Delta measures distance to your habits. A low score does not prove a human wrote the text, and a high score does not prove a machine did. Never use this harness to accuse a person.
8. Trusting one run. Language models vary between calls. Two repeats on six briefs is a floor for exploration, not a verdict. Raise the repeat count before you publish a conclusion.

Troubleshooting: 12 Errors and Fixes #

1. SystemExit: Need at least 8 documents. Fewer than eight files cleared the length filter. Add more writing or lower min_words in the config, but expect a noisier floor if you do.
2. set a real model id for [claude]. The config still holds a REPLACE_WITH placeholder. Paste the exact ID from the vendor’s models page and rerun.
3. 401 or authentication errors. The key is missing, expired or exported in a different shell. Run echo with the variable name in the same terminal that runs the CLI.
4. 404 model not found. The ID is misspelled, not enabled for your account, or retired. OpenAI’s release notes show older models leaving ChatGPT, and API IDs follow their own schedule.
5. RuntimeError: empty completion. A reasoning model spent the whole output budget thinking. Raise max_tokens in the config and delete the empty cache entry if one exists.
6. 429 rate limit errors that survive the retries. Lower repeats, run one model at a time with –models, or wait. Cached generations are not repeated, so progress is kept.
7. ModuleNotFoundError: No module named voicematch. You ran pytest or the CLI from the wrong directory, or init.py is missing. Run from the project root and confirm the empty file exists.
8. ModuleNotFoundError: tomllib. You are on Python 3.10 or older. Upgrade to 3.11 or newer.
9. Closure prints n/a for every row. The generic Delta sits within 0.05 of the floor. Either the model already writes like you at this resolution or your corpus is too uniform. Check the surface column.
10. Closure above 100 percent. The draft landed closer than your own leave-one-out floor. Usually that means copying, so check the copy column, or the held-out set is small and the floor is pessimistic.
11. Delta jumps wildly between reruns of the same prompt. Short outputs or a tiny corpus. Raise target_words, add documents, and confirm the sigma floor is still in stylometry.py.
12. The guide row equals the generic row to two decimals. The system prompt did not reach the model, or the adapter dropped it. Print the prompt for one cell and read it.

Advanced Tips for a Sharper Comparison #

Add confidence intervals. Averages hide spread. A bootstrap over the per-row deltas tells you whether a three-point closure gap is real or noise. This helper drops straight into report.py.

import random
from statistics import mean

def bootstrap_ci(values, n=2000, alpha=0.05, seed=7):
    rng = random.Random(seed)
    means = sorted(mean(rng.choices(values, k=len(values))) for _ in range(n))
    return means[int(n * alpha / 2)], means[int(n * (1 - alpha / 2)) - 1]

Call it on the delta values of each model and condition, and print the interval beside the mean. When two intervals overlap heavily, call the result a tie.

Test a combined condition. Add a fourth branch to build_prompt that includes both the measured guide and the three excerpts. If it beats few-shot alone, you have evidence that rules and examples stack. Keep it as a separate condition so the original three stay comparable.

Cache the long prefix. The few-shot system prompt is identical across briefs, which makes it a good fit for provider-side prompt caching. Anthropic lists a $0.10 per million cache-read price for Sonnet 5.5, and OpenAI lists cached-input rates for GPT-6 Sol and Luna. Caching changes cost, not output, so you can turn it on without invalidating earlier runs.

Add a manual app arm. The API harness cannot see memory, projects or custom GPTs. Create one project or memory on each side with the same style notes, generate the six briefs by hand with the same wording, save each reply as a text file, and score them with a small script that calls burrows_delta directly. Label that arm as app-layer results, never mix it into the API table.

Layer a human check on the finalists. Stylometry is a filter. Once two setups look close, run a blind read of both. The blind test tutorial linked in the introduction already has the blinding code, so reuse it instead of rebuilding it.

Add a third vendor. The adapter shape makes a Gemini arm a ten-line change. The three-vendor API test harness shows one way to wire it, and the same noise floor and closure math apply unchanged.

The Complete Working Project #

The finished project is small enough to hold in your head. Every file below appeared in full above, so copy them in order and the tree will match.

voicematch/
  config.toml
  cli.py
  requirements.txt
  voicematch/
    __init__.py
    textutils.py
    stylometry.py
    prompts.py
    providers.py
    runner.py
    report.py
  scripts/
    make_demo_corpus.py
  tests/
    test_stylometry.py
  corpus/        your .txt files, 8 or more
  out/           results.json, report.md, cache/

The run order is the same every time. Install the packages, drop your writing into corpus/, run the tests, run the offline dry run, fill in the two model IDs, then run the live comparison. Save out/results.json, out/report.md, the pip freeze output and the date alongside each other. That bundle is your evidence, and it is the thing to attach if someone challenges the result.

pip install anthropic openai pytest
python -m pytest -q
python cli.py --models fake
python cli.py --models claude chatgpt
pip freeze > out/environment.txt

FAQ: Claude vs ChatGPT for Writing in Your Voice #

Which is better for writing like me, Claude or ChatGPT?

Neither vendor’s current official pages publish a dated, controlled comparison, so the honest answer depends on your corpus. Run the harness. The result can differ by setup, because one model may respond better to examples and the other to rules.

How many documents do I need?

Eight is the enforced minimum and twelve to twenty is the practical target. More documents make the noise floor and the sigma values more stable. Length matters too, so favor documents above 250 words.

Does a low Delta mean the text was written by a human?

No. Delta measures distance to your habits and nothing more. It is not an AI detector, and it should never be used to judge whether a person cheated.

Can I test the ChatGPT and Claude apps instead of the APIs?

You can, by hand. Generate the six briefs in each app with identical instructions, save the outputs as text files, and score them with burrows_delta. Report those results separately, because app memory, projects and release-note changes sit outside the API arm.

How often should I rerun the test?

After every model release or announced behavior change, and at least monthly. GPT-6 reached ChatGPT on October 7, 2026, Haiku 5.5 shipped the same day, and OpenAI’s release notes describe an October 29 tone update, so a result from September is already history.

Does this method work for judging code?

No. Function-word statistics suit prose. For programming tasks, see the Claude vs ChatGPT for coding comparison, which looks at cost and capability instead.

Is it safe to send my writing to an API?

Read each vendor’s API data-handling terms before you start. The harness only sends the three profile documents, trimmed to excerpts, and never your held-out files. Leave out anything confidential.

source & further reading

shattered.io — original article Fake Job Interview Led to Binance’s $6.3B Loss [2026] Gemini 4 Carbon Tops AA Index at 59, $2 Input [2026] AWS KMS vs Azure vs Google KMS: $1 Keys, PQC [2026]