ChatGPT vs Claude README Test: 12 Steps, 80 Min [2026]

A 12-step, roughly 80-minute harness published by Shattered tests whether ChatGPT and Claude invent facts when writing README documentation, sending the same prompt to each model five times and executing every flag and command the generated READMEs contain against a real tool called logtrim. The test targets errors such as a fabricated --verbose flag, a pip install line for a nonexistent package, or a default value wrong by a factor of two, and the author declines to name a winner, noting outcomes swing with the exact model, plan and day. The article notes the ChatGPT model selector now shows GPT-6 where it previously showed Latest, while users who explicitly selected GPT-5.6 Sol keep that model.

On the OpenAI side, the ChatGPT model selector now shows GPT-6 where it used to show Latest, and SyncAI’s AI news roundup https://syncai.news/article/ainews-it-s-swapping-season-0f7a5a reports that anyone who explicitly picked GPT-5.6 Sol keeps that model. Both product lines moved in recent weeks, so a blog post that declares a winner for writing has a short shelf life. A test you can rerun on the day you read this lasts longer. The cleanest writing test for a developer is not a style contest. It is a README. A README makes claims you can check by running code: which flags exist, what the defaults are, which commands work, which files are there. Earlier tests probed writing rules compliance https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ and voice matching https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ . A README accuracy test goes after a different failure, the confident invention of facts in documentation. The method is small. Send the same prompt to ChatGPT and Claude five times each, pull every flag and command out of the READMEs they write, execute those commands against a real tool, and count errors and gaps per model. Naming a winner here would be premature. Outcomes swing with the exact model, the plan and the day, and the harness exists to settle the question for your own pair. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why a README Test Beats a Style Contest for ChatGPT vs Claude Most ChatGPT vs Claude comparisons for writing rest on taste. One reviewer likes warmer prose, another likes tighter paragraphs, and both are right for themselves. Taste does not transfer to your repository. Documentation is different because a reader acts on it. A wrong flag costs the reader ten minutes, and a wrong install line costs them trust in everything else on the page. A README also forces both models into the same job with the same raw material. You hand over the source files and ask for documentation. The ground truth sits in the code, so there is no need for a human to decide who wrote a better sentence. The logtrim tool used here is small enough to read in two minutes, yet it has enough surface to catch mistakes: five options, a short flag, a stdin convention, a default value, a special value that disables a limit, and three distinct exit codes. The failure modes are familiar to anyone who has used a chatbot on a codebase. Models add a plausible --verbose flag that the tool never had. They write a pip install line for a package that does not exist on any index. They describe a config file because tools like this one usually have one. They get a default wrong by a factor of two. Each of those errors is checkable by a script, which means you can count them across fifty runs without reading a single paragraph. The same method extends the source-grounded writing audit https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/ from prose claims to executable ones. Where a source audit asks whether a sentence matches a cited page, this test asks whether a command exits with the code the README promised. Executable claims are the hardest kind to argue with. One more reason matters for a practical reader. In many teams, documentation is a chore that gets handed to a chatbot and reviewed lightly. People skim a generated README, see headings that look right, and merge it. A harness that fails the build on an invented flag turns that blind spot into a gate. Prerequisites and Versions Set these up before you start. None of it is exotic, and the whole stack uses the standard library plus two vendor SDKs. - Python 3.12 or newer. The project in this tutorial runs on Python 3.12.3. - pip and the built-in venv module. - The anthropic and openai Python packages, installed at their latest versions. The OpenAI call uses the Responses API, so an old SDK that lacks responses will fail. - An OpenAI API key and an Anthropic API key, both with billing enabled. - Two model IDs, one per vendor, stored in environment variables. Step 1 explains how to pick them. - Linux or macOS. On Windows, run everything inside WSL, because the scorer launches Python subprocesses and the shell snippets assume a POSIX shell. - About 80 minutes, with roughly 10 of them going to API wait time. The default run makes 10 API calls, five per model, and each call sends a few hundred lines of source code. On any current plan the bill stays small. If you want exact numbers for your own pair, the harness logs input and output tokens for every call, so you can multiply by the prices on the vendors’ pages. For a longer treatment of cost per accepted output, see the writing cost per 1,000 accepted words https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/ test. You need no GPU, no database and no web framework. You also need no prior experience with either SDK beyond reading an API key from an environment variable. What the Harness Measures The scorer reads each README as text and runs six checks against the real project. Four of them detect errors, meaning the README says something false. Two detect gaps, meaning the README leaves out something a reader needs. | Check | What it catches | How it counts | |---|---|---| | Invented flags | A flag in the README that --help does not list | One error per unknown flag | | Command execution | An example that fails when you run it | One error per command that exits with anything but 0 or 1 | | File references | A backticked file name the repository does not contain | One error per unknown file | | Forbidden claims | Install routes and config files the tool does not have | One error per matching pattern | | Required sections | Missing install, usage, options, exit codes or examples headings | One gap per missing section | | Fact probes | A default, format or exit code the README never states | One gap per missed probe | Exit code 1 counts as a pass because the tool uses it for no matches, a legitimate outcome for an example such as a search that finds nothing. Exit code 2 means invalid input, so an example that triggers it is wrong. Errors versus gaps Keep the two counts separate. A short README with no mistakes has gaps but no errors, and it will never mislead anyone. A long README with a confident invented flag has errors, and that is the worse outcome in practice. Most teams should weigh errors far above gaps, which is why the report prints them in separate columns instead of merging them into one grade. The accuracy percentage still blends them so you can sort a long list quickly, but the decision rule in Step 12 looks at errors first. Why no model judge scores the prose It is tempting to ask a third model to grade the READMEs. Resist that for this test. A judge model can share blind spots with one of the contestants, it adds a new variable you cannot pin down, and nothing external can confirm its verdict. Every number this harness produces comes from running code or matching a pattern, so you can reproduce it. The human part stays human: Step 10 adds a blind read where you rate clarity yourself, with the model names hidden. Steps 1 and 2: Pin the Models and Create the Project Step 1: choose and record the exact models. The product names moved this month, so write down what you actually call. In the ChatGPT app, GPT-6 now sits where Latest used to sit in the selector. GPT-6 uses GPT-6 Sol behind its Instant, Medium, High and Extra High settings, according to OpenAI’s help documentation. The API is a different surface. GPT-6.1 Terra entered the OpenAI API on October 26 under the ID gpt-6.1-terra , listed at $1 per million input tokens and $5 per million output tokens. For a clean test, use the API on both sides, because the apps add system prompts, memory and tool routing you cannot control. | Model | Maker | What the current record confirms | |---|---|---| | GPT-6 | OpenAI | Replaced Latest in the ChatGPT model selector, served by GPT-6 Sol at four reasoning settings | | GPT-6.1 Terra | OpenAI | In the API since October 26, 2026 as gpt-6.1-terra , $1 input and $5 output per million tokens | | GPT-5.6 Sol | OpenAI | Still selectable when chosen explicitly after GPT-6 replaced Latest | | Claude Sonnet 5.5 | Anthropic | Anthropic says it runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work | | Claude Opus 5.5 | Anthropic | Anthropic says it leads in agentic coding and knowledge work and costs 40% less to run than Opus 5 on typical workloads | | Claude Fable 5.2 | Anthropic | Released November 9, 2026 at $8 input and $40 output per million tokens, down from $10 and $50 for Fable 5.1 | | Claude Haiku 5.5 | Anthropic | Announced October 7, 2026 as the fastest small model, aimed at high-volume work | | Claude Mythos 5.2 | Anthropic | Restricted to vetted cyberdefenders and life-sciences researchers, so it is not an option here | Pair models by what you pay, not by name. If you run Claude Sonnet 5.5 against GPT-6.1 Terra, you are comparing two models with callable API IDs that vendors position for everyday work. If you want a premium pair, add Claude Fable 5.2 on one side and a higher OpenAI tier on the other, and record the price of each README as well as its score. Plan prices change often, so check the live pages. As of early November, OpenAI’s pricing page https://chatgpt.com/pricing reportedly listed ChatGPT Plus at $20 a month in the US, while Anthropic keeps current Claude plans on its own pricing page https://claude.com/pricing . The deeper breakdown of tiers lives in the ChatGPT vs Claude pricing and usage limits https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ comparison. Ask each vendor for its model list so you copy IDs rather than guess them: curl -s https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI API KEY" | python3 -m json.tool | grep '"id"' curl -s https://api.anthropic.com/v1/models -H "x-api-key: $ANTHROPIC API KEY" -H "anthropic-version: 2023-06-01" | python3 -m json.tool | grep '"id"' Step 2: create the project and the environment. Make a folder, a virtual environment and the directory layout the later steps assume. mkdir readme-bench && cd readme-bench python3 -m venv .venv && source .venv/bin/activate pip install anthropic openai mkdir -p target prompts runs fixtures/clean fixtures/flawed export OPENAI API KEY="paste-your-key" export ANTHROPIC API KEY="paste-your-key" export OPENAI MODEL="gpt-6.1-terra" export ANTHROPIC MODEL="paste-an-id-from-the-models-list" Write the four exports into a file named .env.sh and source it each session. A forgotten variable is the most common reason the first run fails, and the failure message from Python is a bare KeyError that tells you little. Steps 3 and 4: Build the Target Project and Its Ground Truth Step 3: write the project the models will document. logtrim reads a log file and keeps the lines you ask for. It is deliberately boring. Boring code has an unambiguous right answer, and that is what a ground truth needs. Save this as target/logtrim.py : bash /usr/bin/env python3 """logtrim: cut a log file down to the lines you care about.""" import argparse import json import re import sys from datetime import datetime DEFAULT MAX LINES = 500 TS FORMAT = "%Y-%m-%dT%H:%M:%S" def build parser : p = argparse.ArgumentParser prog="logtrim", description="Trim a log file by pattern, start time and line count.", p.add argument "path", help="log file to read, or - for stdin" p.add argument "-p", "--pattern", help="keep only lines matching this regex" p.add argument "--since", help="keep lines at or after YYYY-MM-DDTHH:MM:SS" p.add argument "--max-lines", type=int, default=DEFAULT MAX LINES, help="keep at most this many lines the newest , 0 means no limit", p.add argument "--json", action="store true", help="print one JSON object per line" p.add argument "--dry-run", action="store true", help="print counts, not lines" return p def read lines path : if path == "-": return sys.stdin.read .splitlines with open path, encoding="utf-8", errors="replace" as fh: return fh.read .splitlines def line time line : try: return datetime.strptime line :19 , TS FORMAT except ValueError: return None def trim lines, pattern=None, since=None, max lines=DEFAULT MAX LINES : rx = re.compile pattern if pattern else None kept = for line in lines: if rx and not rx.search line : continue if since is not None: ts = line time line if ts is None or ts < since: continue kept.append line return kept -max lines: if max lines 0 else kept def main argv=None : args = build parser .parse args argv since = None if args.since: try: since = datetime.strptime args.since, TS FORMAT except ValueError: print "logtrim: --since must look like 2026-11-10T09:00:00", file=sys.stderr return 2 try: lines = read lines args.path except OSError as exc: print f"logtrim: {exc}", file=sys.stderr return 2 try: kept = trim lines, args.pattern, since, args.max lines except re.error as exc: print f"logtrim: bad pattern: {exc}", file=sys.stderr return 2 if args.dry run: print f"read={len lines } kept={len kept }" elif args.json: for n, line in enumerate kept, 1 : print json.dumps {"n": n, "line": line} else: for line in kept: print line return 0 if kept else 1 if name == " main ": sys.exit main Add a nine-line sample log at target/sample.log , which every example command in the READMEs must use: 2026-11-10T08:59:41 INFO service started 2026-11-10T09:00:03 INFO health check ok 2026-11-10T09:04:17 WARN disk at 81 percent 2026-11-10T09:15:02 ERROR upstream timeout after 30s 2026-11-10T09:15:09 ERROR upstream timeout after 30s 2026-11-10T09:31:44 INFO health check ok 2026-11-10T10:02:12 WARN disk at 86 percent 2026-11-10T10:20:55 ERROR payment webhook rejected 2026-11-10T11:00:00 INFO health check ok Add a short target/LICENSE file with any license text you like. The scorer treats it as a known file, so a README that mentions it passes the file check. Run the tool by hand before you trust it. These four commands show every exit path the READMEs have to describe: cd target python3 logtrim.py sample.log --since 2026-11-10T10:00:00 --max-lines 2 python3 logtrim.py sample.log --dry-run python3 logtrim.py sample.log -p NOPE ; echo "exit=$?" python3 logtrim.py nope.log ; echo "exit=$?" cd .. The first command prints the two newest lines from 10:20 and 11:00. The second prints read=9 kept=9 . The third prints nothing and exits 1. The fourth prints a No such file or directory message and exits 2. Step 4: encode the ground truth. The scorer needs a list of things a good README must contain and things it must never claim. Save this as truth.json in the project root: { "sections": {"id": "install", "pattern": "install|requirements|getting started|setup"}, {"id": "usage", "pattern": "usage|quick ?start"}, {"id": "options", "pattern": "options|flags|arguments|reference"}, {"id": "exit-codes", "pattern": "exit codes?|return codes?"}, {"id": "examples", "pattern": "examples?"} , "probes": {"id": "default-500", "pattern": "max-lines ^\\n {0,160}\\b500\\b|\\b500\\b ^\\n {0,160}max-lines"}, {"id": "since-format", "pattern": "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}|YYYY-MM-DDTHH:MM:SS"}, {"id": "exit-1-no-match", "pattern": "\\b1\\b ^\\n {0,80} no lines|nothing|no match|empty|zero | no lines|nothing|no match|empty|zero ^\\n {0,80}\\b1\\b"}, {"id": "exit-2-bad-input", "pattern": "\\b2\\b ^\\n {0,80} invalid|bad|missing|error|unreadable|not found|cannot | invalid|bad|missing|error|unreadable|not found|cannot ^\\n {0,80}\\b2\\b"}, {"id": "stdin-dash", "pattern": "stdin|standard input"}, {"id": "zero-no-limit", "pattern": "\\b0\\b ^\\n {0,80} no limit|unlimited|disable | no limit|unlimited|disable ^\\n {0,80}\\b0\\b"} , "forbidden": {"id": "pip-install-logtrim", "pattern": "pip3? install ^\\n logtrim"}, {"id": "pypi", "pattern": "pypi"}, {"id": "docker", "pattern": "docker run|build "}, {"id": "brew", "pattern": "brew install"}, {"id": "npm", "pattern": "npm install|i "}, {"id": "config-file", "pattern": "\\.logtrimrc|logtrim\\. yaml|yml|toml|conf "}, {"id": "windows-binary", "pattern": "logtrim\\.exe"} } The forbidden list is where the invented claims live. The tool is a single file with no packaging metadata, so any pip install logtrim line is false by construction. The tool has no configuration file, no Docker image and no Windows binary, so those patterns are false too. The probes list holds the facts a reader needs, each as a regular expression that must match somewhere in the text. Keep the patterns loose. A probe that demands one exact phrase punishes a model for wording, and you want to punish it for omissions. Step 5: Write One Prompt Both Models Share Fairness lives in the prompt. Both models receive the same system instruction, the same user message and the same repository snapshot, character for character. The only difference is the vendor endpoint. Save the system prompt as prompts/system.txt : You are a technical writer. Write a README.md for the project the user shows you. Use only facts that appear in the files provided. If something is not in the files, leave it out. Return the README as plain Markdown with no preamble and no wrapping code fence. And the user message as prompts/user.txt . The generator replaces the {snapshot} placeholder with every file in target/ except any existing README: Write the README.md for this project. Cover installation, usage, every option, exit codes and examples. Use sample.log in every example command. Here are all the files in the repository: {snapshot} Three details in these prompts matter more than they look. The instruction to use only facts from the files gives both models a clear rule to break, which is what you want to measure. The instruction to use sample.log in every example makes command execution testable, because a README that points at app.log would fail for a reason unrelated to accuracy. And the instruction to return plain Markdown without a wrapping fence saves you from a parsing step that varies by model. Some models wrap anyway, and the generator strips a markdown fence if it finds one. Do not add a persona, a tone request or formatting tips. Those would make the test a style contest again. A bare prompt shows what each model does by default, which is what most people get when they paste code into a chat window and ask for docs. If you later want to test how much a better prompt helps, change the prompt for both models at once and rerun, as the blind writing test https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ does for prose. Step 6: Generate Five READMEs per Model Five runs per model is the minimum that tells you anything about spread. A single run can land on a lucky or unlucky sample, and neither says much about the model. Save this as generate.py : bash /usr/bin/env python3 """Ask both models for a README, N times each, with identical prompts.""" import json import os import time from pathlib import Path from anthropic import Anthropic from openai import OpenAI ROOT = Path file .parent TARGET = ROOT / "target" RUNS = int os.environ.get "RUNS", "5" MODELS = { "chatgpt": os.environ "OPENAI MODEL" , "claude": os.environ "ANTHROPIC MODEL" , } SYSTEM = ROOT / "prompts" / "system.txt" .read text USER = ROOT / "prompts" / "user.txt" .read text def snapshot : parts = for p in sorted TARGET.rglob " " : if p.is file and p.name = "README.md": body = p.read text encoding="utf-8", errors="replace" parts.append f"=== {p.relative to TARGET } ===\n{body}" return "\n\n".join parts def ask chatgpt model, system, user : r = OpenAI .responses.create model=model, instructions=system, input=user return r.output text, r.usage.input tokens, r.usage.output tokens def ask claude model, system, user : r = Anthropic .messages.create model=model, max tokens=4000, system=system, messages= {"role": "user", "content": user} , text = "".join b.text for b in r.content if b.type == "text" return text, r.usage.input tokens, r.usage.output tokens def unwrap text : t = text.strip if t.startswith " markdown" and t.endswith " " : t = t len " markdown" :-3 .strip return t + "\n" def main : user = USER.format snapshot=snapshot askers = {"chatgpt": ask chatgpt, "claude": ask claude} meta = ROOT / "runs" / "meta.jsonl" .open "a", encoding="utf-8" for label, model in MODELS.items : out = ROOT / "runs" / label out.mkdir parents=True, exist ok=True for i in range 1, RUNS + 1 : start = time.time text, tin, tout = askers label model, SYSTEM, user out / f"run {i:02d}.md" .write text unwrap text , encoding="utf-8" meta.write json.dumps { "label": label, "model": model, "run": i, "seconds": round time.time - start, 1 , "input tokens": tin, "output tokens": tout, } + "\n" meta.flush print f"{label} run {i}/{RUNS} done" if name == " main ": main The two ask functions use each vendor’s current SDK shape. OpenAI’s Responses API takes instructions and input and exposes output text . Anthropic’s Messages API takes system and a message list and returns content blocks, so the code joins the text blocks. The script sets no sampling parameters. Sampling parameters differ across models and API versions, so the script leaves them at vendor defaults and both calls stay valid. Every call appends a line to runs/meta.jsonl with the model ID, elapsed seconds and token counts. That file is the raw material for cost and latency, and it records which exact model produced which README, an audit trail that matters when vendors rename models mid-month. Run it: python generate.py Expected console output, one line per call: chatgpt run 1/5 done chatgpt run 2/5 done chatgpt run 3/5 done chatgpt run 4/5 done chatgpt run 5/5 done claude run 1/5 done claude run 2/5 done claude run 3/5 done claude run 4/5 done claude run 5/5 done The loop runs sequentially on purpose. Parallel calls would finish faster but trigger rate limits on lower tiers, and a failed run halfway through would leave you with uneven sample sizes. If a call raises, fix the cause and rerun the whole script after clearing runs/ , so both models keep five clean samples. The labels chatgpt and claude become folder names under runs/ , and the scorer later reads those folder names as model labels. Steps 7 and 8: Write the Scorer, Then Test It on Fixtures Step 7: write the scorer. This is the core of the project. It extracts flags and commands from the README, runs the commands in a throwaway copy of the target folder, and tallies errors and gaps. Save it as score.py : bash /usr/bin/env python3 """Score README files against the real logtrim project.""" import argparse import json import re import shlex import shutil import subprocess import sys import tempfile from collections import defaultdict from pathlib import Path ROOT = Path file .parent TARGET = ROOT / "target" TRUTH = json.loads ROOT / "truth.json" .read text FENCE = re.compile r" a-zA-Z0-9 - \n . ? ", re.S INLINE = re.compile r" ?< ^ \n + ? " LONG FLAG = re.compile r" ?< \w- -- a-z a-z0-9- " SHORT FLAG = re.compile r" ?< \w- - a-zA-Z ? \w- " PATH REF = re.compile r" \w./- +\. ?:py|log|md|toml|txt|cfg|json|ya?ml " SHELL META = re.compile r" |&;< $ " HEADING = re.compile r"^ {1,6}\s+ . $", re.M def real flags : out = subprocess.run sys.executable, "logtrim.py", "--help" , cwd=TARGET, capture output=True, text=True, timeout=10, .stdout return set LONG FLAG.findall out | set SHORT FLAG.findall out def code lines md : blocks = ln for b in FENCE.findall md for ln in b.splitlines inline = INLINE.findall FENCE.sub "", md return blocks + inline def flags in lines : found = set for ln in lines: s = ln.strip if "logtrim" in s or s.startswith "-" : found |= set LONG FLAG.findall s | set SHORT FLAG.findall s return found def commands md : cmds = for b in FENCE.findall md : for ln in b.splitlines : s = re.sub r"^ $ \s+", "", ln.strip if s.startswith "logtrim ", "python logtrim.py", "python3 logtrim.py" : cmds.append re.split r"\s+ ", s 0 .strip return cmds def run command cmd : if SHELL META.search cmd or " " in cmd: return None try: argv = shlex.split cmd except ValueError: return 2 skip = 1 if argv 0 == "logtrim" else 2 full = sys.executable, "logtrim.py" + argv skip: with tempfile.TemporaryDirectory as tmp: repo = Path tmp / "repo" shutil.copytree TARGET, repo try: r = subprocess.run full, cwd=repo, capture output=True, text=True, timeout=10, stdin=subprocess.DEVNULL, except subprocess.TimeoutExpired: return 124 return r.returncode def score md, real : mentioned = flags in code lines md bad flags = sorted mentioned - real results = c, run command c for c in commands md ran = c, rc for c, rc in results if rc is not None failed = c for c, rc in ran if rc not in 0, 1 known = {p.name for p in TARGET.rglob " " if p.is file } refs = set PATH REF.findall md - {"README.md"} bad refs = sorted r for r in refs if Path r .name not in known forbidden = f "id" for f in TRUTH "forbidden" if re.search f "pattern" , md, re.I heads = " | ".join HEADING.findall md miss sections = s "id" for s in TRUTH "sections" if not re.search s "pattern" , heads, re.I miss probes = p "id" for p in TRUTH "probes" if not re.search p "pattern" , md, re.I | re.S total = len mentioned + len ran + len refs + len TRUTH "forbidden" + len TRUTH "sections" + len TRUTH "probes" errors = len bad flags + len failed + len bad refs + len forbidden gaps = len miss sections + len miss probes return { "words": len md.split , "flags": f"{len mentioned - len bad flags }/{len mentioned }", "cmds": f"{len ran - len failed }/{len ran }", "errors": errors, "gaps": gaps, "accuracy": round 100 total - errors - gaps / total, 1 , "detail": { "bad flags": bad flags, "failed cmds": failed, "bad refs": bad refs, "forbidden": forbidden, "missing sections": miss sections, "missing probes": miss probes, }, } def main : ap = argparse.ArgumentParser ap.add argument "folder", type=Path ap.add argument "--json", type=Path, help="write full results here" args = ap.parse args real = real flags rows = for f in sorted args.folder.rglob " .md" : res = score f.read text encoding="utf-8" , real res "model" , res "file" = f.parent.name, f.name rows.append res if not rows: sys.exit f"no .md files under {args.folder}" print f"{'model':<9}{'file':<12}{'words': 6}{'flags': 7}{'cmds': 7}{'errors': 8}{'gaps': 6}{'acc%': 7}" for r in rows: print f"{r 'model' :<9}{r 'file' :<12}{r 'words' : 6}{r 'flags' : 7}{r 'cmds' : 7}" f"{r 'errors' : 8}{r 'gaps' : 6}{r 'accuracy' : 7}" by = defaultdict list for r in rows: by r "model" .append r print "\nper model mean over runs " for m, rs in by.items : n = len rs print f"{m:<9}runs={n} errors={sum r 'errors' for r in rs / n:.2f} " f"gaps={sum r 'gaps' for r in rs / n:.2f} " f"acc={sum r 'accuracy' for r in rs / n:.1f}% " f"worst={min r 'accuracy' for r in rs }%" if args.json: args.json.write text json.dumps rows, indent=2 if name == " main ": main Four design choices deserve a closer look. First, flag extraction only inspects code spans and fenced code, and only lines that mention logtrim or start with a dash. That avoids flagging a prose word such as well-known as a flag. Second, the scorer asks the real tool for its flags by running --help , so the ground truth is the program itself rather than a hand-maintained list. Third, command execution never uses a shell. It splits the line with shlex , rebuilds the call as python logtrim.py plus the arguments, runs it in a temporary copy with stdin closed and a ten-second timeout, and skips any line containing pipes, redirects or other shell metacharacters. You are executing text that a language model wrote, and that deserves caution. Fourth, the scorer skips lines with square brackets, because they are usage synopses, a fix that comes out of Step 8. Step 8: test the scorer before you trust it on model output. Write two hand-made READMEs. One is accurate. The other plants the errors a careless model would make. Save the accurate one as fixtures/clean/run 01.md : logtrim Cut a log file down to the lines you care about. logtrim filters by regex and start time, caps the line count, and can print JSON. Install logtrim is a single file. Copy logtrim.py into your project and run it with Python 3. bash python3 logtrim.py sample.log Usage bash logtrim PATH -p PATTERN --since TIMESTAMP --max-lines N --json --dry-run Pass - as PATH to read from stdin. Lines are expected to start with a timestamp like 2026-11-10T09:15:02. Options | Flag | Meaning | | --- | --- | | -p , --pattern | Keep only lines matching this regex | | --since | Keep lines at or after YYYY-MM-DDTHH:MM:SS | | --max-lines | Keep at most N lines, the newest. Default 500. Use 0 for no limit | | --json | Print one JSON object per line | | --dry-run | Print counts instead of lines | Examples bash logtrim sample.log --pattern ERROR logtrim sample.log --since 2026-11-10T10:00:00 --max-lines 2 logtrim sample.log --dry-run logtrim sample.log -p WARN --json Exit codes - 0: at least one line was kept - 1: no lines matched - 2: invalid input, such as a missing file, a bad regex or a malformed --since value License MIT, see LICENSE . And the flawed one as fixtures/flawed/run 01.md : logtrim A fast log trimmer for production servers. Installation bash pip install logtrim You can also set defaults in logtrim.yaml . Usage bash logtrim app.log --pattern ERROR logtrim sample.log --since 2026-11-10 --max-lines 1000 logtrim sample.log --follow logtrim sample.log --output out.json logtrim sample.log --dry-run Options - --pattern keep lines matching a regex - --since start date - --max-lines defaults to 1000 - --follow tail the file like tail -f - --output write results to a file License MIT Both fixtures are hand-written by a person, not output from ChatGPT or Claude, so the numbers below describe the scorer, not either model. Run it: python score.py fixtures The real output from this project: model file words flags cmds errors gaps acc% clean run 01.md 213 6/6 5/5 0 0 100.0 flawed run 01.md 83 4/6 1/5 9 8 43.3 per model mean over runs clean runs=1 errors=0.00 gaps=0.00 acc=100.0% worst=100.0% flawed runs=1 errors=9.00 gaps=8.00 acc=43.3% worst=43.3% The flawed README earns nine errors: two invented flags --follow and --output , four commands that exit with code 2, one file reference to logtrim.yaml and two forbidden claims the pip install line and the config file . It also earns eight gaps, because it has no exit codes or examples headings and states none of the six probed facts. The clean README scores 100 percent. A scorer that cannot separate those two documents is broken, and a scorer that can has earned a first look at real model output. Check that your clean fixture scores 100 before you trust the harness. A usage synopsis line such as logtrim PATH -p PATTERN ... can look like a command and exit with code 2. That is a false positive, and it would unfairly punish any model that writes a synopsis, which is nearly all of them. Skipping lines with square brackets removes it. Test on fixtures every time you change a check. Step 9: Run the Live Test and Read the Output With the scorer verified, score the real READMEs: python score.py runs --json results.json The table has the same columns as the fixture output, one row per README, then a summary per model. Read it in this order. Start with errors , because those are false statements. Then check cmds , which shows how many runnable examples passed. Then look at gaps and words . Finally look at the worst figure in the per-model summary. A model with a high mean and a 70 percent worst run is a different bet from one that scores 94 every time, because you will not read all five of your own outputs before merging. Open the full detail for any row you doubt. The JSON file lists the exact flags, commands, files and patterns behind every count: python python3 - <<'EOF' import json for r in json.load open "results.json" : if r "errors" : print r "model" , r "file" , json.dumps r "detail" , indent=1 EOF Expect three kinds of README accuracy test result. In the first, both models land near the same accuracy and differ only in gaps. That tells you the choice should rest on cost, speed or taste, and the three-way API test harness https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/ shows how to extend the same loop. In the second, one model produces errors that the other never does. That is a real signal, and the error type usually repeats, which points to a specific habit such as inventing install lines. In the third, errors vary wildly inside one model from run to run. That is a signal about reliability rather than skill, and you should raise the run count before concluding anything. Do not over-read small gaps between models. With five runs each, a difference of one error in the mean is within what rerunning on another day could erase. A difference of five errors that holds on a second batch is not. Steps 10 and 11: Add a Blind Read and Join the Results Step 10: rate clarity without knowing the author. Accuracy is not everything. A README with zero errors can still be a wall of text. The blind step hides the model names so your preferences cannot leak into the scores. Save this as blind.py : bash /usr/bin/env python3 """Shuffle and rename READMEs so a reviewer cannot see which model wrote them.""" import json import random import shutil import sys from pathlib import Path runs = Path "runs" out = Path "blind" out.mkdir exist ok=True files = sorted runs.glob " /run .md" random.Random int sys.argv 1 if len sys.argv 1 else 7 .shuffle files key = {} for i, f in enumerate files, 1 : tag = f"R{i:02d}" shutil.copy f, out / f"{tag}.md" key tag = f.parent.name Path "blind key.json" .write text json.dumps key, indent=2 Path "ratings.csv" .write text "tag,clarity,skimmability,trust\n" + "".join f"{t},,,\n" for t in key print f"wrote {len key } blind files, blind key.json and an empty ratings.csv" Run python blind.py . It copies every README into blind/ under a shuffled tag such as R07.md , writes the answer key to blind key.json and creates an empty ratings.csv . Open the READMEs in the shuffled order, and fill in three columns per row on a one to five scale: clarity could a new user follow it , skimmability can you find the options in five seconds and trust would you merge this . Do not open blind key.json until you have filled in every row. Give yourself a fixed ten minutes, because rating forty files in a rush produces noise. Step 11: join the ratings and the run log. The last script reads the answer key, averages your ratings per model, and adds latency and output tokens from runs/meta.jsonl . Save it as report.py : bash /usr/bin/env python3 """Join blind ratings with the answer key and print mean scores per model.""" import csv import json from collections import defaultdict from pathlib import Path key = json.loads Path "blind key.json" .read text sums = defaultdict lambda: defaultdict list for row in csv.DictReader open "ratings.csv", encoding="utf-8" : if not row "clarity" : continue for field in "clarity", "skimmability", "trust" : sums key row "tag" field .append int row field for model, fields in sums.items : means = {f: round sum v / len v , 2 for f, v in fields.items } print model, means, f"n={len next iter fields.values }" meta = json.loads ln for ln in open "runs/meta.jsonl", encoding="utf-8" by = defaultdict list for m in meta: by m "label" .append m for label, rows in by.items : n = len rows print label, f"avg seconds={sum r 'seconds' for r in rows / n:.1f}", f"avg output tokens={sum r 'output tokens' for r in rows / n:.0f}" python report.py The output has two blocks. The first block lists mean clarity, skimmability and trust per model with the sample size. The second lists average seconds and average output tokens per model. Output tokens are your cost lever, since vendors charge more for them than for input. A model that writes 30 percent more tokens for the same accuracy costs more per README and gives your reviewers more to read. Treat the blind scores as a tiebreaker, not a headline. Three ratings per file from one reader is a small, subjective sample. If you can persuade a colleague to rate the same shuffled set, average the two sets and look at where you disagree. Step 12: Decide With a Rule Written Before You Look The last step takes five minutes and prevents the most common mistake in any model comparison, including a README accuracy test, which is picking the metric after you see the results. Write the rule first. Copy this table into a note, adjust the thresholds to your own tolerance, and apply it mechanically. | What you see | What to do | |---|---| | One model has a higher mean error count and a worse worst-run score | Use the other model for documentation tasks and keep the harness in CI | | Mean errors are within one of each other, blind trust scores differ by 1.0 or more | Pick the model your reader trusted, then rerun a week later to confirm | | Mean errors are equal, one model writes far fewer output tokens | Pick the cheaper one, because readers gain nothing from extra length | | Errors of one type repeat in one model, such as invented install lines | Add a line to the system prompt forbidding that type, rerun, and compare again | | Errors vary widely inside one model | Raise runs to ten before drawing any conclusion | | Both models score above your threshold on every run | Move to a larger target project, because yours is too easy | Record the model IDs, the date and the run count next to the decision. A verdict without those three items is an opinion. If you want to see how a decision like this fits into a full drafting workflow, the ChatGPT vs Claude writing pipeline https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/ test chains a drafting model and an editing model. Common Pitfalls Most bad results in a README accuracy test come from the setup, not the models. These are the ones that bite most often. - Comparing an app to an API. The ChatGPT and Claude apps add hidden system prompts, memory and tool use. Compare API to API, or app to app with identical settings, never one against the other. - Not recording the exact model. Names shift quickly. GPT-6 replaced the Latest label in the selector, while GPT-5.6 Sol remains available for people who pick it by name. Log the ID with every run, which generate.py does for you. - Running each model once. One sample says nothing about spread. Five is the floor and ten is better. - Executing generated commands carelessly. The scorer runs text that a model wrote. It never invokes a shell, refuses lines with pipes or redirects, runs in a temporary copy and enforces a timeout. Keep every one of those guards if you extend it. - Scoring synopsis lines as commands. The first fixture run showed it. Placeholder syntax such as OPTIONS is documentation, not an example, so skip it. - Leaking hints. If an old README sits in target/ , both models will copy it. The snapshot function excludes README.md for that reason. - Overfitting probes to one phrasing. If your probe patterns match the wording of the first README you read, the other model loses for style. Keep patterns loose and test them on fixtures written in different styles. - Rewarding length. A longer README has more surface to be wrong. The report shows word counts so you can see when a high error count is simply a long document. - Trusting a markdown fence wrapper. Some outputs arrive wrapped in a code fence, which hides every heading from the section check. The unwrap function handles the common case. Troubleshooting Ten problems account for nearly every failed first run. - KeyError: 'OPENAI MODEL' or 'ANTHROPIC MODEL' . The environment variable is not set in this shell. Export it, or source your .env.sh file. - ModuleNotFoundError: No module named 'anthropic' . The virtual environment is not active. Run source .venv/bin/activate and try again. - Authentication error or 401. The API key is missing, mistyped or lacks billing. Print the first six characters with echo ${OPENAI API KEY:0:6} to confirm the shell sees it. - Model not found or 404. The model ID is wrong or your account lacks access. Rerun the two curl commands from Step 1 and copy an ID from the output. - AttributeError on responses or output text . The OpenAI SDK is too old. Run pip install --upgrade openai . - A Claude README stops mid-sentence. The response hit max tokens . Raise the value in ask claude and check stop reason on the response. - Rate limit or 429 errors. Add a short time.sleep between calls, or lower RUNS until your tier allows more. - no .md files under runs . The generator did not finish, or you passed the wrong folder. List runs/ and confirm each model folder holds run 01.md through run 05.md . - cmds shows 0/0 for a README. The README has no fenced blocks with lines that start with logtrim or python logtrim.py , or it uses indented code blocks. Open the file and see how it formats examples. A README with no runnable examples is itself a gap to record. - Every command fails with exit 2. The model used a different file name than sample.log , so each command hits a missing file. Strengthen the prompt line about sample.log , rerun, and keep the change identical for both models. Two more issues come up on the blind step. If report.py raises a KeyError on a tag, you reran blind.py after filling in ratings, which reshuffled the tags and orphaned your scores. Run the blind step once per batch. And if the accuracy figure looks too good, open the JSON detail for one README and confirm that the flag and command counts are not zero, since an empty extraction produces a perfect score with nothing checked. Advanced Tips Once the baseline works, a handful of extensions turn it from a one-off into a habit. Gate documentation in CI. Run the scorer on any pull request that touches a README, and fail the build on a single error. This takes three lines: python python score.py docs candidates --json out.json python3 -c "import json,sys; sys.exit 1 if any r 'errors' for r in json.load open 'out.json' else 0 " Test documentation drift. Change the code, for example rename --dry-run to --count , then ask each model to update an existing README. Score whether the old flag disappears and the new one appears. Updating a document is a harder job than writing one from scratch, and models that excel at the second often slip on the first. The long-form drift test https://shattered.io/claude-vs-chatgpt-long-form-drift-test-12-steps-2026/ covers the same weakness in prose. Raise the difficulty. A single-file tool is a friendly target. Move to a package with subcommands, environment variables and a config file, then extend truth.json to describe them. Harder code is where differences between models tend to open up. Compare tiers, not just vendors. Run Claude Haiku 5.5 or Sonnet 5.5 against GPT-6.1 Terra, then Claude Fable 5.2 against a premium OpenAI model, and log the cost per clean README. The cheapest model that clears your error threshold is often the right default for documentation, with the larger one reserved for hard cases. Report the worst run, not just the mean. If a documentation bot produces one bad README in ten, reviewers will eventually miss it. The worst column in the summary already gives you this figure, so put it in your decision note. Keep the prompt under version control. Commit prompts/ , truth.json and the scorer together, and tag the commit with the model IDs and date of each run. When a vendor ships a new model, you rerun the same harness in the time it takes to drink a coffee, and you compare today’s numbers against a baseline instead of a memory. The wider AI and machine learning coverage https://shattered.io/category/ai-machine-learning/ tracks model releases that should trigger such a rerun. FAQ Which is better for writing documentation, ChatGPT or Claude? The answer to any ChatGPT vs Claude question depends on the exact model, the plan and the task, and it changes as vendors ship updates. This harness measures it for your own pair on a task you can verify by running code. Run five or more samples per model before you decide. How many runs per model do I need? Five is the practical minimum for spotting spread, and ten is better when the means are close. A single run tells you almost nothing, because one lucky or unlucky sample can reverse the ranking. Should I test through the apps or the APIs? Use the APIs for a clean comparison. The apps add system prompts, memory and tool routing that differ by vendor and plan. If you care about the app experience, repeat the test there with the same prompt and treat it as a separate experiment. Is it safe to run commands that a language model wrote? It is safe only with guards. The scorer never starts a shell, skips lines with pipes and redirects, runs inside a throwaway copy of the project, closes stdin and enforces a ten-second timeout. Keep those guards, and do not point the runner at a project that can delete or send data. Why not ask another AI model to judge the READMEs? A judge model adds an uncontrolled variable and can share blind spots with one of the contestants. Executable checks give you numbers you can reproduce and defend. Human reading still has a place, which is why the blind step exists. Do I need a different target project for my own codebase? Yes, ideally. A README accuracy test predicts more when it mirrors your work. Replace target/ with a small project you own, regenerate truth.json from its real flags, defaults and exit codes, and rerun. A target close to your real work predicts real behavior better than any public benchmark. Why does the scorer count exit code 1 as a pass? The tool uses exit code 1 to say that no lines matched, which is a legitimate outcome for an example filter. Only exit code 2, which signals invalid input, and timeouts count as failures.