# ChatGPT vs Claude README Test: 12 Steps, 80 Min [2026] > Source: > Published: 2026-11-11 17:39:26+00:00 On the OpenAI side, the ChatGPT model selector now shows GPT-6 where it used to show Latest, and [SyncAI’s AI news roundup](https://syncai.news/article/ainews-it-s-swapping-season-0f7a5a) reports that anyone who explicitly picked GPT-5.6 Sol keeps that model. Both product lines moved in recent weeks, so a blog post that declares a winner for writing has a short shelf life. A test you can rerun on the day you read this lasts longer. The cleanest writing test for a developer is not a style contest. It is a README. A README makes claims you can check by running code: which flags exist, what the defaults are, which commands work, which files are there. Earlier tests probed [writing rules compliance](https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/) and [voice matching](https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/). A README accuracy test goes after a different failure, the confident invention of facts in documentation. The method is small. Send the same prompt to ChatGPT and Claude five times each, pull every flag and command out of the READMEs they write, execute those commands against a real tool, and count errors and gaps per model. Naming a winner here would be premature. Outcomes swing with the exact model, the plan and the day, and the harness exists to settle the question for your own pair. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. [Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F) ## Why a README Test Beats a Style Contest for ChatGPT vs Claude Most ChatGPT vs Claude comparisons for writing rest on taste. One reviewer likes warmer prose, another likes tighter paragraphs, and both are right for themselves. Taste does not transfer to your repository. Documentation is different because a reader acts on it. A wrong flag costs the reader ten minutes, and a wrong install line costs them trust in everything else on the page. A README also forces both models into the same job with the same raw material. You hand over the source files and ask for documentation. The ground truth sits in the code, so there is no need for a human to decide who wrote a better sentence. The `logtrim` tool used here is small enough to read in two minutes, yet it has enough surface to catch mistakes: five options, a short flag, a stdin convention, a default value, a special value that disables a limit, and three distinct exit codes. The failure modes are familiar to anyone who has used a chatbot on a codebase. Models add a plausible `--verbose` flag that the tool never had. They write a `pip install` line for a package that does not exist on any index. They describe a config file because tools like this one usually have one. They get a default wrong by a factor of two. Each of those errors is checkable by a script, which means you can count them across fifty runs without reading a single paragraph. The same method extends the [source-grounded writing audit](https://shattered.io/chatgpt-vs-claude-source-grounded-writing-audit-12-steps-2026/) from prose claims to executable ones. Where a source audit asks whether a sentence matches a cited page, this test asks whether a command exits with the code the README promised. Executable claims are the hardest kind to argue with. One more reason matters for a practical reader. In many teams, documentation is a chore that gets handed to a chatbot and reviewed lightly. People skim a generated README, see headings that look right, and merge it. A harness that fails the build on an invented flag turns that blind spot into a gate. ## Prerequisites and Versions Set these up before you start. None of it is exotic, and the whole stack uses the standard library plus two vendor SDKs. - Python 3.12 or newer. The project in this tutorial runs on Python 3.12.3. - `pip` and the built-in`venv` module. - The `anthropic` and`openai` Python packages, installed at their latest versions. The OpenAI call uses the Responses API, so an old SDK that lacks`responses` will fail. - An OpenAI API key and an Anthropic API key, both with billing enabled. - Two model IDs, one per vendor, stored in environment variables. Step 1 explains how to pick them. - Linux or macOS. On Windows, run everything inside WSL, because the scorer launches Python subprocesses and the shell snippets assume a POSIX shell. - About 80 minutes, with roughly 10 of them going to API wait time. The default run makes 10 API calls, five per model, and each call sends a few hundred lines of source code. On any current plan the bill stays small. If you want exact numbers for your own pair, the harness logs input and output tokens for every call, so you can multiply by the prices on the vendors’ pages. For a longer treatment of cost per accepted output, see the [writing cost per 1,000 accepted words](https://shattered.io/claude-vs-chatgpt-writing-cost-per-accepted-1000-words-12-steps-2026/) test. You need no GPU, no database and no web framework. You also need no prior experience with either SDK beyond reading an API key from an environment variable. ## What the Harness Measures The scorer reads each README as text and runs six checks against the real project. Four of them detect errors, meaning the README says something false. Two detect gaps, meaning the README leaves out something a reader needs. | Check | What it catches | How it counts | |---|---|---| | Invented flags | A flag in the README that `--help` does not list | One error per unknown flag | | Command execution | An example that fails when you run it | One error per command that exits with anything but 0 or 1 | | File references | A backticked file name the repository does not contain | One error per unknown file | | Forbidden claims | Install routes and config files the tool does not have | One error per matching pattern | | Required sections | Missing install, usage, options, exit codes or examples headings | One gap per missing section | | Fact probes | A default, format or exit code the README never states | One gap per missed probe | Exit code 1 counts as a pass because the tool uses it for no matches, a legitimate outcome for an example such as a search that finds nothing. Exit code 2 means invalid input, so an example that triggers it is wrong. ### Errors versus gaps Keep the two counts separate. A short README with no mistakes has gaps but no errors, and it will never mislead anyone. A long README with a confident invented flag has errors, and that is the worse outcome in practice. Most teams should weigh errors far above gaps, which is why the report prints them in separate columns instead of merging them into one grade. The accuracy percentage still blends them so you can sort a long list quickly, but the decision rule in Step 12 looks at errors first. ### Why no model judge scores the prose It is tempting to ask a third model to grade the READMEs. Resist that for this test. A judge model can share blind spots with one of the contestants, it adds a new variable you cannot pin down, and nothing external can confirm its verdict. Every number this harness produces comes from running code or matching a pattern, so you can reproduce it. The human part stays human: Step 10 adds a blind read where you rate clarity yourself, with the model names hidden. ## Steps 1 and 2: Pin the Models and Create the Project **Step 1: choose and record the exact models.** The product names moved this month, so write down what you actually call. In the ChatGPT app, GPT-6 now sits where Latest used to sit in the selector. GPT-6 uses GPT-6 Sol behind its Instant, Medium, High and Extra High settings, according to OpenAI’s help documentation. The API is a different surface. GPT-6.1 Terra entered the OpenAI API on October 26 under the ID `gpt-6.1-terra`, listed at $1 per million input tokens and $5 per million output tokens. For a clean test, use the API on both sides, because the apps add system prompts, memory and tool routing you cannot control. | Model | Maker | What the current record confirms | |---|---|---| | GPT-6 | OpenAI | Replaced Latest in the ChatGPT model selector, served by GPT-6 Sol at four reasoning settings | | GPT-6.1 Terra | OpenAI | In the API since October 26, 2026 as `gpt-6.1-terra` , $1 input and $5 output per million tokens | | GPT-5.6 Sol | OpenAI | Still selectable when chosen explicitly after GPT-6 replaced Latest | | Claude Sonnet 5.5 | Anthropic | Anthropic says it runs more than 30% faster than Sonnet 5 and costs up to 30% less for most work | | Claude Opus 5.5 | Anthropic | Anthropic says it leads in agentic coding and knowledge work and costs 40% less to run than Opus 5 on typical workloads | | Claude Fable 5.2 | Anthropic | Released November 9, 2026 at $8 input and $40 output per million tokens, down from $10 and $50 for Fable 5.1 | | Claude Haiku 5.5 | Anthropic | Announced October 7, 2026 as the fastest small model, aimed at high-volume work | | Claude Mythos 5.2 | Anthropic | Restricted to vetted cyberdefenders and life-sciences researchers, so it is not an option here | Pair models by what you pay, not by name. If you run Claude Sonnet 5.5 against GPT-6.1 Terra, you are comparing two models with callable API IDs that vendors position for everyday work. If you want a premium pair, add Claude Fable 5.2 on one side and a higher OpenAI tier on the other, and record the price of each README as well as its score. Plan prices change often, so check the live pages. As of early November, OpenAI’s [pricing page](https://chatgpt.com/pricing) reportedly listed ChatGPT Plus at $20 a month in the US, while Anthropic keeps current Claude plans on [its own pricing page](https://claude.com/pricing). The deeper breakdown of tiers lives in the [ChatGPT vs Claude pricing and usage limits](https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/) comparison. Ask each vendor for its model list so you copy IDs rather than guess them: ``` curl -s https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY" | python3 -m json.tool | grep '"id"' curl -s https://api.anthropic.com/v1/models -H "x-api-key: $ANTHROPIC_API_KEY" -H "anthropic-version: 2023-06-01" | python3 -m json.tool | grep '"id"' ``` **Step 2: create the project and the environment.** Make a folder, a virtual environment and the directory layout the later steps assume. ``` mkdir readme-bench && cd readme-bench python3 -m venv .venv && source .venv/bin/activate pip install anthropic openai mkdir -p target prompts runs fixtures/clean fixtures/flawed export OPENAI_API_KEY="paste-your-key" export ANTHROPIC_API_KEY="paste-your-key" export OPENAI_MODEL="gpt-6.1-terra" export ANTHROPIC_MODEL="paste-an-id-from-the-models-list" ``` Write the four exports into a file named `.env.sh` and `source` it each session. A forgotten variable is the most common reason the first run fails, and the failure message from Python is a bare `KeyError` that tells you little. ## Steps 3 and 4: Build the Target Project and Its Ground Truth **Step 3: write the project the models will document.** `logtrim` reads a log file and keeps the lines you ask for. It is deliberately boring. Boring code has an unambiguous right answer, and that is what a ground truth needs. Save this as `target/logtrim.py`: ``` bash #!/usr/bin/env python3 """logtrim: cut a log file down to the lines you care about.""" import argparse import json import re import sys from datetime import datetime DEFAULT_MAX_LINES = 500 TS_FORMAT = "%Y-%m-%dT%H:%M:%S" def build_parser(): p = argparse.ArgumentParser( prog="logtrim", description="Trim a log file by pattern, start time and line count.", ) p.add_argument("path", help="log file to read, or - for stdin") p.add_argument("-p", "--pattern", help="keep only lines matching this regex") p.add_argument("--since", help="keep lines at or after YYYY-MM-DDTHH:MM:SS") p.add_argument( "--max-lines", type=int, default=DEFAULT_MAX_LINES, help="keep at most this many lines (the newest), 0 means no limit", ) p.add_argument("--json", action="store_true", help="print one JSON object per line") p.add_argument("--dry-run", action="store_true", help="print counts, not lines") return p def read_lines(path): if path == "-": return sys.stdin.read().splitlines() with open(path, encoding="utf-8", errors="replace") as fh: return fh.read().splitlines() def line_time(line): try: return datetime.strptime(line[:19], TS_FORMAT) except ValueError: return None def trim(lines, pattern=None, since=None, max_lines=DEFAULT_MAX_LINES): rx = re.compile(pattern) if pattern else None kept = [] for line in lines: if rx and not rx.search(line): continue if since is not None: ts = line_time(line) if ts is None or ts < since: continue kept.append(line) return kept[-max_lines:] if max_lines > 0 else kept def main(argv=None): args = build_parser().parse_args(argv) since = None if args.since: try: since = datetime.strptime(args.since, TS_FORMAT) except ValueError: print("logtrim: --since must look like 2026-11-10T09:00:00", file=sys.stderr) return 2 try: lines = read_lines(args.path) except OSError as exc: print(f"logtrim: {exc}", file=sys.stderr) return 2 try: kept = trim(lines, args.pattern, since, args.max_lines) except re.error as exc: print(f"logtrim: bad pattern: {exc}", file=sys.stderr) return 2 if args.dry_run: print(f"read={len(lines)} kept={len(kept)}") elif args.json: for n, line in enumerate(kept, 1): print(json.dumps({"n": n, "line": line})) else: for line in kept: print(line) return 0 if kept else 1 if __name__ == "__main__": sys.exit(main()) ``` Add a nine-line sample log at `target/sample.log`, which every example command in the READMEs must use: ``` 2026-11-10T08:59:41 INFO service started 2026-11-10T09:00:03 INFO health check ok 2026-11-10T09:04:17 WARN disk at 81 percent 2026-11-10T09:15:02 ERROR upstream timeout after 30s 2026-11-10T09:15:09 ERROR upstream timeout after 30s 2026-11-10T09:31:44 INFO health check ok 2026-11-10T10:02:12 WARN disk at 86 percent 2026-11-10T10:20:55 ERROR payment webhook rejected 2026-11-10T11:00:00 INFO health check ok ``` Add a short `target/LICENSE` file with any license text you like. The scorer treats it as a known file, so a README that mentions it passes the file check. Run the tool by hand before you trust it. These four commands show every exit path the READMEs have to describe: ``` cd target python3 logtrim.py sample.log --since 2026-11-10T10:00:00 --max-lines 2 python3 logtrim.py sample.log --dry-run python3 logtrim.py sample.log -p NOPE ; echo "exit=$?" python3 logtrim.py nope.log ; echo "exit=$?" cd .. ``` The first command prints the two newest lines from 10:20 and 11:00. The second prints `read=9 kept=9`. The third prints nothing and exits 1. The fourth prints a `No such file or directory` message and exits 2. **Step 4: encode the ground truth.** The scorer needs a list of things a good README must contain and things it must never claim. Save this as `truth.json` in the project root: ``` { "sections": [ {"id": "install", "pattern": "install|requirements|getting started|setup"}, {"id": "usage", "pattern": "usage|quick ?start"}, {"id": "options", "pattern": "options|flags|arguments|reference"}, {"id": "exit-codes", "pattern": "exit codes?|return codes?"}, {"id": "examples", "pattern": "examples?"} ], "probes": [ {"id": "default-500", "pattern": "max-lines[^\\n]{0,160}\\b500\\b|\\b500\\b[^\\n]{0,160}max-lines"}, {"id": "since-format", "pattern": "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}|YYYY-MM-DDTHH:MM:SS"}, {"id": "exit-1-no-match", "pattern": "\\b1\\b[^\\n]{0,80}(no lines|nothing|no match|empty|zero)|(no lines|nothing|no match|empty|zero)[^\\n]{0,80}\\b1\\b"}, {"id": "exit-2-bad-input", "pattern": "\\b2\\b[^\\n]{0,80}(invalid|bad|missing|error|unreadable|not found|cannot)|(invalid|bad|missing|error|unreadable|not found|cannot)[^\\n]{0,80}\\b2\\b"}, {"id": "stdin-dash", "pattern": "stdin|standard input"}, {"id": "zero-no-limit", "pattern": "\\b0\\b[^\\n]{0,80}(no limit|unlimited|disable)|(no limit|unlimited|disable)[^\\n]{0,80}\\b0\\b"} ], "forbidden": [ {"id": "pip-install-logtrim", "pattern": "pip3? install[^\\n]*logtrim"}, {"id": "pypi", "pattern": "pypi"}, {"id": "docker", "pattern": "docker (run|build)"}, {"id": "brew", "pattern": "brew install"}, {"id": "npm", "pattern": "npm (install|i) "}, {"id": "config-file", "pattern": "\\.logtrimrc|logtrim\\.(yaml|yml|toml|conf)"}, {"id": "windows-binary", "pattern": "logtrim\\.exe"} ] } ``` The `forbidden` list is where the invented claims live. The tool is a single file with no packaging metadata, so any `pip install logtrim` line is false by construction. The tool has no configuration file, no Docker image and no Windows binary, so those patterns are false too. The `probes` list holds the facts a reader needs, each as a regular expression that must match somewhere in the text. Keep the patterns loose. A probe that demands one exact phrase punishes a model for wording, and you want to punish it for omissions. ## Step 5: Write One Prompt Both Models Share Fairness lives in the prompt. Both models receive the same system instruction, the same user message and the same repository snapshot, character for character. The only difference is the vendor endpoint. Save the system prompt as `prompts/system.txt`: ``` You are a technical writer. Write a README.md for the project the user shows you. Use only facts that appear in the files provided. If something is not in the files, leave it out. Return the README as plain Markdown with no preamble and no wrapping code fence. ``` And the user message as `prompts/user.txt`. The generator replaces the `{snapshot}` placeholder with every file in `target/` except any existing README: ``` Write the README.md for this project. Cover installation, usage, every option, exit codes and examples. Use sample.log in every example command. Here are all the files in the repository: {snapshot} ``` Three details in these prompts matter more than they look. The instruction to use only facts from the files gives both models a clear rule to break, which is what you want to measure. The instruction to use `sample.log` in every example makes command execution testable, because a README that points at `app.log` would fail for a reason unrelated to accuracy. And the instruction to return plain Markdown without a wrapping fence saves you from a parsing step that varies by model. Some models wrap anyway, and the generator strips a `markdown` fence if it finds one. Do not add a persona, a tone request or formatting tips. Those would make the test a style contest again. A bare prompt shows what each model does by default, which is what most people get when they paste code into a chat window and ask for docs. If you later want to test how much a better prompt helps, change the prompt for both models at once and rerun, as the [blind writing test](https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/) does for prose. ## Step 6: Generate Five READMEs per Model Five runs per model is the minimum that tells you anything about spread. A single run can land on a lucky or unlucky sample, and neither says much about the model. Save this as `generate.py`: ``` bash #!/usr/bin/env python3 """Ask both models for a README, N times each, with identical prompts.""" import json import os import time from pathlib import Path from anthropic import Anthropic from openai import OpenAI ROOT = Path(__file__).parent TARGET = ROOT / "target" RUNS = int(os.environ.get("RUNS", "5")) MODELS = { "chatgpt": os.environ["OPENAI_MODEL"], "claude": os.environ["ANTHROPIC_MODEL"], } SYSTEM = (ROOT / "prompts" / "system.txt").read_text() USER = (ROOT / "prompts" / "user.txt").read_text() def snapshot(): parts = [] for p in sorted(TARGET.rglob("*")): if p.is_file() and p.name != "README.md": body = p.read_text(encoding="utf-8", errors="replace") parts.append(f"=== {p.relative_to(TARGET)} ===\n{body}") return "\n\n".join(parts) def ask_chatgpt(model, system, user): r = OpenAI().responses.create(model=model, instructions=system, input=user) return r.output_text, r.usage.input_tokens, r.usage.output_tokens def ask_claude(model, system, user): r = Anthropic().messages.create( model=model, max_tokens=4000, system=system, messages=[{"role": "user", "content": user}], ) text = "".join(b.text for b in r.content if b.type == "text") return text, r.usage.input_tokens, r.usage.output_tokens def unwrap(text): t = text.strip() if t.startswith("``` markdown") and t.endswith("```"): t = t[len("``` markdown"):-3].strip() return t + "\n" def main(): user = USER.format(snapshot=snapshot()) askers = {"chatgpt": ask_chatgpt, "claude": ask_claude} meta = (ROOT / "runs" / "meta.jsonl").open("a", encoding="utf-8") for label, model in MODELS.items(): out = ROOT / "runs" / label out.mkdir(parents=True, exist_ok=True) for i in range(1, RUNS + 1): start = time.time() text, tin, tout = askers[label](model, SYSTEM, user) (out / f"run_{i:02d}.md").write_text(unwrap(text), encoding="utf-8") meta.write(json.dumps({ "label": label, "model": model, "run": i, "seconds": round(time.time() - start, 1), "input_tokens": tin, "output_tokens": tout, }) + "\n") meta.flush() print(f"{label} run {i}/{RUNS} done") if __name__ == "__main__": main() ``` The two ask functions use each vendor’s current SDK shape. OpenAI’s Responses API takes `instructions` and `input` and exposes `output_text`. Anthropic’s Messages API takes `system` and a message list and returns content blocks, so the code joins the text blocks. The script sets no sampling parameters. Sampling parameters differ across models and API versions, so the script leaves them at vendor defaults and both calls stay valid. Every call appends a line to `runs/meta.jsonl` with the model ID, elapsed seconds and token counts. That file is the raw material for cost and latency, and it records which exact model produced which README, an audit trail that matters when vendors rename models mid-month. Run it: ``` python generate.py ``` Expected console output, one line per call: ``` chatgpt run 1/5 done chatgpt run 2/5 done chatgpt run 3/5 done chatgpt run 4/5 done chatgpt run 5/5 done claude run 1/5 done claude run 2/5 done claude run 3/5 done claude run 4/5 done claude run 5/5 done ``` The loop runs sequentially on purpose. Parallel calls would finish faster but trigger rate limits on lower tiers, and a failed run halfway through would leave you with uneven sample sizes. If a call raises, fix the cause and rerun the whole script after clearing `runs/`, so both models keep five clean samples. The labels `chatgpt` and `claude` become folder names under `runs/`, and the scorer later reads those folder names as model labels. ## Steps 7 and 8: Write the Scorer, Then Test It on Fixtures **Step 7: write the scorer.** This is the core of the project. It extracts flags and commands from the README, runs the commands in a throwaway copy of the target folder, and tallies errors and gaps. Save it as `score.py`: ``` bash #!/usr/bin/env python3 """Score README files against the real logtrim project.""" import argparse import json import re import shlex import shutil import subprocess import sys import tempfile from collections import defaultdict from pathlib import Path ROOT = Path(__file__).parent TARGET = ROOT / "target" TRUTH = json.loads((ROOT / "truth.json").read_text()) FENCE = re.compile(r"```[a-zA-Z0-9_-]*\n(.*?)```", re.S) INLINE = re.compile(r"(?$`]") HEADING = re.compile(r"^#{1,6}\s+(.*)$", re.M) def real_flags(): out = subprocess.run( [sys.executable, "logtrim.py", "--help"], cwd=TARGET, capture_output=True, text=True, timeout=10, ).stdout return set(LONG_FLAG.findall(out)) | set(SHORT_FLAG.findall(out)) def code_lines(md): blocks = [ln for b in FENCE.findall(md) for ln in b.splitlines()] inline = INLINE.findall(FENCE.sub("", md)) return blocks + inline def flags_in(lines): found = set() for ln in lines: s = ln.strip() if "logtrim" in s or s.startswith("-"): found |= set(LONG_FLAG.findall(s)) | set(SHORT_FLAG.findall(s)) return found def commands(md): cmds = [] for b in FENCE.findall(md): for ln in b.splitlines(): s = re.sub(r"^[$>]\s+", "", ln.strip()) if s.startswith(("logtrim ", "python logtrim.py", "python3 logtrim.py")): cmds.append(re.split(r"\s+#", s)[0].strip()) return cmds def run_command(cmd): if SHELL_META.search(cmd) or "[" in cmd: return None try: argv = shlex.split(cmd) except ValueError: return 2 skip = 1 if argv[0] == "logtrim" else 2 full = [sys.executable, "logtrim.py"] + argv[skip:] with tempfile.TemporaryDirectory() as tmp: repo = Path(tmp) / "repo" shutil.copytree(TARGET, repo) try: r = subprocess.run( full, cwd=repo, capture_output=True, text=True, timeout=10, stdin=subprocess.DEVNULL, ) except subprocess.TimeoutExpired: return 124 return r.returncode def score(md, real): mentioned = flags_in(code_lines(md)) bad_flags = sorted(mentioned - real) results = [(c, run_command(c)) for c in commands(md)] ran = [(c, rc) for c, rc in results if rc is not None] failed = [c for c, rc in ran if rc not in (0, 1)] known = {p.name for p in TARGET.rglob("*") if p.is_file()} refs = set(PATH_REF.findall(md)) - {"README.md"} bad_refs = sorted(r for r in refs if Path(r).name not in known) forbidden = [f["id"] for f in TRUTH["forbidden"] if re.search(f["pattern"], md, re.I)] heads = " | ".join(HEADING.findall(md)) miss_sections = [s["id"] for s in TRUTH["sections"] if not re.search(s["pattern"], heads, re.I)] miss_probes = [p["id"] for p in TRUTH["probes"] if not re.search(p["pattern"], md, re.I | re.S)] total = (len(mentioned) + len(ran) + len(refs) + len(TRUTH["forbidden"]) + len(TRUTH["sections"]) + len(TRUTH["probes"])) errors = len(bad_flags) + len(failed) + len(bad_refs) + len(forbidden) gaps = len(miss_sections) + len(miss_probes) return { "words": len(md.split()), "flags": f"{len(mentioned) - len(bad_flags)}/{len(mentioned)}", "cmds": f"{len(ran) - len(failed)}/{len(ran)}", "errors": errors, "gaps": gaps, "accuracy": round(100 * (total - errors - gaps) / total, 1), "detail": { "bad_flags": bad_flags, "failed_cmds": failed, "bad_refs": bad_refs, "forbidden": forbidden, "missing_sections": miss_sections, "missing_probes": miss_probes, }, } def main(): ap = argparse.ArgumentParser() ap.add_argument("folder", type=Path) ap.add_argument("--json", type=Path, help="write full results here") args = ap.parse_args() real = real_flags() rows = [] for f in sorted(args.folder.rglob("*.md")): res = score(f.read_text(encoding="utf-8"), real) res["model"], res["file"] = f.parent.name, f.name rows.append(res) if not rows: sys.exit(f"no .md files under {args.folder}") print(f"{'model':<9}{'file':<12}{'words':>6}{'flags':>7}{'cmds':>7}{'errors':>8}{'gaps':>6}{'acc%':>7}") for r in rows: print(f"{r['model']:<9}{r['file']:<12}{r['words']:>6}{r['flags']:>7}{r['cmds']:>7}" f"{r['errors']:>8}{r['gaps']:>6}{r['accuracy']:>7}") by = defaultdict(list) for r in rows: by[r["model"]].append(r) print("\nper model (mean over runs)") for m, rs in by.items(): n = len(rs) print(f"{m:<9}runs={n} errors={sum(r['errors'] for r in rs) / n:.2f} " f"gaps={sum(r['gaps'] for r in rs) / n:.2f} " f"acc={sum(r['accuracy'] for r in rs) / n:.1f}% " f"worst={min(r['accuracy'] for r in rs)}%") if args.json: args.json.write_text(json.dumps(rows, indent=2)) if __name__ == "__main__": main() ``` Four design choices deserve a closer look. First, flag extraction only inspects code spans and fenced code, and only lines that mention `logtrim` or start with a dash. That avoids flagging a prose word such as `well-known` as a flag. Second, the scorer asks the real tool for its flags by running `--help`, so the ground truth is the program itself rather than a hand-maintained list. Third, command execution never uses a shell. It splits the line with `shlex`, rebuilds the call as `python logtrim.py` plus the arguments, runs it in a temporary copy with stdin closed and a ten-second timeout, and skips any line containing pipes, redirects or other shell metacharacters. You are executing text that a language model wrote, and that deserves caution. Fourth, the scorer skips lines with square brackets, because they are usage synopses, a fix that comes out of Step 8. **Step 8: test the scorer before you trust it on model output.** Write two hand-made READMEs. One is accurate. The other plants the errors a careless model would make. Save the accurate one as `fixtures/clean/run_01.md`: ``` # logtrim Cut a log file down to the lines you care about. logtrim filters by regex and start time, caps the line count, and can print JSON. ## Install logtrim is a single file. Copy `logtrim.py` into your project and run it with Python 3. ``` bash python3 logtrim.py sample.log ``` ## Usage ``` bash logtrim PATH [-p PATTERN] [--since TIMESTAMP] [--max-lines N] [--json] [--dry-run] ``` Pass `-` as PATH to read from stdin. Lines are expected to start with a timestamp like 2026-11-10T09:15:02. ## Options | Flag | Meaning | | --- | --- | | `-p`, `--pattern` | Keep only lines matching this regex | | `--since` | Keep lines at or after YYYY-MM-DDTHH:MM:SS | | `--max-lines` | Keep at most N lines, the newest. Default 500. Use 0 for no limit | | `--json` | Print one JSON object per line | | `--dry-run` | Print counts instead of lines | ## Examples ``` bash logtrim sample.log --pattern ERROR logtrim sample.log --since 2026-11-10T10:00:00 --max-lines 2 logtrim sample.log --dry-run logtrim sample.log -p WARN --json ``` ## Exit codes - 0: at least one line was kept - 1: no lines matched - 2: invalid input, such as a missing file, a bad regex or a malformed --since value ## License MIT, see `LICENSE`. ``` And the flawed one as `fixtures/flawed/run_01.md`: ``` # logtrim A fast log trimmer for production servers. ## Installation ``` bash pip install logtrim ``` You can also set defaults in `logtrim.yaml`. ## Usage ``` bash logtrim app.log --pattern ERROR logtrim sample.log --since 2026-11-10 --max-lines 1000 logtrim sample.log --follow logtrim sample.log --output out.json logtrim sample.log --dry-run ``` ## Options - `--pattern` keep lines matching a regex - `--since` start date - `--max-lines` defaults to 1000 - `--follow` tail the file like tail -f - `--output` write results to a file ## License MIT ``` Both fixtures are hand-written by a person, not output from ChatGPT or Claude, so the numbers below describe the scorer, not either model. Run it: ``` python score.py fixtures ``` The real output from this project: ``` model file words flags cmds errors gaps acc% clean run_01.md 213 6/6 5/5 0 0 100.0 flawed run_01.md 83 4/6 1/5 9 8 43.3 per model (mean over runs) clean runs=1 errors=0.00 gaps=0.00 acc=100.0% worst=100.0% flawed runs=1 errors=9.00 gaps=8.00 acc=43.3% worst=43.3% ``` The flawed README earns nine errors: two invented flags (`--follow` and `--output`), four commands that exit with code 2, one file reference to `logtrim.yaml` and two forbidden claims (the `pip install` line and the config file). It also earns eight gaps, because it has no exit codes or examples headings and states none of the six probed facts. The clean README scores 100 percent. A scorer that cannot separate those two documents is broken, and a scorer that can has earned a first look at real model output. Check that your clean fixture scores 100 before you trust the harness. A usage synopsis line such as `logtrim PATH [-p PATTERN] ...` can look like a command and exit with code 2. That is a false positive, and it would unfairly punish any model that writes a synopsis, which is nearly all of them. Skipping lines with square brackets removes it. Test on fixtures every time you change a check. ## Step 9: Run the Live Test and Read the Output With the scorer verified, score the real READMEs: ``` python score.py runs --json results.json ``` The table has the same columns as the fixture output, one row per README, then a summary per model. Read it in this order. Start with `errors`, because those are false statements. Then check `cmds`, which shows how many runnable examples passed. Then look at `gaps` and `words`. Finally look at the `worst` figure in the per-model summary. A model with a high mean and a 70 percent worst run is a different bet from one that scores 94 every time, because you will not read all five of your own outputs before merging. Open the full detail for any row you doubt. The JSON file lists the exact flags, commands, files and patterns behind every count: ``` python python3 - <<'EOF' import json for r in json.load(open("results.json")): if r["errors"]: print(r["model"], r["file"], json.dumps(r["detail"], indent=1)) EOF ``` Expect three kinds of README accuracy test result. In the first, both models land near the same accuracy and differ only in gaps. That tells you the choice should rest on cost, speed or taste, and the [three-way API test harness](https://shattered.io/chatgpt-vs-claude-vs-gemini-api-test-harness-12-steps-2026/) shows how to extend the same loop. In the second, one model produces errors that the other never does. That is a real signal, and the error type usually repeats, which points to a specific habit such as inventing install lines. In the third, errors vary wildly inside one model from run to run. That is a signal about reliability rather than skill, and you should raise the run count before concluding anything. Do not over-read small gaps between models. With five runs each, a difference of one error in the mean is within what rerunning on another day could erase. A difference of five errors that holds on a second batch is not. ## Steps 10 and 11: Add a Blind Read and Join the Results **Step 10: rate clarity without knowing the author.** Accuracy is not everything. A README with zero errors can still be a wall of text. The blind step hides the model names so your preferences cannot leak into the scores. Save this as `blind.py`: ``` bash #!/usr/bin/env python3 """Shuffle and rename READMEs so a reviewer cannot see which model wrote them.""" import json import random import shutil import sys from pathlib import Path runs = Path("runs") out = Path("blind") out.mkdir(exist_ok=True) files = sorted(runs.glob("*/run_*.md")) random.Random(int(sys.argv[1]) if len(sys.argv) > 1 else 7).shuffle(files) key = {} for i, f in enumerate(files, 1): tag = f"R{i:02d}" shutil.copy(f, out / f"{tag}.md") key[tag] = f.parent.name Path("blind_key.json").write_text(json.dumps(key, indent=2)) Path("ratings.csv").write_text("tag,clarity,skimmability,trust\n" + "".join(f"{t},,,\n" for t in key)) print(f"wrote {len(key)} blind files, blind_key.json and an empty ratings.csv") ``` Run `python blind.py`. It copies every README into `blind/` under a shuffled tag such as `R07.md`, writes the answer key to `blind_key.json` and creates an empty `ratings.csv`. Open the READMEs in the shuffled order, and fill in three columns per row on a one to five scale: clarity (could a new user follow it), skimmability (can you find the options in five seconds) and trust (would you merge this). Do not open `blind_key.json` until you have filled in every row. Give yourself a fixed ten minutes, because rating forty files in a rush produces noise. **Step 11: join the ratings and the run log.** The last script reads the answer key, averages your ratings per model, and adds latency and output tokens from `runs/meta.jsonl`. Save it as `report.py`: ``` bash #!/usr/bin/env python3 """Join blind ratings with the answer key and print mean scores per model.""" import csv import json from collections import defaultdict from pathlib import Path key = json.loads(Path("blind_key.json").read_text()) sums = defaultdict(lambda: defaultdict(list)) for row in csv.DictReader(open("ratings.csv", encoding="utf-8")): if not row["clarity"]: continue for field in ("clarity", "skimmability", "trust"): sums[key[row["tag"]]][field].append(int(row[field])) for model, fields in sums.items(): means = {f: round(sum(v) / len(v), 2) for f, v in fields.items()} print(model, means, f"n={len(next(iter(fields.values())))}") meta = [json.loads(ln) for ln in open("runs/meta.jsonl", encoding="utf-8")] by = defaultdict(list) for m in meta: by[m["label"]].append(m) for label, rows in by.items(): n = len(rows) print(label, f"avg_seconds={sum(r['seconds'] for r in rows) / n:.1f}", f"avg_output_tokens={sum(r['output_tokens'] for r in rows) / n:.0f}") python report.py ``` The output has two blocks. The first block lists mean clarity, skimmability and trust per model with the sample size. The second lists average seconds and average output tokens per model. Output tokens are your cost lever, since vendors charge more for them than for input. A model that writes 30 percent more tokens for the same accuracy costs more per README and gives your reviewers more to read. Treat the blind scores as a tiebreaker, not a headline. Three ratings per file from one reader is a small, subjective sample. If you can persuade a colleague to rate the same shuffled set, average the two sets and look at where you disagree. ## Step 12: Decide With a Rule Written Before You Look The last step takes five minutes and prevents the most common mistake in any model comparison, including a README accuracy test, which is picking the metric after you see the results. Write the rule first. Copy this table into a note, adjust the thresholds to your own tolerance, and apply it mechanically. | What you see | What to do | |---|---| | One model has a higher mean error count and a worse worst-run score | Use the other model for documentation tasks and keep the harness in CI | | Mean errors are within one of each other, blind trust scores differ by 1.0 or more | Pick the model your reader trusted, then rerun a week later to confirm | | Mean errors are equal, one model writes far fewer output tokens | Pick the cheaper one, because readers gain nothing from extra length | | Errors of one type repeat in one model, such as invented install lines | Add a line to the system prompt forbidding that type, rerun, and compare again | | Errors vary widely inside one model | Raise runs to ten before drawing any conclusion | | Both models score above your threshold on every run | Move to a larger target project, because yours is too easy | Record the model IDs, the date and the run count next to the decision. A verdict without those three items is an opinion. If you want to see how a decision like this fits into a full drafting workflow, the [ChatGPT vs Claude writing pipeline](https://shattered.io/chatgpt-vs-claude-writing-pipeline-terra-opus-5-5-12-steps-2026/) test chains a drafting model and an editing model. ## Common Pitfalls Most bad results in a README accuracy test come from the setup, not the models. These are the ones that bite most often. - **Comparing an app to an API.** The ChatGPT and Claude apps add hidden system prompts, memory and tool use. Compare API to API, or app to app with identical settings, never one against the other. - **Not recording the exact model.** Names shift quickly. GPT-6 replaced the Latest label in the selector, while GPT-5.6 Sol remains available for people who pick it by name. Log the ID with every run, which`generate.py` does for you. - **Running each model once.** One sample says nothing about spread. Five is the floor and ten is better. - **Executing generated commands carelessly.** The scorer runs text that a model wrote. It never invokes a shell, refuses lines with pipes or redirects, runs in a temporary copy and enforces a timeout. Keep every one of those guards if you extend it. - **Scoring synopsis lines as commands.** The first fixture run showed it. Placeholder syntax such as`[OPTIONS]` is documentation, not an example, so skip it. - **Leaking hints.** If an old README sits in`target/` , both models will copy it. The snapshot function excludes`README.md` for that reason. - **Overfitting probes to one phrasing.** If your probe patterns match the wording of the first README you read, the other model loses for style. Keep patterns loose and test them on fixtures written in different styles. - **Rewarding length.** A longer README has more surface to be wrong. The report shows word counts so you can see when a high error count is simply a long document. - **Trusting a markdown fence wrapper.** Some outputs arrive wrapped in a code fence, which hides every heading from the section check. The`unwrap` function handles the common case. ## Troubleshooting Ten problems account for nearly every failed first run. - **`KeyError: 'OPENAI_MODEL'` or `'ANTHROPIC_MODEL'`.** The environment variable is not set in this shell. Export it, or`source` your`.env.sh` file. - **`ModuleNotFoundError: No module named 'anthropic'`.** The virtual environment is not active. Run` source .venv/bin/activate` and try again. - **Authentication error or 401.** The API key is missing, mistyped or lacks billing. Print the first six characters with`echo ${OPENAI_API_KEY:0:6}` to confirm the shell sees it. - **Model not found or 404.** The model ID is wrong or your account lacks access. Rerun the two`curl` commands from Step 1 and copy an ID from the output. - **`AttributeError` on `responses` or `output_text`.** The OpenAI SDK is too old. Run` pip install --upgrade openai` . - **A Claude README stops mid-sentence.** The response hit`max_tokens` . Raise the value in`ask_claude` and check`stop_reason` on the response. - **Rate limit or 429 errors.** Add a short`time.sleep` between calls, or lower`RUNS` until your tier allows more. - **`no .md files under runs`.** The generator did not finish, or you passed the wrong folder. List`runs/` and confirm each model folder holds`run_01.md` through`run_05.md` . - **`cmds` shows `0/0` for a README.** The README has no fenced blocks with lines that start with`logtrim` or`python logtrim.py` , or it uses indented code blocks. Open the file and see how it formats examples. A README with no runnable examples is itself a gap to record. - **Every command fails with exit 2.** The model used a different file name than`sample.log` , so each command hits a missing file. Strengthen the prompt line about`sample.log` , rerun, and keep the change identical for both models. Two more issues come up on the blind step. If `report.py` raises a `KeyError` on a tag, you reran `blind.py` after filling in ratings, which reshuffled the tags and orphaned your scores. Run the blind step once per batch. And if the accuracy figure looks too good, open the JSON detail for one README and confirm that the flag and command counts are not zero, since an empty extraction produces a perfect score with nothing checked. ## Advanced Tips Once the baseline works, a handful of extensions turn it from a one-off into a habit. **Gate documentation in CI.** Run the scorer on any pull request that touches a README, and fail the build on a single error. This takes three lines: ``` python python score.py docs_candidates --json out.json python3 -c "import json,sys; sys.exit(1 if any(r['errors'] for r in json.load(open('out.json'))) else 0)" ``` **Test documentation drift.** Change the code, for example rename `--dry-run` to `--count`, then ask each model to update an existing README. Score whether the old flag disappears and the new one appears. Updating a document is a harder job than writing one from scratch, and models that excel at the second often slip on the first. The [long-form drift test](https://shattered.io/claude-vs-chatgpt-long-form-drift-test-12-steps-2026/) covers the same weakness in prose. **Raise the difficulty.** A single-file tool is a friendly target. Move to a package with subcommands, environment variables and a config file, then extend `truth.json` to describe them. Harder code is where differences between models tend to open up. **Compare tiers, not just vendors.** Run Claude Haiku 5.5 or Sonnet 5.5 against GPT-6.1 Terra, then Claude Fable 5.2 against a premium OpenAI model, and log the cost per clean README. The cheapest model that clears your error threshold is often the right default for documentation, with the larger one reserved for hard cases. **Report the worst run, not just the mean.** If a documentation bot produces one bad README in ten, reviewers will eventually miss it. The `worst` column in the summary already gives you this figure, so put it in your decision note. **Keep the prompt under version control.** Commit `prompts/`, `truth.json` and the scorer together, and tag the commit with the model IDs and date of each run. When a vendor ships a new model, you rerun the same harness in the time it takes to drink a coffee, and you compare today’s numbers against a baseline instead of a memory. The wider [AI and machine learning coverage](https://shattered.io/category/ai-machine-learning/) tracks model releases that should trigger such a rerun. ## FAQ ### Which is better for writing documentation, ChatGPT or Claude? The answer to any ChatGPT vs Claude question depends on the exact model, the plan and the task, and it changes as vendors ship updates. This harness measures it for your own pair on a task you can verify by running code. Run five or more samples per model before you decide. ### How many runs per model do I need? Five is the practical minimum for spotting spread, and ten is better when the means are close. A single run tells you almost nothing, because one lucky or unlucky sample can reverse the ranking. ### Should I test through the apps or the APIs? Use the APIs for a clean comparison. The apps add system prompts, memory and tool routing that differ by vendor and plan. If you care about the app experience, repeat the test there with the same prompt and treat it as a separate experiment. ### Is it safe to run commands that a language model wrote? It is safe only with guards. The scorer never starts a shell, skips lines with pipes and redirects, runs inside a throwaway copy of the project, closes stdin and enforces a ten-second timeout. Keep those guards, and do not point the runner at a project that can delete or send data. ### Why not ask another AI model to judge the READMEs? A judge model adds an uncontrolled variable and can share blind spots with one of the contestants. Executable checks give you numbers you can reproduce and defend. Human reading still has a place, which is why the blind step exists. ### Do I need a different target project for my own codebase? Yes, ideally. A README accuracy test predicts more when it mirrors your work. Replace `target/` with a small project you own, regenerate `truth.json` from its real flags, defaults and exit codes, and rerun. A target close to your real work predicts real behavior better than any public benchmark. ### Why does the scorer count exit code 1 as a pass? The tool uses exit code 1 to say that no lines matched, which is a legitimate outcome for an example filter. Only exit code 2, which signals invalid input, and timeouts count as failures.