Claude vs ChatGPT Reading Level: 12 Steps, 75 Min [2026] A single-file Python harness tested Claude Sonnet 5.5, Haiku 5.5 and Fable 5.2 against OpenAI's GPT-6.1 Sol and GPT-6.1 Terra on rewriting three passages to grade 6, 9 and 12 reading levels, scoring grade accuracy, fact survival, length drift and run-to-run spread. The test, published by Shattered, uses Flesch-Kincaid grade estimates and flags chatty preambles that distort sentence counts, and it notes Anthropic's Fable 5.2 launched November 9 at $8 per million input tokens and $40 per million output tokens. The harness deliberately does not judge voice, tone or invented claims, so its numbers serve as a screen rather than a full quality verdict. Ask Claude and ChatGPT to rewrite one dense paragraph for a sixth grader, and the two answers rarely land at the same reading level. One model keeps the long clauses and swaps a handful of words. The other chops everything into fragments and quietly drops a number. Claude vs ChatGPT for writing usually gets argued on taste, yet reading-level targeting is something a script can grade in about a second per file. Both lineups changed this autumn, so the test needs current models. Anthropic’s release notes https://support.claude.com/en/articles/12138966-release-notes?ref=vanja.io show Claude Sonnet 5.5 arriving on September 28 and Haiku 5.5 on October 7, and Claude Fable 5.2 followed on November 9 at $8 per million input tokens and $40 per million output tokens. OpenAI’s GPT-6 family now sits behind ChatGPT, with GPT-6.1 Sol and GPT-6.1 Terra as the newest names. A comparison built on last spring’s models measures products you can no longer pick in the menu. The harness is a single Python file. It sends three passages to each model at grade 6, 9 and 12, then reports grade accuracy, fact survival, length drift and run-to-run spread in one table. Where the voice-matching test https://shattered.io/claude-vs-chatgpt-voice-match-stylometry-12-steps-2026/ asks whether a model sounds like you, this one asks whether it can hit a stated target without losing the facts. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F What a Reading-Level Test Measures in Claude vs ChatGPT for Writing Writers hand models reading-level targets all the time. A help-center article should sit around grade 8. A patient letter might need grade 6. A privacy notice for a general audience lands somewhere in between. The model gets a source paragraph and a number, and it must return text that matches. That is a narrower job than “write well,” which makes it testable. Flesch-Kincaid grade estimates the US school year needed to follow a text. It uses two inputs: average words per sentence and average syllables per word. A passage with 28-word sentences and long Latinate vocabulary scores in the high teens. The same facts in 12-word sentences with plain verbs score near grade 6. The formula is blunt. It cannot tell clear prose from murky prose, and it rewards chopped-up sentences. But it gives the same answer every time, and that repeatability is exactly what a model comparison needs. A model can fail this task in two different ways, so the harness scores both. The first is missing the target, such as returning grade 13 prose when you asked for grade 9. The second is hitting the target by cheating, which means deleting the numbers, dates and names that made the passage worth rewriting. A model that shrinks 170 words to 80 will often score a lovely grade 6 and be useless to a reader. Every output therefore gets four checks. Grade error measures the distance from the target. Fact retention counts how many required details survived. Length ratio compares output words with source words. Spread measures how far the grade moves across repeated runs of the same prompt. A fifth flag catches chatty preambles such as “Here is the rewritten passage,” which distort sentence counts if left in. Those checks sit next to two other questions. The writing-rules compliance test https://shattered.io/claude-vs-chatgpt-writing-rules-compliance-test-12-steps-2026/ tracks whether a model obeys a house style guide, and a blind reader test lets humans pick favorites. Reading level lands between them: more objective than taste, narrower than rule-following. What the harness does not do matters just as much. It does not judge voice, tone or charm. It checks only the facts you list, so an invented claim slips through unless you spot it while reading the output. Treat the numbers as a screen. A rewrite that passes still deserves a human read, and one that fails tells you exactly where to look. Prerequisites and Versions to Install Before You Start The scoring half of the harness uses only the Python standard library. Target Python 3.11 or newer. The code in this walkthrough ran on Python 3.12.3. The generation half needs the official anthropic and openai packages, installed at their latest releases from PyPI. Skip them if you plan to paste app outputs by hand. You need one of two access routes. API mode requires an Anthropic API key and an OpenAI API key, exported as ANTHROPIC API KEY and OPENAI API KEY . App mode needs only chat accounts. On the ChatGPT side, OpenAI https://chatgpt.com/pricing/ lists Free at $0 a month and Plus at $20, and its Pro tiers page https://help.openai.com/en/articles/9793128-what-is-chatgpt-pro%252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525252525253F.csv lists three Pro levels at $100, $200 and $500 a month. The Pro tiers matter little here. Plus already includes access to GPT-6.1 Sol according to OpenAI’s pricing page, while Go does not. If you juggle several plans, a rundown of usage limits and plan prices https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ shows what each tier allows. A fair Claude vs ChatGPT run pairs the flagship of each line first, then repeats with the cheaper tier. Claude Fable 5.2 and GPT-6.1 Sol make the main duel, and the small models make the budget duel. | Model | Maker | Date | Published detail | Role in the test | |---|---|---|---|---| | Claude Fable 5.2 | Anthropic | Nov. 9, 2026 | Most capable for technical work, $8 input and $40 output per 1M tokens | Main Claude entry | | Claude Opus 5.5 | Anthropic | Sept. 22, 2026 | Costs 40% less to run than Opus 5 on typical workloads | Optional extra | | Claude Sonnet 5.5 | Anthropic | Sept. 28, 2026 | More than 30% faster than Sonnet 5, up to 30% cheaper for most work | Mid-tier entry | | Claude Haiku 5.5 | Anthropic | Oct. 7, 2026 | Anthropic’s fastest, most capable small model | Budget entry | | GPT-6.1 Sol | OpenAI | Introduced Sept. 29, 2026 | Rolling out to eligible paid plans, included with Plus | Main ChatGPT entry | | GPT-6.1 Terra | OpenAI | Oct. 26, 2026 | Fast, efficient model for high-volume professional work | Budget entry | | GPT-6 Sol and Luna | OpenAI | Oct. 7, 2026 | Plus, Pro, Business and Enterprise use Sol, Free and Go use Luna | What most app users get | Claude Fable 5.1 and Mythos 5.1 are history. Anthropic’s 5.2 releases superseded them on November 9, so Fable 5.2 is the flagship entry. In ChatGPT, GPT-6 replaces the old Latest selector, and existing Latest selections move to GPT-6 at the same reasoning effort. That detail matters because “ChatGPT” is no longer one consistent model. What you get depends on your plan and on what the picker shows. The cost of the default run stays small for each model. Your own run logs the number of calls and the input and output tokens, and you can price them at Fable 5.2’s published $8 and $40 per million tokens. Reasoning-heavy models can bill extra hidden output tokens, so set a one-dollar spending cap in each provider dashboard before the first call. For the ChatGPT side, price the exact model you pick on OpenAI’s own rate card, because API prices differ by model. Steps 1 to 3: Create the Project, the Test Passages and the Fact Lists Step 1: Create the folder and a virtual environment. Keep everything in one directory so the file paths in the script stay short. mkdir reading-level-lab && cd reading-level-lab mkdir passages facts outputs python3 -m venv .venv source .venv/bin/activate Windows: .venv\Scripts\activate pip install anthropic openai only needed for API mode Step 2: Save three test passages. Each passage runs 130 to 170 words of dense adult prose. The text below describes an invented company, Northwind Labs, on purpose. Real events can sit in a model’s training data, which lets it lean on memory instead of the page. Invented numbers also guarantee that every fact is checkable against the file in front of you. cat passages/p01.txt <<'EOF' On the night of March 4, Northwind Labs experienced a 47-minute outage of its customer billing portal after an automated certificate renewal job failed silently. The job, which runs every 60 days, had been migrated to a new scheduler the previous week, and the migration omitted the environment variable that held the credentials for the certificate authority. Because the failure was logged at the informational level rather than as an error, no alert reached the on-call engineer until customers began reporting browser warnings at 11:12 that night. The team restored service by issuing a replacement certificate manually and redeploying the load balancers in two regions. Subsequent analysis showed that 1,380 customers attempted to log in during the outage and that roughly one in five abandoned their session. To prevent recurrence, Northwind Labs has promoted certificate renewal failures to the critical alert tier, added a synthetic check that validates certificate expiry every hour, and required that any scheduler migration include a checklist of inherited secrets reviewed by a second engineer. EOF cat passages/p02.txt <<'EOF' Northwind Labs retains diagnostic logs for 30 days, after which they are automatically deleted unless a legal hold applies. Logs that contain account identifiers are pseudonymized within 24 hours of collection, using a keyed hash whose secret is rotated every 90 days. Customers who request deletion under the company's privacy policy receive confirmation within 10 business days, although backups containing the data may persist for up to 35 additional days until they expire in the normal rotation. Employees may access pseudonymized logs without approval, but any attempt to re-identify an individual requires written sign-off from the privacy office, and each approved request is recorded in an audit register that is reviewed quarterly by an external assessor. In the past year the privacy office received 212 deletion requests and fulfilled all of them within the stated window, while approving only three re-identification requests, each connected to an active fraud investigation. EOF cat passages/p03.txt <<'EOF' Beginning June 1, the Northwind Labs Team plan will move from per-seat billing to a usage-based model that charges $0.004 per processed document, with the first 25,000 documents each month included at no cost. Organizations that currently pay $18 per seat will be migrated automatically, and the company estimates that teams processing fewer than 90,000 documents a month will see their bill decrease. Teams above that threshold may see an increase, and the sales team will offer a transition credit equal to two months of the difference. Existing annual contracts will be honored until their renewal date, at which point the new structure applies. Customers who prefer predictable costs can opt into a committed-volume plan, which fixes the monthly price for twelve months in exchange for a minimum purchase of 500,000 documents per quarter. EOF The passages cover three genres: an incident postmortem, a data-retention policy and a pricing change. Source difficulty varies, which gives the grade targets something to push against. Step 9 prints the exact baseline for each one. Step 3: List the facts that must survive. Each fact is a list of acceptable spellings. The scorer treats a fact as kept when any spelling appears in the output, ignoring case. cat facts/p01.json <<'EOF' "47-minute","47 minutes","47 min" , "March 4" , "60 days","60-day","every 60" , "11:12" , "1,380","1380" , "one in five","1 in 5","20%","20 percent","a fifth","one-fifth" , "two regions","2 regions" , "every hour","hourly","each hour","once an hour" , "second engineer","another engineer","two engineers","second person" EOF cat facts/p02.json <<'EOF' "30 days","30-day" , "24 hours","a day","one day" , "90 days","every 90","90-day" , "10 business days","ten business days","10 working days" , "35 additional days","35 more days","35 extra days","35 days" , "quarterly","every three months","each quarter","every quarter" , "212" , "three","3 " , "fraud" EOF cat facts/p03.json <<'EOF' "June 1" , "$0.004","0.4 cents","four-tenths of a cent" , "25,000","25000" , "$18" , "90,000","90000" , "two months","2 months" , "500,000","500000" , "twelve months","12 months","a year","one year" , "per quarter","each quarter","every quarter","a quarter" EOF Alternatives exist for a reason. A model that writes “one-fifth” for “one in five” has preserved the fact, and a strict string match would punish it. Loose matches carry their own risk, though. The p02 list accepts the bare word “three” for the re-identification count, which is safe because that passage uses “three” once. A passage with several threes would need a tighter pattern. Read the missing-facts report by eye the first time you run it. Each file holds nine facts, so a perfect rewrite keeps 9 of 9. Notice that the lists include small details such as the second engineer who reviews a scheduler migration. Models trim small details first when asked for simpler text, so those entries catch quiet losses that a headline number would not. Steps 4 to 6: Build the Readability and Fact-Retention Scorer Step 4: Write the constants and the prompt. Start the file rlab.py with the imports, the three grade targets, a one-grade tolerance and the instructions both vendors receive. Identical wording for both models is the whole point. Tuning a prompt per vendor is a fair second experiment, but it belongs after the baseline. bash /usr/bin/env python3 """rlab.py - reading-level lab: rewrite to a target grade, then score the result.""" import argparse import csv import functools import json import re import statistics import sys import time from pathlib import Path ROOT = Path file .resolve .parent PASSAGES = ROOT / "passages" FACTS = ROOT / "facts" OUT = ROOT / "outputs" TARGETS = 6, 9, 12 TOLERANCE = 1.0 SYSTEM = "You are a careful editor. Rewrite the passage for the requested reading level. " "Keep every fact, number, name and date. Output only the rewritten passage, " "with no preamble, no headings and no notes." def build prompt passage, grade : return f"Rewrite the passage below so it reads at about US grade {grade} " "on the Flesch-Kincaid scale. Keep the meaning, every number and every " "named entity. Keep the length within 20% of the original.\n\n" f"PASSAGE:\n{passage}" The prompt names the target US grade level, demands every number and named entity, and caps the length either way. The system message forbids preambles and headings. Models still disobey that last rule now and then, which is why the scorer flags it. Step 5: Add the readability calculator. Four design choices live in this block. Words are letter runs or numbers, so a figure with a thousands separator counts as one word. Numbers count as two syllables because spoken digits are rarely one-syllable. The sentence splitter breaks only where a period, question mark or exclamation mark is followed by whitespace and a capital letter or digit, so a price like $0.004 survives intact. And a line that ends without punctuation gets a period appended, so bullet points and headings count as separate sentences instead of merging into one monster. ---------- scoring ---------- WORD RE = re.compile r" A-Za-z A-Za-z'’- |\d \d,. \d|\d" SENT RE = re.compile r" ?<= . ? \"'”’ \s+ ?= A-Z0-9\"'“ " PREAMBLE RE = re.compile r"^ here ?:'s| is| are |sure|certainly|of course|below is ", re.I def count syllables token : if token 0 .isdigit : return 2 w = re.sub r" ^a-z ", "", token.lower if len w <= 3: return 1 w = re.sub r" ?: ^laeiouy es|ed| ^laeiouy e $", "", w w = re.sub r"^y", "", w return max 1, len re.findall r" aeiouy {1,2}", w def clean raw : lines = raw.strip .splitlines pre = bool lines and bool PREAMBLE RE.match lines 0 .strip and lines 0 .rstrip .endswith ":" if pre: lines = lines 1: body = "\n".join lines .strip body = re.sub r" +", "", body body = re.sub r"^\s -• \s+", "", body, flags=re.M return body, pre def readability text : text = re.sub r" ^. ?\"'”’ \s \t \n+", r"\1.\n", text words = WORD RE.findall text sentences = s for s in SENT RE.split text.strip if s.strip if not words or not sentences: raise ValueError "empty text" syllables = sum count syllables w for w in words wps = len words / len sentences spw = syllables / len words return { "words": len words , "sentences": len sentences , "fk grade": 0.39 wps + 11.8 spw - 15.59, "reading ease": 206.835 - 1.015 wps - 84.6 spw, } The clean function strips one leading “Here is…:” line, Markdown symbols and bullet dashes before scoring. Without it, a model that wraps its answer in headers and bullets would score on formatting instead of language. If you care about whether edits stay faithful to the original beyond the facts you list, the editing fidelity test https://shattered.io/claude-vs-chatgpt-editing-fidelity-test-12-steps-2026/ goes deeper on that question. The syllable counter is a heuristic that drops silent endings and counts vowel groups. It will miss on a few words, and it can disagree with other calculators by a fraction of a grade. That is fine as long as every model passes through the same scorer. Compare results inside this harness and do not set them against numbers from a different tool. Step 6: Add fact retention and per-file scoring. This block loads the fact list, checks each spelling, compares word counts against the source and returns one row of metrics per output file. python def fact hits text, facts : low = text.lower return any alt.lower in low for alt in alts for alts in facts def load facts pid : return json.loads FACTS / f"{pid}.json" .read text encoding="utf-8" def load source pid : return PASSAGES / f"{pid}.txt" .read text encoding="utf-8" .strip def score text raw, pid, target : text, pre = clean raw r = readability text facts = load facts pid hits = fact hits text, facts src words = readability load source pid "words" err = r "fk grade" - target return { "passage": pid, "target": target, "fk grade": round r "fk grade" , 2 , "reading ease": round r "reading ease" , 1 , "error": round err, 2 , "hit": abs err <= TOLERANCE, "facts kept": sum hits , "facts total": len hits , "length ratio": round r "words" / src words, 2 , "preamble": pre, "missing": "; ".join f 0 for f, h in zip facts, hits if not h , } The missing field names the first spelling of each lost fact, so the final report can say exactly what a model dropped. A hit means the measured grade landed within one grade of the target. One grade is tight enough to separate models and loose enough to absorb the noise in a heuristic syllable count. Steps 7 and 8: Add the API Generators and the Command-Line Interface Step 7: Write the generation functions. Each provider call is a few lines. The Anthropic call joins every text block in the response, because a reply can carry non-text blocks as well. The OpenAI call falls back to an empty string when the message content is missing. Clients are created once and cached, and creation happens before the loop starts, so a missing package or key fails immediately instead of after three retries per file. python ---------- generation ---------- @functools.cache def anthropic client : import anthropic return anthropic.Anthropic @functools.cache def openai client : from openai import OpenAI return OpenAI def call anthropic model, user : resp = anthropic client .messages.create model=model, max tokens=1500, system=SYSTEM, messages= {"role": "user", "content": user} , return "".join b.text for b in resp.content if b.type == "text" def call openai model, user : resp = openai client .chat.completions.create model=model, messages= {"role": "system", "content": SYSTEM}, {"role": "user", "content": user} , return resp.choices 0 .message.content or "" def cmd gen args : call = call anthropic if args.provider == "anthropic" else call openai anthropic client if args.provider == "anthropic" else openai client dest = OUT / args.label dest.mkdir parents=True, exist ok=True for src in sorted PASSAGES.glob "p .txt" : for grade in TARGETS: for run in range 1, args.runs + 1 : path = dest / f"{src.stem} g{grade} r{run}.txt" if path.exists : continue for attempt in range 3 : try: text = call args.model, build prompt src.read text encoding="utf-8" .strip , grade break except Exception as exc: print f"retry {attempt + 1} for {path.name}: {exc}", file=sys.stderr time.sleep 5 attempt + 1 else: print f"gave up on {path.name}", file=sys.stderr continue path.write text text, encoding="utf-8" print "wrote", path.relative to ROOT Three details deserve attention. First, the code sets no temperature, because some current models reject or ignore that parameter. Second, it sets no token cap for OpenAI and a generous 1,500 for Anthropic, since passages are short. Third, the loop skips files that already exist, so an interrupted run resumes where it stopped and a rerun never pays twice for the same output. The model argument takes the exact model ID from each provider’s model list. Marketing names such as “Claude Fable 5.2” or “GPT-6.1 Sol” are not always valid IDs, and guessing one is the fastest route to a 404. Copy the string from the console or the API documentation instead. Step 8: Add the baseline, scoring and CLI commands. This last block scores the source passages, walks every folder under outputs/ , writes results.csv and prints one summary line per label. It also lists every dropped fact so you can tell a real loss from a spelling your list missed. ---------- reporting ---------- NAME RE = re.compile r"^ p\d+ g \d+ r \d+ \.txt$" def cmd baseline args : print f"{'passage':8} {'words': 5} {'sent': 4} {'FK grade': 8} {'ease': 6}" for src in sorted PASSAGES.glob "p .txt" : r = readability src.read text encoding="utf-8" print f"{src.stem:8} {r 'words' : 5} {r 'sentences' : 4} {r 'fk grade' : 8.2f} {r 'reading ease' : 6.1f}" def cmd score args : rows = for label dir in sorted p for p in OUT.iterdir if p.is dir : for f in sorted label dir.glob " .txt" : m = NAME RE.match f.name if not m: print f"skip {f.name}: name must look like p01 g6 r1.txt", file=sys.stderr continue row = score text f.read text encoding="utf-8" , m.group 1 , int m.group 2 row.update {"label": label dir.name, "run": int m.group 3 } rows.append row if not rows: sys.exit "no output files found under outputs/" with open ROOT / "results.csv", "w", newline="", encoding="utf-8" as fh: writer = csv.DictWriter fh, fieldnames=list rows 0 writer.writeheader writer.writerows rows print f"{'label':14} {'n': 3} {'hit%': 5} {'|err|': 6} {'bias': 6} {'facts%': 7} {'len': 5} {'pre': 3} {'spread': 6}" for label in sorted {r "label" for r in rows} : mine = r for r in rows if r "label" == label groups = {} for r in mine: groups.setdefault r "passage" , r "target" , .append r "fk grade" spreads = statistics.pstdev v for v in groups.values if len v 1 spread = f"{statistics.mean spreads :.2f}" if spreads else "n/a" n = len mine print f"{label:14} {n: 3} " f"{100 sum r 'hit' for r in mine / n: 5.0f} " f"{statistics.mean abs r 'error' for r in mine : 6.2f} " f"{statistics.mean r 'error' for r in mine : +6.2f} " f"{100 sum r 'facts kept' for r in mine / sum r 'facts total' for r in mine : 7.0f} " f"{statistics.mean r 'length ratio' for r in mine : 5.2f} " f"{sum r 'preamble' for r in mine : 3} " f"{spread: 6}" misses = r for r in rows if r "missing" if misses: print "\nDropped facts:" for r in misses: print f" {r 'label' } {r 'passage' } g{r 'target' } r{r 'run' }: {r 'missing' }" def main : parser = argparse.ArgumentParser description= doc sub = parser.add subparsers dest="cmd", required=True g = sub.add parser "gen", help="generate rewrites through an API" g.add argument "--provider", choices= "anthropic", "openai" , required=True g.add argument "--model", required=True, help="exact model ID from the provider's model list" g.add argument "--label", required=True, help="folder name under outputs/" g.add argument "--runs", type=int, default=3 g.set defaults func=cmd gen sub.add parser "baseline", help="score the source passages" .set defaults func=cmd baseline sub.add parser "score", help="score every file under outputs/" .set defaults func=cmd score args = parser.parse args args.func args if name == " main ": main The summary table has nine columns. hit% is the share of rewrites within tolerance. |err| is the mean absolute grade error. bias is the signed mean error, positive when outputs read too hard. facts% is the share of required facts kept. len is the mean length ratio. pre counts preambles. spread is the mean standard deviation of the grade across repeated runs, or n/a for a single run. Step 9: Prove the Scorer Works Before You Spend a Token Run the baseline first. It scores the three source passages and tells you how far each target sits from the starting point. python rlab.py baseline passage words sent FK grade ease p01 171 6 16.92 24.5 p02 149 5 18.13 18.2 p03 134 5 13.97 42.6 The sources sit between grade 14 and grade 18, so a grade 12 request is a mild trim while grade 6 demands a full rewrite. Those gaps are useful. A model that hits grade 12 easily but fails at grade 6 has a different weakness from one that misses everywhere. Now plant two hand-written rewrites, one careful and one sloppy, and confirm the scorer separates them. The good one keeps every fact in short sentences. The bad one opens with a preamble, crams everything into a single 82-word sentence and loses every listed fact. mkdir -p outputs/demo-good outputs/demo-bad cat outputs/demo-good/p01 g6 r1.txt <<'EOF' On March 4, the Northwind Labs billing site went down for 47 minutes. A job that renews the site's security certificate had failed, and nobody knew. The job runs every 60 days. A week earlier, the team had moved it to a new scheduler. The move left out a password the job needed. The failure was only logged as plain information, so no alert went off. The first sign of trouble came at 11:12 that night, when customers saw warnings in their browsers. The team fixed it by hand. They made a new certificate and restarted the load balancers in two regions. Later, the team found that 1,380 customers tried to log in during the outage. About one in five gave up. Now the company treats a failed renewal as a critical alert. It also checks every hour that the certificate is still good. And any move to a new scheduler needs a list of secrets, checked by a second engineer. EOF cat outputs/demo-bad/p01 g6 r1.txt <<'EOF' Here is the rewritten passage: Following a silent failure of an automated certificate renewal job that had been migrated to a new scheduler without the environment variable containing the certificate authority credentials, Northwind Labs experienced a prolonged outage of its customer billing portal, and because the failure had been logged at the informational level rather than as an error, the on-call engineer received no alert until customers started to report browser warnings, at which point the team issued a replacement certificate manually and redeployed the load balancers. EOF python rlab.py score label n hit% |err| bias facts% len pre spread demo-bad 1 0 32.55 +32.55 0 0.48 1 n/a demo-good 1 100 0.40 -0.40 100 0.95 0 n/a Dropped facts: demo-bad p01 g6 r1: 47-minute; March 4; 60 days; 11:12; 1,380; one in five; two regions; every hour; second engineer The good rewrite scores grade 5.60 against a target of 6, keeps 9 of 9 facts and lands at 95% of the source length. The bad one scores a far higher grade because its words form a single long sentence, keeps almost none of the facts and fills well under the source length. The scorer also stripped the preamble line before measuring and flagged it in the pre column. If your own run reproduces that pattern, the scorer is wired correctly. If it does not, fix it now, because every later result depends on it. Each metric answers a different failure, so each gets its own pass rule. The last column shows what the bad demo did to it. | Metric | What it catches | Pass rule | Bad demo result | |---|---|---|---| | Grade error error | Missing the requested reading level | Within 1.0 grade of target | Not listed | | Hit rate hit% | Share of rewrites inside tolerance | As high as possible | 0% | | Facts kept facts% | Numbers, dates and names lost in simplifying | 9 of 9 per passage | 0 of 9 | | Length ratio len | Hitting the grade by deleting content | Not listed | Not listed | | Preamble flag pre | Chatty openers and notes around the text | 0 flagged | 1 flagged | | Spread | Instability across repeated runs | Under about 1 grade | n/a one run | The length band mirrors the limit written into the prompt. The spread threshold is a judgment call and not a standard. Pick a number before you look at results so you do not fit the rule to the outcome. Steps 10 and 11: Collect Rewrites From Claude and ChatGPT Step 10: Generate the Claude outputs. Export the key, set the model ID and run the generator. The label becomes a folder name, so include the model and the date in it. That label is your audit trail when a vendor ships a new version next month. export ANTHROPIC API KEY="sk-ant-..." export CLAUDE MODEL="