cd /news/large-language-models/claude-vs-chatgpt-reading-level-12-s… · home › topics › large-language-models › article
[ARTICLE · art-171443] src=shattered.io ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Claude vs ChatGPT Reading Level: 12 Steps, 75 Min [2026]

A single-file Python harness tested Claude Sonnet 5.5, Haiku 5.5 and Fable 5.2 against OpenAI's GPT-6.1 Sol and GPT-6.1 Terra on rewriting three passages to grade 6, 9 and 12 reading levels, scoring grade accuracy, fact survival, length drift and run-to-run spread. The test, published by Shattered, uses Flesch-Kincaid grade estimates and flags chatty preambles that distort sentence counts, and it notes Anthropic's Fable 5.2 launched November 9 at $8 per million input tokens and $40 per million output tokens. The harness deliberately does not judge voice, tone or invented claims, so its numbers serve as a screen rather than a full quality verdict.

recommendation

When testing reading-level rewrites, log grade error, fact retention, length ratio and spread across repeated runs, and strip chatty preambles before counting sentences.

more in playbook: Evals and testing for LLM apps
by read35 min views5 publishedNov 17, 2026
Claude vs ChatGPT Reading Level: 12 Steps, 75 Min [2026]
Image: Shattered (auto-discovered)

Ask Claude and ChatGPT to rewrite one dense paragraph for a sixth grader, and the two answers rarely land at the same reading level. One model keeps the long clauses and swaps a handful of words. The other chops everything into fragments and quietly drops a number. Claude vs ChatGPT for writing usually gets argued on taste, yet reading-level targeting is something a script can grade in about a second per file.

Both lineups changed this autumn, so the test needs current models. Anthropic’s release notes show Claude Sonnet 5.5 arriving on September 28 and Haiku 5.5 on October 7, and Claude Fable 5.2 followed on November 9 at $8 per million input tokens and $40 per million output tokens. OpenAI’s GPT-6 family now sits behind ChatGPT, with GPT-6.1 Sol and GPT-6.1 Terra as the newest names. A comparison built on last spring’s models measures products you can no longer pick in the menu.

The harness is a single Python file. It sends three passages to each model at grade 6, 9 and 12, then reports grade accuracy, fact survival, length drift and run-to-run spread in one table. Where the voice-matching test asks whether a model sounds like you, this one asks whether it can hit a stated target without losing the facts.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

What a Reading-Level Test Measures in Claude vs ChatGPT for Writing #

Writers hand models reading-level targets all the time. A help-center article should sit around grade 8. A patient letter might need grade 6. A privacy notice for a general audience lands somewhere in between. The model gets a source paragraph and a number, and it must return text that matches. That is a narrower job than “write well,” which makes it testable.

Flesch-Kincaid grade estimates the US school year needed to follow a text. It uses two inputs: average words per sentence and average syllables per word. A passage with 28-word sentences and long Latinate vocabulary scores in the high teens. The same facts in 12-word sentences with plain verbs score near grade 6. The formula is blunt. It cannot tell clear prose from murky prose, and it rewards chopped-up sentences. But it gives the same answer every time, and that repeatability is exactly what a model comparison needs.

A model can fail this task in two different ways, so the harness scores both. The first is missing the target, such as returning grade 13 prose when you asked for grade 9. The second is hitting the target by cheating, which means deleting the numbers, dates and names that made the passage worth rewriting. A model that shrinks 170 words to 80 will often score a lovely grade 6 and be useless to a reader.

Every output therefore gets four checks. Grade error measures the distance from the target. Fact retention counts how many required details survived. Length ratio compares output words with source words. Spread measures how far the grade moves across repeated runs of the same prompt. A fifth flag catches chatty preambles such as “Here is the rewritten passage,” which distort sentence counts if left in.

Those checks sit next to two other questions. The writing-rules compliance test tracks whether a model obeys a house style guide, and a blind reader test lets humans pick favorites. Reading level lands between them: more objective than taste, narrower than rule-following.

What the harness does not do matters just as much. It does not judge voice, tone or charm. It checks only the facts you list, so an invented claim slips through unless you spot it while reading the output. Treat the numbers as a screen. A rewrite that passes still deserves a human read, and one that fails tells you exactly where to look.

Prerequisites and Versions to Install Before You Start #

The scoring half of the harness uses only the Python standard library. Target Python 3.11 or newer. The code in this walkthrough ran on Python 3.12.3. The generation half needs the official anthropic and openai packages, installed at their latest releases from PyPI. Skip them if you plan to paste app outputs by hand.

You need one of two access routes. API mode requires an Anthropic API key and an OpenAI API key, exported as ANTHROPIC_API_KEY and OPENAI_API_KEY. App mode needs only chat accounts. On the ChatGPT side, OpenAI lists Free at $0 a month and Plus at $20, and its Pro tiers page lists three Pro levels at $100, $200 and $500 a month. The Pro tiers matter little here. Plus already includes access to GPT-6.1 Sol according to OpenAI’s pricing page, while Go does not. If you juggle several plans, a rundown of usage limits and plan prices shows what each tier allows.

A fair Claude vs ChatGPT run pairs the flagship of each line first, then repeats with the cheaper tier. Claude Fable 5.2 and GPT-6.1 Sol make the main duel, and the small models make the budget duel.

Model Maker Date Published detail Role in the test
Claude Fable 5.2 Anthropic Nov. 9, 2026 Most capable for technical work, $8 input and $40 output per 1M tokens Main Claude entry
Claude Opus 5.5 Anthropic Sept. 22, 2026 Costs 40% less to run than Opus 5 on typical workloads Optional extra
Claude Sonnet 5.5 Anthropic Sept. 28, 2026 More than 30% faster than Sonnet 5, up to 30% cheaper for most work Mid-tier entry
Claude Haiku 5.5 Anthropic Oct. 7, 2026 Anthropic’s fastest, most capable small model Budget entry
GPT-6.1 Sol OpenAI Introduced Sept. 29, 2026 Rolling out to eligible paid plans, included with Plus Main ChatGPT entry
GPT-6.1 Terra OpenAI Oct. 26, 2026 Fast, efficient model for high-volume professional work Budget entry
GPT-6 Sol and Luna OpenAI Oct. 7, 2026 Plus, Pro, Business and Enterprise use Sol, Free and Go use Luna What most app users get

Claude Fable 5.1 and Mythos 5.1 are history. Anthropic’s 5.2 releases superseded them on November 9, so Fable 5.2 is the flagship entry. In ChatGPT, GPT-6 replaces the old Latest selector, and existing Latest selections move to GPT-6 at the same reasoning effort. That detail matters because “ChatGPT” is no longer one consistent model. What you get depends on your plan and on what the picker shows.

The cost of the default run stays small for each model. Your own run logs the number of calls and the input and output tokens, and you can price them at Fable 5.2’s published $8 and $40 per million tokens. Reasoning-heavy models can bill extra hidden output tokens, so set a one-dollar spending cap in each provider dashboard before the first call. For the ChatGPT side, price the exact model you pick on OpenAI’s own rate card, because API prices differ by model.

Steps 1 to 3: Create the Project, the Test Passages and the Fact Lists #

Step 1: Create the folder and a virtual environment. Keep everything in one directory so the file paths in the script stay short.

mkdir reading-level-lab && cd reading-level-lab
mkdir passages facts outputs
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install anthropic openai     # only needed for API mode

Step 2: Save three test passages. Each passage runs 130 to 170 words of dense adult prose. The text below describes an invented company, Northwind Labs, on purpose. Real events can sit in a model’s training data, which lets it lean on memory instead of the page. Invented numbers also guarantee that every fact is checkable against the file in front of you.

cat > passages/p01.txt <<'EOF'
On the night of March 4, Northwind Labs experienced a 47-minute outage of its customer billing portal after an automated certificate renewal job failed silently. The job, which runs every 60 days, had been migrated to a new scheduler the previous week, and the migration omitted the environment variable that held the credentials for the certificate authority. Because the failure was logged at the informational level rather than as an error, no alert reached the on-call engineer until customers began reporting browser warnings at 11:12 that night. The team restored service by issuing a replacement certificate manually and redeploying the load balancers in two regions. Subsequent analysis showed that 1,380 customers attempted to log in during the outage and that roughly one in five abandoned their session. To prevent recurrence, Northwind Labs has promoted certificate renewal failures to the critical alert tier, added a synthetic check that validates certificate expiry every hour, and required that any scheduler migration include a checklist of inherited secrets reviewed by a second engineer.
EOF

cat > passages/p02.txt <<'EOF'
Northwind Labs retains diagnostic logs for 30 days, after which they are automatically deleted unless a legal hold applies. Logs that contain account identifiers are pseudonymized within 24 hours of collection, using a keyed hash whose secret is rotated every 90 days. Customers who request deletion under the company's privacy policy receive confirmation within 10 business days, although backups containing the data may persist for up to 35 additional days until they expire in the normal rotation. Employees may access pseudonymized logs without approval, but any attempt to re-identify an individual requires written sign-off from the privacy office, and each approved request is recorded in an audit register that is reviewed quarterly by an external assessor. In the past year the privacy office received 212 deletion requests and fulfilled all of them within the stated window, while approving only three re-identification requests, each connected to an active fraud investigation.
EOF

cat > passages/p03.txt <<'EOF'
Beginning June 1, the Northwind Labs Team plan will move from per-seat billing to a usage-based model that charges $0.004 per processed document, with the first 25,000 documents each month included at no cost. Organizations that currently pay $18 per seat will be migrated automatically, and the company estimates that teams processing fewer than 90,000 documents a month will see their bill decrease. Teams above that threshold may see an increase, and the sales team will offer a transition credit equal to two months of the difference. Existing annual contracts will be honored until their renewal date, at which point the new structure applies. Customers who prefer predictable costs can opt into a committed-volume plan, which fixes the monthly price for twelve months in exchange for a minimum purchase of 500,000 documents per quarter.
EOF

The passages cover three genres: an incident postmortem, a data-retention policy and a pricing change. Source difficulty varies, which gives the grade targets something to push against. Step 9 prints the exact baseline for each one.

Step 3: List the facts that must survive. Each fact is a list of acceptable spellings. The scorer treats a fact as kept when any spelling appears in the output, ignoring case.

cat > facts/p01.json <<'EOF'
[["47-minute","47 minutes","47 min"],["March 4"],["60 days","60-day","every 60"],["11:12"],["1,380","1380"],["one in five","1 in 5","20%","20 percent","a fifth","one-fifth"],["two regions","2 regions"],["every hour","hourly","each hour","once an hour"],["second engineer","another engineer","two engineers","second person"]]
EOF

cat > facts/p02.json <<'EOF'
[["30 days","30-day"],["24 hours","a day","one day"],["90 days","every 90","90-day"],["10 business days","ten business days","10 working days"],["35 additional days","35 more days","35 extra days","35 days"],["quarterly","every three months","each quarter","every quarter"],["212"],["three","3 "],["fraud"]]
EOF

cat > facts/p03.json <<'EOF'
[["June 1"],["$0.004","0.4 cents","four-tenths of a cent"],["25,000","25000"],["$18"],["90,000","90000"],["two months","2 months"],["500,000","500000"],["twelve months","12 months","a year","one year"],["per quarter","each quarter","every quarter","a quarter"]]
EOF

Alternatives exist for a reason. A model that writes “one-fifth” for “one in five” has preserved the fact, and a strict string match would punish it. Loose matches carry their own risk, though. The p02 list accepts the bare word “three” for the re-identification count, which is safe because that passage uses “three” once. A passage with several threes would need a tighter pattern. Read the missing-facts report by eye the first time you run it.

Each file holds nine facts, so a perfect rewrite keeps 9 of 9. Notice that the lists include small details such as the second engineer who reviews a scheduler migration. Models trim small details first when asked for simpler text, so those entries catch quiet losses that a headline number would not.

Steps 4 to 6: Build the Readability and Fact-Retention Scorer #

Step 4: Write the constants and the prompt. Start the file rlab.py with the imports, the three grade targets, a one-grade tolerance and the instructions both vendors receive. Identical wording for both models is the whole point. Tuning a prompt per vendor is a fair second experiment, but it belongs after the baseline.

#!/usr/bin/env python3
"""rlab.py - reading-level lab: rewrite to a target grade, then score the result."""
import argparse
import csv
import functools
import json
import re
import statistics
import sys
import time
from pathlib import Path

ROOT = Path(__file__).resolve().parent
PASSAGES = ROOT / "passages"
FACTS = ROOT / "facts"
OUT = ROOT / "outputs"
TARGETS = [6, 9, 12]
TOLERANCE = 1.0

SYSTEM = (
    "You are a careful editor. Rewrite the passage for the requested reading level. "
    "Keep every fact, number, name and date. Output only the rewritten passage, "
    "with no preamble, no headings and no notes."
)

def build_prompt(passage, grade):
    return (
        f"Rewrite the passage below so it reads at about US grade {grade} "
        "on the Flesch-Kincaid scale. Keep the meaning, every number and every "
        "named entity. Keep the length within 20% of the original.\n\n"
        f"PASSAGE:\n{passage}"
    )

The prompt names the target US grade level, demands every number and named entity, and caps the length either way. The system message forbids preambles and headings. Models still disobey that last rule now and then, which is why the scorer flags it.

Step 5: Add the readability calculator. Four design choices live in this block. Words are letter runs or numbers, so a figure with a thousands separator counts as one word. Numbers count as two syllables because spoken digits are rarely one-syllable. The sentence splitter breaks only where a period, question mark or exclamation mark is followed by whitespace and a capital letter or digit, so a price like $0.004 survives intact. And a line that ends without punctuation gets a period appended, so bullet points and headings count as separate sentences instead of merging into one monster.

WORD_RE = re.compile(r"[A-Za-z][A-Za-z'’-]*|\d[\d,.]*\d|\d")
SENT_RE = re.compile(r"(?<=[.!?])[\"'”’)]*\s+(?=[A-Z0-9\"'“(])")
PREAMBLE_RE = re.compile(r"^(here(?:'s| is| are)|sure|certainly|of course|below is)", re.I)

def count_syllables(token):
    if token[0].isdigit():
        return 2
    w = re.sub(r"[^a-z]", "", token.lower())
    if len(w) <= 3:
        return 1
    w = re.sub(r"(?:[^laeiouy]es|ed|[^laeiouy]e)$", "", w)
    w = re.sub(r"^y", "", w)
    return max(1, len(re.findall(r"[aeiouy]{1,2}", w)))

def clean(raw):
    lines = raw.strip().splitlines()
    pre = bool(lines) and bool(PREAMBLE_RE.match(lines[0].strip())) and lines[0].rstrip().endswith(":")
    if pre:
        lines = lines[1:]
    body = "\n".join(lines).strip()
    body = re.sub(r"[*_#`>]+", "", body)
    body = re.sub(r"^\s*[-•]\s+", "", body, flags=re.M)
    return body, pre

def readability(text):
    text = re.sub(r"([^.!?\"'”’)\s])[ \t]*\n+", r"\1.\n", text)
    words = WORD_RE.findall(text)
    sentences = [s for s in SENT_RE.split(text.strip()) if s.strip()]
    if not words or not sentences:
        raise ValueError("empty text")
    syllables = sum(count_syllables(w) for w in words)
    wps = len(words) / len(sentences)
    spw = syllables / len(words)
    return {
        "words": len(words),
        "sentences": len(sentences),
        "fk_grade": 0.39 * wps + 11.8 * spw - 15.59,
        "reading_ease": 206.835 - 1.015 * wps - 84.6 * spw,
    }

The clean function strips one leading “Here is…:” line, Markdown symbols and bullet dashes before scoring. Without it, a model that wraps its answer in headers and bullets would score on formatting instead of language. If you care about whether edits stay faithful to the original beyond the facts you list, the editing fidelity test goes deeper on that question.

The syllable counter is a heuristic that drops silent endings and counts vowel groups. It will miss on a few words, and it can disagree with other calculators by a fraction of a grade. That is fine as long as every model passes through the same scorer. Compare results inside this harness and do not set them against numbers from a different tool.

Step 6: Add fact retention and per-file scoring. This block loads the fact list, checks each spelling, compares word counts against the source and returns one row of metrics per output file.

def fact_hits(text, facts):
    low = text.lower()
    return [any(alt.lower() in low for alt in alts) for alts in facts]

def load_facts(pid):
    return json.loads((FACTS / f"{pid}.json").read_text(encoding="utf-8"))

def load_source(pid):
    return (PASSAGES / f"{pid}.txt").read_text(encoding="utf-8").strip()

def score_text(raw, pid, target):
    text, pre = clean(raw)
    r = readability(text)
    facts = load_facts(pid)
    hits = fact_hits(text, facts)
    src_words = readability(load_source(pid))["words"]
    err = r["fk_grade"] - target
    return {
        "passage": pid,
        "target": target,
        "fk_grade": round(r["fk_grade"], 2),
        "reading_ease": round(r["reading_ease"], 1),
        "error": round(err, 2),
        "hit": abs(err) <= TOLERANCE,
        "facts_kept": sum(hits),
        "facts_total": len(hits),
        "length_ratio": round(r["words"] / src_words, 2),
        "preamble": pre,
        "missing": "; ".join(f[0] for f, h in zip(facts, hits) if not h),
    }

The missing field names the first spelling of each lost fact, so the final report can say exactly what a model dropped. A hit means the measured grade landed within one grade of the target. One grade is tight enough to separate models and loose enough to absorb the noise in a heuristic syllable count.

Steps 7 and 8: Add the API Generators and the Command-Line Interface #

Step 7: Write the generation functions. Each provider call is a few lines. The Anthropic call joins every text block in the response, because a reply can carry non-text blocks as well. The OpenAI call falls back to an empty string when the message content is missing. Clients are created once and cached, and creation happens before the loop starts, so a missing package or key fails immediately instead of after three retries per file.

@functools.cache
def anthropic_client():
    import anthropic

    return anthropic.Anthropic()

@functools.cache
def openai_client():
    from openai import OpenAI

    return OpenAI()

def call_anthropic(model, user):
    resp = anthropic_client().messages.create(
        model=model,
        max_tokens=1500,
        system=SYSTEM,
        messages=[{"role": "user", "content": user}],
    )
    return "".join(b.text for b in resp.content if b.type == "text")

def call_openai(model, user):
    resp = openai_client().chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}],
    )
    return resp.choices[0].message.content or ""

def cmd_gen(args):
    call = call_anthropic if args.provider == "anthropic" else call_openai
    (anthropic_client if args.provider == "anthropic" else openai_client)()
    dest = OUT / args.label
    dest.mkdir(parents=True, exist_ok=True)
    for src in sorted(PASSAGES.glob("p*.txt")):
        for grade in TARGETS:
            for run in range(1, args.runs + 1):
                path = dest / f"{src.stem}__g{grade}__r{run}.txt"
                if path.exists():
                    continue
                for attempt in range(3):
                    try:
                        text = call(args.model, build_prompt(src.read_text(encoding="utf-8").strip(), grade))
                        break
                    except Exception as exc:
                        print(f"retry {attempt + 1} for {path.name}: {exc}", file=sys.stderr)
                        time.sleep(5 * (attempt + 1))
                else:
                    print(f"gave up on {path.name}", file=sys.stderr)
                    continue
                path.write_text(text, encoding="utf-8")
                print("wrote", path.relative_to(ROOT))

Three details deserve attention. First, the code sets no temperature, because some current models reject or ignore that parameter. Second, it sets no token cap for OpenAI and a generous 1,500 for Anthropic, since passages are short. Third, the loop skips files that already exist, so an interrupted run resumes where it stopped and a rerun never pays twice for the same output.

The model argument takes the exact model ID from each provider’s model list. Marketing names such as “Claude Fable 5.2” or “GPT-6.1 Sol” are not always valid IDs, and guessing one is the fastest route to a 404. Copy the string from the console or the API documentation instead.

Step 8: Add the baseline, scoring and CLI commands. This last block scores the source passages, walks every folder under outputs/, writes results.csv and prints one summary line per label. It also lists every dropped fact so you can tell a real loss from a spelling your list missed.

NAME_RE = re.compile(r"^(p\d+)__g(\d+)__r(\d+)\.txt$")

def cmd_baseline(_args):
    print(f"{'passage':8} {'words':>5} {'sent':>4} {'FK grade':>8} {'ease':>6}")
    for src in sorted(PASSAGES.glob("p*.txt")):
        r = readability(src.read_text(encoding="utf-8"))
        print(f"{src.stem:8} {r['words']:>5} {r['sentences']:>4} {r['fk_grade']:>8.2f} {r['reading_ease']:>6.1f}")

def cmd_score(_args):
    rows = []
    for label_dir in sorted(p for p in OUT.iterdir() if p.is_dir()):
        for f in sorted(label_dir.glob("*.txt")):
            m = NAME_RE.match(f.name)
            if not m:
                print(f"skip {f.name}: name must look like p01__g6__r1.txt", file=sys.stderr)
                continue
            row = score_text(f.read_text(encoding="utf-8"), m.group(1), int(m.group(2)))
            row.update({"label": label_dir.name, "run": int(m.group(3))})
            rows.append(row)
    if not rows:
        sys.exit("no output files found under outputs/")
    with open(ROOT / "results.csv", "w", newline="", encoding="utf-8") as fh:
        writer = csv.DictWriter(fh, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)

    print(f"{'label':14} {'n':>3} {'hit%':>5} {'|err|':>6} {'bias':>6} {'facts%':>7} {'len':>5} {'pre':>3} {'spread':>6}")
    for label in sorted({r["label"] for r in rows}):
        mine = [r for r in rows if r["label"] == label]
        groups = {}
        for r in mine:
            groups.setdefault((r["passage"], r["target"]), []).append(r["fk_grade"])
        spreads = [statistics.pstdev(v) for v in groups.values() if len(v) > 1]
        spread = f"{statistics.mean(spreads):.2f}" if spreads else "n/a"
        n = len(mine)
        print(
            f"{label:14} {n:>3} "
            f"{100 * sum(r['hit'] for r in mine) / n:>5.0f} "
            f"{statistics.mean(abs(r['error']) for r in mine):>6.2f} "
            f"{statistics.mean(r['error'] for r in mine):>+6.2f} "
            f"{100 * sum(r['facts_kept'] for r in mine) / sum(r['facts_total'] for r in mine):>7.0f} "
            f"{statistics.mean(r['length_ratio'] for r in mine):>5.2f} "
            f"{sum(r['preamble'] for r in mine):>3} "
            f"{spread:>6}"
        )
    misses = [r for r in rows if r["missing"]]
    if misses:
        print("\nDropped facts:")
        for r in misses:
            print(f"  {r['label']} {r['passage']} g{r['target']} r{r['run']}: {r['missing']}")

def main():
    parser = argparse.ArgumentParser(description=__doc__)
    sub = parser.add_subparsers(dest="cmd", required=True)
    g = sub.add_parser("gen", help="generate rewrites through an API")
    g.add_argument("--provider", choices=["anthropic", "openai"], required=True)
    g.add_argument("--model", required=True, help="exact model ID from the provider's model list")
    g.add_argument("--label", required=True, help="folder name under outputs/")
    g.add_argument("--runs", type=int, default=3)
    g.set_defaults(func=cmd_gen)
    sub.add_parser("baseline", help="score the source passages").set_defaults(func=cmd_baseline)
    sub.add_parser("score", help="score every file under outputs/").set_defaults(func=cmd_score)
    args = parser.parse_args()
    args.func(args)

if __name__ == "__main__":
    main()

The summary table has nine columns. hit% is the share of rewrites within tolerance. |err| is the mean absolute grade error. bias is the signed mean error, positive when outputs read too hard. facts% is the share of required facts kept. len is the mean length ratio. pre counts preambles. spread is the mean standard deviation of the grade across repeated runs, or n/a for a single run.

Step 9: Prove the Scorer Works Before You Spend a Token #

Run the baseline first. It scores the three source passages and tells you how far each target sits from the starting point.

python rlab.py baseline
passage  words sent FK grade   ease
p01        171    6    16.92   24.5
p02        149    5    18.13   18.2
p03        134    5    13.97   42.6

The sources sit between grade 14 and grade 18, so a grade 12 request is a mild trim while grade 6 demands a full rewrite. Those gaps are useful. A model that hits grade 12 easily but fails at grade 6 has a different weakness from one that misses everywhere.

Now plant two hand-written rewrites, one careful and one sloppy, and confirm the scorer separates them. The good one keeps every fact in short sentences. The bad one opens with a preamble, crams everything into a single 82-word sentence and loses every listed fact.

mkdir -p outputs/demo-good outputs/demo-bad

cat > outputs/demo-good/p01__g6__r1.txt <<'EOF'
On March 4, the Northwind Labs billing site went down for 47 minutes. A job that renews the site's security certificate had failed, and nobody knew. The job runs every 60 days. A week earlier, the team had moved it to a new scheduler. The move left out a password the job needed.

The failure was only logged as plain information, so no alert went off. The first sign of trouble came at 11:12 that night, when customers saw warnings in their browsers. The team fixed it by hand. They made a new certificate and restarted the load balancers in two regions.

Later, the team found that 1,380 customers tried to log in during the outage. About one in five gave up. Now the company treats a failed renewal as a critical alert. It also checks every hour that the certificate is still good. And any move to a new scheduler needs a list of secrets, checked by a second engineer.
EOF

cat > outputs/demo-bad/p01__g6__r1.txt <<'EOF'
Here is the rewritten passage:

Following a silent failure of an automated certificate renewal job that had been migrated to a new scheduler without the environment variable containing the certificate authority credentials, Northwind Labs experienced a prolonged outage of its customer billing portal, and because the failure had been logged at the informational level rather than as an error, the on-call engineer received no alert until customers started to report browser warnings, at which point the team issued a replacement certificate manually and redeployed the load balancers.
EOF
python rlab.py score
label            n  hit%  |err|   bias  facts%   len pre spread
demo-bad         1     0  32.55 +32.55       0  0.48   1    n/a
demo-good        1   100   0.40  -0.40     100  0.95   0    n/a

Dropped facts:
  demo-bad p01 g6 r1: 47-minute; March 4; 60 days; 11:12; 1,380; one in five; two regions; every hour; second engineer

The good rewrite scores grade 5.60 against a target of 6, keeps 9 of 9 facts and lands at 95% of the source length. The bad one scores a far higher grade because its words form a single long sentence, keeps almost none of the facts and fills well under the source length. The scorer also stripped the preamble line before measuring and flagged it in the pre column. If your own run reproduces that pattern, the scorer is wired correctly. If it does not, fix it now, because every later result depends on it.

Each metric answers a different failure, so each gets its own pass rule. The last column shows what the bad demo did to it.

Metric What it catches Pass rule Bad demo result
Grade error (error) Missing the requested reading level Within 1.0 grade of target Not listed
Hit rate (hit%) Share of rewrites inside tolerance As high as possible 0%
Facts kept (facts%) Numbers, dates and names lost in simplifying 9 of 9 per passage 0 of 9
Length ratio (len) Hitting the grade by deleting content Not listed Not listed
Preamble flag (pre) Chatty openers and notes around the text 0 flagged 1 flagged
Spread Instability across repeated runs Under about 1 grade n/a (one run)

The length band mirrors the limit written into the prompt. The spread threshold is a judgment call and not a standard. Pick a number before you look at results so you do not fit the rule to the outcome.

Steps 10 and 11: Collect Rewrites From Claude and ChatGPT #

Step 10: Generate the Claude outputs. Export the key, set the model ID and run the generator. The label becomes a folder name, so include the model and the date in it. That label is your audit trail when a vendor ships a new version next month.

export ANTHROPIC_API_KEY="sk-ant-..."
export CLAUDE_MODEL="<exact ID of Claude Fable 5.2 from the model list>"
python rlab.py gen --provider anthropic --model "$CLAUDE_MODEL" --label claude-fable-5-2-nov17 --runs 3

Step 11: Generate the ChatGPT outputs. Use the same command with the OpenAI provider. Pair Fable 5.2 with GPT-6.1 Sol for the flagship duel, then run Haiku 5.5 against GPT-6.1 Terra for the budget duel.

export OPENAI_API_KEY="sk-..."
export OPENAI_MODEL="<exact ID of GPT-6.1 Sol from the model list>"
python rlab.py gen --provider openai --model "$OPENAI_MODEL" --label gpt-6-1-sol-nov17 --runs 3

Each command writes 27 files and prints a line per file. If a call fails, the script retries up to three times with growing s and then moves on, so check the terminal for “gave up” lines and rerun the same command to fill the gaps.

Paste mode for app accounts

No API keys? Paste mode works with any chat account. Create a folder such as outputs/chatgpt-app-plus-nov17, open a brand-new chat for each rewrite, paste the prompt text from build_prompt with the passage attached and save the reply into a file named exactly like the generated ones, for example p01__g6__r1.txt. One repeat per cell means 9 files per model, which takes about 15 minutes by hand.

Clear anything that carries state between chats. Memory features and saved instructions can shift tone and sentence length, and they differ between accounts. If you rely on persistent instructions in daily work, test once with them on and once with them off, using a shared spec such as the custom instructions pack so both vendors see the same text.

Write the model label shown in the picker into a notes file next to each folder. OpenAI said on October 7 that Free and Go chats run GPT-6 Luna while Plus, Pro, Business and Enterprise chats run GPT-6 Sol, and its pricing page says Go does not include GPT-6.1 Sol. Two people on different plans who both say “I tested ChatGPT” have tested different models.

Step 12: Score the Outputs and Read the Table #

python rlab.py score

The command rescans every folder, rewrites results.csv and prints one line per label plus the dropped-facts report. Read it in a fixed order. Start with facts%, because a model that loses facts has disqualified itself regardless of grade. Next check len for deletion. Then look at hit% and bias together, and finish with spread and pre.

Most result tables fall into a handful of patterns. Each pattern points to a likely cause and a next move.

Pattern in the table Likely cause Next action
High hit%, facts% below 100 Model deletes detail to simplify Rerun with a stricter keep-every-number instruction and compare len
Positive bias at grade 6 Model keeps source vocabulary and long clauses Check which passage drives the miss and read its output by eye
Negative bias at grade 12 Model over-simplifies when asked for a higher level Test an audience description such as “college-educated professional”
Not listed Summarizing instead of rewriting Open the files, then tighten the length rule in the prompt
High spread, decent average Good on average but unpredictable per run Add runs, since three may hide a long tail
pre above 0 System instruction ignored or app wrapper text Keep the flag, and trust scores only after cleaning
One passage fails for both vendors Passage is dense with numbers and acronyms Judge on the other two, then re-check the heuristic

To pick a winner in a Claude vs ChatGPT table, count accepted rewrites. Define accepted as inside tolerance, every fact kept and length ratio inside the length band set in the prompt. Divide accepted rewrites by total rewrites for each model. When API cost matters, divide the money spent by accepted rewrites to get a cost per usable output, the same idea the cost per accepted 1,000 words test applies to full drafts.

Finally, read at least three outputs per model with your own eyes. The score cannot see invented content. A rewrite can keep all nine listed facts and still add a claim that the passage never made. The source-grounded audit shows a systematic way to catch that kind of drift.

The Complete Working Project: Files, Commands and a Final Check #

The code blocks from Steps 4 through 8 form one file. Paste them into rlab.py in order: header, readability scorer, fact scoring, generators, reporting. The file ends with a main guard, and nothing else is needed. Scoring uses only the standard library. The SDKs load lazily, so baseline and score run on a machine with neither package installed.

reading-level-lab/
  rlab.py                 # Steps 4-8, one file
  passages/
    p01.txt  p02.txt  p03.txt
  facts/
    p01.json p02.json p03.json
  outputs/
    claude-fable-5-2-nov17/    p01__g6__r1.txt ... p03__g12__r3.txt
    gpt-6-1-sol-nov17/         p01__g6__r1.txt ... p03__g12__r3.txt
  results.csv             # written by: python rlab.py score

A full run from a clean folder takes five commands.

python rlab.py baseline
python rlab.py gen --provider anthropic --model "$CLAUDE_MODEL" --label claude-fable-5-2-nov17 --runs 3
python rlab.py gen --provider openai --model "$OPENAI_MODEL" --label gpt-6-1-sol-nov17 --runs 3
python rlab.py score
head -3 results.csv

The first rows of results.csv look like the line below. Open the file in a spreadsheet to filter by passage or target, which is the quickest way to see whether a model struggles with one genre or one grade level. Close the spreadsheet before the next score run, or the write will fail.

passage,target,fk_grade,reading_ease,error,hit,facts_kept,facts_total,length_ratio,preamble,missing,label,run
p01,6,5.6,75.5,-0.4,True,9,9,0.95,False,,demo-good,1

Delete the two demo folders before the real comparison, or keep them as permanent regression fixtures. They cost nothing and show at a glance whether a later edit to the scorer changed the numbers.

Common Pitfalls That Skew a Claude vs ChatGPT Comparison #

  • Mixing app runs with API runs. Apps add system prompts, memory and formatting that the API does not. Compare app to app or API to API, and label every folder with the route you used.
  • Letting saved instructions leak in. A custom style note in one account and none in the other turns the test into a test of your settings. Clear both, or set both to the same text.
  • Judging from a single run. One rewrite per cell can flatter or sink a model by luck. Three runs expose the spread, and ten runs on a close call are cheap insurance.
  • Rewarding deletion. The bad demo kept far less than the full source length. Real models do the same in milder form, so never read hit% without facts% and len next to it.
  • Letting formatting distort the score. A bullet-style rewrite of the retention passage can shrink the text and drop many of the facts, yet still score well on the readability grade.
  • Using different prompts per vendor on the first pass. Per-model prompt tuning is legitimate later. At the baseline stage it hides which model follows instructions better.
  • Treating “ChatGPT” as one model. Plan, picker and rollout stage decide which GPT-6 variant answers. Record the exact label, or two readers will argue about different models.
  • Trusting number-heavy passages blindly. Figures and acronyms inflate syllable counts and sentence lengths in ways a heuristic handles poorly. If one passage fails for every model, check it before you blame the models.

One more caution concerns scope. Reading level is one writing skill among many. Pair this harness with a human panel such as the blind test before you declare an overall winner in Claude vs ChatGPT for writing. Neither tool replaces the other.

Troubleshooting: 11 Errors and Their Fixes #

  • ModuleNotFoundError: No module named ‘anthropic’ (or ‘openai’). Activate the virtual environment and runpip install anthropic openai . The generator fails fast on purpose, so you see this error right away instead of after a pile of retries.
  • AuthenticationError or HTTP 401. The key is missing from the current shell. Export it in the same terminal that runs the script, and check for stray quotes or spaces. The variable names must be exactlyANTHROPIC_API_KEY andOPENAI_API_KEY .
  • Model not found (HTTP 404). You passed a marketing name instead of an API ID, or your account lacks access to that model. Copy the ID from the provider’s model list and try again.
  • Rate limit errors. The script retries three times with 5, 10 and 15 second s. If files are still missing, rerun the same command. It skips finished files. Lower--runs if the limit persists.
  • “no output files found under outputs/”. The scorer expectsoutputs/LABEL/ folders. Run it from the project root and check that your label folders sit directly insideoutputs .
  • “skip …: name must look like p01__g6__r1.txt”. Pasted files need the exact pattern: passage ID, double underscore,g plus the target grade, double underscore,r plus the run number.
  • ValueError: empty text. The model returned nothing, which happens with refusals or when hidden reasoning consumes the output budget. Delete the empty file, raisemax_tokens for Anthropic, and rerun.
  • Absurd grades above 30. The output has one very long sentence or no sentence punctuation. Open the file and look at the sentence count in the CSV. The bad demo shows the pattern: 82 words, 1 sentence, grade 38.55.
  • FileNotFoundError for facts/p04.json. You added a passage without a matching fact list. Create the JSON file, even if it holds a single fact.
  • UnicodeDecodeError on Windows. A pasted file was saved in a legacy code page. Re-save it as UTF-8. The script reads and writes UTF-8 everywhere, so the file itself is the problem.
  • PermissionError when writing results.csv. A spreadsheet has the file open. Close it and runscore again.

Advanced Tips: Refine Loops, Bigger Matrices and Version Pinning #

Close the gap with a refine loop

Real work rarely stops at the first draft. A refine loop measures the first attempt, tells the model its actual grade and asks for a correction. Because the scorer already exists, the loop takes a dozen lines. Save the first-pass results and the refined results under separate labels, since the first number shows raw accuracy and the second shows how well a model takes feedback.

def refine(call, model, passage, grade, rounds=2):
    prompt = build_prompt(passage, grade)
    text = call(model, prompt)
    for _ in range(rounds):
        body, _pre = clean(text)
        measured = readability(body)["fk_grade"]
        if abs(measured - grade) <= TOLERANCE:
            break
        direction = "shorter sentences and simpler words" if measured > grade else "slightly richer sentences"
        text = call(model, f"{prompt}\n\nYour previous attempt measured grade {measured:.1f}. "
                           f"Revise it toward grade {grade} using {direction}.\n\nPREVIOUS ATTEMPT:\n{body}")
    return text

Cap the rounds at two or three. Each extra round multiplies cost, and a model that needs four rounds to hit grade 6 is giving you a useful negative result already. Watch for overshoot too, because a blunt “simpler words” instruction can push a rewrite far below the target.

Beyond that, widen the matrix. Three passages and three targets give 9 cells per model, which is enough for a screen and thin for a verdict. Add passages from your own field until you have at least ten, and add targets at grade 4, 8 and 14 to see where each model’s range ends. Read the per-target breakdown in results.csv before you trust the pooled averages.

Test audience wording as well. Replace “grade 6” with “a nervous first-time customer” and compare hit rates. Some models track a described reader better than a number, and your real prompts probably use described readers anyway. Keep the grade number out of that variant so the two prompts stay independent.

Length matters too. Whether a model holds its reading level across a 3,000-word document is the question the long-form drift test asks, and the two tests answer different halves of the same worry.

Pin versions and rerun after each release

Both vendors ship quickly. Anthropic replaced Fable 5.1 with Fable 5.2 on November 9, and a November 15 roundup from The Signal covered Claude joining calls and ChatGPT paying in a single issue. Store the model ID, date and account route in each folder name, keep the harness in version control and rerun the full matrix after any release you plan to adopt. A rerun that costs cents beats an assumption built on last month’s results.

For more model release coverage and comparisons, browse the AI and machine learning section.

Frequently Asked Questions #

Which is better for writing, Claude or ChatGPT?

It depends on the job, and the honest answer comes from your own passages, not from a Claude vs ChatGPT verdict someone else wrote. This test settles one narrow question, which is how well each model hits a requested reading level while keeping facts. Pair it with tests for voice, rules and editing before you generalize.

Is the Flesch-Kincaid grade reliable enough for a model comparison?

It is repeatable, and repeatability is what a comparison needs. It is not a measure of clarity. A text full of short, confusing sentences scores low. Use it to compare models against each other inside one scorer, and keep a human read in the loop.

Can I run this without API keys?

Yes. Paste mode needs only chat accounts. Save each reply into a correctly named file under outputs/LABEL/ and run python rlab.py score. Use one repeat per cell if time is short, and note the picker label for every folder.

How many runs per cell are enough?

Three is the minimum that produces a spread number. If two models land within half a grade of each other, add runs until the gap either grows or disappears.

Which models should I compare first?

Start with Claude Fable 5.2 against GPT-6.1 Sol, the flagship general models of each line as of November 17, 2026. Then run Claude Haiku 5.5 against GPT-6.1 Terra for the budget tier. Add Sonnet 5.5 if mid-tier cost matters to you.

Why use invented passages instead of real articles?

Invented text cannot appear in training data, and every fact in it is known to you. That keeps the fact-retention check honest. Once the harness works, swap in your own documents and write fact lists for them.

How much does a full run cost?

Around 30 cents per model for the default 27 calls on Claude Fable 5.2 at its published $8 and $40 per million tokens, by the estimate in the prerequisites. Price the ChatGPT side from the rate card for the model you choose, and set a spending cap before you start.

When should I rerun the test?

Rerun it whenever a vendor ships a version you might adopt or when a plan change swaps the model behind your account. Fable 5.2 replaced Fable 5.1 on November 9, so any result gathered before that date describes a retired model.

── more in #large-language-models 4 stories · sorted by recency
── more on @anthropic 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/claude-vs-chatgpt-re…] indexed:0 read:35min 2026-11-17 · —