{"slug": "show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts", "title": "Show HN: Writing-eval, local deterministic style checks for AI-written drafts", "summary": "Majestic Labs released writing-eval, a local, deterministic tool that scores AI-written drafts against a reusable style profile built from reference prose, with no hosted models or uploaded source material. The Python 3.11+ tool runs on CPU, produces deterministic results, and includes a corpus evaluation pipeline for comparing writing systems. It is designed to help teams measure editorial voice consistency and run regression checks after changes to prompts, models, or reference corpora.", "body_md": "Built by [David Paluy](https://github.com/dpaluy) from [Majestic Labs](https://majesticlabs.dev/?utm_source=github&utm_medium=readme&utm_campaign=writing-eval).\n\n`writing-eval`\n\ngives teams a repeatable way to measure a draft against a chosen\neditorial voice. Build a reusable style profile from reference prose, check a\ndraft against it, and get specific evidence about differences in clarity,\nreadability, sentence rhythm, vocabulary, and detected writing patterns.\n\nThe project also includes a corpus evaluation pipeline for comparing generated outputs with a reference corpus. Use it for local diagnostics, regression checks, and repeatable comparisons between writing systems.\n\nEverything runs locally on the CPU and produces deterministic results. The tool does not call hosted models, upload source material, train models, optimize detectors, or reproduce a proprietary evaluation method.\n\nPython 3.11 or newer and `uv`\n\nare required.\n\n```\nuv sync\n```\n\nThe release version is the `[project].version`\n\nvalue in `pyproject.toml`\n\n. Update\nthis value for every release. To show the installed release version, run:\n\n```\n./writing-eval --version\n```\n\nBuild a style profile from a directory of an author's posts, then check a draft against it:\n\n```\n./writing-eval profile build acme --from posts/acme\n./writing-eval check draft.md --style acme\n```\n\nThe first command ingests the `.md`\n\nand `.txt`\n\nfiles under `posts/acme`\n\nand\nwrites a reusable profile named `acme`\n\ninto `data/profiles/acme/`\n\n. The second\naudits `draft.md`\n\nand produces a scored, profile-relative assessment. The\nhuman-readable report shows four section scores, actionable issues with current\nand target values, editing instructions, success criteria, source locations,\nand general statistics. The profile name is kept out of the report body; it\nremains in JSON metadata for reproducibility.\n\nThe repository includes an agent skill at\n[ skills/writing-eval/SKILL.md](/majesticlabs-dev/writing-eval/blob/master/skills/writing-eval/SKILL.md). Agent harnesses\nthat support\n\n`SKILL.md`\n\ninstructions can load it to select the correct command,\nbuild or choose a profile, interpret exit codes and JSON, protect private source\nmaterial, and report results without overstating the heuristic score.The skill controls the local CLI and does not bundle the executable. Use it from\na repository checkout after running `uv sync`\n\n.\n\nExample requests:\n\n```\nUse writing-eval to check docs/draft.md against the acme profile. Summarize the\nhighest-priority issues with their source locations.\nBuild a writing-eval profile named product-docs from the authorized prose in\ndata/product-docs, then check docs/new-guide.md against it.\nCompare the writing systems in runs/release-candidate against\ndata/reference-corpus.jsonl and explain the report verdict.\n```\n\n| Capability | Use it for |\n|---|---|\n| Style profiles | Build a reusable baseline from approved prose |\n| Draft checks | Compare one Markdown or plain-text file with a profile |\n| Rule-based linting | Locate configurable writing tendencies |\n| Corpus evaluation | Compare several output systems consistently |\n| Markdown and JSON reports | Support human review, automation, and regression gates |\n\nAI-generated writing is easy to demo and difficult to evaluate consistently. A draft can be grammatically correct while still missing an organization's voice, preferred structure, or editorial constraints.\n\n`writing-eval`\n\nturns those expectations into a local, versioned measurement\nprocess. Teams can run the same checks after changing a prompt, model, reference\ncorpus, or rule set and see what improved or regressed.\n\nThis project is one narrow example of a broader Majestic Labs principle: an AI\nworkflow needs a company-controlled definition of acceptable work. Read\n[Building Private AI Evals](https://majesticlabs.dev/blog/202607/building-private-ai-evals?utm_source=github&utm_medium=readme&utm_campaign=writing-eval)\nfor the broader approach.\n\n[Use with LLM agents](#use-with-llm-agents)[Single-document checks](#single-document-check)[Style profiles](#style-profiles)[Metrics](#metrics)[Corpus evaluation and benchmark](#corpus-evaluation-and-benchmark)[Limitations](#limitations)[License and contributions](#license-and-contributions)[Managed hosting](#managed-hosting)[About Majestic Labs](#about-majestic-labs)\n\n`./writing-eval check`\n\naudits one draft the way a linter audits one source file,\nwithout any JSONL wrapping. It takes a Markdown or plain-text file, or `-`\n\nto\nread from standard input.\n\n```\n./writing-eval check draft.md\n./writing-eval check draft.md --references data/reference-corpus.jsonl\ncat draft.md | ./writing-eval check -\n```\n\nOptions:\n\n`--rules`\n\nselects the rule file (default: the builtin rule set that ships with the package). The repository also ships an optional overlay with extra AI-writing tells; see[The anti-ai overlay](#the-anti-ai-overlay).`--references`\n\nis an optional JSONL reference corpus. When omitted, the token 1-gram L2 metric is skipped, rendered as`n/a`\n\n, and a note is printed to standard error.`--style`\n\ncompares the draft against a named style profile and renders a scored assessment (see[Style profiles](#style-profiles)). It is mutually exclusive with`--references`\n\n; passing both is a usage error.`--profiles-root`\n\nlocates profiles for`--style`\n\n(default`data/profiles`\n\n).`--format text|json`\n\nselects human-readable text or JSON on standard output (default`text`\n\n).`--json PATH`\n\nwrites the same JSON result to a file while preserving the selected standard-output format.\n\nWithout `--style`\n\n, text output retains the linter format: one finding per line,\nsorted by position, using 1-indexed line and column offsets computed from the\nreal character positions of each match, followed by a metrics block:\n\n```\ndraft.md:1:1 [warn] metadiscourse_openers: Remove the metadiscourse opener and state the point directly. | span: In this article,\ndraft.md:1:21 [warn] polish_vocab: Replace overused polish vocabulary with specific language. | span: delve\nmetrics:\n  word_count: 15\n  tell_rates_by_severity:\n    warn: 400.000000\n  mean_sentence_length: 7.500000\n  sentence_length_variance: 2.250000\n  repeated_opening_rate: 0.000000\n  token_1gram_l2: n/a\nquality_metrics (informational):\n  flesch_reading_ease: 52.000000\n  flesch_kincaid_grade: 9.000000\n  mtld: 14.000000\n  paragraph_stats:\n    paragraph_count: 1.000000\n    mean_paragraph_sentence_count: 2.000000\n    single_sentence_paragraph_rate: 0.000000\n```\n\nThe example values are illustrative. Scores below 10 tokens render `mtld`\n\nas\n`n/a`\n\n, and text without a sentence renders the readability scores as `n/a`\n\n.\nWith `--style`\n\n, text output uses the scored assessment described in\n[Check a draft against a profile](#check-a-draft-against-a-profile).\n\nExit codes distinguish completed checks from input errors:\n\n`0`\n\n: the check completed, with or without findings.`1`\n\n: a usage or input error (missing file, unreadable rules, invalid JSONL).\n\nA *style profile* is a deterministic fingerprint of one author's writing, built\nfrom a corpus of their prose. Build a profile once, then check any draft against\nit to see how far the draft sits from that voice and which vocabulary and\nstructure differ. The author's own voice is just one profile among many.\n\n```\n./writing-eval profile build <name> --from <dir-or-files...> [--profiles-root data/profiles] [--rules PATH]\n```\n\n`<name>`\n\nmust be exactly one non-absolute path component. `.`\n\n, `..`\n\n, absolute\npaths, and nested names are rejected.\n\n`--from`\n\naccepts directories (their `.md`\n\nand `.txt`\n\nfiles are ingested\nrecursively) or individual files. `--rules`\n\nselects the rule file used to\nprecompute the profile cache (default: the builtin rule set). A leading YAML\nfrontmatter block is stripped from each source, and each document becomes one\nreference record with a stable ID derived from its filename. The command writes\ntwo files into\n`<profiles-root>/<name>/`\n\n:\n\n`references.jsonl`\n\n: one`{\"id\", \"text\", \"file\"}`\n\nrecord per source document, reused as the reference corpus by`check --style`\n\n.`profile.json`\n\n: the profile name, its creation date, a numeric`metrics_version`\n\nfield (currently 2), a`references_sha256`\n\ndigest of the paired`references.jsonl`\n\nfile, per-source word counts, the total word count, and the corpus statistics (mean sentence length and variance, repeated-opening rate, Flesch reading ease and grade, MTLD, paragraph statistics, and the top 20 content tokens after a small stop list).\n\n`metrics_version`\n\npins the metric semantics the stored statistics were computed\nwith. Version 2 covers curly-apostrophe (U+2019) sentence openers,\nmarkdown-aware readability word counts, and MTLD tail, threshold, and\nsequence-input lowercase behavior. Older profiles are rejected with a rebuild\ninstruction until `profile build`\n\nruns again with the current tool.\n`references_sha256`\n\nbinds `profile.json`\n\nto the installed `references.jsonl`\n\n.\nA missing, invalid, or mismatched digest is the same rebuild error, so a mixed\npair from an interrupted write is not loaded.\n\nFor example, put all articles for one author under a dedicated directory:\n\n```\nposts/acme/\n├── choosing-a-market.md\n├── distribution-first.md\n├── founder-notes.txt\n└── archive/\n    └── early-lessons.md\n```\n\nThen build the profile from the directory:\n\n```\n./writing-eval profile build acme --from posts/acme\n```\n\nThe directory is scanned recursively, so this imports all four `.md`\n\nand `.txt`\n\narticles, including `archive/early-lessons.md`\n\n. There is no need to write a\nwildcard or list every file. A successful build reports the number of imported\nsources and words, for example:\n\n``` php\nbuilt profile 'acme': 30 sources, 15742 words -> data/profiles/acme\n```\n\nTo import selected articles instead, list each file after `--from`\n\n:\n\n```\n./writing-eval profile build acme --from \\\n  posts/acme/choosing-a-market.md \\\n  posts/acme/distribution-first.md\n```\n\nThe command accepts a single non-empty article, but a small profile makes the\nscore depend on which articles you happened to include. For a profile used as a\nstyle baseline, use **at least 25 articles, and prefer 40 or more**.\n\nThat number is measured, not assumed. Holding a draft fixed and varying only\nwhich articles form the profile, the standard deviation of the resulting score\nfalls with more articles. Two points is the rubric's smallest unit, the\ndeduction for one excess `warn`\n\noccurrence, so below that threshold the\nsampling noise is smaller than anything the score can express. The mean\nstandard deviation crosses 2 points around N = 20, but individual drafts vary\nwidely, so the number that matters is coverage: the share of drafts that have\nactually settled below 2 points at a given profile size.\n\n| Articles in profile | Drafts at or under 2 points of noise |\n|---|---|\n| 10 | 33.3% |\n| 15 | 37.5% |\n| 20 | 54.2% |\n| 25 | 70.8% |\n| 30 | 66.7% |\n| 40 | 87.5% |\n| 50 | 95.8% |\n| 60 | 95.8% |\n\n25 articles covers 7 in 10 drafts. 30 articles is not reliably better than 25; the difference sits inside trial noise. 40 articles covers 9 in 10, and 50 covers 24 in 25, which is why 40 is the preferred target. As a rule that holds at every corpus size tested: treat a score difference under 3 points as noise, whether between two drafts or between two runs of the same draft.\n\n**Article count drives stability, not word count.** At a fixed word budget, a\nprofile built from more, shorter articles is consistently more stable than one\nbuilt from fewer, longer articles. At 40,000 words, 28 articles gave a standard\ndeviation of 1.3 while 11 articles gave 2.5. Prior versions of this document\nrecommended a 15,000-word minimum; that figure was not supported by measurement\nand has been removed. At a fixed article count, per-article length still\nmatters up to a point: a corpus averaging about 1,200 words per article was\nroughly 0.4 points noisier than one averaging about 2,350, with no further gain\npast about 2,350 words per article. The floor below 517 words per article, the\nshortest article in the study corpus, is unmeasured.\n\nMethod, full results, and the reproduction script are in\n[docs/profile-size-study.md](/majesticlabs-dev/writing-eval/blob/master/docs/profile-size-study.md). The study used one\n101-article corpus of long-form nonfiction by a single author. The direction of\nthe effect should hold generally; the exact crossover point may move for other\ngenres and article lengths. Run the script on your own corpus to check.\n\nPrefer articles from the same author and the kinds of writing the profile should represent; mixing unrelated authors, genres, or registers creates a blended profile.\n\nThere is no incremental append command. Keep the article directory as the authoritative corpus, add the new article to it, and rebuild using the same profile name:\n\n```\nposts/acme/new-article.md\n./writing-eval profile build acme --from posts/acme\n```\n\nRebuilding replaces `data/profiles/acme/references.jsonl`\n\nand\n`data/profiles/acme/profile.json`\n\nwith results computed from every article\ncurrently in `posts/acme`\n\n. Always pass the complete corpus when rebuilding.\nPassing only `posts/acme/new-article.md`\n\nwould replace the profile with a\none-article profile rather than add that article to the existing profile.\n\n```\n./writing-eval profile list [--profiles-root data/profiles]\n```\n\nOne line per profile that `load_profile`\n\naccepts: name, source count, and total\nwords. Directories that fail that load (wrong `metrics_version`\n\n, mismatched\n`references_sha256`\n\n, unreadable metadata) are omitted. The CLI prints a skip\nnote on stderr for each omitted directory.\n\n`profile build`\n\nprecomputes reference statistics into `<profile>/cache/`\n\n.\nAfter a rule change, refresh that cache with:\n\n```\n./writing-eval profile cache <name> [--rules PATH]\n```\n\n`--rules`\n\nmust match the rule file that later `check`\n\nruns will use. Caches\nalso invalidate automatically when detector or tokenizer code changes in a new\nrelease, since each entry records a digest of that code. Checks remain correct\nwith a stale or missing cache; they only get slower until the cache is rebuilt.\n\n```\n./writing-eval check draft.md --style <name> [--profiles-root data/profiles]\n```\n\n`--style`\n\nresolves the profile's `references.jsonl`\n\nas the reference corpus, so\nit is mutually exclusive with `--references`\n\n. The default text report is\nMarkdown-shaped so both a person and an LLM can use it directly:\n\n```\n# Writing Evaluation\n\nFile: `draft.md`\n\n## Article score (heuristic)\n\n**86/100 - Moderate alignment**\n\nThis score measures detected style patterns and alignment with the target\nprofile. It does not measure factual accuracy or overall content quality.\n\n| Section | Score |\n|---|---:|\n| Clarity and directness | 25/25 |\n| Readability | 23/25 |\n| Rhythm and structure | 13/25 |\n| Vocabulary and style | 25/25 |\n| **Total** | **86/100** |\n\n## Issues to improve\n\n### 1. Sentence rhythm differs from the target profile.\n\n- Section: Rhythm and structure\n- Priority: High\n- Deduction: -8 points\n\n| Measure | Article | Target profile | Direction |\n|---|---:|---:|---|\n| Average sentence length | 8.1 words | 16.0 words | increase |\n| Sentence-length variance | 29.3 | 104.8 | increase |\n\nEditing instruction:\n\nCombine selected explanatory sentences and adjust the mixture of short, medium,\nand long sentences toward the target profile. Preserve deliberate emphasis; do\nnot mechanically force every sentence to the target.\n\nSuccess criteria:\n\n- Move average sentence length closer to the target.\n- Move sentence-length variation closer to the target.\n- Preserve short or long sentences that serve a clear rhetorical purpose.\n\n## General statistics\n\n| Statistic | Article | Target profile | Interpretation |\n|---|---:|---:|---|\n| Word count | 485 | n/a | informational |\n| Average sentence length | 8.1 words | 16.0 words | shorter than target |\n| Repeated openings | 18.5% | 8.2% | higher than target |\n| Reading ease | 59.8 | 60.8 | closely aligned |\n```\n\nThe example is abbreviated. Each issue includes stable identifiers in JSON, numeric current and target values, an editing direction, an instruction, success criteria, and any known line and column locations. Consecutive repeated-opening findings are grouped into one run, so deliberate anaphora can be reviewed as a pattern instead of as several disconnected warnings.\n\nRaw rule findings remain in JSON. For a profile check, each rule is also run over the profile references. The profile occurrence rate is scaled to the draft length and rounded up to an allowance. Only occurrences above that allowance appear as rule issues. This lets an author's demonstrated style outrank a generic rule while keeping the evidence visible.\n\nExcess `warn`\n\nfindings appear under **Issues to improve** and can lower the\nscore. Excess `info`\n\nfindings appear separately under **Review\ncandidates** and never lower it. There is no strengths section. If the draft has\nno scorable prose sentence, the report is explicitly `Unscored`\n\ninstead of\nassigning a misleading number.\n\nPrint JSON to standard output:\n\n```\n./writing-eval check draft.md --style acme --format json\n```\n\nKeep the human report on standard output and write the same JSON payload to a file:\n\n```\n./writing-eval check draft.md --style acme --json evaluation.json\n```\n\nProfile checks preserve the raw `file`\n\n, `findings`\n\n, `metrics`\n\n,\n`quality_metrics`\n\n, and `style_gap`\n\nfields and add an `assessment`\n\nobject. This\nabbreviated example shows the shape:\n\n```\n{\n  \"assessment\": {\n    \"schema_version\": 2,\n    \"rubric_version\": \"profile-alignment-v2\",\n    \"basis\": \"rules_and_target_profile\",\n    \"status\": \"scored\",\n    \"profile\": {\n      \"id\": \"acme\"\n    },\n    \"score\": {\n      \"total\": 86,\n      \"maximum\": 100,\n      \"label\": \"Moderate alignment\",\n      \"sections\": [\n        {\n          \"id\": \"clarity_directness\",\n          \"label\": \"Clarity and directness\",\n          \"score\": 25,\n          \"maximum\": 25,\n          \"deduction\": 0\n        },\n        {\n          \"id\": \"readability\",\n          \"label\": \"Readability\",\n          \"score\": 23,\n          \"maximum\": 25,\n          \"deduction\": 2\n        },\n        {\n          \"id\": \"rhythm_structure\",\n          \"label\": \"Rhythm and structure\",\n          \"score\": 13,\n          \"maximum\": 25,\n          \"deduction\": 12\n        },\n        {\n          \"id\": \"vocabulary_style\",\n          \"label\": \"Vocabulary and style\",\n          \"score\": 25,\n          \"maximum\": 25,\n          \"deduction\": 0\n        }\n      ]\n    },\n    \"issues\": [\n      {\n        \"id\": \"sentence_rhythm\",\n        \"kind\": \"improvement\",\n        \"section\": \"rhythm_structure\",\n        \"priority\": \"high\",\n        \"deduction\": 8,\n        \"summary\": \"Sentence rhythm differs from the target profile.\",\n        \"comparisons\": [\n          {\n            \"metric\": \"mean_sentence_length\",\n            \"label\": \"Average sentence length\",\n            \"current\": 8.137931,\n            \"target\": 16.031049,\n            \"delta\": -7.893118,\n            \"direction\": \"increase\",\n            \"unit\": \"words_per_sentence\"\n          }\n        ],\n        \"instruction\": \"Combine selected explanatory sentences while preserving deliberate emphasis.\",\n        \"success_criteria\": [\n          \"Move average sentence length closer to the target.\"\n        ],\n        \"locations\": []\n      }\n    ],\n    \"statistics\": [\n      {\n        \"id\": \"word_count\",\n        \"label\": \"Word count\",\n        \"value\": 485,\n        \"unit\": \"words\",\n        \"target\": null,\n        \"interpretation\": \"informational\"\n      }\n    ]\n  }\n}\n```\n\nJSON numbers retain full precision. The profile ID stays in JSON for\nreproducibility but is rendered only as `Target profile`\n\nin the human report.\nThe unscored form uses `status: \"unscored\"`\n\n, a `reason`\n\n, null total, label, and\nsection scores, an empty issue list, and the available statistics. The\nassessment's `rule_baseline`\n\nobject records the profile and draft word counts and\none sorted entry per observed rule: profile count, profile rate per 1,000 words,\nscaled draft allowance, raw draft count, and excess count.\n\nThe `profile-alignment-v2`\n\nrubric starts four sections at 25 points. Version 2\nadds profile-relative rule allowances; direct library callers that omit rule\nbaseline data retain the version 1 contract.\n\n**Clarity and directness**: directness-related rule findings.** Readability**: Flesch reading ease and Flesch-Kincaid grade relative to the profile.** Rhythm and structure**: sentence length, sentence-length variance, and repeated openings relative to the profile.** Vocabulary and style**: all other style rules.\n\nRule deductions are 2 points per excess `warn`\n\noccurrence and 0 per `info`\n\noccurrence. An excess occurrence is one above the\nallowance derived from the selected profile's aggregate rate. A rule absent from\nthe profile has an allowance of zero. A rule is assigned to exactly one section,\nand repeated openings are scored only in rhythm and structure to prevent\ndouble-counting. Section deductions are capped at 25.\n\nProfile-relative metric deductions use a tolerance before any points are removed, then increase linearly to a cap:\n\n| Metric | Tolerance | Gap at maximum deduction | Maximum deduction |\n|---|---|---|---|\n| Reading ease | 5 points | 30 points | 6 |\n| Reading grade | 1 grade | 4 grades | 6 |\n| Average sentence length | 15% | 100% | 8 |\n| Sentence-length variance | 25% | 100% | 7 |\n| Repeated-opening rate | 3 percentage points | 20 percentage points | 10 |\n\nFor sentence length and variance, the relative gap is\n`abs(article - target) / max(abs(target), 1)`\n\n. Other rows use the absolute gap.\nAt or below the tolerance, the deduction is zero. Above it, the unrounded\ndeduction is:\n\n```\nmaximum × (min(gap, cap) - tolerance) / (cap - tolerance)\n```\n\nEach issue deduction is rounded half up to an integer before the section cap is applied. The displayed arithmetic is invariant:\n\n```\nsum(issue deductions)\n  = sum(section deductions)\n  = 100 - total score\n```\n\nScore labels describe alignment, not universal quality:\n\n| Total | Label |\n|---|---|\n| 90-100 | High alignment |\n| 75-89 | Moderate alignment |\n| 60-74 | Low alignment |\n| 0-59 | Very low alignment |\n\nMTLD, token 1-gram L2, overrepresented terms, and paragraph statistics remain informational. The MTLD comparison is especially sensitive to comparing one draft with an aggregated profile corpus, so it does not affect the score.\n\n`writing-eval`\n\nruns locally and does not send drafts, references, or profiles to\na hosted service. Profiles are stored under the git-ignored `data/`\n\ndirectory\nby default.\n\nOnly use prose that you are permitted to process. Profiles built from third-party published content should remain private and must not be committed or quoted at length in tracked files.\n\n**Tell rate by severity**: style findings in a severity group per 1,000 output words. Lower values indicate fewer detected tendencies.** Token 1-gram L2**: Euclidean distance between normalized output and reference token frequency vectors. Lower values indicate closer vocabulary distributions. Reported as`n/a`\n\nwhen either side has zero tokens.**Overrepresented terms**: tokens whose output frequency most exceeds their reference frequency. Counts or rates explain the ranking.** Shared tokenization**: lowercase word tokens keep ASCII and curly-apostrophe contractions together.** Mean sentence length**: average number of words per sentence.** Sentence length variance**: population variance of sentence word counts. It describes how much sentence lengths vary within the corpus.** Repeated opening rate**: share of adjacent sentence pairs that begin with the same normalized opening. The denominator is the number of adjacent sentence pairs, or zero when fewer than two sentences exist. Openers keep curly apostrophes (U+2019), so a contraction such as We'll is one opener unit, matching tokenization.\n\nThe metrics below broaden the set toward general readability and structure. They\nappear in reports and `check`\n\noutput and never enter the corpus benchmark's\ndecision gate or any pre-registered threshold. In a profile check, reading ease\nand reading grade contribute only to the versioned heuristic alignment score\ndescribed above; MTLD and paragraph statistics remain informational.\n\n**Flesch reading ease** and**Flesch-Kincaid grade**: standard readability scores from word, sentence, and syllable counts. Word and sentence counts use the same markdown-aware segmentation as mean sentence length: headings are excluded and list markers are stripped. Syllable counts use a vowel-group heuristic with silent terminal`-e`\n\n,`-es`\n\n, and`-ed`\n\nadjustments, not a dictionary, so individual word estimates can be wrong. Reported as`n/a`\n\nwhen the text has no sentence.**MTLD**: Measure of Textual Lexical Diversity, the mean length of word runs that keep a type-token ratio above 0.72, averaged over forward and backward passes. Higher means more varied vocabulary. Reported as`n/a`\n\nbelow 10 tokens, where the measure is unreliable. An unfinished tail that stays above the threshold counts as one factor, so a fully unique 10-token input returns 10.0. The threshold must be a finite value in`(0, 1)`\n\n. Token-sequence inputs are lowercased to match text inputs.**Paragraph statistics**: markdown-aware paragraph count, mean sentences per paragraph, and single-sentence paragraph rate. Paragraphs are blank-line separated and headings are excluded, so results depend on the input's markdown formatting.\n\nReports retain the aggregate `tell_rate`\n\nmetric and also emit normalized\n`tell_rates_by_severity`\n\nvalues. Repeated-opening corpus rates count only\nadjacent pairs within each document, so reordering JSONL records does not alter\nthe result.\n\nThese metrics describe observable text patterns. They do not establish factual accuracy, originality, reader preference, or overall writing quality on their own.\n\nThis is the original evaluation pipeline, kept for regression checks and the pre-registered benchmark.\n\nBenchmark corpus generation and revision use the external OpenAI Codex CLI\nthrough `benchmark/generate_runs.py`\n\n. The Codex executable and its\nauthentication are optional benchmark dependencies. They are not required for\nthe `writing-eval`\n\nCLI, evaluation of existing output files, or\n`scripts/dry_run.sh`\n\n.\n\nRun a corpus evaluation with your own output and reference files:\n\n```\nuv run python scripts/run_eval.py --outputs path/to/outputs --references path/to/references.jsonl --report /tmp/writing-eval-report.md\n```\n\nThe command reads each output JSONL file, evaluates its text, compares it with the reference corpus, and writes a Markdown report to the requested path.\n\nThe optional `--json`\n\npath writes the same report data as JSON. JSON\nprovenance includes the tool name `writing-eval`\n\n, the installed package\nversion, and identity for the reference corpus and style rule set: source\npaths, record or rule counts, content SHA-256 hashes, and the rule-set\nversion. The Markdown report renders the reference-corpus and rule-set\nprovenance; it does not render tool identity. Markdown reports keep the\ncaller-supplied system order. JSONL writers keep object key order from dict\ninsertion.\n\nFor a checked repository run against test fixtures, use:\n\n```\nscripts/dry_run.sh\n```\n\nThe corpus evaluation is also available as the `eval`\n\nsubcommand\n(`./writing-eval eval --outputs ... --references ... --report ...`\n\n). The flat\ninvocation shown above is kept for backward compatibility and routes to the same\nbehavior.\n\nRevision runs also record a `literal_preservation`\n\nresult per output in their\nmetadata. The decision-gate report summarizes the same comparison as an\ninformational diagnostic. It checks normalized double-quoted spans, URLs, dates,\nand numeric literals for additions or removals. Quote whitespace is collapsed,\nURL terminal punctuation is ignored, date case and commas are normalized, and\nnumeric grouping commas are ignored. It does not change any registered criterion\nor the gate verdict.\n\nEach JSONL file contains one JSON object per line. IDs must be nonempty, stable, and unique within a file. A reference file may use a different ID namespace than the outputs. All output files in one run must contain the same ID set.\n\n```\n{\"id\":\"prompt-001\",\"use_case\":\"article_section\",\"prompt\":\"Explain why clear release notes reduce support work.\"}\n```\n\nRequired fields:\n\n`id`\n\n: stable prompt identifier`use_case`\n\n:`article_section`\n\n,`product_writing`\n\n, or`exec_communication`\n\n`prompt`\n\n: writing assignment text\n\n```\n{\"id\":\"ref-001\",\"text\":\"Clear release notes help customers understand what changed and what to do next. That context prevents avoidable support requests.\"}\n```\n\nRequired fields:\n\n`id`\n\n: stable reference identifier aligned to a prompt by numeric suffix`text`\n\n: accepted reference prose\n\n```\n{\"id\":\"prompt-001\",\"text\":\"Good release notes explain the change and its effect. Customers can act without opening a support ticket.\"}\n```\n\nRequired fields:\n\n`id`\n\n: prompt identifier`text`\n\n: generated prose to evaluate\n\nEach file in the directory passed to `--outputs`\n\nrepresents one system. Output files must contain the same prompt ID set so system comparisons stay consistent.\n\nStyle rules live in YAML. Each rule has these fields:\n\n| Field | Meaning |\n|---|---|\n`id` |\nStable, unique identifier used in findings and reports. |\n`severity` |\nFinding category used to group and normalize tell rates. |\n`detector` |\nRegex configuration or named detector that identifies the tendency. |\n`message` |\nClear diagnostic text shown when the rule matches. |\n`exceptions` |\nExplicit cases the detector should ignore. An exception suppresses a match only when it overlaps the candidate's own context after Unicode casefold, using whole words or phrases. For regex rules that context is the match span. For named detectors it can be a larger candidate-owned span. Empty or whitespace-only exception strings are rejected at load. Use an empty list when none apply. |\n`enabled` |\nMerge-time directive, valid only in an overlay. `enabled: false` removes the named rule from the effective set. It is not a field of a loaded rule. |\n\n`id`\n\n, `severity`\n\n, `detector`\n\n, and `message`\n\nare required when you define a brand-new\nrule id. When you override an existing rule id in an overlay, supply only the fields\nyou want to change.\n\nRules are validated when loaded, after any overlay merge. Missing fields, unknown keys in standalone rule files, empty exception strings, unsupported detectors, and malformed definitions fail with a clear error. `enabled`\n\nis overlay-only.\n\nOne rule set ships with the project, at a single in-package path. It holds 34\nrules: 16 `warn`\n\nand 18 `info`\n\n. The builtin rules are recommendations, and no\nrule blocks the build. Both `check`\n\nand corpus `eval`\n\nuse it\nby default. The `version:`\n\nkey in the YAML header is a schema version, not a\ncontent version; the reported SHA-256 and rule fingerprint carry content\nidentity. The regular passive-voice pattern is deliberately precision-first:\nit matches a `be`\n\nplus regular `-ed`\n\nparticiple only when the participle ends\nthe sentence or is followed by a likely agent, preposition, or frequency\ndeterminer, so unlisted adjectival participles can pass through undetected.\n\nPoint `--rules`\n\nat your own YAML file. A file without an `extends`\n\nkey replaces\nthe rule set outright. A file with an `extends`\n\nkey overlays the base set:\n\n```\nextends: builtin\nrules:\n  - id: my_jargon_ban      # new id: appended to the end of the rule set\n    severity: warn\n    detector: '(?i)\\b(?:synergy|ideate)\\b'\n    message: Use plain language.\n  - id: em_dash_ban        # existing id: override only the listed fields\n    severity: warn\n  - id: passive_voice      # existing id: removed from the effective set\n    enabled: false\n```\n\nMerge semantics:\n\n`extends: builtin`\n\nresolves through the installed package, so it works from any working directory.- Any other\n`extends`\n\nvalue is a path resolved against the overlay file's own directory, never the current working directory. An overlay can extend another overlay, up to a chain depth of 16. - Ordering is the base order with overrides and removals applied in place, then new rules appended in the order they appear in the overlay.\n- An override replaces only the fields present in the overlay entry. Unlisted fields keep the base values.\n`enabled: false`\n\non an id that is not in the base set is an error, as is a duplicate id inside one overlay file.\n\nUse the bundled overlay when a draft needs a stricter pass for reader-facing\nAI-writing tells than the builtin set provides, for example when reviewing\nmachine-assisted copy before publication. Pass it with `--rules`\n\n; it extends the\nbuiltin set rather than replacing it:\n\n```\n./writing-eval check draft.md --rules rules/anti-ai.yaml\n```\n\nThe command, output format, exit codes, and JSON schema are identical to the default check; only the finding set grows:\n\n```\ndraft.md:1:5 [warn] polish_vocab: Replace overused polish vocabulary with specific language. | span: synergy\ndraft.md:2:1 [info] connector_openers: Replace the formal additive opener with also, and, or a plain sentence start. | span: Furthermore\n```\n\nThe overlay lives at `rules/anti-ai.yaml`\n\nin the repository (outside the\ninstalled package) and begins with `extends: builtin`\n\n, so the effective rule set\nis the builtin rules plus four appended rules:\n\n`narrative_cliches`\n\n: stock narrative phrases such as \"couldn't help but\", \"little did I know\", \"stumbled upon\".`significance_markers`\n\n: meta commentary that labels a moment instead of showing it, such as \"that's the part that got me\", \"let that sink in\".`generation_artifacts`\n\n: bracketed scaffolding tokens (`insert`\n\n,`todo`\n\n,`tbd`\n\n,`placeholder`\n\n,`xxx`\n\n),`utm_source=chatgpt.com`\n\nlinks, and model self-reference such as \"as an AI language model\". Ordinary bracketed phrases such as`[your account]`\n\nare not flagged.`connector_openers`\n\n: sentence-initial \"furthermore\", \"moreover\", \"additionally\".\n\nIt also widens six builtin rules (`polish_vocab`\n\n, `recap_endings`\n\n,\n`throat_clearing`\n\n, `faux_insight`\n\n, `importance_puffery`\n\n, and\n`collaborative_artifacts`\n\n) with further phrases, keeping their severities and\nmessages.\n\nThe overlay combines with every `check`\n\nform, including `--style`\n\n, `--format json`\n\n, and reading from standard input. The builtin rule set stays the default\neverywhere. Because rule fingerprints identify comparable runs, keep the default\nrules for corpus evaluation and benchmark comparisons and use the overlay for\nsingle drafts.\n\n- The bundled fixtures are synthetic and intentionally exaggerated to exercise detectors.\n- Results are diagnostic only and should support, not replace, editorial review.\n- The article score is a heuristic combination of configured style rules and target-profile distances. It measures alignment under the documented rubric, not factual accuracy, argument strength, originality, or overall article quality.\n- Corpus metrics are sensitive to sample size, genre, and reference selection.\n- Simple token and sentence boundaries can differ from linguistic parsers.\n- A rule match identifies a review candidate, not a universal writing error.\n- The harness does not evaluate factual accuracy, coverage, or human preference.\n- Literal preservation is a normalized multiset comparison, not semantic factuality. It can detect a changed number, date, URL, or quoted span but cannot decide whether an unprotected claim is true or whether a paraphrase preserves meaning.\n- Distributional scores (token 1-gram L2 and the top overrepresented terms) depend heavily on the composition and size of the chosen reference corpus or profile; small profiles make them noisy.\n- L2 distance is sensitive to output length, so systems whose word counts differ a lot are not directly comparable on it.\n- Benchmark generation uses a frozen decoding config and measured per-metric noise floors (\n`benchmark/THRESHOLDS.md`\n\n); deltas below the documented floor are inconclusive. - Existing profiles with metric semantics other than version 2 must be rebuilt before use.\n- Readability scores ignore heading text and list markers, so heading-heavy Markdown can differ from a plain-text reading.\n`profile list`\n\nomits directories that`load_profile`\n\nrejects.\n\n`writing-eval`\n\nis source available under the\n[Elastic License 2.0](/majesticlabs-dev/writing-eval/blob/master/LICENSE). You may use, copy, modify, redistribute, and\nself-host the software subject to that license. You may not provide it to third\nparties as a hosted or managed service where the service gives users access to\nany substantial set of the software's features or functionality.\n\nCopyright 2026 Majestic Labs LLC.\n\nContributions are welcome. Read [CONTRIBUTING.md](/majesticlabs-dev/writing-eval/blob/master/CONTRIBUTING.md) before\nsubmitting a pull request. Contributors retain ownership of their work and must\naccept the [Contributor License Agreement](/majesticlabs-dev/writing-eval/blob/master/CLA.md), which gives Majestic Labs\nthe rights needed to keep contributions in the public project and use them in\nan official hosted service.\n\nFor OEM, hosted-service, or managed-service licensing, contact\n[Majestic Labs](https://majesticlabs.dev/ai?utm_source=github&utm_medium=readme&utm_campaign=writing-eval#inquiry).\n\nMajestic Labs is considering an official hosted `writing-eval`\n\nservice for\npeople and teams that cannot or prefer not to deploy and maintain the tool.\nThe local CLI remains complete and self-hostable.\n\nIf managed hosting would help your team,\n[tell us about your workflow](https://majesticlabs.dev/ai?utm_source=github&utm_medium=readme&utm_campaign=writing-eval#inquiry).\n\n[Majestic Labs](https://majesticlabs.dev/?utm_source=github&utm_medium=readme&utm_campaign=writing-eval)\nis a software foundry for operator-built software.\n\n`writing-eval`\n\nis part of our\n[open-source](https://majesticlabs.dev/open-source?utm_source=github&utm_medium=readme&utm_campaign=writing-eval) projects.\n\nIf your team needs to evaluate an AI workflow against company-specific\nstandards, [start with Majestic AI](https://majesticlabs.dev/ai?utm_source=github&utm_medium=readme&utm_campaign=writing-eval#inquiry).\n\nThis project is not a DFT clone or training system. It is a small deterministic harness for writing diagnostics.", "url": "https://wpnews.pro/news/show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts", "canonical_source": "https://github.com/majesticlabs-dev/writing-eval", "published_at": "2026-08-23 14:31:16+00:00", "updated_at": "2026-08-23 14:44:15.492054+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Majestic Labs", "David Paluy", "writing-eval"], "alternates": {"html": "https://wpnews.pro/news/show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts", "markdown": "https://wpnews.pro/news/show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts.md", "text": "https://wpnews.pro/news/show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts.txt", "jsonld": "https://wpnews.pro/news/show-hn-writing-eval-local-deterministic-style-checks-for-ai-written-drafts.jsonld"}}