Show HN: Writing-eval, local deterministic style checks for AI-written drafts

Majestic Labs released writing-eval, a local, deterministic tool that scores AI-written drafts against a reusable style profile built from reference prose, with no hosted models or uploaded source material. The Python 3.11+ tool runs on CPU, produces deterministic results, and includes a corpus evaluation pipeline for comparing writing systems. It is designed to help teams measure editorial voice consistency and run regression checks after changes to prompts, models, or reference corpora.

Built by David Paluy https://github.com/dpaluy from Majestic Labs https://majesticlabs.dev/?utm source=github&utm medium=readme&utm campaign=writing-eval . writing-eval gives teams a repeatable way to measure a draft against a chosen editorial voice. Build a reusable style profile from reference prose, check a draft against it, and get specific evidence about differences in clarity, readability, sentence rhythm, vocabulary, and detected writing patterns. The project also includes a corpus evaluation pipeline for comparing generated outputs with a reference corpus. Use it for local diagnostics, regression checks, and repeatable comparisons between writing systems. Everything runs locally on the CPU and produces deterministic results. The tool does not call hosted models, upload source material, train models, optimize detectors, or reproduce a proprietary evaluation method. Python 3.11 or newer and uv are required. uv sync The release version is the project .version value in pyproject.toml . Update this value for every release. To show the installed release version, run: ./writing-eval --version Build a style profile from a directory of an author's posts, then check a draft against it: ./writing-eval profile build acme --from posts/acme ./writing-eval check draft.md --style acme The first command ingests the .md and .txt files under posts/acme and writes a reusable profile named acme into data/profiles/acme/ . The second audits draft.md and produces a scored, profile-relative assessment. The human-readable report shows four section scores, actionable issues with current and target values, editing instructions, success criteria, source locations, and general statistics. The profile name is kept out of the report body; it remains in JSON metadata for reproducibility. The repository includes an agent skill at skills/writing-eval/SKILL.md /majesticlabs-dev/writing-eval/blob/master/skills/writing-eval/SKILL.md . Agent harnesses that support SKILL.md instructions can load it to select the correct command, build or choose a profile, interpret exit codes and JSON, protect private source material, and report results without overstating the heuristic score.The skill controls the local CLI and does not bundle the executable. Use it from a repository checkout after running uv sync . Example requests: Use writing-eval to check docs/draft.md against the acme profile. Summarize the highest-priority issues with their source locations. Build a writing-eval profile named product-docs from the authorized prose in data/product-docs, then check docs/new-guide.md against it. Compare the writing systems in runs/release-candidate against data/reference-corpus.jsonl and explain the report verdict. | Capability | Use it for | |---|---| | Style profiles | Build a reusable baseline from approved prose | | Draft checks | Compare one Markdown or plain-text file with a profile | | Rule-based linting | Locate configurable writing tendencies | | Corpus evaluation | Compare several output systems consistently | | Markdown and JSON reports | Support human review, automation, and regression gates | AI-generated writing is easy to demo and difficult to evaluate consistently. A draft can be grammatically correct while still missing an organization's voice, preferred structure, or editorial constraints. writing-eval turns those expectations into a local, versioned measurement process. Teams can run the same checks after changing a prompt, model, reference corpus, or rule set and see what improved or regressed. This project is one narrow example of a broader Majestic Labs principle: an AI workflow needs a company-controlled definition of acceptable work. Read Building Private AI Evals https://majesticlabs.dev/blog/202607/building-private-ai-evals?utm source=github&utm medium=readme&utm campaign=writing-eval for the broader approach. Use with LLM agents use-with-llm-agents Single-document checks single-document-check Style profiles style-profiles Metrics metrics Corpus evaluation and benchmark corpus-evaluation-and-benchmark Limitations limitations License and contributions license-and-contributions Managed hosting managed-hosting About Majestic Labs about-majestic-labs ./writing-eval check audits one draft the way a linter audits one source file, without any JSONL wrapping. It takes a Markdown or plain-text file, or - to read from standard input. ./writing-eval check draft.md ./writing-eval check draft.md --references data/reference-corpus.jsonl cat draft.md | ./writing-eval check - Options: --rules selects the rule file default: the builtin rule set that ships with the package . The repository also ships an optional overlay with extra AI-writing tells; see The anti-ai overlay the-anti-ai-overlay . --references is an optional JSONL reference corpus. When omitted, the token 1-gram L2 metric is skipped, rendered as n/a , and a note is printed to standard error. --style compares the draft against a named style profile and renders a scored assessment see Style profiles style-profiles . It is mutually exclusive with --references ; passing both is a usage error. --profiles-root locates profiles for --style default data/profiles . --format text|json selects human-readable text or JSON on standard output default text . --json PATH writes the same JSON result to a file while preserving the selected standard-output format. Without --style , text output retains the linter format: one finding per line, sorted by position, using 1-indexed line and column offsets computed from the real character positions of each match, followed by a metrics block: draft.md:1:1 warn metadiscourse openers: Remove the metadiscourse opener and state the point directly. | span: In this article, draft.md:1:21 warn polish vocab: Replace overused polish vocabulary with specific language. | span: delve metrics: word count: 15 tell rates by severity: warn: 400.000000 mean sentence length: 7.500000 sentence length variance: 2.250000 repeated opening rate: 0.000000 token 1gram l2: n/a quality metrics informational : flesch reading ease: 52.000000 flesch kincaid grade: 9.000000 mtld: 14.000000 paragraph stats: paragraph count: 1.000000 mean paragraph sentence count: 2.000000 single sentence paragraph rate: 0.000000 The example values are illustrative. Scores below 10 tokens render mtld as n/a , and text without a sentence renders the readability scores as n/a . With --style , text output uses the scored assessment described in Check a draft against a profile check-a-draft-against-a-profile . Exit codes distinguish completed checks from input errors: 0 : the check completed, with or without findings. 1 : a usage or input error missing file, unreadable rules, invalid JSONL . A style profile is a deterministic fingerprint of one author's writing, built from a corpus of their prose. Build a profile once, then check any draft against it to see how far the draft sits from that voice and which vocabulary and structure differ. The author's own voice is just one profile among many. ./writing-eval profile build <name --from <dir-or-files... --profiles-root data/profiles --rules PATH <name must be exactly one non-absolute path component. . , .. , absolute paths, and nested names are rejected. --from accepts directories their .md and .txt files are ingested recursively or individual files. --rules selects the rule file used to precompute the profile cache default: the builtin rule set . A leading YAML frontmatter block is stripped from each source, and each document becomes one reference record with a stable ID derived from its filename. The command writes two files into <profiles-root /<name / : references.jsonl : one {"id", "text", "file"} record per source document, reused as the reference corpus by check --style . profile.json : the profile name, its creation date, a numeric metrics version field currently 2 , a references sha256 digest of the paired references.jsonl file, per-source word counts, the total word count, and the corpus statistics mean sentence length and variance, repeated-opening rate, Flesch reading ease and grade, MTLD, paragraph statistics, and the top 20 content tokens after a small stop list . metrics version pins the metric semantics the stored statistics were computed with. Version 2 covers curly-apostrophe U+2019 sentence openers, markdown-aware readability word counts, and MTLD tail, threshold, and sequence-input lowercase behavior. Older profiles are rejected with a rebuild instruction until profile build runs again with the current tool. references sha256 binds profile.json to the installed references.jsonl . A missing, invalid, or mismatched digest is the same rebuild error, so a mixed pair from an interrupted write is not loaded. For example, put all articles for one author under a dedicated directory: posts/acme/ ├── choosing-a-market.md ├── distribution-first.md ├── founder-notes.txt └── archive/ └── early-lessons.md Then build the profile from the directory: ./writing-eval profile build acme --from posts/acme The directory is scanned recursively, so this imports all four .md and .txt articles, including archive/early-lessons.md . There is no need to write a wildcard or list every file. A successful build reports the number of imported sources and words, for example: php built profile 'acme': 30 sources, 15742 words - data/profiles/acme To import selected articles instead, list each file after --from : ./writing-eval profile build acme --from \ posts/acme/choosing-a-market.md \ posts/acme/distribution-first.md The command accepts a single non-empty article, but a small profile makes the score depend on which articles you happened to include. For a profile used as a style baseline, use at least 25 articles, and prefer 40 or more . That number is measured, not assumed. Holding a draft fixed and varying only which articles form the profile, the standard deviation of the resulting score falls with more articles. Two points is the rubric's smallest unit, the deduction for one excess warn occurrence, so below that threshold the sampling noise is smaller than anything the score can express. The mean standard deviation crosses 2 points around N = 20, but individual drafts vary widely, so the number that matters is coverage: the share of drafts that have actually settled below 2 points at a given profile size. | Articles in profile | Drafts at or under 2 points of noise | |---|---| | 10 | 33.3% | | 15 | 37.5% | | 20 | 54.2% | | 25 | 70.8% | | 30 | 66.7% | | 40 | 87.5% | | 50 | 95.8% | | 60 | 95.8% | 25 articles covers 7 in 10 drafts. 30 articles is not reliably better than 25; the difference sits inside trial noise. 40 articles covers 9 in 10, and 50 covers 24 in 25, which is why 40 is the preferred target. As a rule that holds at every corpus size tested: treat a score difference under 3 points as noise, whether between two drafts or between two runs of the same draft. Article count drives stability, not word count. At a fixed word budget, a profile built from more, shorter articles is consistently more stable than one built from fewer, longer articles. At 40,000 words, 28 articles gave a standard deviation of 1.3 while 11 articles gave 2.5. Prior versions of this document recommended a 15,000-word minimum; that figure was not supported by measurement and has been removed. At a fixed article count, per-article length still matters up to a point: a corpus averaging about 1,200 words per article was roughly 0.4 points noisier than one averaging about 2,350, with no further gain past about 2,350 words per article. The floor below 517 words per article, the shortest article in the study corpus, is unmeasured. Method, full results, and the reproduction script are in docs/profile-size-study.md /majesticlabs-dev/writing-eval/blob/master/docs/profile-size-study.md . The study used one 101-article corpus of long-form nonfiction by a single author. The direction of the effect should hold generally; the exact crossover point may move for other genres and article lengths. Run the script on your own corpus to check. Prefer articles from the same author and the kinds of writing the profile should represent; mixing unrelated authors, genres, or registers creates a blended profile. There is no incremental append command. Keep the article directory as the authoritative corpus, add the new article to it, and rebuild using the same profile name: posts/acme/new-article.md ./writing-eval profile build acme --from posts/acme Rebuilding replaces data/profiles/acme/references.jsonl and data/profiles/acme/profile.json with results computed from every article currently in posts/acme . Always pass the complete corpus when rebuilding. Passing only posts/acme/new-article.md would replace the profile with a one-article profile rather than add that article to the existing profile. ./writing-eval profile list --profiles-root data/profiles One line per profile that load profile accepts: name, source count, and total words. Directories that fail that load wrong metrics version , mismatched references sha256 , unreadable metadata are omitted. The CLI prints a skip note on stderr for each omitted directory. profile build precomputes reference statistics into <profile /cache/ . After a rule change, refresh that cache with: ./writing-eval profile cache <name --rules PATH --rules must match the rule file that later check runs will use. Caches also invalidate automatically when detector or tokenizer code changes in a new release, since each entry records a digest of that code. Checks remain correct with a stale or missing cache; they only get slower until the cache is rebuilt. ./writing-eval check draft.md --style <name --profiles-root data/profiles --style resolves the profile's references.jsonl as the reference corpus, so it is mutually exclusive with --references . The default text report is Markdown-shaped so both a person and an LLM can use it directly: Writing Evaluation File: draft.md Article score heuristic 86/100 - Moderate alignment This score measures detected style patterns and alignment with the target profile. It does not measure factual accuracy or overall content quality. | Section | Score | |---|---:| | Clarity and directness | 25/25 | | Readability | 23/25 | | Rhythm and structure | 13/25 | | Vocabulary and style | 25/25 | | Total | 86/100 | Issues to improve 1. Sentence rhythm differs from the target profile. - Section: Rhythm and structure - Priority: High - Deduction: -8 points | Measure | Article | Target profile | Direction | |---|---:|---:|---| | Average sentence length | 8.1 words | 16.0 words | increase | | Sentence-length variance | 29.3 | 104.8 | increase | Editing instruction: Combine selected explanatory sentences and adjust the mixture of short, medium, and long sentences toward the target profile. Preserve deliberate emphasis; do not mechanically force every sentence to the target. Success criteria: - Move average sentence length closer to the target. - Move sentence-length variation closer to the target. - Preserve short or long sentences that serve a clear rhetorical purpose. General statistics | Statistic | Article | Target profile | Interpretation | |---|---:|---:|---| | Word count | 485 | n/a | informational | | Average sentence length | 8.1 words | 16.0 words | shorter than target | | Repeated openings | 18.5% | 8.2% | higher than target | | Reading ease | 59.8 | 60.8 | closely aligned | The example is abbreviated. Each issue includes stable identifiers in JSON, numeric current and target values, an editing direction, an instruction, success criteria, and any known line and column locations. Consecutive repeated-opening findings are grouped into one run, so deliberate anaphora can be reviewed as a pattern instead of as several disconnected warnings. Raw rule findings remain in JSON. For a profile check, each rule is also run over the profile references. The profile occurrence rate is scaled to the draft length and rounded up to an allowance. Only occurrences above that allowance appear as rule issues. This lets an author's demonstrated style outrank a generic rule while keeping the evidence visible. Excess warn findings appear under Issues to improve and can lower the score. Excess info findings appear separately under Review candidates and never lower it. There is no strengths section. If the draft has no scorable prose sentence, the report is explicitly Unscored instead of assigning a misleading number. Print JSON to standard output: ./writing-eval check draft.md --style acme --format json Keep the human report on standard output and write the same JSON payload to a file: ./writing-eval check draft.md --style acme --json evaluation.json Profile checks preserve the raw file , findings , metrics , quality metrics , and style gap fields and add an assessment object. This abbreviated example shows the shape: { "assessment": { "schema version": 2, "rubric version": "profile-alignment-v2", "basis": "rules and target profile", "status": "scored", "profile": { "id": "acme" }, "score": { "total": 86, "maximum": 100, "label": "Moderate alignment", "sections": { "id": "clarity directness", "label": "Clarity and directness", "score": 25, "maximum": 25, "deduction": 0 }, { "id": "readability", "label": "Readability", "score": 23, "maximum": 25, "deduction": 2 }, { "id": "rhythm structure", "label": "Rhythm and structure", "score": 13, "maximum": 25, "deduction": 12 }, { "id": "vocabulary style", "label": "Vocabulary and style", "score": 25, "maximum": 25, "deduction": 0 } }, "issues": { "id": "sentence rhythm", "kind": "improvement", "section": "rhythm structure", "priority": "high", "deduction": 8, "summary": "Sentence rhythm differs from the target profile.", "comparisons": { "metric": "mean sentence length", "label": "Average sentence length", "current": 8.137931, "target": 16.031049, "delta": -7.893118, "direction": "increase", "unit": "words per sentence" } , "instruction": "Combine selected explanatory sentences while preserving deliberate emphasis.", "success criteria": "Move average sentence length closer to the target." , "locations": } , "statistics": { "id": "word count", "label": "Word count", "value": 485, "unit": "words", "target": null, "interpretation": "informational" } } } JSON numbers retain full precision. The profile ID stays in JSON for reproducibility but is rendered only as Target profile in the human report. The unscored form uses status: "unscored" , a reason , null total, label, and section scores, an empty issue list, and the available statistics. The assessment's rule baseline object records the profile and draft word counts and one sorted entry per observed rule: profile count, profile rate per 1,000 words, scaled draft allowance, raw draft count, and excess count. The profile-alignment-v2 rubric starts four sections at 25 points. Version 2 adds profile-relative rule allowances; direct library callers that omit rule baseline data retain the version 1 contract. Clarity and directness : directness-related rule findings. Readability : Flesch reading ease and Flesch-Kincaid grade relative to the profile. Rhythm and structure : sentence length, sentence-length variance, and repeated openings relative to the profile. Vocabulary and style : all other style rules. Rule deductions are 2 points per excess warn occurrence and 0 per info occurrence. An excess occurrence is one above the allowance derived from the selected profile's aggregate rate. A rule absent from the profile has an allowance of zero. A rule is assigned to exactly one section, and repeated openings are scored only in rhythm and structure to prevent double-counting. Section deductions are capped at 25. Profile-relative metric deductions use a tolerance before any points are removed, then increase linearly to a cap: | Metric | Tolerance | Gap at maximum deduction | Maximum deduction | |---|---|---|---| | Reading ease | 5 points | 30 points | 6 | | Reading grade | 1 grade | 4 grades | 6 | | Average sentence length | 15% | 100% | 8 | | Sentence-length variance | 25% | 100% | 7 | | Repeated-opening rate | 3 percentage points | 20 percentage points | 10 | For sentence length and variance, the relative gap is abs article - target / max abs target , 1 . Other rows use the absolute gap. At or below the tolerance, the deduction is zero. Above it, the unrounded deduction is: maximum × min gap, cap - tolerance / cap - tolerance Each issue deduction is rounded half up to an integer before the section cap is applied. The displayed arithmetic is invariant: sum issue deductions = sum section deductions = 100 - total score Score labels describe alignment, not universal quality: | Total | Label | |---|---| | 90-100 | High alignment | | 75-89 | Moderate alignment | | 60-74 | Low alignment | | 0-59 | Very low alignment | MTLD, token 1-gram L2, overrepresented terms, and paragraph statistics remain informational. The MTLD comparison is especially sensitive to comparing one draft with an aggregated profile corpus, so it does not affect the score. writing-eval runs locally and does not send drafts, references, or profiles to a hosted service. Profiles are stored under the git-ignored data/ directory by default. Only use prose that you are permitted to process. Profiles built from third-party published content should remain private and must not be committed or quoted at length in tracked files. Tell rate by severity : style findings in a severity group per 1,000 output words. Lower values indicate fewer detected tendencies. Token 1-gram L2 : Euclidean distance between normalized output and reference token frequency vectors. Lower values indicate closer vocabulary distributions. Reported as n/a when either side has zero tokens. Overrepresented terms : tokens whose output frequency most exceeds their reference frequency. Counts or rates explain the ranking. Shared tokenization : lowercase word tokens keep ASCII and curly-apostrophe contractions together. Mean sentence length : average number of words per sentence. Sentence length variance : population variance of sentence word counts. It describes how much sentence lengths vary within the corpus. Repeated opening rate : share of adjacent sentence pairs that begin with the same normalized opening. The denominator is the number of adjacent sentence pairs, or zero when fewer than two sentences exist. Openers keep curly apostrophes U+2019 , so a contraction such as We'll is one opener unit, matching tokenization. The metrics below broaden the set toward general readability and structure. They appear in reports and check output and never enter the corpus benchmark's decision gate or any pre-registered threshold. In a profile check, reading ease and reading grade contribute only to the versioned heuristic alignment score described above; MTLD and paragraph statistics remain informational. Flesch reading ease and Flesch-Kincaid grade : standard readability scores from word, sentence, and syllable counts. Word and sentence counts use the same markdown-aware segmentation as mean sentence length: headings are excluded and list markers are stripped. Syllable counts use a vowel-group heuristic with silent terminal -e , -es , and -ed adjustments, not a dictionary, so individual word estimates can be wrong. Reported as n/a when the text has no sentence. MTLD : Measure of Textual Lexical Diversity, the mean length of word runs that keep a type-token ratio above 0.72, averaged over forward and backward passes. Higher means more varied vocabulary. Reported as n/a below 10 tokens, where the measure is unreliable. An unfinished tail that stays above the threshold counts as one factor, so a fully unique 10-token input returns 10.0. The threshold must be a finite value in 0, 1 . Token-sequence inputs are lowercased to match text inputs. Paragraph statistics : markdown-aware paragraph count, mean sentences per paragraph, and single-sentence paragraph rate. Paragraphs are blank-line separated and headings are excluded, so results depend on the input's markdown formatting. Reports retain the aggregate tell rate metric and also emit normalized tell rates by severity values. Repeated-opening corpus rates count only adjacent pairs within each document, so reordering JSONL records does not alter the result. These metrics describe observable text patterns. They do not establish factual accuracy, originality, reader preference, or overall writing quality on their own. This is the original evaluation pipeline, kept for regression checks and the pre-registered benchmark. Benchmark corpus generation and revision use the external OpenAI Codex CLI through benchmark/generate runs.py . The Codex executable and its authentication are optional benchmark dependencies. They are not required for the writing-eval CLI, evaluation of existing output files, or scripts/dry run.sh . Run a corpus evaluation with your own output and reference files: uv run python scripts/run eval.py --outputs path/to/outputs --references path/to/references.jsonl --report /tmp/writing-eval-report.md The command reads each output JSONL file, evaluates its text, compares it with the reference corpus, and writes a Markdown report to the requested path. The optional --json path writes the same report data as JSON. JSON provenance includes the tool name writing-eval , the installed package version, and identity for the reference corpus and style rule set: source paths, record or rule counts, content SHA-256 hashes, and the rule-set version. The Markdown report renders the reference-corpus and rule-set provenance; it does not render tool identity. Markdown reports keep the caller-supplied system order. JSONL writers keep object key order from dict insertion. For a checked repository run against test fixtures, use: scripts/dry run.sh The corpus evaluation is also available as the eval subcommand ./writing-eval eval --outputs ... --references ... --report ... . The flat invocation shown above is kept for backward compatibility and routes to the same behavior. Revision runs also record a literal preservation result per output in their metadata. The decision-gate report summarizes the same comparison as an informational diagnostic. It checks normalized double-quoted spans, URLs, dates, and numeric literals for additions or removals. Quote whitespace is collapsed, URL terminal punctuation is ignored, date case and commas are normalized, and numeric grouping commas are ignored. It does not change any registered criterion or the gate verdict. Each JSONL file contains one JSON object per line. IDs must be nonempty, stable, and unique within a file. A reference file may use a different ID namespace than the outputs. All output files in one run must contain the same ID set. {"id":"prompt-001","use case":"article section","prompt":"Explain why clear release notes reduce support work."} Required fields: id : stable prompt identifier use case : article section , product writing , or exec communication prompt : writing assignment text {"id":"ref-001","text":"Clear release notes help customers understand what changed and what to do next. That context prevents avoidable support requests."} Required fields: id : stable reference identifier aligned to a prompt by numeric suffix text : accepted reference prose {"id":"prompt-001","text":"Good release notes explain the change and its effect. Customers can act without opening a support ticket."} Required fields: id : prompt identifier text : generated prose to evaluate Each file in the directory passed to --outputs represents one system. Output files must contain the same prompt ID set so system comparisons stay consistent. Style rules live in YAML. Each rule has these fields: | Field | Meaning | |---|---| id | Stable, unique identifier used in findings and reports. | severity | Finding category used to group and normalize tell rates. | detector | Regex configuration or named detector that identifies the tendency. | message | Clear diagnostic text shown when the rule matches. | exceptions | Explicit cases the detector should ignore. An exception suppresses a match only when it overlaps the candidate's own context after Unicode casefold, using whole words or phrases. For regex rules that context is the match span. For named detectors it can be a larger candidate-owned span. Empty or whitespace-only exception strings are rejected at load. Use an empty list when none apply. | enabled | Merge-time directive, valid only in an overlay. enabled: false removes the named rule from the effective set. It is not a field of a loaded rule. | id , severity , detector , and message are required when you define a brand-new rule id. When you override an existing rule id in an overlay, supply only the fields you want to change. Rules are validated when loaded, after any overlay merge. Missing fields, unknown keys in standalone rule files, empty exception strings, unsupported detectors, and malformed definitions fail with a clear error. enabled is overlay-only. One rule set ships with the project, at a single in-package path. It holds 34 rules: 16 warn and 18 info . The builtin rules are recommendations, and no rule blocks the build. Both check and corpus eval use it by default. The version: key in the YAML header is a schema version, not a content version; the reported SHA-256 and rule fingerprint carry content identity. The regular passive-voice pattern is deliberately precision-first: it matches a be plus regular -ed participle only when the participle ends the sentence or is followed by a likely agent, preposition, or frequency determiner, so unlisted adjectival participles can pass through undetected. Point --rules at your own YAML file. A file without an extends key replaces the rule set outright. A file with an extends key overlays the base set: extends: builtin rules: - id: my jargon ban new id: appended to the end of the rule set severity: warn detector: ' ?i \b ?:synergy|ideate \b' message: Use plain language. - id: em dash ban existing id: override only the listed fields severity: warn - id: passive voice existing id: removed from the effective set enabled: false Merge semantics: extends: builtin resolves through the installed package, so it works from any working directory.- Any other extends value is a path resolved against the overlay file's own directory, never the current working directory. An overlay can extend another overlay, up to a chain depth of 16. - Ordering is the base order with overrides and removals applied in place, then new rules appended in the order they appear in the overlay. - An override replaces only the fields present in the overlay entry. Unlisted fields keep the base values. enabled: false on an id that is not in the base set is an error, as is a duplicate id inside one overlay file. Use the bundled overlay when a draft needs a stricter pass for reader-facing AI-writing tells than the builtin set provides, for example when reviewing machine-assisted copy before publication. Pass it with --rules ; it extends the builtin set rather than replacing it: ./writing-eval check draft.md --rules rules/anti-ai.yaml The command, output format, exit codes, and JSON schema are identical to the default check; only the finding set grows: draft.md:1:5 warn polish vocab: Replace overused polish vocabulary with specific language. | span: synergy draft.md:2:1 info connector openers: Replace the formal additive opener with also, and, or a plain sentence start. | span: Furthermore The overlay lives at rules/anti-ai.yaml in the repository outside the installed package and begins with extends: builtin , so the effective rule set is the builtin rules plus four appended rules: narrative cliches : stock narrative phrases such as "couldn't help but", "little did I know", "stumbled upon". significance markers : meta commentary that labels a moment instead of showing it, such as "that's the part that got me", "let that sink in". generation artifacts : bracketed scaffolding tokens insert , todo , tbd , placeholder , xxx , utm source=chatgpt.com links, and model self-reference such as "as an AI language model". Ordinary bracketed phrases such as your account are not flagged. connector openers : sentence-initial "furthermore", "moreover", "additionally". It also widens six builtin rules polish vocab , recap endings , throat clearing , faux insight , importance puffery , and collaborative artifacts with further phrases, keeping their severities and messages. The overlay combines with every check form, including --style , --format json , and reading from standard input. The builtin rule set stays the default everywhere. Because rule fingerprints identify comparable runs, keep the default rules for corpus evaluation and benchmark comparisons and use the overlay for single drafts. - The bundled fixtures are synthetic and intentionally exaggerated to exercise detectors. - Results are diagnostic only and should support, not replace, editorial review. - The article score is a heuristic combination of configured style rules and target-profile distances. It measures alignment under the documented rubric, not factual accuracy, argument strength, originality, or overall article quality. - Corpus metrics are sensitive to sample size, genre, and reference selection. - Simple token and sentence boundaries can differ from linguistic parsers. - A rule match identifies a review candidate, not a universal writing error. - The harness does not evaluate factual accuracy, coverage, or human preference. - Literal preservation is a normalized multiset comparison, not semantic factuality. It can detect a changed number, date, URL, or quoted span but cannot decide whether an unprotected claim is true or whether a paraphrase preserves meaning. - Distributional scores token 1-gram L2 and the top overrepresented terms depend heavily on the composition and size of the chosen reference corpus or profile; small profiles make them noisy. - L2 distance is sensitive to output length, so systems whose word counts differ a lot are not directly comparable on it. - Benchmark generation uses a frozen decoding config and measured per-metric noise floors benchmark/THRESHOLDS.md ; deltas below the documented floor are inconclusive. - Existing profiles with metric semantics other than version 2 must be rebuilt before use. - Readability scores ignore heading text and list markers, so heading-heavy Markdown can differ from a plain-text reading. profile list omits directories that load profile rejects. writing-eval is source available under the Elastic License 2.0 /majesticlabs-dev/writing-eval/blob/master/LICENSE . You may use, copy, modify, redistribute, and self-host the software subject to that license. You may not provide it to third parties as a hosted or managed service where the service gives users access to any substantial set of the software's features or functionality. Copyright 2026 Majestic Labs LLC. Contributions are welcome. Read CONTRIBUTING.md /majesticlabs-dev/writing-eval/blob/master/CONTRIBUTING.md before submitting a pull request. Contributors retain ownership of their work and must accept the Contributor License Agreement /majesticlabs-dev/writing-eval/blob/master/CLA.md , which gives Majestic Labs the rights needed to keep contributions in the public project and use them in an official hosted service. For OEM, hosted-service, or managed-service licensing, contact Majestic Labs https://majesticlabs.dev/ai?utm source=github&utm medium=readme&utm campaign=writing-eval inquiry . Majestic Labs is considering an official hosted writing-eval service for people and teams that cannot or prefer not to deploy and maintain the tool. The local CLI remains complete and self-hostable. If managed hosting would help your team, tell us about your workflow https://majesticlabs.dev/ai?utm source=github&utm medium=readme&utm campaign=writing-eval inquiry . Majestic Labs https://majesticlabs.dev/?utm source=github&utm medium=readme&utm campaign=writing-eval is a software foundry for operator-built software. writing-eval is part of our open-source https://majesticlabs.dev/open-source?utm source=github&utm medium=readme&utm campaign=writing-eval projects. If your team needs to evaluate an AI workflow against company-specific standards, start with Majestic AI https://majesticlabs.dev/ai?utm source=github&utm medium=readme&utm campaign=writing-eval inquiry . This project is not a DFT clone or training system. It is a small deterministic harness for writing diagnostics.