{"slug": "building-a-generic-context-aware-scoring-engine", "title": "Building a Generic Context-Aware Scoring Engine", "summary": "Ts-profanity-filter, a TypeScript library for context-aware profanity filtering, now offers a streaming batch runner with a CLI, PII detection, and optional AI checks, supporting English and German out of the box with custom language registration. The library splits text into segments for UI rendering, never mutating the string, and includes a DSA Art. 17 statement generator. Version 1.0.0 is available on npm, with zero runtime dependencies and optional React, Vue, and Angular adapters.", "body_md": "A strict TypeScript profanity filter that splits text into segments so your UI can render the redaction itself — the library never mutates or masks your string.\n\n**English and German ship pre-registered; any other language is a\nregisterLanguage() call away.** Leet spellings, lookalike letters, spaced-out\nwords and repetition are matched; a cross-check keeps ordinary words like\n\n`Klassik`\n\nand `classic`\n\nout of the results.Zero runtime dependencies. Optional adapters for React, Vue and Angular, an\noptional AI check, a **PII detector** for e-mail addresses, phone numbers,\nIBANs and cards, a **streaming batch runner** with a CLI for corpora that do not\nfit in memory, and an optional generator for the **DSA Art. 17** statement of\nreasons you owe whoever you moderated. Each is its own subpath, so nothing you\ndo not import reaches your bundle.\n\n```\nnpm install ts-profanity-filter\n```\n\n**Analysing one comment is a function call. Analysing two million is a different\nproblem** — and the difference is not speed. The obvious version holds the whole\ncorpus in memory, dies at row 900 000 with nothing written, and makes one paid\nmodel call per row.\n\n``` js\nimport { runBatch, formatSummary } from 'ts-profanity-filter/batch';\nimport { ndjsonFrom } from 'ts-profanity-filter/batch/node';\n\nconst summary = await runBatch(ndjsonFrom('comments.ndjson'), {\n  filter: { languages: ['en', 'de'] },\n  pii: true,\n  ai: { provider: 'gemini', when: 'matched', maxCalls: 500 },\n  onResult: (result) => { if (result.flagged) hold(result.id); },\n});\n\nconsole.log(formatSummary(summary));\n```\n\nIterable in, results streamed out: **peak memory is one record**, whatever the\nfile size. Every stage is wrapped per record, so one hostile input costs a result\nand not the run. And the model is *gated* rather than called per row — `when: 'matched'`\n\nasks only about records a word list already hit, and `maxCalls`\n\nis a\nhard ceiling, because a batch is exactly where one call per row becomes a bill.\n\n**There is a command now:**\n\n```\nnpx ts-profanity-filter scan comments.ndjson --pii --out flagged.ndjson --pdf report.pdf\n```\n\nIt reads NDJSON, CSV, TSV or plain lines, guesses the text column from the CSV\nheader and says which one it picked, writes the flagged records back as NDJSON,\nand exits 1 under `--fail-on-findings`\n\nfor CI. Progress goes to stderr and the\nsummary to stdout, so it composes.\n\n[The full section →](#batch-processing) ·\n[The CLI guide →](/Kevinci/ts-profanity-filter/blob/main/docs/cli.md) ·\n[Try it in the playground →](https://kevinci.github.io/ts-profanity-filter/#sec-batch)\n\n**Also in this release:**\n\n**A chat log to test against.** is 25 messages with a documented expected verdict for every row — twelve of them deliberately clean, which is the half that catches a detector getting eager.`examples/batch/chat-log.csv`\n\n**The CSV reader no longer scans the wrong column quietly.** It used to default to column 0, so pointing it at a file whose first column is an id reported`0 flagged`\n\n— indistinguishable from a genuinely clean file. It now guesses from the header, announces the choice, and stops and asks when it cannot.**PDF reports** via`renderSummaryPdf()`\n\n, throughas an`fast-pdf`\n\n**optional** peer dependency loaded by dynamic import in that one function.`dependencies`\n\nstays empty.lists every feature with one line of what it is and one of why it works that way, in English and German.[FEATURES.md](/Kevinci/ts-profanity-filter/blob/main/FEATURES.md)\n\n**A moderation filter that cannot see an IBAN is half a filter.** The same\ncomment box that collects insults collects phone numbers, bank details and card\nnumbers, and `ts-profanity-filter/pii`\n\nreports those the way this library reports\neverything — as spans, so the redaction stays yours to render.\n\n``` js\nimport { detectPii } from 'ts-profanity-filter/pii';\n\ndetectPii('IBAN DE44 5001 0517 5407 3249 31, Tel. 030 12345678');\n// [\n//   { kind: 'iban',  confidence: 0.99, evidence: ['structure', 'checksum', 'context'], … },\n//   { kind: 'phone', confidence: 0.99, evidence: ['structure', 'context'], … },\n// ]\n```\n\n**The admission criterion is that a finding can be verified.** An IBAN passes\nmod-97 and its country's length, a card passes Luhn and owns its issuer prefix, a\nGerman tax id passes ISO 7064 *and* the repetition rule the BZSt guarantees.\nNames, postal addresses and dates of birth are missing on purpose: nothing inside\nthe string can confirm them, and a detector that guesses at those turns every\ncapitalised word into a finding.\n\n**It is one pass, not six regexes.** The text is walked once for anchors, digit\nclusters are built once and interpreted by three recognizers, every candidate is\n*scored* rather than accepted, and overlaps are settled by weighted interval\nscheduling — because `::ffff:192.168.1.1`\n\nis an IPv6 address containing an IPv4\none, and resolving greedily from the left picks the earliest candidate rather\nthan the best one.\n\n[The full section →](#personal-data) ·\n[Try it in the playground →](https://kevinci.github.io/ts-profanity-filter/#sec-pii)\n\n**Also in 1.4.0, alongside this:**\n\n**The playground shows what it suppresses.** The panel has a switch that drops`minConfidence`\n\nto 0.2, so the findings that scored too low to be reported become visible in grey instead of being invisible.**The demo build now catches a clash the old guard could not.** Its modules share one script scope on the page, and two of them declaring the same top-level`const`\n\nis a`SyntaxError`\n\nthat blanks the whole page. The check compared the bundle against the page script but never against itself — it now does, which is how`SEPARATOR`\n\nand`ALNUM`\n\nwere caught before shipping.\n\n**In the EU, deleting the comment is only half the obligation.** Article 17 of\nthe Digital Services Act requires that whoever is moderated gets a *statement of\nreasons*: what was done, on which ground, on which facts, whether an automated\nsystem was involved, how long it lasts, and where to contest it — in their\nlanguage, on a durable medium they can keep.\n\nA filter that returns `flagged: true`\n\ngives you none of that. So the new\n`ts-profanity-filter/compliance`\n\nsubpath builds the notice out of the moderation\nresult you already have:\n\n``` js\nimport { moderateText } from 'ts-profanity-filter/ai';\nimport {\n  generateJustification,\n  formatJustificationAsText,\n} from 'ts-profanity-filter/compliance';\n\nconst result = await moderateText(comment, {\n  languages: ['de'],\n  ai: { provider: 'gemini', enabled: true },   // the graded verdict\n});\n\nconst notice = await generateJustification(comment, result, {\n  action: 'CONTENT_REMOVED',\n  policyBases: [{ name: 'Community Guidelines', section: '§4.2' }],\n  appealUrl: 'https://example.com/appeal/8f21',\n  ai: { provider: 'gemini', enabled: true },   // wording only — optional\n});\n\nformatJustificationAsText(notice);   // the text you send the user\nexportJustification(notice);         // the JSON you keep for your records\n```\n\n**The facts are never the model's to decide.** Action, policy basis, categories,\nseverity, confidence, the quoted excerpt and the timestamp are all fixed by the\ncode before any model is asked. What a model contributes is the two things a\ntemplate cannot write: a `reason`\n\nthat names the measure and the behaviour in one\nbreath, and an `assessment`\n\nthat weighs the case — and says so plainly when the\ncall is uncertain. Leave `ai`\n\nout and the built-in German and English templates\ncarry the notice on their own.\n\n[The full section →](#dsa-art-17-justifications) ·\n[See one generated →](https://kevinci.github.io/ts-profanity-filter/#sec-compliance)\n\n**Also in 1.3.0, alongside this:**\n\n**A third AI provider that needs no network.**`ollama`\n\nruns the check on your own machine — no key, no third party, the same JSON Schema constraining the answer, so switching is a config change and not a second code path. See[Nothing leaves the building](#nothing-leaves-the-building).**Three false positives and a Cyrillic**, found by pointing an adversarial benchmark of 81 attacks at this filter.`к`\n\nfixed`Cockburn`\n\n,`Lightwater`\n\nand`Matsushita`\n\nare names, which is the most expensive kind of false positive;`к`\n\nwas never in the expandable set, so no pattern could reach it. English went from 82/83 to 94/100, German to 74/100, both with regression tests.\n\n**Optionally, a model reads the whole sentence** — for what no word list can\nsee. A message can be a threat without containing a single listed word, and it\ncan be full of them and still be a quotation. Word lists cannot tell the\ndifference; this can.\n\nIt reports hate, threats, harassment, racism, obscenity, sexual content involving minors and pressure toward self-harm — with a severity, a confidence, one sentence of reasoning in the language of the text, and the exact stretch it objected to, so you can highlight it.\n\n``` js\nimport { moderateText } from 'ts-profanity-filter/ai';\n\nconst result = await moderateText(comment, {\n  languages: ['en', 'de'],\n  ai: { provider: 'gemini', enabled: true },   // key from GEMINI_API_KEY\n});\n\nresult.matchedList   // a word list matched\nresult.ai.flagged    // the model flagged the sentence as a whole\nresult.flagged       // either of the two\n```\n\n**Google Gemini** needs nothing installed — it is a plain `fetch`\n\n, and the free\ntier covers this. **Anthropic Claude** works through the optional SDK. Or bring\nany model at all with `ai.complete`\n\n.\n\n**Off unless you ask for it.** No `ai`\n\noption means no model is contacted and\nnothing leaves your machine — the word-list half never calls out at all.\n\n[The full section →](#ai-check-optional) ·\n[Try it with your own key →](https://kevinci.github.io/ts-profanity-filter/#sec-ai)\n\n**Evasion is a Unicode problem**, and the matching path treats it as one.\nCompatibility spellings fold with NFKC to the letters the patterns are written\nin, so `Ｄｒｅｃｋｓａｕ`\n\n, `𝐃𝐫𝐞𝐜𝐤𝐬𝐚𝐮`\n\nand `Ⓓⓡⓔⓒⓚⓢⓐⓤ`\n\nstop walking past the\nlist. Whole-word anchors use Unicode boundaries instead of `\\b`\n\n— which is\ndefined in terms of `\\w`\n\nand stays ASCII even under the `u`\n\nflag, so every\numlaut and every `ß`\n\nread as a word boundary and `Straußschwanz`\n\ncame back\nflagged.\n\nIteration is by code point rather than code unit, and the offset map carries one entry per output character, so a folded character that expands still points back at the one it came from — and a segment boundary can no longer land inside a surrogate pair.\n\nSeventeen cases assert the **offsets**, not the round trip. Rebuilding the\nstring intact proves only that nothing was dropped; it says nothing about\nwhether the flagged span still covers the right characters, which is exactly\nwhere a filter holding three representations of the input — original, folded\nhaystack, segments — goes wrong.\n\n``` js\nimport { filterFWordsToSegments } from 'ts-profanity-filter';\n\nconst output = filterFWordsToSegments('This is bullsh1t.', { languages: ['en'] });\n\n// [\n//   { text: 'This is bull', isProfane: false },\n//   { text: 'sh1t',         isProfane: true  },\n//   { text: '.',            isProfane: false },\n// ]\n```\n\nConcatenating every `segment.text`\n\nalways reproduces the original input exactly,\nso rendering is lossless:\n\n``` js\n<p>\n  {filterFWordsToSegments(comment).map((seg, i) =>\n    seg.isProfane ? <span key={i} className=\"redacted\">{seg.text}</span> : seg.text,\n  )}\n</p>\n```\n\nMatching is **substring-based**. That is what catches `asshole`\n\nfrom `ass`\n\nand\nsurvives obfuscation — but on its own it also flags `class`\n\n, `Klassik`\n\nand\n`Massage`\n\n.\n\nSo every hit is checked against an allowlist of ordinary words before it counts.\nThe allowlist is anchored against the **whole surrounding word**, and an allowed\nword always beats a blocked pattern:\n\n```\nfilterFWordsToSegments('Der Klassiker war klasse.', { languages: ['en', 'de'] });\n// -> one clean segment; the two `ass` hits are dropped\n\nfilterFWordsToSegments('Der Klassiker war klasse.', {\n  languages: ['en', 'de'],\n  crossCheck: false,          // raw substring matching\n});\n// -> `ass` flagged twice\n```\n\nCross-checked out of the box, among others:\n\n| Language | Blocked pattern | Ordinary words it would otherwise hit |\n|---|---|---|\n| en | `ass` |\nclass, pass, assistant, embarrass, potassium |\n| en | `cunt` |\nScunthorpe |\n| en | `cock` |\ncocktail, cockpit, peacock |\n| en | `spic` |\nspicy, suspicious, conspicuous |\n| de | `ass` |\nKlassik, klassisch, Massage, Sparkasse, Tasse |\n| de | `arsch` |\nMarsch, marschieren, Barsch, harsch |\n| de | `anal` |\nAnalyse, Kanal, banal, Analphabet |\n| de | `cum` (via `k` ) |\nDokument, Kumpel, Publikum, Vakuum |\n\nAdd your own with `allowList`\n\n— entries are regex sources matched against the\nwhole word:\n\n```\nfilterFWordsToSegments('Die Assmann GmbH', {\n  languages: ['en', 'de'],\n  allowList: ['assmann', 'meine-firma\\\\p{L}*'],\n});\n```\n\n| Option | Type | Default | Description |\n|---|---|---|---|\n`languages` |\n`string[] | '*'` |\n`['en']` |\nRegistered languages to match against. `'*'` uses every registered one. |\n`crossCheck` |\n`boolean` |\n`true` |\nDrop a hit when the surrounding word is allowlisted. `false` = raw substrings. |\n`allowList` |\n`string[]` |\n— | Extra allowed words, added on top of the built-in allowlist. Regex sources. |\n`customList` |\n`string[]` |\n— | Replaces the built-in patterns entirely. Regex sources. Empty array = fall back. |\n`aggressive` |\n`boolean` |\n`true` |\nAlso match lookalike spellings — see below. |\n\nPatterns are compiled with the `u`\n\nflag, so case-insensitive matching uses full\nUnicode case folding — `SCHEIẞE`\n\nfolds to `scheiße`\n\nand is caught. It also means\nyour patterns must be valid in unicode mode: a stray identity escape like\n`\\\\-`\n\nis an error there. `registerLanguage`\n\nreports that up front.\n\n```\ninterface TextSegment {\n  text: string;\n  isProfane: boolean;\n}\n```\n\nWith `aggressive`\n\non (the default) every letter is expanded into the things\npeople actually type to get past a filter:\n\n| Kind | Example |\n|---|---|\n| leet | `Dr3cks4u` , `a$$hole` , `$hit` , `fu(k` |\n| diacritics | `DräckSAU` , `ärschloch` |\n| cross-script lookalikes | `Аrschloch` (Cyrillic А), `Sсheiße` (Cyrillic с) |\n\nThe diacritics are deliberately ambiguous: `ä`\n\ncounts as an **a** *and* as an\n**e**, because `Dräck`\n\nuses it as an e while `ärsch`\n\nuses it as an a. A\none-to-one normalisation would have to pick a side and get one of them wrong.\n\n**The allowlist gets the same expansion.** That symmetry is the whole point —\nwithout it `ass`\n\nmatches the `4ss`\n\nin `Kl4ssik`\n\nwhile the allow entry still\nonly spells `klass`\n\n, and an ordinary word comes back flagged. `Kl4ssik`\n\n,\n`M4ssage`\n\n, `Cl4ss`\n\nand `Fässer`\n\nall stay clean.\n\nSeparators, repetition and invisible characters are handled differently,\nbecause they change the *length* of the text and no character class can reach\nthem. The text is rewritten for matching, and every rewritten character\nremembers which slice of the original it came from — so matches are found in\nthe rewritten copy and sliced out of the original, and the segments still add\nup to the input exactly.\n\n| Kind | Example | Rule |\n|---|---|---|\n| spaced out | `D r e c k s a u` , `D-r-e-c-k-s-a-u` |\nthree or more whole one-letter words in a row |\n| repetition | `Dreeecksau` , `fuuuuck` |\nruns of three or more identical characters collapse |\n| invisible | `Dreck<ZWSP>sau` |\nformatting characters are dropped |\n| decomposed | `a` + combining diaeresis |\ncomposed into `ä` first |\n| compatibility forms | `Ｄｒｅｃｋｓａｕ` , `𝐃𝐫𝐞𝐜𝐤𝐬𝐚𝐮` , `Ⓓⓡⓔⓒⓚⓢⓐⓤ` |\nfolded with NFKC to the plain letters |\n\nDoubles are left alone — `Klasse`\n\nand `Fässer`\n\nare ordinary spelling, and\ncollapsing them would break the allowlist. The spaced-out rule needs *whole*\none-letter words, which is what keeps `next to a cockroach`\n\nfrom collapsing\ninto `next toacockroach`\n\nand inventing a `cock`\n\n.\n\n`en`\n\nand `de`\n\nare simply the two that ship pre-registered. Nothing about the\nlibrary is limited to them.\n\n``` js\nimport { registerLanguage } from 'ts-profanity-filter';\n\nregisterLanguage('fr', {\n  profanity: ['merde', 'connard', 'salope', 'putain'],\n  allow: ['\\\\p{L}*connaiss\\\\p{L}*'],   // connaissance, connaisseur\n});\n\nfilterFWordsToSegments('Quelle merde', { languages: ['fr'] });\n```\n\n**Regional variants inherit instead of duplicating.** A parent's patterns come\nfirst, yours are added on top — including its allowlist, so the false positives\nit already solved stay solved:\n\n```\nregisterLanguage('de-AT', { extends: 'de', profanity: ['oasch', 'gschissana'] });\n\nfilterFWordsToSegments('Du Oasch, du Trottel!', { languages: ['de-AT'] });\n// both flagged: 'oasch' is the variant's, 'Trottel' is inherited from 'de'\n```\n\n**Lookups fall back along BCP-47 subtags.** `de-AT-1996`\n\ntries `de-at-1996`\n\n,\nthen `de-at`\n\n, then `de`\n\n— so an unregistered `de-CH`\n\nstill works, and codes\nare case-insensitive.\n\n**Use everything at once** with `'*'`\n\n:\n\n```\nfilterFWordsToSegments(text, { languages: '*' });\n```\n\n**Patterns are validated when you register them**, not when text is filtered, so\na typo fails at startup naming the offending entry rather than throwing inside a\nmoderation request:\n\n```\nSyntaxError: registerLanguage('fr'): profanity[1] \"(unclosed\" is not a\nvalid regular expression — Invalid regular expression: /(unclosed/gi:\nUnterminated group\n```\n\nAn **unknown language throws** rather than being ignored — silently matching\nnothing is the worst way for a moderation filter to fail.\n\n| Function | Purpose |\n|---|---|\n`registerLanguage(code, def)` |\nAdd or replace a language. `def` is `{ profanity?, allow?, extends? }` . |\n`unregisterLanguage(code)` |\nRemove one. Refuses while another language extends it. |\n`resetLanguages()` |\nBack to just the built-in `en` and `de` . |\n`getLanguage(code)` |\nResolved lists, `extends` flattened and subtags applied. |\n`hasLanguage(code)` / `resolveKey(code)` |\nExistence check / which code it resolves to. |\n`listLanguages()` |\nEvery registered code. |\n\nThe built-in packs are importable on their own, which is also the shape a third-party language pack should export:\n\n``` js\nimport { en, EN_PROFANITY, EN_ALLOWLIST } from 'ts-profanity-filter/lang/en';\nimport { de } from 'ts-profanity-filter/lang/de';\n\nregisterLanguage('en-custom', { extends: 'en', profanity: ourExtraWords });\n```\n\nBoth are registered by the main entry point, so they are in your bundle whether\nor not you use them — `unregisterLanguage`\n\nchanges behaviour, not bundle size.\n\nThe package is plain ESM with no dependencies, so a module script and a CDN import are the whole setup — no build step, nothing to install.\n\n[Run this example on JSFiddle →](https://jsfiddle.net/7x6mtawo/)\n\n```\n<textarea id=\"draft\"></textarea>\n<p id=\"output\"></p>\n\n<style>\n  .redacted { background: currentColor; border-radius: 1px; }\n</style>\n\n<script type=\"module\">\n  import { filterFWordsToSegments } from 'https://cdn.jsdelivr.net/npm/ts-profanity-filter@1.5.0/+esm';\n\n  const draft = document.getElementById('draft');\n  const output = document.getElementById('output');\n\n  function render() {\n    const segments = filterFWordsToSegments(draft.value, { languages: ['en', 'de'] });\n\n    output.replaceChildren(\n      ...segments.map((seg) => {\n        const node = document.createElement('span');\n        if (seg.isProfane) node.className = 'redacted';\n        node.textContent = seg.text;   // textContent, never innerHTML\n        return node;\n      }),\n    );\n  }\n\n  draft.addEventListener('input', render);\n  render();\n</script>\n```\n\n`node.textContent = seg.text`\n\nis the line that matters. What you are rendering\nis whatever a stranger typed, and pushing it through `innerHTML`\n\nwould hand\nthem your page. Because the API returns segments rather than a marked-up\nstring, building nodes is both the safe route and the obvious one.\n\nNothing to set up — every npm CDN serves the package automatically, subpath\nimports included. [The jsDelivr package page](https://www.jsdelivr.com/package/npm/ts-profanity-filter)\nlists every published file and version.\n\n| CDN | URL |\n|---|---|\n| jsDelivr | `https://cdn.jsdelivr.net/npm/ts-profanity-filter@1.5.0/+esm` |\n| esm.sh | `https://esm.sh/ts-profanity-filter@1.5.0` |\n| unpkg | `https://unpkg.com/ts-profanity-filter@1.5.0/dist/index.js` |\n\n``` js\nimport { useProfanitySegments } from 'https://esm.sh/ts-profanity-filter@1.5.0/react';\nimport { de } from 'https://esm.sh/ts-profanity-filter@1.5.0/lang/de';\n```\n\n**Pin the version.** An unpinned URL like `https://esm.sh/ts-profanity-filter`\n\nalways resolves to the newest release, so your page starts running different\ncode the next time this package is published — without you changing anything.\nFine for a playground, not for production.\n\nTo use a local install instead of a CDN, point an import map at it:\n\n```\n<script type=\"importmap\">\n  {\n    \"imports\": {\n      \"ts-profanity-filter\": \"/node_modules/ts-profanity-filter/dist/index.js\"\n    }\n  }\n</script>\n```\n\nIf all you need is the verdict or a masked string:\n\n``` js\nconst segments = filterFWordsToSegments(text, { languages: ['en', 'de'] });\n\nconst isProfane = segments.some((seg) => seg.isProfane);\n\nconst masked = segments\n  .map((seg) => (seg.isProfane ? '*'.repeat(seg.text.length) : seg.text))\n  .join('');\n```\n\nWord lists catch **words**. They cannot tell that a sentence containing no\nlisted word at all is a threat, or that one full of them is a quotation. That\njudgement is what a model adds.\n\nNothing to install for the Gemini path — it is a plain `fetch`\n\n, and Google's\nfree tier covers this use case:\n\n``` js\nimport { moderateText } from 'ts-profanity-filter/ai';\n\nconst result = await moderateText(comment, {\n  languages: ['en', 'de'],\n  ai: { provider: 'gemini', enabled: true },   // key read from GEMINI_API_KEY\n});\n\nresult.matchedList    // a word list matched\nresult.ai.flagged     // the model flagged the sentence as a whole\nresult.flagged        // either of the two\n```\n\n**It is off unless you ask for it.** No `ai`\n\noption means no model is contacted,\nand `moderateText`\n\nis then just the local filter in a wrapper. `enabled: false`\n\nkeeps the configuration around with the check switched off.\n\n```\nai: { provider: 'gemini', enabled: true }   // key from GEMINI_API_KEY\nai: { enabled: true }                        // anthropic, key from ANTHROPIC_API_KEY\nai: { provider: 'ollama', model: 'llama3.2' } // your machine, no key, no network\n```\n\n| Needs | Default model | Key from | |\n|---|---|---|---|\n`gemini` |\nnothing — plain `fetch` |\n`gemini-flash-lite-latest` |\n|\n\n`anthropic`\n\n`@anthropic-ai/sdk`\n\n`claude-opus-5`\n\n[console.anthropic.com](https://console.anthropic.com/settings/keys)— paid`ollama`\n\n[Ollama](https://ollama.com)`llama3.2`\n\n`ollama`\n\nanswers the objection that makes this whole feature a non-starter for\nsome deployments: that moderating a message means handing it to a third party.\n\n``` js\nconst result = await moderateText(comment, {\n  languages: ['de'],\n  ai: { provider: 'ollama', model: 'gemma3' },\n});\n```\n\nSame prompt, same schema, same verdict shape as the hosted providers — Ollama\nconstrains decoding to the JSON Schema exactly as they do, so switching is a\nconfig change rather than a different code path. Point `ai.baseUrl`\n\nat another\nhost, or set `OLLAMA_HOST`\n\n; a bearer token is sent only if you supply one, for\nthe case where the server sits behind an authenticating proxy.\n\nTwo things to expect. It is **slower** — a first call also loads the weights, and\na 9 GB model took 34 seconds on a warm laptop, which is why the default timeout\nfor this provider is 120 s rather than 20 s. And it is **only as good as the\nmodel you pulled**: a small instruct model handles clear-cut cases well and gets\nvaguer at the edges, which is the trade you are making for the text never\nleaving the host.\n\nFor anything else — transformers.js, a hosted open-weights endpoint, your own\nfine-tune — `ai.complete`\n\ntakes any function that returns JSON matching the\nschema. See [Any model, not just Claude](#any-model-not-just-claude).\n\nGemini is the cheapest way to try this: its free tier covers the use case and it\nadds no dependency at all. `ai.model`\n\ntakes any id the provider accepts;\n`AI_MODELS`\n\nlists a few per provider for populating a picker.\n\nThe Gemini list is the one verified against a fresh free-tier key, which is not\nthe same as the list the models endpoint returns: `gemini-2.5-flash`\n\nis\nadvertised there but rejected for new accounts (\"no longer available to new\nusers\"), and `gemini-2.0-flash`\n\nis out of free quota. The default is\n`gemini-flash-lite-latest`\n\nbecause an alias cannot go stale that way.\n\nOne thing worth knowing about the Gemini provider: it sends `BLOCK_NONE`\n\nfor\nGoogle's own safety categories. For a moderation classifier that default is\nbackwards — the text you need it to read is exactly the text it otherwise\nrefuses to look at. It is safe here *because* the output is a verdict rather\nthan generated content: the model labels text you already have, it never\nproduces any.\n\n| Category | Covers |\n|---|---|\n`racism` |\nracial or ethnic slurs; dehumanising by origin or skin colour |\n`hate` |\ncontempt toward a group — religion, ethnicity, nationality, sexuality, gender, disability |\n`violence` |\nthreats, calls to harm, approval of harm; incitement against a whole group is the most severe form |\n`harassment` |\ninsults and degradation aimed at a specific person |\n`sexual` |\nexplicit or obscene sexual content |\n`sexual_minors` |\nsexualisation of a minor, grooming, predatory approaches |\n`self_harm` |\nencouraging suicide or self-injury |\n\nAlongside them: a `severity`\n\n(`none`\n\n… `critical`\n\n), a `confidence`\n\n, one\nsentence of `reason`\n\nin the language of the text, and a `quote`\n\n— the stretch\nof the input the model objected to, copied verbatim so you can locate and\nhighlight it:\n\n``` js\nconst at = result.ai.quote ? text.indexOf(result.ai.quote) : -1;\nif (at !== -1) highlight(at, at + result.ai.quote.length);\n```\n\nCheck it against the original before using it, as above. A model can paraphrase despite being told not to, and a span that does not match must not be invented.\n\n| Option | Type | Default | Description |\n|---|---|---|---|\n`enabled` |\n`boolean` |\n`true` when `ai` is present |\nThe switch. No `ai` at all means no model is contacted. |\n`provider` |\n`'anthropic' | 'gemini' | 'ollama'` |\n`'anthropic'` |\n`gemini` needs no SDK and has a free tier; `ollama` needs no key and no network. |\n`apiKey` |\n`string` |\n`ANTHROPIC_API_KEY` / `GEMINI_API_KEY` |\nKeep it server-side. |\n`model` |\n`string` |\nper provider | Any id the provider accepts; `AI_MODELS` lists a few. |\n`categories` |\n`AiCategory[]` |\nall seven | Narrow what is checked. |\n`prompt` |\n`string` |\nbuilt-in | Replaces the system prompt entirely. |\n`extraInstructions` |\n`string` |\n— | Appended to the built-in prompt. |\n`languageHint` |\n`string` |\nauto-detect | e.g. `'German'` . |\n`effort` |\n`'low'…'max'` |\n`'low'` |\nAnthropic only. This is a classification, not an essay. |\n`maxTokens` |\n`number` |\n`4096` |\n|\n`timeoutMs` |\n`number` |\n`20000` |\n|\n`fallback` |\n`boolean` |\n`true` |\nAnthropic only: retry on another model if its safety layer declines. |\n`onError` |\n`'return' | 'throw'` |\n`'return'` |\nA failed check is a decision, not an exception. |\n`complete` |\n`AiCompletion` |\n— | Bring your own model; bypasses both built-in providers. |\n\nBoth functions are exported: `moderateText(text, options)`\n\nruns the local\nfilter *and* the model, `analyzeWithAi(text, aiOptions)`\n\nruns only the model.\n\nThe built-in prompt describes each category rather than listing example slurs — spelling them out would ship those words in every request and teach the filter one exact wording. Extend it, or replace it:\n\n```\nai: {\n  enabled: true,\n  categories: ['racism', 'hate', 'violence'],   // narrow the check\n  extraInstructions: 'Football banter is fine here.',  // added to the default\n  prompt: myOwnSystemPrompt,                    // replaces it entirely\n  languageHint: 'German',\n  effort: 'low',                                // cheap by default\n}\n```\n\n`moderateText`\n\nnever throws by default — a moderation call that fails should be\nsomething you decide about, not something that takes down the request:\n\n```\nswitch (result.ai.status) {\n  case 'ok':       break;                        // a verdict was produced\n  case 'disabled': break;                        // the check is switched off\n  case 'refused':  hold(result.ai.error); break; // the provider's own safety layer declined\n  case 'error':    hold(result.ai.error); break; // the call failed\n}\n```\n\nA failed or refused check **never** reports `flagged: true`\n\n— absence of a\nverdict is not a clean bill of health. Set `onError: 'throw'`\n\nif a missing\nverdict should stop the request instead. The API key is stripped from error\nmessages before they are returned.\n\n`ai.complete`\n\nreplaces the built-in provider with anything that takes a system\nprompt plus text and returns JSON matching the schema. It is also how the test\nsuite runs without a network:\n\n``` python\nimport type { AiCompletion } from 'ts-profanity-filter/ai';\n\nconst myModel: AiCompletion = async ({ system, text, schema }) => ({\n  json: await callWhateverYouLike(system, text, schema),\n});\n```\n\n**This package sends nothing on its own.** It ships no key and has no account.\nWithout an `ai`\n\noption it makes no network call at all, and the word-list half\nnever does under any circumstances. When you switch the check on, the call goes\nout under *your* key, to *your* account, on *your* decision.\n\nThat is worth being precise about, because it decides who answers for it: you are the controller of that processing, and the provider is your processor. Under the GDPR that means you need a lawful basis for sending user-submitted text to them, a data processing agreement with them, and a privacy notice that says so. None of that is something a library can do for you.\n\n**The free Gemini tier is not suitable for production data.** From\n[Google's Gemini API terms](https://ai.google.dev/gemini-api/terms):\n\nTo help with quality and improve our products, human reviewers may read, annotate, and process your API input and output.\n\nOn the paid tier the same terms say the opposite — \"Google doesn't use your\nprompts … or responses to improve our products\". The free tier is the right way\nto *try* this feature; it is the wrong way to run it on real user messages.\nCheck Anthropic's current terms the same way before relying on either.\n\nThe moderation case makes this sharper than usual: the text you send is, by definition, the text somebody wrote when they were at their worst. Three ways to keep that proportionate:\n\n**Filter first, ask second.** Run the local lists on everything and reach for the model only on what they cannot settle. Most messages never leave.**Send the message, not the person.** No names, no ids, no metadata — the check takes a string and nothing else.**Or keep it in-house.**`ai.complete`\n\ntakes any model, including one you run yourself, and nothing about the rest of the feature changes.\n\n`examples/server`\n\nis the whole shape in one file: a browser page, an endpoint,\nand the key staying in the server process.\n\n```\ncd examples/server\ncp .env.example .env        # put your key in it — .env is gitignored\nnpm install && npm start    # http://localhost:8787\n```\n\nIt reads the key from the environment rather than the request body, returns a verdict rather than the machinery behind it, caps the body size, and rate-limits per IP — an endpoint that spends money per call needs all four. See its README for what it deliberately leaves out (auth, above all).\n\nAn API key shipped to a browser is readable by everyone who loads the page and spendable by all of them — no amount of obfuscation changes that. Run the check on your server and send the verdict to the client, never the key. The word-list half of this library runs happily in the browser; the model half does not belong there.\n\nDefaults worth knowing: `effort: 'low'`\n\n(this is a\nclassification, not an essay), a 20-second timeout, and provider-side retry on\nanother model if Anthropic's own safety layer declines the request — moderation\ntext is exactly the kind of input that trips those classifiers, and a refusal is\nnot a verdict. Turn that off with `fallback: false`\n\n.\n\nArticle 17 of the [Digital Services\nAct](https://eur-lex.europa.eu/eli/reg/2022/2065/oj) applies to any hosting\nprovider that restricts something a user posted — removal, demotion, a feature\nlock, a suspension. The user is owed a statement of reasons, and the article\nlists what has to be in it: the measure and its scope and duration, the facts\nthe decision rests on, the ground relied on, whether automated means were used,\nand how to contest it.\n\n`ts-profanity-filter/compliance`\n\nassembles that from a `ModerationResult`\n\n.\n\n``` js\nimport {\n  generateJustification,\n  exportJustification,\n  formatJustificationAsText,\n  InMemoryJustificationStore,\n} from 'ts-profanity-filter/compliance';\n```\n\n| Field | Comes from | Art. 17 point it answers |\n|---|---|---|\n`action` |\nyou | which measure — `CONTENT_REMOVED` , `CONTENT_DEMOTION` , `ACCOUNT_SUSPENSION` , `ACCOUNT_TERMINATION` , `FEATURE_RESTRICTION` |\n`duration` |\nyou | its scope in time — `'7d'` , `'permanent'` , … |\n`policyBases` |\nyou | the ground: a name, an optional `section` , an optional `url` to the rule |\n`facts.quote` |\nthe model's excerpt, else the words the lists matched | which content, verbatim |\n`facts.categories` |\nthe AI check | what it was classified as |\n`facts.severity` |\nthe AI check | how heavily it weighs |\n`facts.confidence` |\nthe AI check | how sure the classification was |\n`facts.automatedDetection` |\ncomputed | whether automation was involved — Art. 17(3)(c) |\n`facts.humanReview` |\nalways `false` |\nsee below |\n`reason` |\ntemplate or model | one sentence, in the user's language |\n`assessment` |\ntemplate or model | two or three sentences weighing the case |\n`appealUrl` |\nyou | the redress route |\n`timestamp` , `language` |\ncomputed | when, and in which language |\n\n`language`\n\nis auto-detected between `de`\n\nand `en`\n\nfrom the text unless you pass\none. Pass it explicitly if you know the user's language — which you usually do,\nand it is *their* language the notice owes, not the language they happened to\nwrite that sentence in.\n\n**A model may word the notice. It may not decide anything in it.** The facts are\nhanded to it as data, together with an instruction that quoted text is material\nunder judgement and not instructions to follow. It writes two fields and nothing\nelse; a reply that fails to parse leaves the templates in place.\n\nThat split is what makes the optional model safe here. An invented category or a made-up date in a notice like this is exactly the error that loses an appeal.\n\n```\nai: {\n  enabled: true,\n  provider: 'gemini',\n  extraInstructions: 'Sign off as the Beispiel.de moderation team.',\n}\n```\n\nSame rule as the AI check: **no ai option means no model is contacted.** The\ntemplates then write both fields, in German or English, and that is a complete\nnotice — blunter, not incomplete.\n\nWhen only a word list matched, no model graded anything. Writing \"severity: none\" into the notice would read as a considered finding that the content was fine, which is not what happened — so the severity line is simply absent from the rendered text, and the template assessment says what was actually established:\n\nDie zitierte Stelle entspricht einem Begriff aus der Wortliste der geprüften Sprachen. Damit steht fest, welche Wörter gefallen sind — nicht, was der Satz mit ihnen tut. Eine solche Feststellung wird hier auch nicht behauptet.\n\nOverstating there would mean inventing grounds. The same reticence applies to\ncategories and confidence: those lines are omitted rather than printed as\n`(none)`\n\nand `0%`\n\n.\n\nArt. 17 wants the statement on a durable medium — retrievable next month, not a\ntoast that disappears. `JustificationStore`\n\nis a two-method interface for that,\nand `InMemoryJustificationStore`\n\nimplements it for demos:\n\n```\ninterface JustificationStore {\n  save(id: string, justification: ComplianceJustification): Promise<void>;\n  get(id: string): Promise<ComplianceJustification | null>;\n  list?(): Promise<string[]>;\n}\n```\n\n**The in-memory one is not a production store** — it is a `Map`\n\n, and a restart\nerases every notice you owe. Implement the interface against your database.\n`exportJustification`\n\ngives you the JSON to put in a column.\n\n`examples/server`\n\nshows the round trip: `POST /api/justifications`\n\nmoderates,\ngenerates and stores, returning an id; `GET /api/justifications/:id`\n\nis the\ndurable link you put in the notice.\n\nA failed model call must not stop a legal notification from going out.\n`generateJustification`\n\ncatches everything: on a refusal, a timeout or\nunparseable JSON it returns the template wording. There is no code path where\nyou get no notice at all.\n\n**Not legal advice, and not compliance in a box.** It produces the artefact; whether your process around it satisfies the DSA is your assessment to make, with your own counsel.The library cannot know whether a person looked at the case. If one did, set it on the object before you store it — and note that Art. 17 is`humanReview`\n\nis hardcoded`false`\n\n.*why*you would want to: a purely automated restriction has to say so.**Nothing distinguishes \"illegal content\" from \"incompatible with your terms\".** Art. 17(3) treats those as different grounds with different consequences. The module has one`policyBases`\n\nlist, so that distinction is yours to encode —`{ name: 'Legal', section: '§ 130 StGB' }`\n\nversus your house rules.**No complaint handling (Art. 20), no transparency database submission (Art. 24(5)).** Those are systems, not strings. This module writes the notice that both of them start from.\n\n`ts-profanity-filter/pii`\n\nfinds e-mail addresses, phone numbers, IBANs, payment\ncards, IP addresses and German tax ids — as spans, like everything else here, so\nthe redaction stays yours to render.\n\n``` js\nimport { detectPii, piiToSegments } from 'ts-profanity-filter/pii';\n\ndetectPii('Meine IBAN ist DE44 5001 0517 5407 3249 31, Tel. 030 12345678');\n// [\n//   { kind: 'iban',  text: 'DE44 5001 0517 5407 3249 31', start: 15, end: 42,\n//     confidence: 0.99, evidence: ['structure', 'checksum', 'context'] },\n//   { kind: 'phone', text: '030 12345678', start: 49, end: 61,\n//     confidence: 0.99, evidence: ['structure', 'context'] },\n// ]\n```\n\n| Kind | What confirms it |\n|---|---|\n`email` |\nstructure — local part, labels, TLD, all length-checked |\n`iban` |\nISO 13616 mod-97, plus the country's published length |\n`card` |\nLuhn, plus an issuer prefix that owns that length |\n`taxid-de` |\nISO 7064 MOD 11,10, plus the BZSt repetition rule |\n`ip` |\noctet ranges; IPv6 group counting, compression and embedded IPv4 |\n`phone` |\nE.164 length, a trunk zero, or a word nearby that says so |\n\n**Names, postal addresses and dates of birth are missing on purpose.** They are\npersonal data too, but nothing inside the string can confirm them, and a\ndetector that guesses turns every capitalised word into a finding. Everything on\nthat list either verifies arithmetically or is honest about leaning on context.\n\nFour stages, and each exists because the obvious alternative is worse:\n\n**One O(n) scan** records the only three things worth anchoring on:`@`\n\npositions, digit runs, colons. Six regexes over the text would walk it six times and still miss the grouped spellings.**Digit clusters are built once.**`030 12 34 56`\n\nbecomes one object with its groups and separators intact. A phone number, a card and a tax id are the same object at this stage; telling them apart is not the scanner's job.**Recognizers score, they do not vote.** A checksum is a fact, a shape is an argument, punctuation like`+`\n\nis a hint, and a nearby word only adjusts what the string already said.**Overlaps are resolved optimally** by weighted interval scheduling — confidence × length,`O(m log m)`\n\n. This is not decoration:`::ffff:192.168.1.1`\n\nis an IPv6 address containing an IPv4 one, and a grouped IBAN contains digit runs that pass Luhn. Resolving left to right greedily picks the earliest candidate, which is not the same as the best one.\n\nA verified IBAN sits at `0.98`\n\nbecause the arithmetic says so. A bare\nten-digit number sits at `0.3`\n\nuntil a word next to it agrees, and then at\n`0.65`\n\n. The threshold is `0.6`\n\n, so:\n\n```\ndetectPii('Die Zahl 1701234567 steht hier');   // []\ndetectPii('Telefon 1701234567');                // phone, 0.65\ndetectPii('1.2.3.4');                           // [] — that is a version number\ndetectPii('8.8.8.8');                           // ip, 0.8 — no version repeats one component\n```\n\nLower `minConfidence`\n\nto audit what is being suppressed rather than wondering.\nWhat a nearby word is worth also differs per kind: `Telefon`\n\nin front of ten\ndigits is nearly the whole case, while `IBAN:`\n\nin front of a string that already\npasses mod-97 adds almost nothing.\n\n| Function | Returns |\n|---|---|\n`detectPii(text, options?)` |\n`PiiMatch[]` — non-overlapping, in reading order |\n`hasPii(text, options?)` |\n`boolean` |\n`piiToSegments(text, options?)` |\n`PiiSegment[]` , same shape as the filter's — concatenates back to the input exactly |\n`isValidIban` · `isValidLuhn` · `isValidGermanTaxId` · `iso7064Mod1110` |\nthe checksums on their own |\n\n| Option | Type | Default | Description |\n|---|---|---|---|\n`kinds` |\n`PiiKind[] | '*'` |\n`'*'` |\nNarrow the search. An unknown kind throws. |\n`minConfidence` |\n`number` |\n`0.6` |\nFindings below this are dropped. |\n`contextWindow` |\n`number` |\n`48` |\nHow many characters either side count as context. |\n\nBecause segments come back in the same shape as the profanity filter's, one renderer handles both:\n\n``` js\n{piiToSegments(comment).map((seg, i) =>\n  seg.isPii ? <span key={i} className=\"redacted\" title={seg.kind}>{seg.text}</span> : seg.text,\n)}\n```\n\nThe cross-check keeps `class`\n\n, `Klassik`\n\nand `Scunthorpe`\n\nout of the results, and\nthe built-in allowlist was written against English and German as they are\ngenerally used — **not against your domain**. A medical corpus, a Dutch user\nbase, a supplier called Assmann: each finds holes the shipped lists never saw.\n\n`ts-profanity-filter/allowlist`\n\nfinds them in your text and proposes entries to\nclose them.\n\n``` js\nimport { tuneAllowlist, formatAllowEntries } from 'ts-profanity-filter/allowlist';\n\nconst report = await tuneAllowlist(() => ndjsonTexts('comments.ndjson'), {\n  languages: ['en', 'de'],\n  ai: { provider: 'gemini' },   // optional — see below\n});\n\nreport.before;    // 412 records were flagged\nreport.after;     //  37 still are\nreport.entries;   // ['assmann\\\\p{L}*', 'diagnos\\\\p{L}*', …]\nconsole.log(formatAllowEntries(report, 'en-house'));\n```\n\n**Scan**— read the corpus, collect the*whole words*the lists flagged, count them. Entirely local. Frequency is the first signal: an ordinary word appears everywhere, a slur appears rarely.**Judge**— ordinary word or genuine insult? A model can answer, or you can through`verdicts`\n\n, or both. Without an`ai`\n\noption**no model is contacted** and only your own verdicts are used.**Verify**— compile each proposal and*test*it.\n\n**Stage 3 is the whole point.** A model asked to fix `Klassiker`\n\nmay answer\n`\\p{L}*ass\\p{L}*`\n\n, which technically clears it and destroys the filter for the\nentire language. No amount of prompting reliably prevents that; compiling the\nsuggestion and measuring it does. Every proposed entry must\n\n- clear the word it was written for — otherwise\n`why: 'no-effect'`\n\n; **not** clear any word judged offensive — otherwise`why: 'too-broad'`\n\n, naming the word it would have cleared;- be a valid pattern under the\n`u`\n\nflag — otherwise`why: 'invalid'`\n\n.\n\nNothing is dropped silently: rejected proposals come back in `report.rejected`\n\nwith the reason.\n\nOnce to find what gets flagged, once to prove the accepted entries helped. So pass\nan **array or a factory**, not a spent iterator:\n\n``` js\ntuneAllowlist(() => linesFrom('comments.txt'), { … });   // before/after measured\ntuneAllowlist(oneShotGenerator, { … });                  // rerun: false, after unmeasured\n```\n\nThe model may say it cannot tell, and that verdict is never acted on. The same goes for a word you did not judge at all: no verdict, no entry.\n\n| Option | Type | Default | Description |\n|---|---|---|---|\n`languages` |\n`Language[] | '*'` |\n`['en']` |\nWhich lists to tune against. |\n`minCount` |\n`number` |\n`1` |\nIgnore words appearing fewer times than this. |\n`limit` |\n`number` |\n`50` |\nJudge at most this many distinct words, most frequent first. |\n`sampleLimit` |\n`number` |\n`3` |\nRecords kept per word as evidence. |\n`verdicts` |\n`Record<string, WordVerdict>` |\n— | Judgements by hand. Never sent to a model. |\n`ai` |\n`AiOptions` |\nabsent | Absent means no model is contacted. |\n\n`findFlaggedWords()`\n\nis exported on its own if all you want is the scan.\n\n**A generated entry is a proposal, not a policy.** It has been proved not to\nbreak the corpus you gave it, which is a real guarantee and a narrow one — a\nnative speaker should still read the list before it ships.\n\nAnalysing one comment is a function call. Analysing two million is a different problem, and the difference is not speed — it is that the obvious version holds the whole corpus in memory, dies at row 900 000 with nothing written, and makes one paid model call per row.\n\n``` js\nimport { runBatch, formatSummary } from 'ts-profanity-filter/batch';\nimport { ndjsonFrom } from 'ts-profanity-filter/batch/node';\n\nconst summary = await runBatch(ndjsonFrom('comments.ndjson'), {\n  filter: { languages: ['en', 'de'] },\n  pii: true,\n  onResult: (result) => { if (result.flagged) hold(result.id); },\n});\n\nconsole.log(formatSummary(summary));\n```\n\nNothing is buffered: the file is read a chunk at a time, results are handed to\n`onResult`\n\nand dropped, and peak memory is one record regardless of file size.\n\n`runBatch(source, options)` |\nruns to completion, returns the `BatchSummary` , hands each result to `onResult` . For large input. |\n`streamBatch(source, options)` |\nan `AsyncGenerator` yielding each result. Its return value is the summary, which `for await` discards — drive `.next()` yourself if you want both. |\n\nThe source is anything iterable: an array, a generator, a database cursor, one of\nthe Node readers below. A record is a `string`\n\nor `{ text, id }`\n\n, and the `id`\n\ncomes back on the result so you can join to your own data.\n\n```\nai: {\n  provider: 'gemini',\n  when: 'matched',   // only records a word list already hit — the default\n  maxCalls: 500,     // a hard ceiling for the whole run\n  retries: 2,        // exponential backoff on failure\n}\n```\n\n`when: 'matched'`\n\nis the cheap and usually correct reading: it is the quotation\ncheck on the records the lists flagged. ** when: 'unmatched' is the expensive\none** — most records in any real corpus are clean, so it sends nearly all of\nthem.\n\n`'all'`\n\nsends everything, and a predicate lets you decide per record.`maxCalls`\n\nexists because a batch is exactly where one call per row becomes a\nbill. Once it is reached the run continues locally and the summary says\n`aiBudgetExhausted: true`\n\n, so a truncated run can never read as a complete one.\n\nA refusal is not retried — the provider's safety layer declining is a decision, not a transient fault, and asking again in 500 ms will not change its mind.\n\nEvery stage is wrapped per record. A pattern that throws, a hostile input, a\nprovider that is down: the result carries `error: { stage, message }`\n\n, the other\nstages still ran, and the run continues. `signal`\n\nstops it early and the summary\ncomes back with `aborted: true`\n\nrather than throwing away the work already done.\n\n`ts-profanity-filter/batch/node`\n\nis a **separate subpath** so that importing the\nrunner itself never drags a Node API into a browser bundle.\n\n| Function | For |\n|---|---|\n`ndjsonFrom(path, { textField, idField, onBadLine })` |\none JSON object per line |\n`csvFrom(path, { column, idColumn, delimiter, header })` |\none column of a CSV |\n`csvRowsFrom(path, delimiter)` |\nraw rows |\n`linesFrom(path)` |\none text per line |\n`recordsFrom(path)` |\npicks the reader from the extension |\n`createNdjsonWriter(path)` |\nappend results, respecting backpressure |\n\nThe CSV reader is a character-level state machine, not `split('\\n')`\n\n— a quoted\nfield may contain the delimiter, a newline or an escaped `\"\"`\n\n, and splitting on\nlines first makes the embedded-newline case unrecoverable rather than merely\nwrong.\n\n```\nnpx ts-profanity-filter scan comments.ndjson --languages en,de --pii \\\n  --out flagged.ndjson --pdf report.pdf\nRecords processed           5\n  Flagged                     3 (60.0%)\n  Matched a word list         1\n  Records with personal data  2 (3 findings)\n  Duration                    50 ms · 100/s\n```\n\nProgress goes to stderr and the summary to stdout, so the command composes.\n`--fail-on-findings`\n\nexits 1 for CI, `--json`\n\nprints the summary as JSON, and\n`--max-calls`\n\ndefaults to **100** when `--ai`\n\nis used — a command that can spend\nmoney should not spend an unbounded amount of it by default.\n\n** The full CLI guide →** — every format, every flag, exit codes,\nrecipes for CI and nightly runs, and what to check when a scan comes back empty.\n\n[is 25 chat messages with a documented expected verdict for every row, so you can point the command at something real before pointing it at your own data.](/Kevinci/ts-profanity-filter/blob/main/examples/batch/chat-log.csv)\n\n`examples/batch/chat-log.csv`\n\n`renderSummaryPdf(summary)`\n\nreturns the bytes of a report — the same rows as the\ntext version, from the same function, so the two can never disagree.\n\n``` js\nimport { renderSummaryPdf } from 'ts-profanity-filter/batch';\nawait writeFile('report.pdf', await renderSummaryPdf(summary, { title: 'Nightly scan' }));\n```\n\nIt needs [ fast-pdf](https://www.npmjs.com/package/fast-pdf), and\n\n**this does not change the dependency count.** It is an\n\n*optional*peer dependency reached through a dynamic import in that one function, exactly as the Anthropic SDK is:\n\n`dependencies`\n\nstays empty, an install that never renders a PDF pulls nothing,\nand the call throws with an install hint if the package is absent. Pass\n`deterministic: true`\n\nfor byte-identical output — useful for hashing or\narchiving a report.| Option | Type | Default | Description |\n|---|---|---|---|\n`filter` |\n`FilterOptions | false` |\n`{}` |\nWord lists, or `false` to skip them. |\n`pii` |\n`PiiOptions | true | false` |\noff | Off unless asked, like the AI check. |\n`ai` |\n`BatchAiOptions` |\nabsent | Absent means no model is contacted. |\n`segments` |\n`boolean` |\n`false` |\nInclude the segments per record — the one part of a result whose size grows with the text. |\n`concurrency` |\n`number` |\n`8` |\nIn-flight records. Only matters with a model. |\n`ordered` |\n`boolean` |\n`true` |\n`false` is faster when durations vary: nothing waits behind a slow neighbour. |\n`onProgress` |\n`(p) => void` |\n— | Called every `progressEvery` records, and once at the end. |\n`progressEvery` |\n`number` |\n`500` |\nPer-record callbacks are their own cost at this scale. |\n`signal` |\n`AbortSignal` |\n— | Stops pulling; the summary reports `aborted` . |\n`sampleLimit` |\n`number` |\n`20` |\nFlagged records kept in the summary. A summary must not grow with the input. |\n\nEach adapter is a separate subpath import, so nothing you do not use reaches\nyour bundle. `react`\n\nand `vue`\n\nare **optional** peer dependencies.\n\n``` js\nimport { useProfanitySegments, useIsProfane } from 'ts-profanity-filter/react';\n\nfunction Comment({ body }: { body: string }) {\n  const segments = useProfanitySegments(body, { languages: ['en', 'de'] });\n  return (\n    <p>\n      {segments.map((seg, i) =>\n        seg.isProfane ? <mark key={i}>{seg.text}</mark> : <span key={i}>{seg.text}</span>,\n      )}\n    </p>\n  );\n}\n```\n\nMemoised by the **value** of the options, not their identity — an inline object\nliteral will not re-run the filter on every render.\n\n``` js\n<script setup lang=\"ts\">\nimport { ref } from 'vue';\nimport { useProfanitySegments } from 'ts-profanity-filter/vue';\n\nconst body = ref('');\nconst segments = useProfanitySegments(body, { languages: ['en', 'de'] });\n</script>\n```\n\nBoth arguments accept a plain value, a ref, or a getter; you get back a computed\nref. Needs Vue 3.3+ (for `toValue`\n\n).\n\nA class carrying a real `@Pipe()`\n\ndecorator has to be compiled by Angular's own\ncompiler, and a plain `tsc`\n\nbuild cannot produce that — an AOT build in your app\nwould reject it. So this package ships the **logic without decorators**, and you\nadd the decorator in your app where `ngtsc`\n\ncompiles it properly:\n\n``` js\n// profanity-segments.pipe.ts — the whole file\nimport { Pipe } from '@angular/core';\nimport { ProfanitySegmentsPipeBase } from 'ts-profanity-filter/angular';\n\n@Pipe({ name: 'profanitySegments', standalone: true })\nexport class ProfanitySegmentsPipe extends ProfanitySegmentsPipeBase {}\njs\n<span *ngFor=\"let seg of body | profanitySegments:{ languages: ['en','de'] }\"\n      [class.redacted]=\"seg.isProfane\">{{ seg.text }}</span>\n```\n\nThe base class caches by value, which matters because a pure pipe re-runs whenever a template's object literal gets a new identity.\n\nThere is also a plain service class (no `@Injectable()`\n\n, so provide it\nexplicitly):\n\n``` js\nimport { ProfanityFilter } from 'ts-profanity-filter/angular';\n\nproviders: [\n  { provide: ProfanityFilter, useFactory: () => new ProfanityFilter({ languages: ['en', 'de'] }) },\n]\n\nfilter.mask('Du Trottel!'); // 'Du *******!'\n```\n\nBecause it imports nothing from `@angular/core`\n\n, Angular is not a peer\ndependency of this package at all.\n\n[https://kevinci.github.io/ts-profanity-filter/](https://kevinci.github.io/ts-profanity-filter/)\n\nBilingual UI (English/German), live segmentation, three render modes, and a table showing exactly which false positives the cross-check suppressed and which allowlist rule cleared them.\n\nThe AI panel runs the check from the page itself: pick a provider and model,\npaste your **own** key, and see both signals for the same sentence — what the\nword lists caught, and what the model made of it. The key is never stored and\ngoes only to the provider. That is a playground pattern, not a production one;\nsee *Keep the key server-side* above.\n\nThe page is a single self-contained file generated from the compiled `dist/`\n\n,\nso it always runs the same code npm ships. To work on it locally:\n\n```\nnpm run demo        # tsc, then regenerate docs/index.html\nopen docs/index.html\n```\n\nEdit `demo/template.html`\n\n, never `docs/index.html`\n\n— the latter is generated.\nGitHub Pages serves it straight from `main`\n\n, so a push updates the live page.\n\nThe core needs Unicode property escapes (`\\p{L}`\n\n) — Chrome 64, Firefox 78,\nSafari 11.1, Node 10. That is a hard floor: they are used throughout, and\nnothing can degrade without them.\n\nLookbehind is the one construct beyond that, and it is optional. It powers the\nspaced-out detection (`D r e c k s a u`\n\n) and the whole-word anchors — `\\b`\n\nis\nuseless for those, because it is defined in terms of `\\w`\n\nand stays ASCII even\nunder the `u`\n\nflag, so every umlaut and every `ß`\n\nreads as a word boundary and\n`Straußschwanz`\n\ncomes back flagged. Engines shipped lookbehind late — Safari\nonly from 16.4 — so it is compiled with `new RegExp`\n\ninside a `try`\n\n, never\nwritten as a literal:\n\n| Safari 16.4+, Chrome 62+, Firefox 78+, Node 18+ | older engines | |\n|---|---|---|\n| word lists, cross-check, leet, lookalikes, repetition, zero-width, NFKC folding | ✅ | ✅ |\nspaced-out words (`D r e c k s a u` ) |\n✅ | not detected |\n| whole-word anchors | Unicode boundaries | fall back to ASCII `\\b` |\n\nThe distinction matters more than it looks. A regex literal is compiled when\nthe *script* is parsed, so one the engine cannot handle is a syntax error for\nthe whole module — importing the package would fail outright rather than\nlosing one feature. A test asserts that no shipped file contains such a\nliteral.\n\n**React Native / Hermes is untested.** Hermes has had gaps in Unicode regex\nsupport; verify before shipping.\n\n**ESM only.** `require()`\n\nworks on Node 20.19+ / 22.12+; on Node 18 a\nCommonJS caller cannot load it.\n\n**The AI check belongs on a server.** A key shipped to a browser is readable\nand spendable by every visitor.\n\n**German compounding** forces permissive allow entries like`\\p{L}*klass\\p{L}*`\n\n. A contrived word containing both a slur and an allowed stem comes out clean. Allowed always beats blocked.With the German list active it is allowlisted, which necessarily clears the English word too. Use`dick`\n\nis German for \"thick\".`languages: ['en']`\n\nfor English-only text.A custom`customList`\n\nreplaces the patterns but not the allowlist.`ass`\n\npattern still loses against an allowlisted`Klassik`\n\n. Combine with`crossCheck: false`\n\nif you want nothing suppressed., letters and all, so a pattern carrying its own syntax breaks:`aggressive`\n\nrewrites the regex source`[abc]`\n\nbecomes`[[a@4]b[c(k<]]`\n\nand`(?<word>…)`\n\nbecomes`(?<w[o0]rd>…)`\n\n. Both are rejected at registration. Write patterns as plain words, or turn`aggressive`\n\noff for hand-written regexes.**Word lists are a starting point, not a policy.** Extend them for your domain.**The AI check costs money or quota, and adds latency.** It is one network round trip per call. Run the local filter first and reach for the model only when it matters — or use`ai: false`\n\non the cheap path.**A model verdict is a second opinion, not ground truth.** It has a`confidence`\n\nfor a reason. Treat a flag as a reason to hold a message for review, not as a conviction, and keep a human in the loop for anything consequential.**A failed or refused check reports** Absence of a verdict is not a clean bill of health — branch on`flagged: false`\n\n.`ai.status`\n\n, do not read`ai.flagged`\n\nalone.**The Gemini provider disables Google's own safety filtering.** It has to: the text a moderation classifier must read is the text those filters refuse to look at. Safe here because the output is a verdict, never generated content.**A batch is one thread with bounded concurrency, not a thread pool.** That is the right answer for the model path, where the network waits rather than the CPU. For millions of records through the word lists alone, the ceiling is one core — shard the input across processes if that is not enough.and to 100 in the CLI. A batch with a model and no ceiling is the easiest way to spend real money by accident; set it explicitly.`maxCalls`\n\ndefaults to unlimited in the library**PII detection finds only what a string can prove.** No names, no postal addresses, no dates of birth — see[Only what can be verified](#only-what-can-be-verified). A phone number without a`+`\n\n, a trunk zero or a word next to it stays below the threshold, which means bare digit runs in prose are missed on purpose.It is equally a version number. A bare one appears once`1.2.3.4`\n\nis not reported as an IP address.`minConfidence`\n\ndrops below 0.5 — but with the word*version*in front of it the candidate is**discarded**, not scored low, so no threshold will surface that one.** The Art. 17 module writes the notice, not the process.**It does not store anything durably on its own, does not know whether a human reviewed the case, and draws no line between illegal content and a breach of your terms. See[What this is not](#what-this-is-not).\n\n``` php\nnpm run build         # tsc -> dist/ (.js, .d.ts, .d.ts.map, .js.map)\nnpm test              # builds, then runs node --test\nnpm run typecheck     # type-checks src + test\nnpm run lint          # eslint, type-aware\nnpm run lint:fix      # the same, applying what it can fix\nnpm run format        # prettier --write\nnpm run format:check  # prettier --check, for CI\nnpm run demo          # builds and regenerates docs/index.html\n```\n\n`prepublishOnly`\n\nruns all of them, so a release cannot ship code that fails its\nown checks.\n\n**Prettier leaves three things alone**, on purpose: the Markdown, because the\nprose is hand-wrapped and it would rewrite every `*word*`\n\nto `_word_`\n\nfor no\ngain; the playground template, because it is one hand-built page rather than a\nmodule; and a handful of `// prettier-ignore`\n\ntables — the IBAN lengths, the PII\ncontext words, the 81-attack corpus — which are data laid out to be read across,\nnot down. The corpus alone would go from 226 lines to 711.\n\nThe warranty disclaimer and the limitation of liability are in there, in the\nlast two paragraphs: the software is provided *as is*, without warranty of any\nkind, and the authors are not liable for claims or damages arising from its\nuse. That is the whole of what is offered and the whole of what is disclaimed —\nthere is no separate agreement anywhere.\n\nTwo things it does not do, because no wording can. It cannot exclude liability\nfor intent or gross negligence, or for injury to life, body or health, where\nmandatory law says otherwise. And it does not decide anything about *your*\nobligations to *your* users — see\n*What leaves your machine, and whose problem it is* above.", "url": "https://wpnews.pro/news/building-a-generic-context-aware-scoring-engine", "canonical_source": "https://github.com/Kevinci/ts-profanity-filter", "published_at": "2026-08-11 12:56:45+00:00", "updated_at": "2026-08-11 13:13:20.539480+00:00", "lang": "en", "topics": ["developer-tools", "natural-language-processing", "ai-tools"], "entities": ["ts-profanity-filter", "Kevinci", "React", "Vue", "Angular", "Gemini", "DSA Art. 17"], "alternates": {"html": "https://wpnews.pro/news/building-a-generic-context-aware-scoring-engine", "markdown": "https://wpnews.pro/news/building-a-generic-context-aware-scoring-engine.md", "text": "https://wpnews.pro/news/building-a-generic-context-aware-scoring-engine.txt", "jsonld": "https://wpnews.pro/news/building-a-generic-context-aware-scoring-engine.jsonld"}}