{"slug": "open-pii-models-for-hebrew-english-first-filters-miss-most-first-names", "title": "Open PII models for Hebrew: English-first filters miss most first names", "summary": "A developer posting as yinonov released Seter, an MIT-licensed PII model for Hebrew, Arabic and English, after finding that openai/privacy-filter caught only 14% and GLiNER 33% of standalone Hebrew first names on a synthetic test set. Seter reaches 88% recall on the same set, with lenient char-F1 of 0.904 on Hebrew, 0.917 on Arabic and 0.878 on English, and runs fully in the browser via ONNX so text never leaves the machine. In follow-up probes the developer reported that titles before names were all caught (6 of 6) but that Israeli case numbers such as 45127-09-23 ת״א and 8954/11 ע״א are mislabeled as account numbers, and a 9-digit file number is treated the same as a Teudat Zehut.", "body_md": "[yinonov](https://discuss.huggingface.co/u/yinonov)\n1\n \nEnglish-first PII filters miss most Hebrew first names.\n\nI send Hebrew legal material to LLMs, so redaction has to work before anything leaves the building. Off-the-shelf tools did not hold up. On a synthetic set of standalone Hebrew first names, openai/privacy-filter caught 14% and GLiNER 33%. Names that also function as ordinary words (שחר, אור, נועה) are where they fail most.\n\nI trained Seter, an open (MIT) PII model for Hebrew, Arabic and English. It catches 88% on the same set, with lenient char-F1 of 0.904 on Hebrew, 0.917 on Arabic and 0.878 on English, and it runs fully in the browser via ONNX.\n\nDemo: [Seter — Hebrew/Arabic PII redaction - a Hugging Face Space by yinonov](https://huggingface.co/spaces/yinonov/seter-demo)\n\nModel: [shoresh-labs/seter-pii-he-ar · Hugging Face](https://huggingface.co/shoresh-labs/seter-pii-he-ar)\n\nFailure cases from real-world Hebrew text are the most useful thing you could send back.\n\n \n \n[DHEIVER](https://discuss.huggingface.co/u/DHEIVER)\n2\n \n[@yinonov](https://discuss.huggingface.co/u/yinonov) The drop in performance on names that double as common nouns (shachar, or, noa) usually happens because standard NER models rely on subword tokenization. In Hebrew, a single word might split into tokens where the entity boundary is lost, or the model sees a generic token with high prior probability for a common noun, suppressing the entity label. The 88% recall on the synthetic set suggests the model likely shifted focus to character-level features or used a different tokenization strategy that preserved those boundaries better than generic subword approaches. The high char-F1 confirms that handling diacritics and script-specific morphology is the critical factor here. Running via ONNX in the browser is a smart architectural choice for legal workflows: it guarantees the PII never traverses a network, eliminating the transmission risk entirely.\n\n \n \n[yinonov](https://discuss.huggingface.co/u/yinonov)\n3\n \nThanks. One correction: Seter does not use character-level features or a special tokenizer. Both models are ordinary subword token classifiers (XLM-R large, and an mmBERT-base model distilled from it), and nothing in them handles diacritics specially.\n\nThe difference is the training data. It is synthetic and built around the cases that fail: first names on their own, names that are also ordinary words, attached prefixes (ל/ו/ש), titles and kunyas, plus negatives such as hospital names and UUIDs.\n\nSubword boundaries do show up, but as a smaller problem: a prefix letter attached to a name sometimes ends up inside the span. More data did not fix that.\n\n \n \n[dbabel](https://discuss.huggingface.co/u/dbabel)\n4\n \nSince you asked for failure cases, a few patterns from Hebrew legal text worth adding to the eval set. Rulings that already pseudonymize parties as פלוני / אלמונית (those shouldn’t count as misses), titles directly before the name like עו\"ד or כב’ השופט, and Teudat Zehut numbers, which are 9 digits with a check digit you can validate to tell them apart from case or file numbers.\n\nText that went through OCR is a different distribution from typed text: final letters (ם/ס, ן/ו) get confused and line breaks split names, so recall on scanned PDFs can come in well below the synthetic-set number. Mixed-script lines, with an English or transliterated name inside a Hebrew sentence, are another case worth measuring separately.\n\n \n \n[yinonov](https://discuss.huggingface.co/u/yinonov)\n5\n \nThanks, these were worth testing.\n\nI ran the fast model on a small hand written probe (25 sentences, so indicative only):\n\n- Titles before the name (עו״ד, כב׳ השופט, רו״ח)): all 6 names caught, including ones that are ordinary words.\n- Case numbers: this is a real weakness. 45127-09-23 ת״א and 8954/11 ע״א are tagged as account numbers, and a 9 digit file number is treated the same as a Teudat Zehut. The check-digit test is a good idea for a step after the model. mostly tagged as a person. Harmless for redaction, but you are right that they should not count either אלמונית / פלוני - way in scoring.\n- OCR-like text: final-letter swaps and hyphenated line breaks held up; a name split mid-word by a line break without a hyphen was missed. I have not measured real scans yet.\n- Latin-script names inside Hebrew: fine, except a bare “Or”.\n\nCase-number negatives and a scanned-text set are going on the list.\n\n \n \n[dbabel](https://discuss.huggingface.co/u/dbabel)\n6\n \nUseful results, thanks for sharing them. For the name split by a line break without a hyphen, a cheap fix is a pre-pass that joins lines before the model runs: if a line ends mid-word (no sentence punctuation, and the joined token is in your name lexicon or the model scores it higher joined), merge it, then map the spans back to the original offsets so redaction still lands on both fragments. For the scanned-text set, one way to separate the OCR effect from the model effect is to render your existing typed probe sentences to page images, degrade them at a few levels (blur, low DPI, skew, JPEG), OCR them, and run the model on the OCR output. Because you already know the gold spans, the recall drop at each level is attributable to OCR alone.", "url": "https://wpnews.pro/news/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names", "canonical_source": "https://discuss.huggingface.co/t/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names/191772#post_6", "published_at": "2026-11-18 05:46:10+00:00", "updated_at": "2026-11-18 06:11:55.126557+00:00", "lang": "en", "topics": ["natural-language-processing", "ai-safety", "ai-tools", "machine-learning"], "entities": ["Seter", "yinonov", "shoresh-labs", "Hugging Face", "openai/privacy-filter", "GLiNER", "XLM-R", "mmBERT"], "also_reported_by": [], "takeaway": "Seter, an MIT-licensed ONNX model for Hebrew, Arabic and English PII, reaches 88% recall on standalone Hebrew first names versus 14% for openai/privacy-filter and 33% for GLiNER, but still mislabels Israeli case numbers like 45127-09-23 ת״א as account numbers, so add a check-digit validation step af", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names", "markdown": "https://wpnews.pro/news/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names.md", "text": "https://wpnews.pro/news/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names.txt", "jsonld": "https://wpnews.pro/news/open-pii-models-for-hebrew-english-first-filters-miss-most-first-names.jsonld"}}