Open PII models for Hebrew: English-first filters miss most first names A developer posting as yinonov released Seter, an MIT-licensed PII model for Hebrew, Arabic and English, after finding that openai/privacy-filter caught only 14% and GLiNER 33% of standalone Hebrew first names on a synthetic test set. Seter reaches 88% recall on the same set, with lenient char-F1 of 0.904 on Hebrew, 0.917 on Arabic and 0.878 on English, and runs fully in the browser via ONNX so text never leaves the machine. In follow-up probes the developer reported that titles before names were all caught (6 of 6) but that Israeli case numbers such as 45127-09-23 ת״א and 8954/11 ע״א are mislabeled as account numbers, and a 9-digit file number is treated the same as a Teudat Zehut. yinonov https://discuss.huggingface.co/u/yinonov 1 English-first PII filters miss most Hebrew first names. I send Hebrew legal material to LLMs, so redaction has to work before anything leaves the building. Off-the-shelf tools did not hold up. On a synthetic set of standalone Hebrew first names, openai/privacy-filter caught 14% and GLiNER 33%. Names that also function as ordinary words שחר, אור, נועה are where they fail most. I trained Seter, an open MIT PII model for Hebrew, Arabic and English. It catches 88% on the same set, with lenient char-F1 of 0.904 on Hebrew, 0.917 on Arabic and 0.878 on English, and it runs fully in the browser via ONNX. Demo: Seter — Hebrew/Arabic PII redaction - a Hugging Face Space by yinonov https://huggingface.co/spaces/yinonov/seter-demo Model: shoresh-labs/seter-pii-he-ar · Hugging Face https://huggingface.co/shoresh-labs/seter-pii-he-ar Failure cases from real-world Hebrew text are the most useful thing you could send back. DHEIVER https://discuss.huggingface.co/u/DHEIVER 2 @yinonov https://discuss.huggingface.co/u/yinonov The drop in performance on names that double as common nouns shachar, or, noa usually happens because standard NER models rely on subword tokenization. In Hebrew, a single word might split into tokens where the entity boundary is lost, or the model sees a generic token with high prior probability for a common noun, suppressing the entity label. The 88% recall on the synthetic set suggests the model likely shifted focus to character-level features or used a different tokenization strategy that preserved those boundaries better than generic subword approaches. The high char-F1 confirms that handling diacritics and script-specific morphology is the critical factor here. Running via ONNX in the browser is a smart architectural choice for legal workflows: it guarantees the PII never traverses a network, eliminating the transmission risk entirely. yinonov https://discuss.huggingface.co/u/yinonov 3 Thanks. One correction: Seter does not use character-level features or a special tokenizer. Both models are ordinary subword token classifiers XLM-R large, and an mmBERT-base model distilled from it , and nothing in them handles diacritics specially. The difference is the training data. It is synthetic and built around the cases that fail: first names on their own, names that are also ordinary words, attached prefixes ל/ו/ש , titles and kunyas, plus negatives such as hospital names and UUIDs. Subword boundaries do show up, but as a smaller problem: a prefix letter attached to a name sometimes ends up inside the span. More data did not fix that. dbabel https://discuss.huggingface.co/u/dbabel 4 Since you asked for failure cases, a few patterns from Hebrew legal text worth adding to the eval set. Rulings that already pseudonymize parties as פלוני / אלמונית those shouldn’t count as misses , titles directly before the name like עו"ד or כב’ השופט, and Teudat Zehut numbers, which are 9 digits with a check digit you can validate to tell them apart from case or file numbers. Text that went through OCR is a different distribution from typed text: final letters ם/ס, ן/ו get confused and line breaks split names, so recall on scanned PDFs can come in well below the synthetic-set number. Mixed-script lines, with an English or transliterated name inside a Hebrew sentence, are another case worth measuring separately. yinonov https://discuss.huggingface.co/u/yinonov 5 Thanks, these were worth testing. I ran the fast model on a small hand written probe 25 sentences, so indicative only : - Titles before the name עו״ד, כב׳ השופט, רו״ח : all 6 names caught, including ones that are ordinary words. - Case numbers: this is a real weakness. 45127-09-23 ת״א and 8954/11 ע״א are tagged as account numbers, and a 9 digit file number is treated the same as a Teudat Zehut. The check-digit test is a good idea for a step after the model. mostly tagged as a person. Harmless for redaction, but you are right that they should not count either אלמונית / פלוני - way in scoring. - OCR-like text: final-letter swaps and hyphenated line breaks held up; a name split mid-word by a line break without a hyphen was missed. I have not measured real scans yet. - Latin-script names inside Hebrew: fine, except a bare “Or”. Case-number negatives and a scanned-text set are going on the list. dbabel https://discuss.huggingface.co/u/dbabel 6 Useful results, thanks for sharing them. For the name split by a line break without a hyphen, a cheap fix is a pre-pass that joins lines before the model runs: if a line ends mid-word no sentence punctuation, and the joined token is in your name lexicon or the model scores it higher joined , merge it, then map the spans back to the original offsets so redaction still lands on both fragments. For the scanned-text set, one way to separate the OCR effect from the model effect is to render your existing typed probe sentences to page images, degrade them at a few levels blur, low DPI, skew, JPEG , OCR them, and run the model on the OCR output. Because you already know the gold spans, the recall drop at each level is attributable to OCR alone.