{"slug": "our-pii-model-comparison-reversed-when-we-changed-the-dataset", "title": "Our PII model comparison reversed when we changed the dataset", "summary": "A comparison of two personal-data detection models by an unnamed company reversed when the evaluation dataset changed, showing that benchmark choice can flip conclusions. On the company's own 1,800-example corpus, its existing GLiNER-based setup leaked 0.04% versus 3.28% for OpenAI's Privacy Filter, but on the PII-Masking-300k corpus Privacy Filter outperformed by roughly three to one, and on Gretel's synthetic PII corpus Privacy Filter was worse (26.6% vs 23.5%). The company nearly published the misleading result and warns that using a model's reference benchmark biases the comparison.", "body_md": "# Our PII model comparison reversed when we changed the dataset\n\nWe build a tool that finds personal data in text and masks it before it reaches a model. In April, OpenAI released [Privacy Filter](https://huggingface.co/openai/privacy-filter), an open-weight model that does the detection half of that job. We already run [GLiNER](https://huggingface.co/onnx-community/gliner_multi_pii-v1) for the same purpose, so we measured one against the other.\n\nThe first answer we got was clear and wrong. This is how it went wrong, because the mistake is easy to make and we came within about a day of publishing it.\n\n## The setup\n\nTwo token-level detectors, same harness, same machine, same metric.\n\nThe metric matters, so: we score **surface leak**. Run the whole pipeline end to end, then ask whether each thing that should have been masked is still sitting in the output as a literal substring. Not span F1, not per-token accuracy. The reason is that our pipeline merges overlapping detections into one span, and span-level scoring undercounts that: when a merged span covers two adjacent expected spans it can only be credited with one, so the other reads as a miss even though the text is fully masked. Span scoring told us one configuration was leaking 2.9% on a corpus where nothing leaked at all. We tune on span F1 because it is more sensitive; we report surface leak because it is what a user could check.\n\n## The first result\n\nOur own evaluation corpus is 1,800 examples across nine languages, generated by our own builder and validated by our own oracle. On it, our existing setup leaks 0.04% and swapping in Privacy Filter made things worse, 3.28%, concentrated in Catalan and Spanish person names.\n\nThat is a home-field result and we knew it, so we needed a corpus we had not written. We picked the obvious one: **PII-Masking-300k**, the standard reference in this space.\n\nOn that corpus the ordering flipped hard. Privacy Filter alone beat our existing setup by roughly three to one. Same code, same harness, same metric, opposite conclusion.\n\nWe drafted that up. A model from a large lab, beating what we ship, on a neutral dataset. It is a good story and it was nearly a public one.\n\n## Why it was wrong\n\nPII-Masking-300k is the corpus OpenAI reports Privacy Filter against.\n\nThat is not an accusation of anything. Nobody outside OpenAI can say whether it is in the training data, and we are not claiming it is. The point is narrower and it does not require contamination: **a corpus that a model's authors selected as their reference is a corpus that model was steered toward**, through architecture choices, label taxonomy, threshold calibration and everything else that gets tuned while looking at a benchmark. It was Privacy Filter's home field in precisely the way our corpus is ours.\n\nWe had swapped one biased corpus for another and called the second one neutral.\n\nSo we picked a third: [Gretel's synthetic PII corpus](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual). Nine hundred contracts and statements across six languages, averaging about 1.2k characters, chosen by neither party.\n\nOn that corpus, Privacy Filter alone is **worse** than what we already ran: 26.6% against 23.5%.\n\nThe three-to-one win did not survive the dataset change. Nothing else changed.\n\n## The licence problem, which we should have hit first\n\nThere is a second reason not to use PII-Masking-300k, and it is the one we should have checked before running anything.\n\nIts licence grants access \"exclusively for academic research and non-commercial purposes,\" requires written permission for redistribution or derivative works, and states that no licence is available to companies without prior discussion. We are a commercial product. Benchmark figures derived from it, published on a company blog, are not ours to publish.\n\nThat is why this post describes the reversal in prose and gives you exact numbers only from the Apache-2.0 corpus. It is an awkward shape for an article and it is the correct one.\n\nIf you are evaluating PII models, check this before you build an argument on a dataset. The permissively licensed options we found usable were the Gretel corpus above and [beki/privy](https://huggingface.co/datasets/beki/privy) under MIT.\n\n## What our own corpus was hiding\n\nThe uncomfortable part is not what the third corpus said about Privacy Filter. It is what it said about our benchmark.\n\nOn our corpus, every configuration we ship scores between 0.0% and 1.1% leak. That reads like a strong result. It is closer to a measurement failure: the corpus cannot distinguish our configurations from each other. On the Gretel corpus the same three score 51.2%, 23.5% and 15.1%.\n\nBoth are real numbers. The first describes clean text in languages we have specifically tuned for, the second describes document formats nobody tuned for. But only one of them can tell you whether a change you just made was an improvement, and it is not the one we had been using alone.\n\n## Two bugs that needed a second model to surface\n\nRunning two detectors over the same corpora exposed problems one detector had been hiding.\n\n**A policy interaction that leaked every date of birth.** Privacy Filter has no date-of-birth category; it reports birth dates as a generic private date. Our policy keeps generic dates by default, because masking every date destroys a contract for no privacy gain. Compose those two correct behaviours and every date of birth passes through in plaintext. The model finds 99.7% of them and the policy waves them through. Neither component is wrong on its own. We only caught it because the new corpus labels dates of birth as a distinct type and ours does not.\n\n**Form headings masked as people.** \"T.C. Kimlik\", \"Vergi\", \"IBAN\", \"Póliza\": the label printed next to an identifier rather than the identifier. Twelve distinct surfaces accounted for 84% of one model's over-masking. Fixing it improved our existing shipped tier too, whose over-masking fell from 1.04% to 0.31% with no change in leak. That fix had been available the whole time and needed a second model to become visible.\n\n## Where the two models actually differ\n\nThey fail in different places, which is the only interesting result here.\n\nOn source files both leak nothing on our code corpus, but Privacy Filter is **3.4 times faster** on real files: 562 ms against 1,919 ms at the median, on 47 files averaging 5 KB.\n\nThe mechanism is chunking. GLiNER is a zero-shot span model: it takes a label list as a prompt and re-runs a forward pass per chunk. Privacy Filter uses banded attention with a 128-token band and reads the whole file in one pass.\n\n**The obvious objection is that we handicapped GLiNER with a small chunk size, and it is worth answering directly.** Our chunk is 1,000 characters. That is not a tuning oversight, it is the model's context window: GLiNER here has a 384-token limit, and 1,000 characters of multilingual text plus a roughly 40-token label prompt already sits close to it. Raising the chunk size truncates the input. The re-prompting cost is structural to how the model works, not a configuration we neglected.\n\nThe ordering also reverses by file size. On 300-character fragments GLiNER is the faster of the two, 85 ms against 105 ms. Privacy Filter only wins once files are big enough for chunking overhead to dominate. Any latency number in this comparison is meaningless without the document size attached, which is a thing we would have gotten wrong if we had measured only one size.\n\n## What we shipped\n\nNeither model replaced the other. Detection now routes by what is being read: source files to Privacy Filter, prose to GLiNER. Running both on everything has the lowest leak we measured and costs 4.4 GB resident, so it exists and is not the default.\n\nNumbers, methodology and the corpus build script are on our [benchmark page](/benchmarks). The Gretel corpus is Apache-2.0, so the figures in this post are reproducible without asking us for anything.\n\n## Limitations\n\nOne machine, Apple silicon, twelve cores, single run, no variance bands. Latency numbers in particular would benefit from repetition and we have not done it.\n\nThe comparison is between two specific models at specific quantizations in one pipeline, with our policy layer and our regex tier underneath both. It is not a general claim about either model.\n\nAnd the finding that matters generalizes past all of that: **if you benchmark a model on the dataset its authors report it against, you have measured its home field.** Pick a third corpus. Read the licence first.", "url": "https://wpnews.pro/news/our-pii-model-comparison-reversed-when-we-changed-the-dataset", "canonical_source": "https://velumprivacy.com/blog/pii-benchmark-reversed-changing-dataset", "published_at": "2026-08-04 10:47:23+00:00", "updated_at": "2026-08-04 11:23:29.853654+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-ethics"], "entities": ["OpenAI", "Privacy Filter", "GLiNER", "PII-Masking-300k", "Gretel"], "alternates": {"html": "https://wpnews.pro/news/our-pii-model-comparison-reversed-when-we-changed-the-dataset", "markdown": "https://wpnews.pro/news/our-pii-model-comparison-reversed-when-we-changed-the-dataset.md", "text": "https://wpnews.pro/news/our-pii-model-comparison-reversed-when-we-changed-the-dataset.txt", "jsonld": "https://wpnews.pro/news/our-pii-model-comparison-reversed-when-we-changed-the-dataset.jsonld"}}