PrivacyBench: An open benchmark for de-identifying text that scores synthesis Tonic AI released PrivacyBench, an open benchmark for de-identifying semi-structured text from work tools like Slack and email. The benchmark introduces novel metrics for evaluating synthesis quality and found that Tonic Textual paired with Opus 4.8 outperformed LLMs in PII detection and replacement coherence. Datasets: /datasets The dataset viewer is not available for this dataset. Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer https://huggingface.co/docs/hub/datasets-data-files-configuration , and open a discussion /datasets/TonicAI/Privacy-Bench/discussions/new?title=Dataset+Viewer+issue&description=The+dataset+viewer+is+not+working.%0A%0Acc+%40lhoestq+%40cfahlgren1. for direct support. PrivacyBench PrivacyBench is a benchmark for de-identifying semi-structured data exports from work tools like email, messaging, calendar, and so forth. The benchmark focuses on the identification and synthesis of PII in the unstructured text fields of the data export, and introduces novel metrics for evaluating synthesis quality. This initial version consists of data exports of Slack and email messages from 21 distinct personas generated by the Fabricate https://fabricate.tonic.ai/ synthetic data tool using a seed list of characters. Ground truth labels are created using said seed list of characters, so the benchmark requires no human annotators. Across every configuration we tested, using Tonic Textual for detection improved PII recall over an LLM, and the strongest pipeline overall paired Tonic Textual with Opus 4.8. De-identifying unstructured text is difficult in two separate directions: detection of sensitive entities and coherent replacement of detected entities. Both of these directions are necessary to have private and high utility synthetic data. Detecting sensitive entities covers the privacy aspect of the synthetic data, and is done by named entity recognition NER . This is well studied with many benchmarks out there, CoNLL https://huggingface.co/datasets/eriktks/conll2003 , Ontonotes https://catalog.ldc.upenn.edu/LDC2013T19 , and TAB https://github.com/NorskRegnesentral/text-anonymization-benchmark to state a few. Choosing coherent replacements of the detected entities determines the utility of the synthetic data, and no benchmarks exist that measure the accuracy of chosen synthetic replacements. This is where PrivacyBench comes in, as it measure both the privacy and the utility of synthetic data by examining what entities are detected and what their replacements are. Consistent synthesis of replacement PII is difficult, because it requires us to consistently treat different textual representations of a character's identity. Consider a simple example with one character, Joseph Ferrara who often goes by Joe and has two email addresses jferrara@gmail.com mailto:jferrara@gmail.com and joe@tonic.ai mailto:joe@tonic.ai . These different textual representations may occur within the same document or across many different documents, and the challenge is to synthesize replacement PII for all instances coherently so that the synthetic PII all align with a single synthetic character. When there are many documents of different types with a lot of cross references, character PII linking and synthesis across documents becomes very difficult. You already see this with just email and slack data, as many different types of cross references happen in different email threads and slack channels. Measuring synthesis quality is difficult due to the abundance of possible good synthetic replacement possibilities making the creation of ground truth labels difficult. PrivacyBench addresses this by cleverly creating character PII ground truth labels as part of the synthetic data generation process, and then using a simple LLM-as-a-judge to measure synthesis quality for each character in the ground truth. Rather than a single correct replacement, the ground truth records which spans belong to each character, and the judge checks whether the pipeline's replacements for those spans form one coherent synthetic identity instead of matching a fixed value. PrivacyBench uses this process to define a new metric called synthesis accuracy. As a POC, PrivacyBench is evaluated on six synthesis pipelines consisting of NER done by either an LLM or Tonic Textual https://textual.tonic.ai/ , and then synthesis done by an LLM. The LLMs used are the three Anthropic models Opus 4.8, Sonnet 4.6, and Haiku 4.5. Using Tonic Textual improves NER performance over an LLM in all cases. Tonic Textual improves NER performance in three directions: accuracy, speed, and cost. Combining Tonic Textual NER with LLM synthesis shows that using efficient NER models such as Tonic Textual allows for more accurate and more cost effective synthesis than using just LLMs. The code for the metrics is on GitHub: TonicAI/privacy bench metrics https://github.com/TonicAI/privacy bench metrics . Dataset overview 21 slack and email data exports : one per fictional protagonist, each set in a different industry pharma, retail, finance, airlines, tech, manufacturing, … . 17,917 messages total. On average about 25,000 words, 115 emails, 740 Slack messages, and 24 characters per data export. 5 PII entity types : NAME GIVEN , NAME FAMILY , EMAIL ADDRESS , USERNAME , ORGANIZATION . Gold ground truth for every message . The ground truth is stored as PII spans and a character roster. Each PII span is an instance of one of the above entity types appearing in a message and is associated to one of the characters in the data. The character roster states each of the characters and their associated PII. Two-stage task : NER detect PII + synthesis replace coherently , scored by three metrics see below .- All data is fully synthetic . Metrics The evaluation reports three headline scores and a per-entity-type breakdown of each : NER recall : the fraction of ground truth PII spans that the synthesizer detected. recall = TP / TP + FN . A recall miss is a privacy leak real PII left in the output . Synthesis accuracy : of the ground truth spans that were detected, the fraction whose synthetic value is coherent, as determined by an LLM-as-a-judge. The LLM-as-a-judge gets one call per character, evaluating each synthetic replacement against the character's inferred synthetic identity. synthesis accuracy = coherent / coherent + incoherent . Synthesis + NER accuracy : the overall synthesis pipeline score. This score folds detection misses into the denominator of synthesis accuracy. It scores the overall pipeline, not distinguishing synthesis misses or NER misses. synthesis + NER accuracy = coherent / coherent + incoherent + missed . The denominator of synthesis + NER accuracy is the same as the denominator of NER recall, it is the number of ground truth spans. The below image is a simple example of how a synthesis pipeline works and how the metrics are calculated on it. Baseline results As a POC PrivacyBench is evaluated on six synthesis pipelines. Each pipeline consists of - NER done either by Tonic Textual or by an LLM annotating the input text inline with detected entities. - Synthetic replacements chosen by prompting an LLM with input text, detected entities, and previous replacement choices. The three Anthropic model tiers were used for NER and synthesis: Opus 4.8, Sonnet 4.6, and Haiku 4.5. PrivacyBench successfully stratifies the Anthropic models by capability Opus Sonnet Haiku across the defined metrics, and using Tonic Textual always improves NER performance. The scores in this table are the macro-average across all 21 datasets each dataset weighted equally , ordered by the bottom-line Synthesis + NER accuracy metric. | NER model | Synthesis model | NER recall | Synthesis accuracy | Synthesis + NER accuracy | |---|---|---|---|---| | Textual | Opus 4.8 | 95.0% | 97.0% | 92.0% | | Opus 4.8 | Opus 4.8 | 88.8% | 99.0% | 87.7% | | Textual | Sonnet 4.6 | 95.0% | 90.5% | 85.9% | | Textual | Haiku 4.5 | 95.0% | 87.3% | 82.8% | | Sonnet 4.6 | Sonnet 4.6 | 83.8% | 90.5% | 76.2% | | Haiku 4.5 | Haiku 4.5 | 79.9% | 88.5% | 71.1% | The strongest pipeline pairs Tonic Textual with Opus 4.8, reaching 92.0% Synthesis + NER accuracy. This is 4.3 percentage points higher than Opus 4.8 doing both steps on its own 87.7% . The difference comes from NER detection where Textual finds 95% of the PII, 6.2 points higher than the 88.8% achieved by Opus. Textual holds 95.0% NER recall no matter which LLM performs synthesis, decoupling NER quality from the synthesis model. This raises the Synthesis + NER accuracy of cheaper synthesis models as Textual + Sonnet 85.9% lands within 1.8 points of Opus alone 87.7% , and Textual + Haiku 82.8% outscores Sonnet alone 76.2% by 6.6 points. One number to point out directly, because it looks like a point in favor of the LLM-only pipeline is the 99.0% synthesis accuracy for the Opus 4.8 only pipeline. This number is higher than the 97.0% for Textual + Opus. Synthesis accuracy is measured only over the entities a pipeline actually detects. Opus alone detects fewer entities, and the ones it misses tend to be the harder ones, so it is effectively synthesizing an easier subset and scoring itself on that. Textual detects more of the difficult entities and is then judged on that harder set, which is why its synthesis accuracy is fractionally lower even though its output is both more private and more complete. The overall metrics, Synthesis + NER accuracy, puts detection misses back into Synthesis accuracy, and on that measure Textual + Opus is the clear leader. Tonic Textual provides better detection at a fraction of the cost. It isn't just more accurate at NER than an LLM, it is dramatically cheaper. Swapping the LLM detection step for Tonic Textual cuts the total cost of the pipeline NER and synthesis by more than 60% Textual + Opus 4.8 vs just Opus 4.8 . The scatter plot below plots Synthesis + NER accuracy against total cost for each pipeline. Textual-based pipelines sit up and to the left showing higher accuracy and lower cost. Layout tasks/ one