The dataset viewer is not available for this dataset. #
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
PrivacyBench is a benchmark for de-identifying semi-structured data exports from work tools like email, messaging, calendar, and so forth. The benchmark focuses on the identification and synthesis of PII in the unstructured text fields of the data export, and introduces novel metrics for evaluating synthesis quality. This initial version consists of data exports of Slack and email messages from 21 distinct personas generated by the Fabricate synthetic data tool using a seed list of characters. Ground truth labels are created using said seed list of characters, so the benchmark requires no human annotators.
Across every configuration we tested, using Tonic Textual for detection improved PII recall over an LLM, and the strongest pipeline overall paired Tonic Textual with Opus 4.8.
De-identifying unstructured text is difficult in two separate directions: detection of sensitive entities and coherent replacement of detected entities. Both of these directions are necessary to have private and high utility synthetic data. Detecting sensitive entities covers the privacy aspect of the synthetic data, and is done by named entity recognition (NER). This is well studied with many benchmarks out there, CoNLL, Ontonotes, and TAB to state a few. Choosing coherent replacements of the detected entities determines the utility of the synthetic data, and no benchmarks exist that measure the accuracy of chosen synthetic replacements. This is where PrivacyBench comes in, as it measure both the privacy and the utility of synthetic data by examining what entities are detected and what their replacements are.
Consistent synthesis of replacement PII is difficult, because it requires us to consistently treat different textual representations of a character's identity. Consider a simple example with one character, Joseph Ferrara (who often goes by Joe) and has two email addresses jferrara@gmail.com and joe@tonic.ai. These different textual representations may occur within the same document or across many different documents, and the challenge is to synthesize replacement PII for all instances coherently so that the synthetic PII all align with a single synthetic character. When there are many documents of different types with a lot of cross references, character PII linking and synthesis across documents becomes very difficult. You already see this with just email and slack data, as many different types of cross references happen in different email threads and slack channels.
Measuring synthesis quality is difficult due to the abundance of possible good synthetic replacement possibilities making the creation of ground truth labels difficult. PrivacyBench addresses this by cleverly creating character PII ground truth labels as part of the synthetic data generation process, and then using a simple LLM-as-a-judge to measure synthesis quality for each character in the ground truth. Rather than a single correct replacement, the ground truth records which spans belong to each character, and the judge checks whether the pipeline's replacements for those spans form one coherent synthetic identity instead of matching a fixed value. PrivacyBench uses this process to define a new metric called synthesis accuracy.
As a POC, PrivacyBench is evaluated on six synthesis pipelines consisting of NER done by either an LLM or Tonic Textual, and then synthesis done by an LLM. The LLMs used are the three Anthropic models Opus 4.8, Sonnet 4.6, and Haiku 4.5. Using Tonic Textual improves NER performance over an LLM in all cases. Tonic Textual improves NER performance in three directions: accuracy, speed, and cost. Combining Tonic Textual NER with LLM synthesis shows that using efficient NER models such as Tonic Textual allows for more accurate and more cost effective synthesis than using just LLMs.
The code for the metrics is on GitHub: TonicAI/privacy_bench_metrics.
Dataset overview #
21 slack and email data exports: one per fictional protagonist, each set in a different industry (pharma, retail, finance, airlines, tech, manufacturing, …).17,917 messages total. On average about 25,000 words, 115 emails, 740 Slack messages, and 24 characters per data export.5 PII entity types:NAME_GIVEN
,NAME_FAMILY
,EMAIL_ADDRESS
,USERNAME
,ORGANIZATION
.Gold ground truth for every message. The ground truth is stored as PII spans and a character roster. Each PII span is an instance of one of the above entity types appearing in a message and is associated to one of the characters in the data. The character roster states each of the characters and their associated PII.Two-stage task: NER (detect PII) + synthesis (replace coherently), scored by three metrics (see below).- All data is fully synthetic.
Metrics #
The evaluation reports three headline scores (and a per-entity-type breakdown of each):
NER recall: the fraction of ground truth PII spans that the synthesizer detected.recall = TP / (TP + FN)
. A recall miss is a privacy leak (real PII left in the output).Synthesis accuracy: of the ground truth spans that were detected, the fraction whose synthetic value is coherent, as determined by an LLM-as-a-judge. The LLM-as-a-judge gets one call per character, evaluating each synthetic replacement against the character's inferred synthetic identity.synthesis accuracy = coherent / (coherent + incoherent)
.Synthesis + NER accuracy: the overall synthesis pipeline score. This score folds detection misses into the denominator of synthesis accuracy. It scores the overall pipeline, not distinguishing synthesis misses or NER misses.synthesis + NER accuracy = coherent / (coherent + incoherent + missed)
. The denominator of synthesis + NER accuracy is the same as the denominator of NER recall, it is the number of ground truth spans.
The below image is a simple example of how a synthesis pipeline works and how the metrics are calculated on it.
Baseline results #
As a POC PrivacyBench is evaluated on six synthesis pipelines. Each pipeline consists of
- NER done either by Tonic Textual or by an LLM annotating the input text inline with detected entities.
- Synthetic replacements chosen by prompting an LLM with input text, detected entities, and previous replacement choices.
The three Anthropic model tiers were used for NER and synthesis: Opus 4.8, Sonnet 4.6, and Haiku 4.5. PrivacyBench successfully stratifies the Anthropic models by capability (Opus > Sonnet > Haiku) across the defined metrics, and using Tonic Textual always improves NER performance. The scores in this table are the macro-average across all 21 datasets (each dataset weighted equally), ordered by the bottom-line Synthesis + NER accuracy metric.
| NER model | Synthesis model | NER recall | Synthesis accuracy | Synthesis + NER accuracy |
|---|---|---|---|---|
| Textual | Opus 4.8 | 95.0% | ||
| 97.0% | 92.0% | |||
| Opus 4.8 | Opus 4.8 | 88.8% | 99.0% | 87.7% |
| Textual | Sonnet 4.6 | 95.0% | ||
| 90.5% | 85.9% | |||
| Textual | Haiku 4.5 | 95.0% | ||
| 87.3% | 82.8% | |||
| Sonnet 4.6 | Sonnet 4.6 | 83.8% | 90.5% | 76.2% |
| Haiku 4.5 | Haiku 4.5 | 79.9% | 88.5% | 71.1% |
The strongest pipeline pairs Tonic Textual with Opus 4.8, reaching 92.0% Synthesis + NER accuracy. This is 4.3 percentage points higher than Opus 4.8 doing both steps on its own (87.7%). The difference comes from NER detection where Textual finds 95% of the PII, 6.2 points higher than the 88.8% achieved by Opus.
Textual holds 95.0% NER recall no matter which LLM performs synthesis, decoupling NER quality from the synthesis model. This raises the Synthesis + NER accuracy of cheaper synthesis models as Textual + Sonnet (85.9%) lands within 1.8 points of Opus alone (87.7%), and Textual + Haiku (82.8%) outscores Sonnet alone (76.2%) by 6.6 points.
One number to point out directly, because it looks like a point in favor of the LLM-only pipeline is the 99.0% synthesis accuracy for the Opus 4.8 only pipeline. This number is higher than the 97.0% for Textual + Opus. Synthesis accuracy is measured only over the entities a pipeline actually detects. Opus alone detects fewer entities, and the ones it misses tend to be the harder ones, so it is effectively synthesizing an easier subset and scoring itself on that. Textual detects more of the difficult entities and is then judged on that harder set, which is why its synthesis accuracy is fractionally lower even though its output is both more private and more complete. The overall metrics, Synthesis + NER accuracy, puts detection misses back into Synthesis accuracy, and on that measure Textual + Opus is the clear leader.
Tonic Textual provides better detection at a fraction of the cost. It isn't just more accurate at NER than an LLM, it is dramatically cheaper. Swapping the LLM detection step for Tonic Textual cuts the total cost of the pipeline (NER and synthesis) by more than 60% (Textual + Opus 4.8 vs just Opus 4.8). The scatter plot below plots Synthesis + NER accuracy against total cost for each pipeline. Textual-based pipelines sit up and to the left showing higher accuracy and lower cost.
Layout #
tasks/ one <set>_messages.jsonl per dataset — the INPUT to synthesize
ground_truth/ per dataset, two files:
<set>_ground_truth_spans.jsonl — gold PII spans
<set>_characters_ground_truth.json — the character roster
example_output/ an example synthesizer OUTPUT (xml/haiku on megan_donovan_eli_lilly)
The scoring code lives at TonicAI/privacy_bench_metrics.
<set>
is the dataset name, which is formatted as {protagonist}_{organization}
(e.g. megan_donovan_eli_lilly
, aaron_pfizer
,renee_tyson_foods
) — {protagonist}_{organization}
.
There are 21 datasets, each a distinct fictional protagonist and their coworkers in a different industry (e.g. megan_donovan_eli_lilly
, aaron_pfizer
, renee_tyson_foods
). All data is synthetic, generated by Tonic Fabricate from a seed that declares each character's PII up front — which is why ground-truth labels exist without any human annotation.
the data #
This is a file-tree dataset: each dataset's inputs, gold spans, and character roster are separate files sharing a <set>
prefix (rather than one unified table), so the most reliable way to use it is to download the files and read them directly.
from huggingface_hub import snapshot_download
local = snapshot_download(
repo_id="TonicAI/Privacy-Bench",
repo_type="dataset"
)
The per-dataset JSONL files (tasks/*_messages.jsonl
, ground_truth/*_ground_truth_spans.jsonl
) can also be loaded individually with the datasets
library; the *_characters_ground_truth.json
rosters are JSON objects (character_id → PII
), not record tables:
from datasets import load_dataset
msgs = load_dataset(
"json",
data_files="tasks/megan_donovan_eli_lilly_messages.jsonl",
split="train"
)
File schemas #
tasks/<dataset>_messages.jsonl
— input
One message per line. This is what a synthesizer consumes.
{
"meta": {
"row_id": "1768498320.332959",
"from": {"name": "...", "email": "..."} | {"user_id": "...", "name": "..."},
"to": [ ... ],
"datetime": "2026-01-15T18:12:00Z",
"thread_ts": "..." | "thread_id": "..."
},
"text": "the free-text email body or Slack message that may contain PII"
}
ground_truth/<set>_ground_truth_spans.jsonl
— gold PII spans
Same rows (joined to messages by meta.row_id
), with gold PII spans.
{
"meta": { ... same as above ... },
"text": "...",
"ground_truth_spans": [
{
"text": "<@U02CARLOS>",
"start": 46, "end": 58,
"label": "USERNAME",
"characters": ["carlos"]
}
]
}
start
/end
are character offsets into text
. label
is one of: NAME_GIVEN
, NAME_FAMILY
, EMAIL_ADDRESS
, USERNAME
, ORGANIZATION
.
characters
identifies which person (or, for organizations, which org group) the span refers to. This is what lets the evaluation check that one real identity maps to one consistent synthetic identity.
ground_truth/<set>_characters_ground_truth.json
character roster
A map of character_id → declared PII
for every character in the dataset, derived from the data generator's seed. Synthesis accuracy needs this: the LLM judge groups each character's PII (the names, emails, Slack handle, and organization the characters
field on the spans points at) and checks the synthetic replacements form a single coherent synthetic identity.
{
"carlos": {
"canonical_name": "Carlos Mendez",
"first_names": ["Carlos"], "last_names": ["Mendez"],
"nicknames": [],
"emails": [{"value": "carlos.mendez@lilly.com", "category": "professional"}],
"slack_handles": ["U02CARLOS"],
"organizations": ["Eli Lilly and Company"]
}
}
Synthesizer output (what you produce, and what example_output/
contains)
One line per message, carrying only the detected entities and their synthetic replacements, joined to the ground truth by row_id
:
{
"row_id": "1768498320.332959",
"entities": [
{
"start": 46, "end": 58,
"label": "USERNAME",
"text": "<@U02CARLOS>",
"new_text": "<@U02MIGUEL_SANTOS>"
}
]
}
start
/end
/label
/text
describe the detected PII span (offsets into the original message text).new_text
is the synthetic replacement. An entity counts as "detected and replaced" only whennew_text
differs fromtext
.
That is all the evaluation needs from a synthesizer; combined with the two ground-truth files (spans + character roster) it computes every metric.
Running the evaluation #
The scoring code lives in a separate repo, TonicAI/privacy_bench_metrics. It scores a synthesizer output file against a dataset's two ground-truth files (the gold spans and the character roster). Clone the metrics repo and point it at the dataset files you downloaded from here:
git clone git@github.com:TonicAI/privacy_bench_metrics.git
cd privacy_bench_metrics
pip install anthropic # for the LLM judge (synthesis accuracy)
DATA=/path/to/this/dataset # where you downloaded the dataset files
python -m synthesis_evaluation.run_eval \
--predictions "$DATA/example_output/megan_donovan_eli_lilly_xml_haiku_output.jsonl" \
--ground-truth "$DATA/ground_truth/megan_donovan_eli_lilly_ground_truth_spans.jsonl" \
--characters "$DATA/ground_truth/megan_donovan_eli_lilly_characters_ground_truth.json" \
--run-name megan_donovan_eli_lilly_xml_haiku
This writes synthesis_evaluation/runs/megan_donovan_eli_lilly_xml_haiku/
containing:
results.json
— the metrics as structured data,summary.md
— a human-readable report (the three headline scores + per-entity-type tables),viewer.html
— an interactive view of the missed PII and the judge's incoherent verdicts.
Requirements
**Python 3.10+.**NER recall is computed offline — pass--skip-llm-judge
(and drop--characters
) and no dependencies beyond the standard library are needed.Synthesis accuracy uses an LLM judge, so it needs the character roster (--characters
),pip install anthropic
, andANTHROPIC_API_KEY
. (The judge call costs a few cents per dataset.)
To score only detection, with no API key and no roster:
python -m synthesis_evaluation.run_eval \
--predictions "$DATA/example_output/megan_donovan_eli_lilly_xml_haiku_output.jsonl" \
--ground-truth "$DATA/ground_truth/megan_donovan_eli_lilly_ground_truth_spans.jsonl" \
--run-name megan_donovan_eli_lilly_recall_only \
--skip-llm-judge
Scoring your own synthesizer
- Read
tasks/<set>_messages.jsonl
. - For each message, detect PII spans and choose synthetic replacements.
- Write an output JSONL in the schema above (one
{row_id, entities}
per message;row_id
must matchmeta.row_id
in the ground truth). - Run the command above with
--predictions
pointing at your file.
The example output (example_output/megan_donovan_eli_lilly_xml_haiku_output.jsonl
) was produced by a reference pipeline that does NER and synthesis with Anthropic's Claude Haiku; use it to sanity-check your setup before scoring your own.
Synthetic data disclaimer #
Every dataset in PrivacyBench is entirely synthetic. The protagonists, coworkers, companies, emails, Slack handles, and message contents were generated by Tonic Fabricate and do not describe real people, organizations, or events. Any resemblance to a real person or company is coincidental. The "PII" the benchmark asks systems to detect and replace is fictional PII embedded in fictional documents; no real personal data is present.
Intended use #
PrivacyBench is intended for evaluating and benchmarking PII-synthesis systems — measuring how well a pipeline detects PII and replaces it with coherent synthetic values. The ground truth (spans + character rosters) is the answer key for that evaluation.
Citation #
If you use PrivacyBench, please cite it:
@misc{privacybench2026,
title = {PrivacyBench: A Benchmark for Synthesizing Personal Workplace Data},
author = {Tonic AI},
year = {2026},
howpublished = {Hugging Face Datasets},
url = {https://huggingface.co/datasets/TonicAI/Privacy-Bench}
}
Contact #
Built by Tonic AI. Questions and issues are welcome on the dataset's Hugging Face Community tab.
- Downloads last month
- 540