Member-only story
Every trace restates the patient’s note — in extractions, tool calls, and reasoning. #
I added observability to my local clinical AI agent, opened the trace viewer, and stared at a patient’s full name, date of birth, and phone number sitting in plain text in a database I’d just spun up.
With synthetic patients, that’s a shrug. Wire this into a real clinical pipeline and your observability backend quietly becomes a PHI database nobody threat-modeled. The standard fix is “redact your traces before export.” So I did: I wired in Presidio, Microsoft’s open-source PII detection library, built a ground-truth set of 752 PHI entities across 40 synthetic notes, and measured what actually leaked.
The masking pipeline never failed once. The detection underneath it failed 135 times. Refining the PHI detection got it down to 17 but that was not the most interesting finding of the experiment.
During the detection evaluation, I realized that redacting the note is not redacting the trace. The same patient name gets…