# Show HN: AgentLeak – open-source privacy testing for multi-agent LLM systems

> Source: <https://github.com/Privatris/AgentLeak>
> Published: 2026-11-13 01:36:36+00:00

Benchmark for privacy leakage in multi-agent LLM systems.

This repository accompanies the IEEE Access paper: *AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems*.

Preprint (v3): [https://arxiv.org/abs/2602.11510v3](https://arxiv.org/abs/2602.11510v3)

IEEE Access (2026): [https://doi.org/10.1109/ACCESS.2026.3704541](https://doi.org/10.1109/ACCESS.2026.3704541)

For the installable privacy-testing tool, see **[yagobski/agentleak](https://github.com/yagobski/agentleak)** and **[agentleak.org](https://www.agentleak.org/?utm_source=research_repo&utm_medium=readme&utm_campaign=oss_visibility_202609)**.

The practical tool provides a Python SDK, CLI, local web UI and CI checks for sensitive data disclosed through tool calls, shared memory, inter-agent messages, logs, generated files and final outputs. Core trace analysis runs locally without an LLM dependency; optional live-agent runs use the endpoint you configure.

Try a bundled synthetic example with Python 3.10+:

```
pip install agentleak
agentleak run --scenario healthcare_patient_summary
```

The example illustrates a clean final answer alongside internal-channel disclosures. It is a demonstration, not a production leakage-rate estimate. Follow the [tool quickstart](https://github.com/yagobski/agentleak/blob/main/docs/quickstart.md) to analyze your own traces.

This repository remains the research benchmark and reproduction code. Its experiments, channel definitions and detection pipeline should be cited separately from the practical tool's deterministic analysis and bundled examples.

The [paper (v3)](https://arxiv.org/abs/2602.11510v3) evaluates **1,000 scenarios** across healthcare, finance, legal, and corporate domains, with **4,979 validated execution traces** across five models. AgentLeak instruments seven channels: **C1** final outputs, **C2** inter-agent messages, **C3** tool inputs, **C4** tool outputs, **C5** shared memory, **C6** logs, and **C7** artifacts. The large-scale analysis covers **C1/C2/C5**; **C3/C6** are evaluated on a sample; **C4/C7** are examined in an integration study. These channels therefore do not all have the same empirical coverage. The validated trace count is also recorded in [`paper_stats.json`](https://github.com/Privatris/AgentLeak/blob/main/benchmarks/ieee_repro/results/paper_stats.json).

The table and findings below are preserved from the existing README as a separate repository summary. Their stated 5,694-trace basis and reported rates have **not been reconciled** with the paper's 4,979 validated traces. Do not interpret them as the paper's validated results or combine the two denominators; consult the paper for its reported results and evaluation scope.

| Model | C1 (Output) | C2 (Internal) | H1 (Audit Gap) | Total Leak | 
|---|---|---|---|---|
| **Claude-3.5-Sonnet** | 8.2% | 53.9% | 45.7% | 55.2% | 
| GPT-4o | 17.2% | 76.8% | 59.6% | 77.6% | 
| GPT-4o-mini | 41.2% | 75.3% | 34.2% | 76.3% | 
| Llama-3.3-70B | 26.9% | 67.8% | 41.3% | 89.9% | 
| Mistral-Large | 47.5% | 96.2% | 48.7% | 99.3% | 
| **Average** | **28.2%** | **74.0%** | **45.9%** | **79.7%** | 

- **Internal channels leak 2.6× more** than external (74.0% vs 28.2%)
- **Output-only audits miss 45.9%** of violations
- **Claude 3.5 Sonnet paradox** : Lowest C1 leakage (8.2%) but 6.6× internal/external ratio—the highest among all models
- **Finding 7 (Tool Leakage)** : Tool inputs (C3) and system logs (C6) exhibit extremely high leakage rates (up to**85%** on Claude 3.5), even when the final agent output (C1) is perfectly sanitized.
- Pattern C2 > C1 holds **across all 5 models** tested

- 1,000 scenarios (healthcare, finance, legal, corporate)
- 7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts
- 32 attack classes, 6 families
- SDK: CrewAI, LangChain, AutoGPT, MetaGPT

To reproduce the main results (Output, Internal, Memory):

```
cd benchmarks/ieee_repro
python benchmark.py --n 1000 --traces --model openai/gpt-4o
```

Targets "Secondary Channel" leakage where sensitive data is sent to external tools or dumped in logs.

```
cd benchmarks/ieee_repro
# Run for a specific model (e.g., Claude 3.5)
python benchmark_tools.py --n 100 --model anthropic/claude-3.5-sonnet

# Or run the automated multi-model test suite
./run_tools_benchmark.sh
```

Results are saved in `benchmarks/ieee_repro/results/tools/`.

- `agentleak/` : The core framework SDK
- `agentleak_data/` : The dataset of 1000 scenarios
- `benchmarks/ieee_repro/` : Scripts to reproduce the paper's findings, including Finding 7 (Tools & Logs).
- `benchmarks/showcase/` : Real-world CrewAI integration demo showing the SDK in action.
- `paper/` : The LaTeX source of the IEEE Access paper

```
git clone https://github.com/Privatris/AgentLeak
cd AgentLeak
pip install -e .
pytest tests/ -v
python
from agentleak import AgentLeakTester, DetectionMode

tester = AgentLeakTester(mode=DetectionMode.HYBRID)
result = tester.check(
    vault={"ssn": "123-45-6789"},
    output="The SSN is 123-45-6789",
    channel="C1"
)
print(f"Leak: {result.leaked}, Confidence: {result.confidence}")
```

CLI:

```
python -m agentleak run --quick --dry-run
python -m agentleak run --full
cd benchmarks/ieee_repro
python benchmark.py --n 100 --traces --model openai/gpt-4o-mini
```

Traces are in `benchmarks/ieee_repro/results/traces/`.

```
@article{el2026agentleak,
  title   = {AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems},
  author  = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa},
  journal = {IEEE Access},
  year    = {2026},
  doi     = {10.1109/ACCESS.2026.3704541},
  url     = {https://doi.org/10.1109/ACCESS.2026.3704541}
}
```

MIT