Benchmark for privacy leakage in multi-agent LLM systems.
This repository accompanies the IEEE Access paper: AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems.
Preprint (v3): https://arxiv.org/abs/2602.11510v3
IEEE Access (2026): https://doi.org/10.1109/ACCESS.2026.3704541
For the installable privacy-testing tool, see yagobski/agentleak and agentleak.org.
The practical tool provides a Python SDK, CLI, local web UI and CI checks for sensitive data disclosed through tool calls, shared memory, inter-agent messages, logs, generated files and final outputs. Core trace analysis runs locally without an LLM dependency; optional live-agent runs use the endpoint you configure.
Try a bundled synthetic example with Python 3.10+:
pip install agentleak
agentleak run --scenario healthcare_patient_summary
The example illustrates a clean final answer alongside internal-channel disclosures. It is a demonstration, not a production leakage-rate estimate. Follow the tool quickstart to analyze your own traces.
This repository remains the research benchmark and reproduction code. Its experiments, channel definitions and detection pipeline should be cited separately from the practical tool's deterministic analysis and bundled examples.
The paper (v3) evaluates 1,000 scenarios across healthcare, finance, legal, and corporate domains, with 4,979 validated execution traces across five models. AgentLeak instruments seven channels: C1 final outputs, C2 inter-agent messages, C3 tool inputs, C4 tool outputs, C5 shared memory, C6 logs, and C7 artifacts. The large-scale analysis covers C1/C2/C5; C3/C6 are evaluated on a sample; C4/C7 are examined in an integration study. These channels therefore do not all have the same empirical coverage. The validated trace count is also recorded in paper_stats.json.
The table and findings below are preserved from the existing README as a separate repository summary. Their stated 5,694-trace basis and reported rates have not been reconciled with the paper's 4,979 validated traces. Do not interpret them as the paper's validated results or combine the two denominators; consult the paper for its reported results and evaluation scope.
| Model | C1 (Output) | C2 (Internal) | H1 (Audit Gap) | Total Leak |
|---|---|---|---|---|
| Claude-3.5-Sonnet | 8.2% | 53.9% | 45.7% | 55.2% |
| GPT-4o | 17.2% | 76.8% | 59.6% | 77.6% |
| GPT-4o-mini | 41.2% | 75.3% | 34.2% | 76.3% |
| Llama-3.3-70B | 26.9% | 67.8% | 41.3% | 89.9% |
| Mistral-Large | 47.5% | 96.2% | 48.7% | 99.3% |
| Average | 28.2% | 74.0% | 45.9% | 79.7% |
-
Internal channels leak 2.6× more than external (74.0% vs 28.2%)
-
Output-only audits miss 45.9% of violations
-
Claude 3.5 Sonnet paradox : Lowest C1 leakage (8.2%) but 6.6× internal/external ratio—the highest among all models
-
Finding 7 (Tool Leakage) : Tool inputs (C3) and system logs (C6) exhibit extremely high leakage rates (up to85% on Claude 3.5), even when the final agent output (C1) is perfectly sanitized.
-
Pattern C2 > C1 holds across all 5 models tested
-
1,000 scenarios (healthcare, finance, legal, corporate)
-
7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts
-
32 attack classes, 6 families
-
SDK: CrewAI, LangChain, AutoGPT, MetaGPT
To reproduce the main results (Output, Internal, Memory):
cd benchmarks/ieee_repro
python benchmark.py --n 1000 --traces --model openai/gpt-4o
Targets "Secondary Channel" leakage where sensitive data is sent to external tools or dumped in logs.
cd benchmarks/ieee_repro
python benchmark_tools.py --n 100 --model anthropic/claude-3.5-sonnet
./run_tools_benchmark.sh
Results are saved in benchmarks/ieee_repro/results/tools/.
agentleak/: The core framework SDKagentleak_data/: The dataset of 1000 scenariosbenchmarks/ieee_repro/: Scripts to reproduce the paper's findings, including Finding 7 (Tools & Logs).benchmarks/showcase/: Real-world CrewAI integration demo showing the SDK in action.paper/: The LaTeX source of the IEEE Access paper
git clone https://github.com/Privatris/AgentLeak
cd AgentLeak
pip install -e .
pytest tests/ -v
python
from agentleak import AgentLeakTester, DetectionMode
tester = AgentLeakTester(mode=DetectionMode.HYBRID)
result = tester.check(
vault={"ssn": "123-45-6789"},
output="The SSN is 123-45-6789",
channel="C1"
)
print(f"Leak: {result.leaked}, Confidence: {result.confidence}")
CLI:
python -m agentleak run --quick --dry-run
python -m agentleak run --full
cd benchmarks/ieee_repro
python benchmark.py --n 100 --traces --model openai/gpt-4o-mini
Traces are in benchmarks/ieee_repro/results/traces/.
@article{el2026agentleak,
title = {AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems},
author = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa},
journal = {IEEE Access},
year = {2026},
doi = {10.1109/ACCESS.2026.3704541},
url = {https://doi.org/10.1109/ACCESS.2026.3704541}
}
MIT