Show HN: AgentLeak – open-source privacy testing for multi-agent LLM systems

The AgentLeak research benchmark, published in IEEE Access (2026) and accompanied by arXiv preprint v3, evaluated 1,000 scenarios across healthcare, finance, legal and corporate domains over 4,979 validated execution traces from five models, finding that internal channels leak at a 74.0% rate versus 28.2% for final outputs. The benchmark instruments seven channels (C1 final outputs, C2 inter-agent messages, C3 tool inputs, C4 tool outputs, C5 shared memory, C6 logs, C7 artifacts), with large-scale analysis covering only C1/C2/C5, and reports that output-only audits miss 45.9% of violations. A separate installable tool, agentleak, provides a Python SDK, CLI, local web UI and CI checks for detecting sensitive data disclosed through tool calls, shared memory, inter-agent messages, logs and generated files, with core trace analysis running locally without an LLM dependency.

Benchmark for privacy leakage in multi-agent LLM systems. This repository accompanies the IEEE Access paper: AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems . Preprint v3 : https://arxiv.org/abs/2602.11510v3 https://arxiv.org/abs/2602.11510v3 IEEE Access 2026 : https://doi.org/10.1109/ACCESS.2026.3704541 https://doi.org/10.1109/ACCESS.2026.3704541 For the installable privacy-testing tool, see yagobski/agentleak https://github.com/yagobski/agentleak and agentleak.org https://www.agentleak.org/?utm source=research repo&utm medium=readme&utm campaign=oss visibility 202609 . The practical tool provides a Python SDK, CLI, local web UI and CI checks for sensitive data disclosed through tool calls, shared memory, inter-agent messages, logs, generated files and final outputs. Core trace analysis runs locally without an LLM dependency; optional live-agent runs use the endpoint you configure. Try a bundled synthetic example with Python 3.10+: pip install agentleak agentleak run --scenario healthcare patient summary The example illustrates a clean final answer alongside internal-channel disclosures. It is a demonstration, not a production leakage-rate estimate. Follow the tool quickstart https://github.com/yagobski/agentleak/blob/main/docs/quickstart.md to analyze your own traces. This repository remains the research benchmark and reproduction code. Its experiments, channel definitions and detection pipeline should be cited separately from the practical tool's deterministic analysis and bundled examples. The paper v3 https://arxiv.org/abs/2602.11510v3 evaluates 1,000 scenarios across healthcare, finance, legal, and corporate domains, with 4,979 validated execution traces across five models. AgentLeak instruments seven channels: C1 final outputs, C2 inter-agent messages, C3 tool inputs, C4 tool outputs, C5 shared memory, C6 logs, and C7 artifacts. The large-scale analysis covers C1/C2/C5 ; C3/C6 are evaluated on a sample; C4/C7 are examined in an integration study. These channels therefore do not all have the same empirical coverage. The validated trace count is also recorded in paper stats.json https://github.com/Privatris/AgentLeak/blob/main/benchmarks/ieee repro/results/paper stats.json . The table and findings below are preserved from the existing README as a separate repository summary. Their stated 5,694-trace basis and reported rates have not been reconciled with the paper's 4,979 validated traces. Do not interpret them as the paper's validated results or combine the two denominators; consult the paper for its reported results and evaluation scope. | Model | C1 Output | C2 Internal | H1 Audit Gap | Total Leak | |---|---|---|---|---| | Claude-3.5-Sonnet | 8.2% | 53.9% | 45.7% | 55.2% | | GPT-4o | 17.2% | 76.8% | 59.6% | 77.6% | | GPT-4o-mini | 41.2% | 75.3% | 34.2% | 76.3% | | Llama-3.3-70B | 26.9% | 67.8% | 41.3% | 89.9% | | Mistral-Large | 47.5% | 96.2% | 48.7% | 99.3% | | Average | 28.2% | 74.0% | 45.9% | 79.7% | - Internal channels leak 2.6× more than external 74.0% vs 28.2% - Output-only audits miss 45.9% of violations - Claude 3.5 Sonnet paradox : Lowest C1 leakage 8.2% but 6.6× internal/external ratio—the highest among all models - Finding 7 Tool Leakage : Tool inputs C3 and system logs C6 exhibit extremely high leakage rates up to 85% on Claude 3.5 , even when the final agent output C1 is perfectly sanitized. - Pattern C2 C1 holds across all 5 models tested - 1,000 scenarios healthcare, finance, legal, corporate - 7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts - 32 attack classes, 6 families - SDK: CrewAI, LangChain, AutoGPT, MetaGPT To reproduce the main results Output, Internal, Memory : cd benchmarks/ieee repro python benchmark.py --n 1000 --traces --model openai/gpt-4o Targets "Secondary Channel" leakage where sensitive data is sent to external tools or dumped in logs. cd benchmarks/ieee repro Run for a specific model e.g., Claude 3.5 python benchmark tools.py --n 100 --model anthropic/claude-3.5-sonnet Or run the automated multi-model test suite ./run tools benchmark.sh Results are saved in benchmarks/ieee repro/results/tools/ . - agentleak/ : The core framework SDK - agentleak data/ : The dataset of 1000 scenarios - benchmarks/ieee repro/ : Scripts to reproduce the paper's findings, including Finding 7 Tools & Logs . - benchmarks/showcase/ : Real-world CrewAI integration demo showing the SDK in action. - paper/ : The LaTeX source of the IEEE Access paper git clone https://github.com/Privatris/AgentLeak cd AgentLeak pip install -e . pytest tests/ -v python from agentleak import AgentLeakTester, DetectionMode tester = AgentLeakTester mode=DetectionMode.HYBRID result = tester.check vault={"ssn": "123-45-6789"}, output="The SSN is 123-45-6789", channel="C1" print f"Leak: {result.leaked}, Confidence: {result.confidence}" CLI: python -m agentleak run --quick --dry-run python -m agentleak run --full cd benchmarks/ieee repro python benchmark.py --n 100 --traces --model openai/gpt-4o-mini Traces are in benchmarks/ieee repro/results/traces/ . @article{el2026agentleak, title = {AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems}, author = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa}, journal = {IEEE Access}, year = {2026}, doi = {10.1109/ACCESS.2026.3704541}, url = {https://doi.org/10.1109/ACCESS.2026.3704541} } MIT