# Compiled open-source servers with injected backdoors: BinaryAudit: testing whether AI agents can find backdoors in compiled binaries

> Source: <https://reversedwithai.com/p/binaryaudit-backdoor-detection-benchmark>
> Published: 2026-11-10 08:56:59+00:00

## Evidence at a glance

Based on the linked sources. The author's reported result and independent confirmation are separate checks.

- AI involvement
- Reported models: Claude Opus 4.6, Gemini 3 Pro, GPT-5.2. Tools: Ghidra, Radare2, Harbor, Docker. The writeup below describes the work attributed to AI.
- Reported result
- Research. This is the directory's source-based outcome classification.
- Independent confirmation
- Not reported.
- Human work and limitations
- Quesma staff (mostly Piotr Grabowski and Piotr Migdal in the commit log) wrote the tasks and harness by hand. The benchmark measures AI agents; it is not a record of one agent finishing a single reverse engineering job. Results and trajectories are published on the Quesma site.

### Original sources

## What was done

Quesma built tasks from five open-source programs by injecting backdoors, compiling stripped binaries and asking AI agents to decide whether each is tampered. Its blog reports Claude Opus 4.6 found 49 percent of the backdoors.

BinaryAudit is a benchmark that checks whether AI agents can find backdoors in compiled software. The authors took five open-source programs (lighttpd, dnsmasq, Dropbear, Sozu and Caddy), injected backdoors by hand, built stripped binaries, and put each into a Docker task where an agent gets Ghidra and Radare2 but no source code. The agent must say whether the binary was modified.

The injected backdoors range, per the README, from remote code execution through hidden HTTP headers to timebombs that fire after a given date. Some tasks add obfuscation, some use syscalls or process spawning, and some mix several binaries or several architectures. There are also negative tasks with clean binaries, so false positives are measured too. The README warns that every backdoor is artificial and says nothing about real vulnerabilities in these projects.

Tasks run through the Harbor framework, by default with the terminus-2 agent via OpenRouter. The README's examples compare Claude Opus 4.6, Gemini 3 Pro and GPT-5.2, and the linked blog post reports that Opus 4.6 found 49 percent of the planted backdoors. Commit trailers name Claude Opus 4.6 on 8 commits and Cursor on 42.

## Reproduce it

Install Harbor with uv, build the shared Docker image from docker/base.Dockerfile, then run for example harbor run --path tasks --task-name lighttpd-backdoor-detect with your own model API key.

## Community

Ask a question, share what worked, or follow the next milestone.

[Your library](https://reversedwithai.com/library)