A host-file disclosure vulnerability in Harbor's benchmark regrading

A malicious benchmark task could cause Harbor, an open-source framework for evaluating agents in sandboxed environments, to copy a private file from the operator's host machine into the task's verification environment when the operator regrades a saved run, exposing files such as cloud credentials to the task author's verification code. The researchers reproduced the leak with a single-step Docker task by placing a symbolic link in the task's output directory pointing to a host file containing "hello world"; during the original run the verifier could not read the target, but regrading followed the link on the host and copied the file's contents into the new verifier. The finding matters because labs and enterprises run public benchmark tasks and verification code in their RLVR training and evaluation pipelines, making agent benchmarks a potential supply-chain attack route akin to the XZ Utils backdoor (CVE-2024-3094).

From frontier labs training the next generation of models to enterprise teams deciding which agents to deploy, more organizations are running agent benchmark tasks on their infrastructure. Labs large and small use tasks with verifiable outcomes to train models through reinforcement learning from verifiable rewards RLVR . Enterprise teams run evaluations to find the model and agent harness that work best for their use cases. Both workflows depend on executing tasks and the verification code that scores their results. These workflows draw on a shared corpus of publicly available datasets: collections of tasks, environments, and verification code that teams can download and run. This lets labs and enterprises build on the community’s work instead of creating every task themselves. It also brings someone else’s code into their training and evaluation pipelines. As those workflows scale, so does the opportunity for a malicious task to reach infrastructure with valuable access. That makes agent benchmarks an interesting target. Get malicious code into something people willingly run, and you have the beginnings of a supply-chain attack. The XZ Utils backdoor https://openssf.org/blog/2024/03/30/xz-backdoor-cve-2024-3094/ got us thinking about this. An attacker had used an upstream software dependency to target SSH access. Could a public benchmark offer a similar route into the organizations running it? Suppose an attacker could contribute a task to a public benchmark. They would control the environment and the code that checks the agent’s work. What could they reach beyond the sandbox? Where the sandbox ends We explored that question in Harbor https://www.harborframework.com/ , a framework for evaluating agents in sandboxed environments. A task includes instructions, an environment for the agent to work in, and verification code that scores the result. Through Harbor Hub https://hub.harborframework.com/ , people can publish and share tasks, datasets, and recorded runs. The sandbox is supposed to contain the task’s code. A benchmark author’s code should not be able to read private files on the machine managing the evaluation. But the control plane runs outside the task’s sandbox. It starts environments, collects outputs, and moves files between them, handling files created by untrusted code with access to the host machine. A benchmark task that could steal credentials We found that a malicious task could cause Harbor to copy a private file from the operator’s machine into the task’s verification environment. The task author could then read that file through their verification code. If the file contained cloud credentials, those credentials would be exposed. The leak occurred when the operator regraded a saved run, applying updated verification tests to existing outputs without running the agent again. To test this safely, we used a host file containing “hello world” as a stand-in for credentials. The file did not exist inside the task’s sandbox. The task created a symbolic link in its output directory. A symbolic link stores a path to another file. Ours pointed to the path of the host file. Harbor collected the task’s outputs and saved the link. During the original run, the verification code tried to read it and failed: the target file was absent from the verifier’s sandbox. Regrading the saved run changed the result. As Harbor prepared the saved outputs, it followed the symbolic link on the host machine and copied the host file’s contents into the new verifier. This time, the same verification code read “hello world.” Code supplied by the task author had received the contents of a file outside its sandbox. This reproduction confirmed the leak when regrading a single-step task with Docker. The exposed file belonged to the machine performing the regrade. An attacker could use the same mechanism to target credentials or other sensitive files at known paths, provided the Harbor process could read them. The file crosses the trust boundary through Harbor’s output handling. The task does not need to break out of its sandbox to obtain it. Our investigation also included a broad sweep of publicly available benchmark tasks on Harbor. The review found no evidence of exploitation through this mechanism in the tasks examined. That finding is limited to the sweep; it does not establish that exploitation has never occurred. Private task collections deserve the same scrutiny. Labs buy RL tasks from providers that, in turn, employ third-party contractors to produce them. A private dataset can therefore contain code from multiple outside contributors. Our public sweep does not cover those collections, and we cannot rule out use of this mechanism in private datasets. After disclosing the vulnerability to the Harbor team, we collaborated with them on the fix, which they merged in PR 3602 https://github.com/harbor-framework/harbor/pull/3602 . Before copying saved outputs, Harbor now checks where each symbolic link leads. It rejects links outside the recorded agent and artifact directories for that trial step, along with broken links and cycles. Links to valid recorded outputs are copied as files or directories so they remain usable in the verifier. The fix is included in Harbor v0.25.0 https://github.com/harbor-framework/harbor/releases/tag/v0.25.0 , released on October 25, 2026. Update to v0.25.0 or later before regrading tasks. Keeping benchmarks useful and secure Public benchmarks help labs train better models, researchers understand and evaluate them, and enterprises decide which agents to deploy. Their value depends on people being able to contribute tasks, reproduce results, and build on each other’s work. Keeping that process secure takes the same community effort: task authors, framework maintainers, researchers, and users examining how untrusted code and its outputs move through the system. Sandboxing the task is only part of securing an evaluation pipeline. The orchestration code that collects, copies, and replays task outputs must also treat them as untrusted. Security reviews need to cover those interactions across the execution infrastructure, including operations performed after the agent has finished. The other responsibility is supply-chain security for the tasks themselves. Public benchmarks and private RL datasets contain executable contributions from outside authors. Teams need to examine where that code comes from and how it enters their pipelines, while continuing to treat its outputs as untrusted even after a task has been accepted. Thank you to the Harbor team for their prompt response and resolution. As agent training and evaluation grow, we want to help keep benchmarks useful for progress without compromising the security of the labs and enterprises running them.