Read the preprint (arXiv:2611.18553)
Can an AI agent read, fix and write neural-network weights directly, the way an expert reads code?
WeightBench gives an agent the raw weights of a model and a task (find a backdoor, repair a behaviour, write a network from scratch, prune a model for hardware) and only a small, metered budget of forward passes. The agent works in a sandbox, every model call is counted in a hash-chained ledger, and the score is computed from that ledger, so it can't be faked. 42 tasks in 5 tracks, all generated from a seed.
| Agent | READ | FIX | META | WRITE | SYSTEMS | Score (42 tasks) | Score (38 tasks) |
|---|---|---|---|---|---|---|---|
| GPT-6.1-sol † | 0.928 | 0.571 † | 0.260 | 0.863 | 0.220 | – | 0.568 |
| Claude Opus 5.5 | 0.744 | 0.404 | 0.492 | 0.829 | 0.207 | 0.535 | 0.546 |
| Claude Sonnet 5.5 | 0.656 | 0.381 | 0.500 | 0.569 | 0.225 | 0.466 | 0.491 |
| Claude Haiku 5.5 | 0.671 | 0.300 | 0.491 | 0.369 | 0.177 | 0.402 | 0.428 |
| DeepSeek v4-flash | 0.470 | 0.187 | 0.181 | 0.000 | 0.238 | 0.215 | 0.232 |
Score = mean of the five track means of S_WB (0 to 1), seed 0. Claude models ran two samples per task, DeepSeek and GPT-6.1-sol one. Quarantined runs and runs that ended without a valid submission count as 0; no quarantined run has been reviewed yet, so every score is a lower bound. The 38-task score leaves out the four TR-02 reasoning-loop tasks. † GPT-6.1-sol was not run on the reasoning-loop tasks (its FIX mean covers 9 of 13 tasks), and 10 of its 38 runs used the Codex CLI instead of Claude Code. Claude Fable 5.1 is being evaluated. How these numbers were produced, and how to rerun them: docs/reproducing-the-paper.md.
Linux, macOS, or Windows through WSL2, with Python 3.11 or 3.12 and uv.
- Windows: run
wsl --install -d Ubuntuin an administrator PowerShell, restart, open Ubuntu and do every step below inside it (clone into your Linux home, notC:\). - Ubuntu / WSL:
sudo apt update && sudo apt install -y git curl bubblewrap - macOS:
xcode-select --install. The sandbox uses the built-insandbox-exec.
curl -LsSf https://astral.sh/uv/install.sh | sh # skip if you have uv
git clone https://github.com/Aayush-Marasini/weightbench-oss && cd weightbench-oss
bash scripts/quickstart.sh # install + score one demo task, no account needed
source .venv/bin/activate
The quickstart builds TR05-DEAD-001 and scores its baseline (0.000) and reference solution (1.000). Details, extras
and troubleshooting (including Ubuntu 24.04's AppArmor setting): docs/getting-started.md.
bash scripts/install.sh --agent # agent tool environment + sandbox self-test (once per machine)
weightbench bench --agent claude --model claude-sonnet-5-5 --samples 3
weightbench bench --agent codex --model <model-id> --tracks TR04
weightbench bench --agent cmd --model my-model \
--agent-cmd "my-agent --model {model} --prompt-file {prompt_file}" --allow-host api.example.com
Narrow a run with --tracks TR01,TR04, --tiers foundational or --tasks 'TR05-*'; add --with-floors to score
the baseline and reference next to your agent. Results land in runs/<run_id>/ (result.json, the transcript, the
audit and the ledger). weightbench-report prints a leaderboard for your runs, and weightbench-verify --all runs/
re-checks every score from its ledger. Full guide: docs/running-agents.md.
WeightBench ships no API keys and never calls a model itself.
- Agent: your agent runs with your own login or API key (for example
ANTHROPIC_API_KEY, or a gateway via the CLI's own endpoint variables). The sandbox blocks all network traffic except your agent's own API host and any host you add with--allow-host. - Hugging Face: tasks built on gated models need your own token. Accept the model licences on huggingface.co
(Gemma-3-270M-it ,Mistral-7B-v0.3 ), then run
hf auth login. The token file stays outside the sandbox, so agents can't read it. An exportedHF_TOKENis inherited by the agent process, so preferhf auth login. - Language-model tasks need the extras:
uv pip install -e '.[lm]' --torch-backend cpu -c constraints.txt.
| Track | Tasks | The agent must |
|---|---|---|
| TR-01 READ Static diagnostics | 10 | find backdoors, hidden behaviours and planted circuits from the weights |
| TR-02 FIX Surgical patching | 13 | repair a flaw (sycophancy, control failures, reasoning loops) with a small weight edit |
| TR-03 META Trap verification | 5 | tell a fixable defect from one that a weight edit can't fix |
| TR-04 WRITE Weight synthesis | 6 | write a working network (game policies, Go, grafted skills) with no training |
| TR-05 SYSTEMS Hardware-aware | 8 | prune or quantize a model while keeping its behaviour |
weightbench list prints every task with its track and tier; docs/tasks.md lists subject models and
budgets. Scoring, budgets and integrity rules are in docs/SPEC.md.
Most tasks run on a CPU laptop. The exceptions:
| Tasks | Need |
|---|---|
| TR02-SYC-001..004, TR03-PAIR-001..005 | Hugging Face token with the Gemma licence accepted |
| TR03-PAIR-001 | about 15 GB disk (Mistral-7B) and the Mistral licence |
| TR02-LOOP-001..004, TR03-PAIR-003/005 | a GPU, or many CPU hours (long thinking-mode generation) |
- Getting started : install per OS,
weightbench doctor, extras, troubleshooting - Running agents : adapters, custom agents, task selection, outputs, quarantine review
- Tasks : every task with its subject model, budgets and requirements
- Reports, leaderboard and verification :
weightbench-report,weightbench-leaderboard,weightbench-verify, manifests, calibration, access studies - Specification : harness enforcement, tracks, scoring, tiers, knobs and integrity protocols
- Automated quarantine review
- Reproducing the paper's agent study
- Adding a task
- Contributing andsecurity policy
MIT. See LICENSE.
@misc{marasini2026weightbench,
title = {WeightBench: Evaluating Language Models as Weight-Space Programmers},
author = {Marasini, Aayush and Khadka, Barsat and Rahimi, Nick},
year = {2026},
eprint = {2611.18553},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arXiv.2611.18553},
url = {https://arxiv.org/abs/2611.18553}
}