# WeightBench: Evaluating LLMs as weight-space programmers

> Source: <https://github.com/Aayush-Marasini/WeightBench>
> Published: 2026-11-18 19:29:42+00:00

**Recommendation:** Score agents on weight-space tasks from a hash-chained ledger of metered forward passes and report per-track means, since overall scores hide near-zero WRITE and SYSTEMS results.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

[Read the preprint (arXiv:2611.18553)](https://arxiv.org/pdf/2611.18553)

**Can an AI agent read, fix and write neural-network weights directly, the way an expert reads code?**

WeightBench gives an agent the raw weights of a model and a task (find a backdoor, repair a behaviour, write a network from scratch, prune a model for hardware) and only a small, metered budget of forward passes. The agent works in a sandbox, every model call is counted in a hash-chained ledger, and the score is computed from that ledger, so it can't be faked. 42 tasks in 5 tracks, all generated from a seed.

| Agent | READ | FIX | META | WRITE | SYSTEMS | **Score (42 tasks)** | Score (38 tasks) | 
|---|---|---|---|---|---|---|---|
| GPT-6.1-sol † | 0.928 | 0.571 † | 0.260 | 0.863 | 0.220 | – | **0.568** | 
| Claude Opus 5.5 | 0.744 | 0.404 | 0.492 | 0.829 | 0.207 | **0.535** | 0.546 | 
| Claude Sonnet 5.5 | 0.656 | 0.381 | 0.500 | 0.569 | 0.225 | **0.466** | 0.491 | 
| Claude Haiku 5.5 | 0.671 | 0.300 | 0.491 | 0.369 | 0.177 | **0.402** | 0.428 | 
| DeepSeek v4-flash | 0.470 | 0.187 | 0.181 | 0.000 | 0.238 | **0.215** | 0.232 | 

Score = mean of the five track means of S_WB (0 to 1), seed 0. Claude models ran two samples per task, DeepSeek and
GPT-6.1-sol one. Quarantined runs and runs that ended without a valid submission count as 0; no quarantined run has
been reviewed yet, so every score is a lower bound. The 38-task score leaves out the four TR-02 reasoning-loop tasks.
† GPT-6.1-sol was not run on the reasoning-loop tasks (its FIX mean covers 9 of 13 tasks), and 10 of its 38 runs used
the Codex CLI instead of Claude Code. Claude Fable 5.1 is being evaluated. How these numbers were produced, and how
to rerun them: [docs/reproducing-the-paper.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/reproducing-the-paper.md).

Linux, macOS, or Windows through WSL2, with Python 3.11 or 3.12 and [uv](https://docs.astral.sh/uv/).

- **Windows:** run`wsl --install -d Ubuntu` in an administrator PowerShell, restart, open Ubuntu and do every step
below inside it (clone into your Linux home, not`C:\` ).
- **Ubuntu / WSL:**`sudo apt update && sudo apt install -y git curl bubblewrap`
- **macOS:**`xcode-select --install` . The sandbox uses the built-in`sandbox-exec` .

```
curl -LsSf https://astral.sh/uv/install.sh | sh     # skip if you have uv
git clone https://github.com/Aayush-Marasini/weightbench-oss && cd weightbench-oss
bash scripts/quickstart.sh                          # install + score one demo task, no account needed
source .venv/bin/activate
```

The quickstart builds `TR05-DEAD-001` and scores its baseline (0.000) and reference solution (1.000). Details, extras
and troubleshooting (including Ubuntu 24.04's AppArmor setting): [docs/getting-started.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/getting-started.md).

```
bash scripts/install.sh --agent          # agent tool environment + sandbox self-test (once per machine)

# Claude Code (log in once with `claude`), all tracks, 3 samples per task
weightbench bench --agent claude --model claude-sonnet-5-5 --samples 3

# OpenAI Codex CLI (log in once with `codex`), one track
weightbench bench --agent codex --model <model-id> --tracks TR04

# any command-line agent
weightbench bench --agent cmd --model my-model \
    --agent-cmd "my-agent --model {model} --prompt-file {prompt_file}" --allow-host api.example.com
```

Narrow a run with `--tracks TR01,TR04`, `--tiers foundational` or `--tasks 'TR05-*'`; add `--with-floors` to score
the baseline and reference next to your agent. Results land in `runs/<run_id>/` (`result.json`, the transcript, the
audit and the ledger). `weightbench-report` prints a leaderboard for your runs, and `weightbench-verify --all runs/`
re-checks every score from its ledger. Full guide: [docs/running-agents.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/running-agents.md).

WeightBench ships no API keys and never calls a model itself.

- **Agent:** your agent runs with your own login or API key (for example`ANTHROPIC_API_KEY` , or a gateway via the
CLI's own endpoint variables). The sandbox blocks all network traffic except your agent's own API host and any host
you add with`--allow-host` .
- **Hugging Face:** tasks built on gated models need your own token. Accept the model licences on huggingface.co
([Gemma-3-270M-it](https://huggingface.co/google/gemma-3-270m-it) ,[Mistral-7B-v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3) ), then run`hf auth login` . The token file
stays outside the sandbox, so agents can't read it. An exported`HF_TOKEN` is inherited by the agent process, so
prefer`hf auth login` .
- Language-model tasks need the extras: `uv pip install -e '.[lm]' --torch-backend cpu -c constraints.txt` .

| Track | Tasks | The agent must | 
|---|---|---|
| **TR-01 READ** Static diagnostics | 10 | find backdoors, hidden behaviours and planted circuits from the weights | 
| **TR-02 FIX** Surgical patching | 13 | repair a flaw (sycophancy, control failures, reasoning loops) with a small weight edit | 
| **TR-03 META** Trap verification | 5 | tell a fixable defect from one that a weight edit can't fix | 
| **TR-04 WRITE** Weight synthesis | 6 | write a working network (game policies, Go, grafted skills) with no training | 
| **TR-05 SYSTEMS** Hardware-aware | 8 | prune or quantize a model while keeping its behaviour | 

`weightbench list` prints every task with its track and tier; [docs/tasks.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/tasks.md) lists subject models and
budgets. Scoring, budgets and integrity rules are in [docs/SPEC.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/SPEC.md).

Most tasks run on a CPU laptop. The exceptions:

| Tasks | Need | 
|---|---|
| TR02-SYC-001..004, TR03-PAIR-001..005 | Hugging Face token with the Gemma licence accepted | 
| TR03-PAIR-001 | about 15 GB disk (Mistral-7B) and the Mistral licence | 
| TR02-LOOP-001..004, TR03-PAIR-003/005 | a GPU, or many CPU hours (long thinking-mode generation) | 

- [Getting started](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/getting-started.md) : install per OS,`weightbench doctor` , extras, troubleshooting
- [Running agents](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/running-agents.md) : adapters, custom agents, task selection, outputs, quarantine review
- [Tasks](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/tasks.md) : every task with its subject model, budgets and requirements
- [Reports, leaderboard and verification](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/leaderboard-and-verification.md) :`weightbench-report` ,`weightbench-leaderboard` ,`weightbench-verify` , manifests, calibration, access studies
- [Specification](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/SPEC.md) : harness enforcement, tracks, scoring, tiers, knobs and integrity protocols
- [Automated quarantine review](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/AUTO_REVIEW.md)
- [Reproducing the paper's agent study](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/reproducing-the-paper.md)
- [Adding a task](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/adding-a-task.md)
- [Contributing](https://github.com/Aayush-Marasini/WeightBench/blob/main/CONTRIBUTING.md) and[security policy](https://github.com/Aayush-Marasini/WeightBench/blob/main/SECURITY.md)

MIT. See [LICENSE](https://github.com/Aayush-Marasini/WeightBench/blob/main/LICENSE).

```
@misc{marasini2026weightbench,
  title         = {WeightBench: Evaluating Language Models as Weight-Space Programmers},
  author        = {Marasini, Aayush and Khadka, Barsat and Rahimi, Nick},
  year          = {2026},
  eprint        = {2611.18553},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  doi           = {10.48550/arXiv.2611.18553},
  url           = {https://arxiv.org/abs/2611.18553}
}
```


