WeightBench: Evaluating LLMs as weight-space programmers A preprint (arXiv:2611.18553) introduces WeightBench, a benchmark that gives AI agents raw neural-network weights and a metered budget of forward passes to perform 42 tasks across five tracks — READ, FIX, META, WRITE and SYSTEMS — with every model call recorded in a hash-chained ledger so scores can be re-verified. On the full 42-task set, Claude Opus 5.5 scored highest at 0.535, followed by Claude Sonnet 5.5 at 0.466, Claude Haiku 5.5 at 0.402 and DeepSeek v4-flash at 0.215; GPT-6.1-sol was not run on the four TR-02 reasoning-loop tasks and scored 0.568 on the remaining 38. The authors state that quarantined runs and runs without a valid submission count as zero and that no quarantined run has been reviewed yet, so every score is a lower bound. Read the preprint arXiv:2611.18553 https://arxiv.org/pdf/2611.18553 Can an AI agent read, fix and write neural-network weights directly, the way an expert reads code? WeightBench gives an agent the raw weights of a model and a task find a backdoor, repair a behaviour, write a network from scratch, prune a model for hardware and only a small, metered budget of forward passes. The agent works in a sandbox, every model call is counted in a hash-chained ledger, and the score is computed from that ledger, so it can't be faked. 42 tasks in 5 tracks, all generated from a seed. | Agent | READ | FIX | META | WRITE | SYSTEMS | Score 42 tasks | Score 38 tasks | |---|---|---|---|---|---|---|---| | GPT-6.1-sol † | 0.928 | 0.571 † | 0.260 | 0.863 | 0.220 | – | 0.568 | | Claude Opus 5.5 | 0.744 | 0.404 | 0.492 | 0.829 | 0.207 | 0.535 | 0.546 | | Claude Sonnet 5.5 | 0.656 | 0.381 | 0.500 | 0.569 | 0.225 | 0.466 | 0.491 | | Claude Haiku 5.5 | 0.671 | 0.300 | 0.491 | 0.369 | 0.177 | 0.402 | 0.428 | | DeepSeek v4-flash | 0.470 | 0.187 | 0.181 | 0.000 | 0.238 | 0.215 | 0.232 | Score = mean of the five track means of S WB 0 to 1 , seed 0. Claude models ran two samples per task, DeepSeek and GPT-6.1-sol one. Quarantined runs and runs that ended without a valid submission count as 0; no quarantined run has been reviewed yet, so every score is a lower bound. The 38-task score leaves out the four TR-02 reasoning-loop tasks. † GPT-6.1-sol was not run on the reasoning-loop tasks its FIX mean covers 9 of 13 tasks , and 10 of its 38 runs used the Codex CLI instead of Claude Code. Claude Fable 5.1 is being evaluated. How these numbers were produced, and how to rerun them: docs/reproducing-the-paper.md https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/reproducing-the-paper.md . Linux, macOS, or Windows through WSL2, with Python 3.11 or 3.12 and uv https://docs.astral.sh/uv/ . - Windows: run wsl --install -d Ubuntu in an administrator PowerShell, restart, open Ubuntu and do every step below inside it clone into your Linux home, not C:\ . - Ubuntu / WSL: sudo apt update && sudo apt install -y git curl bubblewrap - macOS: xcode-select --install . The sandbox uses the built-in sandbox-exec . curl -LsSf https://astral.sh/uv/install.sh | sh skip if you have uv git clone https://github.com/Aayush-Marasini/weightbench-oss && cd weightbench-oss bash scripts/quickstart.sh install + score one demo task, no account needed source .venv/bin/activate The quickstart builds TR05-DEAD-001 and scores its baseline 0.000 and reference solution 1.000 . Details, extras and troubleshooting including Ubuntu 24.04's AppArmor setting : docs/getting-started.md https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/getting-started.md . bash scripts/install.sh --agent agent tool environment + sandbox self-test once per machine Claude Code log in once with claude , all tracks, 3 samples per task weightbench bench --agent claude --model claude-sonnet-5-5 --samples 3 OpenAI Codex CLI log in once with codex , one track weightbench bench --agent codex --model