{"slug": "weightbench-evaluating-llms-as-weight-space-programmers", "title": "WeightBench: Evaluating LLMs as weight-space programmers", "summary": "A preprint (arXiv:2611.18553) introduces WeightBench, a benchmark that gives AI agents raw neural-network weights and a metered budget of forward passes to perform 42 tasks across five tracks — READ, FIX, META, WRITE and SYSTEMS — with every model call recorded in a hash-chained ledger so scores can be re-verified. On the full 42-task set, Claude Opus 5.5 scored highest at 0.535, followed by Claude Sonnet 5.5 at 0.466, Claude Haiku 5.5 at 0.402 and DeepSeek v4-flash at 0.215; GPT-6.1-sol was not run on the four TR-02 reasoning-loop tasks and scored 0.568 on the remaining 38. The authors state that quarantined runs and runs without a valid submission count as zero and that no quarantined run has been reviewed yet, so every score is a lower bound.", "body_md": "[Read the preprint (arXiv:2611.18553)](https://arxiv.org/pdf/2611.18553)\n\n**Can an AI agent read, fix and write neural-network weights directly, the way an expert reads code?**\n\nWeightBench gives an agent the raw weights of a model and a task (find a backdoor, repair a behaviour, write a network from scratch, prune a model for hardware) and only a small, metered budget of forward passes. The agent works in a sandbox, every model call is counted in a hash-chained ledger, and the score is computed from that ledger, so it can't be faked. 42 tasks in 5 tracks, all generated from a seed.\n\n| Agent | READ | FIX | META | WRITE | SYSTEMS | **Score (42 tasks)** | Score (38 tasks) | \n|---|---|---|---|---|---|---|---|\n| GPT-6.1-sol † | 0.928 | 0.571 † | 0.260 | 0.863 | 0.220 | – | **0.568** | \n| Claude Opus 5.5 | 0.744 | 0.404 | 0.492 | 0.829 | 0.207 | **0.535** | 0.546 | \n| Claude Sonnet 5.5 | 0.656 | 0.381 | 0.500 | 0.569 | 0.225 | **0.466** | 0.491 | \n| Claude Haiku 5.5 | 0.671 | 0.300 | 0.491 | 0.369 | 0.177 | **0.402** | 0.428 | \n| DeepSeek v4-flash | 0.470 | 0.187 | 0.181 | 0.000 | 0.238 | **0.215** | 0.232 | \n\nScore = mean of the five track means of S_WB (0 to 1), seed 0. Claude models ran two samples per task, DeepSeek and\nGPT-6.1-sol one. Quarantined runs and runs that ended without a valid submission count as 0; no quarantined run has\nbeen reviewed yet, so every score is a lower bound. The 38-task score leaves out the four TR-02 reasoning-loop tasks.\n† GPT-6.1-sol was not run on the reasoning-loop tasks (its FIX mean covers 9 of 13 tasks), and 10 of its 38 runs used\nthe Codex CLI instead of Claude Code. Claude Fable 5.1 is being evaluated. How these numbers were produced, and how\nto rerun them: [docs/reproducing-the-paper.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/reproducing-the-paper.md).\n\nLinux, macOS, or Windows through WSL2, with Python 3.11 or 3.12 and [uv](https://docs.astral.sh/uv/).\n\n- **Windows:** run`wsl --install -d Ubuntu` in an administrator PowerShell, restart, open Ubuntu and do every step\nbelow inside it (clone into your Linux home, not`C:\\` ).\n- **Ubuntu / WSL:**`sudo apt update && sudo apt install -y git curl bubblewrap`\n- **macOS:**`xcode-select --install` . The sandbox uses the built-in`sandbox-exec` .\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh     # skip if you have uv\ngit clone https://github.com/Aayush-Marasini/weightbench-oss && cd weightbench-oss\nbash scripts/quickstart.sh                          # install + score one demo task, no account needed\nsource .venv/bin/activate\n```\n\nThe quickstart builds `TR05-DEAD-001` and scores its baseline (0.000) and reference solution (1.000). Details, extras\nand troubleshooting (including Ubuntu 24.04's AppArmor setting): [docs/getting-started.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/getting-started.md).\n\n```\nbash scripts/install.sh --agent          # agent tool environment + sandbox self-test (once per machine)\n\n# Claude Code (log in once with `claude`), all tracks, 3 samples per task\nweightbench bench --agent claude --model claude-sonnet-5-5 --samples 3\n\n# OpenAI Codex CLI (log in once with `codex`), one track\nweightbench bench --agent codex --model <model-id> --tracks TR04\n\n# any command-line agent\nweightbench bench --agent cmd --model my-model \\\n    --agent-cmd \"my-agent --model {model} --prompt-file {prompt_file}\" --allow-host api.example.com\n```\n\nNarrow a run with `--tracks TR01,TR04`, `--tiers foundational` or `--tasks 'TR05-*'`; add `--with-floors` to score\nthe baseline and reference next to your agent. Results land in `runs/<run_id>/` (`result.json`, the transcript, the\naudit and the ledger). `weightbench-report` prints a leaderboard for your runs, and `weightbench-verify --all runs/`\nre-checks every score from its ledger. Full guide: [docs/running-agents.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/running-agents.md).\n\nWeightBench ships no API keys and never calls a model itself.\n\n- **Agent:** your agent runs with your own login or API key (for example`ANTHROPIC_API_KEY` , or a gateway via the\nCLI's own endpoint variables). The sandbox blocks all network traffic except your agent's own API host and any host\nyou add with`--allow-host` .\n- **Hugging Face:** tasks built on gated models need your own token. Accept the model licences on huggingface.co\n([Gemma-3-270M-it](https://huggingface.co/google/gemma-3-270m-it) ,[Mistral-7B-v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3) ), then run`hf auth login` . The token file\nstays outside the sandbox, so agents can't read it. An exported`HF_TOKEN` is inherited by the agent process, so\nprefer`hf auth login` .\n- Language-model tasks need the extras: `uv pip install -e '.[lm]' --torch-backend cpu -c constraints.txt` .\n\n| Track | Tasks | The agent must | \n|---|---|---|\n| **TR-01 READ** Static diagnostics | 10 | find backdoors, hidden behaviours and planted circuits from the weights | \n| **TR-02 FIX** Surgical patching | 13 | repair a flaw (sycophancy, control failures, reasoning loops) with a small weight edit | \n| **TR-03 META** Trap verification | 5 | tell a fixable defect from one that a weight edit can't fix | \n| **TR-04 WRITE** Weight synthesis | 6 | write a working network (game policies, Go, grafted skills) with no training | \n| **TR-05 SYSTEMS** Hardware-aware | 8 | prune or quantize a model while keeping its behaviour | \n\n`weightbench list` prints every task with its track and tier; [docs/tasks.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/tasks.md) lists subject models and\nbudgets. Scoring, budgets and integrity rules are in [docs/SPEC.md](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/SPEC.md).\n\nMost tasks run on a CPU laptop. The exceptions:\n\n| Tasks | Need | \n|---|---|\n| TR02-SYC-001..004, TR03-PAIR-001..005 | Hugging Face token with the Gemma licence accepted | \n| TR03-PAIR-001 | about 15 GB disk (Mistral-7B) and the Mistral licence | \n| TR02-LOOP-001..004, TR03-PAIR-003/005 | a GPU, or many CPU hours (long thinking-mode generation) | \n\n- [Getting started](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/getting-started.md) : install per OS,`weightbench doctor` , extras, troubleshooting\n- [Running agents](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/running-agents.md) : adapters, custom agents, task selection, outputs, quarantine review\n- [Tasks](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/tasks.md) : every task with its subject model, budgets and requirements\n- [Reports, leaderboard and verification](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/leaderboard-and-verification.md) :`weightbench-report` ,`weightbench-leaderboard` ,`weightbench-verify` , manifests, calibration, access studies\n- [Specification](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/SPEC.md) : harness enforcement, tracks, scoring, tiers, knobs and integrity protocols\n- [Automated quarantine review](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/AUTO_REVIEW.md)\n- [Reproducing the paper's agent study](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/reproducing-the-paper.md)\n- [Adding a task](https://github.com/Aayush-Marasini/WeightBench/blob/main/docs/adding-a-task.md)\n- [Contributing](https://github.com/Aayush-Marasini/WeightBench/blob/main/CONTRIBUTING.md) and[security policy](https://github.com/Aayush-Marasini/WeightBench/blob/main/SECURITY.md)\n\nMIT. See [LICENSE](https://github.com/Aayush-Marasini/WeightBench/blob/main/LICENSE).\n\n```\n@misc{marasini2026weightbench,\n  title         = {WeightBench: Evaluating Language Models as Weight-Space Programmers},\n  author        = {Marasini, Aayush and Khadka, Barsat and Rahimi, Nick},\n  year          = {2026},\n  eprint        = {2611.18553},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG},\n  doi           = {10.48550/arXiv.2611.18553},\n  url           = {https://arxiv.org/abs/2611.18553}\n}\n```\n\n", "url": "https://wpnews.pro/news/weightbench-evaluating-llms-as-weight-space-programmers", "canonical_source": "https://github.com/Aayush-Marasini/WeightBench", "published_at": "2026-11-18 19:29:42+00:00", "updated_at": "2026-11-18 19:46:16.800576+00:00", "lang": "en", "topics": ["ai-research", "ai-agents", "machine-learning", "ai-safety"], "entities": ["WeightBench", "Claude Opus 5.5", "Claude Sonnet 5.5", "Claude Haiku 5.5", "GPT-6.1-sol", "DeepSeek v4-flash", "Claude Fable 5.1", "arXiv"], "also_reported_by": [], "takeaway": "On WeightBench's 42 weight-space tasks, Claude Opus 5.5 leads at 0.535 overall but scores only 0.207 on SYSTEMS and 0.404 on FIX, while DeepSeek v4-flash scores 0.000 on WRITE.", "recommendation": {"advice": "Score agents on weight-space tasks from a hash-chained ledger of metered forward passes and report per-track means, since overall scores hide near-zero WRITE and SYSTEMS results.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/weightbench-evaluating-llms-as-weight-space-programmers", "markdown": "https://wpnews.pro/news/weightbench-evaluating-llms-as-weight-space-programmers.md", "text": "https://wpnews.pro/news/weightbench-evaluating-llms-as-weight-space-programmers.txt", "jsonld": "https://wpnews.pro/news/weightbench-evaluating-llms-as-weight-space-programmers.jsonld"}}