cd /news/ai-research/weightbench-evaluating-llms-as-weigh… · home › topics › ai-research › article
[ARTICLE · art-172085] src=github.com ↗ pub= topic=ai-research verified=true sentiment=· neutral

WeightBench: Evaluating LLMs as weight-space programmers

A preprint (arXiv:2611.18553) introduces WeightBench, a benchmark that gives AI agents raw neural-network weights and a metered budget of forward passes to perform 42 tasks across five tracks — READ, FIX, META, WRITE and SYSTEMS — with every model call recorded in a hash-chained ledger so scores can be re-verified. On the full 42-task set, Claude Opus 5.5 scored highest at 0.535, followed by Claude Sonnet 5.5 at 0.466, Claude Haiku 5.5 at 0.402 and DeepSeek v4-flash at 0.215; GPT-6.1-sol was not run on the four TR-02 reasoning-loop tasks and scored 0.568 on the remaining 38. The authors state that quarantined runs and runs without a valid submission count as zero and that no quarantined run has been reviewed yet, so every score is a lower bound.

recommendation

Score agents on weight-space tasks from a hash-chained ledger of metered forward passes and report per-track means, since overall scores hide near-zero WRITE and SYSTEMS results.

more in playbook: Evals and testing for LLM apps
read5 min views4 publishedNov 18, 2026
WeightBench: Evaluating LLMs as weight-space programmers
Image: Michielbdejong (auto-discovered)

Read the preprint (arXiv:2611.18553)

Can an AI agent read, fix and write neural-network weights directly, the way an expert reads code?

WeightBench gives an agent the raw weights of a model and a task (find a backdoor, repair a behaviour, write a network from scratch, prune a model for hardware) and only a small, metered budget of forward passes. The agent works in a sandbox, every model call is counted in a hash-chained ledger, and the score is computed from that ledger, so it can't be faked. 42 tasks in 5 tracks, all generated from a seed.

Agent READ FIX META WRITE SYSTEMS Score (42 tasks) Score (38 tasks)
GPT-6.1-sol † 0.928 0.571 † 0.260 0.863 0.220 – 0.568
Claude Opus 5.5 0.744 0.404 0.492 0.829 0.207 0.535 0.546
Claude Sonnet 5.5 0.656 0.381 0.500 0.569 0.225 0.466 0.491
Claude Haiku 5.5 0.671 0.300 0.491 0.369 0.177 0.402 0.428
DeepSeek v4-flash 0.470 0.187 0.181 0.000 0.238 0.215 0.232

Score = mean of the five track means of S_WB (0 to 1), seed 0. Claude models ran two samples per task, DeepSeek and GPT-6.1-sol one. Quarantined runs and runs that ended without a valid submission count as 0; no quarantined run has been reviewed yet, so every score is a lower bound. The 38-task score leaves out the four TR-02 reasoning-loop tasks. † GPT-6.1-sol was not run on the reasoning-loop tasks (its FIX mean covers 9 of 13 tasks), and 10 of its 38 runs used the Codex CLI instead of Claude Code. Claude Fable 5.1 is being evaluated. How these numbers were produced, and how to rerun them: docs/reproducing-the-paper.md.

Linux, macOS, or Windows through WSL2, with Python 3.11 or 3.12 and uv.

  • Windows: runwsl --install -d Ubuntu in an administrator PowerShell, restart, open Ubuntu and do every step below inside it (clone into your Linux home, notC:\ ).
  • Ubuntu / WSL:sudo apt update && sudo apt install -y git curl bubblewrap
  • macOS:xcode-select --install . The sandbox uses the built-insandbox-exec .
curl -LsSf https://astral.sh/uv/install.sh | sh     # skip if you have uv
git clone https://github.com/Aayush-Marasini/weightbench-oss && cd weightbench-oss
bash scripts/quickstart.sh                          # install + score one demo task, no account needed
source .venv/bin/activate

The quickstart builds TR05-DEAD-001 and scores its baseline (0.000) and reference solution (1.000). Details, extras and troubleshooting (including Ubuntu 24.04's AppArmor setting): docs/getting-started.md.

bash scripts/install.sh --agent          # agent tool environment + sandbox self-test (once per machine)

weightbench bench --agent claude --model claude-sonnet-5-5 --samples 3

weightbench bench --agent codex --model <model-id> --tracks TR04

weightbench bench --agent cmd --model my-model \
    --agent-cmd "my-agent --model {model} --prompt-file {prompt_file}" --allow-host api.example.com

Narrow a run with --tracks TR01,TR04, --tiers foundational or --tasks 'TR05-*'; add --with-floors to score the baseline and reference next to your agent. Results land in runs/<run_id>/ (result.json, the transcript, the audit and the ledger). weightbench-report prints a leaderboard for your runs, and weightbench-verify --all runs/ re-checks every score from its ledger. Full guide: docs/running-agents.md.

WeightBench ships no API keys and never calls a model itself.

  • Agent: your agent runs with your own login or API key (for exampleANTHROPIC_API_KEY , or a gateway via the CLI's own endpoint variables). The sandbox blocks all network traffic except your agent's own API host and any host you add with--allow-host .
  • Hugging Face: tasks built on gated models need your own token. Accept the model licences on huggingface.co (Gemma-3-270M-it ,Mistral-7B-v0.3 ), then runhf auth login . The token file stays outside the sandbox, so agents can't read it. An exportedHF_TOKEN is inherited by the agent process, so preferhf auth login .
  • Language-model tasks need the extras: uv pip install -e '.[lm]' --torch-backend cpu -c constraints.txt .
Track Tasks The agent must
TR-01 READ Static diagnostics 10 find backdoors, hidden behaviours and planted circuits from the weights
TR-02 FIX Surgical patching 13 repair a flaw (sycophancy, control failures, reasoning loops) with a small weight edit
TR-03 META Trap verification 5 tell a fixable defect from one that a weight edit can't fix
TR-04 WRITE Weight synthesis 6 write a working network (game policies, Go, grafted skills) with no training
TR-05 SYSTEMS Hardware-aware 8 prune or quantize a model while keeping its behaviour

weightbench list prints every task with its track and tier; docs/tasks.md lists subject models and budgets. Scoring, budgets and integrity rules are in docs/SPEC.md.

Most tasks run on a CPU laptop. The exceptions:

Tasks Need
TR02-SYC-001..004, TR03-PAIR-001..005 Hugging Face token with the Gemma licence accepted
TR03-PAIR-001 about 15 GB disk (Mistral-7B) and the Mistral licence
TR02-LOOP-001..004, TR03-PAIR-003/005 a GPU, or many CPU hours (long thinking-mode generation)

MIT. See LICENSE.

@misc{marasini2026weightbench,
  title         = {WeightBench: Evaluating Language Models as Weight-Space Programmers},
  author        = {Marasini, Aayush and Khadka, Barsat and Rahimi, Nick},
  year          = {2026},
  eprint        = {2611.18553},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  doi           = {10.48550/arXiv.2611.18553},
  url           = {https://arxiv.org/abs/2611.18553}
}
── more in #ai-research 4 stories · sorted by recency
── more on @weightbench 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/weightbench-evaluati…] indexed:0 read:5min 2026-11-18 · —