Open-source benchmark for long-horizon web-app coding

Researchers released EvoWeb, an open-source benchmark (arXiv:2610.16402) that tests coding agents on 32 web applications, each requiring a first build plus nine follow-up feature requests, with a browser agent (WebVoyager driven by Claude Sonnet 4.6) checking the running app after every request. EvoWeb scores agents on Grade, which summarizes how much working functionality survives the full 10-request sequence, and Survival, which counts how many consecutive requests stay usable. The release includes the harness Docker image, scoring via `python3 -m evoweb.score_run`, and support for Anthropic, OpenAI, OpenRouter, any OpenAI-compatible API, or locally served weights via vLLM.

Code, data and traces for EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications arXiv:2610.16402 https://arxiv.org/abs/2610.16402 . Left: how the 32 apps and their 10-request sequences were built. Right: how an agent is evaluated on one app, request after request. EvoWeb gives a coding agent 32 web apps to build and then extend, request by request: a first build and nine follow-up requests per app, each adding a feature that may depend on earlier ones. After every request a browser agent WebVoyager driven by Claude Sonnet 4.6 tests the running app. Grade summarises how much working functionality survives the whole sequence; Survival counts how many requests in a row stay usable. | <sub One app, request after request. Left: the user asks, the coding agent builds. Right: the browser agent judges the live app, check by check, and the request's pass rate lands on the timeline. Every request, tool call, screenshot and judge step is from the released trace of app 000001 Opus 4.8 ; request 1's verdicts are adjusted to show every grade. Video https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-teaser.mp4 , the 80-second version https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-demo.mp4 with the scoring, and the source https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-demo . </sub | Build the harness image once, then serve the model and run it. A run takes a name, which apps 1-32 for all, or 1,4,6,11 and how many run in parallel; results land in runs/<run name / , and python3 -m evoweb.score run scores them exactly like the paper. Any provider or model works in their place: Anthropic, OpenAI, OpenRouter, any OpenAI-compatible API, or weights served locally with vLLM; harness/README.md https://github.com/two-delta/EvoWeb/blob/main/harness/README.md lists every variable. docker build -f harness/Dockerfile -t evoweb-harness:latest . pip install -r requirements.txt scoring: Python = 3.10 pip install -e . the evoweb command Example for vLLM recipe for your model https://docs.vllm.ai/projects/recipes/en/latest/ : docker pull vllm/vllm-openai:v0.31.0 docker run --rm --gpus '"device=0,1,2,3"' --ipc=host --network=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.31.0 Qwen/Qwen3.5-397B-A17B-FP8 \ --tensor-parallel-size 4 \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice --tool-call-parser qwen3 coder \ --reasoning-parser qwen3 \ --port 8000 Check that it answers: curl -s localhost:8000/v1/models The model and the judge are each set by a name plus where it's served. With a local vLLM server you only need the port; the host defaults to localhost . export ANTHROPIC API KEY=<key for the judge OPENHANDS SECURITY RISK OPTIONAL=1 evoweb run --name qwen397b --model Qwen/Qwen3.5-397B-A17B-FP8 --model-port 8000 OPENHANDS SECURITY RISK OPTIONAL=1 accepts a tool call that omits OpenHands' security risk field instead of rejecting it; Qwen 3.5 omits it and stalls without this. The paper's rows were run without it, and it also raises Claude Haiku 4.5 above its paper score, so leave it unset when reproducing the paper. The judge defaults to Claude Sonnet 4.6 on the Anthropic API. To serve the judge yourself a vision model: it reads screenshots, so add --limit-mm-per-prompt '{"image":4}' to its vLLM command : evoweb run --name qwen-both \ --model Qwen/Qwen3.5-397B-A17B-FP8 --model-port 8000 \ --judge Qwen/Qwen3.5-397B-A17B-FP8 --judge-port 8001 A server on another machine: add --model-host 10.0.0.5 or --judge-host . Skip step 2. Give the model with its provider prefix and no port; the key comes from that provider's usual environment variable: export ANTHROPIC API KEY=<key evoweb run --name haiku --model anthropic/claude-haiku-4-5 export OPENAI API KEY=<key evoweb run --name gpt5 --model openai/gpt-5 export OPENROUTER API KEY=<key evoweb run --name glm52 --model openrouter/z-ai/glm-5.2 Any other OpenAI-compatible API: --model-url https://<host /v1 --model <name , with the key in MODEL API KEY . | Flag | Default | | |---|---|---| | --name | required | run name; results go to runs/<name / | | --model | required | served name, or provider/model for an API | | --model-port | | port of a vLLM / OpenAI-compatible server | | --model-host | localhost | host of that server | | --model-url | | full base URL instead of host + port | | --judge , --judge-port , --judge-host , --judge-url | anthropic/claude-sonnet-4-6 | same, for the judge | | --apps | 1-32 | subset, e.g. 1,4,6 | | --parallel | 8 | apps run at once | To compare a run with the paper's runs of the same model released for the four Anthropic models : python3 -m evoweb.compare to released "Haiku 4.5" runs/haiku . infra/README.md https://github.com/two-delta/EvoWeb/blob/main/infra/README.md covers running many models at once on Kubernetes. harness/ the agent OpenHands + judge WebVoyager runner, its Dockerfile and launchers data/ the benchmark evoweb benchmark.jsonl and its translations multilingual/ , both downloaded from Hugging Face on use evoweb/paths.py , the no-grounding ablation set, the WebGen-Bench base apps and data gen/ with the scripts that generated them; data/README.md documents the format evoweb/ scoring scoring.py holds the metric definitions, metrics.py the paper's aggregates and the runnable modules: score run score your run , compare to released next to the paper's runs , score zip unpack and score a full-trace archive examples artifacts/ traces.zip the paper's runs of the four Anthropic models Fable 5, Opus 4.8, Sonnet 4.6, Haiku 4.5 , three each, plus the multilingual Sonnet 4.6 and Haiku 4.5 runs: <run-id /<app /result.json judge verdicts + token logs; the other models' runs are not released full traces/ Fable 5's first run complete: all conversations, code, judge sessions, logs examples.zip one Opus 4.8 app with everything, screenshots included reproductions/ our full Haiku re-run and its comparison with the paper ablations/ the paper's DAG, grounding and multilingual analyses tests/ offline tests scoring, datasets, harness + the opt-in live reproduction test infra/ running at scale on Kubernetes: launcher, job template, node pool, README media/ the method figure and the judge demo video @article{nukrai2026evoweb, title = {EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications}, author = {Nukrai, David and Alon, Bar and Levy, Nativ and Atad, Ido Andrew and Cohen, Ido and Zimerman, Itamar and Wolf, Lior and David, Eli and Naiman, Ilan}, journal = {arXiv preprint arXiv:2610.16402}, year = {2026}, url = {https://arxiv.org/abs/2610.16402} } Apache License 2.0; see LICENSE https://github.com/two-delta/EvoWeb/blob/main/LICENSE .