{"slug": "open-source-benchmark-for-long-horizon-web-app-coding", "title": "Open-source benchmark for long-horizon web-app coding", "summary": "Researchers released EvoWeb, an open-source benchmark (arXiv:2610.16402) that tests coding agents on 32 web applications, each requiring a first build plus nine follow-up feature requests, with a browser agent (WebVoyager driven by Claude Sonnet 4.6) checking the running app after every request. EvoWeb scores agents on Grade, which summarizes how much working functionality survives the full 10-request sequence, and Survival, which counts how many consecutive requests stay usable. The release includes the harness Docker image, scoring via `python3 -m evoweb.score_run`, and support for Anthropic, OpenAI, OpenRouter, any OpenAI-compatible API, or locally served weights via vLLM.", "body_md": "Code, data and traces for **EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications**\n([arXiv:2610.16402](https://arxiv.org/abs/2610.16402)).\n\n*Left: how the 32 apps and their 10-request sequences were built. Right: how an agent is\nevaluated on one app, request after request.*\n\nEvoWeb gives a coding agent 32 web apps to build and then extend, request by request: a first\nbuild and nine follow-up requests per app, each adding a feature that may depend on earlier ones.\nAfter every request a browser agent (WebVoyager driven by Claude Sonnet 4.6) tests the running\napp. **Grade** summarises how much working functionality survives the whole sequence;\n**Survival** counts how many requests in a row stay usable.\n\n|  <sub>*One app, request after request. Left: the user asks, the coding agent builds. Right: the browser       agent judges the live app, check by check, and the request's pass rate lands on the timeline. Every       request, tool call, screenshot and judge step is from the released trace of app 000001 (Opus 4.8);       request 1's verdicts are adjusted to show every grade. [Video](https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-teaser.mp4),       [the 80-second version](https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-demo.mp4) with the scoring, and       [the source](https://github.com/two-delta/EvoWeb/blob/main/media/evaluation-demo).*</sub> | \n\nBuild the harness image once, then serve the model and run it. A run takes a\nname, which apps (`1-32` for all, or `1,4,6,11`) and how many run in parallel; results land in\n`runs/<run name>/`, and `python3 -m evoweb.score_run` scores them exactly like the paper. Any provider or model works in\ntheir place: Anthropic, OpenAI, OpenRouter, any OpenAI-compatible API, or weights served\nlocally with vLLM; [`harness/README.md`](https://github.com/two-delta/EvoWeb/blob/main/harness/README.md) lists every variable.\n\n```\ndocker build -f harness/Dockerfile -t evoweb-harness:latest .\npip install -r requirements.txt        # scoring: Python >= 3.10\npip install -e .                       # the evoweb command\n```\n\nExample for vLLM ([recipe for your model](https://docs.vllm.ai/projects/recipes/en/latest/)):\n\n```\ndocker pull vllm/vllm-openai:v0.31.0\n\ndocker run --rm --gpus '\"device=0,1,2,3\"' --ipc=host --network=host \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:v0.31.0 Qwen/Qwen3.5-397B-A17B-FP8 \\\n  --tensor-parallel-size 4 \\\n  --max-model-len 262144 \\\n  --kv-cache-dtype fp8 \\\n  --enable-auto-tool-choice --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --port 8000\n```\n\nCheck that it answers:\n\n```\ncurl -s localhost:8000/v1/models\n```\n\nThe model and the judge are each set by a name plus where it's served. With a local vLLM server\nyou only need the port; the host defaults to `localhost`.\n\n```\nexport ANTHROPIC_API_KEY=<key>      # for the judge\n\nOPENHANDS_SECURITY_RISK_OPTIONAL=1 evoweb run --name qwen397b --model Qwen/Qwen3.5-397B-A17B-FP8 --model-port 8000\n```\n\n`OPENHANDS_SECURITY_RISK_OPTIONAL=1` accepts a tool call that omits OpenHands' `security_risk`\nfield instead of rejecting it; Qwen 3.5 omits it and stalls without this. The paper's rows were\nrun without it, and it also raises Claude Haiku 4.5 above its paper score, so leave it unset\nwhen reproducing the paper.\n\nThe judge defaults to Claude Sonnet 4.6 on the Anthropic API. To serve the judge yourself\n(a vision model: it reads screenshots, so add `--limit-mm-per-prompt '{\"image\":4}'` to its vLLM command):\n\n```\nevoweb run --name qwen-both \\\n  --model Qwen/Qwen3.5-397B-A17B-FP8 --model-port 8000 \\\n  --judge Qwen/Qwen3.5-397B-A17B-FP8 --judge-port 8001\n```\n\nA server on another machine: add `--model-host 10.0.0.5` (or `--judge-host`).\n\nSkip step 2. Give the model with its provider prefix and no port; the key comes from that provider's usual environment variable:\n\n```\nexport ANTHROPIC_API_KEY=<key>\nevoweb run --name haiku --model anthropic/claude-haiku-4-5\n\nexport OPENAI_API_KEY=<key>\nevoweb run --name gpt5 --model openai/gpt-5\n\nexport OPENROUTER_API_KEY=<key>\nevoweb run --name glm52 --model openrouter/z-ai/glm-5.2\n```\n\nAny other OpenAI-compatible API: `--model-url https://<host>/v1 --model <name>`,\nwith the key in `MODEL_API_KEY`.\n\n| Flag | Default |  | \n|---|---|---|\n| `--name` | required | run name; results go to `runs/<name>/` | \n| `--model` | required | served name, or `provider/model` for an API | \n| `--model-port` |  | port of a vLLM / OpenAI-compatible server | \n| `--model-host` | `localhost` | host of that server | \n| `--model-url` |  | full base URL instead of host + port | \n| `--judge` ,`--judge-port` ,`--judge-host` ,`--judge-url` | `anthropic/claude-sonnet-4-6` | same, for the judge | \n| `--apps` | `1-32` | subset, e.g. `1,4,6` | \n| `--parallel` | `8` | apps run at once | \n\nTo compare a run with the paper's runs of the same model (released for the four Anthropic\nmodels): `python3 -m evoweb.compare_to_released \"Haiku 4.5\" runs/haiku`.\n[`infra/README.md`](https://github.com/two-delta/EvoWeb/blob/main/infra/README.md) covers running many models at once on Kubernetes.\n\n```\nharness/        the agent (OpenHands) + judge (WebVoyager) runner, its Dockerfile and launchers\ndata/           the benchmark (evoweb_benchmark.jsonl) and its translations (multilingual/), both downloaded\n                from Hugging Face on use (evoweb/paths.py), the no-grounding\n                ablation set, the WebGen-Bench base apps and data_gen/ with the scripts that\n                generated them; data/README.md documents the format\nevoweb/         scoring (scoring.py holds the metric definitions, metrics.py the paper's aggregates) and the\n                runnable modules: score_run (score your run), compare_to_released (next to the\n                paper's runs), score_zip (unpack and score a full-trace archive)\nexamples_artifacts/\n  traces.zip         the paper's runs of the four Anthropic models (Fable 5, Opus 4.8, Sonnet 4.6, Haiku 4.5),\n                     three each, plus the multilingual Sonnet 4.6 and Haiku 4.5 runs: <run-id>/<app>/result.json\n                     (judge verdicts) + token logs; the other models' runs are not released\n  full_traces/       Fable 5's first run complete: all conversations, code, judge sessions, logs\n  examples.zip       one Opus 4.8 app with everything, screenshots included\n  reproductions/     our full Haiku re-run and its comparison with the paper\nablations/      the paper's DAG, grounding and multilingual analyses\ntests/          offline tests (scoring, datasets, harness) + the opt-in live reproduction test\ninfra/          running at scale on Kubernetes: launcher, job template, node pool, README\nmedia/          the method figure and the judge demo video\n@article{nukrai2026evoweb,\n  title   = {EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications},\n  author  = {Nukrai, David and Alon, Bar and Levy, Nativ and Atad, Ido Andrew and Cohen, Ido\n             and Zimerman, Itamar and Wolf, Lior and David, Eli and Naiman, Ilan},\n  journal = {arXiv preprint arXiv:2610.16402},\n  year    = {2026},\n  url     = {https://arxiv.org/abs/2610.16402}\n}\n```\n\nApache License 2.0; see [`LICENSE`](https://github.com/two-delta/EvoWeb/blob/main/LICENSE).", "url": "https://wpnews.pro/news/open-source-benchmark-for-long-horizon-web-app-coding", "canonical_source": "https://github.com/two-delta/evoWeb", "published_at": "2026-10-20 08:51:24+00:00", "updated_at": "2026-10-20 09:17:13.532913+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-tools", "developer-tools", "large-language-models"], "entities": ["EvoWeb", "arXiv:2610.16402", "WebVoyager", "Claude Sonnet 4.6", "Anthropic", "OpenAI", "OpenRouter", "vLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-source-benchmark-for-long-horizon-web-app-coding", "markdown": "https://wpnews.pro/news/open-source-benchmark-for-long-horizon-web-app-coding.md", "text": "https://wpnews.pro/news/open-source-benchmark-for-long-horizon-web-app-coding.txt", "jsonld": "https://wpnews.pro/news/open-source-benchmark-for-long-horizon-web-app-coding.jsonld"}}