{"slug": "swe-touch-benchmarking-coding-agents-when-users-touch-the-code", "title": "SWE-Touch: Benchmarking Coding Agents When Users Touch the Code", "summary": "Researchers introduced SWE-Touch, a benchmark evaluating coding agents' ability to repair software after users directly edit code, finding that most models' performance drops significantly when user edits conflict with the task. Claude Opus 4.8 retained 96.0% of its Vanilla-solved tasks under Counter-Edit, while Qwen3-Coder-480B retained only 60.8%, and MiniMax M2.7 dropped 13.8 percentage points. The benchmark includes tasks from SWE-Bench Pro and DeepSWE, with aggregate resolve rates declining by 4.9 and 3.4 percentage points respectively.", "body_md": "### The agent repairs the original checkout\n\nThe issue, repository, tools, verifier, and interaction budget follow the standard autonomous setting.\n\nSWE-Touch tests whether a coding agent can complete a software repair after a user has directly changed the code in their shared workspace.\n\nWhen a user contributes code during an ongoing task, can the agent **interpret that edit against the issue, repository, and tests** and still reach a correct solution?\n\nEach task is evaluated under matched conditions. The user edit is the only change to the live task state.\n\nThe issue, repository, tools, verifier, and interaction budget follow the standard autonomous setting.\n\nA small, plausible but task-conflicting edit and a contextual user message enter the same live workspace.\n\nResolve rates are percentages. Verified results report mean ± standard deviation.\n\nRetention is the fraction of tasks solved under Vanilla that remain solved under Counter-Edit.\n\nCounter-Edit minus Vanilla, percentage points\n\n| Model | Vanilla | Counter-Edit | Δ | Retention |\n|---|---|---|---|---|\n| Claude Opus 4.8 | 85.2 ± 1.8 | 83.3 ± 0.6 | −1.8 | 96.0 |\n| GPT 5.5 | 80.5 ± 1.0 | 79.2 ± 0.6 | −1.3 | 95.0 |\n| GLM 5.1 | 72.7 ± 2.0 | 68.3 ± 0.8 | −4.3 | 83.3 |\n| MiniMax M2.7 | 76.5 ± 1.5 | 62.7 ± 2.4 | −13.8 | 78.1 |\n| MiniMax M2.5 | 75.7 ± 3.3 | 66.2 ± 1.0 | −9.5 | 78.3 |\n| Qwen 3.7 Max | 75.2 ± 1.0 | 70.3 ± 0.8 | −4.8 | 90.3 |\n| Qwen3-Coder-480B | 57.2 ± 3.5 | 40.7 ± 1.0 | −16.5 | 60.8 |\n| Kimi K2.6 | 70.3 ± 2.0 | 64.3 ± 3.4 | −6.0 | 87.2 |\n| DeepSeek V4 Pro | 74.8 ± 0.8 | 63.8 ± 1.8 | −11.0 | 81.5 |\n\n**Finding.** The ordering changes substantially: Qwen3-Coder-480B and MiniMax M2.7 lose more than 13 points, while GPT 5.5 and Claude Opus 4.8 retain over 95% of their Vanilla-solved tasks.\n\nSelected SWE-Bench Pro and DeepSWE tasks use a longer budget and trajectory-relative delivery.\n\n| Model | SWE-Bench Pro | DeepSWE | ||||\n|---|---|---|---|---|---|---|\n| Vanilla | Counter | Δ | Vanilla | Counter | Δ | |\n| Claude Opus 4.8 | 68.0 | 68.0 | 0.0 | 56.0 | 46.0 | −10.0 |\n| GPT 5.5 | 38.0 | 38.0 | 0.0 | 64.0 | 56.0 | −8.0 |\n| GLM 5.1 | 43.1 | 32.8 | −10.3 | 19.4 | 16.8 | −2.5 |\n| MiniMax M2.7 | 30.6 | 24.6 | −6.0 | 2.2 | 2.2 | 0.0 |\n| MiniMax M2.5 | 32.6 | 24.6 | −8.0 | 0.0 | 0.0 | 0.0 |\n| Qwen 3.7 Max | 36.0 | 26.0 | −10.0 | 4.1 | 2.1 | −2.0 |\n| Qwen3-Coder-480B | 20.0 | 14.0 | −6.0 | 0.0 | 0.0 | 0.0 |\n| Kimi K2.6 | 50.0 | 48.0 | −2.0 | 18.0 | 12.0 | −6.0 |\n| DeepSeek V4 Pro | 34.0 | 32.0 | −2.0 | 4.1 | 2.0 | −2.1 |\n| Mean | 39.1 | 34.2 | −4.9 | 18.6 | 15.2 | −3.4 |\n\n**Finding.** The aggregate direction persists on both harder benchmarks, although the effect varies by model and task family.\n\n| Condition | GPT 5.5 | GLM 5.1 | M2.7 | Qwen 3.7 |\n|---|---|---|---|---|\n| Vanilla | 81.5 | 70.5 | 76.5 | 74.0 |\n| Message, K=3 | 79.5 | 73.0 | 76.5 | 77.0 |\n| Code edit, K=3 | 80.5 | 66.5 | 67.0 | 71.5 |\n| Both, K=1 | 78.5 | 72.0 | 64.5 | 71.5 |\n| Both, K=3 | 79.5 | 69.0 | 64.5 | 71.0 |\n| Both, K=5 | 78.0 | 69.0 | 60.0 | 69.0 |\n\nExploratory resolve rates on SWE-bench Verified. K is the maximum number of interventions.\n\n**Finding.** Messages alone do not reproduce the effect consistently. Direct changes to executable code are the stronger intervention in this comparison.\n\n| Benchmark | Reference lines / files | Counter lines / files |\n|---|---|---|\n| Verified | 13.3 / 1.20 | 7.0 / 1.04 |\n| Pro | 361.0 / 5.44 | 13.0 / 1.40 |\n| DeepSWE | 730.2 / 7.24 | 10.8 / 1.52 |\n\nMean changed lines and files per patch.\n\n**Finding.** Counter-Edits remain small even when the underlying reference repair spans hundreds of lines and several files.\n\nAgents sometimes remove or replace the inserted edit and still fail. Recognizing a problematic contribution does not guarantee that the model can restore the repository to a correct state.\n\nThe release separates reusable benchmark records from the Harbor-based evaluation implementation.\n\nRunner, schemas, user simulator, validation gates, and reproducibility commands.\n\nTask records, critical regions, Counter-Edits, and validation evidence.\n\nInstall the package, validate the release, and launch Vanilla or SWE-Touch evaluation.", "url": "https://wpnews.pro/news/swe-touch-benchmarking-coding-agents-when-users-touch-the-code", "canonical_source": "https://trae1oung.github.io/SWE-Touch/", "published_at": "2026-08-04 04:01:32+00:00", "updated_at": "2026-08-04 04:22:26.864610+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-tools"], "entities": ["SWE-Touch", "Claude Opus 4.8", "GPT 5.5", "GLM 5.1", "MiniMax M2.7", "Qwen3-Coder-480B", "DeepSeek V4 Pro", "SWE-Bench Pro"], "alternates": {"html": "https://wpnews.pro/news/swe-touch-benchmarking-coding-agents-when-users-touch-the-code", "markdown": "https://wpnews.pro/news/swe-touch-benchmarking-coding-agents-when-users-touch-the-code.md", "text": "https://wpnews.pro/news/swe-touch-benchmarking-coding-agents-when-users-touch-the-code.txt", "jsonld": "https://wpnews.pro/news/swe-touch-benchmarking-coding-agents-when-users-touch-the-code.jsonld"}}