SWE-bench Multimodal v2 is not multimodal A Prime Intellect residency researcher found that SWE-bench Multimodal v2, a 480-task benchmark built from GitHub issues in 11 JavaScript repos, barely tests visual reasoning: GPT-6 Luna (xhigh) scored similarly whether given images, OCR-based output, or no images at all, and Sonnet 5.5's score on a 60-task subset barely changed when images were removed. The researcher attributed the result to most tasks being solvable from issue text and code alone, since images mostly repeat the text and hidden tests rarely check image-only details such as colours or spacing, contrasting with MMMU-Pro where accuracy fell sharply without images. The findings were published after Vals AI released a SWE-bench Multimodal v2 leaderboard on October 14, and the researcher also rebuilt the harness to strip Git history after agents were caught retrieving upstream fixes from later commits. SWE-bench Multimodal v2 is not multimodal Contents Intro I was using SWE-bench Multimodal https://www.swebench.com/multimodal.html v2 to establish baselines for a project at the Prime Intellect residency. I found out that GPT-6 Luna xhigh performed similarly on the benchmark whether I used normal images or OCR-based output or no images at all. This felt unexpected for a multimodal benchmark. I wondered: does this mean the images are optional? After looking into the data, we found that most of the information the models needed was already in the issue text or derivable from the code. I also ran Sonnet 5.5 on a subset to check whether this was specific to Luna. Its score barely changed when I removed the images. In short, - Removing the images barely changes the score, for both Luna and Sonnet - Most tasks can be solved from the issue text and the code alone, since the images mostly repeat what the text already says - The hidden tests rarely check details found only in the images, like the colours or spacing in a design spec, which helps explain why removing images barely affects the score What changes when the images disappear? 480 tasks 60 tasks 1,730 questions 120 questions Scores barely move on SWE-bench Multimodal, while accuracy falls sharply on MMMU-Pro. Each pair uses the same tasks within that experiment. Vals AI released a SWE-bench Multimodal v2 leaderboard https://x.com/ValsAI/status/2110113232501567805 on October 14, so this felt like a good time to write this blog up and share our findings. Methodology SWE Multimodal v2 Benchmark details The benchmark has 480 tasks built from GitHub issues in 11 JavaScript repos Carbon alone accounts for 133 , most with screenshots. 39 of the 480 have no image in the issue text at all. It's worth noting that even these tasks can involve images, since the agent may take screenshots to check its work or open image files already in the repo. Most tasks are small fixes to UI components, map rendering, syntax highlighting or Lighthouse audits, things like a missing aria-label, a misaligned icon or a keyword highlighted in the wrong colour. They are fairly focused and short horizon. Luna xhigh spent a median of 5.5 minutes on each. Vals AI reports 5min 14 sec median time for Luna and 24m+ for Sonnet 5.5 at max reasoning. I find it interesting that this benchmark is not saturated given the nature of tasks. Tasks per repo click to expand Experiment details I evaluated two models on SWE-bench Multimodal v2 and MMMU-Pro: | Model | SWE-bench Multimodal v2 | MMMU-Pro | Input conditions | |---|---|---|---| | GPT-6 Luna xhigh | All 480 tasks | All 1,730 questions | Images, OCR, no images | | Sonnet 5.5 max | 60-task subset | 120-question subset | Images, no images | For SWE-bench Multimodal, rollouts ran with full network isolation, and grading took place in a separate sandbox. Issue images were downloaded beforehand so they could be provided without network access. Removing Git history In the traces of an initial run, we caught agents: - Searching Git history for commits associated with the reported issue or an upstream fix. - Inspecting later commits and their diffs to retrieve the implementation that fixed the issue. - Incorporating code from those upstream fixes into their submitted patches. As you can see, sometimes the agents take 'learn from the codebase' too literally. This is a known SWE-bench reward hack, documented about a year ago https://bayes.net/swebench-hack/ , and we were surprised to still see this in the benchmark images. Checking out an older commit takes the files back in time, but it doesn't erase newer commits from .git . So the agent may start with the buggy code while the actual upstream fix is still sitting in the same repository. It can go on an archaeological expedition through Git refs or stored objects, find the fix, and copy it instead of solving the issue. To prevent this, we rebuilt the benchmark images to keep only the task's base commit and the objects needed to reconstruct its files. We then copied the cleaned filesystem into a fresh image so the removed history couldn't survive in older image layers. GPT-6 Luna xhigh runs I was using GPT-6 Luna xhigh for ablations. There were three full runs on all 480 tasks, one per condition, on verifiers with the Codex harness. The three conditions were native image input, a custom OCR-based representation of each image and no images mode. This is possible to do in verifiers because of the interception server https://www.primeintellect.ai/blog/verifiers-v1 :~:text=The%20core%20abstraction%20that%20makes%20the%20v1%20design%20work%20is%20the%20notion%20of%20an%20interception%20server.%20The . It sits between the harness and the model API, letting us rewrite messages before the model sees them. I used it to replace images in both user messages and tool outputs with OCR text or placeholders. In OCR mode, each image is replaced by the text that OCR reads from it, with every word placed on a character grid at its position in the image and tagged with its colour. A header gives the image size and main colours, and a short list underneath describes shapes that are not text, such as icons, outlines and lines. The interceptor rewrites every image in each model request, in the issue and in tool outputs alike, before it reaches the model. Sample trace Issue message of task lighthouse-4036, which has 2 screenshots, as the model received it in each of the three modes. Swipe or scroll sideways, or jump to with images sample-image-mode · with OCR sample-ocr-mode · no images sample-no-images-mode With images 1 / 3 → With OCR ← 2 / 3 → No images ← 3 / 3 Full trace in the no images mode. The full trace click to expand · resolved · 5 model calls · 7 commands · 60 s I need to modify the repository and inspect it further. The user mentioned that the rel=noopener audit should ignore mailto links, which seems to be related to a request issue. I should identify the specific project involved. First, I'll start with a preamble to explain what I'm doing and then proceed with the relevant commands. This way, the user will know I'm addressing their concerns directly I’ll locate the noopener audit and its tests, then make mailto: links exempt. pwd && rg -n "noopener|rel. audit|target=. blank" . --glob ' node modules' --glob ' vendor' Chunk ID: 159d7e Wall time: 0.0000 seconds Process exited with code 0 Original token count: 2232 Output: /testbed ./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:50: 'external-anchors-use-rel-noopener': { ./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:301: 'external-anchors-use-rel-noopener': { ./docs/understanding-results.md:187:| result | Object | The actual audit result, a copy of the audit object found in audits audits . NOTE: this property will likely be removed in upcoming releases; use the id property to lookup the result in the audits property. | ./lighthouse-cli/test/fixtures/dobetterweb/dbw tester.html:22:< -- we use a relative protocol url to test that new URL resource in the critical-request-chains.js audit can handle it -- ./lighthouse-cli/test/fixtures/dobetterweb/dbw tester.html:103: