New post

I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 for performing ablations. I found out that gpt 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was bit baffling at first but based on some further slop-investigation, I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the multimodality. In this post, I will briefly go through the methodology of the evaluation and the investigation itself.

Methodology

I was using GPT 6 luna xhigh for ablations. Cheapest model and higher rate limits. Did 3 runs on verifiers with codex harness with full network isolation. The issue images were downloaded in advance. We (Florian helped) also made new images with the git history object removed from the images provided by the benchmark because in an initial run, we found out that models were looking up “future” commits via the git object.

Evaluation runs

I did 3 runs - with native image input, customised OCR and no images mode.

This is possible to do in verifiers because of the interception server which allows to fetch the user message and tool related messages and replace stuff in them with what you want. I replaced the images with OCR or just placeholder text in the messages.

Here is what a task looks like in the placeholder (no images) run. This is lighthouse-4036 from the latest run, which Luna resolved in 5 model calls and about a minute. The issue has two screenshots; the model gets the raw <img> markup, a note that the images were removed, and one marker per image.

rel="noopener" audit should ignore href="mailto:x@y.com" links
<img width="699" alt="screen shot 2017-11-01 at 2 34 04 pm" src="https://user-images.githubusercontent.com/3506071/32291560-....png">

Following this suggestion for these `mailto:` links causes the links to break on Chrome for Android and iOS Safari. Unfortunately, we only found this issue in production :(

![screenshot_20171101-151030](https://user-images.githubusercontent.com/3506071/32292580-....png)

Reproduction:
<ul>
  <li><a href="mailto:sales@marketamplified.com" target="_blank">Sales</a></li>
  <li><a href="mailto:support@marketamplified.com" target="_blank" rel="noopener">Support</a></li>
</ul>
The support link fails as seen above on Chrome for Android. The sales link works fine and launches Gmail from Chrome for Android.

You cannot see the images in this message; each was removed and only its position is marked.

[Image 1 of 2]
[Image 2 of 2]

And the whole trace that followed, condensed (reasoning summaries in italics, tool calls in code):

rg -n "noopener|rel.*audit|target=.*_blank" .`` sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js plus the matching test file and the smoke fixture.javascript: only; I'll add a case-insensitive mailto: exemption and regression test."apply_patch:

-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));

mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js → 5 passing. Done. The screenshots (a Lighthouse audit flagging the mailto link, and a broken link on Android) never come up. The title alone says what to fix, and the existing javascript: exemption in the code says how.

Roughly same results were obtained in the three cases. Around 175-178 pass scores out of 480 which turns out to be roughly 37%.

To further check, if the model was looking up images via some other means that I was missing, I sacrificed my claude max sub and put on several sonnet 5.5 reviewers to go through the traces line by line to find out. The sonnet 5.5 reviewers pointed out

In 403 of 480 tasks (84%) the issue text fully specifies the bug and the images only illustrate it. Resolution rates are flat across the three runs in every category, so the images don't change outcomes even where they add information.

No task needed a repo image. 39 tasks were judged not solvable without images, but mostly because nothing could solve them. The gold patch depends on image information in 7 tasks and partly in 65.

required tasks

Here are four resolved tasks where you can judge for yourself. Each was solved in all runs, including the no-images run, and the no-images trace was checked for any image access (none). Two Fable 5.1 reviewers rated each one independently, one shown the gold patch and hidden tests, the other shown only the trace with the outcome stripped; all rated the images "not needed". The screenshot is the original from the GitHub issue, which the model never saw. Swipe or scroll sideways for the next task.

highlight.js-2958 — one-character regex fix; the issue labels the failing line FAIL in the code sample

What the screenshot shows: jsbin output with three XML tags highlighted and <a:FAIL xmlns:a="..." /> rendered as plain text.

What the text already says: title "(xml) line with single letter namespace prefix not highlighted since 10.4.0", plus a sample where the lines are literally named <abc:OK>, <ab:OK>, <a:FAIL>, <OK>.

What Luna did without the image: read src/languages/xml.js, noted "the tag-name regex requires a colon prefix to exceed one character", reproduced it with hljs.highlight on the sample, and changed one character. Identical to the upstream fix.

-  const TAG_NAME_RE = regex.concat(/[A-Z_]/, regex.optional(/[A-Z0-9_.-]+:/), /[A-Z0-9_.-]*/);
+  const TAG_NAME_RE = regex.concat(/[A-Z_]/, regex.optional(/[A-Z0-9_.-]*:/), /[A-Z0-9_.-]*/);

Reviewer: "The images only confirm visually what the issue title and the OK/FAIL labels in the sample code already state."

openlayers-14627 — the screenshots are a stack trace and a debugger view; the issue text names the cause and the function

What the screenshots show: a console TypeError: Cannot read properties of undefined (reading 'length') in the WebGL tile renderer, and a debugger d in fromEPSG4326 with y = NaN on the Math.tan line.

What the text already says: "applyTransform for empty extent will get a extent with NaN", a repro using createEmpty() + applyTransform(extent, getTransform('EPSG:4326','EPSG:3857')), and "it's caused by Math.tan will transform Infinity to NaN".

What Luna did without the images: read fromEPSG4326 (the exact code in the debugger screenshot), found isEmpty and createOrUpdateEmpty already in ol/extent.js, and added a guard. Identical to the upstream fix; 102 tests pass.

 export function applyTransform(extent, transformFn, dest, stops) {
+  if (isEmpty(extent)) {
+    return createOrUpdateEmpty(dest);
+  }

Reviewer: "Only incidental detail: the exact downstream stack trace and that applyTransform internally calls the transform with 8 coordinates. Neither is needed for the fix or tests."

eslint-14242 — a terminal screenshot; the issue text states the bug and the acceptable fix in two sentences

What the screenshot shows: a PowerShell session: eslint --max-warnings 3 exits 1 with "ESLint found too many warnings", then the same command with --quiet prints nothing and exits 0.

What the text already says: "Using the --max-warnings cli option is useless in combination with --quiet" and "Also acceptable: The exit code should be 1 when max-warnings is exceeded."

What Luna did without the image: read lib/cli.js, saw that quiet mode filters results before countErrors(results) runs, so the warning count is always 0, and moved the count above the filter. Same behaviour as the upstream fix; 83 tests pass.

+        const { errorCount, warningCount } = countErrors(results);
+
         if (options.quiet) {
             results = ESLint.getErrorResults(results);
         }
         if (await printResults(engine, results, options.format, options.outputFile)) {
-            const { errorCount, warningCount } = countErrors(results);

Reviewer: "The image adds only incidental detail: the exact command used, that the exit code is observed via $LASTEXITCODE, and the warning count 26. None of this is needed; the test uses its own fixture and threshold 5."

prism-1602 — a rendered-output screenshot; the text already contains the two-line reproducer

What the screenshot shows: Prism's output for the YAML snippet: the first "world" (followed by # test) is unhighlighted, the second is green.

What the text already says: "YAML string highlighting does not work when a string has a comment on the same line", with the snippet - "world" # test vs - "world".

What Luna did without the image: read components/prism-yaml.js, found the string pattern's lookahead only allows end-of-line, ,, ] or } after the closing quote, and added a trailing-comment case. The upstream fix adds \s*#; Luna's adds [ \t]+#. Hidden tests pass either way.

-  ...\2(?=[ \t]*(?:$|,|]|}))/m,
+  ...\2(?=[ \t]*(?:$|,|]|}|[ \t]+#))/m,

Reviewer: "The image only shows the symptom the text already states (first "world" unhighlighted, second highlighted)."

MMMU Pro is a good control: it is not agentic, just single-turn multiple-choice questions across college subjects, with no repo or tools to fall back on. In its standard split, each question comes with a diagram, chart or photo that carries information the text does not restate, so a model that cannot see the image should actually lose points there.

And it does. Same model, same effort, same interceptor, 1,730 questions of the MMMU-Pro standard split (10 options, so chance is 10%):

Every gap is large and paired (McNemar p < 1e-49 for all three pairs). Taking the images away costs 43 points here; on SWE-bench Multimodal it cost nothing. The OCR run also burns more than twice the reasoning tokens of the native run, which is the model trying to reconstruct the figure from text.

The drop is biggest where the image carries structure that words don't: Art goes from 42/53 to 3/53 without images, Finance from 59/60 to 15/60, Accounting from 51/58 to 10/58. It is smallest where the text already carries the question: Literature 43/52 → 38/52, Energy and Power 50/58 → 38/58.