SWE-bench Multimodal v2 is not multimodal

Contents

Intro #

I was using SWE-bench Multimodal v2 to establish baselines for a project at the Prime Intellect residency. I found out that GPT-6 Luna (xhigh) performed similarly on the benchmark whether I used normal images or OCR-based output or no images at all. This felt unexpected for a multimodal benchmark. I wondered: does this mean the images are optional?

After looking into the data, we found that most of the information the models needed was already in the issue text or derivable from the code. I also ran Sonnet 5.5 on a subset to check whether this was specific to Luna. Its score barely changed when I removed the images. In short,

• Removing the images barely changes the score, for both Luna and Sonnet
• Most tasks can be solved from the issue text and the code alone, since the images mostly repeat what the text already says
• The hidden tests rarely check details found only in the images, like the colours or spacing in a design spec, which helps explain why removing images barely affects the score

What changes when the images disappear?

480 tasks

60 tasks

1,730 questions

120 questions

Scores barely move on SWE-bench Multimodal, while accuracy falls sharply on MMMU-Pro. Each pair uses the same tasks within that experiment.

Vals AI released a SWE-bench Multimodal v2 leaderboard on October 14, so this felt like a good time to write this blog up and share our findings.

Methodology #

SWE Multimodal v2 Benchmark details

The benchmark has 480 tasks built from GitHub issues in 11 JavaScript repos (Carbon alone accounts for 133), most with screenshots. 39 of the 480 have no image in the issue text at all. It's worth noting that even these tasks can involve images, since the agent may take screenshots to check its work or open image files already in the repo.

Most tasks are small fixes to UI components, map rendering, syntax highlighting or Lighthouse audits, things like a missing aria-label, a misaligned icon or a keyword highlighted in the wrong colour. They are fairly focused and short horizon. Luna xhigh spent a median of 5.5 minutes on each. Vals AI reports 5min 14 sec median time for Luna and 24m+ for Sonnet 5.5 at max reasoning. I find it interesting that this benchmark is not saturated given the nature of tasks.

Tasks per repo (click to expand) #

Experiment details

I evaluated two models on SWE-bench Multimodal v2 and MMMU-Pro:

Model	SWE-bench Multimodal v2	MMMU-Pro	Input conditions
GPT-6 Luna (xhigh)	All 480 tasks	All 1,730 questions	Images, OCR, no images
Sonnet 5.5 (max)	60-task subset	120-question subset	Images, no images

For SWE-bench Multimodal, rollouts ran with full network isolation, and grading took place in a separate sandbox. Issue images were downloaded beforehand so they could be provided without network access.

Removing Git history

In the traces of an initial run, we caught agents:

• Searching Git history for commits associated with the reported issue or an upstream fix.
• Inspecting later commits and their diffs to retrieve the implementation that fixed the issue.
• Incorporating code from those upstream fixes into their submitted patches.

As you can see, sometimes the agents take 'learn from the codebase' too literally. This is a known SWE-bench reward hack, documented about a year ago, and we were surprised to still see this in the benchmark images.

Checking out an older commit takes the files back in time, but it doesn't erase newer commits from .git. So the agent may start with the buggy code while the actual upstream fix is still sitting in the same repository. It can go on an archaeological expedition through Git refs or stored objects, find the fix, and copy it instead of solving the issue.

To prevent this, we rebuilt the benchmark images to keep only the task's base commit and the objects needed to reconstruct its files. We then copied the cleaned filesystem into a fresh image so the removed history couldn't survive in older image layers.

GPT-6 Luna xhigh runs

I was using GPT-6 Luna xhigh for ablations. There were three full runs on all 480 tasks, one per condition, on verifiers with the Codex harness.

The three conditions were native image input, a custom OCR-based representation of each image and no images mode.

This is possible to do in verifiers because of the interception server. It sits between the harness and the model API, letting us rewrite messages before the model sees them. I used it to replace images in both user messages and tool outputs with OCR text or placeholders.

In OCR mode, each image is replaced by the text that OCR reads from it, with every word placed on a character grid at its position in the image and tagged with its colour. A header gives the image size and main colours, and a short list underneath describes shapes that are not text, such as icons, outlines and lines.

The interceptor rewrites every image in each model request, in the issue and in tool outputs alike, before it reaches the model.

Sample trace

Issue message of task lighthouse-4036, which has 2 screenshots, as the model received it in each of the three modes.

Swipe or scroll sideways, or jump to with images · with OCR · no images

With images 1 / 3 →

With OCR← 2 / 3 →

No images← 3 / 3

Full trace in the no images mode.

The full trace (click to expand) · resolved · 5 model calls · 7 commands · 60 s #

I need to modify the repository and inspect it further. The user mentioned that the rel=noopener audit should ignore mailto links, which seems to be related to a request issue. I should identify the specific project involved. First, I'll start with a preamble to explain what I'm doing and then proceed with the relevant commands. This way, the user will know I'm addressing their concerns directly!

I’ll locate the noopener audit and its tests, then make mailto: links exempt.

pwd && rg -n "noopener|rel.*audit|target=.*_blank" . --glob '!node_modules' --glob '!vendor'
Chunk ID: 159d7e
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 2232
Output:
/testbed
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:50:      'external-anchors-use-rel-noopener': {
./lighthouse-cli/test/smokehouse/dobetterweb/dbw-expectations.js:301:      'external-anchors-use-rel-noopener': {
./docs/understanding-results.md:187:| result | `Object` | The actual audit result, a copy of the audit object found in [audits](#audits). *NOTE: this property will likely be removed in upcoming releases; use the `id` property to lookup the result in the `audits` property.* |
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:22:<!-- we use a relative protocol url to test that `new URL(resource)` in the critical-request-chains.js audit can handle it -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:103:<template id="noopener-links-tmpl">
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:104:  <!-- FAIL - does not use rel="noopener" to open external link -->
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:105:  <a href="https://www.google.com/" target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:106:  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:108:  <a target="_blank">external link</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:110:  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:112:  <a href="./doesnotexist" target="_blank">internal link is ok</a>
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:294:  stampTemplate('noopener-links-tmpl', document.body);
./lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html:374:  if (params.has('relnoopener')) {
./changelog.md:24:* audit: Ignore `href=javascript:.*` for `rel=noopener` audit ([#3574](https://github.com/googlechrome/lighthouse/pull/3574))
./changelog.md:298:Thanks to Katie Hempenius, Artur M, Kevin Farrugia, Piper Chester, Robin Drexler, and Mike Deverell. Due to the effort of these developers we now have an audit for checking for paste-blocked password fields, unified usage of toLocaleString(), a correct gulp recipe, a well-functioning chrome-launcher module, and improved clarity in the audit test descriptions.  **We truly appreciate all these improvements!**
./changelog.md:882:- **Audit:** Handle invalid URLs in `external-anchors-use-rel-noopener` audit - #1358
./changelog.md:927:- **Audit:** handle anchor tags with no href in 'external-anchors-use-rel-noopener' audit - #1238
./changelog.md:1030:- **Audit:** identify `noopener` when in list of link types in `rel` - #1035
./changelog.md:1089:- **Audit:** Add DBW audit for `rel=noopener` on external links - #912
./lighthouse-core/test/results/sample_v2.json:1198:      "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:1533:      "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:2135:    "external-anchors-use-rel-noopener": {
./lighthouse-core/test/results/sample_v2.json:2146:            "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2152:            "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2158:            "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:2163:      "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:2165:      "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:2166:      "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/results/sample_v2.json:5125:            "helpText": "When a button doesn't have an accessible name, screen readers announce it as \"button\", making it unusable for users who rely on screen readers. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/button-name)."
./lighthouse-core/test/results/sample_v2.json:5580:            "helpText": "A value greater than 0 implies an explicit navigation ordering. Although technically valid, this often creates frustrating experiences for users who rely on assistive technologies. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/tabindex)."
./lighthouse-core/test/results/sample_v2.json:6392:          "id": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6405:                  "url": "<a href=\"https://www.google.com/\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6411:                  "url": "<a target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6417:                  "url": "<a href=\"./doesnotexist\" target=\"_blank\">"
./lighthouse-core/test/results/sample_v2.json:6422:            "name": "external-anchors-use-rel-noopener",
./lighthouse-core/test/results/sample_v2.json:6424:            "description": "Opens external anchors using `rel=\"noopener\"`",
./lighthouse-core/test/results/sample_v2.json:6425:            "helpText": "Open new tabs using `rel=\"noopener\"` to improve performance and prevent security vulnerabilities. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).",
./lighthouse-core/test/report/v2/renderer/dom-test.js:82:          'Some <a rel="noopener" target="_blank" href="https://example.com/foo">link</a>. ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:83:          '<a rel="noopener" target="_blank" href="http://example.com/">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/dom-test.js:87:          '<a rel="noopener" target="_blank" href="https://example.com/foo">link</a>',
./lighthouse-core/test/report/v2/renderer/dom-test.js:93:          '<a rel="noopener" target="_blank" href="https://example.com/foo"> Link </a> ' +
./lighthouse-core/test/report/v2/renderer/dom-test.js:115:          'good. <a rel="noopener" target="_blank" href="https://dequeuniversity.com/rules/axe/2.2/td-headers-attr">Learn more</a>.');
./lighthouse-core/test/report/v2/renderer/details-renderer-test.js:199:      assert.equal(el.rel, 'noopener');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:9:  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
./lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:16:describe('External anchors use rel="noopener"', () => {
./lighthouse-core/report/v2/templates.html:324:      <div class="lh-metadata__results">Results for: <a href="" class="lh-metadata__url" target="_blank" rel="noopener"><!-- fill me --></a></div>
./lighthouse-core/report/v2/templates.html:386:      <a href="https://github.com/GoogleChrome/Lighthouse/issues" target="_blank" rel="noopener">File an issue</a>
./lighthouse-core/report/v2/renderer/dom.js:108:        a.rel = 'noopener';
./lighthouse-core/report/v2/renderer/details-renderer.js:107:    a.rel = 'noopener';
./lighthouse-core/gather/gatherers/dobetterweb/anchors-with-no-rel-noopener.js:19:      const selector = 'a[target="_blank"]:not([rel~="noopener"])';
./lighthouse-core/config/default.js:30:      'dobetterweb/anchors-with-no-rel-noopener',
./lighthouse-core/config/default.js:153:    'dobetterweb/external-anchors-use-rel-noopener',
./lighthouse-core/config/default.js:340:        {id: 'external-anchors-use-rel-noopener', weight: 1},
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:17:      name: 'external-anchors-use-rel-noopener',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:18:      description: 'Opens external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:19:      failureDescription: 'Does not open external anchors using `rel="noopener"`',
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:20:      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
./lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:22:          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',

I need to take a closer look at the instructions and the files available. I'm thinking maybe there are root AGENTS present in the current working directory, but I’m not sure. The developer mentioned that the root AGENT is included, but I can’t seem to find anything. So, I guess I’ll need to use ls or find to search for any applicable files or indications related to the AGENTS. Let's see what I can uncover!

rg --files -g 'AGENTS.md' -g '!node_modules' -g '!vendor' /testbed / 2>/dev/null | head -80
Chunk ID: 065d07
Wall time: 0.0480 seconds
Process exited with code 0
Original token count: 0
Output:
sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js && printf '\n--- TEST ---\n' && sed -n '1,280p' lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
Chunk ID: 338291
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1478
Output:
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const URL = require('../../lib/url-shim');
const Audit = require('../audit');
​
class ExternalAnchorsUseRelNoopenerAudit extends Audit {
  /**
   * @return {!AuditMeta}
   */
  static get meta() {
    return {
      name: 'external-anchors-use-rel-noopener',
      description: 'Opens external anchors using `rel="noopener"`',
      failureDescription: 'Does not open external anchors using `rel="noopener"`',
      helpText: 'Open new tabs using `rel="noopener"` to improve performance and ' +
          'prevent security vulnerabilities. ' +
          '[Learn more](https://developers.google.com/web/tools/lighthouse/audits/noopener).',
      requiredArtifacts: ['URL', 'AnchorsWithNoRelNoopener'],
    };
  }
​
  /**
   * @param {!Artifacts} artifacts
   * @return {!AuditResult}
   */
  static audit(artifacts) {
    let debugString;
    const pageHost = new URL(artifacts.URL.finalUrl).host;
    // Filter usages to exclude anchors that are same origin
    // TODO: better extendedInfo for anchors with no href attribute:
    // https://github.com/GoogleChrome/lighthouse/issues/1233
    // https://github.com/GoogleChrome/lighthouse/issues/1345
    const failingAnchors = artifacts.AnchorsWithNoRelNoopener
      .filter(anchor => {
        try {
          return new URL(anchor.href).host !== pageHost;
        } catch (err) {
          debugString = 'Lighthouse was unable to determine the destination ' +
              'of some anchor tags. If they are not used as hyperlinks, ' +
              'consider removing the _blank target.';
          return true;
        }
      })
      .filter(anchor => {
        // Ignore href's that are not real links
        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
      })
      .map(anchor => {
        return {
          href: anchor.href || 'Unknown',
          target: anchor.target || '',
          rel: anchor.rel || '',
          url: '<a' +
              (anchor.href ? ` href="${anchor.href}"` : '') +
              (anchor.target ? ` target="${anchor.target}"` : '') +
              (anchor.rel ? ` rel="${anchor.rel}"` : '') + '>',
        };
      });
​
    const headings = [
      {key: 'href', itemType: 'url', text: 'URL'},
      {key: 'target', itemType: 'text', text: 'Target'},
      {key: 'rel', itemType: 'text', text: 'Rel'},
    ];
​
    const details = Audit.makeTableDetails(headings, failingAnchors);
​
    return {
      rawValue: failingAnchors.length === 0,
      extendedInfo: {
        value: failingAnchors,
      },
      details,
      debugString,
    };
  }
}
​
module.exports = ExternalAnchorsUseRelNoopenerAudit;
​
--- TEST ---
/**
 * @license Copyright 2016 Google Inc. All Rights Reserved.
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 */
'use strict';
​
const ExternalAnchorsAudit =
  require('../../../audits/dobetterweb/external-anchors-use-rel-noopener.js');
const assert = require('assert');
​
const URL = 'https://google.com/test';
​
/* eslint-env mocha */
​
describe('External anchors use rel="noopener"', () => {
  it('passes when links are from same hosts as the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://google.com/test'},
        {href: 'https://google.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
    assert.equal(auditResult.details.items.length, 0);
  });
​
  it('fails when links are from different hosts than the page host', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'https://example.com/test'},
        {href: 'https://example.com/test1'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 2);
    assert.equal(auditResult.details.items.length, 2);
  });
​
  it('handles links with no href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: ''},
        {href: 'http://'},
        {href: 'http:'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, false);
    assert.equal(auditResult.details.items.length, 3);
    assert.equal(auditResult.details.items.length, 3);
    assert.ok(auditResult.debugString, 'includes debugString');
  });
​
  it('does not fail for links with javascript in href attribute', () => {
    const auditResult = ExternalAnchorsAudit.audit({
      AnchorsWithNoRelNoopener: [
        {href: 'javascript:void(0)'},
        {href: 'JAVASCRIPT:void(0)'},
      ],
      URL: {finalUrl: URL},
    });
    assert.equal(auditResult.rawValue, true);
    assert.equal(auditResult.details.items.length, 0);
  });
});
sed -n '90,125p' lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html && cat package.json | head -100 && git status --short
Chunk ID: 6b5fe4
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 1393
Output:
  </style>
</template>
​
</head>
<body>
​
<div>
  <h2>Do better web tester page</h2>
  <span>Hi there!</span>
</div>
​
<section id="touchmove-section">touchmove section</section>
​
<template id="noopener-links-tmpl">
  <!-- FAIL - does not use rel="noopener" to open external link -->
  <a href="https://www.google.com/" target="_blank">external link</a>
  <!-- FAIL - does not use rel="noopener" and has no href attribute, giving an
    href value of '' when read, which will throw in a `new URL('')` constructor -->
  <a target="_blank">external link</a>
  <!-- PASS -->
  <a href="https://www.google.com/" target="_blank" rel="noopener nofollow">external link that uses rel noopener and another unrelated rel attribute</a>
  <!-- PASS -->
  <a href="./doesnotexist" target="_blank">internal link is ok</a>
</template>
​
<template id="password-inputs-can-be-pasted-into">
  <!-- FAIL - calls preventDefault on event -->
  <input type="password" onpaste="event.preventDefault();" />
  <!-- PASS -->
  <input type="password" />
  <!-- FAIL - returns false in event handler -->
  <input type="password" onpaste="return false;" />
​
</template>
​
<!-- FAIL(image-aspect-ratio): image is naturally 480x318 -->
{
  "name": "lighthouse",
  "version": "2.6.0",
  "description": "Lighthouse",
  "main": "./lighthouse-core/index.js",
  "bin": {
    "lighthouse": "./lighthouse-cli/index.js",
    "chrome-debug": "./chrome-launcher/manual-chrome-launcher.js"
  },
  "engines": {
    "node": ">=6"
  },
  "scripts": {
    "install-all": "npm-run-posix-or-windows install-all:task",
    "install-all:task": "yarn & yarn install-extension & yarn install-viewer & wait",
    "install-all:task:windows": "yarn && yarn install-extension && yarn install-viewer",
    "install-extension": "cd ./lighthouse-extension && yarn install",
    "install-viewer": "cd ./lighthouse-viewer && yarn install",
    "build-all": "npm-run-posix-or-windows build-all:task",
    "build-all:task": "yarn build-extension & yarn build-viewer & wait",
    "build-all:task:windows": "yarn build-extension && yarn build-viewer",
    "build-extension": "cd ./lighthouse-extension && yarn build",
    "build-viewer": "cd ./lighthouse-viewer && yarn build",
    "clean": "rimraf *.report.html *.report.dom.html *.report.json *.screenshots.html *.devtoolslog.json *.trace.json || true",
    "lint": "[ \"$CI\" = true ] && eslint --quiet -f codeframe . || eslint .",
    "smoke": "bash lighthouse-cli/test/smokehouse/run-all-tests.sh",
    "coverage": "istanbul cover -x \"**/third_party/**\" _mocha -- $(find */test -name '*-test.js') --timeout 10000 --reporter progress --report lcovonly",
    "coveralls": "yarn coverage && cat ./coverage/lcov.info | coveralls && ./node_modules/codecov/bin/codecov",
    "start": "node ./lighthouse-cli/index.js",
    "test": "yarn lint --quiet && yarn unit && yarn type-check && yarn closure",
    "unit-core": "bash lighthouse-core/scripts/run-mocha.sh --core",
    "unit-cli": "bash lighthouse-core/scripts/run-mocha.sh --cli",
    "unit-viewer": "bash lighthouse-core/scripts/run-mocha.sh --viewer",
    "unit": "bash lighthouse-core/scripts/run-mocha.sh --default",
    "core-unit": "yarn unit-core",
    "cli-unit": "yarn unit-cli",
    "viewer-unit": "yarn unit-viewer",
    "closure": "cd lighthouse-core && node closure/closure-type-checking.js",
    "devtools": "bash lighthouse-core/scripts/roll-to-devtools.sh",
    "compile-devtools": "bash lighthouse-core/scripts/compile-against-devtools.sh",
    "watch": "bash lighthouse-core/scripts/run-mocha.sh --watch",
    "chrome": "node chrome-launcher/manual-chrome-launcher.js",
    "fast": "yarn start -- --disable-device-emulation --disable-cpu-throttling --disable-network-throttling",
    "smokehouse": "node lighthouse-cli/test/smokehouse/smokehouse.js",
    "deploy-viewer": "cd lighthouse-viewer && gulp deploy",
    "bundlesize": "bundlesize",
    "plots-smoke": "bash plots/test/smoke.sh",
    "changelog": "conventional-changelog --config ./build/changelog-generator/index.js --infile changelog.md --same-file",
    "type-check": "tsc -p ."
  },
  "devDependencies": {
    "@types/configstore": "^2.1.1",
    "@types/inquirer": "^0.0.35",
    "@types/node": "*",
    "@types/opn": "^3.0.28",
    "@types/update-notifier": "^1.0.2",
    "@types/yargs": "^8.0.2",
    "babel-core": "^6.16.0",
    "bundlesize": "^0.14.4",
    "codecov": "^2.2.0",
    "commitizen": "^2.9.6",
    "conventional-changelog-cli": "^1.3.4",
    "coveralls": "^2.11.9",
    "cz-customizable": "^5.2.0",
    "eslint": "^4.8.0",
    "eslint-config-google": "^0.9.1",
    "google-closure-compiler": "^20170521.0.0",
    "gulp": "^3.9.1",
    "gulp-replace": "^0.5.4",
    "gulp-util": "^3.0.7",
    "istanbul": "^0.4.3",
    "jsdom": "^9.12.0",
    "mocha": "^3.2.0",
    "npm-run-posix-or-windows": "^2.0.2",
    "typescript": "^2.6.1",
    "zone.js": "^0.7.3"
  },
  "dependencies": {
    "axe-core": "2.4.1",
    "chrome-devtools-frontend": "1.0.422034",
    "chrome-launcher": "0.8.1",
    "configstore": "^3.1.1",
    "devtools-timeline-model": "1.1.6",
    "http-link-header": "^0.8.0",
    "inquirer": "^3.3.0",
    "jpeg-js": "0.1.2",
    "js-library-detector": "^4.0.0",
    "json-stringify-safe": "5.0.1",
    "lighthouse-logger": "^1.0.0",
    "metaviewport-parser": "0.1.0",
    "mkdirp": "0.5.1",
    "opn": "4.0.2",
    "parse-cache-control": "1.0.1",
    "raven": "^2.2.1",
    "rimraf": "^2.6.1",
    "semver": "^5.3.0",
    "speedline": "1.3.0",
    "update-notifier": "^2.1.0",
    "whatwg-url": "^6.3.0",
    "ws": "3.3.2",

I need to update the audit implementation to specifically exclude mailto links. The current filter ordering is causing issues since it's checking new URL mailto cases, and I need a way to parse this correctly. I should ensure JavaScript handling is included too. Maybe I can separate this with conditions to ignore actual links while still ensuring that mailto links aren't parsed incorrectly. Overall, I'm focused on simplifying the logic to avoid unnecessary URL parsing, while still being thorough in the audit checks.Updating Mailto Protocol Handling

It seems the new URL for mailto likely returns an empty host, which isn’t producing an exception. I should implement an extension in the predicate and set up a unit test that checks mixed-case mailto links, since that’s likely a requirement. There might be a need to create separate tests for mailto specifically. Also, I should consider adding a changelog entry for clarity and check if existing tests cover this. Focused changes are the goal, so I'll aim for simplicity while ensuring thorough testing.

The audit currently exempts javascript: only; I’ll add a case-insensitive mailto: exemption and regression test.

apply_patch <<'PATCH'
*** Begin Patch
*** Update File: lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
*** Update File: lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@
   it('does not fail for links with javascript in href attribute', () => {
     const auditResult = ExternalAnchorsAudit.audit({
       AnchorsWithNoRelNoopener: [
         {href: 'javascript:void(0)'},
@@
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
*** End Patch
PATCH
Wall time: 0.0000 seconds
Output:
Exit code: 0
Wall time: 0 seconds
Output:
Success. Updated the following files:
M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js

Patch is in; I’ll run the focused audit test and inspect the final diff.

./node_modules/.bin/mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js --timeout 10000
Chunk ID: 648f78
Wall time: 0.2552 seconds
Process exited with code 0
Original token count: 92
Output:
​
​
  External anchors use rel="noopener"
    ✓ passes when links are from same hosts as the page host
    ✓ fails when links are from different hosts than the page host
    ✓ handles links with no href attribute
    ✓ does not fail for links with javascript in href attribute
    ✓ does not fail for links with mailto in href attribute
​
​
  5 passing (20ms)
git diff --check && git diff -- lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js && git status --short
Chunk ID: 59373a
Wall time: 0.0000 seconds
Process exited with code 0
Original token count: 510
Output:
diff --git a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
index 296f94d..21b1147 100644
--- a/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
+++ b/lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
@@ -47,8 +47,9 @@ class ExternalAnchorsUseRelNoopenerAudit extends Audit {
         }
       })
       .filter(anchor => {
-        // Ignore href's that are not real links
-        return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');
+        // Ignore href's that are not real links or do not navigate to a page.
+        const href = (anchor.href || '').toLowerCase();
+        return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));
       })
       .map(anchor => {
         return {
diff --git a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
index c89aab4..ea8d777 100644
--- a/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
+++ b/lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js
@@ -66,4 +66,16 @@ describe('External anchors use rel="noopener"', () => {
     assert.equal(auditResult.rawValue, true);
     assert.equal(auditResult.details.items.length, 0);
   });
+
+  it('does not fail for links with mailto in href attribute', () => {
+    const auditResult = ExternalAnchorsAudit.audit({
+      AnchorsWithNoRelNoopener: [
+        {href: 'mailto:sales@example.com'},
+        {href: 'MAILTO:support@example.com'},
+      ],
+      URL: {finalUrl: URL},
+    });
+    assert.equal(auditResult.rawValue, true);
+    assert.equal(auditResult.details.items.length, 0);
+  });
 });
 M lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js
 M lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js

The unit test shows it can handle both lower and uppercase, and I have 5 validation tests passing, which is great! I don't see the need for any more tests at this point. However, I'm a bit concerned that the status output indicates only two modifications, but I'll keep an eye on that. Overall, it seems I can finalize with up to 10 lines.

Updated the rel="noopener" audit to ignore mailto: links, case-insensitively, in lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js:50. Added lowercase and uppercase regression coverage in lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js:69.

Validation: all 5 focused audit tests pass.

The model didn't need the screenshots here (a Lighthouse audit flagging the mailto link, and a broken link on Android). The issue title already says to ignore mailto links, and the audit already skipped javascript: links, so it just added mailto: next to that.

Sonnet 5.5 max runs

It was interesting to see that Claude models were ahead of all OpenAI models on the leaderboard, which made me and worry a bit. Maybe Luna just doesn't make good use of images, and a model that does would show a gap. I shared my concern with Florian and he suggested we could run Sonnet 5.5 as well.

Vals AI's SWE-bench Multimodal leaderboard as of October 20.

Sonnet 5.5 at max reasoning is slow and pricey, so I picked a 60-task subset. Those 60 tasks cost about $170, more than the whole 480-task Luna run (about $110, inference plus sandboxes).

Sonnet runs at max effort (adaptive thinking) in the Claude Code harness, on 60 randomly sampled tasks spread over all 11 repos. The same tasks were used for both runs, native images and no images, with one rollout per task. No OCR run this time.

The rest of the setup was the same as the Luna runs, except for the agent timeout limit, which was raised from 20 to 50 minutes since Vals AI reports Sonnet at max effort taking 24m+ per task.

Results #

GPT-6 Luna xhigh (480 tasks)

All three runs resolved 175 to 178 of 480 tasks (36.5 to 37.1%). This is in line with the 37.5% that Vals AI's SWE-bench Multimodal leaderboard reports for GPT-6 Luna on max effort (as of October 2026), even though they run it in a different harness (mini-swe-agent).

Run	Resolved	Rate
Native images	177 / 480	36.9%
OCR text	175 / 480	36.5%
No images	178 / 480	37.1%

Pair	Both resolved	Only the first	Only the second	Both failed
Images vs OCR	163	14	12	291
Images vs no images	167	10	11	292
OCR vs no images	161	14	17	288

157 tasks were resolved in all three runs and 284 failed in all three, so 441 of 480 (92%) got the same result every time. The remaining 39 change outcomes, with gains and losses roughly balancing out (images vs no images is −0.2 points, 95% CI −2.1 to +1.7, McNemar p = 1.0).

Observations

Some observations by my trace police (Sonnet 5.5, sometimes Opus 5.5 reviewer agents) after going through the traces:

• In the native and no-image runs, Luna's network lookups all failed, and most went looking for the upstream issue, PR or commit rather than the images. The only images it opened itself were rendering-test baselines from the repo.
• In the no-image run, Luna only now and then mentions the missing images in its thinking. When it does, it works around them ("Since I can't see the image, I should infer what the expected outcome is" on carbon-4431, "I can't access screenshot URLs since images are removed, and I respect that" on lighthouse-9151), and once in a while it guesses what an image shows. Otherwise it gets on with the issue text and the code.
• In the native-image run, Luna mostly ignores the images. Going by its reasoning summaries, it cites a specific visual detail in well under half the tasks that have images, and those tasks resolve just as often in the runs without images.

Sonnet 5.5 max (60-task subset)

Run	Sonnet 5.5 max	GPT-6 Luna xhigh on the same 60
Native images	21 / 60 (35.0%)	14 / 60 (23.3%)
OCR text	not run	10 / 60 (16.7%)
No images	24 / 60 (40.0%)	14 / 60 (23.3%)

19 tasks were resolved in both Sonnet runs and 34 failed in both. 2 flipped towards images and 5 towards no images (−5 points, 95% CI roughly −15 to +5, McNemar p = 0.45). With only 60 tasks, this shows no sign of an image benefit but can't rule out a moderate one.

Median thinking per task was 69k tokens with images and 67k without them. The no-image runs took a few more tool calls (median 82 vs 67) and a bit longer (18 vs 14 minutes). In contrast, Sonnet used 3x the output tokens on MMMU-Pro when images were gone.

Observations

Some observations by my trace police after going through the traces:

• In 27 of 60 no-image tasks Sonnet tried to get at the images or the upstream issue (curl, WebFetch, opening png files), with no luck.
• Funnily, the thinking often says "I can't see the image" and then guesses what it shows anyway. The guesses mostly restated the issue text, so they often matched the real image (27 of 46 guesses).Three guesses about images Sonnet never saw (click to expand)Thinking quotes are Anthropic's summarized thinking, not the raw chain of thought. All three guesses match the real image, and all three tasks still failed, with and without images. lighthouse-9451 The issue says "A page like the below should be visible during the offline pass, not the desktop version of the site." Looking at the screenshot description, the expected behavior is that during the offline pass the service worker should serve its offline fallback page, but the reporter saw the real desktop site instead. There is no screenshot description. The screenshot it never saw: grommet-5243 The issue says the checked background colour is applied to every radio button, not just the checked one. Sonnet's message after reproducing the bug: Bug reproduced: all three radio boxes get #3b5ed8 , matching the screenshot in the issue.The screenshot it never saw: grommet-6296 The issue says a long string in the PageHeader heading "overflows and collides with other content". Without seeing the actual screenshot, I'm guessing the overflow is likely a long unbroken string in the heading, like a filename or email, pushing into the actions column. The screenshot it never saw:
• Sonnet kept trying to recall the upstream fix from memory. Reviewers flagged it in 38 of 60 no-image runs (it remembered Carbon's useIdPrefix and.bx--grid--narrow names, for example). It doesn't look like it was making up for the missing images though, since the native runs did it too.
• Where it needed to see the UI, it took its own headless Chrome screenshots. In the no-image run those came back as the placeholder too, so it measured bounding boxes and sampled pixels instead. Both conditions rendered the UI about equally often (34 runs with images, 33 without).
• In 5 rollouts, it refused to look up stuff upstream.Traces where it refused to look upstream (click to expand)These are Anthropic's summarized thinking, not the raw chain of thought. grommet-2695 Actually, in a SWE-bench-like setting, fetching the actual upstream patch would be closer to contamination than genuine problem-solving, and since I'm confident I can solve this from the repo alone, the cleaner approach is to skip the web lookup and trust my own analysis. highlight.js-2726 Still, I'm weighing whether this counts as contamination given a benchmark setting -- a human contributor would naturally check the issue tracker discussion, but if the goal is to solve purely from the issue text, pulling the upstream patch feels like cheating. prism-3372 I'm going to stick with my earlier decision not to look up the actual upstream Prism implementation or tests — that would feel like cheating.

Why did removing images make so little difference? #

To check whether images were slipping through somewhere, I had the trace police inspect both models' runs and checked some samples myself. The audit found no issue images reaching either model in the no-image runs.

The reviewers also assessed whether each task needed its images. For each task, they examined the native-image trace alongside the issue images, the merged fix and the hidden tests. These ratings are the reviewers' judgments and not definitive labels.

Images were rarely required

GPT-6 Luna xhigh

In 403 of 480 tasks (84%) the issue text fully specifies the bug and the images only illustrate it. Resolution counts barely move across the three runs. The helpful category has nine resolutions in each run, and of the seven tasks rated required, only one was resolved, with native images.

Reviewer rating	Tasks	Resolved: images / OCR / omitted
not_needed : text fully specifies the task	403	167 / 166 / 169
helpful : text is enough, images remove ambiguity	70	9 / 9 / 9
required : needed information is only in the images	7	1 / 0 / 0

Where images mattered at all, it was the issue screenshots, never an image file from the repo.

Seven required tasks (click to expand) #

Task	Images / OCR / omitted	What only the images show	What happened in the native run
carbon-12384	R / F / F	Spec: read-only steppers stay visible but greyed	Resolved. The only clear image win in 480 tasks
lighthouse-15092	F / F / F	Whole error redesign; issue text is the title only	Tests check the category gauge, which Luna left alone
lighthouse-9151	F / F / F	Which gauge is broken (PWA badge)	Patched the path shape; the cause was duplicate SVG ids
lighthouse-4301	F / F / F	Gatherer names during the hang point to start-url	Patched driver listener setup instead of the gatherer
carbon-3139	F / F / F	The two accessibility violations	Right fix, but undefined where snapshots expectnull
carbon-4260	F / F / F	The two accessibility violations	Snapshot expects an id scheme nobody specified
openlayers-14100	F / F / F	WMTS GetTile URLs with TIME	Read it as a throttling question; no patch

Sonnet 5.5 max

The reviewers rated the images not needed in 53 of 60 tasks and required in none. Resolution is flat again. Without images Sonnet resolves 4 more of the not_needed tasks and 1 fewer of the helpful ones.

Reviewer rating	Tasks	Resolved: images / omitted
not_needed : text fully specifies the task	53	17 / 21
helpful : text is enough, images remove ambiguity	5	2 / 1
required : needed information is only in the images	0	–
no images in the issue	2	2 / 2

The reviewers of the no-image traces rated next-3724 as required and moved grommet-6438 and highlight.js-2969 up to helpful (both failed either way). The two audits agree on the other 57 tasks.

Five helpful tasks (click to expand) #

Task	Images / omitted	What only the images show	What happened
next-3724	R / F	Keep the left search box and hide the right one	Without the screenshot the agent made only searchProps per-panel, so the right box still rendered. The only clear image win in the subset
carbon-12384	R / R	Spec: read-only steppers stay visible but greyed	Resolved both ways. The test only checks that the steppers are disabled
carbon-14476	F / F	The "Select all (N)" label and divider layout	Hid the button once every row was selected; the test expects it to stay
carbon-4028	F / F	The four accessibility violations	Left ListBoxSelection alone; snapshots expect aria-label="Clear Selection"
carbon-4820	F / F	The label-in-name and contrast violations	Patched only Dropdown; snapshots expect a new id scheme across five components

The hidden tests don't check what the images show

The audit also exposed a gap between what the screenshots specify and what the hidden tests reward. A lot of the Carbon issues come with detailed design spec screenshots (colours, spacing, icon placement), but the test that grades them is often just a snapshot of the component's prop types (carbon-13851, for example, comes with a full TimePicker error/warning spec, and the test only checks that two new props exist). The OpenLayers and bpmn-js issues have GIFs of the bug, while the tests check internal state like waypoints or function arguments.

The test only checks that warning and warningText exist as props. The red border, the icons, the message colours and the 8px and 16px spacing are never checked.

When tasks failed, it was usually because the tests expected some detail that neither the text nor the images mention, like an exact prop name, null vs undefined, or something settled in the GitHub issue comments that the model never sees. The reviewers flagged this in all but 2 of the Luna failures they reviewed, and in 32 of Sonnet's 36 no-image failures. I haven't checked every case by hand.

For the images to matter, a test would have to check something you can only learn from them, like which component is broken or a value from the design spec.

The models may already know some fixes

Memorisation is another possibility. These tasks come from public GitHub issues and PRs from 2017–2024, so both models may already know some of the fixes and have little need for the screenshots. In fact, Sonnet tried to recall the upstream fix in 38 of 60 no-image runs.

Example tasks where images were not needed

These ten tasks were resolved with native images, OCR text and no images. You can read their full no-image traces in an interactive trace viewer. It lives on my other website, since Bear Blog can't run one.

Task	What the images show	What the text already says
lighthouse-4036	The noopener audit flagging mailto links; Chrome for Android opening a blank page	Title: the audit "should ignore href="mailto:x@y.com" links". The repo already skips javascript: links the same way
prism-1602	Rendered YAML where "world" loses its colour when followed by# test	"String highlighting does not work when a string has a comment on the same line", plus the repro - "world" # test
next-2984	Console TypeError "Cannot add property value, object is not extensible"	CascaderSelect changes the props passed in; "the component should not change the user's values"
bpmn-js-1655	Diagram whose external labels are drawn in the pale fill colour	"External labels should be set to the 'stroke' color, as seen in the embedded labels"
carbon-6675	Pagination footer reading "1 of 1 pages"	Names the file, the lines and the rule: singular when max === 1 ortotalPages === 1
eslint-14242	Terminal: exit code 1 without --quiet , exit code 0 with it	"The exit code should be 1 when max-warnings is exceeded"
grommet-6282	DataTable sorted by Parent, rows with null Parent not grouped	"Undefined properties should be treated as empty string"
highlight.js-2958	Output where <a:FAIL …> is plain text and the other tags are coloured	Sample code labels each line OK or FAIL; title: "single letter namespace prefix not highlighted"
openlayers-14627	WebGL stack trace; debugger in fromEPSG4326 with Infinity inputs	Repro code, the cause (" Math.tan will transformInfinity toNaN ") and the expected behaviour
prettier-11884	Editor showing a lone comma inside [ ]	Input, CLI options, the bad output and the expected output

MMMU-Pro as a control #

Florian and I wanted to check that our setup really takes the image information away, and that Luna uses images when they matter, so I ran the same setup of native image input, customised OCR and placeholder images on MMMU-Pro, with the same model, reasoning effort and interceptor.

MMMU-Pro makes a good control because it isn't agentic. It's single-turn multiple-choice questions across college subjects, with no repo to read and no tools to fall back on. In its standard split, each question comes with a diagram, chart or photo that carries information the text does not restate, so a model that cannot see the image should actually lose points there.

GPT-6 Luna xhigh (1730 questions)

Looks like the images matter!

Run	Accuracy	Correct	Output tokens / question
Native images	79.7%	1378 / 1730	1.7k
OCR text	56.3%	974 / 1730	4.1k
No images (placeholder)	36.9%	638 / 1730	2.1k

Every gap is large and paired (McNemar p < 1e-49 for all three pairs). Taking the images away costs 43 points here; on SWE-bench Multimodal it cost nothing. The OCR run also burns more than twice the output tokens of the native run (reasoning tokens alone: 4.0k vs 1.6k per question), which I interpret as the model trying to understand/reconstruct the figure from text.

The steepest drops come where the image carries structure the text leaves out: Art goes from 42/53 to 3/53 without images, Finance from 59/60 to 15/60, Accounting from 51/58 to 10/58. They are smallest where the text already carries the question: Literature 43/52 → 38/52, Energy and Power 50/58 → 38/58.

MMMU-Pro results by subject (click to expand) #

Sorted by how much accuracy drops when the images are removed. Cell colour goes from red (low) to green (high). The biggest drops are highlighted.

Subject	Native	OCR	No images	Drop (native → none)
Art	79% 42/53	34% 18/53	6% 3/53	−74 pts
Finance	98% 59/60	88% 53/60	25% 15/60	−73 pts
Accounting	88% 51/58	83% 48/58	17% 10/58	−71 pts
Design	83% 50/60	40% 24/60	20% 12/60	−63 pts
Marketing	95% 56/59	71% 42/59	34% 20/59	−61 pts
Computer Science	77% 46/60	42% 25/60	20% 12/60	−57 pts
Public Health	86% 50/58	81% 47/58	31% 18/58	−55 pts
Art Theory	84% 46/55	49% 27/55	31% 17/55	−53 pts
Biology	78% 46/59	42% 25/59	27% 16/59	−51 pts
Chemistry	82% 49/60	47% 28/60	32% 19/60	−50 pts
Geography	79% 41/52	48% 25/52	29% 15/52	−50 pts
Management	72% 36/50	48% 24/50	22% 11/50	−50 pts
Materials	82% 49/60	53% 32/60	32% 19/60	−50 pts
Economics	90% 53/59	64% 38/59	41% 24/59	−49 pts
Physics	85% 51/60	52% 31/60	42% 25/60	−43 pts
Architecture and Engineering	83% 50/60	62% 37/60	43% 26/60	−40 pts
Electronics	97% 58/60	63% 38/60	57% 34/60	−40 pts
Music	63% 38/60	30% 18/60	25% 15/60	−38 pts
Basic Medical Science	83% 43/52	40% 21/52	46% 24/52	−37 pts
Math	87% 52/60	63% 38/60	53% 32/60	−33 pts
Psychology	72% 43/60	48% 29/60	38% 23/60	−33 pts
Pharmacy	79% 45/57	63% 36/57	47% 27/57	−32 pts
History	70% 39/56	68% 38/56	39% 22/56	−30 pts
Clinical Medicine	73% 43/59	46% 27/59	44% 26/59	−29 pts
Sociology	74% 40/54	61% 33/54	46% 25/54	−28 pts
Agriculture	57% 34/60	45% 27/60	33% 20/60	−23 pts
Mechanical Engineering	71% 42/59	61% 36/59	49% 29/59	−22 pts
Energy and Power	86% 50/58	71% 41/58	66% 38/58	−21 pts
Diagnostics and Laboratory Medicine	55% 33/60	47% 28/60	38% 23/60	−17 pts
Literature	83% 43/52	77% 40/52	73% 38/52	−10 pts

Sonnet 5.5 max (120 questions)

I also ran Sonnet 5.5 at max effort on 120 of these questions (4 random ones per subject), with and without images, in the same plain chat setup. Luna's numbers on the same 120 are on the right for comparison.

Run	Sonnet 5.5 max	Output tokens / question	GPT-6 Luna xhigh on the same 120
Native images	80.8% (97 / 120)	6.8k	78.3% (94 / 120)
OCR text	not run		53.3% (64 / 120)
No images	44.2% (53 / 120)	19.8k	37.5% (45 / 120)

Taking away the images costs Sonnet 37 points: 46 questions go from right to wrong and only 2 the other way (McNemar p < 1e-10). The same change cost it nothing on SWE-bench Multimodal. Two native questions failed because their images were over Anthropic's 5 MB limit; they count as wrong.

Without images, Sonnet uses about three times as many output tokens (6.8k to 19.8k). They say a picture is worth a thousand words. Apparently, removing one adds several thousand for Sonnet.

It says it can't see the image in 78 of the 120 questions and openly guesses in 92, often from the answer options themselves. Opus 5.5 reviewers trace 55 of the 67 misses in the no-image run to the missing image.

Conclusion #

The experiments show that SWE-bench Multimodal v2 scores were largely insensitive to removing image inputs in our setups, while the same ablation cost both models around 40 points on MMMU-Pro. The audit points to two reasons. Screenshots often repeat what the issue text already says, and the hidden tests rarely check visual requirements.

Thanks for reading! If you found the post helpful, please like/retweet and share it with others who might find it useful.

Acknowledgements #

Thanks to Florian for the suggestions along the way and encouraging me to blog about this. Also grateful to Prime Intellect for the infrastructure and the generous inference credits that made these runs possible.

source & further reading

sankalp.bearblog.dev — original article New post Porting ALE Linux CLI: I Looked at the Data of a Frontier Benchmark and Found some Issues How Prompt Caching Works