{"slug": "new-post", "title": "New post", "summary": "A Prime Intellect residency project found that GPT 6 Luna xhigh scored roughly the same on the SWE-Multimodal v2 benchmark whether it received native images, OCR text, or no images at all, passing about 175-178 of 480 tasks (roughly 37%) across three runs. The author concluded SWE-Multimodal v2 is not actually testing multimodality, since most task information was present in the text or derivable from code, and noted that an initial run showed models looking up \"future\" commits via the git object, prompting new images with the git history object removed.", "body_md": "I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 for performing ablations. I found out that gpt 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was bit baffling at first but based on some further slop-investigation, I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the multimodality. In this post, I will briefly go through the methodology of the evaluation and the investigation itself.\n\nMethodology\n\nI was using GPT 6 luna xhigh for ablations. Cheapest model and higher rate limits. Did 3 runs on verifiers with codex harness with full network isolation. The issue images were downloaded in advance. We (Florian helped) also made new images with the git history object removed from the images provided by the benchmark because in an initial run, we found out that models were looking up “future” commits via the git object.\n\nEvaluation runs\n\nI did 3 runs - with native image input, customised OCR and no images mode.\n\nThis is possible to do in verifiers because of the interception server which allows to fetch the user message and tool related messages and replace stuff in them with what you want. I replaced the images with OCR or just placeholder text in the messages.\n\nHere is what a task looks like in the placeholder (no images) run. This is lighthouse-4036 from the latest run, which Luna resolved in 5 model calls and about a minute. The issue has two screenshots; the model gets the raw `![]()
` markup, a note that the images were removed, and one marker per image.\n\n```\nrel=\"noopener\" audit should ignore href=\"mailto:x@y.com\" links\n![\"screen](\"https://user-images.githubusercontent.com/3506071/32291560-....png\")
\n\nFollowing this suggestion for these `mailto:` links causes the links to break on Chrome for Android and iOS Safari. Unfortunately, we only found this issue in production :(\n\n\n\nReproduction:\n\nThe support link fails as seen above on Chrome for Android. The sales link works fine and launches Gmail from Chrome for Android.\n\nYou cannot see the images in this message; each was removed and only its position is marked.\n\n[Image 1 of 2]\n[Image 2 of 2]\n```\n\nAnd the whole trace that followed, condensed (reasoning summaries in italics, tool calls in code):\n\n`rg -n \"noopener|rel.*audit|target=.*_blank\" .`` sed -n '1,240p' lighthouse-core/audits/dobetterweb/external-anchors-use-rel-noopener.js` plus the matching test file and the smoke fixture.`javascript:` only; I'll add a case-insensitive `mailto:` exemption and regression test.\"`apply_patch`:\n\n```\n- // Ignore href's that are not real links\n- return !anchor.href || !anchor.href.toLowerCase().startsWith('javascript:');\n+ // Ignore href's that are not real links or do not navigate to a page.\n+ const href = (anchor.href || '').toLowerCase();\n+ return !anchor.href || (!href.startsWith('javascript:') && !href.startsWith('mailto:'));\n```\n\n`mocha lighthouse-core/test/audits/dobetterweb/external-anchors-use-rel-noopener-test.js` → 5 passing. Done.\nThe screenshots (a Lighthouse audit flagging the mailto link, and a broken link on Android) never come up. The title alone says what to fix, and the existing `javascript:` exemption in the code says how.\n\nRoughly same results were obtained in the three cases. Around 175-178 pass scores out of 480 which turns out to be roughly 37%.\n\nTo further check, if the model was looking up images via some other means that I was missing, I sacrificed my claude max sub and put on several sonnet 5.5 reviewers to go through the traces line by line to find out. The sonnet 5.5 reviewers pointed out\n\nIn 403 of 480 tasks (84%) the issue text fully specifies the bug and the images only illustrate it. Resolution rates are flat across the three runs in every category, so the images don't change outcomes even where they add information.\n\n| Reviewer rating | Tasks | Resolved: images / OCR / omitted | \n|---|---|---|\n| `not_needed` : text fully specifies the task | 403 | 167 / 166 / 169 | \n| `helpful` : text is enough, images remove ambiguity | 70 | 9 / 9 / 9 | \n| `required` : needed information is only in the images | 7 | 1 / 0 / 0 | \n\nNo task needed a **repo** image. 39 tasks were judged not solvable without images, but mostly because nothing could solve them. The gold patch depends on image information in 7 tasks and partly in 65.\n\n`required` tasks\n| Task | Images / OCR / omitted | What only the images show | Why the native run still failed | \n|---|---|---|---|\n| carbon-12384 | R / F / F | Spec: read-only steppers stay visible but greyed | Resolved. The only clear image win in 480 tasks | \n| lighthouse-15092 | F / F / F | Whole error redesign; issue text is the title only | Tests check the category gauge, which Luna left alone | \n| lighthouse-9151 | F / F / F | Which gauge is broken (PWA badge) | Patched the path shape; the cause was duplicate SVG ids | \n| lighthouse-4301 | F / F / F | Gatherer names during the hang point to start-url | Patched driver listener setup instead of the gatherer | \n| carbon-3139 | F / F / F | The two accessibility violations | Right fix, but `undefined` where snapshots expect`null` | \n| carbon-4260 | F / F / F | The two accessibility violations | Snapshot expects an id scheme nobody specified | \n| openlayers-14100 | F / F / F | WMTS GetTile URLs with TIME | Read it as a throttling question; no patch | \n\nHere are four resolved tasks where you can judge for yourself. Each was solved in all runs, including the no-images run, and the no-images trace was checked for any image access (none). Two Fable 5.1 reviewers rated each one independently, one shown the gold patch and hidden tests, the other shown only the trace with the outcome stripped; all rated the images \"not needed\". The screenshot is the original from the GitHub issue, which the model never saw. Swipe or scroll sideways for the next task.\n\n**highlight.js-2958** — one-character regex fix; the issue labels the failing line `FAIL` in the code sample\n\n*What the screenshot shows:* jsbin output with three XML tags highlighted and `` rendered as plain text.\n\n*What the text already says:* title \"(xml) line with single letter namespace prefix not highlighted since 10.4.0\", plus a sample where the lines are literally named ``, ``, ``, ``.\n\n*What Luna did without the image:* read `src/languages/xml.js`, noted \"the tag-name regex requires a colon prefix to exceed one character\", reproduced it with `hljs.highlight` on the sample, and changed one character. Identical to the upstream fix.\n\n``` js\n- const TAG_NAME_RE = regex.concat(/[A-Z_]/, regex.optional(/[A-Z0-9_.-]+:/), /[A-Z0-9_.-]*/);\n+ const TAG_NAME_RE = regex.concat(/[A-Z_]/, regex.optional(/[A-Z0-9_.-]*:/), /[A-Z0-9_.-]*/);\n```\n\n*Reviewer:* \"The images only confirm visually what the issue title and the OK/FAIL labels in the sample code already state.\"\n\n**openlayers-14627** — the screenshots are a stack trace and a debugger view; the issue text names the cause and the function\n\n*What the screenshots show:* a console `TypeError: Cannot read properties of undefined (reading 'length')` in the WebGL tile renderer, and a debugger paused in `fromEPSG4326` with `y = NaN` on the `Math.tan` line.\n\n*What the text already says:* \"applyTransform for empty extent will get a extent with NaN\", a repro using `createEmpty()` + `applyTransform(extent, getTransform('EPSG:4326','EPSG:3857'))`, and \"it's caused by `Math.tan` will transform `Infinity` to `NaN`\".\n\n*What Luna did without the images:* read `fromEPSG4326` (the exact code in the debugger screenshot), found `isEmpty` and `createOrUpdateEmpty` already in `ol/extent.js`, and added a guard. Identical to the upstream fix; 102 tests pass.\n\n```\n export function applyTransform(extent, transformFn, dest, stops) {\n+ if (isEmpty(extent)) {\n+ return createOrUpdateEmpty(dest);\n+ }\n```\n\n*Reviewer:* \"Only incidental detail: the exact downstream stack trace and that applyTransform internally calls the transform with 8 coordinates. Neither is needed for the fix or tests.\"\n\n**eslint-14242** — a terminal screenshot; the issue text states the bug and the acceptable fix in two sentences\n\n*What the screenshot shows:* a PowerShell session: `eslint --max-warnings 3` exits 1 with \"ESLint found too many warnings\", then the same command with `--quiet` prints nothing and exits 0.\n\n*What the text already says:* \"Using the `--max-warnings` cli option is useless in combination with `--quiet`\" and \"Also acceptable: The exit code should be 1 when `max-warnings` is exceeded.\"\n\n*What Luna did without the image:* read `lib/cli.js`, saw that quiet mode filters `results` before `countErrors(results)` runs, so the warning count is always 0, and moved the count above the filter. Same behaviour as the upstream fix; 83 tests pass.\n\n```\n+ const { errorCount, warningCount } = countErrors(results);\n+\n if (options.quiet) {\n results = ESLint.getErrorResults(results);\n }\n if (await printResults(engine, results, options.format, options.outputFile)) {\n- const { errorCount, warningCount } = countErrors(results);\n```\n\n*Reviewer:* \"The image adds only incidental detail: the exact command used, that the exit code is observed via $LASTEXITCODE, and the warning count 26. None of this is needed; the test uses its own fixture and threshold 5.\"\n\n**prism-1602** — a rendered-output screenshot; the text already contains the two-line reproducer\n\n*What the screenshot shows:* Prism's output for the YAML snippet: the first `\"world\"` (followed by `# test`) is unhighlighted, the second is green.\n\n*What the text already says:* \"YAML string highlighting does not work when a string has a comment on the same line\", with the snippet `- \"world\" # test` vs `- \"world\"`.\n\n*What Luna did without the image:* read `components/prism-yaml.js`, found the string pattern's lookahead only allows end-of-line, `,`, `]` or `}` after the closing quote, and added a trailing-comment case. The upstream fix adds `\\s*#`; Luna's adds `[ \\t]+#`. Hidden tests pass either way.\n\n```\n- ...\\2(?=[ \\t]*(?:$|,|]|}))/m,\n+ ...\\2(?=[ \\t]*(?:$|,|]|}|[ \\t]+#))/m,\n```\n\n*Reviewer:* \"The image only shows the symptom the text already states (first \"world\" unhighlighted, second highlighted).\"\n\nMMMU Pro is a good control: it is not agentic, just single-turn multiple-choice questions across college subjects, with no repo or tools to fall back on. In its standard split, each question comes with a diagram, chart or photo that carries information the text does not restate, so a model that cannot see the image should actually lose points there.\n\nAnd it does. Same model, same effort, same interceptor, 1,730 questions of the MMMU-Pro standard split (10 options, so chance is 10%):\n\n| Run | Accuracy | Correct | Output tokens / question | \n|---|---|---|---|\n| Native images | **79.7%** | 1378 / 1730 | 1.7k | \n| OCR text | 56.3% | 974 / 1730 | 4.1k | \n| No images (placeholder) | 36.9% | 638 / 1730 | 2.1k | \n\nEvery gap is large and paired (McNemar p < 1e-49 for all three pairs). Taking the images away costs 43 points here; on SWE-bench Multimodal it cost nothing. The OCR run also burns more than twice the reasoning tokens of the native run, which is the model trying to reconstruct the figure from text.\n\nThe drop is biggest where the image carries structure that words don't: Art goes from 42/53 to 3/53 without images, Finance from 59/60 to 15/60, Accounting from 51/58 to 10/58. It is smallest where the text already carries the question: Literature 43/52 → 38/52, Energy and Power 50/58 → 38/58.\n\n| Subject | Native | OCR | No images | \n|---|---|---|---|\n| Accounting | 51/58 | 48/58 | 10/58 | \n| Agriculture | 34/60 | 27/60 | 20/60 | \n| Architecture and Engineering | 50/60 | 37/60 | 26/60 | \n| Art | 42/53 | 18/53 | 3/53 | \n| Art Theory | 46/55 | 27/55 | 17/55 | \n| Basic Medical Science | 43/52 | 21/52 | 24/52 | \n| Biology | 46/59 | 25/59 | 16/59 | \n| Chemistry | 49/60 | 28/60 | 19/60 | \n| Clinical Medicine | 43/59 | 27/59 | 26/59 | \n| Computer Science | 46/60 | 25/60 | 12/60 | \n| Design | 50/60 | 24/60 | 12/60 | \n| Diagnostics and Laboratory Medicine | 33/60 | 28/60 | 23/60 | \n| Economics | 53/59 | 38/59 | 24/59 | \n| Electronics | 58/60 | 38/60 | 34/60 | \n| Energy and Power | 50/58 | 41/58 | 38/58 | \n| Finance | 59/60 | 53/60 | 15/60 | \n| Geography | 41/52 | 25/52 | 15/52 | \n| History | 39/56 | 38/56 | 22/56 | \n| Literature | 43/52 | 40/52 | 38/52 | \n| Manage | 36/50 | 24/50 | 11/50 | \n| Marketing | 56/59 | 42/59 | 20/59 | \n| Materials | 49/60 | 32/60 | 19/60 | \n| Math | 52/60 | 38/60 | 32/60 | \n| Mechanical Engineering | 42/59 | 36/59 | 29/59 | \n| Music | 38/60 | 18/60 | 15/60 | \n| Pharmacy | 45/57 | 36/57 | 27/57 | \n| Physics | 51/60 | 31/60 | 25/60 | \n| Psychology | 43/60 | 29/60 | 23/60 | \n| Public Health | 50/58 | 47/58 | 18/58 | \n| Sociology | 40/54 | 33/54 | 25/54 |", "url": "https://wpnews.pro/news/new-post", "canonical_source": "https://sankalp.bearblog.dev/new-post-new-new-new-new-new-new/", "published_at": "2026-10-15 11:40:00+00:00", "updated_at": "2026-10-15 12:47:45.032222+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-safety"], "entities": ["Prime Intellect", "GPT 6 Luna xhigh", "SWE-Multimodal v2", "Florian", "verifiers", "codex"], "also_reported_by": [{"url": "https://sankalp.bearblog.dev/swe_multidmoal/", "source": "sankalp.bearblog.dev", "title": "swe multimodal v2 bench may be cooked", "published_at": "2026-10-15T11:40:00+00:00"}], "alternates": {"html": "https://wpnews.pro/news/new-post", "markdown": "https://wpnews.pro/news/new-post.md", "text": "https://wpnews.pro/news/new-post.txt", "jsonld": "https://wpnews.pro/news/new-post.jsonld"}}