New post A Prime Intellect residency project found that GPT 6 Luna xhigh scored roughly the same on the SWE-Multimodal v2 benchmark whether it received native images, OCR text, or no images at all, passing about 175-178 of 480 tasks (roughly 37%) across three runs. The author concluded SWE-Multimodal v2 is not actually testing multimodality, since most task information was present in the text or derivable from code, and noted that an initial run showed models looking up "future" commits via the git object, prompting new images with the git history object removed. I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 for performing ablations. I found out that gpt 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was bit baffling at first but based on some further slop-investigation, I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the multimodality. In this post, I will briefly go through the methodology of the evaluation and the investigation itself. Methodology I was using GPT 6 luna xhigh for ablations. Cheapest model and higher rate limits. Did 3 runs on verifiers with codex harness with full network isolation. The issue images were downloaded in advance. We Florian helped also made new images with the git history object removed from the images provided by the benchmark because in an initial run, we found out that models were looking up “future” commits via the git object. Evaluation runs I did 3 runs - with native image input, customised OCR and no images mode. This is possible to do in verifiers because of the interception server which allows to fetch the user message and tool related messages and replace stuff in them with what you want. I replaced the images with OCR or just placeholder text in the messages. Here is what a task looks like in the placeholder no images run. This is lighthouse-4036 from the latest run, which Luna resolved in 5 model calls and about a minute. The issue has two screenshots; the model gets the raw