MLIRBench – A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning

A November 11, 2026 arXiv paper by Pranava Sai Maganti introduces MLIRBench, a benchmark and dataset of 1,768 execution-validated program pairs (888 linalg, 880 torch) for evaluating how large language models reason about MLIR compiler intermediate representations. Across nine models — five open-weight and four proprietary — Gemini 3.1 Pro posted the highest hard-label accuracy at 92.65% on all 1,768 pairs, yet scored only 74.18 in aggregate and 5.04/10 on a judge-assigned result-behavior score, showing that strong semantic classification does not imply accurate prediction of detailed execution behavior. The authors present the gap as a reproducible foundation for studying semantic reasoning over compiler IRs, program repair and semantic classification.

Computer Science Programming Languages Submitted on 11 Nov 2026 Title:MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning View PDF https://arxiv.org/pdf/2611.14881 HTML experimental https://arxiv.org/html/2611.14881v1 Abstract:In recent years, large language models LLMs have become increasingly popular in software engineering and have performed well on different programming tasks. However, it is still difficult to know whether these models truly understand program behavior or whether they mainly rely on patterns learned from their training data. This distinction is particularly important for intermediate representations IRs , where operations, dataflow, and execution semantics are explicitly represented. In this paper, we introduce MLIRBench, a benchmark for evaluating how LLMs reason about MLIR programs. MLIRBench uses a multi-agent pipeline to generate program mutations, validate their behavior through execution, and construct reasoning tasks from torch and linalg program pairs. The dataset contains 1,768 validated pairs, including 888 linalg pairs and 880 torch pairs, labeled as EQUIV, DIVERGE, or ERROR. We evaluated nine models, including five open-weight and four proprietary models. Gemini 3.1 Pro achieved the highest hard-label accuracy of 92.65% across all 1,768 pairs, counting unsuccessful predictions as incorrect. Across its 1,768 scored predictions, it achieved an aggregate score of 74.18 and a judge-assigned result-behavior score of 5.04/10. These results show that strong semantic classification does not necessarily imply accurate prediction of detailed execution behavior, highlighting a gap in current LLM reasoning over compiler IRs. By exposing this gap through execution-validated program pairs, MLIRBench provides a reproducible foundation for studying semantic reasoning over compiler IRs and related tasks such as program repair and semantic classification. Submission history From: Pranava Sai Maganti view email https://arxiv.org/show-email/8b3883fd/2611.14881 v1 Wed, 11 Nov 2026 01:12:50 UTC 2,061 KB Additional Features Current browse context: cs.PL References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .