{"slug": "mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning", "title": "MLIRBench – A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning", "summary": "A November 11, 2026 arXiv paper by Pranava Sai Maganti introduces MLIRBench, a benchmark and dataset of 1,768 execution-validated program pairs (888 linalg, 880 torch) for evaluating how large language models reason about MLIR compiler intermediate representations. Across nine models — five open-weight and four proprietary — Gemini 3.1 Pro posted the highest hard-label accuracy at 92.65% on all 1,768 pairs, yet scored only 74.18 in aggregate and 5.04/10 on a judge-assigned result-behavior score, showing that strong semantic classification does not imply accurate prediction of detailed execution behavior. The authors present the gap as a reproducible foundation for studying semantic reasoning over compiler IRs, program repair and semantic classification.", "body_md": "# Computer Science > Programming Languages\n\n  [Submitted on 11 Nov 2026]\n\n# Title:MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning\n\n[View PDF](https://arxiv.org/pdf/2611.14881)\n\n[HTML (experimental)](https://arxiv.org/html/2611.14881v1)\n\nAbstract:In recent years, large language models (LLMs) have become increasingly popular in software engineering and have performed well on different programming tasks. However, it is still difficult to know whether these models truly understand program behavior or whether they mainly rely on patterns learned from their training data. This distinction is particularly important for intermediate representations (IRs), where operations, dataflow, and execution semantics are explicitly represented. In this paper, we introduce MLIRBench, a benchmark for evaluating how LLMs reason about MLIR programs. MLIRBench uses a multi-agent pipeline to generate program mutations, validate their behavior through execution, and construct reasoning tasks from torch and linalg program pairs. The dataset contains 1,768 validated pairs, including 888 linalg pairs and 880 torch pairs, labeled as EQUIV, DIVERGE, or ERROR. We evaluated nine models, including five open-weight and four proprietary models. Gemini 3.1 Pro achieved the highest hard-label accuracy of 92.65% across all 1,768 pairs, counting unsuccessful predictions as incorrect. Across its 1,768 scored predictions, it achieved an aggregate score of 74.18 and a judge-assigned result-behavior score of 5.04/10. These results show that strong semantic classification does not necessarily imply accurate prediction of detailed execution behavior, highlighting a gap in current LLM reasoning over compiler IRs. By exposing this gap through execution-validated program pairs, MLIRBench provides a reproducible foundation for studying semantic reasoning over compiler IRs and related tasks such as program repair and semantic classification.\n    \n\n## Submission history\n\nFrom: Pranava Sai Maganti [\n[view email](https://arxiv.org/show-email/8b3883fd/2611.14881)]\n\n**[v1]** Wed, 11 Nov 2026 01:12:50 UTC (2,061 KB)\n\n### Additional Features\n\n### Current browse context:\n\ncs.PL\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning", "canonical_source": "https://arxiv.org/abs/2611.14881", "published_at": "2026-11-13 19:17:04+00:00", "updated_at": "2026-11-13 19:20:10.439380+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["MLIRBench", "Pranava Sai Maganti", "Gemini 3.1 Pro", "arXiv", "MLIR"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning", "markdown": "https://wpnews.pro/news/mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning.md", "text": "https://wpnews.pro/news/mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning.txt", "jsonld": "https://wpnews.pro/news/mlirbench-a-benchmark-and-dataset-for-compiler-and-hardware-aware-reasoning.jsonld"}}