IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval

A new arXiv paper (2610.30533v1) proposes IFC-MCQ, a method that automatically generates multiple-choice benchmarks from Industry Foundation Classes (IFC) files so that LLM agents retrieving building data can be graded by exact option-set matching instead of costly human or LLM review of free-text answers. The taxonomy spans 15 categories and 36 subtypes across four groups, with answer keys derived from the file and distractors built from file-derived values or predefined rules. Applied to IFC models of varying sizes and modelling characteristics, the method exposed category-specific failure modes and showed that agent performance can vary with the source IFC model and retrieval interface.

arXiv:2610.30533v1 Announce Type: new Abstract: Industry Foundation Classes IFC , the standard for building information exchange in architecture, engineering, and construction AEC , is increasingly queried through large language models LLMs using natural language. Evaluating these systems at scale is difficult because free-text answers require costly human or LLM-based review. We propose IFC-MCQ, a method that automatically generates multiple-choice benchmarks from IFC files for exact option-set grading. The taxonomy covers 15 categories and 36 subtypes across four groups, with answer keys derived from the file and distractors constructed from file-derived values or predefined rules. We applied IFC-MCQ to IFC models of varying sizes and modelling characteristics to assess generation coverage and evaluated two agentic configurations on selected models using tool- and code-based retrieval. The evaluation exposed category-specific failure modes and showed that performance can vary with the source IFC model and retrieval interface. IFC-MCQ enables scalable benchmarking of LLM-based IFC retrieval in AEC workflows.