SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models

Researchers introduced SPARC-Rad, a manually curated multimodal benchmark dataset and evaluation pipeline for assessing spatial and anatomical reasoning in radiology vision-language models (VLMs). The dataset includes 300 image-question pairs from healthy control imaging studies in The Cancer Imaging Archive (TCIA), spanning CT, MRI, and radiography across five anatomical categories. The pipeline supports standardized prompting, LLM-as-judge grading, and subgroup analysis, providing a reusable framework for evaluating VLMs' radiologic anatomy reasoning.

arXiv:2608.00100v1 Announce Type: new Abstract: Vision-language models VLMs are increasingly being evaluated for medical imaging, but many available benchmarks emphasize disease classification, report generation, or broad visual question answering rather than the spatial and anatomical reasoning required for radiology. We developed the Spatial Perception and Anatomical Reasoning in Clinical Radiology SPARC-Rad Benchmark, a manually curated multimodal benchmark dataset and evaluation pipeline for assessing these capabilities in radiology VLMs. SPARC-Rad includes 300 image-question pairs derived from healthy control imaging studies in The Cancer Imaging Archive TCIA , spanning CT, MRI, and radiography across the abdomen, chest, breast, neuro, and musculoskeletal categories. Radiology trainees manually designed and annotated questions to evaluate anatomical identification, localization, laterality, regional recognition, device identification, and inter-structure spatial relationships. The evaluation pipeline supports standardized prompting, structured output collection, response normalization, LLM-as-judge grading, human quality review, binary correctness scoring, and subgroup analysis by modality, anatomy, and reasoning type. SPARC-Rad provides a reusable framework for evaluating whether VLMs can provide reasoning for radiologic anatomy as a spatial system, supporting future model development, failure-mode analysis, and pre-deployment assessment.