VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Researchers released VELD-QA, a benchmark of 15,000 human-authored question-answer pairs spanning 5,859 real-world PDFs and 372,445 pages, designed to test visual evidence grounding and question answering in long documents. Evaluating nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems, the authors found oracle accuracy peaks at 81.7% and that retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages, with full-document access failing to close the gap. Enriching inputs with the dataset's visual-element annotations narrowed the performance gap by 17-24%, and the code is available at https://github.com/oracle-samples/veld-qa.

arXiv:2610.20964v1 Announce Type: new Abstract: Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.