AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation

Researchers released AGIDefect-4K, a dataset of 4,000 AI-generated images from 15 generative models with hierarchical annotations for defect detection, segmentation, explanation, and quality scoring, and introduced AGIDA, a baseline framework using multimodal large language models for joint defect analysis. Benchmarking shows AGI defect understanding remains challenging, highlighting the dataset's value for improving AI image reliability.

arXiv:2608.20713v1 Announce Type: new Abstract: Generative AI can now produce highly realistic images, yet current models still exhibit subtle but critical defects that undermine their reliability. While existing AI-generated image AGI evaluation benchmarks have made notable progress, comprehensive AGI defect diagnosis remains underexplored. To bridge this gap, we introduce AGIDefect-4K, a richly annotated dataset of 4,000 images from 15 state-of-the-art generative models spanning both open-source and closed-source systems. AGIDefect-4K features hierarchical defect annotations: 1 detection labels identifying whether defects exist, 2 pixel-level segmentation masks localizing defective regions, and 3 detailed textual explanations characterizing defect types and their perceptual impact. Each image is further annotated with an overall quality score. Building on this, we present AGIDA AGI Defect Assistant , a baseline framework leveraging Multimodal Large Language Models MLLMs for joint defect detection, localization, explanation, and quality prediction. Comprehensive benchmarking on AGIDefect-4K reveals that AGI defect understanding remains challenging, underscoring the value of this dataset. The dataset is publicly available at https://github.com/sxfly99/AGIDefect-4K.