CoMedBench: A Multi-Source Benchmark of Synthetic Medical Data Fidelity and Downstream Utility

CoMedBench, a new benchmark introduced in arXiv:2608.12805v1, evaluates synthetic medical data fidelity and downstream utility across 37 dataset-task pairs from seven public sources, including MIMIC-III, MIMIC-IV, and eICU. The reference generator CoMed-CTGAN retains 90.6% mean AUROC utility on tabular tasks, while CoMed-TVAE achieves 97.3%; temporal ICU tasks are harder, with CoMed-CTGAN retaining 81.6% AUROC and 64.0% AUPRC, whereas CoMed-TVAE retains ~95% AUROC.

arXiv:2608.12805v1 Announce Type: new Abstract: Access to clinical data is essential for developing reliable healthcare machine learning systems, but direct use of electronic health records is constrained by privacy regulation, institutional review, data-use agreements, and the risk of re-identification. Synthetic data promises a practical alternative: it can preserve useful statistical and clinical structure while reducing exposure of sensitive patient records. Prior studies often evaluate a single generator, one dataset, or a narrow downstream task, making it difficult to know when synthetic data can support model development and when it fails to preserve task-critical signal. We introduce CoMedBench, a reproducible benchmark that evaluates a family of generators under a common clinical-validity framework and one shared training and evaluation engine, spanning static tabular and temporal downstream tasks on established critical-care datasets. In total the benchmark spans 37 dataset-task pairs across two modalities consists of 20 static tabular and 17 temporal ICU time-series-drawn from seven public data sources: three intensive-care databases MIMIC-III, MIMIC-IV, and eICU together with the UCI Machine Learning Repository, the CDC BRFSS diabetes cohort 2015 , NHANES 1999-2014 , and the pycox survival datasets GBSG and METABRIC . The benchmark evaluates both statistical fidelity and task utility by comparing models trained and tested across real and synthetic data. In these settings, synthetic training data preserves most of the downstream signal: on tabular tasks the reference generator CoMed-CTGAN retains a mean AUROC utility the synthetic-to-real performance ratio of 90.6%, rising to 97.3% for the strongest generator, CoMed-TVAE. Temporal ICU tasks are harder and more generator-sensitive: CoMed-CTGAN retains 81.6% AUROC and only 64.0% under the imbalance-sensitive AUPRC, whereas CoMed-TVAE still retains ~95% AUROC .