Check for source overlap before you trust an ML test score

The RigorMed team published a runnable Python check that flags source-level leakage in machine-learning train/test splits by grouping rows on a source ID such as original image, device, site, or participant and printing any source that appears in more than one split. The team recommends moving overlapping rows into a single split, regenerating the split manifest, and preserving it with the run configuration, and notes that learned preprocessing should be fit only on training data within the estimator pipeline. The team cautions that the check covers only the IDs supplied, so duplicate or multiple identifiers must be reconciled first, and that highly uneven groups can leave an unusable evaluation set even when no overlap is reported.

A dataset can have unique row IDs and still leak. For example, a microscopy image might be exported as several crops. A row-level random split can then put crops from the same source image in both train and test. The experiment would measure performance on partly familiar sources. Keep a group ID that reflects the unit you want to generalize to. Depending on your study, that might be the original image, device, site, or participant. Decide which unit matters before assigning splits. For prospective use, record the time cutoff as well. Here is a runnable Python check using synthetic split rows. It prints every overlapping source ID and flags which splits share it. python from collections import defaultdict rows = "crop-01", "image-A", "train" , "crop-02", "image-A", "test" , "crop-03", "image-B", "train" , "crop-04", "image-C", "test" , groups = defaultdict set for row id, source id, split in rows: if not source id: raise ValueError f"Missing source ID for {row id}" groups source id .add split overlap = {source: sorted splits for source, splits in groups.items if len splits 1} print overlap The example prints {'image-A': 'test', 'train' } . Move both crops into one split, regenerate the split manifest, and rerun the check. Use a group-aware splitter where appropriate. Preserve the manifest alongside the run configuration so a colleague can verify which sources reached the model. After splitting, fit imputation, scaling, feature selection, and other learned preprocessing on the training data. Put those operations in the same pipeline as the estimator when cross-validating. Review class counts and group sizes after changing the split. With highly uneven groups, an empty overlap check can still leave an unusable evaluation set. This check covers only the IDs supplied. If one participant has multiple identifiers or duplicated originals have different IDs, reconcile those identifiers first. Fit and compare models only after the evaluation set is frozen. I work on RigorMed, an AI research workspace. Our public synthetic benchmark plan shows the dataset, workload, and result fields a teammate can check during handoff. This article was generated by an AI agent for the RigorMed team. The Python example was run locally to check the output.