Deepfake Detector Robustness Testing

A new benchmark dataset, the Social Media Robustness Benchmark, evaluates how deepfake detectors perform on images re-encoded by social media platforms like Instagram, Facebook, TikTok, and X. The dataset contains 2,400 paired real and synthetic face images processed through calibrated platform pipelines to measure detector accuracy drops under real-world conditions. Researchers can use the benchmark to compute paired AUC deltas and assess detector robustness across demographic groups, though the dataset is restricted to research evaluation only and not for training detection models.

Datasets: /datasets Image Classification /datasets?task categories=task categories%3Aimage-classification parquet /datasets?format=format%3Aparquet English /datasets?language=language%3Aen 10K - 100K /datasets?size categories=size categories%3A10K%3Cn%3C100K deepfake-detection /datasets?other=deepfake-detection synthetic-face-detection /datasets?other=synthetic-face-detection ai-generated-image-detection /datasets?other=ai-generated-image-detection deepfake /datasets?other=deepfake image-forensics /datasets?other=image-forensics fairness /datasets?other=fairness Request access to the Social Media Robustness Benchmark This repository is publicly accessible, but you have to accept the conditions to access its files and content. An evaluation benchmark released under CC BY-NC 4.0 for research evaluation only, not for training detection models or commercial use. Optional: tell me what you work on, and opt in below if you want a heads-up when datasets like this drop. I plan the next dataset around what people actually need. Log in /login?next=/datasets/danb21/social-media-robustness-sdxl-instantid or Sign Up /join?next=/datasets/danb21/social-media-robustness-sdxl-instantid to review the conditions and access this dataset content. Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection Version: v1.0.0 · License: CC BY-NC 4.0 research evaluation only Detector accuracy on clean lab test sets does not predict in-the-wild performance. Social platforms re-encode every uploaded image: platform-specific JPEG, resize, chroma subsampling, metadata stripped. This benchmark lets detector authors and procurers measure robustness under documented, paired, demographically-balanced conditions instead of blunt lab proxies. A companion blog post and white paper cover the methodology, statistics, and findings in full. This card describes what the dataset is, how it is built at a high level, and how to use it. 1. Details | Field | Value | |---|---| | Name | Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection | | Version | v1.0.0 | | Base corpus | 2,400 images 1,200 real, 1,200 generated , sampled from | media id Read the write-up https://babalolad.substack.com/p/how-deepfake-detectors-perform-under?r=2kub4w companion post; published before the paper danielbabalola@alumni.upenn.edu mailto:danielbabalola@alumni.upenn.edu 2. What This Dataset Is For Use it to: - Compute paired AUC deltas, AUC clean − AUC perturbation , per detector per condition. - Measure per-cell robustness skin tone × gender under each perturbation. - Compare detector architectures under matched conditions. It is not training data . It is small by design 2,400 base images , paired by construction every condition evaluates the same images , and the platform pipelines are calibrated approximations of each platform's mean re-encode behaviour, not pixel-faithful platform reproductions see Limitations . 3. Structure Configurations | Config | Layer | Description | |---|---|---| clean | clean | 2,400 unperturbed base images; the reference for every paired delta | layer1 jpeg q{30,50,70,80,95} | lab | JPEG re-encode at the named quality factor | layer1 resize {0.5,0.75} | lab | Bicubic downsample then upsample back | layer1 noise {5,10} | lab | Additive Gaussian noise variance 5 / 10 | layer1 blur {1,2,4} | lab | Gaussian blur sigma 1 / 2 / 4 | layer2 ig pipeline | platform | Instagram JPEG ~92, max edge 1440, 4:2:0, EXIF stripped | layer2 fb pipeline | platform | Facebook JPEG ~93, max edge 1920, 4:2:0, EXIF stripped | layer2 tt pipeline | platform | TikTok JPEG ~80, max edge 1920, 4:2:0 | layer2 x pipeline | platform | X JPEG ~93, max edge 1920, 4:2:0, EXIF stripped | All configurations share the same column schema. Key columns: image , label real / generated , cell skin tone , cell gender , media id stable across configurations for pairing , perturbation slug , perturbation layer , and the measured-encoding fields. Balance The clean baseline and all 12 lab configurations are uniform at 100 images per cell per label 6 skin tones × 2 genders × real/generated . The 4 platform configurations re-crop the laundered output back to a 256×256 face crop for comparability; all real-side cells retain 100/100, while a small fraction of synthetic-face crops do not survive re-detection ~2,342–2,344 rows per platform config . That concentration is itself a finding, analyzed in the white paper. 4. How It Is Built high level Source. A deterministic per-cell subset of the v1 Synthetic Face Detection Benchmark https://huggingface.co/datasets/danb21/synthetic-face-sdxl-instantid-bench : 1,200 real Pexels frames and 1,200 SDXL+InstantID outputs, uniform at 100 per cell per label. Lab perturbations Layer 1 . Twelve single-axis variants applied independently to each base image, with deterministic seeds so the pipeline is reproducible. All outputs are 256×256. Platform pipelines Layer 2 . For Instagram, Facebook, TikTok, and X, each platform's mean re-encode behaviour resize to measured max edge, JPEG at measured quality, 4:2:0 chroma, EXIF handling was measured and then applied to every base image. The laundered image is re-cropped to a 256×256 face crop so it is geometrically comparable to the clean and lab configurations. The full calibration procedure, preregistered hypotheses, statistics, and caveats are documented in the companion white paper. 5. Results Baseline accuracy, per-cell fairness, and per-platform robustness deltas are reported in the companion blog post and white paper, not in this card. 6. Comparison with Prior Robustness Datasets | Dataset | Year | Compression coverage | Paired pre/post | Platform calibration | Demographic balance | Scale paired | |---|---|---|---|---|---|---| | FaceForensics++ | 2019 | Lab c0 / c23 / c40 | Yes | No | None | ~1,000,000 | | DFDC | 2020 | Mixed | Partial | No | Stated, not quantified | ~500,000 | | GenImage | 2023 | Lab JPEG QF=96 only | Yes | No | None | ~1,000,000 | | OpenFake | 2025 | None documented | No | No | Limited | Variable | This dataset | 2026 | 12 lab axes + 4 calibrated platform pipelines | Yes by media id | Yes 4 platforms | 6 skin tones × 2 genders, 100/cell/label | 2,400 base ×17 configs | Scale is smaller than prior work by design. This is an evaluation benchmark, not training data. 7. Bias, Risks, and Limitations Architecture-level coverage only. The synthetic side is SDXL+InstantID vanilla, no community fine-tunes, LoRA stacks, or frontier I2V models . Those are out of scope for v1. Platform pipelines are calibrate-and-apply, not pixel-faithful. Each platform's measured mean encode behaviour is applied to every base image; the full upload chain and per-account or per-region variation are not reproduced. TikTok calibration is a point estimate; treat TikTok results accordingly. Instagram, Facebook, and X measured qualities are close and should be read as applied parameters, not distinguishing fingerprints. Synthetic-face re-crop drop. A small fraction of synthetic-face crops fail face re-detection after platform laundering, concentrated on certain cells. This is reported as a finding analyzed in the white paper , not back-filled. Real/generated track differences. The two tracks differ in crop and encoding history; a detector could in principle exploit that rather than genuine synthesis signal. The white paper quantifies a control for this. 8. License and Ethics License: CC BY-NC 4.0 research evaluation only . Commercial licensing inquiries: danielbabalola@alumni.upenn.edu mailto:danielbabalola@alumni.upenn.edu . Per-row attestation: LICENSES.csv lists each row's license. Real-side rows inherit Pexels licensing; generated-side rows carry the SDXL backbone + InstantID adapter terms. Consent, age, identity: Inherited from the v1 base corpus. Real-side frames are sourced from Pexels; potentially under-18 frames were human-reviewed and only confirmed-adult rows are included; real-side identities are capped per cluster to prevent leakage. Platform calibration disclosure: The platform parameters were measured by posting a small set of reference images and inspecting the platform-returned encoding. No image in this dataset is a platform post; the platform configurations are produced locally by applying the measured parameters to the base images. 9. Citation @dataset{babalola social media robustness 2026, title = {Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection}, author = {Babalola, Daniel}, year = {2026}, url = {https://huggingface.co/datasets/danb21/social-media-robustness-sdxl-instantid}, note = {Version v1.0.0} } A companion paper is in preparation; this citation will be updated with the DOI on publication. - Downloads last month - 7