{"slug": "deepfake-detector-robustness-testing", "title": "Deepfake Detector Robustness Testing", "summary": "A new benchmark dataset, the Social Media Robustness Benchmark, evaluates how deepfake detectors perform on images re-encoded by social media platforms like Instagram, Facebook, TikTok, and X. The dataset contains 2,400 paired real and synthetic face images processed through calibrated platform pipelines to measure detector accuracy drops under real-world conditions. Researchers can use the benchmark to compute paired AUC deltas and assess detector robustness across demographic groups, though the dataset is restricted to research evaluation only and not for training detection models.", "body_md": "[ Datasets:](/datasets)\n\n[ Image Classification ](/datasets?task_categories=task_categories%3Aimage-classification)\n\n[ parquet ](/datasets?format=format%3Aparquet)\n\n[ English ](/datasets?language=language%3Aen)\n\n[ 10K - 100K ](/datasets?size_categories=size_categories%3A10K%3Cn%3C100K)\n\n[ deepfake-detection ](/datasets?other=deepfake-detection)\n\n[ synthetic-face-detection ](/datasets?other=synthetic-face-detection)\n\n[ ai-generated-image-detection ](/datasets?other=ai-generated-image-detection)\n\n[ deepfake ](/datasets?other=deepfake)\n\n[ image-forensics ](/datasets?other=image-forensics)\n\n[ fairness ](/datasets?other=fairness)\n\n## Request access to the Social Media Robustness Benchmark\n\nThis repository is publicly accessible, but you have to accept the conditions to access its files and content.\n\nAn evaluation benchmark released under CC BY-NC 4.0 for research evaluation only, not for training detection models or commercial use. Optional: tell me what you work on, and opt in below if you want a heads-up when datasets like this drop. I plan the next dataset around what people actually need.\n\n[Log in](/login?next=/datasets/danb21/social-media-robustness-sdxl-instantid) or [Sign Up](/join?next=/datasets/danb21/social-media-robustness-sdxl-instantid) to review the conditions and access this dataset content.\n\n# Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection\n\n**Version:** v1.0.0 · **License:** CC BY-NC 4.0 (research evaluation only)\n\nDetector accuracy on clean lab test sets does not predict in-the-wild performance. Social platforms re-encode every uploaded image: platform-specific JPEG, resize, chroma subsampling, metadata stripped. This benchmark lets detector authors and procurers measure robustness under documented, paired, demographically-balanced conditions instead of blunt lab proxies.\n\nA companion blog post and white paper cover the methodology, statistics, and findings in full. This card describes what the dataset is, how it is built at a high level, and how to use it.\n\n## 1. Details\n\n| Field | Value |\n|---|---|\n| Name | Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection |\n| Version | v1.0.0 |\n| Base corpus | 2,400 images (1,200 real, 1,200 generated), sampled from\n|\n\n`media_id`\n\n)[Read the write-up](https://babalolad.substack.com/p/how-deepfake-detectors-perform-under?r=2kub4w)(companion post; published before the paper)[danielbabalola@alumni.upenn.edu](mailto:danielbabalola@alumni.upenn.edu)##\n\n2. What This Dataset Is For\n\nUse it to:\n\n- Compute paired AUC deltas, AUC(clean) − AUC(perturbation), per detector per condition.\n- Measure per-cell robustness (skin tone × gender) under each perturbation.\n- Compare detector architectures under matched conditions.\n\nIt is **not training data**. It is small by design (2,400 base images), paired by construction\n(every condition evaluates the same images), and the platform pipelines are calibrated\napproximations of each platform's mean re-encode behaviour, not pixel-faithful platform\nreproductions (see Limitations).\n\n## 3. Structure\n\n### Configurations\n\n| Config | Layer | Description |\n|---|---|---|\n`clean` |\nclean | 2,400 unperturbed base images; the reference for every paired delta |\n`layer1_jpeg_q{30,50,70,80,95}` |\nlab | JPEG re-encode at the named quality factor |\n`layer1_resize_{0.5,0.75}` |\nlab | Bicubic downsample then upsample back |\n`layer1_noise_{5,10}` |\nlab | Additive Gaussian noise (variance 5 / 10) |\n`layer1_blur_{1,2,4}` |\nlab | Gaussian blur (sigma 1 / 2 / 4) |\n`layer2_ig_pipeline` |\nplatform | Instagram (JPEG ~92, max edge 1440, 4:2:0, EXIF stripped) |\n`layer2_fb_pipeline` |\nplatform | Facebook (JPEG ~93, max edge 1920, 4:2:0, EXIF stripped) |\n`layer2_tt_pipeline` |\nplatform | TikTok (JPEG ~80, max edge 1920, 4:2:0) |\n`layer2_x_pipeline` |\nplatform | X (JPEG ~93, max edge 1920, 4:2:0, EXIF stripped) |\n\nAll configurations share the same column schema. Key columns: `image`\n\n, `label`\n\n(`real`\n\n/`generated`\n\n), `cell_skin_tone`\n\n, `cell_gender`\n\n, `media_id`\n\n(stable across\nconfigurations for pairing), `perturbation_slug`\n\n, `perturbation_layer`\n\n, and the\nmeasured-encoding fields.\n\n### Balance\n\nThe clean baseline and all 12 lab configurations are uniform at **100 images per cell per\nlabel** (6 skin tones × 2 genders × real/generated). The 4 platform configurations re-crop the\nlaundered output back to a 256×256 face crop for comparability; all real-side cells retain\n100/100, while a small fraction of synthetic-face crops do not survive re-detection (~2,342–2,344\nrows per platform config). That concentration is itself a finding, analyzed in the white paper.\n\n## 4. How It Is Built (high level)\n\n**Source.** A deterministic per-cell subset of the[v1 Synthetic Face Detection Benchmark](https://huggingface.co/datasets/danb21/synthetic-face-sdxl-instantid-bench): 1,200 real Pexels frames and 1,200 SDXL+InstantID outputs, uniform at 100 per cell per label.**Lab perturbations (Layer 1).** Twelve single-axis variants applied independently to each base image, with deterministic seeds so the pipeline is reproducible. All outputs are 256×256.**Platform pipelines (Layer 2).** For Instagram, Facebook, TikTok, and X, each platform's mean re-encode behaviour (resize to measured max edge, JPEG at measured quality, 4:2:0 chroma, EXIF handling) was measured and then applied to every base image. The laundered image is re-cropped to a 256×256 face crop so it is geometrically comparable to the clean and lab configurations.\n\nThe full calibration procedure, preregistered hypotheses, statistics, and caveats are documented in the companion white paper.\n\n## 5. Results\n\nBaseline accuracy, per-cell fairness, and per-platform robustness deltas are reported in the companion blog post and white paper, not in this card.\n\n## 6. Comparison with Prior Robustness Datasets\n\n| Dataset | Year | Compression coverage | Paired pre/post | Platform calibration | Demographic balance | Scale (paired) |\n|---|---|---|---|---|---|---|\n| FaceForensics++ | 2019 | Lab c0 / c23 / c40 | Yes | No | None | ~1,000,000 |\n| DFDC | 2020 | Mixed | Partial | No | Stated, not quantified | ~500,000 |\n| GenImage | 2023 | Lab JPEG QF=96 only | Yes | No | None | ~1,000,000 |\n| OpenFake | 2025 | None documented | No | No | Limited | Variable |\nThis dataset |\n2026 |\n12 lab axes + 4 calibrated platform pipelines |\nYes (by `media_id` ) |\nYes (4 platforms) |\n6 skin tones × 2 genders, 100/cell/label |\n2,400 base (×17 configs) |\n\nScale is smaller than prior work by design. This is an evaluation benchmark, not training data.\n\n## 7. Bias, Risks, and Limitations\n\n**Architecture-level coverage only.** The synthetic side is SDXL+InstantID (vanilla, no community fine-tunes, LoRA stacks, or frontier I2V models). Those are out of scope for v1.**Platform pipelines are calibrate-and-apply, not pixel-faithful.** Each platform's measured mean encode behaviour is applied to every base image; the full upload chain and per-account or per-region variation are not reproduced.**TikTok calibration is a point estimate;** treat TikTok results accordingly. Instagram, Facebook, and X measured qualities are close and should be read as applied parameters, not distinguishing fingerprints.**Synthetic-face re-crop drop.** A small fraction of synthetic-face crops fail face re-detection after platform laundering, concentrated on certain cells. This is reported as a finding (analyzed in the white paper), not back-filled.**Real/generated track differences.** The two tracks differ in crop and encoding history; a detector could in principle exploit that rather than genuine synthesis signal. The white paper quantifies a control for this.\n\n## 8. License and Ethics\n\n**License:** CC BY-NC 4.0 (research evaluation only). Commercial licensing inquiries:[danielbabalola@alumni.upenn.edu](mailto:danielbabalola@alumni.upenn.edu).**Per-row attestation:**`LICENSES.csv`\n\nlists each row's license. Real-side rows inherit Pexels licensing; generated-side rows carry the SDXL backbone + InstantID adapter terms.**Consent, age, identity:** Inherited from the v1 base corpus. Real-side frames are sourced from Pexels; potentially under-18 frames were human-reviewed and only confirmed-adult rows are included; real-side identities are capped per cluster to prevent leakage.**Platform calibration disclosure:** The platform parameters were measured by posting a small set of reference images and inspecting the platform-returned encoding. No image in this dataset is a platform post; the platform configurations are produced locally by applying the measured parameters to the base images.\n\n## 9. Citation\n\n```\n@dataset{babalola_social_media_robustness_2026,\n  title  = {Social Media Robustness Benchmark: SDXL+InstantID Synthetic Face Detection},\n  author = {Babalola, Daniel},\n  year   = {2026},\n  url    = {https://huggingface.co/datasets/danb21/social-media-robustness-sdxl-instantid},\n  note   = {Version v1.0.0}\n}\n```\n\nA companion paper is in preparation; this citation will be updated with the DOI on publication.\n\n- Downloads last month\n- 7", "url": "https://wpnews.pro/news/deepfake-detector-robustness-testing", "canonical_source": "https://huggingface.co/datasets/danb21/social-media-robustness-sdxl-instantid", "published_at": "2026-06-07 01:12:57+00:00", "updated_at": "2026-06-07 01:46:57.557647+00:00", "lang": "en", "topics": ["computer-vision", "ai-safety", "ai-research", "generative-ai", "ai-ethics"], "entities": ["Social Media Robustness Benchmark", "SDXL", "InstantID", "CC BY-NC 4.0"], "alternates": {"html": "https://wpnews.pro/news/deepfake-detector-robustness-testing", "markdown": "https://wpnews.pro/news/deepfake-detector-robustness-testing.md", "text": "https://wpnews.pro/news/deepfake-detector-robustness-testing.txt", "jsonld": "https://wpnews.pro/news/deepfake-detector-robustness-testing.jsonld"}}