{"slug": "end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for", "title": "End-to-End Multimodal Data Augmentation and Adversarial Robustness Benchmark with AugLy for Images, Text, Audio, and PyTorch", "summary": "A tutorial demonstrates an end-to-end multimodal data augmentation and adversarial robustness workflow using Meta's AugLy library for images, text, and audio, connected directly to PyTorch datasets and DataLoaders. The workflow benchmarks perceptual-hash copy detection under image distortions and evaluates text classifiers against adversarial perturbations, Unicode obfuscation, sanitization, and adversarial training, with deterministic synthetic datasets seeded at 1234 for reproducibility. It also covers AugLy's functional and class-based APIs, metadata and intensity tracking, probabilistic composition, bounding-box-aware transformations, and a queryable metadata warehouse.", "body_md": "In this tutorial, we build a comprehensive multimodal augmentation and robustness workflow with [**AugLy**](https://github.com/facebookresearch/AugLy) for images, text, and audio. We start by addressing modern dependency compatibility issues and generating deterministic synthetic datasets so the experiments remain self-contained and reproducible. We then explore AugLy’s functional and class-based APIs, metadata, and intensity tracking, probabilistic composition, bounding-box-aware transformations, and custom transforms. We extend the workflow into practical robustness experiments by benchmarking perceptual-hash copy detection under image distortions and evaluating text classifiers against adversarial perturbations, Unicode obfuscation, sanitization, and adversarial training. We also integrate audio augmentation, build a queryable metadata warehouse, and connect AugLy transformations directly to PyTorch datasets and DataLoaders, giving us an end-to-end view of augmentation as both a data-generation mechanism and a measurable robustness tool.\n\n``` python\nimport subprocess, sys, importlib\ndef _sh(cmd):\n   print(f\"$ {cmd}\")\n   subprocess.run(cmd, shell=True, check=False,\n                  stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)\ndef _need(mod):\n   try:\n       importlib.import_module(mod)\n       return False\n   except ImportError:\n       return True\nif _need(\"augly\"):\n   _sh(\"apt-get -qq install -y libmagic1 > /dev/null 2>&1\")\n   _sh(f'\"{sys.executable}\" -m pip install -q --no-deps augly')\n   _sh(f'\"{sys.executable}\" -m pip install -q \"iopath>=0.1.8\" \"python-magic>=0.4.22\" '\n       f'\"regex>=2021.4.4\" \"nlpaug==1.1.3\"')\nimport numpy as np\nfrom PIL import Image, ImageDraw, ImageFont, ImageFilter\nfor _name, _builtin in ((\"float\", float), (\"int\", int), (\"bool\", bool)):\n   if not hasattr(np, _name):\n       setattr(np, _name, _builtin)\ndef _size(font, text):\n   left, top, right, bottom = font.getbbox(text)\n   return (right, bottom)\nif not hasattr(ImageFont.FreeTypeFont, \"getsize\"):\n   ImageFont.FreeTypeFont.getsize = lambda self, t, *a, **k: _size(self, t)\nif not hasattr(ImageFont.FreeTypeFont, \"getsize_multiline\"):\n   def _getsize_multiline(self, text, direction=None, spacing=4, features=None,\n                          language=None, stroke_width=0):\n       lines = text.split(\"\\n\")\n       w = max((_size(self, ln)[0] for ln in lines), default=0)\n       h = sum(_size(self, ln)[1] for ln in lines) + spacing * (len(lines) - 1)\n       return (w, h)\n   ImageFont.FreeTypeFont.getsize_multiline = _getsize_multiline\nimport os, io, json, math, random, string, textwrap, unicodedata, warnings\nfrom dataclasses import dataclass\nfrom typing import Any, Dict, List, Optional, Tuple\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport augly.image as imaugs\nimport augly.text as textaugs\nimport augly.utils as augutils\nfrom augly.image.transforms import BaseTransform as ImageBaseTransform\nwarnings.filterwarnings(\"ignore\")\npd.set_option(\"display.width\", 160)\nSEED = 1234\nrandom.seed(SEED)\nnp.random.seed(SEED)\nprint(\"\\n\" + \"=\" * 78)\nprint(\"AugLy ready.  assets at:\", augutils.ASSETS_BASE_DIR)\nprint(\"image augs :\", len([f for f in dir(imaugs) if f[0].islower()]))\nprint(\"text  augs :\", len([f for f in dir(textaugs) if f[0].islower()]))\nprint(\"=\" * 78 + \"\\n\")\ndef make_image(idx: int, w: int = 320, h: int = 240) -> Tuple[Image.Image, Tuple[int, int, int, int]]:\n   \"\"\"Procedurally generated 'photo' + a ground-truth bbox in pascal_voc format.\"\"\"\n   rng = random.Random(SEED + idx)\n   img = Image.new(\"RGB\", (w, h), tuple(rng.randint(20, 90) for _ in range(3)))\n   d = ImageDraw.Draw(img)\n   for _ in range(70):\n       x0, y0 = rng.randint(0, w), rng.randint(0, h)\n       d.line([x0, y0, x0 + rng.randint(-60, 60), y0 + rng.randint(-60, 60)],\n              fill=tuple(rng.randint(60, 160) for _ in range(3)), width=rng.randint(1, 3))\n   ow, oh = rng.randint(70, 130), rng.randint(60, 110)\n   ox, oy = rng.randint(10, w - ow - 10), rng.randint(10, h - oh - 10)\n   box = (ox, oy, ox + ow, oy + oh)\n   colour = tuple(rng.randint(150, 255) for _ in range(3))\n   if idx % 3 == 0:\n       d.ellipse(box, fill=colour, outline=(255, 255, 255), width=3)\n   elif idx % 3 == 1:\n       d.rectangle(box, fill=colour, outline=(255, 255, 255), width=3)\n   else:\n       d.polygon([(ox + ow // 2, oy), (ox + ow, oy + oh), (ox, oy + oh)],\n                 fill=colour, outline=(255, 255, 255))\n   return img, box\nN_IMAGES = 24\nIMAGES, BOXES = zip(*[make_image(i) for i in range(N_IMAGES)])\nIMAGES, BOXES = list(IMAGES), list(BOXES)\nDEMO_IMG, DEMO_BOX = IMAGES[0], BOXES[0]\ndef make_text_dataset(n_per_class: int = 260):\n   \"\"\"Tiny sentiment corpus built from templates -> learnable but not trivial.\"\"\"\n   rng = random.Random(SEED)\n   pos_adj = [\"excellent\", \"delightful\", \"superb\", \"charming\", \"brilliant\",\n              \"flawless\", \"wonderful\", \"outstanding\", \"impressive\", \"lovely\"]\n   neg_adj = [\"terrible\", \"awful\", \"dreadful\", \"disappointing\", \"clumsy\",\n              \"broken\", \"miserable\", \"useless\", \"painful\", \"sloppy\"]\n   subj = [\"the movie\", \"this restaurant\", \"the hotel room\", \"their support team\",\n           \"the new phone\", \"the sequel\", \"this laptop\", \"the delivery service\"]\n   tail_p = [\"and I would recommend it to anyone\", \"worth every rupee\",\n             \"I left completely satisfied\", \"easily the best of the year\",\n             \"it exceeded all my expectations\"]\n   tail_n = [\"and I want a refund\", \"a total waste of money\",\n             \"I left extremely frustrated\", \"easily the worst of the year\",\n             \"it failed every expectation\"]\n   rows = []\n   for _ in range(n_per_class):\n       rows.append((f\"{rng.choice(subj)} was {rng.choice(pos_adj)} {rng.choice(tail_p)}\", 1))\n       rows.append((f\"{rng.choice(subj)} was {rng.choice(neg_adj)} {rng.choice(tail_n)}\", 0))\n   rng.shuffle(rows)\n   return [r[0] for r in rows], [r[1] for r in rows]\nTEXTS, LABELS = make_text_dataset()\nDEMO_TEXT = \"The quick brown fox jumps over the lazy dog near the river bank\"\ndef make_audio(seconds: float = 2.0, sr: int = 16000) -> Tuple[np.ndarray, int]:\n   \"\"\"A chirp + harmonics + a little noise = something you can actually hear change.\"\"\"\n   t = np.linspace(0, seconds, int(sr * seconds), endpoint=False)\n   f = np.linspace(220, 880, t.size)\n   sig = 0.5 * np.sin(2 * np.pi * f * t) + 0.2 * np.sin(2 * np.pi * 2 * f * t)\n   sig += 0.02 * np.random.RandomState(SEED).randn(t.size)\n   env = np.minimum(1.0, np.minimum(t * 8, (seconds - t) * 8))\n   return (sig * env).astype(np.float32), sr\nAUDIO, SR = make_audio()\ndef show_grid(pairs, cols=4, title=\"\", figsize_scale=2.9):\n   \"\"\"pairs: list of (caption, PIL.Image).\"\"\"\n   rows = math.ceil(len(pairs) / cols)\n   fig, axes = plt.subplots(rows, cols, figsize=(cols * figsize_scale, rows * figsize_scale))\n   axes = np.atleast_1d(axes).ravel()\n   for ax, (cap, im) in zip(axes, pairs):\n       ax.imshow(im)\n       ax.set_title(cap, fontsize=8)\n       ax.axis(\"off\")\n   for ax in axes[len(pairs):]:\n       ax.axis(\"off\")\n   if title:\n       fig.suptitle(title, fontsize=13, y=1.0)\n   plt.tight_layout()\n   plt.show()\ndef as_str(out) -> str:\n   \"\"\"AugLy text augs return str for str input in some transforms, list in others.\"\"\"\n   return out[0] if isinstance(out, list) else out\nprint(\"\\n### §2  IMAGE AUGMENTATION + METADATA \" + \"#\" * 38)\nfunctional_result = imaugs.pixelization(DEMO_IMG, ratio=0.25)\nclass_result = imaugs.Pixelization(ratio=0.25, p=1.0)(DEMO_IMG)\nprint(\"functional == class:\", np.array_equal(np.array(functional_result), np.array(class_result)))\nIMAGE_ZOO = {\n   \"blur\":               lambda im, m: imaugs.blur(im, radius=3.0, metadata=m),\n   \"brightness\":         lambda im, m: imaugs.brightness(im, factor=1.7, metadata=m),\n   \"color_jitter\":       lambda im, m: imaugs.color_jitter(im, brightness_factor=1.3,\n                                                           contrast_factor=1.4,\n                                                           saturation_factor=1.6, metadata=m),\n   \"crop\":               lambda im, m: imaugs.crop(im, x1=.15, y1=.15, x2=.85, y2=.85, metadata=m),\n   \"encoding_quality\":   lambda im, m: imaugs.encoding_quality(im, quality=8, metadata=m),\n   \"grayscale\":          lambda im, m: imaugs.grayscale(im, metadata=m),\n   \"hflip\":              lambda im, m: imaugs.hflip(im, metadata=m),\n   \"meme_format\":        lambda im, m: imaugs.meme_format(im, text=\"TOP TEXT\",\n                                                          caption_height=90, metadata=m),\n   \"opacity\":            lambda im, m: imaugs.opacity(im, level=0.45, metadata=m),\n   \"overlay_emoji\":      lambda im, m: imaugs.overlay_emoji(im, opacity=0.9,\n                                                            emoji_size=0.35, metadata=m),\n   \"overlay_screenshot\": lambda im, m: imaugs.overlay_onto_screenshot(im, metadata=m),\n   \"overlay_stripes\":    lambda im, m: imaugs.overlay_stripes(im, line_width=0.4,\n                                                              line_opacity=0.7, metadata=m),\n   \"overlay_text\":       lambda im, m: imaugs.overlay_text(im, opacity=0.9, metadata=m),\n   \"pad_square\":         lambda im, m: imaugs.pad_square(im, metadata=m),\n   \"perspective\":        lambda im, m: imaugs.perspective_transform(im, sigma=40.0, metadata=m),\n   \"pixelization\":       lambda im, m: imaugs.pixelization(im, ratio=0.15, metadata=m),\n   \"random_noise\":       lambda im, m: imaugs.random_noise(im, var=0.03, metadata=m),\n   \"rotate\":             lambda im, m: imaugs.rotate(im, degrees=17, metadata=m),\n   \"saturation\":         lambda im, m: imaugs.saturation(im, factor=3.0, metadata=m),\n   \"scale\":              lambda im, m: imaugs.scale(im, factor=0.35, metadata=m),\n   \"sharpen\":            lambda im, m: imaugs.sharpen(im, factor=8.0, metadata=m),\n   \"shuffle_pixels\":     lambda im, m: imaugs.shuffle_pixels(im, factor=0.15, metadata=m),\n   \"skew\":               lambda im, m: imaugs.skew(im, skew_factor=0.35, metadata=m),\n   \"vflip\":              lambda im, m: imaugs.vflip(im, metadata=m),\n}\ngallery, image_meta = [(\"ORIGINAL\", DEMO_IMG)], []\nfor name, fn in IMAGE_ZOO.items():\n   m = []\n   try:\n       out = fn(DEMO_IMG, m)\n       gallery.append((f\"{name}\\nintensity={m[0]['intensity']:.1f}\", out))\n       image_meta.append(m[0])\n   except Exception as e:\n       print(f\"  [skip] {name}: {type(e).__name__}: {e}\")\nshow_grid(gallery, cols=5, title=\"§2  AugLy image augmentations (with AugLy's own intensity score)\")\nmeta_df = pd.DataFrame(image_meta)[[\"name\", \"intensity\", \"src_width\", \"src_height\",\n                                   \"dst_width\", \"dst_height\"]]\nprint(meta_df.sort_values(\"intensity\", ascending=False).head(10).to_string(index=False))\n```\n\nWe set up AugLy in a modern Colab environment while adding compatibility shims for NumPy and Pillow. We generate deterministic synthetic image, text, and audio datasets without external downloads. We also initialize reusable visualization and utility functions before exploring image augmentation and metadata.\n\n```\nprint(\"\\n### §3  COMPOSITION & REPRODUCIBILITY \" + \"#\" * 39)\nREUPLOAD_PIPELINE = imaugs.Compose([\n   imaugs.OneOf([\n       imaugs.OverlayOntoScreenshot(),\n       imaugs.MemeFormat(text=\"LOL\", caption_height=80),\n       imaugs.OverlayStripes(line_width=0.3, line_opacity=0.5),\n   ], p=0.9),\n   imaugs.RandomAspectRatio(min_ratio=0.7, max_ratio=1.4, p=0.5),\n   imaugs.RandomEmojiOverlay(p=0.7),\n   imaugs.RandomBrightness(min_factor=0.7, max_factor=1.4, p=0.6),\n   imaugs.EncodingQuality(quality=12, p=1.0),\n])\ndef run_pipeline(img, seed=None):\n   \"\"\"AugLy image transforms use the global `random` module -> seed it for determinism.\"\"\"\n   if seed is not None:\n       random.seed(seed)\n       np.random.seed(seed)\n   meta = []\n   return REUPLOAD_PIPELINE(img, metadata=meta), meta\na, meta_a = run_pipeline(DEMO_IMG, seed=7)\nb, meta_b = run_pipeline(DEMO_IMG, seed=7)\nc, _ = run_pipeline(DEMO_IMG, seed=99)\nprint(\"same seed -> identical output:\", np.array_equal(np.array(a), np.array(b)))\nprint(\"applied chain (seed=7)      :\", \" -> \".join(m[\"name\"] for m in meta_a))\nshow_grid([(\"original\", DEMO_IMG), (\"seed=7\", a), (\"seed=7 again\", b), (\"seed=99\", c)],\n         cols=4, title=\"§3  Seeded, reproducible augmentation pipelines\")\nprint(\"\\n### §4  BBOX-AWARE AUGMENTATION \" + \"#\" * 45)\nBBOX_OPS = [\n   (\"crop\",        lambda im, m, bb: imaugs.crop(im, x1=.1, y1=.1, x2=.9, y2=.9,\n                                                 metadata=m, bboxes=bb, bbox_format=\"pascal_voc\")),\n   (\"hflip\",       lambda im, m, bb: imaugs.hflip(im, metadata=m, bboxes=bb,\n                                                  bbox_format=\"pascal_voc\")),\n   (\"rotate 20\",   lambda im, m, bb: imaugs.rotate(im, degrees=20, metadata=m, bboxes=bb,\n                                                   bbox_format=\"pascal_voc\")),\n   (\"pad\",         lambda im, m, bb: imaugs.pad(im, w_factor=0.25, h_factor=0.25,\n                                                metadata=m, bboxes=bb, bbox_format=\"pascal_voc\")),\n   (\"meme_format\", lambda im, m, bb: imaugs.meme_format(im, text=\"BOXED\", caption_height=80,\n                                                        metadata=m, bboxes=bb,\n                                                        bbox_format=\"pascal_voc\")),\n]\ndef draw_box(img, box, colour=(0, 255, 0)):\n   out = img.copy().convert(\"RGB\")\n   ImageDraw.Draw(out).rectangle([float(v) for v in box], outline=colour, width=4)\n   return out\nbbox_panels = [(\"original\", draw_box(DEMO_IMG, DEMO_BOX))]\nfor label, op in BBOX_OPS:\n   m = []\n   try:\n       out = op(DEMO_IMG, m, [DEMO_BOX])\n       dst = m[0][\"dst_bboxes\"][0]\n       bbox_panels.append((f\"{label}\\n{tuple(round(v) for v in dst)}\", draw_box(out, dst)))\n       print(f\"  {label:12s} {DEMO_BOX} -> {tuple(round(v, 1) for v in dst)}\")\n   except Exception as e:\n       print(f\"  [skip] {label}: {type(e).__name__}: {e}\")\nshow_grid(bbox_panels, cols=3, title=\"§4  Boxes follow the pixels automatically\")\nprint(\"\\n### §5  CUSTOM TRANSFORMS \" + \"#\" * 51)\nclass RecompressionChain(ImageBaseTransform):\n   \"\"\"Simulate an image surviving N rounds of platform re-encoding.\n   Subclassing BaseTransform (rather than using ApplyLambda) buys you: the `p`\n   probability gate, `force=True`, and full participation in Compose/OneOf.\n   \"\"\"\n   def __init__(self, n_rounds: int = 3, min_q: int = 12, max_q: int = 45,\n                downscale: float = 0.85, p: float = 1.0):\n       super().__init__(p)\n       self.n_rounds, self.min_q, self.max_q, self.downscale = n_rounds, min_q, max_q, downscale\n   def apply_transform(self, image, metadata=None, bboxes=None, bbox_format=None):\n       src_w, src_h = image.size\n       out, qualities = image, []\n       for _ in range(self.n_rounds):\n           q = random.randint(self.min_q, self.max_q)\n           qualities.append(q)\n           out = imaugs.encoding_quality(out, quality=q)\n           out = imaugs.scale(out, factor=self.downscale)\n       out = out.resize((src_w, src_h), Image.BILINEAR)\n       if metadata is not None:\n           metadata.append({\n               \"name\": \"recompression_chain\",\n               \"src_width\": src_w, \"src_height\": src_h,\n               \"dst_width\": out.size[0], \"dst_height\": out.size[1],\n               \"n_rounds\": self.n_rounds, \"qualities\": qualities,\n               \"intensity\": float(100 * (1 - np.mean(qualities) / 100)),\n           })\n       return out\nvignette = imaugs.ApplyLambda(aug_function=lambda im: Image.composite(\n   im, Image.new(\"RGB\", im.size, (0, 0, 0)),\n   Image.radial_gradient(\"L\").resize(im.size).point(lambda v: 255 - v)))\nrandom.seed(SEED)\ncustom_meta = []\nshow_grid([\n   (\"original\", DEMO_IMG),\n   (\"RecompressionChain(n=3)\", RecompressionChain(n_rounds=3)(DEMO_IMG, metadata=custom_meta)),\n   (\"RecompressionChain(n=6)\", RecompressionChain(n_rounds=6, min_q=5, max_q=20)(DEMO_IMG)),\n   (\"ApplyLambda vignette\", vignette(DEMO_IMG)),\n], cols=4, title=\"§5  Custom transforms drop straight into the AugLy API\")\nprint(\"  custom metadata:\", custom_meta[0])\nCUSTOM_PIPELINE = imaugs.Compose([RecompressionChain(n_rounds=2, p=1.0),\n                                 imaugs.RandomEmojiOverlay(p=1.0)])\n_ = CUSTOM_PIPELINE(DEMO_IMG)\nprint(\"  composed with built-ins: OK\")\n```\n\nWe construct probabilistic augmentation pipelines with Compose and OneOf while controlling reproducibility through explicit random seeds. We demonstrate how AugLy automatically propagates bounding-box coordinates through spatial transformations. We then implement a custom BaseTransform and combine it with built-in AugLy transforms.\n\n```\nprint(\"\\n### §6  COPY-DETECTION ROBUSTNESS BENCHMARK \" + \"#\" * 33)\nfrom scipy.fftpack import dct\ndef phash(img: Image.Image, hash_size: int = 8, highfreq: int = 4) -> np.ndarray:\n   \"\"\"Classic DCT perceptual hash -> 64-bit signature as a bool array.\"\"\"\n   size = hash_size * highfreq\n   px = np.asarray(img.convert(\"L\").resize((size, size), Image.LANCZOS), dtype=np.float64)\n   d = dct(dct(px, axis=0, norm=\"ortho\"), axis=1, norm=\"ortho\")[:hash_size, :hash_size]\n   return (d > np.median(d[1:, 1:])).ravel()\ndef hamming(a, b) -> int:\n   return int(np.count_nonzero(a != b))\nINDEX = np.stack([phash(im) for im in IMAGES])\nATTACKS = {\n   \"brightness x1.6\":    lambda im: imaugs.brightness(im, factor=1.6),\n   \"blur r=3\":           lambda im: imaugs.blur(im, radius=3.0),\n   \"jpeg q=8\":           lambda im: imaugs.encoding_quality(im, quality=8),\n   \"crop 80%\":           lambda im: imaugs.crop(im, x1=.1, y1=.1, x2=.9, y2=.9),\n   \"rotate 12\":          lambda im: imaugs.rotate(im, degrees=12),\n   \"hflip\":              lambda im: imaugs.hflip(im),\n   \"grayscale\":          lambda im: imaugs.grayscale(im),\n   \"pixelize 0.2\":       lambda im: imaugs.pixelization(im, ratio=0.2),\n   \"noise var=.03\":      lambda im: imaugs.random_noise(im, var=0.03),\n   \"emoji overlay\":      lambda im: imaugs.overlay_emoji(im, emoji_size=0.35, opacity=0.9),\n   \"meme format\":        lambda im: imaugs.meme_format(im, text=\"LOL\", caption_height=70),\n   \"screenshot\":         lambda im: imaugs.overlay_onto_screenshot(im),\n   \"perspective s=40\":  lambda im: imaugs.perspective_transform(im, sigma=40.0),\n   \"scale 0.35\":         lambda im: imaugs.scale(im, factor=0.35),\n   \"stripes\":            lambda im: imaugs.overlay_stripes(im, line_width=0.4, line_opacity=0.7),\n   \"re-encode chain\":    lambda im: RecompressionChain(n_rounds=3)(im),\n   \"REUPLOAD pipeline\":  lambda im: REUPLOAD_PIPELINE(im),\n}\nrows = []\nfor attack, fn in ATTACKS.items():\n   random.seed(SEED)\n   np.random.seed(SEED)\n   hits, dists, failures = 0, [], 0\n   for i, im in enumerate(IMAGES):\n       try:\n           q = phash(fn(im))\n       except Exception:\n           failures += 1\n           continue\n       d = np.array([hamming(q, h) for h in INDEX])\n       hits += int(d.argmin() == i)\n       dists.append(int(d[i]))\n   n = len(IMAGES) - failures\n   rows.append({\"attack\": attack,\n                \"top1_recall\": hits / max(n, 1),\n                \"mean_hamming\": float(np.mean(dists)) if dists else np.nan,\n                \"errors\": failures})\nbench = pd.DataFrame(rows).sort_values(\"top1_recall\")\nprint(bench.to_string(index=False, float_format=lambda v: f\"{v:.3f}\"))\nfig, ax = plt.subplots(1, 2, figsize=(14, 6))\ncolours = [\"#c0392b\" if r < .5 else \"#e67e22\" if r < .9 else \"#27ae60\"\n          for r in bench.top1_recall]\nax[0].barh(bench.attack, bench.top1_recall, color=colours)\nax[0].set_xlabel(\"top-1 retrieval recall\"); ax[0].set_xlim(0, 1.05)\nax[0].axvline(0.9, ls=\"--\", c=\"k\", lw=1)\nax[0].set_title(\"pHash survival per AugLy attack\")\nax[1].scatter(bench.mean_hamming, bench.top1_recall, s=70, c=colours)\nfor _, r in bench.iterrows():\n   ax[1].annotate(r.attack, (r.mean_hamming, r.top1_recall), fontsize=7,\n                  xytext=(3, 3), textcoords=\"offset points\")\nax[1].set_xlabel(\"mean Hamming distance to the true match (0-64)\")\nax[1].set_ylabel(\"top-1 recall\")\nax[1].set_title(\"Distortion vs. retrieval failure\")\nplt.tight_layout(); plt.show()\nworst = bench.head(3).attack.tolist()\nprint(f\"\\n  pHash breaks under: {worst}\")\nprint(\"  -> exactly the augmentations you'd add to training, or handle with a\")\nprint(\"     geometry-invariant embedding instead of a hash.\")\n```\n\nWe build a perceptual-hash index over the synthetic image corpus and evaluate its robustness against a broad collection of AugLy distortions. We measure top-1 retrieval recall and Hamming distance for every attack to quantify how different transformations affect copy detection. We visualize the results to identify the augmentations that most strongly degrade perceptual matching.\n\n```\nprint(\"\\n### §7  TEXT ATTACK / DEFEND / HARDEN \" + \"#\" * 39)\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nX_tr, X_te, y_tr, y_te = train_test_split(TEXTS, LABELS, test_size=0.3,\n                                         random_state=SEED, stratify=LABELS)\ndef new_model():\n   return make_pipeline(\n       TfidfVectorizer(analyzer=\"word\", ngram_range=(1, 2), sublinear_tf=True),\n       LogisticRegression(max_iter=1000, C=4.0),\n   )\nbaseline = new_model().fit(X_tr, y_tr)\nclean_acc = accuracy_score(y_te, baseline.predict(X_te))\nprint(f\"clean test accuracy: {clean_acc:.3f}\\n\")\nATTACK_SUITE = {\n   \"typos (keyboard+misspell)\": textaugs.SimulateTypos(aug_word_p=0.45, typo_type=\"all\"),\n   \"unicode homoglyphs\":        textaugs.ReplaceSimilarUnicodeChars(aug_word_p=0.8, aug_char_p=0.4),\n   \"leetspeak lookalikes\":      textaugs.ReplaceSimilarChars(aug_word_p=0.8, aug_char_p=0.4),\n   \"zero-width injection\":      textaugs.InsertZeroWidthChars(granularity=\"word\", cadence=2.0),\n   \"punctuation injection\":     textaugs.InsertPunctuationChars(granularity=\"word\", cadence=2.0,\n                                                                vary_chars=True),\n   \"whitespace injection\":      textaugs.InsertWhitespaceChars(granularity=\"word\", cadence=3.0),\n   \"fun fonts\":                 textaugs.ReplaceFunFonts(aug_p=0.8, granularity=\"word\",\n                                                         vary_fonts=True),\n   \"upside down\":               textaugs.ReplaceUpsideDown(aug_p=0.6, granularity=\"word\"),\n   \"bidirectional\":             textaugs.ReplaceBidirectional(granularity=\"word\"),\n   \"split words\":               textaugs.SplitWords(aug_word_p=0.5),\n   \"merge words\":               textaugs.MergeWords(aug_word_p=0.5),\n   \"CaSe ChAoS\":                textaugs.ChangeCase(granularity=\"word\", cadence=2.0, case=\"upper\"),\n}\nprint(\"what the attacks look like on one sentence:\")\nfor name, aug in ATTACK_SUITE.items():\n   random.seed(SEED)\n   print(f\"  {name:26s} {as_str(aug(DEMO_TEXT))[:72]}\")\nZERO_WIDTH = dict.fromkeys(\n   [0x200B, 0x200C, 0x200D, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0xFEFF,\n    0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F]\n)\nPUNCT_TABLE = str.maketrans(\"\", \"\", \"\".join(c for c in string.punctuation if c not in \"'\"))\ndef sanitize(text: str) -> str:\n   \"\"\"Cheap, high-yield preprocessing against Unicode-obfuscation attacks.\"\"\"\n   t = text.translate(ZERO_WIDTH)\n   t = unicodedata.normalize(\"NFKD\", t)\n   t = \"\".join(ch for ch in t if not unicodedata.combining(ch))\n   t = t.translate(PUNCT_TABLE)\n   return \" \".join(t.lower().split())\nrandom.seed(SEED)\nTRAIN_AUGS = [ATTACK_SUITE[k] for k in\n             [\"typos (keyboard+misspell)\", \"unicode homoglyphs\", \"leetspeak lookalikes\",\n              \"zero-width injection\", \"punctuation injection\", \"fun fonts\", \"CaSe ChAoS\"]]\naug_X, aug_y = list(X_tr), list(y_tr)\nfor aug in TRAIN_AUGS:\n   for x, y in zip(X_tr, y_tr):\n       aug_X.append(as_str(aug(x)))\n       aug_y.append(y)\nprint(f\"\\ntrain set: {len(X_tr)} -> {len(aug_X)} examples after AugLy adversarial training\")\nhardened = make_pipeline(\n   TfidfVectorizer(analyzer=\"char_wb\", ngram_range=(3, 5), sublinear_tf=True,\n                   preprocessor=sanitize),\n   LogisticRegression(max_iter=2000, C=4.0),\n).fit(aug_X, aug_y)\nresults = []\nfor name, aug in ATTACK_SUITE.items():\n   random.seed(SEED)\n   X_atk = [as_str(aug(x)) for x in X_te]\n   results.append({\n       \"attack\": name,\n       \"baseline\": accuracy_score(y_te, baseline.predict(X_atk)),\n       \"+ sanitize\": accuracy_score(y_te, baseline.predict([sanitize(x) for x in X_atk])),\n       \"+ sanitize + adv-train\": accuracy_score(y_te, hardened.predict(X_atk)),\n   })\nres = pd.DataFrame(results).set_index(\"attack\").sort_values(\"baseline\")\nres.loc[\"-- CLEAN --\"] = [clean_acc,\n                         accuracy_score(y_te, baseline.predict([sanitize(x) for x in X_te])),\n                         accuracy_score(y_te, hardened.predict(X_te))]\nprint(\"\\n\" + res.to_string(float_format=lambda v: f\"{v:.3f}\"))\nprint(f\"\\nmean accuracy under attack:  baseline {res.iloc[:-1,0].mean():.3f}\"\n     f\"  ->  sanitized {res.iloc[:-1,1].mean():.3f}\"\n     f\"  ->  hardened {res.iloc[:-1,2].mean():.3f}\")\nplot_df = res.drop(index=\"-- CLEAN --\")\nax = plot_df.plot.barh(figsize=(11, 7), width=0.78,\n                      color=[\"#c0392b\", \"#e6a817\", \"#27ae60\"])\nax.axvline(clean_acc, ls=\"--\", c=\"k\", lw=1.2, label=f\"clean acc = {clean_acc:.2f}\")\nax.set_xlabel(\"accuracy under attack\"); ax.set_xlim(0, 1.05)\nax.set_title(\"§7  AugLy as a red-team suite — and as the fix\")\nax.legend(loc=\"lower right\", fontsize=8)\nplt.tight_layout(); plt.show()\n```\n\nWe create a text classification baseline and systematically expose it to typos, Unicode homoglyphs, invisible characters, punctuation changes, and other adversarial transformations. We implement Unicode normalization and sanitization to remove several classes of obfuscation. We then use AugLy-generated adversarial examples during training and compare the resulting hardened model against the baseline.\n\n``` python\nprint(\"\\n### §8  AUDIO AUGMENTATION \" + \"#\" * 50)\ntry:\n   import augly.audio as audaugs\n   AUDIO_ZOO = {\n       \"pitch_shift +4\":     lambda a, sr, m: audaugs.pitch_shift(a, sr, n_steps=4.0, metadata=m),\n       \"time_stretch 1.5x\":  lambda a, sr, m: audaugs.time_stretch(a, sr, rate=1.5, metadata=m),\n       \"change_volume -12dB\":lambda a, sr, m: audaugs.change_volume(a, sr, volume_db=-12.0, metadata=m),\n       \"clicks\":             lambda a, sr, m: audaugs.clicks(a, sr, seconds_between_clicks=0.25,\n                                                             metadata=m),\n       \"background_noise\":   lambda a, sr, m: audaugs.add_background_noise(a, sr, snr_level_db=5.0,\n                                                                           metadata=m),\n       \"low_pass 1kHz\":      lambda a, sr, m: audaugs.low_pass_filter(a, sr, cutoff_hz=1000.0,\n                                                                      metadata=m),\n       \"high_pass 2kHz\":     lambda a, sr, m: audaugs.high_pass_filter(a, sr, cutoff_hz=2000.0,\n                                                                       metadata=m),\n       \"peaking_eq\":         lambda a, sr, m: audaugs.peaking_equalizer(a, sr, center_hz=800.0,\n                                                                        gain_db=-12.0, metadata=m),\n       \"harmonic\":           lambda a, sr, m: audaugs.harmonic(a, sr, metadata=m),\n       \"percussive\":         lambda a, sr, m: audaugs.percussive(a, sr, metadata=m),\n       \"clip (50%)\":         lambda a, sr, m: audaugs.clip(a, sr, duration_factor=0.5, metadata=m),\n       \"loop x2\":            lambda a, sr, m: audaugs.loop(a, sr, n=1, metadata=m),\n       \"normalize\":          lambda a, sr, m: audaugs.normalize(a, sr, metadata=m),\n       \"speed 1.4x\":         lambda a, sr, m: audaugs.speed(a, sr, factor=1.4, metadata=m),\n       \"tempo 0.8x\":         lambda a, sr, m: audaugs.tempo(a, sr, factor=0.8, metadata=m),\n       \"reverb\":             lambda a, sr, m: audaugs.reverb(a, sr, reverberance=80.0, metadata=m),\n   }\n   ok, audio_meta = [], []\n   for name, fn in AUDIO_ZOO.items():\n       m = []\n       try:\n           out, out_sr = fn(AUDIO.copy(), SR, m)\n           ok.append((name, np.asarray(out).squeeze(), out_sr))\n           audio_meta.append({\"name\": m[0][\"name\"], \"intensity\": m[0].get(\"intensity\"),\n                              \"dst_duration\": np.asarray(out).squeeze().shape[-1] / out_sr})\n       except Exception as e:\n           print(f\"  [skip] {name}: {type(e).__name__}: {str(e)[:90]}\")\n   print(\"\\n\" + pd.DataFrame(audio_meta).to_string(index=False,\n                                                   float_format=lambda v: f\"{v:.3f}\"))\n   n = min(len(ok), 8)\n   fig, axes = plt.subplots(n + 1, 1, figsize=(11, 1.5 * (n + 1)), sharex=False)\n   axes[0].plot(AUDIO, lw=.5, color=\"k\"); axes[0].set_ylabel(\"orig\", fontsize=7)\n   for ax, (name, sig, _) in zip(axes[1:], ok[:n]):\n       ax.plot(sig, lw=.5); ax.set_ylabel(name, fontsize=6, rotation=0, ha=\"right\", va=\"center\")\n   for ax in axes:\n       ax.set_xticks([]); ax.set_yticks([])\n   fig.suptitle(\"§8  Audio waveforms after AugLy augmentation\")\n   plt.tight_layout(); plt.show()\n   try:\n       from IPython.display import Audio, display\n       print(\"original:\"); display(Audio(AUDIO, rate=SR))\n       for name, sig, sr_out in ok[:3]:\n           print(name); display(Audio(sig, rate=sr_out))\n   except Exception:\n       pass\nexcept ImportError as e:\n   print(f\"  audio module unavailable ({e}).\")\n   print(\"  On Colab librosa/torch/torchaudio are preinstalled; elsewhere run:\")\n   print('    pip install \"librosa>=0.8.1\" soundfile audioread torch torchaudio')\nprint(\"\\n### §9  METADATA WAREHOUSE \" + \"#\" * 50)\nwarehouse = []\nrandom.seed(SEED)\nfor idx, im in enumerate(IMAGES[:8]):\n   for name, fn in IMAGE_ZOO.items():\n       meta = []\n       try:\n           out = fn(im, meta)\n       except Exception:\n           continue\n       m = meta[0]\n       warehouse.append({\n           \"source_id\": idx,\n           \"augmentation\": m[\"name\"],\n           \"intensity\": m.get(\"intensity\"),\n           \"src_w\": m.get(\"src_width\"), \"src_h\": m.get(\"src_height\"),\n           \"dst_w\": m.get(\"dst_width\"), \"dst_h\": m.get(\"dst_height\"),\n           \"area_ratio\": (m.get(\"dst_width\", 0) * m.get(\"dst_height\", 0)) /\n                         max(m.get(\"src_width\", 1) * m.get(\"src_height\", 1), 1),\n       })\nwh = pd.DataFrame(warehouse)\nprint(wh.head(8).to_string(index=False, float_format=lambda v: f\"{v:.2f}\"))\nprint(f\"\\nlogged {len(wh)} augmented samples from {wh.source_id.nunique()} sources\")\nprint(\"\\nhardest augmentations by mean intensity:\")\nprint(wh.groupby(\"augmentation\").intensity.mean().sort_values(ascending=False)\n       .head(8).to_string(float_format=lambda v: f\"{v:.1f}\"))\nwh.to_csv(\"augly_metadata.csv\", index=False)\nprint(\"\\nwrote augly_metadata.csv  (join this to your training manifest)\")\n```\n\nWe extend the augmentation workflow to audio by applying transformations such as pitch shifting, time stretching, filtering, noise injection, and reverb while gracefully skipping unavailable dependencies. We inspect the resulting waveforms and, where supported, play augmented samples directly in Colab. We also build a metadata warehouse that records augmentation type, intensity, dimensions, and area changes for downstream analysis.\n\n``` python\nprint(\"\\n### §10  PYTORCH DATASET / DATALOADER \" + \"#\" * 39)\ntry:\n   import torch\n   from torch.utils.data import Dataset, DataLoader\n   import torchvision.transforms as T\n   class AugLyDataset(Dataset):\n       \"\"\"AugLy transforms are drop-in for torchvision because they are PIL->PIL.\"\"\"\n       def __init__(self, images, labels, train=True):\n           self.images, self.labels = images, labels\n           aug = imaugs.Compose([\n               imaugs.OneOf([imaugs.RandomBlur(min_radius=0.5, max_radius=2.5),\n                             imaugs.RandomPixelization(min_ratio=0.3, max_ratio=1.0),\n                             imaugs.EncodingQuality(quality=20)], p=0.8),\n               imaugs.RandomBrightness(min_factor=0.7, max_factor=1.4, p=0.6),\n               imaugs.RandomEmojiOverlay(p=0.3),\n               RecompressionChain(n_rounds=1, p=0.3),\n           ])\n           steps = ([aug] if train else []) + [\n               T.Resize((128, 128)),\n               T.ToTensor(),\n               T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n           ]\n           self.tf = T.Compose(steps)\n       def __len__(self):\n           return len(self.images)\n       def __getitem__(self, i):\n           return self.tf(self.images[i]), self.labels[i]\n   ds = AugLyDataset(IMAGES, [i % 3 for i in range(len(IMAGES))], train=True)\n   dl = DataLoader(ds, batch_size=8, shuffle=True, num_workers=0)\n   xb, yb = next(iter(dl))\n   print(f\"  batch tensor {tuple(xb.shape)}  dtype={xb.dtype}  labels={yb.tolist()}\")\n   denorm = (xb[:8] * torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)\n             + torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)).clamp(0, 1)\n   show_grid([(f\"batch[{i}] y={yb[i].item()}\", denorm[i].permute(1, 2, 0).numpy())\n              for i in range(min(8, len(denorm)))],\n             cols=4, title=\"§10  One augmented batch straight out of the DataLoader\")\n   np_img = np.asarray(DEMO_IMG)\n   np_out = imaugs.aug_np_wrapper(np_img, imaugs.overlay_emoji,\n                                  **{\"opacity\": 0.8, \"y_pos\": 0.4})\n   print(f\"  aug_np_wrapper: {np_img.shape} -> {np_out.shape} ({np_out.dtype})\")\nexcept ImportError:\n   print(\"  torch/torchvision not installed — skipping (Colab has them by default).\")\nprint(\"\\n\" + \"=\" * 78)\nprint(\"DONE. Ideas from here:\")\nprint(\"  * swap pHash in §6 for a real embedding (CLIP / DINOv2) and re-run the table\")\nprint(\"  * feed §9's CSV into a curriculum: train on low-intensity augs first\")\nprint(\"  * add augly.video (pip install 'augly' + apt install ffmpeg) and\")\nprint(\"    benchmark frame-level robustness the same way\")\nprint(\"=\" * 78)\n```\n\nWe integrate AugLy directly into a PyTorch Dataset and DataLoader, allowing augmentations to run as part of the training-time preprocessing pipeline. We apply normalization and tensor conversion after augmentation and visualize a generated training batch to verify the complete data path. We also demonstrate AugLy’s NumPy-native wrapper and summarize practical extensions for embedding-based and video robustness benchmarks.\n\nIn conclusion, we showed how to use AugLy as more than a collection of independent augmentation functions by treating it as a systematic framework for robustness engineering. We measured how different image transformations affect copy-detection retrieval, show how adversarial text transformations expose weaknesses in conventional classifiers, and evaluate sanitization and adversarial training as complementary defenses. We also preserved augmentation metadata and intensity information so every generated sample remains traceable and analyzable, while custom transforms let us model application-specific distortions. By integrating image, text, audio, and PyTorch workflows within one reproducible pipeline, we established a foundation for building augmentation-aware training systems, robustness benchmarks, and production data pipelines.\n\nCheck out the **[FULL CODES here](https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Data%20Augmentation/AugLy_Multimodal_Augmentation_and_Robustness_Benchmark_Marktechpost.ipynb)**. All credit goes to the researcher of this project. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/MJjjVDPS7whH8Ngs6)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for", "canonical_source": "https://www.marktechpost.com/2026/09/26/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-with-augly-for-images-text-audio-and-pytorch/", "published_at": "2026-09-26 07:22:46+00:00", "updated_at": "2026-09-26 07:29:34.151106+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "developer-tools", "natural-language-processing", "computer-vision"], "entities": ["AugLy", "Meta", "PyTorch", "Python", "NumPy", "Pillow", "pandas", "matplotlib"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for", "markdown": "https://wpnews.pro/news/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for.md", "text": "https://wpnews.pro/news/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for.txt", "jsonld": "https://wpnews.pro/news/end-to-end-multimodal-data-augmentation-and-adversarial-robustness-benchmark-for.jsonld"}}