End-to-End Multimodal Data Augmentation and Adversarial Robustness Benchmark with AugLy for Images, Text, Audio, and PyTorch

A tutorial demonstrates an end-to-end multimodal data augmentation and adversarial robustness workflow using Meta's AugLy library for images, text, and audio, connected directly to PyTorch datasets and DataLoaders. The workflow benchmarks perceptual-hash copy detection under image distortions and evaluates text classifiers against adversarial perturbations, Unicode obfuscation, sanitization, and adversarial training, with deterministic synthetic datasets seeded at 1234 for reproducibility. It also covers AugLy's functional and class-based APIs, metadata and intensity tracking, probabilistic composition, bounding-box-aware transformations, and a queryable metadata warehouse.

In this tutorial, we build a comprehensive multimodal augmentation and robustness workflow with AugLy https://github.com/facebookresearch/AugLy for images, text, and audio. We start by addressing modern dependency compatibility issues and generating deterministic synthetic datasets so the experiments remain self-contained and reproducible. We then explore AugLy’s functional and class-based APIs, metadata, and intensity tracking, probabilistic composition, bounding-box-aware transformations, and custom transforms. We extend the workflow into practical robustness experiments by benchmarking perceptual-hash copy detection under image distortions and evaluating text classifiers against adversarial perturbations, Unicode obfuscation, sanitization, and adversarial training. We also integrate audio augmentation, build a queryable metadata warehouse, and connect AugLy transformations directly to PyTorch datasets and DataLoaders, giving us an end-to-end view of augmentation as both a data-generation mechanism and a measurable robustness tool. python import subprocess, sys, importlib def sh cmd : print f"$ {cmd}" subprocess.run cmd, shell=True, check=False, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL def need mod : try: importlib.import module mod return False except ImportError: return True if need "augly" : sh "apt-get -qq install -y libmagic1 /dev/null 2 &1" sh f'"{sys.executable}" -m pip install -q --no-deps augly' sh f'"{sys.executable}" -m pip install -q "iopath =0.1.8" "python-magic =0.4.22" ' f'"regex =2021.4.4" "nlpaug==1.1.3"' import numpy as np from PIL import Image, ImageDraw, ImageFont, ImageFilter for name, builtin in "float", float , "int", int , "bool", bool : if not hasattr np, name : setattr np, name, builtin def size font, text : left, top, right, bottom = font.getbbox text return right, bottom if not hasattr ImageFont.FreeTypeFont, "getsize" : ImageFont.FreeTypeFont.getsize = lambda self, t, a, k: size self, t if not hasattr ImageFont.FreeTypeFont, "getsize multiline" : def getsize multiline self, text, direction=None, spacing=4, features=None, language=None, stroke width=0 : lines = text.split "\n" w = max size self, ln 0 for ln in lines , default=0 h = sum size self, ln 1 for ln in lines + spacing len lines - 1 return w, h ImageFont.FreeTypeFont.getsize multiline = getsize multiline import os, io, json, math, random, string, textwrap, unicodedata, warnings from dataclasses import dataclass from typing import Any, Dict, List, Optional, Tuple import matplotlib.pyplot as plt import pandas as pd import augly.image as imaugs import augly.text as textaugs import augly.utils as augutils from augly.image.transforms import BaseTransform as ImageBaseTransform warnings.filterwarnings "ignore" pd.set option "display.width", 160 SEED = 1234 random.seed SEED np.random.seed SEED print "\n" + "=" 78 print "AugLy ready. assets at:", augutils.ASSETS BASE DIR print "image augs :", len f for f in dir imaugs if f 0 .islower print "text augs :", len f for f in dir textaugs if f 0 .islower print "=" 78 + "\n" def make image idx: int, w: int = 320, h: int = 240 - Tuple Image.Image, Tuple int, int, int, int : """Procedurally generated 'photo' + a ground-truth bbox in pascal voc format.""" rng = random.Random SEED + idx img = Image.new "RGB", w, h , tuple rng.randint 20, 90 for in range 3 d = ImageDraw.Draw img for in range 70 : x0, y0 = rng.randint 0, w , rng.randint 0, h d.line x0, y0, x0 + rng.randint -60, 60 , y0 + rng.randint -60, 60 , fill=tuple rng.randint 60, 160 for in range 3 , width=rng.randint 1, 3 ow, oh = rng.randint 70, 130 , rng.randint 60, 110 ox, oy = rng.randint 10, w - ow - 10 , rng.randint 10, h - oh - 10 box = ox, oy, ox + ow, oy + oh colour = tuple rng.randint 150, 255 for in range 3 if idx % 3 == 0: d.ellipse box, fill=colour, outline= 255, 255, 255 , width=3 elif idx % 3 == 1: d.rectangle box, fill=colour, outline= 255, 255, 255 , width=3 else: d.polygon ox + ow // 2, oy , ox + ow, oy + oh , ox, oy + oh , fill=colour, outline= 255, 255, 255 return img, box N IMAGES = 24 IMAGES, BOXES = zip make image i for i in range N IMAGES IMAGES, BOXES = list IMAGES , list BOXES DEMO IMG, DEMO BOX = IMAGES 0 , BOXES 0 def make text dataset n per class: int = 260 : """Tiny sentiment corpus built from templates - learnable but not trivial.""" rng = random.Random SEED pos adj = "excellent", "delightful", "superb", "charming", "brilliant", "flawless", "wonderful", "outstanding", "impressive", "lovely" neg adj = "terrible", "awful", "dreadful", "disappointing", "clumsy", "broken", "miserable", "useless", "painful", "sloppy" subj = "the movie", "this restaurant", "the hotel room", "their support team", "the new phone", "the sequel", "this laptop", "the delivery service" tail p = "and I would recommend it to anyone", "worth every rupee", "I left completely satisfied", "easily the best of the year", "it exceeded all my expectations" tail n = "and I want a refund", "a total waste of money", "I left extremely frustrated", "easily the worst of the year", "it failed every expectation" rows = for in range n per class : rows.append f"{rng.choice subj } was {rng.choice pos adj } {rng.choice tail p }", 1 rows.append f"{rng.choice subj } was {rng.choice neg adj } {rng.choice tail n }", 0 rng.shuffle rows return r 0 for r in rows , r 1 for r in rows TEXTS, LABELS = make text dataset DEMO TEXT = "The quick brown fox jumps over the lazy dog near the river bank" def make audio seconds: float = 2.0, sr: int = 16000 - Tuple np.ndarray, int : """A chirp + harmonics + a little noise = something you can actually hear change.""" t = np.linspace 0, seconds, int sr seconds , endpoint=False f = np.linspace 220, 880, t.size sig = 0.5 np.sin 2 np.pi f t + 0.2 np.sin 2 np.pi 2 f t sig += 0.02 np.random.RandomState SEED .randn t.size env = np.minimum 1.0, np.minimum t 8, seconds - t 8 return sig env .astype np.float32 , sr AUDIO, SR = make audio def show grid pairs, cols=4, title="", figsize scale=2.9 : """pairs: list of caption, PIL.Image .""" rows = math.ceil len pairs / cols fig, axes = plt.subplots rows, cols, figsize= cols figsize scale, rows figsize scale axes = np.atleast 1d axes .ravel for ax, cap, im in zip axes, pairs : ax.imshow im ax.set title cap, fontsize=8 ax.axis "off" for ax in axes len pairs : : ax.axis "off" if title: fig.suptitle title, fontsize=13, y=1.0 plt.tight layout plt.show def as str out - str: """AugLy text augs return str for str input in some transforms, list in others.""" return out 0 if isinstance out, list else out print "\n §2 IMAGE AUGMENTATION + METADATA " + " " 38 functional result = imaugs.pixelization DEMO IMG, ratio=0.25 class result = imaugs.Pixelization ratio=0.25, p=1.0 DEMO IMG print "functional == class:", np.array equal np.array functional result , np.array class result IMAGE ZOO = { "blur": lambda im, m: imaugs.blur im, radius=3.0, metadata=m , "brightness": lambda im, m: imaugs.brightness im, factor=1.7, metadata=m , "color jitter": lambda im, m: imaugs.color jitter im, brightness factor=1.3, contrast factor=1.4, saturation factor=1.6, metadata=m , "crop": lambda im, m: imaugs.crop im, x1=.15, y1=.15, x2=.85, y2=.85, metadata=m , "encoding quality": lambda im, m: imaugs.encoding quality im, quality=8, metadata=m , "grayscale": lambda im, m: imaugs.grayscale im, metadata=m , "hflip": lambda im, m: imaugs.hflip im, metadata=m , "meme format": lambda im, m: imaugs.meme format im, text="TOP TEXT", caption height=90, metadata=m , "opacity": lambda im, m: imaugs.opacity im, level=0.45, metadata=m , "overlay emoji": lambda im, m: imaugs.overlay emoji im, opacity=0.9, emoji size=0.35, metadata=m , "overlay screenshot": lambda im, m: imaugs.overlay onto screenshot im, metadata=m , "overlay stripes": lambda im, m: imaugs.overlay stripes im, line width=0.4, line opacity=0.7, metadata=m , "overlay text": lambda im, m: imaugs.overlay text im, opacity=0.9, metadata=m , "pad square": lambda im, m: imaugs.pad square im, metadata=m , "perspective": lambda im, m: imaugs.perspective transform im, sigma=40.0, metadata=m , "pixelization": lambda im, m: imaugs.pixelization im, ratio=0.15, metadata=m , "random noise": lambda im, m: imaugs.random noise im, var=0.03, metadata=m , "rotate": lambda im, m: imaugs.rotate im, degrees=17, metadata=m , "saturation": lambda im, m: imaugs.saturation im, factor=3.0, metadata=m , "scale": lambda im, m: imaugs.scale im, factor=0.35, metadata=m , "sharpen": lambda im, m: imaugs.sharpen im, factor=8.0, metadata=m , "shuffle pixels": lambda im, m: imaugs.shuffle pixels im, factor=0.15, metadata=m , "skew": lambda im, m: imaugs.skew im, skew factor=0.35, metadata=m , "vflip": lambda im, m: imaugs.vflip im, metadata=m , } gallery, image meta = "ORIGINAL", DEMO IMG , for name, fn in IMAGE ZOO.items : m = try: out = fn DEMO IMG, m gallery.append f"{name}\nintensity={m 0 'intensity' :.1f}", out image meta.append m 0 except Exception as e: print f" skip {name}: {type e . name }: {e}" show grid gallery, cols=5, title="§2 AugLy image augmentations with AugLy's own intensity score " meta df = pd.DataFrame image meta "name", "intensity", "src width", "src height", "dst width", "dst height" print meta df.sort values "intensity", ascending=False .head 10 .to string index=False We set up AugLy in a modern Colab environment while adding compatibility shims for NumPy and Pillow. We generate deterministic synthetic image, text, and audio datasets without external downloads. We also initialize reusable visualization and utility functions before exploring image augmentation and metadata. print "\n §3 COMPOSITION & REPRODUCIBILITY " + " " 39 REUPLOAD PIPELINE = imaugs.Compose imaugs.OneOf imaugs.OverlayOntoScreenshot , imaugs.MemeFormat text="LOL", caption height=80 , imaugs.OverlayStripes line width=0.3, line opacity=0.5 , , p=0.9 , imaugs.RandomAspectRatio min ratio=0.7, max ratio=1.4, p=0.5 , imaugs.RandomEmojiOverlay p=0.7 , imaugs.RandomBrightness min factor=0.7, max factor=1.4, p=0.6 , imaugs.EncodingQuality quality=12, p=1.0 , def run pipeline img, seed=None : """AugLy image transforms use the global random module - seed it for determinism.""" if seed is not None: random.seed seed np.random.seed seed meta = return REUPLOAD PIPELINE img, metadata=meta , meta a, meta a = run pipeline DEMO IMG, seed=7 b, meta b = run pipeline DEMO IMG, seed=7 c, = run pipeline DEMO IMG, seed=99 print "same seed - identical output:", np.array equal np.array a , np.array b print "applied chain seed=7 :", " - ".join m "name" for m in meta a show grid "original", DEMO IMG , "seed=7", a , "seed=7 again", b , "seed=99", c , cols=4, title="§3 Seeded, reproducible augmentation pipelines" print "\n §4 BBOX-AWARE AUGMENTATION " + " " 45 BBOX OPS = "crop", lambda im, m, bb: imaugs.crop im, x1=.1, y1=.1, x2=.9, y2=.9, metadata=m, bboxes=bb, bbox format="pascal voc" , "hflip", lambda im, m, bb: imaugs.hflip im, metadata=m, bboxes=bb, bbox format="pascal voc" , "rotate 20", lambda im, m, bb: imaugs.rotate im, degrees=20, metadata=m, bboxes=bb, bbox format="pascal voc" , "pad", lambda im, m, bb: imaugs.pad im, w factor=0.25, h factor=0.25, metadata=m, bboxes=bb, bbox format="pascal voc" , "meme format", lambda im, m, bb: imaugs.meme format im, text="BOXED", caption height=80, metadata=m, bboxes=bb, bbox format="pascal voc" , def draw box img, box, colour= 0, 255, 0 : out = img.copy .convert "RGB" ImageDraw.Draw out .rectangle float v for v in box , outline=colour, width=4 return out bbox panels = "original", draw box DEMO IMG, DEMO BOX for label, op in BBOX OPS: m = try: out = op DEMO IMG, m, DEMO BOX dst = m 0 "dst bboxes" 0 bbox panels.append f"{label}\n{tuple round v for v in dst }", draw box out, dst print f" {label:12s} {DEMO BOX} - {tuple round v, 1 for v in dst }" except Exception as e: print f" skip {label}: {type e . name }: {e}" show grid bbox panels, cols=3, title="§4 Boxes follow the pixels automatically" print "\n §5 CUSTOM TRANSFORMS " + " " 51 class RecompressionChain ImageBaseTransform : """Simulate an image surviving N rounds of platform re-encoding. Subclassing BaseTransform rather than using ApplyLambda buys you: the p probability gate, force=True , and full participation in Compose/OneOf. """ def init self, n rounds: int = 3, min q: int = 12, max q: int = 45, downscale: float = 0.85, p: float = 1.0 : super . init p self.n rounds, self.min q, self.max q, self.downscale = n rounds, min q, max q, downscale def apply transform self, image, metadata=None, bboxes=None, bbox format=None : src w, src h = image.size out, qualities = image, for in range self.n rounds : q = random.randint self.min q, self.max q qualities.append q out = imaugs.encoding quality out, quality=q out = imaugs.scale out, factor=self.downscale out = out.resize src w, src h , Image.BILINEAR if metadata is not None: metadata.append { "name": "recompression chain", "src width": src w, "src height": src h, "dst width": out.size 0 , "dst height": out.size 1 , "n rounds": self.n rounds, "qualities": qualities, "intensity": float 100 1 - np.mean qualities / 100 , } return out vignette = imaugs.ApplyLambda aug function=lambda im: Image.composite im, Image.new "RGB", im.size, 0, 0, 0 , Image.radial gradient "L" .resize im.size .point lambda v: 255 - v random.seed SEED custom meta = show grid "original", DEMO IMG , "RecompressionChain n=3 ", RecompressionChain n rounds=3 DEMO IMG, metadata=custom meta , "RecompressionChain n=6 ", RecompressionChain n rounds=6, min q=5, max q=20 DEMO IMG , "ApplyLambda vignette", vignette DEMO IMG , , cols=4, title="§5 Custom transforms drop straight into the AugLy API" print " custom metadata:", custom meta 0 CUSTOM PIPELINE = imaugs.Compose RecompressionChain n rounds=2, p=1.0 , imaugs.RandomEmojiOverlay p=1.0 = CUSTOM PIPELINE DEMO IMG print " composed with built-ins: OK" We construct probabilistic augmentation pipelines with Compose and OneOf while controlling reproducibility through explicit random seeds. We demonstrate how AugLy automatically propagates bounding-box coordinates through spatial transformations. We then implement a custom BaseTransform and combine it with built-in AugLy transforms. print "\n §6 COPY-DETECTION ROBUSTNESS BENCHMARK " + " " 33 from scipy.fftpack import dct def phash img: Image.Image, hash size: int = 8, highfreq: int = 4 - np.ndarray: """Classic DCT perceptual hash - 64-bit signature as a bool array.""" size = hash size highfreq px = np.asarray img.convert "L" .resize size, size , Image.LANCZOS , dtype=np.float64 d = dct dct px, axis=0, norm="ortho" , axis=1, norm="ortho" :hash size, :hash size return d np.median d 1:, 1: .ravel def hamming a, b - int: return int np.count nonzero a = b INDEX = np.stack phash im for im in IMAGES ATTACKS = { "brightness x1.6": lambda im: imaugs.brightness im, factor=1.6 , "blur r=3": lambda im: imaugs.blur im, radius=3.0 , "jpeg q=8": lambda im: imaugs.encoding quality im, quality=8 , "crop 80%": lambda im: imaugs.crop im, x1=.1, y1=.1, x2=.9, y2=.9 , "rotate 12": lambda im: imaugs.rotate im, degrees=12 , "hflip": lambda im: imaugs.hflip im , "grayscale": lambda im: imaugs.grayscale im , "pixelize 0.2": lambda im: imaugs.pixelization im, ratio=0.2 , "noise var=.03": lambda im: imaugs.random noise im, var=0.03 , "emoji overlay": lambda im: imaugs.overlay emoji im, emoji size=0.35, opacity=0.9 , "meme format": lambda im: imaugs.meme format im, text="LOL", caption height=70 , "screenshot": lambda im: imaugs.overlay onto screenshot im , "perspective s=40": lambda im: imaugs.perspective transform im, sigma=40.0 , "scale 0.35": lambda im: imaugs.scale im, factor=0.35 , "stripes": lambda im: imaugs.overlay stripes im, line width=0.4, line opacity=0.7 , "re-encode chain": lambda im: RecompressionChain n rounds=3 im , "REUPLOAD pipeline": lambda im: REUPLOAD PIPELINE im , } rows = for attack, fn in ATTACKS.items : random.seed SEED np.random.seed SEED hits, dists, failures = 0, , 0 for i, im in enumerate IMAGES : try: q = phash fn im except Exception: failures += 1 continue d = np.array hamming q, h for h in INDEX hits += int d.argmin == i dists.append int d i n = len IMAGES - failures rows.append {"attack": attack, "top1 recall": hits / max n, 1 , "mean hamming": float np.mean dists if dists else np.nan, "errors": failures} bench = pd.DataFrame rows .sort values "top1 recall" print bench.to string index=False, float format=lambda v: f"{v:.3f}" fig, ax = plt.subplots 1, 2, figsize= 14, 6 colours = " c0392b" if r < .5 else " e67e22" if r < .9 else " 27ae60" for r in bench.top1 recall ax 0 .barh bench.attack, bench.top1 recall, color=colours ax 0 .set xlabel "top-1 retrieval recall" ; ax 0 .set xlim 0, 1.05 ax 0 .axvline 0.9, ls="--", c="k", lw=1 ax 0 .set title "pHash survival per AugLy attack" ax 1 .scatter bench.mean hamming, bench.top1 recall, s=70, c=colours for , r in bench.iterrows : ax 1 .annotate r.attack, r.mean hamming, r.top1 recall , fontsize=7, xytext= 3, 3 , textcoords="offset points" ax 1 .set xlabel "mean Hamming distance to the true match 0-64 " ax 1 .set ylabel "top-1 recall" ax 1 .set title "Distortion vs. retrieval failure" plt.tight layout ; plt.show worst = bench.head 3 .attack.tolist print f"\n pHash breaks under: {worst}" print " - exactly the augmentations you'd add to training, or handle with a" print " geometry-invariant embedding instead of a hash." We build a perceptual-hash index over the synthetic image corpus and evaluate its robustness against a broad collection of AugLy distortions. We measure top-1 retrieval recall and Hamming distance for every attack to quantify how different transformations affect copy detection. We visualize the results to identify the augmentations that most strongly degrade perceptual matching. print "\n §7 TEXT ATTACK / DEFEND / HARDEN " + " " 39 from sklearn.feature extraction.text import TfidfVectorizer from sklearn.linear model import LogisticRegression from sklearn.pipeline import make pipeline from sklearn.model selection import train test split from sklearn.metrics import accuracy score X tr, X te, y tr, y te = train test split TEXTS, LABELS, test size=0.3, random state=SEED, stratify=LABELS def new model : return make pipeline TfidfVectorizer analyzer="word", ngram range= 1, 2 , sublinear tf=True , LogisticRegression max iter=1000, C=4.0 , baseline = new model .fit X tr, y tr clean acc = accuracy score y te, baseline.predict X te print f"clean test accuracy: {clean acc:.3f}\n" ATTACK SUITE = { "typos keyboard+misspell ": textaugs.SimulateTypos aug word p=0.45, typo type="all" , "unicode homoglyphs": textaugs.ReplaceSimilarUnicodeChars aug word p=0.8, aug char p=0.4 , "leetspeak lookalikes": textaugs.ReplaceSimilarChars aug word p=0.8, aug char p=0.4 , "zero-width injection": textaugs.InsertZeroWidthChars granularity="word", cadence=2.0 , "punctuation injection": textaugs.InsertPunctuationChars granularity="word", cadence=2.0, vary chars=True , "whitespace injection": textaugs.InsertWhitespaceChars granularity="word", cadence=3.0 , "fun fonts": textaugs.ReplaceFunFonts aug p=0.8, granularity="word", vary fonts=True , "upside down": textaugs.ReplaceUpsideDown aug p=0.6, granularity="word" , "bidirectional": textaugs.ReplaceBidirectional granularity="word" , "split words": textaugs.SplitWords aug word p=0.5 , "merge words": textaugs.MergeWords aug word p=0.5 , "CaSe ChAoS": textaugs.ChangeCase granularity="word", cadence=2.0, case="upper" , } print "what the attacks look like on one sentence:" for name, aug in ATTACK SUITE.items : random.seed SEED print f" {name:26s} {as str aug DEMO TEXT :72 }" ZERO WIDTH = dict.fromkeys 0x200B, 0x200C, 0x200D, 0x2060, 0x2061, 0x2062, 0x2063, 0x2064, 0xFEFF, 0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F PUNCT TABLE = str.maketrans "", "", "".join c for c in string.punctuation if c not in "'" def sanitize text: str - str: """Cheap, high-yield preprocessing against Unicode-obfuscation attacks.""" t = text.translate ZERO WIDTH t = unicodedata.normalize "NFKD", t t = "".join ch for ch in t if not unicodedata.combining ch t = t.translate PUNCT TABLE return " ".join t.lower .split random.seed SEED TRAIN AUGS = ATTACK SUITE k for k in "typos keyboard+misspell ", "unicode homoglyphs", "leetspeak lookalikes", "zero-width injection", "punctuation injection", "fun fonts", "CaSe ChAoS" aug X, aug y = list X tr , list y tr for aug in TRAIN AUGS: for x, y in zip X tr, y tr : aug X.append as str aug x aug y.append y print f"\ntrain set: {len X tr } - {len aug X } examples after AugLy adversarial training" hardened = make pipeline TfidfVectorizer analyzer="char wb", ngram range= 3, 5 , sublinear tf=True, preprocessor=sanitize , LogisticRegression max iter=2000, C=4.0 , .fit aug X, aug y results = for name, aug in ATTACK SUITE.items : random.seed SEED X atk = as str aug x for x in X te results.append { "attack": name, "baseline": accuracy score y te, baseline.predict X atk , "+ sanitize": accuracy score y te, baseline.predict sanitize x for x in X atk , "+ sanitize + adv-train": accuracy score y te, hardened.predict X atk , } res = pd.DataFrame results .set index "attack" .sort values "baseline" res.loc "-- CLEAN --" = clean acc, accuracy score y te, baseline.predict sanitize x for x in X te , accuracy score y te, hardened.predict X te print "\n" + res.to string float format=lambda v: f"{v:.3f}" print f"\nmean accuracy under attack: baseline {res.iloc :-1,0 .mean :.3f}" f" - sanitized {res.iloc :-1,1 .mean :.3f}" f" - hardened {res.iloc :-1,2 .mean :.3f}" plot df = res.drop index="-- CLEAN --" ax = plot df.plot.barh figsize= 11, 7 , width=0.78, color= " c0392b", " e6a817", " 27ae60" ax.axvline clean acc, ls="--", c="k", lw=1.2, label=f"clean acc = {clean acc:.2f}" ax.set xlabel "accuracy under attack" ; ax.set xlim 0, 1.05 ax.set title "§7 AugLy as a red-team suite — and as the fix" ax.legend loc="lower right", fontsize=8 plt.tight layout ; plt.show We create a text classification baseline and systematically expose it to typos, Unicode homoglyphs, invisible characters, punctuation changes, and other adversarial transformations. We implement Unicode normalization and sanitization to remove several classes of obfuscation. We then use AugLy-generated adversarial examples during training and compare the resulting hardened model against the baseline. python print "\n §8 AUDIO AUGMENTATION " + " " 50 try: import augly.audio as audaugs AUDIO ZOO = { "pitch shift +4": lambda a, sr, m: audaugs.pitch shift a, sr, n steps=4.0, metadata=m , "time stretch 1.5x": lambda a, sr, m: audaugs.time stretch a, sr, rate=1.5, metadata=m , "change volume -12dB":lambda a, sr, m: audaugs.change volume a, sr, volume db=-12.0, metadata=m , "clicks": lambda a, sr, m: audaugs.clicks a, sr, seconds between clicks=0.25, metadata=m , "background noise": lambda a, sr, m: audaugs.add background noise a, sr, snr level db=5.0, metadata=m , "low pass 1kHz": lambda a, sr, m: audaugs.low pass filter a, sr, cutoff hz=1000.0, metadata=m , "high pass 2kHz": lambda a, sr, m: audaugs.high pass filter a, sr, cutoff hz=2000.0, metadata=m , "peaking eq": lambda a, sr, m: audaugs.peaking equalizer a, sr, center hz=800.0, gain db=-12.0, metadata=m , "harmonic": lambda a, sr, m: audaugs.harmonic a, sr, metadata=m , "percussive": lambda a, sr, m: audaugs.percussive a, sr, metadata=m , "clip 50% ": lambda a, sr, m: audaugs.clip a, sr, duration factor=0.5, metadata=m , "loop x2": lambda a, sr, m: audaugs.loop a, sr, n=1, metadata=m , "normalize": lambda a, sr, m: audaugs.normalize a, sr, metadata=m , "speed 1.4x": lambda a, sr, m: audaugs.speed a, sr, factor=1.4, metadata=m , "tempo 0.8x": lambda a, sr, m: audaugs.tempo a, sr, factor=0.8, metadata=m , "reverb": lambda a, sr, m: audaugs.reverb a, sr, reverberance=80.0, metadata=m , } ok, audio meta = , for name, fn in AUDIO ZOO.items : m = try: out, out sr = fn AUDIO.copy , SR, m ok.append name, np.asarray out .squeeze , out sr audio meta.append {"name": m 0 "name" , "intensity": m 0 .get "intensity" , "dst duration": np.asarray out .squeeze .shape -1 / out sr} except Exception as e: print f" skip {name}: {type e . name }: {str e :90 }" print "\n" + pd.DataFrame audio meta .to string index=False, float format=lambda v: f"{v:.3f}" n = min len ok , 8 fig, axes = plt.subplots n + 1, 1, figsize= 11, 1.5 n + 1 , sharex=False axes 0 .plot AUDIO, lw=.5, color="k" ; axes 0 .set ylabel "orig", fontsize=7 for ax, name, sig, in zip axes 1: , ok :n : ax.plot sig, lw=.5 ; ax.set ylabel name, fontsize=6, rotation=0, ha="right", va="center" for ax in axes: ax.set xticks ; ax.set yticks fig.suptitle "§8 Audio waveforms after AugLy augmentation" plt.tight layout ; plt.show try: from IPython.display import Audio, display print "original:" ; display Audio AUDIO, rate=SR for name, sig, sr out in ok :3 : print name ; display Audio sig, rate=sr out except Exception: pass except ImportError as e: print f" audio module unavailable {e} ." print " On Colab librosa/torch/torchaudio are preinstalled; elsewhere run:" print ' pip install "librosa =0.8.1" soundfile audioread torch torchaudio' print "\n §9 METADATA WAREHOUSE " + " " 50 warehouse = random.seed SEED for idx, im in enumerate IMAGES :8 : for name, fn in IMAGE ZOO.items : meta = try: out = fn im, meta except Exception: continue m = meta 0 warehouse.append { "source id": idx, "augmentation": m "name" , "intensity": m.get "intensity" , "src w": m.get "src width" , "src h": m.get "src height" , "dst w": m.get "dst width" , "dst h": m.get "dst height" , "area ratio": m.get "dst width", 0 m.get "dst height", 0 / max m.get "src width", 1 m.get "src height", 1 , 1 , } wh = pd.DataFrame warehouse print wh.head 8 .to string index=False, float format=lambda v: f"{v:.2f}" print f"\nlogged {len wh } augmented samples from {wh.source id.nunique } sources" print "\nhardest augmentations by mean intensity:" print wh.groupby "augmentation" .intensity.mean .sort values ascending=False .head 8 .to string float format=lambda v: f"{v:.1f}" wh.to csv "augly metadata.csv", index=False print "\nwrote augly metadata.csv join this to your training manifest " We extend the augmentation workflow to audio by applying transformations such as pitch shifting, time stretching, filtering, noise injection, and reverb while gracefully skipping unavailable dependencies. We inspect the resulting waveforms and, where supported, play augmented samples directly in Colab. We also build a metadata warehouse that records augmentation type, intensity, dimensions, and area changes for downstream analysis. python print "\n §10 PYTORCH DATASET / DATALOADER " + " " 39 try: import torch from torch.utils.data import Dataset, DataLoader import torchvision.transforms as T class AugLyDataset Dataset : """AugLy transforms are drop-in for torchvision because they are PIL- PIL.""" def init self, images, labels, train=True : self.images, self.labels = images, labels aug = imaugs.Compose imaugs.OneOf imaugs.RandomBlur min radius=0.5, max radius=2.5 , imaugs.RandomPixelization min ratio=0.3, max ratio=1.0 , imaugs.EncodingQuality quality=20 , p=0.8 , imaugs.RandomBrightness min factor=0.7, max factor=1.4, p=0.6 , imaugs.RandomEmojiOverlay p=0.3 , RecompressionChain n rounds=1, p=0.3 , steps = aug if train else + T.Resize 128, 128 , T.ToTensor , T.Normalize 0.485, 0.456, 0.406 , 0.229, 0.224, 0.225 , self.tf = T.Compose steps def len self : return len self.images def getitem self, i : return self.tf self.images i , self.labels i ds = AugLyDataset IMAGES, i % 3 for i in range len IMAGES , train=True dl = DataLoader ds, batch size=8, shuffle=True, num workers=0 xb, yb = next iter dl print f" batch tensor {tuple xb.shape } dtype={xb.dtype} labels={yb.tolist }" denorm = xb :8 torch.tensor 0.229, 0.224, 0.225 .view 3, 1, 1 + torch.tensor 0.485, 0.456, 0.406 .view 3, 1, 1 .clamp 0, 1 show grid f"batch {i} y={yb i .item }", denorm i .permute 1, 2, 0 .numpy for i in range min 8, len denorm , cols=4, title="§10 One augmented batch straight out of the DataLoader" np img = np.asarray DEMO IMG np out = imaugs.aug np wrapper np img, imaugs.overlay emoji, {"opacity": 0.8, "y pos": 0.4} print f" aug np wrapper: {np img.shape} - {np out.shape} {np out.dtype} " except ImportError: print " torch/torchvision not installed — skipping Colab has them by default ." print "\n" + "=" 78 print "DONE. Ideas from here:" print " swap pHash in §6 for a real embedding CLIP / DINOv2 and re-run the table" print " feed §9's CSV into a curriculum: train on low-intensity augs first" print " add augly.video pip install 'augly' + apt install ffmpeg and" print " benchmark frame-level robustness the same way" print "=" 78 We integrate AugLy directly into a PyTorch Dataset and DataLoader, allowing augmentations to run as part of the training-time preprocessing pipeline. We apply normalization and tensor conversion after augmentation and visualize a generated training batch to verify the complete data path. We also demonstrate AugLy’s NumPy-native wrapper and summarize practical extensions for embedding-based and video robustness benchmarks. In conclusion, we showed how to use AugLy as more than a collection of independent augmentation functions by treating it as a systematic framework for robustness engineering. We measured how different image transformations affect copy-detection retrieval, show how adversarial text transformations expose weaknesses in conventional classifiers, and evaluate sanitization and adversarial training as complementary defenses. We also preserved augmentation metadata and intensity information so every generated sample remains traceable and analyzable, while custom transforms let us model application-specific distortions. By integrating image, text, audio, and PyTorch workflows within one reproducible pipeline, we established a foundation for building augmentation-aware training systems, robustness benchmarks, and production data pipelines. Check out the FULL CODES here https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Data%20Augmentation/AugLy Multimodal Augmentation and Robustness Benchmark Marktechpost.ipynb . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter https://x.com/intent/follow?screen name=marktechpost and don’t forget to join our 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ and Subscribe to our Newsletter https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}} . Wait are you on telegram? now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/MJjjVDPS7whH8Ngs6 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.