Testing AI image watermarks: C2PA and SynthID

An independent experiment found that OpenAI's C2PA Content Credentials can be removed from an AI-generated image by stripping its metadata with ImageMagick, while Google's SynthID pixel watermark survived metadata stripping, a 512x512 center crop and JPEG re-encoding at quality 75. The tester confirmed the C2PA removal was lossless by decoding both the original and stripped files to raw RGBA and matching SHA-256 hash ef994f090d047a181138e41a0e44bdec7cc22e4982bda529eecc2240c21c3cb2, and noted OpenAI did not embed SynthID in a simple generated rectangle.

Testing AI image watermarks: C2PA and SynthID C2PA and SynthID are two technologies used to identify AI-generated images. C2PA Content Credentials is pretty simple, as it just adds signed metadata to a file. SynthID, on the other hand, is a different beast. It adds a watermark directly into the pixels of images. I decided to experiment with these to understand more. Test 1: Plain rectangle Prompt: Let's inspect the image: c2patool ai-generated-images/pure-white-blank-canvas1.png --info The important part here is "One manifest". Let's extract the manifest and search for interesting fields. It is clear that this image has C2PA metadata. Stripping is pretty easy with magick. Let's inspect again, after stripping. At this point, the metadata is gone. </ javascript { "type": "c2pa", "outcome": "not detected" } Did we change the image, though? Let's decode these files into raw RGBA pixels and check the hash. </ javascript magick rectangle-original.png rgba:- | shasum -a 256 magick rectangle-stripped.png rgba:- | shasum -a 256 // Both output ef994f090d047a181138e41a0e44bdec7cc22e4982bda529eecc2240c21c3cb2 They both output the same hash, so the images are identical. Interestingly enough, OpenAI does not seem to add SynthID in simple shapes like these. Let's generate something better. Test 2: Apple I used the prompt "Draw a red apple, realistic studio photo, on a plain white background": Let's check for provenance signals. We could either use the web's version https://openai.com/research/verify/ or just call the API. I stripped the metadata and ran the check again. </ javascript { "object": "content provenance check", "results": { "type": "c2pa", "generated at": null, "issuer": null, "model": null, "outcome": "not detected", "validation state": "not present" }, { "type": "SynthID", "generated at": null, "model": null, "outcome": "detected" } } This is expected, as removing metadata does not remove SynthID. What happens with the crop? magick Pure-White-Canvas.png -gravity center -crop 512x512+0+0 +repage ai crop 512.png SynthID survives. JPEG conversion? magick apple-stripped.png -quality 75 apple-q75.jpg SynthID survives. Maybe a diffusion attack, then? ... pipe = StableDiffusionImg2ImgPipeline.from pretrained model id, torch dtype=torch.float16 if device == "mps" else torch.float32 prompt = "A high-quality, detailed, photorealistic macro photograph of a single glossy red apple with a brown stem, natural yellow streaks, isolated on a plain white background with a soft shadow" image = pipe prompt=prompt, image=init image, strength=0.25, guidance scale=7.5 .images 0 ... That is, convert the original image to numbers latents , add some noise, then denoise based on a prompt. But this is cheating. I am essentially regenerating the image from the original , not removing SynthID. This type of diffusion reconstruction will not exactly preserve the original image texture, color, edges, lighting, etc. . Guess I am back to analyzing pixels, and there is no better candidate for this than a nearly white image. Test 3: Plain white background Prompt: How about we compare this plain white image with a plain white image that I can create locally ? Let's analyze the AI image for spatial patterns or uniformities. python import numpy as np import matplotlib.pyplot as plt from PIL import Image Compute a windowed, 0 mean 2D FFT spectrum def fft spectrum channel : centered = channel - channel.mean dtype=np.float64 zero mean h, w = centered.shape window = np.outer np.hanning h , np.hanning w , windowed = centered window fft raw = np.fft.fft2 windowed fft shifted = np.fft.fftshift fft raw magnitude = np.log1p np.abs fft shifted return magnitude analyze spatial variation load image, calculate mean RGB independently, subtract mean color from every pixel and compute FFT spectra def analyze zero mean residual image path, output path="zero mean analysis.png", amp factor=50, : img = Image.open image path .convert "RGB" ai = np.array img, dtype=np.float64 height, width, channels = ai.shape print "--- Image ---" print f"Path: {image path}" print f"Dimensions: {width} x {height}" print f"Shape: {ai.shape}" print "--- Raw Pixel Statistics ---" print f"Overall min: {ai.min :.0f}" print f"Overall max: {ai.max :.0f}" print f"Overall mean: {ai.mean dtype=np.float64 :.6f}" unique values = np.unique ai.astype np.uint8 print f"Unique channel values: {len unique values }" if len unique values <= 30: print f"Values: {unique values}" else: print f"Value range: " f"{unique values 0 } ... {unique values -1 }" mean rgb = ai.mean axis= 0, 1 , dtype=np.float64, print "--- Mean RGB ---" print f"R = {mean rgb 0 :.9f}" print f"G = {mean rgb 1 :.9f}" print f"B = {mean rgb 2 :.9f}" channel min = ai.min axis= 0, 1 channel max = ai.max axis= 0, 1 print "--- Channel Ranges ---" print f"R: {channel min 0 :.0f} to {channel max 0 :.0f}" print f"G: {channel min 1 :.0f} to {channel max 1 :.0f}" print f"B: {channel min 2 :.0f} to {channel max 2 :.0f}" if np.any mean rgb < channel min or np.any mean rgb channel max : raise RuntimeError "Calculated RGB mean is outside the observed pixel range." f"Mean: {mean rgb}" f"Min: {channel min}" f"Max: {channel max}" residual = ai - mean rgb residual channel means = residual.mean axis= 0, 1 , dtype=np.float64, print "--- Residual ---" print f"Min: {residual.min :.9f}" print f"Max: {residual.max :.9f}" print f"Overall mean: " f"{residual.mean dtype=np.float64 :.12f}" print "--- Residual Mean By Channel ---" print f"R = {residual channel means 0 :.12f}" print f"G = {residual channel means 1 :.12f}" print f"B = {residual channel means 2 :.12f}" if not np.allclose residual channel means, 0.0, atol=1e-10, : raise RuntimeError "Residual was not centered correctly." f"Residual means: {residual channel means}" channel names = "Red", "Green", "Blue" print "--- Per-channel Residual Statistics ---" for index, name in enumerate channel names : channel = residual :, :, index print f"{name: 5} | " f"mean={channel.mean dtype=np.float64 : .12f} " f"std={channel.std dtype=np.float64 : .9f} " f"min={channel.min : .9f} " f"max={channel.max : .9f}" Overall RMS residual rms = np.sqrt np.mean residual 2, dtype=np.float64, print f"Overall residual RMS: {rms:.9f}" red fft = fft spectrum residual :, :, 0 green fft = fft spectrum residual :, :, 1 blue fft = fft spectrum residual :, :, 2 all fft values = np.concatenate red fft.ravel , green fft.ravel , blue fft.ravel , fft vmin = np.percentile all fft values, 1.0, fft vmax = np.percentile all fft values, 99.5, print "--- Shared FFT Display Scale ---" print f"vmin = {fft vmin:.9f}" print f"vmax = {fft vmax:.9f}" signed residual visualization residual vis = np.clip 128.0 + residual amp factor, 0, 255, .astype np.uint8 flat mean-color reference for visualization flat rgb uint8 = np.clip np.round mean rgb , 0, 255, .astype np.uint8 flat vis = np.empty like ai, dtype=np.uint8, flat vis : = flat rgb uint8 plot fig, axes = plt.subplots 2, 3, figsize= 18, 11 , constrained layout=True, Original image axes 0, 0 .set title "Original AI image" axes 0, 0 .imshow img axes 0, 0 .axis "off" Mean-matched flat reference axes 0, 1 .set title "Mean-matched flat reference" f"Display RGB {tuple flat rgb uint8.tolist }" axes 0, 1 .imshow flat vis axes 0, 1 .axis "off" axes 0, 2 .set title f"Signed residual ×{amp factor}" "128 = zero difference" axes 0, 2 .imshow residual vis axes 0, 2 .axis "off" fft image = axes 1, 0 .imshow red fft, cmap="magma", vmin=fft vmin, vmax=fft vmax, red axes 1, 0 .set title "FFT: red residual" axes 1, 0 .axis "off" axes 1, 1 .imshow green fft, cmap="magma", vmin=fft vmin, vmax=fft vmax, green axes 1, 1 .set title "FFT: green residual" axes 1, 1 .axis "off" axes 1, 2 .imshow blue fft, cmap="magma", vmin=fft vmin, vmax=fft vmax, axes 1, 2 .set title "FFT: blue residual" axes 1, 2 .axis "off" Shared colorbar fig.colorbar fft image, ax=axes 1, : , fraction=0.025, pad=0.02, label="log 1 + FFT magnitude ", All done plt.savefig output path, dpi=200, bbox inches="tight", plt.close fig print f"Saved: {output path}" if name == " main ": analyze zero mean residual image path="ai-generated-images/pure-white-blank-canvas1.png", output path="zero mean analysis.png", amp factor=50, Output: There is a lot of data to unpack, but we are mostly interested in mean RGB values and the residual standard deviation. Green seems to have a lot more residual energy, but we can't conclude that it is carrying the watermark. We can use Pillow to generate a-like image. We will get the exact value distribution per channel and generate a random image with the same histogram. random control = np.empty like source, dtype=np.uint8 channel names = "Red", "Green", "Blue" for i, channel name in enumerate channel names : channel = source :, :, i unique vals, counts = np.unique channel, return counts=True probs = counts / counts.sum random channel = rng.choice unique vals, size= h, w , p=probs, .astype np.uint8 random control :, :, i = random channel Image.fromarray random control .save output path Checking this newly generated image through the provenance API shows "SynthID not detected". So, just having the matching RGB distributions is not enough. The AI image contains spatial information that matters. We will do one final spectral comparison here. ... for i, channel name in enumerate channel names : ai mag, ai log, ai power = fft data ai residual :, :, i control mag, control log, control power = fft data control residual :, :, i ai fft.append ai log control fft.append control log ai entropy = spectral entropy ai power control entropy = spectral entropy control power ai peak = peak prominence ai power control peak = peak prominence control power corr = spectrum correlation ai log, control log, metrics.append { "channel": channel name, "ai entropy": ai entropy, "control entropy": control entropy, "ai peak": ai peak, "control peak": control peak, "correlation": corr, } radial ai.append radial power spectrum ai power radial control.append radial power spectrum control power ... This is an interesting result. The AI image and control are identical but very different spectrally. - Spectral entropy: AI image: ~0.889-0.900 Control image: ~0.970 - Peak prominence: AI image: ~716-1078 Control image: ~19-25 - Spectrum correlation: Zero Running it a few more times showed: | Metric | Run 1 | Run 2 | Run 3 | Controls | |---|---|---|---|---| | Red entropy | 0.9000 | 0.9011 | 0.8895 | ~0.9704 | | Green entropy | 0.8893 | 0.8818 | 0.8702 | ~0.9704 | | Blue entropy | 0.8970 | 0.8896 | 0.8817 | ~0.9704 | | Red peak | 715.7 | 554.5 | 485.2 | ~21-25 | | Green peak | 1077.8 | 884.5 | 633.6 | ~19-21 | | Blue peak | 777.5 | 718.3 | 811.3 | ~19-24 | In these samples, SynthID images show much lower spectral entropy than their matched random controls. I am now curious to see how much shaving off or limiting high-frequency spikes affects the image's appearance. We can convert each color channel into the frequency domain via a 2D Fast Fourier Transform and shift the zero-frequency component to the center. Then we iterate over some magnitude thresholds, cap the ones that exceed and reconstruct the image back. python import numpy as np from PIL import Image THRESHOLDS = 300, 500, 750, 1000, 1500, 2000, 2500, 3000 def evaluate threshold arr, threshold, dc radius=15 : h, w, c = arr.shape reconstructed = np.zeros like arr y, x = np.ogrid :h, :w cy, cx = h // 2, w // 2 dc mask = x - cx 2 + y - cy 2 <= dc radius 2 per channel affected = max imag residue = 0.0 for i in range c : channel = arr :, :, i fft raw = np.fft.fft2 channel fft shifted = np.fft.fftshift fft raw magnitude = np.abs fft shifted phase = np.angle fft shifted capped mask = magnitude threshold & ~dc mask affected count = np.sum capped mask affected fraction = affected count / magnitude.size 100.0 per channel affected.append affected fraction magnitude capped mask = threshold fft capped = magnitude np.exp 1j phase inverse = np.fft.ifft2 np.fft.ifftshift fft capped max imag residue = max max imag residue, np.max np.abs inverse.imag reconstructed :, :, i = inverse.real overall affected fraction = float np.mean per channel affected Float-domain error diff float = reconstructed - arr rmse float = np.sqrt np.mean diff float 2 mae float = np.mean np.abs diff float max abs float = np.max np.abs diff float 8-bit / saved-image error arr 8bit = np.clip np.round arr , 0, 255 .astype np.uint8 recon 8bit = np.clip np.round reconstructed , 0, 255 .astype np.uint8 diff 8bit = arr 8bit.astype np.float64 - recon 8bit.astype np.float64 rmse 8bit = np.sqrt np.mean diff 8bit 2 pixel changed mask = np.any arr 8bit = recon 8bit, axis=2 pixels changed pct = np.sum pixel changed mask / h w 100.0 subpixels changed pct = np.sum arr 8bit = recon 8bit / arr 8bit.size 100.0 return { "threshold": threshold, "affected overall pct": overall affected fraction, "affected red pct": per channel affected 0 , "affected green pct": per channel affected 1 , "affected blue pct": per channel affected 2 , "rmse float": rmse float, "mae float": mae float, "max abs float": max abs float, "rmse 8bit": rmse 8bit, "pixels changed pct": pixels changed pct, "subpixels changed pct": subpixels changed pct, "max imag residue": max imag residue, } def run sweep input path, dc radius=15 : print "--- Running Threshold Sweep ---" print f"Image: {input path}" print f"DC protection radius: {dc radius}" img = Image.open input path .convert "RGB" arr = np.array img, dtype=np.float64 print f"{'Threshold':<10} | " f"{'FFT Avg':<9} | " f"{'R%':<7} | " f"{'G%':<7} | " f"{'B%':<7} | " f"{'Float RMSE':<11} | " f"{'8-bit RMSE':<11} | " f"{'RGB Px':<9} | " f"{'Subpx':<9} | " f"{'Max Imag'}" print "-" 120 for t in THRESHOLDS: result = evaluate threshold arr, t, dc radius=dc radius print f"{result 'threshold' :<10} | " f"{result 'affected overall pct' : 7.2f}% | " f"{result 'affected red pct' : 6.2f}% | " f"{result 'affected green pct' : 6.2f}% | " f"{result 'affected blue pct' : 6.2f}% | " f"{result 'rmse float' : 10.4f} | " f"{result 'rmse 8bit' : 10.4f} | " f"{result 'pixels changed pct' : 7.2f}% | " f"{result 'subpixels changed pct' : 7.2f}% | " f"{result 'max imag residue' :.2e}" if name == " main ": run sweep "ai-generated-images/pure-white-blank-canvas1.png", dc radius=15, This result ruled out my naive but tempting shortcut. At 1500, I am changing about 5% of FFT bins, but only ~0.5% of RGB pixels survive quantization. But at 2000, I am changing about 2.6% of the FFT bins, but the 8-bit image is unchanged. This probably means threshold sweeps aren't a good way to target recurring peaks. The strong FFT peaks are interesting, but blindly capping them was not a good way to isolate whatever the detector was seeing. Comparing frequencies Finally, I took the FFT area that were very strong in the AI image and treated them as candidates. I can now compare them with similar non candidate frequencies. First is Frequency radius. I start from the center FFT , and compare the radius of AI candidate and control frequencies. The result was: - Median radius difference: 0.73 bins - 95th percentile: 2.67 bins So they were pretty close in frequency location. Then I matched FFT magnitude. For every candidate, I looked for a control frequency within 0.95x and 1.05x FTT magnitude. The result was that the median control magnitude was about 95.6% of the candidate magnitude. After matching radius and magnitude, I got RMSE of candidate: 0.105326 and RMSE of control: 0.100709, i.e. just 4.6% difference. In other words, once I matched both sides properly, most of that difference disappeared. Final words I started this experiment thinking the watermark was simple stuff. But the data decided to consume my evening. The takeaway is that the AI-generated images contained small pixel variations, and they do not behave like random noise. When I compared them with random controls, the AI-generated ones showed lower spectral entropy and near-zero spectrum correlations with our controls. And my attempt to suppress the high-energy FFT coefficients didn't turn out the way I hoped. So I didn't find the SynthID frequency yet . But there was clearly some spectral structure hiding inside a plain white image.