Deep Learning & Computer Vision in Web Diffing: Solving Layout Shifts with Neural Embeddings and SSIM

PageWatch.tech has developed a computer vision diff pipeline for web change monitoring that combines SSIM, ORB feature matching, and a Siamese ResNet-18 neural network to handle layout shifts and dynamic content. The approach uses homography correction to align pages before diffing and projects regions into a 128-dimensional latent space for robust comparison.

When engineers talk about visual regression or website change monitoring, pixel-level diffing algorithms like pixelmatch or Euclidean RGB distance are usually the default solution. However, in real-world web environments, pixel-by-pixel comparisons fundamentally fail under normal user interactions and dynamic rendering conditions: At PageWatch.tech , we solved this by combining classical In this article, I will dive into the mathematics, neural network architectures, and TypeScript implementation of our computer vision diff pipeline. Unlike raw Mean Squared Error MSE , SSIM measures visual change based on human perception across three dimensions: Luminance , Contrast , and Structure . Mathematically, the SSIM between two image windows $x$ and $y$ is defined as: $$\text{SSIM} x, y = \frac{ 2\mu x\mu y + C 1 2\sigma {xy} + C 2 }{ \mu x^2 + \mu y^2 + C 1 \sigma x^2 + \sigma y^2 + C 2 }$$ Where: Below is a snippet of how SSIM local window sliding is implemented over screenshot canvas buffers: / Calculates localized Structural Similarity Index SSIM map across two image buffers using an 8x8 Gaussian sliding window. / export function calculateSSIMMap img1: Float32Array, img2: Float32Array, width: number, height: number, windowSize = 8 : { meanSSIM: number; ssimMap: Float32Array } { const C1 = 0.01 255 2; const C2 = 0.03 255 2; const numWindowsX = Math.floor width / windowSize ; const numWindowsY = Math.floor height / windowSize ; const ssimMap = new Float32Array numWindowsX numWindowsY ; let totalSSIM = 0; for let wy = 0; wy < numWindowsY; wy++ { for let wx = 0; wx < numWindowsX; wx++ { let sumX = 0, sumY = 0, sumX2 = 0, sumY2 = 0, sumXY = 0; const count = windowSize windowSize; for let dy = 0; dy < windowSize; dy++ { for let dx = 0; dx < windowSize; dx++ { const px = wx windowSize + dx; const py = wy windowSize + dy; const idx = py width + px; const v1 = img1 idx ; const v2 = img2 idx ; sumX += v1; sumY += v2; sumX2 += v1 v1; sumY2 += v2 v2; sumXY += v1 v2; } } const muX = sumX / count; const muY = sumY / count; const varX = sumX2 / count - muX muX; const varY = sumY2 / count - muY muY; const covXY = sumXY / count - muX muY; const num = 2 muX muY + C1 2 covXY + C2 ; const den = muX muX + muY muY + C1 varX + varY + C2 ; const ssim = num / den; const windowIdx = wy numWindowsX + wx; ssimMap windowIdx = ssim; totalSSIM += ssim; } } const meanSSIM = totalSSIM / numWindowsX numWindowsY ; return { meanSSIM, ssimMap }; } When a web page shifts down due to a new top element, SSIM alone will still flag the shifted area. To fix this, we apply Oriented FAST and Rotated BRIEF ORB feature matching to compute a homography matrix that aligns dynamic layout offsets before diffing: Baseline Image Shifted Image Homography Corrected ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Header │ │ NEW BANNER │ │ Header │ │ Article │ ──Offset── │ Header │ ──Warp─── │ Article │ Aligned │ Footer │ │ Article │ │ Footer │ └──────────────┘ └──────────────┘ └──────────────┘ For complex web components e.g., dynamic graphs, changing avatars, or styled typography , pixel or SSIM comparisons can still be overly sensitive. We solved this by projecting screenshot regions into a 128-dimensional latent feature space using a lightweight Siamese ResNet-18 Neural Network running in ONNX Runtime. ┌─────────────────────┐ │ Baseline Image Patch│ ────► ResNet-18 ────► Embedding Vector A 128d └─────────────────────┘ │ ▼ Cos-Similarity Loss ▲ ┌─────────────────────┐ │ │ Candidate Image Patch│ ───► ResNet-18 ────► Embedding Vector B 128d └─────────────────────┘ If the cosine distance between Embedding Vector A and Vector B is less than a threshold $\epsilon$, the system treats the change as cosmetic/non-semantic e.g. anti-aliased font rendering or minor color balance adjustments . js import as orb from "onnxruntime-node"; let inferenceSession: orb.InferenceSession | null = null; export async function getModelSession : Promise<orb.InferenceSession { if inferenceSession { // Load quantized ResNet-18 model optimized for layout embedding inferenceSession = await orb.InferenceSession.create "./models/visual embedding resnet18 quantized.onnx" ; } return inferenceSession; } / Computes 128-dimensional latent space feature embeddings for a given visual patch. / export async function computeSemanticEmbedding patchFloat32Tensor: orb.Tensor : Promise<Float32Array { const session = await getModelSession ; const feeds: Record<string, orb.Tensor = { input: patchFloat32Tensor }; const results = await session.run feeds ; const embedding = results.output.data as Float32Array; return embedding; } / Calculates Cosine Similarity between two 128d embeddings. / export function cosineSimilarity a: Float32Array, b: Float32Array : number { let dotProduct = 0; let normA = 0; let normB = 0; for let i = 0; i < a.length; i++ { dotProduct += a i b i ; normA += a i a i ; normB += b i b i ; } return dotProduct / Math.sqrt normA Math.sqrt normB ; } Here is how the complete Computer Vision Pipeline executes in PageWatch.tech when comparing two snapshots: Combining classical SSIM algorithms , ORB rigid feature alignment , and Siamese Neural Networks allowed us to eliminate over 99% of false-positive visual alerts while keeping monitoring instant and reliable. If you are interested in trying out an intelligent, noise-free website change monitoring tool, check out PageWatch.tech Have questions about our SSIM implementation or ONNX Model quantization? Drop a comment below 🚀