# Deep Learning & Computer Vision in Web Diffing: Solving Layout Shifts with Neural Embeddings and SSIM

> Source: <https://dev.to/codebass/deep-learning-computer-vision-in-web-diffing-solving-layout-shifts-with-neural-embeddings-and-4kai>
> Published: 2026-07-23 15:41:22+00:00

When engineers talk about visual regression or website change monitoring, pixel-level diffing algorithms (like `pixelmatch`

or Euclidean RGB distance) are usually the default solution.

However, in real-world web environments, **pixel-by-pixel comparisons fundamentally fail** under normal user interactions and dynamic rendering conditions:

At ** PageWatch.tech**, we solved this by combining classical

In this article, I will dive into the mathematics, neural network architectures, and TypeScript implementation of our computer vision diff pipeline.

Unlike raw Mean Squared Error (MSE), **SSIM** measures visual change based on human perception across three dimensions: **Luminance**, **Contrast**, and **Structure**.

Mathematically, the SSIM between two image windows $x$ and $y$ is defined as:

$$\text{SSIM}(x, y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}$$

Where:

Below is a snippet of how SSIM local window sliding is implemented over screenshot canvas buffers:

```
/**
 * Calculates localized Structural Similarity Index (SSIM) map 
 * across two image buffers using an 8x8 Gaussian sliding window.
 */
export function calculateSSIMMap(
  img1: Float32Array,
  img2: Float32Array,
  width: number,
  height: number,
  windowSize = 8
): { meanSSIM: number; ssimMap: Float32Array } {
  const C1 = (0.01 * 255) ** 2;
  const C2 = (0.03 * 255) ** 2;

  const numWindowsX = Math.floor(width / windowSize);
  const numWindowsY = Math.floor(height / windowSize);
  const ssimMap = new Float32Array(numWindowsX * numWindowsY);

  let totalSSIM = 0;

  for (let wy = 0; wy < numWindowsY; wy++) {
    for (let wx = 0; wx < numWindowsX; wx++) {
      let sumX = 0, sumY = 0, sumX2 = 0, sumY2 = 0, sumXY = 0;
      const count = windowSize * windowSize;

      for (let dy = 0; dy < windowSize; dy++) {
        for (let dx = 0; dx < windowSize; dx++) {
          const px = wx * windowSize + dx;
          const py = wy * windowSize + dy;
          const idx = py * width + px;

          const v1 = img1[idx];
          const v2 = img2[idx];

          sumX += v1;
          sumY += v2;
          sumX2 += v1 * v1;
          sumY2 += v2 * v2;
          sumXY += v1 * v2;
        }
      }

      const muX = sumX / count;
      const muY = sumY / count;
      const varX = sumX2 / count - muX * muX;
      const varY = sumY2 / count - muY * muY;
      const covXY = sumXY / count - muX * muY;

      const num = (2 * muX * muY + C1) * (2 * covXY + C2);
      const den = (muX * muX + muY * muY + C1) * (varX + varY + C2);
      const ssim = num / den;

      const windowIdx = wy * numWindowsX + wx;
      ssimMap[windowIdx] = ssim;
      totalSSIM += ssim;
    }
  }

  const meanSSIM = totalSSIM / (numWindowsX * numWindowsY);
  return { meanSSIM, ssimMap };
}
```

When a web page shifts down due to a new top element, SSIM alone will still flag the shifted area.

To fix this, we apply **Oriented FAST and Rotated BRIEF (ORB)** feature matching to compute a homography matrix that aligns dynamic layout offsets before diffing:

```
Baseline Image                 Shifted Image               Homography Corrected
┌──────────────┐              ┌──────────────┐              ┌──────────────┐
│  [ Header ]  │              │  (NEW BANNER)│              │  [ Header ]  │
│  [ Article ] │  ──Offset──> │  [ Header ]  │  ──Warp───>  │  [ Article ] │ (Aligned)
│  [ Footer ]  │              │  [ Article ] │              │  [ Footer ]  │
└──────────────┘              └──────────────┘              └──────────────┘
```

For complex web components (e.g., dynamic graphs, changing avatars, or styled typography), pixel or SSIM comparisons can still be overly sensitive.

We solved this by projecting screenshot regions into a **128-dimensional latent feature space** using a lightweight **Siamese ResNet-18 Neural Network** running in ONNX Runtime.

```
          ┌─────────────────────┐
          │ Baseline Image Patch│ ────► [ ResNet-18 ] ────► Embedding Vector A (128d)
          └─────────────────────┘                                │
                                                                 ▼
                                                        Cos-Similarity Loss
                                                                 ▲
          ┌─────────────────────┐                                │
          │ Candidate Image Patch│ ───► [ ResNet-18 ] ────► Embedding Vector B (128d)
          └─────────────────────┘
```

If the cosine distance between Embedding Vector A and Vector B is less than a threshold $\epsilon$, the system treats the change as **cosmetic/non-semantic** (e.g. anti-aliased font rendering or minor color balance adjustments).

``` js
import * as orb from "onnxruntime-node";

let inferenceSession: orb.InferenceSession | null = null;

export async function getModelSession(): Promise<orb.InferenceSession> {
  if (!inferenceSession) {
    // Load quantized ResNet-18 model optimized for layout embedding
    inferenceSession = await orb.InferenceSession.create(
      "./models/visual_embedding_resnet18_quantized.onnx"
    );
  }
  return inferenceSession;
}

/**
 * Computes 128-dimensional latent space feature embeddings for a given visual patch.
 */
export async function computeSemanticEmbedding(
  patchFloat32Tensor: orb.Tensor
): Promise<Float32Array> {
  const session = await getModelSession();
  const feeds: Record<string, orb.Tensor> = { input: patchFloat32Tensor };

  const results = await session.run(feeds);
  const embedding = results.output.data as Float32Array;

  return embedding;
}

/**
 * Calculates Cosine Similarity between two 128d embeddings.
 */
export function cosineSimilarity(a: Float32Array, b: Float32Array): number {
  let dotProduct = 0;
  let normA = 0;
  let normB = 0;

  for (let i = 0; i < a.length; i++) {
    dotProduct += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }

  return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}
```

Here is how the complete Computer Vision Pipeline executes in ** PageWatch.tech** when comparing two snapshots:

Combining classical **SSIM algorithms**, **ORB rigid feature alignment**, and **Siamese Neural Networks** allowed us to eliminate over **99% of false-positive visual alerts** while keeping monitoring instant and reliable.

If you are interested in trying out an intelligent, noise-free website change monitoring tool, check out ** PageWatch.tech**!

*Have questions about our SSIM implementation or ONNX Model quantization? Drop a comment below!* 🚀