Building a Browser-Based ID Photo Maker: Face Detection, Head Estimation & Background Removal

The browser has become a surprisingly capable platform for computer vision. In this post, I'll walk through building a fully client-side ID photo maker — covering multi-tier face detection, anthropometric head measurement, segmentation-refined cropping, and background removal with color decontamination.

ID photos have strict requirements: head height ratios, eye-line positions, margin rules. Tools like HivisionIDPhotos, dpar39/ppp, and commercial services like dreamega.ai and photoaid.com all need to solve the same core problem: given an uploaded portrait, detect the face accurately enough to auto-crop to spec.

Server-side tools can afford heavyweight models (HivisionIDPhotos uses ONNX runtime with dedicated face parsing networks). But if you want to run everything client-side — no upload, no server cost, instant preview — the browser environment is unpredictable. Not every user has Chrome's experimental FaceDetector

API, and WebGL support varies.

Our solution: a three-tier detection chain that tries the best option first and gracefully falls back.

// Detection chain: MediaPipe (478 landmarks) → Browser FaceDetector → BlazeFace
export async function detectFace(image: HTMLImageElement): Promise<FaceDetectResult> {
  if (typeof window !== "undefined" && !window.isSecureContext) {
    return { supported: false, face: null, reason: "insecure-context" };
  }

  // 1. MediaPipe Face Landmarker – best quality, 478 landmarks
  try {
    const mpResult = await detectWithMediaPipe(image);
    if (mpResult.face || mpResult.reason === "no-face-found") {
      return mpResult;
    }
  } catch { /* Fall through */ }

  // 2. Browser FaceDetector API (Chrome)
  const detector = await getDetector();
  if (detector) {
    try {
      const nativeResult = await detectWithNative(detector, image);
      if (nativeResult.face || nativeResult.reason === "no-face-found") {
        return nativeResult;
      }
    } catch { /* Fall through */ }
  }

  // 3. BlazeFace TensorFlow model
  try {
    const fallbackResult = await detectWithBlazeFace(image);
    if (fallbackResult.supported || fallbackResult.reason === "no-face-found") {
      return fallbackResult;
    }
  } catch { /* Fall through */ }

  return { supported: false, face: null, reason: "api-missing" };
}

Each tier returns a FaceBox

with bounding coordinates and optional landmarks (eyes, forehead, chin). The key insight: each detector gives you different landmark richness, so the geometry estimation adapts.

Accurate head measurement is crucial — a few pixels off and the photo fails compliance. The approach here borrows from dpar39/ppp's CrownChinEstimator

and extends it with a MediaPipe-first path:

export function estimateHeadHeight(face: FaceBox): number {
  // Best: MediaPipe forehead-to-chin + hair offset (1.35x multiplier)
  // HivisionIDPhotos avoids this problem by using alpha-channel contour
  // detection after background removal — a neat trick if you have a server.
  if (face.foreheadTop && face.chinBottom) {
    const chinY = Math.max(face.chinBottom.y, face.y + face.height);
    const foreheadToChin = chinY - face.foreheadTop.y;
    return foreheadToChin * 1.35;
  }

  // Good: ppp formula (chinCrownCoeff = 1.7699)
  if (face.leftEye && face.rightEye) {
    const ipd = Math.hypot(
      face.rightEye.x - face.leftEye.x,
      face.rightEye.y - face.leftEye.y
    );
    const eyeMidY = (face.leftEye.y + face.rightEye.y) / 2;
    const chinY = face.y + face.height;
    const frownToChin = chinY - eyeMidY;
    return 1.77 * (ipd + frownToChin);
  }

  // Fallback: anthropometric multiplier (Farkas LG reference)
  return face.height * 1.35;
}

The 1.35x multiplier accounts for hair volume above the forehead — MediaPipe landmarks don't detect hair, so this is intentionally an overestimate to prevent head cropping. The IPD formula comes from anthropometric research (Farkas LG, "Anthropometry of the Head and Face").

Landmarks estimate where the head top should be, but segmentation gives you pixel-accurate results. This is where combining both outperforms services like passport-photo.online that rely purely on face detection — especially for people with voluminous hair:

export function refineWithSegmentation(
  geom: HeadGeometry,
  segBounds: { topY: number; bottomY: number } | null
): HeadGeometry {
  if (!segBounds) return geom;

  // Only use segmentation topY — bottomY detects body silhouette, not chin
  const refinedTopY = Math.min(segBounds.topY, geom.topY);

  return {
    ...geom,
    topY: refinedTopY,
    headHeight: geom.bottomY - refinedTopY,
  };
}

An important gotcha: segmentation bottomY

is the body bottom (shoulders, torso), not the chin. Using it would over-extend the head measurement. We only take topY

from segmentation (hair boundary) and keep the chin position from face landmarks.

With accurate head geometry, auto-cropping becomes a three-step process:

export function computeAutoFitCropState({ imageWidth, imageHeight, face, spec, canvasWidth, canvasHeight, segBounds }) {
  const rules = getNormalizedHeadRules(spec);
  const head = refineWithSegmentation(estimateHeadGeometry(face), segBounds);

  // 1. SCALE — head matches target height ratio
  const targetH = guide.headBottom - guide.headTop;
  const scale = clampedTarget / Math.max(1, head.headHeight);

  // 2. PLACE — two anchor strategies
  if (rules.hasEyeLineRule && hasEyes) {
    // Strategy A: eyes at target Y (used by ICAO/US passport specs)
    crop = cropFromAnchor(eyeImg, { x: cw/2, y: eyeTargetY }, ...);
  } else {
    // Strategy B: head top at target margin (common in Asian passport specs)
    crop = cropFromAnchor({ x: head.centerX, y: head.topY }, ...);
  }

  // 3. GUARD — prevent head from being cropped
  crop = guard(crop, head, face, ...);
  return crop;
}

The guard pass handles edge cases: crown too close to the top, chin below frame, or both overflowing (in which case it scales down to fit).

When replacing backgrounds, edge pixels bleed the original background color. We use @imgly/background-removal

(ISNet model) — the same engine behind remove.bg's open-source alternative — then apply color decontamination:

// Decontaminate semi-transparent edge pixels
for (let i = 0; i < rPx.length; i += 4) {
  let a = rPx[i + 3] / 255;

  // Erode very low alpha pixels — eliminates faint halo
  if (a < 0.15) { rPx[i + 3] = 0; continue; }

  // Squeeze alpha: remap [0.15, 1] → [0, 1]
  a = Math.min(1, (a - 0.15) / 0.85);
  rPx[i + 3] = Math.round(a * 255);

  // Color decontamination for edge pixels
  if (a > 0.01 && a < 0.95) {
    const inv = 1 - a;
    rPx[i]     = clamp255((rPx[i]     - inv * bgR) / a);
    rPx[i + 1] = clamp255((rPx[i + 1] - inv * bgG) / a);
    rPx[i + 2] = clamp255((rPx[i + 2] - inv * bgB) / a);
  }
}

The math reverses the alpha compositing formula: if the composited color is C = α·F + (1-α)·B

, then the true foreground is F = (C - (1-α)·B) / α

. The alpha squeeze ([0.15, 1] → [0, 1]

) acts as an erosion filter that kills the soft halo around hair.

Progressive enhancement works for ML: The three-tier face detection chain means every user gets the best experience their browser supports, with no manual feature-flag management.

Separate detection from estimation: Face detection gives you coordinates; head geometry estimation interprets them. Keeping these separate lets you swap in better models without touching the layout logic.

Segmentation complements, doesn't replace landmarks: Use segmentation for what it's good at (hair boundaries, background separation) and landmarks for what they're good at (facial feature positions). Mixing their strengths produces better results than either alone.

Color decontamination matters: Without the alpha-unpremultiply step, every background replacement shows a visible color fringe. The three lines of math are worth it.

Browser ML is production-ready: Between MediaPipe, TensorFlow.js, and @imgly/background-removal, you can build a full ID photo pipeline that runs entirely on the client — no server round-trips, no privacy concerns.

Hope this helps if you're building similar browser-based CV tools. Happy to answer questions in the comments.