Synthetic Checks & SLOs for the Paths Nobody Clicks

A client's status page showed 99.98% uptime for the quarter. In the same quarter, their password reset flow was broken for 11 days. The homepage returned 200, the API health endpoint returned 200, and the load balancer was happy. But a queue worker that sent reset emails had been pinned to an old mail driver after a config change, and the reset page itself rendered fine; it just never sent anything. Nobody alerted because nobody clicks "forgot password" often enough for error rates to move. We found out from a support ticket written in capital letters.

I wrote this with help from AI tools and checked every script, config and query myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we run and maintain production servers for SaaS clients. This is Part 6, the finale of the series Silent Failures in Production SaaS. This last part is the safety net under the earlier five: actively exercising the paths real users rarely touch and measuring them against an objective.

📚 Series Navigation — Silent Failures in Production SaaS

Part 1: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS Part 2: Webhooks That Stopped Arriving: Gap Detection & Reconciliation Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines Part 4: Config & Feature Flags That Quietly Diverged Across Environments Part 5: Backups That Never Restored: Automated Restore Drills Part 6 (This Post, Finale): Synthetic Checks & SLOs for the Paths Nobody Clicks Related earlier posts: Production Observability on a Budget · Observability for Async Work: Correlation IDs & Job Traces

Direct answer: Run synthetic checks: scheduled scripts that perform the real user journey end to end (request a password reset, receive the email, follow the link, export an invoice) using a dedicated test tenant. Record each run as pass or fail with timing, define an SLO per journey, and alert on error-budget burn rather than on single failures or plain uptime.

Uptime checks answer one question: does a URL respond? Low-traffic paths fail in ways that never touch that question. Here is how the three common layers compare on the failures from this series:

The bottom row is the only one uptime monitoring is designed for, and it is also the failure you would notice anyway.

I list journeys by asking two questions: if this broke for a week, would it cost money or trust? And does it get fewer than a few hundred uses a day? Anything that answers yes to both gets a synthetic check. For a typical B2B SaaS that is:

Use a dedicated synthetic tenant with clearly labelled data, excluded from analytics and billing. Never point checks at real customer accounts.

For flows that involve a UI, I use Playwright. This is a trimmed version of our password reset check. The test mailbox is a catch-all on a subdomain we control, read through the mail provider's inbound API:

// synthetics/password-reset.spec.ts
import { test, expect } from '@playwright/test';
import { waitForEmail } from './mailbox';

test('password reset end to end', async ({ page }) => {
  const email = process.env.SYNTH_USER_EMAIL!;
  const started = Date.now();

  await page.goto(`${process.env.APP_URL}/forgot-password`);
  await page.getByLabel('Email').fill(email);
  await page.getByRole('button', { name: /send reset link/i }).click();
  await expect(page.getByText(/we have emailed/i)).toBeVisible();

  const msg = await waitForEmail({ to: email, subject: /reset/i, sinceMs: started, timeoutMs: 120_000 });
  const link = msg.html.match(/href="([^"]+reset-password[^"]+)"/)?.[1];
  expect(link, 'reset link present in email').toBeTruthy();

  await page.goto(link!);
  await page.getByLabel('Password', { exact: true }).fill(process.env.SYNTH_USER_PASSWORD!);
  await page.getByLabel('Confirm Password').fill(process.env.SYNTH_USER_PASSWORD!);
  await page.getByRole('button', { name: /reset password/i }).click();
  await expect(page).toHaveURL(/dashboard|login/);
});

The email step is what would have caught the 11-day outage. A check that stopped at "we have emailed you" would have passed every single run.

For API-only journeys, a plain script that posts a signed test event and polls until the record exists is enough.

Each run writes one row: journey, pass or fail, duration, timestamp. In Laravel I keep a tiny table and a scheduled command, but pushing to Prometheus via a Pushgateway works just as well:

// app/Console/Commands/RunSynthetics.php (handle method, trimmed)
foreach (config('synthetics.journeys') as $journey => $cmd) {
    $start = microtime(true);
    $result = Process::timeout(180)->run($cmd);
    SyntheticRun::create([
        'journey'     => $journey,
        'ok'          => $result->successful(),
        'duration_ms' => (int) ((microtime(true) - $start) * 1000),
        'output'      => Str::limit($result->errorOutput() ?: $result->output(), 2000),
    ]);
}
php
// routes/console.php
Schedule::command('synthetics:run')
    ->everyFiveMinutes()
    ->withoutOverlapping()
    ->onOneServer()
    ->pingOnSuccess(config('services.heartbeats.synthetics'));

Note the heartbeat. As Part 1 argued, a monitor that silently stops running is the worst silent failure of all, so the synthetic runner itself reports in.

A single failed run is often noise: a slow mail provider, a deploy in progress. Paging on every failure trains people to ignore pages. Instead I set an SLO per journey and alert when the error budget burns too fast.

Example: password reset should succeed in 99% of synthetic runs over 30 days. At one run every five minutes, that is 8,640 runs and an error budget of about 86 failures. The query for the current failure ratio:

SELECT journey,
       1 - AVG(ok::int)       AS failure_ratio_1h,
       PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms
FROM synthetic_runs
WHERE created_at > now() - interval '1 hour'
GROUP BY journey;

Alerting rules I use as defaults, borrowed from the multi-window burn-rate approach in Google's SRE workbook:

With a 99% target the budget rate is 1%, so "fast burn" means more than about 14% of runs failing in the last hour. Two consecutive failed runs of a five-minute check will usually trip it, which is the behaviour you want for a fully broken path.

Synthetic checks rot like any other code. Run them against production from outside your network so DNS, TLS and CDN problems show up, keep the synthetic user's credentials in your secret manager, fix or delete flaky checks instead of muting them, and add a check for every incident where a customer told you first.

We set up and run these as part of our server setup and maintenance work. If you want help choosing journeys and SLO targets for your own product, get in touch with us.

They share tooling, but the purpose differs. E2E tests run before deploy against a test environment. Synthetic checks run continuously against production to catch what changes after deploy: config, third parties, certificates and infrastructure.

Only if you let it. Tag the synthetic tenant, exclude it in analytics and billing, and add a request header your logs can filter on.

Three to five. Password reset, signup and your main export or integration path cover most of the silent failures I see.

Start lower than you think, often 99% for synthetic journeys, then tighten once a month of data shows what is realistic. A target you constantly miss gets ignored.

Yes. Checkly, Datadog Synthetics, Grafana Synthetic Monitoring and similar tools handle scheduling and storage. The important work, choosing journeys and writing the full flow including email, is the same.

That closes Silent Failures in Production SaaS. Cron jobs, webhooks, email, config drift, backups and now unclicked paths: the common thread is that a system with no errors is not the same as a system that works. Give each critical behaviour a positive signal, alert when that signal goes quiet, and measure it against a target someone agreed to.