Synthetic Checks & SLOs for the Paths Nobody Clicks A principal software engineer at Loop Origin described how a client's password reset flow stayed broken for 11 days despite 99.98% reported uptime, because uptime pings and error-rate alerts never exercised the low-traffic path. The engineer's fix, the finale of a six-part series on silent failures in production SaaS, is to run scheduled synthetic journey checks against a dedicated test tenant, record each run as pass or fail with timing, set an SLO per journey, and alert on error-budget burn rather than single failures. A client's status page showed 99.98% uptime for the quarter. In the same quarter, their password reset flow was broken for 11 days. The homepage returned 200, the API health endpoint returned 200, and the load balancer was happy. But a queue worker that sent reset emails had been pinned to an old mail driver after a config change, and the reset page itself rendered fine; it just never sent anything. Nobody alerted because nobody clicks "forgot password" often enough for error rates to move. We found out from a support ticket written in capital letters. I wrote this with help from AI tools and checked every script, config and query myself. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and at Loop Origin we run and maintain production servers for SaaS clients. This is Part 6, the finale of the series Silent Failures in Production SaaS. This last part is the safety net under the earlier five: actively exercising the paths real users rarely touch and measuring them against an objective. ๐Ÿ“š Series Navigation โ€” Silent Failures in Production SaaS Part 1: Cron Jobs That Never Ran: Heartbeats & Dead-Man Switches for SaaS https://dev.to/muhammadalimughal/cron-jobs-that-never-ran-heartbeats-dead-man-switches-for-saas-1gk6 Part 2: Webhooks That Stopped Arriving: Gap Detection & Reconciliation https://dev.to/muhammadalimughal/webhooks-that-stopped-arriving-gap-detection-reconciliation-jo7 Part 3: Email That Never Landed: SPF/DKIM/DMARC Drift & Bounce Pipelines https://dev.to/muhammadalimughal/email-that-never-landed-spfdkimdmarc-drift-bounce-pipelines-999 Part 4: Config & Feature Flags That Quietly Diverged Across Environments https://dev.to/muhammadalimughal/config-feature-flags-that-quietly-diverged-across-environments-4m38 Part 5: Backups That Never Restored: Automated Restore Drills https://dev.to/muhammadalimughal/backups-that-never-restored-automated-restore-drills-1amb Part 6 This Post, Finale : Synthetic Checks & SLOs for the Paths Nobody Clicks Related earlier posts: Production Observability on a Budget https://dev.to/muhammadalimughal/production-observability-on-a-budget-opentelemetry-traces-structured-logs-tail-sampling-360a ยท Observability for Async Work: Correlation IDs & Job Traces https://dev.to/muhammadalimughal/observability-for-async-work-correlation-ids-job-traces-16f3 Direct answer: Run synthetic checks: scheduled scripts that perform the real user journey end to end request a password reset, receive the email, follow the link, export an invoice using a dedicated test tenant. Record each run as pass or fail with timing, define an SLO per journey, and alert on error-budget burn rather than on single failures or plain uptime. Uptime checks answer one question: does a URL respond? Low-traffic paths fail in ways that never touch that question. Here is how the three common layers compare on the failures from this series: | Failure | Uptime ping | Error-rate alert on real traffic | Synthetic journey check | |---|---|---|---| | Password reset email never sent | Misses page renders | Misses no exception, few users | Catches no email within 2 min | | SSO login broken for one IdP | Misses | Misses until a big customer logs in | Catches test IdP login fails | | Invoice PDF export times out | Misses | Maybe, if volume is high | Catches export 30 s | | Webhook endpoint returns 200 but drops data | Misses | Misses | Catches round-trip record not found | | Signup blocked by a new CAPTCHA config | Misses | Misses users just leave | Catches signup step fails | | Homepage down | Catches | Catches | Catches | The bottom row is the only one uptime monitoring is designed for, and it is also the failure you would notice anyway. I list journeys by asking two questions: if this broke for a week, would it cost money or trust? And does it get fewer than a few hundred uses a day? Anything that answers yes to both gets a synthetic check. For a typical B2B SaaS that is: Use a dedicated synthetic tenant with clearly labelled data, excluded from analytics and billing. Never point checks at real customer accounts. For flows that involve a UI, I use Playwright. This is a trimmed version of our password reset check. The test mailbox is a catch-all on a subdomain we control, read through the mail provider's inbound API: js // synthetics/password-reset.spec.ts import { test, expect } from '@playwright/test'; import { waitForEmail } from './mailbox'; test 'password reset end to end', async { page } = { const email = process.env.SYNTH USER EMAIL ; const started = Date.now ; await page.goto ${process.env.APP URL}/forgot-password ; await page.getByLabel 'Email' .fill email ; await page.getByRole 'button', { name: /send reset link/i } .click ; await expect page.getByText /we have emailed/i .toBeVisible ; const msg = await waitForEmail { to: email, subject: /reset/i, sinceMs: started, timeoutMs: 120 000 } ; const link = msg.html.match /href=" ^" +reset-password ^" + "/ ?. 1 ; expect link, 'reset link present in email' .toBeTruthy ; await page.goto link ; await page.getByLabel 'Password', { exact: true } .fill process.env.SYNTH USER PASSWORD ; await page.getByLabel 'Confirm Password' .fill process.env.SYNTH USER PASSWORD ; await page.getByRole 'button', { name: /reset password/i } .click ; await expect page .toHaveURL /dashboard|login/ ; } ; The email step is what would have caught the 11-day outage. A check that stopped at "we have emailed you" would have passed every single run. For API-only journeys, a plain script that posts a signed test event and polls until the record exists is enough. Each run writes one row: journey, pass or fail, duration, timestamp. In Laravel I keep a tiny table and a scheduled command, but pushing to Prometheus via a Pushgateway works just as well: // app/Console/Commands/RunSynthetics.php handle method, trimmed foreach config 'synthetics.journeys' as $journey = $cmd { $start = microtime true ; $result = Process::timeout 180 - run $cmd ; SyntheticRun::create 'journey' = $journey, 'ok' = $result- successful , 'duration ms' = int microtime true - $start 1000 , 'output' = Str::limit $result- errorOutput ?: $result- output , 2000 , ; } php // routes/console.php Schedule::command 'synthetics:run' - everyFiveMinutes - withoutOverlapping - onOneServer - pingOnSuccess config 'services.heartbeats.synthetics' ; Note the heartbeat. As Part 1 argued, a monitor that silently stops running is the worst silent failure of all, so the synthetic runner itself reports in. A single failed run is often noise: a slow mail provider, a deploy in progress. Paging on every failure trains people to ignore pages. Instead I set an SLO per journey and alert when the error budget burns too fast. Example: password reset should succeed in 99% of synthetic runs over 30 days. At one run every five minutes, that is 8,640 runs and an error budget of about 86 failures. The query for the current failure ratio: SELECT journey, 1 - AVG ok::int AS failure ratio 1h, PERCENTILE CONT 0.95 WITHIN GROUP ORDER BY duration ms AS p95 ms FROM synthetic runs WHERE created at now - interval '1 hour' GROUP BY journey; Alerting rules I use as defaults, borrowed from the multi-window burn-rate approach in Google's SRE workbook: | Alert | Condition | Meaning | Action | |---|---|---|---| | Fast burn | Failure ratio over 1 h 14.4ร— budget rate, confirmed over 5 min | Budget gone in about 2 days | Page on-call | | Slow burn | Failure ratio over 6 h 6ร— budget rate, confirmed over 30 min | Budget gone in about 5 days | Ticket, fix this week | | Latency | p95 duration 2ร— baseline for 1 h | Degrading, not broken | Ticket | | Silence | No synthetic heartbeat for 15 min | Monitoring is down | Page on-call | With a 99% target the budget rate is 1%, so "fast burn" means more than about 14% of runs failing in the last hour. Two consecutive failed runs of a five-minute check will usually trip it, which is the behaviour you want for a fully broken path. Synthetic checks rot like any other code. Run them against production from outside your network so DNS, TLS and CDN problems show up, keep the synthetic user's credentials in your secret manager, fix or delete flaky checks instead of muting them, and add a check for every incident where a customer told you first. We set up and run these as part of our server setup and maintenance https://looporigin.com/services/server-setup-maintenance?utm source=devto&utm medium=series&utm campaign=synthetic-checks-slos-unclicked-paths work. If you want help choosing journeys and SLO targets for your own product, get in touch with us https://looporigin.com/contact?utm source=devto&utm medium=series&utm campaign=synthetic-checks-slos-unclicked-paths . They share tooling, but the purpose differs. E2E tests run before deploy against a test environment. Synthetic checks run continuously against production to catch what changes after deploy: config, third parties, certificates and infrastructure. Only if you let it. Tag the synthetic tenant, exclude it in analytics and billing, and add a request header your logs can filter on. Three to five. Password reset, signup and your main export or integration path cover most of the silent failures I see. Start lower than you think, often 99% for synthetic journeys, then tighten once a month of data shows what is realistic. A target you constantly miss gets ignored. Yes. Checkly, Datadog Synthetics, Grafana Synthetic Monitoring and similar tools handle scheduling and storage. The important work, choosing journeys and writing the full flow including email, is the same. That closes Silent Failures in Production SaaS. Cron jobs, webhooks, email, config drift, backups and now unclicked paths: the common thread is that a system with no errors is not the same as a system that works. Give each critical behaviour a positive signal, alert when that signal goes quiet, and measure it against a target someone agreed to.