False Green: My Incident Agent Said “Healthy” Without Live Evidence

A developer's incident agent falsely reported a Fivetran pipeline as 'Healthy' after a live check failed, causing the app to fall back to cached evidence that still indicated a healthy state. The false green occurred in deterministic application logic before any AI model was involved, and it could misdirect the investigation's next steps. The developer fixed the issue by ensuring cached evidence cannot support a current 'Healthy' verdict when live evidence is unavailable.

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry. The pipeline was healthy. At least, that was what my incident agent told me. There was one problem: it could not see the live Fivetran pipeline. The live check had failed, so the app fell back to cached evidence. That snapshot still said connected and on schedule . The final verdict was still: Healthy. Nothing crashed. The request succeeded. The fallback data looked healthy. The conclusion was still wrong. The application had taken the last known state and quietly treated it like the current one. That was the false green. Current confidence requires current evidence. Pipeline Rescue Agent is a Next.js incident-investigation app that combines connector-health and data-freshness signals to decide what still needs investigation. Gemini runs later in the recovery-planning flow. The false-green verdict occurred in deterministic application logic before the model was involved. The question at this decision point was simple: Is this pipeline healthy right now? I found a case where the application was more confident than its evidence allowed. When live Fivetran telemetry became unavailable, Pipeline Rescue Agent fell back to cached connector evidence. I wanted to keep that fallback. The last known connector state can still be useful during an incident. The failed live path was the trigger condition. It was not the application defect. The defect was what happened afterward: cached evidence was still allowed to support a current Healthy verdict. Before the fix, the decision state looked like this: evidence.mode = cached fivetran evidence evidence.live = false fivetran.mcp ok = false fivetran.setup state = connected fivetran.update state = on schedule pipeline.health verdict = healthy false green.detected = true There are three intentional repository states in this investigation: | State | Purpose | |---|---| 6194a3b14efd33a0d5f8235f703187005123a0f9 | Historical tagged baseline, before Sentry instrumentation | b3a45110 | Sentry instrumentation added while the decision bug was still intact | b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 | Final fixed state | The sequence was deliberate: reproduce → instrument the still-broken decision → inspect → diagnose → fix → verify Pre-fix Sentry decision span: cached, non-live evidence still produced a Healthy verdict. Almost every individual status value looked reasonable. The /api/investigate request itself completed successfully. The fallback returned data. The cached connector status said connected and on schedule . There were no active warnings or tasks. The endpoint completed successfully; the semantic failure itself raised no exception. The contradiction only appeared when evidence provenance and the final decision were inspected together: cached evidence + live = false + MCP unavailable → Healthy The cache itself was doing its job. The health gate was asking the cache to prove something it could not prove. Cached evidence could answer: What was last known? It could not answer: Is the pipeline healthy right now? Healthy was not only a displayed status. The connector decision feeds the agent's final pipelineStatus , which getLikelyIssue uses when choosing what the investigation should examine next. When connector health cannot be established: Needs review → likely issue: Connector health When the connector is considered healthy while destination data is stale: Healthy + stale data → likely issue: Upstream source freshness A false green did not merely change the displayed status; it could change what the investigation examined next. I did not need the application to assume Fivetran was broken. I needed it to stop clearing connector health when current connector evidence was unavailable. I first reproduced the suspicious Healthy result through /api/investigate . Before fixing it, I added a custom Sentry pipeline.decision span containing the runtime state behind that decision: evidence.mode evidence.live fivetran.mcp ok fivetran.setup state fivetran.update state pipeline.health verdict false green.detected This was a silent correctness bug. The request worked. The fallback worked. The values looked plausible. The failure was in the conclusion drawn from them. Sentry put the healthy-looking status content, its non-live provenance, and the final verdict in the same trace. That changed the debugging question from: Why does this endpoint say Healthy ? to: Why is non-live evidence allowed to support Healthy ? I used Seer to analyze the captured failing trace. It highlighted evidence currentness as the likely missing decision constraint: healthy-looking cached values were reaching a verdict path that did not enforce live provenance. Seer RCA: the failing trace pointed to evidence currentness/provenance as the missing decision constraint. I treated that as a hypothesis and verified it against the source. I also did not copy the generated suggestion literally. Pipeline Rescue Agent supports two legitimate live evidence paths: mcp live live A direct live Fivetran API response can still be current evidence even when the MCP path is unavailable. So the correct invariant was not: Healthy requires MCP success It was: Healthy requires live evidence The bug reduced to one safety property: A Healthy verdict requires live evidence. Healthy ⇒ evidence.live = true The failing Sentry trace violated it: evidence.live = false pipeline.health verdict = healthy The cache could remain; only the decision boundary had to change. PR: Fix false-green pipeline health verdicts from cached Fivetran evidence https://github.com/mneang/pipeline-rescue-agent/pull/1 I isolated the health decision in a small helper used by the production route: export function evaluateFivetranHealth { mode, setupState, updateState, hasWarnings, hasTasks, paused, } { const hasLiveEvidence = mode === "mcp live" || mode === "live"; const isHealthy = hasLiveEvidence && setupState === "connected" && updateState === "on schedule" && hasWarnings && hasTasks && paused === false; return { hasLiveEvidence, isHealthy, }; } The important distinction is small: | Evidence | What it may establish | |---|---| Live | May certify current health | Cached | Historical context only; cannot independently certify current health | Historical evidence can still inform the investigation. Only current evidence may certify current health. I checked the surrounding investigation flow for the same provenance assumption. Cached Fivetran evidence no longer appears as a successful current connector check, no longer counts as live evidence, and is no longer described in recovery reasoning as if it represented current health. The production rule now lives in the small pure helper above, which also makes the decision boundary directly testable. The final Sentry trace from commit b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 shows: evidence.mode = cached fivetran evidence evidence.live = false fivetran.mcp ok = false fivetran.setup state = connected fivetran.update state = on schedule provenance.guard applied = true pipeline.health verdict = needs review Final Sentry decision span: the same non-live evidence class now produces Needs review. The status values still look superficially healthy. The live telemetry is still unavailable. The fix did not make unavailable telemetry available. It changed what conclusion the application was allowed to derive from that uncertainty. cached + non-live → Needs review I wanted stronger verification than: The patched application passes a test written after patching it. So I moved the decisive assertion outside the repository and ran it unchanged against the tagged historical application and the final branch. The script imports no application decision logic. It intentionally reads only response fields available in both versions: timeline .evidence.mode agentRun.decision.pipelineStatus It also refuses to count a run if the expected cached-fallback condition was not reproduced. In simplified form, the external assertion enforced three conditions: if mode == "cached fivetran evidence" { throw new Error "TEST INVALID: cached fallback was not reproduced" ; } if verdict === "Healthy" { process.exit 1 ; } if verdict == "Needs review" { throw new Error Unexpected verdict: ${verdict} ; } I ran that same external assertion against the historical baseline: Commit: 6194a3b14efd33a0d5f8235f703187005123a0f9 Evidence mode: cached fivetran evidence Pipeline verdict: Healthy Result: FAIL Exit code: 1 Then I ran it against the final branch: Commit: b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 Evidence mode: cached fivetran evidence Pipeline verdict: Needs review Result: PASS Exit code: 0 External black-box check: historical baseline fails; final branch passes. This comparison does not claim that every low-level cause of telemetry unavailability was identical between the two executions. That is not what the assertion tests. The controlled application condition is: Fivetran evidence is cached rather than live. The question is: Can that evidence still certify current health? Historical application: Yes. Final application: No. There was one more bad fix I wanted to rule out: What if Healthy simply became impossible? The same pure predicate imported by the production route was checked on both sides of the boundary: | Scenario | Expected | Result | |---|---|---| cached fivetran evidence + clean status | Healthy = false | PASS | mcp live + clean status | Healthy = true | PASS | live + clean status | Healthy = true | PASS | This is intentionally a decision-boundary check. It does not claim to contact a live MCP server or the live Fivetran API. Its purpose is narrower: both supported live evidence modes remain eligible for Healthy , while cached evidence does not. The non-live application behavior is separately verified through /api/investigate and the external black-box assertion. Final branch verification: npm run test:health-invariant PASS PIPELINE RESCUE URL=http://localhost:3001 npm run test:false-green PASS External black-box check PASS exit 0 npm run lint PASS npm run build PASS The HTTP regression check confirms: Evidence mode: cached fivetran evidence Live evidence: false Pipeline verdict: Needs review PASS: non-live cached evidence cannot produce a Healthy verdict. This was exactly where Sentry mattered: /api/investigate completed successfully, so ordinary error monitoring did not expose the semantic failure. The pipeline.decision span made provenance, liveness, connector state, and final verdict observable together inside that successful request. Sentry did not merely verify the repair after the fact: I captured the contradiction from the still-buggy release, used that failing trace as runtime context for Seer, and then inspected the same decision again after the fix. Seer then helped narrow that runtime contradiction to evidence currentness, which I verified against the source before implementing the narrower application invariant. After the fix, the final trace shows: evidence.live = false provenance.guard applied = true pipeline.health verdict = needs review Sentry did not replace the debugging. It exposed the hidden decision state that ordinary success/failure monitoring would not have explained. BEFORE No live Fivetran telemetry → cached healthy-looking evidence → Healthy AFTER No live Fivetran telemetry → cached evidence retained as context → live-evidence guard → Needs review The cache can still tell an investigator what was last known. It just no longer gets to answer: Is the pipeline healthy right now? Current confidence requires current evidence.