{"slug": "false-green-my-incident-agent-said-healthy-without-live-evidence", "title": "False Green: My Incident Agent Said “Healthy” Without Live Evidence", "summary": "A developer's incident agent falsely reported a Fivetran pipeline as 'Healthy' after a live check failed, causing the app to fall back to cached evidence that still indicated a healthy state. The false green occurred in deterministic application logic before any AI model was involved, and it could misdirect the investigation's next steps. The developer fixed the issue by ensuring cached evidence cannot support a current 'Healthy' verdict when live evidence is unavailable.", "body_md": "*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*\n\nThe pipeline was healthy.\n\nAt least, that was what my incident agent told me.\n\nThere was one problem: it could not see the live Fivetran pipeline.\n\nThe live check had failed, so the app fell back to cached evidence. That snapshot still said `connected`\n\nand `on_schedule`\n\n.\n\nThe final verdict was still:\n\n**Healthy.**\n\nNothing crashed.\n\nThe request succeeded.\n\nThe fallback data looked healthy.\n\n**The conclusion was still wrong.**\n\nThe application had taken the last known state and quietly treated it like the current one.\n\nThat was the false green.\n\nCurrent confidence requires current evidence.\n\nPipeline Rescue Agent is a Next.js incident-investigation app that combines connector-health and data-freshness signals to decide what still needs investigation.\n\nGemini runs later in the recovery-planning flow. The false-green verdict occurred in deterministic application logic before the model was involved.\n\nThe question at this decision point was simple:\n\n**Is this pipeline healthy right now?**\n\nI found a case where the application was more confident than its evidence allowed.\n\nWhen live Fivetran telemetry became unavailable, Pipeline Rescue Agent fell back to cached connector evidence.\n\nI wanted to keep that fallback. The last known connector state can still be useful during an incident.\n\nThe failed live path was the trigger condition.\n\nIt was not the application defect.\n\nThe defect was what happened afterward: cached evidence was still allowed to support a current `Healthy`\n\nverdict.\n\nBefore the fix, the decision state looked like this:\n\n```\nevidence.mode = cached_fivetran_evidence\nevidence.live = false\n\nfivetran.mcp_ok = false\nfivetran.setup_state = connected\nfivetran.update_state = on_schedule\n\npipeline.health_verdict = healthy\nfalse_green.detected = true\n```\n\nThere are three intentional repository states in this investigation:\n\n| State | Purpose |\n|---|---|\n`6194a3b14efd33a0d5f8235f703187005123a0f9` |\nHistorical tagged baseline, before Sentry instrumentation |\n`b3a45110` |\nSentry instrumentation added while the decision bug was still intact |\n`b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86` |\nFinal fixed state |\n\nThe sequence was deliberate:\n\n```\nreproduce\n→ instrument the still-broken decision\n→ inspect\n→ diagnose\n→ fix\n→ verify\n```\n\n*Pre-fix Sentry decision span: cached, non-live evidence still produced a Healthy verdict.*\n\nAlmost every individual status value looked reasonable.\n\nThe `/api/investigate`\n\nrequest itself completed successfully. The fallback returned data. The cached connector status said `connected`\n\nand `on_schedule`\n\n. There were no active warnings or tasks.\n\nThe endpoint completed successfully; the semantic failure itself raised no exception.\n\nThe contradiction only appeared when evidence provenance and the final decision were inspected together:\n\n```\ncached evidence\n+ live = false\n+ MCP unavailable\n→ Healthy\n```\n\nThe cache itself was doing its job.\n\n**The health gate was asking the cache to prove something it could not prove.**\n\nCached evidence could answer:\n\nWhat was last known?\n\nIt could not answer:\n\nIs the pipeline healthy right now?\n\n`Healthy`\n\nwas not only a displayed status.\n\nThe connector decision feeds the agent's final `pipelineStatus`\n\n, which `getLikelyIssue`\n\nuses when choosing what the investigation should examine next.\n\nWhen connector health cannot be established:\n\n```\nNeeds review\n→ likely issue: Connector health\n```\n\nWhen the connector is considered healthy while destination data is stale:\n\n```\nHealthy + stale data\n→ likely issue: Upstream source freshness\n```\n\n**A false green did not merely change the displayed status; it could change what the investigation examined next.**\n\nI did not need the application to assume Fivetran was broken. I needed it to stop clearing connector health when current connector evidence was unavailable.\n\nI first reproduced the suspicious `Healthy`\n\nresult through `/api/investigate`\n\n.\n\nBefore fixing it, I added a custom Sentry `pipeline.decision`\n\nspan containing the runtime state behind that decision:\n\n```\nevidence.mode\nevidence.live\nfivetran.mcp_ok\nfivetran.setup_state\nfivetran.update_state\npipeline.health_verdict\nfalse_green.detected\n```\n\nThis was a silent correctness bug.\n\nThe request worked. The fallback worked. The values looked plausible. The failure was in the conclusion drawn from them.\n\nSentry put the healthy-looking status content, its non-live provenance, and the final verdict in the same trace.\n\nThat changed the debugging question from:\n\nWhy does this endpoint say\n\n`Healthy`\n\n?\n\nto:\n\nWhy is non-live evidence allowed to support\n\n`Healthy`\n\n?\n\nI used Seer to analyze the captured failing trace.\n\nIt highlighted evidence currentness as the likely missing decision constraint: healthy-looking cached values were reaching a verdict path that did not enforce live provenance.\n\n*Seer RCA: the failing trace pointed to evidence currentness/provenance as the missing decision constraint.*\n\nI treated that as a hypothesis and verified it against the source.\n\nI also did not copy the generated suggestion literally.\n\nPipeline Rescue Agent supports two legitimate live evidence paths:\n\n```\nmcp_live\nlive\n```\n\nA direct live Fivetran API response can still be current evidence even when the MCP path is unavailable.\n\nSo the correct invariant was not:\n\n```\nHealthy requires MCP success\n```\n\nIt was:\n\n```\nHealthy requires live evidence\n```\n\nThe bug reduced to one safety property:\n\nA`Healthy`\n\nverdict requires live evidence.\n\n```\nHealthy ⇒ evidence.live = true\n```\n\nThe failing Sentry trace violated it:\n\n```\nevidence.live = false\npipeline.health_verdict = healthy\n```\n\nThe cache could remain; only the decision boundary had to change.\n\nPR: [Fix false-green pipeline health verdicts from cached Fivetran evidence](https://github.com/mneang/pipeline-rescue-agent/pull/1)\n\nI isolated the health decision in a small helper used by the production route:\n\n```\nexport function evaluateFivetranHealth({\n  mode,\n  setupState,\n  updateState,\n  hasWarnings,\n  hasTasks,\n  paused,\n}) {\n  const hasLiveEvidence = mode === \"mcp_live\" || mode === \"live\";\n\n  const isHealthy =\n    hasLiveEvidence &&\n    setupState === \"connected\" &&\n    updateState === \"on_schedule\" &&\n    !hasWarnings &&\n    !hasTasks &&\n    paused === false;\n\n  return {\n    hasLiveEvidence,\n    isHealthy,\n  };\n}\n```\n\nThe important distinction is small:\n\n| Evidence | What it may establish |\n|---|---|\nLive |\nMay certify current health |\nCached |\nHistorical context only; cannot independently certify current health |\n\nHistorical evidence can still inform the investigation.\n\nOnly current evidence may certify current health.\n\nI checked the surrounding investigation flow for the same provenance assumption. Cached Fivetran evidence no longer appears as a successful current connector check, no longer counts as live evidence, and is no longer described in recovery reasoning as if it represented current health. The production rule now lives in the small pure helper above, which also makes the decision boundary directly testable.\n\nThe final Sentry trace from commit `b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86`\n\nshows:\n\n```\nevidence.mode = cached_fivetran_evidence\nevidence.live = false\n\nfivetran.mcp_ok = false\nfivetran.setup_state = connected\nfivetran.update_state = on_schedule\n\nprovenance.guard_applied = true\npipeline.health_verdict = needs_review\n```\n\n*Final Sentry decision span: the same non-live evidence class now produces Needs review.*\n\nThe status values still look superficially healthy.\n\nThe live telemetry is still unavailable.\n\n**The fix did not make unavailable telemetry available. It changed what conclusion the application was allowed to derive from that uncertainty.**\n\n```\ncached + non-live\n→ Needs review\n```\n\nI wanted stronger verification than:\n\nThe patched application passes a test written after patching it.\n\nSo I moved the decisive assertion outside the repository and ran it unchanged against the tagged historical application and the final branch.\n\nThe script imports no application decision logic. It intentionally reads only response fields available in both versions:\n\n```\ntimeline[].evidence.mode\nagentRun.decision.pipelineStatus\n```\n\nIt also refuses to count a run if the expected cached-fallback condition was not reproduced.\n\nIn simplified form, the external assertion enforced three conditions:\n\n```\nif (mode !== \"cached_fivetran_evidence\") {\n  throw new Error(\"TEST INVALID: cached fallback was not reproduced\");\n}\n\nif (verdict === \"Healthy\") {\n  process.exit(1);\n}\n\nif (verdict !== \"Needs review\") {\n  throw new Error(`Unexpected verdict: ${verdict}`);\n}\n```\n\nI ran that same external assertion against the historical baseline:\n\n```\nCommit:\n6194a3b14efd33a0d5f8235f703187005123a0f9\n\nEvidence mode:\ncached_fivetran_evidence\n\nPipeline verdict:\nHealthy\n\nResult:\nFAIL\n\nExit code:\n1\n```\n\nThen I ran it against the final branch:\n\n```\nCommit:\nb05d2cccf93713f6a52a4a4ec6fa6b1600cbec86\n\nEvidence mode:\ncached_fivetran_evidence\n\nPipeline verdict:\nNeeds review\n\nResult:\nPASS\n\nExit code:\n0\n```\n\n*External black-box check: historical baseline fails; final branch passes.*\n\nThis comparison does not claim that every low-level cause of telemetry unavailability was identical between the two executions.\n\nThat is not what the assertion tests.\n\nThe controlled application condition is:\n\nFivetran evidence is cached rather than live.\n\nThe question is:\n\nCan that evidence still certify current health?\n\nHistorical application:\n\n**Yes.**\n\nFinal application:\n\n**No.**\n\nThere was one more bad fix I wanted to rule out:\n\nWhat if\n\n`Healthy`\n\nsimply became impossible?\n\nThe same pure predicate imported by the production route was checked on both sides of the boundary:\n\n| Scenario | Expected | Result |\n|---|---|---|\n`cached_fivetran_evidence` + clean status |\n`Healthy = false` |\nPASS |\n`mcp_live` + clean status |\n`Healthy = true` |\nPASS |\n`live` + clean status |\n`Healthy = true` |\nPASS |\n\nThis is intentionally a decision-boundary check.\n\nIt does **not** claim to contact a live MCP server or the live Fivetran API.\n\nIts purpose is narrower: both supported live evidence modes remain eligible for `Healthy`\n\n, while cached evidence does not.\n\nThe non-live application behavior is separately verified through `/api/investigate`\n\nand the external black-box assertion.\n\nFinal branch verification:\n\n```\nnpm run test:health-invariant\nPASS\n\nPIPELINE_RESCUE_URL=http://localhost:3001 npm run test:false-green\nPASS\n\nExternal black-box check\nPASS\nexit 0\n\nnpm run lint\nPASS\n\nnpm run build\nPASS\n```\n\nThe HTTP regression check confirms:\n\n```\nEvidence mode: cached_fivetran_evidence\nLive evidence: false\nPipeline verdict: Needs review\n\nPASS: non-live cached evidence cannot produce a Healthy verdict.\n```\n\nThis was exactly where Sentry mattered: `/api/investigate`\n\ncompleted successfully, so ordinary error monitoring did not expose the semantic failure.\n\nThe `pipeline.decision`\n\nspan made provenance, liveness, connector state, and final verdict observable together inside that successful request.\n\n**Sentry did not merely verify the repair after the fact: I captured the contradiction from the still-buggy release, used that failing trace as runtime context for Seer, and then inspected the same decision again after the fix.**\n\nSeer then helped narrow that runtime contradiction to evidence currentness, which I verified against the source before implementing the narrower application invariant.\n\nAfter the fix, the final trace shows:\n\n```\nevidence.live = false\nprovenance.guard_applied = true\npipeline.health_verdict = needs_review\n```\n\nSentry did not replace the debugging.\n\nIt exposed the hidden decision state that ordinary success/failure monitoring would not have explained.\n\n```\nBEFORE\n\nNo live Fivetran telemetry\n→ cached healthy-looking evidence\n→ Healthy\n\nAFTER\n\nNo live Fivetran telemetry\n→ cached evidence retained as context\n→ live-evidence guard\n→ Needs review\n```\n\nThe cache can still tell an investigator what was last known.\n\nIt just no longer gets to answer:\n\nIs the pipeline healthy right now?\n\nCurrent confidence requires current evidence.", "url": "https://wpnews.pro/news/false-green-my-incident-agent-said-healthy-without-live-evidence", "canonical_source": "https://dev.to/mneang/false-green-my-incident-agent-said-healthy-without-live-evidence-2di3", "published_at": "2026-08-24 04:08:29+00:00", "updated_at": "2026-08-24 04:43:49.824797+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools"], "entities": ["Fivetran", "Pipeline Rescue Agent", "Gemini", "Sentry"], "alternates": {"html": "https://wpnews.pro/news/false-green-my-incident-agent-said-healthy-without-live-evidence", "markdown": "https://wpnews.pro/news/false-green-my-incident-agent-said-healthy-without-live-evidence.md", "text": "https://wpnews.pro/news/false-green-my-incident-agent-said-healthy-without-live-evidence.txt", "jsonld": "https://wpnews.pro/news/false-green-my-incident-agent-said-healthy-without-live-evidence.jsonld"}}