This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
The pipeline was healthy.
At least, that was what my incident agent told me.
There was one problem: it could not see the live Fivetran pipeline.
The live check had failed, so the app fell back to cached evidence. That snapshot still said connected
and on_schedule
.
The final verdict was still:
Healthy.
Nothing crashed.
The request succeeded.
The fallback data looked healthy.
The conclusion was still wrong.
The application had taken the last known state and quietly treated it like the current one.
That was the false green.
Current confidence requires current evidence.
Pipeline Rescue Agent is a Next.js incident-investigation app that combines connector-health and data-freshness signals to decide what still needs investigation.
Gemini runs later in the recovery-planning flow. The false-green verdict occurred in deterministic application logic before the model was involved.
The question at this decision point was simple:
Is this pipeline healthy right now?
I found a case where the application was more confident than its evidence allowed.
When live Fivetran telemetry became unavailable, Pipeline Rescue Agent fell back to cached connector evidence.
I wanted to keep that fallback. The last known connector state can still be useful during an incident.
The failed live path was the trigger condition.
It was not the application defect.
The defect was what happened afterward: cached evidence was still allowed to support a current Healthy
verdict.
Before the fix, the decision state looked like this:
evidence.mode = cached_fivetran_evidence
evidence.live = false
fivetran.mcp_ok = false
fivetran.setup_state = connected
fivetran.update_state = on_schedule
pipeline.health_verdict = healthy
false_green.detected = true
There are three intentional repository states in this investigation:
| State | Purpose |
|---|---|
6194a3b14efd33a0d5f8235f703187005123a0f9 |
|
| Historical tagged baseline, before Sentry instrumentation | |
b3a45110 |
|
| Sentry instrumentation added while the decision bug was still intact | |
b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86 |
|
| Final fixed state |
The sequence was deliberate:
reproduce
→ instrument the still-broken decision
→ inspect
→ diagnose
→ fix
→ verify
Pre-fix Sentry decision span: cached, non-live evidence still produced a Healthy verdict.
Almost every individual status value looked reasonable.
The /api/investigate
request itself completed successfully. The fallback returned data. The cached connector status said connected
and on_schedule
. There were no active warnings or tasks.
The endpoint completed successfully; the semantic failure itself raised no exception.
The contradiction only appeared when evidence provenance and the final decision were inspected together:
cached evidence
+ live = false
+ MCP unavailable
→ Healthy
The cache itself was doing its job.
The health gate was asking the cache to prove something it could not prove.
Cached evidence could answer:
What was last known?
It could not answer:
Is the pipeline healthy right now?
Healthy
was not only a displayed status.
The connector decision feeds the agent's final pipelineStatus
, which getLikelyIssue
uses when choosing what the investigation should examine next.
When connector health cannot be established:
Needs review
→ likely issue: Connector health
When the connector is considered healthy while destination data is stale:
Healthy + stale data
→ likely issue: Upstream source freshness
A false green did not merely change the displayed status; it could change what the investigation examined next.
I did not need the application to assume Fivetran was broken. I needed it to stop clearing connector health when current connector evidence was unavailable.
I first reproduced the suspicious Healthy
result through /api/investigate
.
Before fixing it, I added a custom Sentry pipeline.decision
span containing the runtime state behind that decision:
evidence.mode
evidence.live
fivetran.mcp_ok
fivetran.setup_state
fivetran.update_state
pipeline.health_verdict
false_green.detected
This was a silent correctness bug.
The request worked. The fallback worked. The values looked plausible. The failure was in the conclusion drawn from them.
Sentry put the healthy-looking status content, its non-live provenance, and the final verdict in the same trace.
That changed the debugging question from:
Why does this endpoint say
Healthy
?
to:
Why is non-live evidence allowed to support
Healthy
?
I used Seer to analyze the captured failing trace.
It highlighted evidence currentness as the likely missing decision constraint: healthy-looking cached values were reaching a verdict path that did not enforce live provenance.
Seer RCA: the failing trace pointed to evidence currentness/provenance as the missing decision constraint.
I treated that as a hypothesis and verified it against the source.
I also did not copy the generated suggestion literally.
Pipeline Rescue Agent supports two legitimate live evidence paths:
mcp_live
live
A direct live Fivetran API response can still be current evidence even when the MCP path is unavailable.
So the correct invariant was not:
Healthy requires MCP success
It was:
Healthy requires live evidence
The bug reduced to one safety property:
AHealthy
verdict requires live evidence.
Healthy ⇒ evidence.live = true
The failing Sentry trace violated it:
evidence.live = false
pipeline.health_verdict = healthy
The cache could remain; only the decision boundary had to change.
PR: Fix false-green pipeline health verdicts from cached Fivetran evidence
I isolated the health decision in a small helper used by the production route:
export function evaluateFivetranHealth({
mode,
setupState,
updateState,
hasWarnings,
hasTasks,
d,
}) {
const hasLiveEvidence = mode === "mcp_live" || mode === "live";
const isHealthy =
hasLiveEvidence &&
setupState === "connected" &&
updateState === "on_schedule" &&
!hasWarnings &&
!hasTasks &&
d === false;
return {
hasLiveEvidence,
isHealthy,
};
}
The important distinction is small:
| Evidence | What it may establish |
|---|---|
| Live | |
| May certify current health | |
| Cached | |
| Historical context only; cannot independently certify current health |
Historical evidence can still inform the investigation.
Only current evidence may certify current health.
I checked the surrounding investigation flow for the same provenance assumption. Cached Fivetran evidence no longer appears as a successful current connector check, no longer counts as live evidence, and is no longer described in recovery reasoning as if it represented current health. The production rule now lives in the small pure helper above, which also makes the decision boundary directly testable.
The final Sentry trace from commit b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86
shows:
evidence.mode = cached_fivetran_evidence
evidence.live = false
fivetran.mcp_ok = false
fivetran.setup_state = connected
fivetran.update_state = on_schedule
provenance.guard_applied = true
pipeline.health_verdict = needs_review
Final Sentry decision span: the same non-live evidence class now produces Needs review.
The status values still look superficially healthy.
The live telemetry is still unavailable.
The fix did not make unavailable telemetry available. It changed what conclusion the application was allowed to derive from that uncertainty.
cached + non-live
→ Needs review
I wanted stronger verification than:
The patched application passes a test written after patching it.
So I moved the decisive assertion outside the repository and ran it unchanged against the tagged historical application and the final branch.
The script imports no application decision logic. It intentionally reads only response fields available in both versions:
timeline[].evidence.mode
agentRun.decision.pipelineStatus
It also refuses to count a run if the expected cached-fallback condition was not reproduced.
In simplified form, the external assertion enforced three conditions:
if (mode !== "cached_fivetran_evidence") {
throw new Error("TEST INVALID: cached fallback was not reproduced");
}
if (verdict === "Healthy") {
process.exit(1);
}
if (verdict !== "Needs review") {
throw new Error(`Unexpected verdict: ${verdict}`);
}
I ran that same external assertion against the historical baseline:
Commit:
6194a3b14efd33a0d5f8235f703187005123a0f9
Evidence mode:
cached_fivetran_evidence
Pipeline verdict:
Healthy
Result:
FAIL
Exit code:
1
Then I ran it against the final branch:
Commit:
b05d2cccf93713f6a52a4a4ec6fa6b1600cbec86
Evidence mode:
cached_fivetran_evidence
Pipeline verdict:
Needs review
Result:
PASS
Exit code:
0
External black-box check: historical baseline fails; final branch passes.
This comparison does not claim that every low-level cause of telemetry unavailability was identical between the two executions.
That is not what the assertion tests.
The controlled application condition is:
Fivetran evidence is cached rather than live.
The question is:
Can that evidence still certify current health?
Historical application:
Yes.
Final application:
No.
There was one more bad fix I wanted to rule out:
What if
Healthy
simply became impossible?
The same pure predicate imported by the production route was checked on both sides of the boundary:
| Scenario | Expected | Result |
|---|---|---|
cached_fivetran_evidence + clean status |
||
Healthy = false |
||
| PASS | ||
mcp_live + clean status |
||
Healthy = true |
||
| PASS | ||
live + clean status |
||
Healthy = true |
||
| PASS |
This is intentionally a decision-boundary check.
It does not claim to contact a live MCP server or the live Fivetran API.
Its purpose is narrower: both supported live evidence modes remain eligible for Healthy
, while cached evidence does not.
The non-live application behavior is separately verified through /api/investigate
and the external black-box assertion.
Final branch verification:
npm run test:health-invariant
PASS
PIPELINE_RESCUE_URL=http://localhost:3001 npm run test:false-green
PASS
External black-box check
PASS
exit 0
npm run lint
PASS
npm run build
PASS
The HTTP regression check confirms:
Evidence mode: cached_fivetran_evidence
Live evidence: false
Pipeline verdict: Needs review
PASS: non-live cached evidence cannot produce a Healthy verdict.
This was exactly where Sentry mattered: /api/investigate
completed successfully, so ordinary error monitoring did not expose the semantic failure.
The pipeline.decision
span made provenance, liveness, connector state, and final verdict observable together inside that successful request.
Sentry did not merely verify the repair after the fact: I captured the contradiction from the still-buggy release, used that failing trace as runtime context for Seer, and then inspected the same decision again after the fix.
Seer then helped narrow that runtime contradiction to evidence currentness, which I verified against the source before implementing the narrower application invariant.
After the fix, the final trace shows:
evidence.live = false
provenance.guard_applied = true
pipeline.health_verdict = needs_review
Sentry did not replace the debugging.
It exposed the hidden decision state that ordinary success/failure monitoring would not have explained.
BEFORE
No live Fivetran telemetry
→ cached healthy-looking evidence
→ Healthy
AFTER
No live Fivetran telemetry
→ cached evidence retained as context
→ live-evidence guard
→ Needs review
The cache can still tell an investigator what was last known.
It just no longer gets to answer:
Is the pipeline healthy right now?
Current confidence requires current evidence.