# The Month My AI Pipeline Failed 100% of the Time and Every Log Said "Success"

> Source: <https://dev.to/suyashdev/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success-58ba>
> Published: 2026-09-07 07:51:00+00:00

[MyVitals](https://myvitals.co.in) extracts lab values with one LLM call, then reconciles raw names against a shared dictionary — exact match, fuzzy match, then an LLM call for the leftovers. If that call fails outright, the pipeline doesn't fail the upload — it degrades: unmatched metrics auto-create a new dictionary entry instead of finding the right one. Good design, on its own.

Here's the problem: an env var meant for the OCR endpoint got set on the variable that goes to Chat Completions instead. Every metric-matching call, for about a month, hit `400 Invalid Model`. Every single one. And because of the graceful degradation above, **every upload still reported success.**

The visible symptom wasn't an error — it was a canonical-metrics dictionary slowly fragmenting into near-duplicates. Nothing about that looks like a bug from the outside. The metric everyone watches, "did uploads succeed," stayed at 100%.

**Lesson**: when a stage is designed to degrade instead of fail loudly, the success of the operation it's embedded in tells you nothing about whether that stage ran. You need a metric for the stage itself.

Not an alert, not a test — there's no integration harness exercising the real pipeline. A routine cost audit noticed the LLM-matching cost line had gone quiet, which prompted someone to actually read the error logs. The diagnosis — `400 Invalid Model` — had been sitting in the database the whole time, inside a raw error string, in a tooltip nobody had reason to open.

The fix wasn't more data, it was structuring what already existed: typed failure classification (error type, HTTP status, retryability) grouped by that classification instead of raw message text. `4× · 36% of matching calls · invalid_model` is a bug report. Four chronological stack traces is a wall nobody reads.

Extraction runs synchronously inside the upload request — no job queue yet. Nginx's `proxy_read_timeout` is 120s. If retry logic doesn't know that number: three 90-second retries run for 4.5 minutes against a socket Nginx already closed. Client gets a 504, server keeps working on nothing, report sits at `"processing"` forever.

Fix: a shared wall-clock budget (105s, deliberately under 120s) passed through both extraction and matching, so retries stop before the proxy would kill the connection anyway. Retries only happen for genuinely transient failures — a 401 isn't going to succeed on attempt two, and every retry is a real charge. A provider's `Retry-After` is honored as sent, not clamped — ignoring a rate limiter's instructions just earns another 429 faster.

**Your retry budget and your proxy's timeout are the same constraint wearing two config files.** Tune only one and you haven't fixed anything — you've just moved where the zombie request dies.

👉 **[Try MyVitals now](https://myvitals.co.in)** — the pipeline now tells on itself when something's wrong.