{"slug": "the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success", "title": "The Month My AI Pipeline Failed 100% of the Time and Every Log Said \"Success\"", "summary": "An engineer at MyVitals detailed a month-long incident where an AI pipeline silently failed 100% of the time while logs reported success. A misconfigured environment variable caused every metric-matching LLM call to return a 400 Invalid Model error, but graceful degradation masked the failure, fragmenting the canonical dictionary. The fix involved typed failure classification and a shared wall-clock budget to align retries with Nginx's proxy timeout.", "body_md": "[MyVitals](https://myvitals.co.in) extracts lab values with one LLM call, then reconciles raw names against a shared dictionary — exact match, fuzzy match, then an LLM call for the leftovers. If that call fails outright, the pipeline doesn't fail the upload — it degrades: unmatched metrics auto-create a new dictionary entry instead of finding the right one. Good design, on its own.\n\nHere's the problem: an env var meant for the OCR endpoint got set on the variable that goes to Chat Completions instead. Every metric-matching call, for about a month, hit `400 Invalid Model`. Every single one. And because of the graceful degradation above, **every upload still reported success.**\n\nThe visible symptom wasn't an error — it was a canonical-metrics dictionary slowly fragmenting into near-duplicates. Nothing about that looks like a bug from the outside. The metric everyone watches, \"did uploads succeed,\" stayed at 100%.\n\n**Lesson**: when a stage is designed to degrade instead of fail loudly, the success of the operation it's embedded in tells you nothing about whether that stage ran. You need a metric for the stage itself.\n\nNot an alert, not a test — there's no integration harness exercising the real pipeline. A routine cost audit noticed the LLM-matching cost line had gone quiet, which prompted someone to actually read the error logs. The diagnosis — `400 Invalid Model` — had been sitting in the database the whole time, inside a raw error string, in a tooltip nobody had reason to open.\n\nThe fix wasn't more data, it was structuring what already existed: typed failure classification (error type, HTTP status, retryability) grouped by that classification instead of raw message text. `4× · 36% of matching calls · invalid_model` is a bug report. Four chronological stack traces is a wall nobody reads.\n\nExtraction runs synchronously inside the upload request — no job queue yet. Nginx's `proxy_read_timeout` is 120s. If retry logic doesn't know that number: three 90-second retries run for 4.5 minutes against a socket Nginx already closed. Client gets a 504, server keeps working on nothing, report sits at `\"processing\"` forever.\n\nFix: a shared wall-clock budget (105s, deliberately under 120s) passed through both extraction and matching, so retries stop before the proxy would kill the connection anyway. Retries only happen for genuinely transient failures — a 401 isn't going to succeed on attempt two, and every retry is a real charge. A provider's `Retry-After` is honored as sent, not clamped — ignoring a rate limiter's instructions just earns another 429 faster.\n\n**Your retry budget and your proxy's timeout are the same constraint wearing two config files.** Tune only one and you haven't fixed anything — you've just moved where the zombie request dies.\n\n👉 **[Try MyVitals now](https://myvitals.co.in)** — the pipeline now tells on itself when something's wrong.", "url": "https://wpnews.pro/news/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success", "canonical_source": "https://dev.to/suyashdev/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success-58ba", "published_at": "2026-09-07 07:51:00+00:00", "updated_at": "2026-09-07 07:57:40.492111+00:00", "lang": "en", "topics": ["artificial-intelligence", "mlops", "developer-tools"], "entities": ["MyVitals", "OpenAI", "Nginx"], "alternates": {"html": "https://wpnews.pro/news/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success", "markdown": "https://wpnews.pro/news/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success.md", "text": "https://wpnews.pro/news/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success.txt", "jsonld": "https://wpnews.pro/news/the-month-my-ai-pipeline-failed-100-of-the-time-and-every-log-said-success.jsonld"}}