{"slug": "nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline", "title": "Nobody Alerts on Silence: Wiring Sentry Into an LLM Pipeline", "summary": "A developer wired Sentry error monitoring into the LLM pipeline of TextStack, an open-source .NET reader, after discovering that a user's PDF fell onto a CPU-only Ollama container, causing 390% CPU usage for an hour with zero alerts. The fix includes route resolution that records intent, alert throttling, and a per-book re-check that caught a hole in the initial fix. The first 24 hours in production caught HTTP 429 errors on the paid translation and explanation endpoints.", "body_md": "*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*\n\n🔨 #bugsmash, week by week:\n\n[a 390% CPU hour nobody noticed],[a state machine with no exit],[a backup that leaked 156 GB]. Week four is the finale: I wired monitoring into the pipeline that produced all three — and its best catch was itself.\n\nTextStack is an open-source reader for technical books, built in .NET: an ASP.NET Core API, a background Worker, PostgreSQL + pgvector, React on top. The LLM pipeline does translation, word explanations, \"Ask this book\" RAG, and three production agents (Enrichment, Librarian, Tutor), routed between a local Ollama and OpenAI by a config-driven router. The code is public: [github.com/mrviduus/textstack](https://github.com/mrviduus/textstack).\n\nThree weeks ago a user's PDF fell through my LLM router onto a CPU-only Ollama container instead of GPT-4.1, and my CPU sat at 390% for an hour. Zero exceptions. Zero error logs. Zero alerts. And when I went to see what my existing observability had recorded, the answer was *nothing at all*: the OTLP exporter pointed at an Aspire dashboard container that is profile-gated and doesn't run in production. Every span my services had ever produced in prod had been fired into a closed socket.\n\n**Observability you never read is indistinguishable from observability you never installed.**\n\nThe one-line config fix was submission #1. This submission is the fix for the *class* of bug — a system that has no way to make a sound when it does the wrong thing successfully:\n\nFour PRs, all merged to main; 1,363 unit tests, full CI green:\n\n`23505`\n\non reading-progress upserts, real users losing their place in books**The router now says why.** Route resolution was a `??`\n\nchain that produced a string — *identical* whether an operator deliberately routed a task or it fell off the end onto the default. That chain doesn't just fail to record intent; it destroys it. So it returns two things now:\n\n``` js\nprivate RouteDecision ResolveRoute(string? featureTag)\n{\n    var matched = RegistryKey(featureTag) ?? ConfigRouteKey(featureTag);\n    return matched is not null\n        ? new RouteDecision(matched, RouteReason.RouteMatched)\n        : new RouteDecision(config[\"Ai:DefaultProvider\"] ?? \"openai\",\n                            RouteReason.DefaultFallback);\n}\n```\n\nEvery LLM call tags its span with `ai.task`\n\n, `ai.provider.resolved`\n\n, and `ai.provider.reason`\n\n= `route_matched | default_fallback`\n\n. \"Which model answered this, and did anyone choose it on purpose?\" is now a trace query instead of a CPU graph.\n\n**Alert arithmetic.** `pdf.parse`\n\nresolves a route once per *page* with parallelism six — my first version would have turned the original incident into 106 identical Sentry events. Every alarm goes through a throttle keyed on `(task, provider, reason)`\n\n: first hit fires immediately, then one event per hour per distinct problem. The unit test literally counts to 106 and asserts one claim.\n\n**No silent fallback, ever — in either direction.** When the breaker finds Ollama dead, tasks are skipped and stay queued; nothing auto-switches to a paid provider, because that converts an outage into unbounded spend. Provider choice stays 100% config-driven.\n\n**And the first live run found a hole in my own fix.** The startup probe opens the circuit on a one-minute backoff; the backfill worker wakes after a two-minute start delay — by then the circuit is legitimately half-open, and my single up-front gate waved the whole batch through. A per-book re-check turned 38 calls into one:\n\n```\nMetadata backfill: enriching 38 user books\nMetadata backfill: aborting after 0 enriched / 1 failed — provider 'ollama'\n  is unavailable; the remaining candidates stay queued\n```\n\nTests check what you imagined; a live run checks what's there.\n\n**Error Monitoring — what the first 24 hours in production caught:**\n\n`HTTP 429 (insufficient_quota)`\n\non `/translate`\n\nand `/explain`\n\n— the entire paid surface had been failing for readers for twelve hours. No version of my logs would have surfaced that before a user complained.`PUT /me/progress`\n\nthrew `23505: duplicate key value violates unique constraint`\n\nten times in four hours: a textbook read-then-insert race (session heartbeat + `sendBeacon`\n\non unload + second device), milliseconds wide, invisible in tests. Fixed in **Custom tags** (`ai.task`\n\n, `ai.provider`\n\n, `ai.failure`\n\n, `agent.name`\n\n, `agent.outcome`\n\n) go through an **allowlist scrubber** — every tag not explicitly blessed dies at the edge, so a future `SetTag(\"prompt\", userText)`\n\ncan never leak. A Sentry issue answers \"which feature, on which model, is broken?\" without opening a trace.\n\n**Tracing** covers agent runs and RAG indexing at 100% sampling (they're the reason I installed this), HTTP at 20%, health checks at 0%. I rejected the deprecated OTel bridge *specifically because* spans leaving through the OpenTelemetry SDK bypass `BeforeSend`\n\n— my OTel pipeline carries raw client IPs and full SQL text that must never leave the box. A tiny `TraceScope`\n\ndual-writes an `Activity`\n\nand a Sentry span instead, so everything Sentry receives passes my scrubber.\n\n**Breadcrumbs caught my scrubber lying — twice.** A live event's breadcrumb trail contained SQL: EF Core interpolates the query into the breadcrumb *message*, not the structured `data`\n\nbag my scrubber nulled (and my unit tests were green the whole time, asserting exactly the wrong thing). Fixed by dropping EF command breadcrumbs outright — then production found the *same* leak in a second channel: EF logs a failed command at `Error`\n\nlevel and Sentry's `ILogger`\n\nintegration promotes it to an event, SQL in the message again. A scrubber written against one egress path will be bypassed by the next one. Both doors are closed in [#446](https://github.com/mrviduus/textstack/pull/446), and dropping loses no signal — the exception middleware already reports the same failure with the SQLSTATE and constraint name, no SQL.\n\n**Release + environment tags as forensics.** The most interesting issue of the first day showed a dead Ollama starving a metadata pipeline: thirty events, tagged `environment: Production`\n\n. I read it as an outage and started writing the fix. It was my laptop — a dev `.env`\n\nwith `ASPNETCORE_ENVIRONMENT=Production`\n\nplus the production DSN I'd pasted in to verify the integration. What broke the spell was Sentry's own metadata: `linux-arm64`\n\nruntime on an x86_64 prod, and a `release`\n\ntag pointing at a commit that had never been deployed. An environment tag is a claim a process makes about itself, not a fact. Now `SENTRY_RELEASE`\n\ncomes from the `GIT_SHA`\n\nbuild arg — every CI-built image has one, no `dotnet run`\n\never does — and a Production claim without a release gets renamed `production-unverified`\n\n([#448](https://github.com/mrviduus/textstack/pull/448)).\n\nAnd the meta-lesson that justified the whole exercise: I verified the integration by sending real events at the real DSN and *reading the captured payloads in the UI* — that's how the leaks, the inferred-geo surprise, and the middleware capture path all surfaced. A monitoring system whose first act is to indict itself is one you can start trusting.\n\nWhat's the most embarrassing thing your monitoring has ever caught — and was it in the code, or in you?\n\n*I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on my blog. github.com/mrviduus/textstack*", "url": "https://wpnews.pro/news/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline", "canonical_source": "https://dev.to/mrviduus/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline-12lo", "published_at": "2026-08-11 12:00:00+00:00", "updated_at": "2026-08-11 12:15:58.237087+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "large-language-models", "mlops"], "entities": ["Sentry", "TextStack", "Ollama", "OpenAI", "Aspire", "pgvector", "React", "mrviduus"], "alternates": {"html": "https://wpnews.pro/news/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline", "markdown": "https://wpnews.pro/news/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline.md", "text": "https://wpnews.pro/news/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline.txt", "jsonld": "https://wpnews.pro/news/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline.jsonld"}}