{"slug": "ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz", "title": "ur rag can return 200 and still be cooked — how we built Goose on SigNoz", "summary": "At Agents of SigNoz, a team built Goose, an observability tool that catches silent AI quality failures in RAG systems by monitoring quality scores and tool output validity, alerting when quality drops, and using an MCP agent to auto-fix retrieval breaks. The project, demonstrated at HackRx RAG 6.0 where the team placed in the top 30, addresses the gap where standard RED metrics remain green even when answers become wrong.", "body_md": "ok so real talk. your latency graph looks clean. error rate is chilling. every `/chat`\n\nis HTTP 200.\n\nand the answers are just… wrong. like confidently wrong. mid.\n\nthat’s the thing that lowkey haunted us after **HackRx RAG 6.0**. like ~** 1000 teams**, we somehow landed **top 30**, we were deep in evals / retrieval / “is this grounded tho”, and the whole time our “monitoring” was basically vibes + batch scripts. RED metrics (rate, errors, duration) stay green while the model yoinks the wrong KB. for a startup shipping RAG that gap is actually lethal cuz users bounce before your SLO even flinches.\n\nso for Agents of SigNoz we built **Goose** — catch silent AI quality fails, put them in SigNoz, alert on the cliff, then an MCP agent writes an RCA and can auto-fix the break. this post is what we actually did / hit, not a docs remix. you’ll see the signals, the gotchas, and a real e2e run with numbers.\n\nnormal API: wrong = 5xx or timeout. rag chatbot: wrong = success arc.\n\nbut asked category ≠ queried category. user asks a custom fact, pipeline silently pulls billing docs. or tool output is empty/invalid and the model just freestyles from “general knowledge”.\n\non hackrx that silent miss is the diff between “cool demo” and “i would never ship this”. we needed signals for **is this answer actually grounded / routed right** — not just **is the process alive**.\n\nshort version:\n\n`quality.score`\n\ntanks`POST /admin/fix-retriever`\n\nfor our injected break)track is **AI & Agent Observability**.\n\ndemo flow we used: run ~100 custom-fact questions healthy (answers deliberately weird — “we live on **Mars**” — so you can tell its grounded in our chroma store not wikipedia). then break retrieval. same questions. still 200s. scores go brrr into the floor. answers flip. signoz shows the cliff. agent investigates + can fix.\n\nrag app ships otlp into signoz. each chat turn ain’t just a request span — we slap quality attrs + metrics on it:\n\n`quality.score`\n\n— the main cliff metric`quality.entity_match`\n\n— asked category match retrieved?`tool.output_valid`\n\n— was retriever/tool output even usable`gen_ai.*`\n\n+ zone attrs (`asked`\n\n/ `queried`\n\n)silent fails are cross-signal by nature. trace can look fine (OK) while the metrics panel shows `quality.score`\n\nfalling like 1.0 → 0.2. in signoz we open the dashboard cliff then jump a span that still says OK but `tool.output_valid=false`\n\n. that combo is the whole thesis.\n\ndrop your dashboard screenshot here — quality.score healthy then cliff\n\nwe didn’t want another “cpu go up” board. ours is basically:\n\n`quality.score`\n\n`tool.output_valid`\n\nrateif volume stays up and quality dies, you know its not “the box is down”, its “the answers got dumb”.\n\nscreenshot: healthy avg ~1.0 vs broken ~0.2\n\nalert: `quality.score`\n\n**below 0.6**. channel = webhook into our agent on `:8020/webhook/alert`\n\n.\n\nthis is the bit that made it feel real. signoz isn’t just a museum of charts. its the pager for quality. same muscle memory as a classic outage page, except the outage is semantic.\n\nscreenshot: alert rule / fired alert\n\nalert fires → goose doesn’t invent lore. it talks to **signoz mcp**, grabs recent quality evidence, dumps a markdown rca (what tanked, misroute hints like asked vs queried zones, what to do).\n\nfor the hackathon remediator we call `POST /admin/fix-retriever`\n\nwhen the report / alert value smells like our injected misroute. irl you’d swap that for reindex / model rollback / feature flag whatever — pattern stays: **signoz evidence → agent decides → action**.\n\nalso being honest: auto-fix does **not** magically fix “we changed embeddings and everything drifted”. our remediator is for the routing break we demo. don’t overclaim that part, judges can smell it.\n\nscreenshot: agent report markdown\n\nhealthy traffic → break → webhook → fix. autoheal **off** on purpose so the agent had to be the one that healed it.\n\n`tool.output_valid=false`\n\n, answers drift to earth/general knowledge etc`POST /admin/fix-retriever`\n\nthis is literally the loop we wished we had during hackrx crunch. not only “did the batch score drop in a csv” but “did prod quality just cliff and can something investigate without me clicking around for 40 mins”.\n\nevery turn we roughly do this then export otlp:\n\n```\nspan.set_attribute(\"quality.score\", quality_score)\nspan.set_attribute(\"quality.entity_match\", 1.0 if asked == queried else 0.0)\nspan.set_attribute(\"tool.output_valid\", tool_output_valid)\nspan.set_attribute(\"quality.asked_zone\", asked)\nspan.set_attribute(\"quality.queried_zone\", queried)\n\nquality_score_gauge.set(quality_score)\n```\n\npoint `OTEL_EXPORTER_OTLP_ENDPOINT`\n\nat signoz otlp.\n\n**gotcha we ate so you don’t have to:** if you start the rag *before* signoz is up, otel can basically no-op / disable and then your dashboards look haunted (empty) even tho the app “works”. restart rag after signoz is healthy. we wasted time on this. not proud.\n\nanother dumb one: `localhost`\n\nvs `127.0.0.1`\n\non mac — ipv6 being spicy — our e2e died with connection refused until we pinned 127.0.0.1. tiny, but its the kind of detail you only get from actually running the loop at 3am.\n\n**worked**\n\n`quality.score`\n\nfeels correct. cpu alerts would’ve slept through this incident**didn’t / don’t lie about it**\n\nwhat i’d tell hackrx-me: instrument semantic quality on day 1. batch evals are cute. production still needs a cliff on a graph.\n\ngoose exists cuz top 30 at hackrx still left a hole — strong rag, blind when it fails quiet. for this hackathon we closed it: quality signals in, dashboards + alerts in signoz, mcp agent out with rca (+ optional fix). we’re using the same mindset for our startup rag stuff too, because “http 200 but wrong” is not a hypothetical, its the default failure mode.\n\none question for your stack: **when the answer is wrong but http is 200, what breaks on the graph?**\n\nif the answer is “nothing”… yeah. that’s the bug.", "url": "https://wpnews.pro/news/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz", "canonical_source": "https://dev.to/karthikeya_atthavelli_a64/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz-1k2g", "published_at": "2026-07-25 22:47:44+00:00", "updated_at": "2026-07-25 23:31:31.815157+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "ai-tools", "developer-tools"], "entities": ["SigNoz", "Goose", "HackRx RAG 6.0", "MCP"], "alternates": {"html": "https://wpnews.pro/news/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz", "markdown": "https://wpnews.pro/news/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz.md", "text": "https://wpnews.pro/news/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz.txt", "jsonld": "https://wpnews.pro/news/ur-rag-can-return-200-and-still-be-cooked-how-we-built-goose-on-signoz.jsonld"}}