ur rag can return 200 and still be cooked — how we built Goose on SigNoz

At Agents of SigNoz, a team built Goose, an observability tool that catches silent AI quality failures in RAG systems by monitoring quality scores and tool output validity, alerting when quality drops, and using an MCP agent to auto-fix retrieval breaks. The project, demonstrated at HackRx RAG 6.0 where the team placed in the top 30, addresses the gap where standard RED metrics remain green even when answers become wrong.

ok so real talk. your latency graph looks clean. error rate is chilling. every /chat is HTTP 200. and the answers are just… wrong. like confidently wrong. mid. that’s the thing that lowkey haunted us after HackRx RAG 6.0 . like ~ 1000 teams , we somehow landed top 30 , we were deep in evals / retrieval / “is this grounded tho”, and the whole time our “monitoring” was basically vibes + batch scripts. RED metrics rate, errors, duration stay green while the model yoinks the wrong KB. for a startup shipping RAG that gap is actually lethal cuz users bounce before your SLO even flinches. so for Agents of SigNoz we built Goose — catch silent AI quality fails, put them in SigNoz, alert on the cliff, then an MCP agent writes an RCA and can auto-fix the break. this post is what we actually did / hit, not a docs remix. you’ll see the signals, the gotchas, and a real e2e run with numbers. normal API: wrong = 5xx or timeout. rag chatbot: wrong = success arc. but asked category ≠ queried category. user asks a custom fact, pipeline silently pulls billing docs. or tool output is empty/invalid and the model just freestyles from “general knowledge”. on hackrx that silent miss is the diff between “cool demo” and “i would never ship this”. we needed signals for is this answer actually grounded / routed right — not just is the process alive . short version: quality.score tanks POST /admin/fix-retriever for our injected break track is AI & Agent Observability . demo flow we used: run ~100 custom-fact questions healthy answers deliberately weird — “we live on Mars ” — so you can tell its grounded in our chroma store not wikipedia . then break retrieval. same questions. still 200s. scores go brrr into the floor. answers flip. signoz shows the cliff. agent investigates + can fix. rag app ships otlp into signoz. each chat turn ain’t just a request span — we slap quality attrs + metrics on it: quality.score — the main cliff metric quality.entity match — asked category match retrieved? tool.output valid — was retriever/tool output even usable gen ai. + zone attrs asked / queried silent fails are cross-signal by nature. trace can look fine OK while the metrics panel shows quality.score falling like 1.0 → 0.2. in signoz we open the dashboard cliff then jump a span that still says OK but tool.output valid=false . that combo is the whole thesis. drop your dashboard screenshot here — quality.score healthy then cliff we didn’t want another “cpu go up” board. ours is basically: quality.score tool.output valid rateif volume stays up and quality dies, you know its not “the box is down”, its “the answers got dumb”. screenshot: healthy avg ~1.0 vs broken ~0.2 alert: quality.score below 0.6 . channel = webhook into our agent on :8020/webhook/alert . this is the bit that made it feel real. signoz isn’t just a museum of charts. its the pager for quality. same muscle memory as a classic outage page, except the outage is semantic. screenshot: alert rule / fired alert alert fires → goose doesn’t invent lore. it talks to signoz mcp , grabs recent quality evidence, dumps a markdown rca what tanked, misroute hints like asked vs queried zones, what to do . for the hackathon remediator we call POST /admin/fix-retriever when the report / alert value smells like our injected misroute. irl you’d swap that for reindex / model rollback / feature flag whatever — pattern stays: signoz evidence → agent decides → action . also being honest: auto-fix does not magically fix “we changed embeddings and everything drifted”. our remediator is for the routing break we demo. don’t overclaim that part, judges can smell it. screenshot: agent report markdown healthy traffic → break → webhook → fix. autoheal off on purpose so the agent had to be the one that healed it. tool.output valid=false , answers drift to earth/general knowledge etc POST /admin/fix-retriever this is literally the loop we wished we had during hackrx crunch. not only “did the batch score drop in a csv” but “did prod quality just cliff and can something investigate without me clicking around for 40 mins”. every turn we roughly do this then export otlp: span.set attribute "quality.score", quality score span.set attribute "quality.entity match", 1.0 if asked == queried else 0.0 span.set attribute "tool.output valid", tool output valid span.set attribute "quality.asked zone", asked span.set attribute "quality.queried zone", queried quality score gauge.set quality score point OTEL EXPORTER OTLP ENDPOINT at signoz otlp. gotcha we ate so you don’t have to: if you start the rag before signoz is up, otel can basically no-op / disable and then your dashboards look haunted empty even tho the app “works”. restart rag after signoz is healthy. we wasted time on this. not proud. another dumb one: localhost vs 127.0.0.1 on mac — ipv6 being spicy — our e2e died with connection refused until we pinned 127.0.0.1. tiny, but its the kind of detail you only get from actually running the loop at 3am. worked quality.score feels correct. cpu alerts would’ve slept through this incident didn’t / don’t lie about it what i’d tell hackrx-me: instrument semantic quality on day 1. batch evals are cute. production still needs a cliff on a graph. goose exists cuz top 30 at hackrx still left a hole — strong rag, blind when it fails quiet. for this hackathon we closed it: quality signals in, dashboards + alerts in signoz, mcp agent out with rca + optional fix . we’re using the same mindset for our startup rag stuff too, because “http 200 but wrong” is not a hypothetical, its the default failure mode. one question for your stack: when the answer is wrong but http is 200, what breaks on the graph? if the answer is “nothing”… yeah. that’s the bug.