ok so real talk. your latency graph looks clean. error rate is chilling. every /chat
is HTTP 200.
and the answers are just… wrong. like confidently wrong. mid.
that’s the thing that lowkey haunted us after HackRx RAG 6.0. like ~** 1000 teams**, we somehow landed top 30, we were deep in evals / retrieval / “is this grounded tho”, and the whole time our “monitoring” was basically vibes + batch scripts. RED metrics (rate, errors, duration) stay green while the model yoinks the wrong KB. for a startup shipping RAG that gap is actually lethal cuz users bounce before your SLO even flinches.
so for Agents of SigNoz we built Goose — catch silent AI quality fails, put them in SigNoz, alert on the cliff, then an MCP agent writes an RCA and can auto-fix the break. this post is what we actually did / hit, not a docs remix. you’ll see the signals, the gotchas, and a real e2e run with numbers.
normal API: wrong = 5xx or timeout. rag chatbot: wrong = success arc.
but asked category ≠ queried category. user asks a custom fact, pipeline silently pulls billing docs. or tool output is empty/invalid and the model just freestyles from “general knowledge”.
on hackrx that silent miss is the diff between “cool demo” and “i would never ship this”. we needed signals for is this answer actually grounded / routed right — not just is the process alive.
short version:
quality.score
tanksPOST /admin/fix-retriever
for our injected break)track is AI & Agent Observability.
demo flow we used: run ~100 custom-fact questions healthy (answers deliberately weird — “we live on Mars” — so you can tell its grounded in our chroma store not wikipedia). then break retrieval. same questions. still 200s. scores go brrr into the floor. answers flip. signoz shows the cliff. agent investigates + can fix.
rag app ships otlp into signoz. each chat turn ain’t just a request span — we slap quality attrs + metrics on it:
quality.score
— the main cliff metricquality.entity_match
— asked category match retrieved?tool.output_valid
— was retriever/tool output even usablegen_ai.*
- zone attrs (
asked
/ queried
)silent fails are cross-signal by nature. trace can look fine (OK) while the metrics panel shows quality.score
falling like 1.0 → 0.2. in signoz we open the dashboard cliff then jump a span that still says OK but tool.output_valid=false
. that combo is the whole thesis.
drop your dashboard screenshot here — quality.score healthy then cliff
we didn’t want another “cpu go up” board. ours is basically:
quality.score
tool.output_valid
rateif volume stays up and quality dies, you know its not “the box is down”, its “the answers got dumb”.
screenshot: healthy avg ~1.0 vs broken ~0.2
alert: quality.score
below 0.6. channel = webhook into our agent on :8020/webhook/alert
.
this is the bit that made it feel real. signoz isn’t just a museum of charts. its the pager for quality. same muscle memory as a classic outage page, except the outage is semantic.
screenshot: alert rule / fired alert
alert fires → goose doesn’t invent lore. it talks to signoz mcp, grabs recent quality evidence, dumps a markdown rca (what tanked, misroute hints like asked vs queried zones, what to do).
for the hackathon remediator we call POST /admin/fix-retriever
when the report / alert value smells like our injected misroute. irl you’d swap that for reindex / model rollback / feature flag whatever — pattern stays: signoz evidence → agent decides → action.
also being honest: auto-fix does not magically fix “we changed embeddings and everything drifted”. our remediator is for the routing break we demo. don’t overclaim that part, judges can smell it.
screenshot: agent report markdown
healthy traffic → break → webhook → fix. autoheal off on purpose so the agent had to be the one that healed it.
tool.output_valid=false
, answers drift to earth/general knowledge etcPOST /admin/fix-retriever
this is literally the loop we wished we had during hackrx crunch. not only “did the batch score drop in a csv” but “did prod quality just cliff and can something investigate without me clicking around for 40 mins”.
every turn we roughly do this then export otlp:
span.set_attribute("quality.score", quality_score)
span.set_attribute("quality.entity_match", 1.0 if asked == queried else 0.0)
span.set_attribute("tool.output_valid", tool_output_valid)
span.set_attribute("quality.asked_zone", asked)
span.set_attribute("quality.queried_zone", queried)
quality_score_gauge.set(quality_score)
point OTEL_EXPORTER_OTLP_ENDPOINT
at signoz otlp.
gotcha we ate so you don’t have to: if you start the rag before signoz is up, otel can basically no-op / disable and then your dashboards look haunted (empty) even tho the app “works”. restart rag after signoz is healthy. we wasted time on this. not proud.
another dumb one: localhost
vs 127.0.0.1
on mac — ipv6 being spicy — our e2e died with connection refused until we pinned 127.0.0.1. tiny, but its the kind of detail you only get from actually running the loop at 3am.
worked
quality.score
feels correct. cpu alerts would’ve slept through this incidentdidn’t / don’t lie about it
what i’d tell hackrx-me: instrument semantic quality on day 1. batch evals are cute. production still needs a cliff on a graph.
goose exists cuz top 30 at hackrx still left a hole — strong rag, blind when it fails quiet. for this hackathon we closed it: quality signals in, dashboards + alerts in signoz, mcp agent out with rca (+ optional fix). we’re using the same mindset for our startup rag stuff too, because “http 200 but wrong” is not a hypothetical, its the default failure mode.
one question for your stack: when the answer is wrong but http is 200, what breaks on the graph?
if the answer is “nothing”… yeah. that’s the bug.