{"slug": "we-made-signoz-s-llm-observability-actually-reachable", "title": "We made SigNoz's LLM observability actually reachable…", "summary": "SigNoz's LLM observability feature, shipped in v0.133.0, is nearly unreachable due to three silent configuration gates and a bug that prevents processors from being wired into pipelines. A hackathon team identified that the feature requires two flags to be enabled together, seeded registries for pricing and span mapping, and a fix for the pipeline wiring bug. The team also added measured answer quality as a missing signal.", "body_md": "-and added the one signal it was missing\n\nTrack 01 — AI & Agent Observability.A hackathon project built from scratch, July 20–26, 2026.\n\nThere's a sentence we said to each other on day one and then kept saying all week, because it kept being true:\n\nAn API 200 is not evidence that a feature works.\n\nThis post is the story of a project that lives entirely inside that sentence. It's about a feature SigNoz already shipped that almost nobody can reach, a real bug that makes part of it impossible, and a missing signal — *measured answer quality* — that we built on top. And it's about how we used SigNoz not as a black box but as the thing under test.\n\nMost \"add observability to LLMs\" projects start from zero. Ours didn't, and that's the interesting part.\n\nSigNoz **v0.133.0** already ships a complete LLM-observability product in open source: a dedicated frontend module, two custom OpenTelemetry collector processors (an LLM-pricing processor and a span-mapper processor), and CRUD APIs behind them. The machinery to ingest `gen_ai.*`\n\nspans, price them, normalize vendor-specific shapes, and render them on a dashboard is *there*, in the OSS repo, today.\n\nSo our thesis wasn't \"SigNoz is missing LLM observability.\" It was sharper and, we think, more useful:\n\nSigNoz shipped the machinery. We make the shipped feature actually reachable — and add the one signal it's missing: measured answer quality.\n\nThat framing mattered to us ethically, too. Nothing in this project reads \"SigNoz is broken.\" It reads: they built something good, three defaults and one bug stand between a self-hoster and it, and here's the fix — upstreamed.\n\nHere's what actually stands between a fresh `docker compose up`\n\nand a working LLM-observability feature. Every one of them fails *silently* — the backend returns success while the frontend hides the result, and nothing tells you why.\n\n`enable_ai_observability`\n\nis default-disabled\nThe whole feature sits behind an experimental flag that ships off. With it off, the UI route for LLM observability **silently redirects to /home**. No error, no \"coming soon,\" no toast. You click the thing and you're just... somewhere else. At the time we hit it, there were\n\n`use_dashboard_v2`\n\nis default-disabled\nThe seeded LLM dashboards are v2 dashboards. Import one over the API with this flag off and it succeeds — `200 OK`\n\n— and then renders as an empty **\"Welcome to your new dashboard\"** shell. The data is flowing. The panels are defined. The screen is blank.\n\nThis is where the day-one sentence was born. Both gates let the backend succeed while the frontend hides the result. We got bitten by *both, in one afternoon*, before we understood the pattern.\n\n**The fix for gates 1 and 2 is to ship both flags together — and it has to be both:**\n\n```\nSIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY=true\nSIGNOZ_FLAGGER_CONFIG_BOOLEAN_USE__DASHBOARD__V2=true\n```\n\nSetting only one still gives you an empty shell — the exact failure we're fixing. So any \"dashboards-as-code\" deliverable that sets one flag is quietly reproducing the bug.\n\nTwo registries back the feature — `llm_pricing_rules`\n\n(model prices) and `span_mapper_groups`\n\n(vendor→spec mapping presets). **Both ship with zero rows.** No seeded prices, no mapping presets. And there's a nasty second-order trap here: SigNoz's own CrewAI docs tell users to instrument with OpenInference — which emits *zero* `gen_ai.*`\n\nattributes — so a user who follows the docs faithfully lands on a seeded dashboard that shows nothing, and has no way to know the mapping layer is what's missing.\n\nThe three gates are configuration. This one is a bug.\n\nSigNoz's LLM-pricing processor and span-mapper processor are generated into the collector config correctly — we verified the rendered YAML. But neither of them is ever **wired into a pipeline**:\n\n`GenerateCollectorConfigWithLLMPricingProcessor`\n\nand `GenerateCollectorConfigWithSpanMapperProcessor`\n\nboth write only `collectorConf[\"processors\"]`\n\n.`service.pipelines.traces.processors`\n\n.In OpenTelemetry Collector terms, that's *dead config*: a processor that exists but is in no pipeline never runs. So even with the flag on and a correctly-rendered `gpt-4o`\n\npricing rule, the effective traces pipeline was still `[signozspanmetrics/delta, batch]`\n\n, and ClickHouse showed **no cost attributes** — not `_signoz.gen_ai.total_cost`\n\n, not `_signoz.gen_ai.cost_input`\n\n/ `cost_output`\n\n(those are the real emitted names; there is no `gen_ai.usage.cost`\n\noutput attribute).\n\nThe good news: the codebase already does this right somewhere. The **logs** feature wires its generated processor straight into the pipeline. So the fix is small, it follows in-repo prior art, and it's the kind of change a maintainer can verify in two minutes:\n\nenable\n\n`enable_ai_observability`\n\n→ add a pricing rule → observe the effective traces pipeline is still`[signozspanmetrics/delta, batch]`\n\n→ cost attributes never appear.\n\nWe filed it upstream as ** SigNoz/signoz#12268** — a small, focused change that cites the prior-art file and ships a clean reproduction. A teammate's fix, not a hackathon drive-by.\n\nEverything above makes the shipped feature reachable. But there's a signal SigNoz's LLM product genuinely does not have, and it's the one that matters most for agents in production: **is the model's answer any good, and did the last release make it worse?**\n\nCost, latency, and token counts tell you the model *ran*. They say nothing about whether it was *right*. So we built an eval layer.\n\nOpenTelemetry specs an evaluation signal — `gen_ai.evaluation.result`\n\n(name, score, label, explanation). It has **zero references in the SigNoz codebase**. We emit it: an LLM-as-judge scores an answer, and the score travels into SigNoz over OTLP as spec-shaped telemetry.\n\nCrucially, *how* you carry it decides whether it's usable. We learned this the hard way and built a probe (`probe_eval_carriers.py`\n\n) that emits the same score three ways side by side to prove which one survives:\n\n`Array(String)`\n\n— you `GROUP BY`\n\nis not auditable.So we carry every score on the two that SigNoz can actually aggregate, never on a span event.\n\nThe judge is the load-bearing piece: every downstream claim is only as trustworthy as the score it produces. So we picked it by measuring it against our actual task, and we picked it to run **locally on open weights** — `gpt-oss:20b`\n\non Ollama, free and self-hostable.\n\nThat last choice is architecture, not thrift. This entire project exists to *remove* a gate between a self-hoster and a feature. A judge that needed a paid API key would reintroduce exactly that kind of gate — now you need a billing account before you can evaluate anything. A free, self-hostable judge works for the same person we're unblocking.\n\nWe benchmarked five candidates. The results were not what \"bigger is better\" would predict:\n\n| model | valid JSON | grounded | fabricated | separation | jitter (temp 0) | p50 latency |\n|---|---|---|---|---|---|---|\ngpt-oss:20b |\n3/3 | 1.000 | 0.000 | +1.000 |\n0.000 |\n3.16s |\n| gpt-oss:120b | 3/3 | 1.000 | 0.133 | +0.867 | 0.250 | 4.66s |\n| nemotron-3-nano:30b | flaky | 1.000 | 0.270 | +0.730 | 0.180 | 5.38s |\n| deepseek-v4-flash | — | HTTP 403 (not in tier) | ||||\n| gemma4:31b | — | returned non-JSON |\n\n**The bigger model was the noisier one.** On a repeated fabricated input at temperature 0, `gpt-oss:120b`\n\nreturned 0.25 / 0.00 / 0.15 — a 0.25 spread — while `gpt-oss:20b`\n\nreturned 0.00 every single time. A quality-regression alert lives or dies on that stability: a judge that wobbles ±0.25 makes \"quality dropped 12%\" indistinguishable from measurement noise, and can name the *wrong* release as the culprit. We chose the model our alert can actually trust, and it happened to be the smaller, faster, free one.\n\nEvery score also carries the two attributes that make it reproducible after the fact:\n\n`gen_ai.evaluation.judge.model`\n\n— which model scored it`judge.prompt.version`\n\n— which rubric version scored itSo \"which judge and which rubric produced this score?\" is a `GROUP BY`\n\n, not a shrug.\n\nThe payoff is a threshold alert: `avg(gen_ai.evaluation.score.value)`\n\nbelow `0.7`\n\n, over a 30-minute window, **grouped by service.version**. When a release's answer quality slides under the floor, the alert fires\n\nWe measured the full loop end to end against a live instance, not estimated it:\n\n`16:21:43Z`\n\n→ alert transitioned to `16:25:40Z`\n\nAnd here's the part that matters for anyone building on this: that time is almost entirely SigNoz's own machinery — the 120s `eval_delay`\n\nplus the 1-minute evaluation interval. The judge adds only ~2.5s per call, and because spans are timestamped when the run *starts*, even that overlaps the `eval_delay`\n\nrather than stacking on it. **The eval layer adds essentially nothing to detection time; SigNoz's evaluation interval is the floor.**\n\nFor the demo we use a deliberately loud drop — measured mean **0.958 → 0.167** — so the mechanism is legible in one glance. That's a demo choice, not a limitation: the alert fires on the same 0.7 floor whether the slide is a cliff or a subtle crossing.\n\nWe ran SigNoz self-hosted in WSL2 (Ubuntu 24.04) on native Docker Engine — *not* Docker Desktop, whose ClickHouse Keeper is known to segfault. Deployment is via **SigNoz Foundry** (`foundryctl`\n\n), their \"one config, one command\" tool: you declare the whole deployment in a `casting.yaml`\n\n, `forge`\n\nexpands it into platform manifests plus a checksum-pinned `casting.yaml.lock`\n\n, and `cast`\n\ndeploys it.\n\nBecause the judges of this hackathon may re-run Foundry to reproduce our deployment, we didn't hand-edit generated files — we put the fix *in the source of truth*. Our `casting.yaml`\n\nsets both gates as first-class env overrides on the SigNoz service:\n\n```\nspec:\n  deployment:\n    flavor: compose\n    mode: docker\n  signoz:\n    spec:\n      version: v0.133.0\n      image: signoz/signoz:v0.133.0\n      env:\n        SIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY: \"true\"\n        SIGNOZ_FLAGGER_CONFIG_BOOLEAN_USE__DASHBOARD__V2: \"true\"\n```\n\nWe verified this is the right way to do it: `forge`\n\n*merges* those overrides into the defaults (it doesn't wipe the DB/store env), the flags render into the generated `compose.yaml`\n\n, and — critically — **forge is deterministic**: re-running it produces a byte-identical lock. So a judge who clones the repo and runs `foundryctl forge && foundryctl cast`\n\ngets the *feature-reachable* deployment, not the default empty shell. The casting file is part of the argument, not a checkbox.\n\n`GROUP BY`\n\n. Know your storage model before you emit.SigNoz built a genuinely good LLM-observability product. We made it reachable, filed the bug that was blocking cost, and added the measured-quality signal it was missing — all in OSS, all reproducible from one `casting.yaml`\n\n.\n\n*Repo (with casting.yaml + casting.yaml.lock for reproduction): *", "url": "https://wpnews.pro/news/we-made-signoz-s-llm-observability-actually-reachable", "canonical_source": "https://dev.to/gajanand_vdhayagode_55da/we-made-signozs-llm-observability-actually-reachable-ldk", "published_at": "2026-07-26 17:01:23+00:00", "updated_at": "2026-07-26 17:30:41.083788+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["SigNoz", "OpenTelemetry", "OpenInference", "CrewAI"], "alternates": {"html": "https://wpnews.pro/news/we-made-signoz-s-llm-observability-actually-reachable", "markdown": "https://wpnews.pro/news/we-made-signoz-s-llm-observability-actually-reachable.md", "text": "https://wpnews.pro/news/we-made-signoz-s-llm-observability-actually-reachable.txt", "jsonld": "https://wpnews.pro/news/we-made-signoz-s-llm-observability-actually-reachable.jsonld"}}