{"slug": "signoz-for-ai-agent-observability-a-real-world-deployment", "title": "SigNoz for AI Agent Observability: A Real-World Deployment", "summary": "SigNoz v0.133.0 was deployed via Foundry for AI agent observability, but a developer discovered that the OpenInference instrumentation used by openinference-instrumentation-agno does not follow the standard gen_ai.* attribute spec, causing dashboards to remain empty without errors. The fix required manually inspecting raw span attribute keys before writing queries, and structured span attributes (e.g., arcnet.guard.checkpoint, arcnet.guard.risk_score) were added to turn guardrail verdicts into actionable telemetry, catching a hidden prompt injection during a send_email call.", "body_md": "# SigNoz for AI Agent Observability: A Real-World Deployment\n\n## The Technical Stack\n\nI used Agno for the agents, wrapped in an in-process SDK that handles OpenTelemetry instrumentation and `unplug-ai`\n\nguardrails at four specific checkpoints: input, retrieved content, tool calls, and final output. The traces flow to SigNoz via OTLP, with a FastAPI server and React UI handling the data readout.\n\nGetting SigNoz running was surprisingly painless via Foundry.\n\n```\napiVersion: v1alpha1\nkind: Installation\nmetadata:\n name: signoz\nspec:\n deployment:\n flavor: compose\n mode: docker\n signoz:\n spec:\n image: signoz/signoz:v0.133.0\nfoundryctl cast -f casting.yaml\n```\n\n## The \"Invisible Failure\" Trap\n\nHere is where I almost wasted a week: assuming the instrumentation followed the GenAI spec. I spent time drafting dashboard queries for `gen_ai.usage.input_tokens`\n\nbefore the system was even live.\n\nWhen I actually ran the agents, I discovered that `openinference-instrumentation-agno`\n\nuses OpenInference conventions, not the standard `gen_ai.*`\n\nattributes. The result? My dashboards didn't crash—they just stayed empty. Empty charts are a nightmare because they don't trigger errors; they just lie to you.\n\nThe fix was a deep dive into raw spans. I spent an evening manually verifying every attribute key in the SigNoz trace view. If you're setting up an AI workflow, send one real request and manually inspect the attribute names before you write a single query.\n\n## Turning Guardrails into Data\n\nA boolean \"true/false\" from a guardrail is useless for monitoring. You can't alert on a boolean if you don't know why it triggered. To make this observable, I started pushing structured span attributes for every verdict:\n\n```\nspan.set_attribute(\"arcnet.guard.checkpoint\", \"tool_call\")\nspan.set_attribute(\"arcnet.guard.action\", \"block\")\nspan.set_attribute(\"arcnet.guard.rule\", \"retrieved_source_in_side_effect\")\nspan.set_attribute(\"arcnet.guard.risk_score\", 0.85)\n```\n\nThis changed everything. I caught a case where an agent scraped a page containing a hidden prompt injection. The ingest scan passed, but the \"tainted\" content triggered a block when the agent tried to execute a `send_email`\n\ncall. Because I had the checkpoint as an attribute, I could see exactly where the failure happened. Without that telemetry, I would have been guessing.\n\n[Next Why I stopped obsessing over model benchmarks →](/en/threads/3783/)", "url": "https://wpnews.pro/news/signoz-for-ai-agent-observability-a-real-world-deployment", "canonical_source": "https://promptcube3.com/en/threads/3805/", "published_at": "2026-07-26 18:55:18+00:00", "updated_at": "2026-07-26 19:10:44.026417+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools", "ai-infrastructure"], "entities": ["SigNoz", "Agno", "OpenTelemetry", "OpenInference", "Foundry", "FastAPI", "React"], "alternates": {"html": "https://wpnews.pro/news/signoz-for-ai-agent-observability-a-real-world-deployment", "markdown": "https://wpnews.pro/news/signoz-for-ai-agent-observability-a-real-world-deployment.md", "text": "https://wpnews.pro/news/signoz-for-ai-agent-observability-a-real-world-deployment.txt", "jsonld": "https://wpnews.pro/news/signoz-for-ai-agent-observability-a-real-world-deployment.jsonld"}}