Manual security code review costs roughly $85/hour of engineer time and doesn't scale with commit velocity. Most AI code-scanners stop at "here's a vulnerability" — they don't verify their own fixes, and they're completely opaque about what they're actually doing under the hood.
DevGuard AI is my attempt at fixing both problems for the "Agents of SigNoz" hackathon: an autonomous pipeline that detects vulnerabilities, patches them, adversarially reviews its own work, and — critically — observes itself through SigNoz closely enough to change its own behavior mid-request.
Request flow: Browser → POST /scan → FastAPI backend → circuit-breaker-wrapped pipeline → Scanner Agent (RAG-augmented) → Fixer/Validator reflection loop (up to 3 attempts) → response.
Three agents do the actual security work:
If the Validator rejects a fix, its feedback goes straight back into the Fixer's next prompt. This loops up to 3 times before the pipeline gives up and returns its best attempt.
Every one of those steps — Scanner, each Fixer/Validator retry, the circuit breaker's state transitions — is wrapped in an OpenTelemetry span via a custom @traced
decorator. Open a trace for a single scan in SigNoz and you see the entire reflection loop as a flame graph: which attempt failed, how long each agent took, and exactly where an LLM call retried after a transient failure.
On top of tracing, I built custom OTel metrics feeding a "DevGuard AI Command Center" dashboard in SigNoz:
devguard.llm.tokens_total
/ devguard.llm.cost_total
— real token and cost accounting per scan, tagged by agent and modeldevguard.scan.latency
— a histogram driving p50/p95/p99 panelsdevguard.cache.hit_total
/ miss_total
— Redis cache efficiencydevguard.circuit_breaker.state_changes_total
— how often the resilience layer tripsI also configured three SigNoz Alert Rules so the pipeline monitors itself in production terms, not just in a dashboard someone has to remember to check:
This is the part I'm most excited about. Most "observability for AI agents" projects are one-directional — the agent runs, SigNoz watches. DevGuard closes the loop: the pipeline reads its own recent telemetry back out through SigNoz's MCP server before making key decisions, via a small mcp_client.py
that speaks the real MCP protocol (streamable-HTTP, authenticated) against SigNoz's own MCP server — confirmed working with a live session.initialize()
list_tools()handshake returning real SigNoz tool names.
Two adaptations run off that telemetry today:
Every adaptation is (a) returned in the API response so the frontend can show it, and (b) stamped onto the active OpenTelemetry span as an attribute — so a routing override is visible both to the end user and inside the SigNoz trace that produced it. And the whole layer is deliberately fail-safe: if SigNoz or its MCP server is unreachable, every function degrades to "behave exactly as if this layer didn't exist" rather than blocking a scan.
A hand-rolled circuit breaker (CLOSED/OPEN/HALF_OPEN) sits between the pipeline and the LLM provider. If the provider starts failing, the breaker trips, the pipeline falls back to a cheaper model automatically, and a state transition event gets logged onto the active span — visible directly in the SigNoz trace, not buried in application logs.
Building this for the hackathon reinforced something I didn't fully appreciate going in: instrumenting an AI pipeline for observability is the easy 80%. The hard 20% — and the actually interesting part — is closing the loop so the system can act on what it's observing about itself, without that self-observation ever being allowed to break the thing it's trying to help.
Repo: github.com/akashbichukale0111/devguard-ai Built for the SigNoz "Agents of SigNoz" hackathon by WeMakeDevs.