{"slug": "ai-root-cause-analysis-architecture-testing-and-evaluation", "title": "AI Root Cause Analysis: Architecture, Testing, and Evaluation", "summary": "A new technical post details the architecture of an AI Root Cause Analysis system that connects to an existing LGTM observability stack via MCP and uses LangChain4j @RegisterAiService interfaces backed by a roughly 290-line system prompt to correlate traces, logs, and metrics. The system exposes five tools to the LLM, including provideLastNTraceIds, traceById, logsWithTraceId, getRootSpanStartTime, and getAllMetricsForDatetime, and a StripMcpClient decorator filters telemetry to cut Prometheus metrics from 586 entries to about 35 to stay within context limits. The project tests analysis quality with chaos engineering across 11 chaos types exposed via GET /chaos?type={type}&intensity={value}, scored with LLM-as-judge evaluation.", "body_md": "In the [introductory post](https://quarkus.io/blog/ai-root-cause-analysis/), we showed what AI Root Cause Analysis does and how to get started. This post goes deeper — into the architecture, the analysis pipeline, how we keep telemetry data within LLM context limits, and how we test analysis quality using chaos engineering with LLM-as-judge evaluation.\n\nThe system has a layered architecture connecting monitored applications to the AI analysis engine through an observability stack:\n\nYour applications just need OpenTelemetry instrumentation — no code changes required. The AI module connects to your existing LGTM stack via MCP and optionally to your running Quarkus applications via Quarkus Dev MCP for source examination and dashboard generation.\n\nThe AI module uses [LangChain4j](https://docs.langchain4j.dev/) `@RegisterAiService` interfaces backed by a ~290-line system prompt that instructs the LLM on three-way telemetry correlation, chaos detection patterns, severity classification, and structured output formatting.\n\nFive tools are exposed to the LLM during analysis:\n\n| Tool | What It Provides | \n|---|---|\n| `provideLastNTraceIds` | Recent trace IDs from Tempo (TraceQL) | \n| `traceById` | Full trace data for a specific trace | \n| `logsWithTraceId` | Correlated Loki logs for a trace | \n| `getRootSpanStartTime` | Timestamp for metric queries | \n| `getAllMetricsForDatetime` | Prometheus metrics at a point in time | \n\nThe LLM orchestrates these tools autonomously — it decides which traces look interesting, pulls their logs, checks metrics, and builds the analysis iteratively.\n\nWhen you hit `GET /analyze/{n}`, the following sequence unfolds:\n\n**Tool dispatch** — The AI module sends the system prompt and tool definitions to the LLM\n\n**Trace retrieval** — The LLM calls `provideLastNTraceIds` to get the N most recent traces\n\n**Per-trace investigation** — For each trace, the LLM pulls full trace data and correlated logs through `StripMcpClient`, which strips SDK noise, localhost attributes, and redundant fields\n\n**Metrics context** — The LLM queries Prometheus metrics at the root span’s start time; the filtering layer removes Netty/OTel internals, static counters, and histogram bucket breakdowns\n\n**Analysis synthesis** — The LLM produces a structured report with per-trace sections covering root cause, slowest operations, error patterns, and cross-trace correlation\n\nIf source examination or dashboard generation is enabled, a post-analysis pipeline kicks in using `DevMcpAiService` — a separate `@RegisterAiService` that uses Dev MCP tools to read application source code and write Grafana dashboard JSON back to each workspace.\n\nOne of the trickiest engineering challenges is keeping telemetry data within LLM context limits while preserving signal. The `StripMcpClient` decorator applies per-tool filtering functions:\n\n**Traces**: Removes SDK metadata (`telemetry.sdk.`, `otel.`), localhost attributes, and instrumentation library details\n\n**Logs**: Strips SDK metadata, redundant fields, and duplicate information already present in trace spans\n\n**Metrics**: Removes Netty internals, static/constant values, histogram bucket breakdowns, and per-region metric breakdowns — reducing 586 entries to ~35\n\nThe system prompt also instructs the LLM to treat chaos-related log entries as authoritative evidence, even when Prometheus metrics don’t confirm them — because point-in-time metric snapshots can miss transient pressure spikes that logs capture.\n\nHow do you test an AI-powered analysis tool? You need realistic failure scenarios with known root causes, and you need an objective way to score the analysis quality. AI Root Cause Analysis solves this with chaos engineering + LLM-as-judge evaluation.\n\nThe project includes companion test applications that simulate distributed microservices with configurable chaos failure modes. These expose 11 chaos types via `GET /chaos?type={type}&intensity={value}`:\n\n| Chaos Type | What It Does | \n|---|---|\n| `delay` | Thread.sleep for configurable milliseconds | \n| `memory` | Allocate N MB (released after request) | \n| `cpu` | CPU burn loop for N ms | \n| `leak` | Allocate N MB (never freed) | \n| `error` | Random 5xx WebApplicationException | \n| `exception` | Unhandled RuntimeException (HTTP 500) | \n| `threadpool` | Block 10 threads via CountDownLatch | \n| `contention` | 10 threads competing for a synchronized lock | \n| `gc` | Rapid alloc/dealloc → GC pressure | \n| `intermittent` | Random failures at configurable rate | \n| `deadlock` | Two threads deadlocked on competing locks | \n\nBeyond application-level chaos, the test suite also covers infrastructure-level failures using [**Toxiproxy**](https://github.com/Shopify/toxiproxy) — a TCP proxy that can inject latency, cut connections, and throttle bandwidth at the network layer:\n\n**Database chaos** routes JDBC connections through Toxiproxy to MySQL:\n\nTests include slow queries (3s latency injection), connection pool exhaustion (4s latency + small pool + concurrent requests), and full database outages (connection cut). JDBC telemetry creates separate database query spans, so the AI sees `GET /poke (3019ms) → SELECT … (3015ms)` and can identify database-level root causes.\n\n**Weather API chaos** routes REST client calls through Toxiproxy to an external API:\n\nTests include slow external API responses and full API outages.\n\nNo application code changes are required — chaos is injected purely at the network layer via the Toxiproxy REST API.\n\nEach integration test follows the same pattern:\n\n**Inject chaos** into the test application (or do nothing for baseline tests)\n\n**Wait** for telemetry to propagate to LGTM\n\n**Run analysis** via `GET /analyze/{n}`\n\n**Capture** all tool outputs (raw telemetry data the LLM saw)\n\n**Score** the analysis using a separate LLM call\n\nThe scoring uses an `AnalysisEvaluationStrategy` that builds a detailed evaluation prompt containing:\n\nThe original telemetry data (traces, logs, metrics) the AI analyzed\n\nThe system prompt used for analysis\n\nThe analysis output being evaluated\n\nScenario-specific evaluation criteria\n\nA judge LLM (`EvaluationJudge`) then scores the analysis on four dimensions, each worth 0-25 points:\n\n| Dimension | What It Measures | \n|---|---|\n| **Completeness** | Did it find all the issues present in the telemetry? | \n| **Accuracy** | Are the identified root causes correct? | \n| **Correlation Quality** | Did it properly link traces, logs, and metrics? | \n| **Actionability** | Are the remediation suggestions useful and specific? | \n\nThe passing threshold defaults to **70/100**. Set `EVALUATION_THRESHOLD` to override it for any test script:\n\n```\nexport EVALUATION_THRESHOLD=80\n./run-integration-test.sh chaos                    # chaos tests with threshold 80\n./run-integration-test.sh db grok grok             # DB tests with threshold 80\n```\n\nIf the score falls below the threshold, the evaluation also outputs **prompt improvement suggestions** — concrete changes to the system prompt that would improve analysis quality. This creates a feedback loop: run tests, get scores, improve the prompt, run tests again.\n\nA critical design choice is that the **analyzer LLM** and the **scorer LLM** can be different providers. This prevents self-evaluation bias — an LLM scoring its own output might be systematically lenient.\n\nThe test scripts support all combinations:\n\n```\n./run-integration-test.sh chaos openai grok     # AI=OpenAI, scorer=Grok\n./run-integration-test.sh chaos grok openai     # AI=Grok, scorer=OpenAI\n./run-integration-test.sh db openai grok        # DB tests, cross-provider\n./run-integration-test.sh weather grok grok     # Weather tests, same provider\n```\n\nAll 19 test scenarios pass across provider combinations:\n\n| AI Provider | Scorer | Chaos | DB | Weather | \n|---|---|---|---|---|\n| OpenAI | Grok | 12/12 | 4/4 | 3/3 | \n| Grok | OpenAI | 12/12 | 4/4 | 3/3 | \n| OpenAI | OpenAI | 12/12 | 4/4 | 3/3 | \n| Grok | Grok | 12/12 | 4/4 | 3/3 | \n\nThe cross-provider combinations (OpenAI/Grok, Grok/OpenAI) are the most important — they validate that analysis quality is genuine, not an artifact of one LLM being generous about its own or a peer’s output.\n\nThe project is designed for extensibility — adding a new LLM provider is a Maven profile + Quarkus configuration, adding new chaos types is a method in the test application’s chaos endpoint, and the evaluation framework automatically validates any changes against the full test matrix.\n\nAI Root Cause Analysis is a [Quarkiverse project](https://github.com/quarkiverse/quarkus-ai-rca) — contributions, feedback, and testing across different application architectures are welcome.", "url": "https://wpnews.pro/news/ai-root-cause-analysis-architecture-testing-and-evaluation", "canonical_source": "https://quarkus.io/blog/ai-root-cause-analysis-deep-dive/", "published_at": "2026-11-17 00:00:00+00:00", "updated_at": "2026-11-17 20:30:51.430979+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models", "mlops", "ai-tools"], "entities": ["LangChain4j", "Quarkus", "OpenTelemetry", "Grafana", "Prometheus", "Loki", "Tempo", "MCP"], "also_reported_by": [], "takeaway": "A StripMcpClient decorator with per-tool filtering cuts Prometheus metric context from 586 entries to about 35 while preserving signal, and chaos engineering with LLM-as-judge scoring provides an objective way to test analysis quality.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/ai-root-cause-analysis-architecture-testing-and-evaluation", "markdown": "https://wpnews.pro/news/ai-root-cause-analysis-architecture-testing-and-evaluation.md", "text": "https://wpnews.pro/news/ai-root-cause-analysis-architecture-testing-and-evaluation.txt", "jsonld": "https://wpnews.pro/news/ai-root-cause-analysis-architecture-testing-and-evaluation.jsonld"}}