{"slug": "silent-failures-in-ai-agents-how-to-detect-a-wrong-answer", "title": "Silent failures in AI agents: how to detect a wrong answer", "summary": "A developer outlined a method for catching silent failures in AI agents—cases where a request returns HTTP 200 with no error span yet the answer is wrong—by tracing every agent step, separating technical errors from content errors, and attaching evaluation results to traces. The approach relies on OpenTelemetry GenAI semantic conventions, including invoke_agent and execute_tool spans, the OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT variable for capturing tool arguments and results, and the gen_ai.evaluation.result event, and notes that Dynatrace SaaS documents online LLM-as-a-judge evaluations on production traces. The writeup stresses that the specification defines how to carry an evaluation score but not who computes it, and that judge models must be calibrated on annotated cases.", "body_md": "Silent failures in AI agents are the cases where everything looks successful: the request returns HTTP 200, no exception is raised, no span is in error, and yet the answer is wrong, incomplete or built on a bad tool result. A classic dashboard stays green. This is the main trap when operating agents.\n\nThis article offers a method to detect a wrong answer: trace every agent step, separate technical errors from content errors, then attach an evaluation result to the trace. Attribute names come from the OpenTelemetry GenAI specification, read at the source. They apply to any OTLP backend, including **Dynatrace SaaS**. No percentage is given here, because none was read in a primary source.\n\nThe OpenTelemetry specification asks for the span status to be set to `Error` when the operation ends in an error, along with `error.type`. It also makes clear there is no fixed rule for « valid but wrong » results: whether an outcome counts as an error depends on context. For an agent, a well-formed but wrong answer therefore raises no error signal by default.\n\nCommon shapes of silent failure, to adapt to your agent:\n\nNone of these raises an HTTP error. For the cost side of an agent loop, see [the token cost of an AI agent](https://span.observer/en/ai-agent-cost-tokens-opentelemetry/); the signals are the same spans.\n\nThe GenAI conventions define agent spans, with *Development* status (so subject to change). The invocation span is named `invoke_agent {gen_ai.agent.name}`: kind `CLIENT` for a remote agent, `INTERNAL` when it runs in the same process. Each tool call produces an `execute_tool {gen_ai.tool.name}` span of kind `INTERNAL`, with `gen_ai.tool.name` required.\n\nThis structure answers the first diagnostic question: which step went wrong? An `execute_tool` span with an empty result points at the tool. An inference span with an unexpected `gen_ai.response.finish_reasons` value (a recommended attribute, an array of reasons the model stopped) points at the model. To link these spans across services, the [OpenTelemetry Collector](https://span.observer/en/opentelemetry-collector-kubernetes-dynatrace/) remains the usual entry point.\n\nTo judge an answer, you must see it. The specification provides the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT` variable, with values `NO_CONTENT` (default), `SPAN_ONLY`, `EVENT_ONLY` and `SPAN_AND_EVENT`. An unrecognised value falls back to `NO_CONTENT`.\n\nTool arguments and results (`gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`) are *Opt-In*: the documentation warns that some attributes may contain sensitive information. Enable capture on a sample, with personal data masked, rather than everywhere. Tool-side abuse detection is covered in the article on [MCP tool poisoning](https://span.observer/en/mcp-tool-poisoning-detection/).\n\nThe specification defines the `gen_ai.evaluation.result` event (*Development* status, *Recommended* level). It carries the result of evaluating model output for quality, accuracy or other characteristics. It should be parented to the evaluated span when possible; otherwise set `gen_ai.response.id`.\n\nThe evaluation name (`gen_ai.evaluation.name`, for example `Relevance`) is required. The numeric score (` gen_ai.evaluation.score.value`) and the label (` gen_ai.evaluation.score.label`) are required « if applicable »; the label must stay low-cardinality. If the evaluator itself fails, `error.type` says so: a broken evaluator is not an answer judged correct.\n\nWho computes the score? The specification does not say: it only describes how to carry it. A deterministic check (does the answer cite a retrieved source? is the JSON valid?), human review on a sample or a second model are all possible. A judge model is itself fallible and must be calibrated on annotated cases.\n\nOn Dynatrace SaaS, the « Dynatrace for AI Observability » page states, as read in the documentation, online LLM-as-a-judge evaluations on production traces, with scores written as business events linked to the source traces, and OpenTelemetry ingestion using the GenAI conventions. No equivalence with the attributes above is claimed: check in your tenant.\n\n`invoke_agent` and `execute_tool`; check the names actually produced, as the specification is in development.`gen_ai.response.id`.\nTo see why a raw failure rate is not enough, read the [difference between observability and monitoring](https://span.observer/en/observability-vs-monitoring-difference/). And if trace volume gets heavy, [tail sampling](https://span.observer/en/opentelemetry-tail-sampling-dynatrace/) can keep the traces with bad evaluations first.\n\nReplay a request whose expected answer is known, forcing a tool failure (a tool returning an empty list). The trace should show the `execute_tool` span, and the evaluation should come out as a failure. If it stays « pass », the criterion is too loose. If no event arrives, check the link to the span or the `gen_ai.response.id`.\n\n`gen_ai.evaluation.explanation`) for human review.` gen_ai.agent.version`) to compare before and after a prompt change.\nIt is a run that ends without a technical error but whose result is wrong or unusable. Spans carry no error status, hence the need to evaluate the content.\n\nNo: the pages read show *Development* status for agent spans, the evaluation event and most attributes. Names may change; only `error.type` is marked stable there.\n\nNo. Capture is off by default (`NO_CONTENT`) and content can be sensitive. Enable it on a sample and protect access.\n\nDetecting a wrong answer needs a signal that availability metrics do not provide: an evaluation of the content, attached to the trace. OpenTelemetry offers a format to carry it; the quality criterion remains for the team to define. Next, see the [guardrails for AI agents in production](https://span.observer/en/ai-agent-guardrails-production/).\n\nSources: [semantic-conventions-genai repository (Apache 2.0)](https://github.com/open-telemetry/semantic-conventions-genai), [OpenTelemetry: Recording errors](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/), [Dynatrace documentation (SaaS)](https://docs.dynatrace.com/docs/observe/dynatrace-for-ai-observability).\n\n*This is an independent blog, not affiliated with Dynatrace LLC. Product names belong to their owners. The conventions cited are in development: check the specification version before relying on them.*\n\n*Originally published on [span.observer](https://span.observer/en/ai-agent-silent-failures-detect/). French version: [https://span.observer/fr/echecs-silencieux-agents-ia-detecter/](https://span.observer/fr/echecs-silencieux-agents-ia-detecter/).*", "url": "https://wpnews.pro/news/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer", "canonical_source": "https://dev.to/spanobserver/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer-1l1m", "published_at": "2026-11-10 11:30:14+00:00", "updated_at": "2026-11-10 11:51:50.647190+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "developer-tools"], "entities": ["OpenTelemetry", "Dynatrace", "OpenTelemetry Collector"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer", "markdown": "https://wpnews.pro/news/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer.md", "text": "https://wpnews.pro/news/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer.txt", "jsonld": "https://wpnews.pro/news/silent-failures-in-ai-agents-how-to-detect-a-wrong-answer.jsonld"}}