# Silent failures in AI agents: how to detect a wrong answer > Source: > Published: 2026-11-10 11:30:14+00:00 Silent failures in AI agents are the cases where everything looks successful: the request returns HTTP 200, no exception is raised, no span is in error, and yet the answer is wrong, incomplete or built on a bad tool result. A classic dashboard stays green. This is the main trap when operating agents. This article offers a method to detect a wrong answer: trace every agent step, separate technical errors from content errors, then attach an evaluation result to the trace. Attribute names come from the OpenTelemetry GenAI specification, read at the source. They apply to any OTLP backend, including **Dynatrace SaaS**. No percentage is given here, because none was read in a primary source. The OpenTelemetry specification asks for the span status to be set to `Error` when the operation ends in an error, along with `error.type`. It also makes clear there is no fixed rule for « valid but wrong » results: whether an outcome counts as an error depends on context. For an agent, a well-formed but wrong answer therefore raises no error signal by default. Common shapes of silent failure, to adapt to your agent: None of these raises an HTTP error. For the cost side of an agent loop, see [the token cost of an AI agent](https://span.observer/en/ai-agent-cost-tokens-opentelemetry/); the signals are the same spans. The GenAI conventions define agent spans, with *Development* status (so subject to change). The invocation span is named `invoke_agent {gen_ai.agent.name}`: kind `CLIENT` for a remote agent, `INTERNAL` when it runs in the same process. Each tool call produces an `execute_tool {gen_ai.tool.name}` span of kind `INTERNAL`, with `gen_ai.tool.name` required. This structure answers the first diagnostic question: which step went wrong? An `execute_tool` span with an empty result points at the tool. An inference span with an unexpected `gen_ai.response.finish_reasons` value (a recommended attribute, an array of reasons the model stopped) points at the model. To link these spans across services, the [OpenTelemetry Collector](https://span.observer/en/opentelemetry-collector-kubernetes-dynatrace/) remains the usual entry point. To judge an answer, you must see it. The specification provides the `OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT` variable, with values `NO_CONTENT` (default), `SPAN_ONLY`, `EVENT_ONLY` and `SPAN_AND_EVENT`. An unrecognised value falls back to `NO_CONTENT`. Tool arguments and results (`gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`) are *Opt-In*: the documentation warns that some attributes may contain sensitive information. Enable capture on a sample, with personal data masked, rather than everywhere. Tool-side abuse detection is covered in the article on [MCP tool poisoning](https://span.observer/en/mcp-tool-poisoning-detection/). The specification defines the `gen_ai.evaluation.result` event (*Development* status, *Recommended* level). It carries the result of evaluating model output for quality, accuracy or other characteristics. It should be parented to the evaluated span when possible; otherwise set `gen_ai.response.id`. The evaluation name (`gen_ai.evaluation.name`, for example `Relevance`) is required. The numeric score (` gen_ai.evaluation.score.value`) and the label (` gen_ai.evaluation.score.label`) are required « if applicable »; the label must stay low-cardinality. If the evaluator itself fails, `error.type` says so: a broken evaluator is not an answer judged correct. Who computes the score? The specification does not say: it only describes how to carry it. A deterministic check (does the answer cite a retrieved source? is the JSON valid?), human review on a sample or a second model are all possible. A judge model is itself fallible and must be calibrated on annotated cases. On Dynatrace SaaS, the « Dynatrace for AI Observability » page states, as read in the documentation, online LLM-as-a-judge evaluations on production traces, with scores written as business events linked to the source traces, and OpenTelemetry ingestion using the GenAI conventions. No equivalence with the attributes above is claimed: check in your tenant. `invoke_agent` and `execute_tool`; check the names actually produced, as the specification is in development.`gen_ai.response.id`. To see why a raw failure rate is not enough, read the [difference between observability and monitoring](https://span.observer/en/observability-vs-monitoring-difference/). And if trace volume gets heavy, [tail sampling](https://span.observer/en/opentelemetry-tail-sampling-dynatrace/) can keep the traces with bad evaluations first. Replay a request whose expected answer is known, forcing a tool failure (a tool returning an empty list). The trace should show the `execute_tool` span, and the evaluation should come out as a failure. If it stays « pass », the criterion is too loose. If no event arrives, check the link to the span or the `gen_ai.response.id`. `gen_ai.evaluation.explanation`) for human review.` gen_ai.agent.version`) to compare before and after a prompt change. It is a run that ends without a technical error but whose result is wrong or unusable. Spans carry no error status, hence the need to evaluate the content. No: the pages read show *Development* status for agent spans, the evaluation event and most attributes. Names may change; only `error.type` is marked stable there. No. Capture is off by default (`NO_CONTENT`) and content can be sensitive. Enable it on a sample and protect access. Detecting a wrong answer needs a signal that availability metrics do not provide: an evaluation of the content, attached to the trace. OpenTelemetry offers a format to carry it; the quality criterion remains for the team to define. Next, see the [guardrails for AI agents in production](https://span.observer/en/ai-agent-guardrails-production/). Sources: [semantic-conventions-genai repository (Apache 2.0)](https://github.com/open-telemetry/semantic-conventions-genai), [OpenTelemetry: Recording errors](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/), [Dynatrace documentation (SaaS)](https://docs.dynatrace.com/docs/observe/dynatrace-for-ai-observability). *This is an independent blog, not affiliated with Dynatrace LLC. Product names belong to their owners. The conventions cited are in development: check the specification version before relying on them.* *Originally published on [span.observer](https://span.observer/en/ai-agent-silent-failures-detect/). French version: [https://span.observer/fr/echecs-silencieux-agents-ia-detecter/](https://span.observer/fr/echecs-silencieux-agents-ia-detecter/).*