Silent failures in AI agents: how to detect a wrong answer

A developer outlined a method for catching silent failures in AI agents—cases where a request returns HTTP 200 with no error span yet the answer is wrong—by tracing every agent step, separating technical errors from content errors, and attaching evaluation results to traces. The approach relies on OpenTelemetry GenAI semantic conventions, including invoke_agent and execute_tool spans, the OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT variable for capturing tool arguments and results, and the gen_ai.evaluation.result event, and notes that Dynatrace SaaS documents online LLM-as-a-judge evaluations on production traces. The writeup stresses that the specification defines how to carry an evaluation score but not who computes it, and that judge models must be calibrated on annotated cases.

Silent failures in AI agents are the cases where everything looks successful: the request returns HTTP 200, no exception is raised, no span is in error, and yet the answer is wrong, incomplete or built on a bad tool result. A classic dashboard stays green. This is the main trap when operating agents. This article offers a method to detect a wrong answer: trace every agent step, separate technical errors from content errors, then attach an evaluation result to the trace. Attribute names come from the OpenTelemetry GenAI specification, read at the source. They apply to any OTLP backend, including Dynatrace SaaS . No percentage is given here, because none was read in a primary source. The OpenTelemetry specification asks for the span status to be set to Error when the operation ends in an error, along with error.type . It also makes clear there is no fixed rule for « valid but wrong » results: whether an outcome counts as an error depends on context. For an agent, a well-formed but wrong answer therefore raises no error signal by default. Common shapes of silent failure, to adapt to your agent: None of these raises an HTTP error. For the cost side of an agent loop, see the token cost of an AI agent https://span.observer/en/ai-agent-cost-tokens-opentelemetry/ ; the signals are the same spans. The GenAI conventions define agent spans, with Development status so subject to change . The invocation span is named invoke agent {gen ai.agent.name} : kind CLIENT for a remote agent, INTERNAL when it runs in the same process. Each tool call produces an execute tool {gen ai.tool.name} span of kind INTERNAL , with gen ai.tool.name required. This structure answers the first diagnostic question: which step went wrong? An execute tool span with an empty result points at the tool. An inference span with an unexpected gen ai.response.finish reasons value a recommended attribute, an array of reasons the model stopped points at the model. To link these spans across services, the OpenTelemetry Collector https://span.observer/en/opentelemetry-collector-kubernetes-dynatrace/ remains the usual entry point. To judge an answer, you must see it. The specification provides the OTEL INSTRUMENTATION GENAI CAPTURE CONTENT variable, with values NO CONTENT default , SPAN ONLY , EVENT ONLY and SPAN AND EVENT . An unrecognised value falls back to NO CONTENT . Tool arguments and results gen ai.tool.call.arguments , gen ai.tool.call.result are Opt-In : the documentation warns that some attributes may contain sensitive information. Enable capture on a sample, with personal data masked, rather than everywhere. Tool-side abuse detection is covered in the article on MCP tool poisoning https://span.observer/en/mcp-tool-poisoning-detection/ . The specification defines the gen ai.evaluation.result event Development status, Recommended level . It carries the result of evaluating model output for quality, accuracy or other characteristics. It should be parented to the evaluated span when possible; otherwise set gen ai.response.id . The evaluation name gen ai.evaluation.name , for example Relevance is required. The numeric score gen ai.evaluation.score.value and the label gen ai.evaluation.score.label are required « if applicable »; the label must stay low-cardinality. If the evaluator itself fails, error.type says so: a broken evaluator is not an answer judged correct. Who computes the score? The specification does not say: it only describes how to carry it. A deterministic check does the answer cite a retrieved source? is the JSON valid? , human review on a sample or a second model are all possible. A judge model is itself fallible and must be calibrated on annotated cases. On Dynatrace SaaS, the « Dynatrace for AI Observability » page states, as read in the documentation, online LLM-as-a-judge evaluations on production traces, with scores written as business events linked to the source traces, and OpenTelemetry ingestion using the GenAI conventions. No equivalence with the attributes above is claimed: check in your tenant. invoke agent and execute tool ; check the names actually produced, as the specification is in development. gen ai.response.id . To see why a raw failure rate is not enough, read the difference between observability and monitoring https://span.observer/en/observability-vs-monitoring-difference/ . And if trace volume gets heavy, tail sampling https://span.observer/en/opentelemetry-tail-sampling-dynatrace/ can keep the traces with bad evaluations first. Replay a request whose expected answer is known, forcing a tool failure a tool returning an empty list . The trace should show the execute tool span, and the evaluation should come out as a failure. If it stays « pass », the criterion is too loose. If no event arrives, check the link to the span or the gen ai.response.id . gen ai.evaluation.explanation for human review. gen ai.agent.version to compare before and after a prompt change. It is a run that ends without a technical error but whose result is wrong or unusable. Spans carry no error status, hence the need to evaluate the content. No: the pages read show Development status for agent spans, the evaluation event and most attributes. Names may change; only error.type is marked stable there. No. Capture is off by default NO CONTENT and content can be sensitive. Enable it on a sample and protect access. Detecting a wrong answer needs a signal that availability metrics do not provide: an evaluation of the content, attached to the trace. OpenTelemetry offers a format to carry it; the quality criterion remains for the team to define. Next, see the guardrails for AI agents in production https://span.observer/en/ai-agent-guardrails-production/ . Sources: semantic-conventions-genai repository Apache 2.0 https://github.com/open-telemetry/semantic-conventions-genai , OpenTelemetry: Recording errors https://opentelemetry.io/docs/specs/semconv/general/recording-errors/ , Dynatrace documentation SaaS https://docs.dynatrace.com/docs/observe/dynatrace-for-ai-observability . This is an independent blog, not affiliated with Dynatrace LLC. Product names belong to their owners. The conventions cited are in development: check the specification version before relying on them. Originally published on span.observer https://span.observer/en/ai-agent-silent-failures-detect/ . French version: https://span.observer/fr/echecs-silencieux-agents-ia-detecter/ https://span.observer/fr/echecs-silencieux-agents-ia-detecter/ .