{"slug": "datadog-brings-production-experimentation-to-ai-agent-observability", "title": "Datadog Brings Production Experimentation to AI Agent Observability", "summary": "Datadog extended its Agent Observability platform with Online Experiments, a capability that splits production traffic between AI agent versions to compare evaluation scores alongside latency, token consumption, execution cost and errors. In a Datadog example comparing GPT-5.6 Luna against GPT-6 Luna with traffic split evenly, the newer model showed a 44.3% reduction in estimated cost while quality scores stayed close at 3.84 and 3.86, but trace inspection revealed more frequent invalid tool parameters that the overall quality score did not surface. Datadog said it used that finding to improve tool handling before proceeding with the model change, and the failed calls became candidates for additional offline regression tests.", "body_md": "[Datadog](https://www.datadoghq.com/) is extending its [Agent Observability platform](https://www.datadoghq.com/products/ai/agent-observability/) with [Online Experiments](https://www.datadoghq.com/blog/online-experiments-agent-observability/), allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. The capability combines agent evaluation scores with operational measurements such as latency, token consumption, execution cost, and errors, helping teams establish whether changes to models, prompts, retrieval strategies, or tools actually improve an agent.\n\nThe announcement addresses a growing challenge in agent development: a change that performs better in a controlled test may behave very differently when exposed to unpredictable user requests. Datadog's approach brings controlled experimentation, production telemetry, and execution traces together, giving teams evidence of both the outcome and the behaviour behind it.\n\nOffline evaluations remain an important part of agent development. Teams can run different versions against predefined datasets, assess responses and identify known regressions before deployment.\n\nHowever, these evaluations are only as representative as the datasets and scoring criteria behind them. New user workflows, unexpected requests, and changing tool responses can expose weaknesses that existing test cases never anticipated.\n\nOnline experiments address this limitation by evaluating agent behaviour against real user interactions. Teams can compare a control version with a modified variant by randomly distributing production traffic between them, reducing the influence of differences in user requests.\n\nThis provides a more realistic view of how an agent performs under actual workloads, while also helping teams identify new scenarios that should eventually become part of their offline regression datasets.\n\nDatadog integrates its Agent Observability evaluators with its Experiments capability, allowing evaluation scores to be compared alongside conventional operational metrics.\n\nTeams can measure response quality, accuracy, and completeness while tracking execution duration, token consumption, estimated cost, and workflow errors.[Feature Flags](https://en.wikipedia.org/wiki/Feature_toggle) manage traffic allocation, letting engineers expose a change to a small percentage of users before rolling it out more widely.\n\nThis is important because improvements rarely come without trade-offs. A cheaper model might require more retries, while a revised retrieval strategy could improve evidence quality but increase execution time.\n\nDatadog also connects experiment results to individual agent traces. Engineers can inspect tool calls, supplied arguments, and returned results to understand why a particular variant behaved differently.\n\nAccording to the article, the experiment compared GPT-5.6 Luna against GPT-6 Luna, splitting traffic evenly between the two. An LLM-based evaluator scored investigation quality from one to five, considering accuracy, completeness, and whether conclusions were supported by available evidence.\n\nThey claim the results showed a 44.3% reduction in estimated cost for the newer model, with quality scores remaining close at 3.84 and 3.86, respectively.\n\nHowever, the operational results revealed potential concerns. The newer model showed estimated increases in execution duration and workflow errors, although the confidence intervals were too broad to establish definitive regressions.\n\nInspecting the traces revealed that the newer model also supplied invalid tool parameters more frequently, usually recovering after retries. This behaviour was not sufficiently visible in the overall quality score.\n\nDatadog used that finding to improve tool handling before proceeding with the model change. The failed calls also provided candidates for additional offline regression tests.\n\nDatadog is not alone in trying to solve this challenge. Tools such as [LangSmith](https://smith.langchain.com/) and [Langfuse](https://langfuse.com/) provide agent tracing and evaluation capabilities, while experimentation platforms such as [LaunchDarkly](https://launchdarkly.com/) support controlled feature rollouts. Datadog's approach connects these concerns within its existing observability ecosystem, linking evaluation results directly to production experiments and execution evidence.\n\nDatadog's broader proposition is that online experiments can create a feedback loop between production behaviour and development testing. Unexpected failures become new regression cases, while subsequent experiments establish whether changes actually resolve them.\n\nThe important shift is from asking whether an agent passes a predefined evaluation to establishing whether it performs better under real conditions. For agent engineering, production experimentation may become as important as traditional software testing.", "url": "https://wpnews.pro/news/datadog-brings-production-experimentation-to-ai-agent-observability", "canonical_source": "https://www.infoq.com/news/2026/11/datadog-ai-agent-observability/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global", "published_at": "2026-11-17 12:00:00+00:00", "updated_at": "2026-11-17 19:35:48.066613+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "large-language-models"], "entities": ["Datadog", "Agent Observability", "Online Experiments", "GPT-5.6 Luna", "GPT-6 Luna"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/datadog-brings-production-experimentation-to-ai-agent-observability", "markdown": "https://wpnews.pro/news/datadog-brings-production-experimentation-to-ai-agent-observability.md", "text": "https://wpnews.pro/news/datadog-brings-production-experimentation-to-ai-agent-observability.txt", "jsonld": "https://wpnews.pro/news/datadog-brings-production-experimentation-to-ai-agent-observability.jsonld"}}