{"slug": "observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and", "title": "Observability in Microsoft Foundry: Tracing Agent Runs, Continuous Evaluation, and the OpenTelemetry Data Plane", "summary": "Microsoft Foundry's agent observability stack captures agent execution as OpenTelemetry traces, using GenAI semantic conventions such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.tool.name, and stores them in Azure Monitor Application Insights. Server-side tracing is enabled by connecting an Application Insights resource to a Foundry project rather than modifying agent code, so it applies uniformly to Prompt Agents and Hosted Agents. The same telemetry feeds a continuous evaluation loop that can automatically grade production traffic, letting teams debug, dashboard, and auto-grade from a single instrumentation pass.", "body_md": "*Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series.*\n\nYou shipped an agent. It calls a model, invokes two tools, retrieves a few documents, and returns an answer. It works in your dev loop. Three weeks later, a support ticket lands on your desk: \"the assistant gave a wrong price for SKU-4471.\" You have no idea which of the six internal steps produced that number, whether the tool returned stale data, whether the model hallucinated over a truncated retrieval result, or whether a retry silently doubled a side effect. You have logs, but logs are flat — they don't tell you *which* LLM call belongs to *which* tool result, nested inside *which* user turn.\n\nThis is the problem agent tracing exists to solve, and it's the problem this article is about: how Microsoft Foundry captures, stores, and lets you query the execution anatomy of an agent run — not as a marketing feature, but as a distributed-systems observability pipeline built on OpenTelemetry (OTel) semantic conventions, backed by Azure Monitor Application Insights, and wired into a continuous evaluation loop that can automatically grade production traffic.\n\nAgents are not stateless request/response functions. A single \"agent run\" can fan out into a tree of operations: a planning call to the model, a tool call to an MCP server, a retrieval call to a vector index, a second model call to synthesize the tool result, and possibly a handoff to another agent. Each of those steps has its own latency, its own token cost, its own failure mode, and its own opportunity to introduce an error that only becomes visible several hops later at the top of the tree.\n\nWithout structured tracing, debugging an agent regresses to grep-ing logs and guessing. With structured tracing, you get:\n\nFoundry treats this as a first-class capability area, not an afterthought bolted onto logging. It sits on three pillars: **evaluation**, **monitoring**, and **tracing** — and this article focuses primarily on tracing and its downstream monitoring/evaluation consumers, because that's where the architecturally interesting decisions live.\n\nFoundry's tracing model is not a proprietary format — it's built directly on **OpenTelemetry**, the CNCF standard for distributed tracing, metrics, and logs. If you've instrumented a microservice with OTel before, the mental model transfers almost directly:\n\n`trace_id`.` gen_ai.request.model`, `gen_ai.usage.input_tokens`, and `gen_ai.tool.name` so that tooling built for one GenAI framework can render traces from another.\nThe reason semantic conventions matter architecturally: they decouple the *producer* of telemetry (your agent code, or Foundry's own hosted runtime) from the *consumer* (the Foundry portal's trace viewer, Application Insights, or a third-party OTel-compatible tool like Grafana Tempo or Honeycomb). As long as both sides speak the same attribute vocabulary, you can swap the visualization layer without touching instrumentation.\n\nAt a high level, the data plane looks like this:\n\nThree things are worth calling out about this architecture:\n\n`response.completed` events) rather than requiring a second instrumentation pass. This is the architectural insight that makes the whole system compose well: instrument once, consume three ways (debug, dashboard, auto-grade).\nWhen tracing is enabled and an agent runs, the sequence is roughly:\n\nBecause server-side tracing is enabled by *connecting* an Application Insights resource to the project — not by changing agent code — this works uniformly for both Prompt Agents (defined via `PromptAgentDefinition`) and Hosted Agents (custom runtimes deployed behind the Responses/Invocations protocols), which is a meaningfully different design point from \"add an SDK decorator to every function,\" the model most bespoke agent frameworks use.\n\nFoundry gives you two complementary instrumentation paths, and the recommended sequence is deliberate:\n\nThis requires zero code changes. You connect an Application Insights resource to your Foundry project (via the **Agents → Traces → Connect** flow, or **Manage → Project details → Connected resources**), and Foundry automatically starts logging traces for any Prompt Agent, Hosted Agent, or workflow running in that project. You get 90 days of out-of-the-box trace history the moment it's wired up.\n\n```\n# There's no CLI step for the connection itself (it's a portal action today),\n# but you can verify the Application Insights resource exists and is linked\n# via Azure CLI as part of your provisioning pipeline:\naz monitor app-insights component show \\\n  --app my-foundry-project-insights \\\n  --resource-group rg-foundry-prod \\\n  --query \"{name:name, connectionString:connectionString}\" \\\n  -o table\n```\n\nIf your application wraps the Foundry SDK with custom orchestration logic — retries, pre/post-processing, business rule branching — you'll want spans for *that* code too, not just what happens inside the agent runtime. This is standard OpenTelemetry instrumentation layered on top of the Azure SDK's tracing plugin:\n\n```\npip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry\n# main.py — client-side tracing for custom orchestration code around a Foundry agent call\nimport os\nfrom azure.identity import DefaultAzureCredential\nfrom azure.ai.projects import AIProjectClient\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter\n\n# 1. Wire up an OTel TracerProvider that exports to the same\n#    Application Insights resource connected to your Foundry project.\nprovider = TracerProvider()\nexporter = AzureMonitorTraceExporter(\n    connection_string=os.environ[\"APPLICATIONINSIGHTS_CONNECTION_STRING\"]\n)\nprovider.add_span_processor(BatchSpanProcessor(exporter))\ntrace.set_tracer_provider(provider)\ntracer = trace.get_tracer(__name__)\n\nendpoint = os.environ[\"AZURE_AI_PROJECT_ENDPOINT\"]\n\nwith (\n    DefaultAzureCredential() as credential,\n    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,\n):\n    # 2. Wrap your own business logic in a span. This nests correctly\n    #    alongside the server-side spans Foundry emits for the agent call,\n    #    because both use the same OTel trace-context propagation.\n    with tracer.start_as_current_span(\"pricing_lookup_orchestration\") as span:\n        span.set_attribute(\"customer.tier\", \"enterprise\")\n        span.set_attribute(\"sku.id\", \"SKU-4471\")\n\n        response = project_client.get_openai_client().responses.create(\n            model=os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"],\n            input=\"What is the current price for SKU-4471?\",\n        )\n\n        span.set_attribute(\"response.id\", response.id)\n        print(response.output_text)\n```\n\nThe important architectural detail: because the Azure SDK's tracing plugin (`azure-core-tracing-opentelemetry`) and your manual span both register against the *same* global `TracerProvider`, the resulting trace has your custom span as a parent (or sibling) of the SDK's auto-generated GenAI spans — you get one coherent tree, not two disconnected traces you have to mentally stitch together.\n\nThere's also a **Foundry Toolkit for VS Code** extension that spins up a local OTLP collector so you can view traces during development without needing an Application Insights resource at all — useful for the inner dev loop before you've provisioned cloud infrastructure.\n\nHere's a more complete example — a Prompt Agent with a tool call, instrumented end-to-end, followed by a script that queries the resulting trace back out of Application Insights using KQL.\n\n```\n# create_traced_agent.py\n# Production-adjacent pattern: create an agent, run it, and confirm\n# the run is traceable. Requires AZURE_AI_PROJECT_ENDPOINT and\n# AZURE_AI_MODEL_DEPLOYMENT_NAME to already be connected to an\n# Application Insights resource in the Foundry portal.\nimport os\nfrom azure.identity import DefaultAzureCredential\nfrom azure.ai.projects import AIProjectClient\nfrom azure.ai.projects.models import PromptAgentDefinition\n\nendpoint = os.environ[\"AZURE_AI_PROJECT_ENDPOINT\"]\nmodel = os.environ[\"AZURE_AI_MODEL_DEPLOYMENT_NAME\"]\n\nwith (\n    DefaultAzureCredential() as credential,\n    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,\n    project_client.get_openai_client() as openai_client,\n):\n    agent = project_client.agents.create_version(\n        agent_name=\"pricing-assistant\",\n        definition=PromptAgentDefinition(\n            model=model,\n            instructions=(\n                \"You are a pricing assistant. Use the get_price tool \"\n                \"to answer questions about SKU pricing. Never guess a price.\"\n            ),\n            tools=[\n                {\n                    \"type\": \"function\",\n                    \"name\": \"get_price\",\n                    \"description\": \"Look up the current price for a SKU.\",\n                    \"parameters\": {\n                        \"type\": \"object\",\n                        \"properties\": {\"sku_id\": {\"type\": \"string\"}},\n                        \"required\": [\"sku_id\"],\n                    },\n                }\n            ],\n        ),\n    )\n\n    response = openai_client.responses.create(\n        model=agent.name,\n        input=\"What is the current price for SKU-4471?\",\n        extra_body={\"agent\": {\"name\": agent.name, \"type\": \"agent_reference\"}},\n    )\n\n    # response.id is the correlation key you'll search for in\n    # Foundry's Traces tab or in Application Insights.\n    print(f\"Response ID (search this in Traces tab): {response.id}\")\n```\n\nTo pull the resulting trace back out programmatically (useful for CI gates that assert \"no run in this test suite exceeded 5 seconds of model latency\"), query Application Insights with KQL:\n\n```\n// Find all GenAI spans for a given response/run, ordered by start time,\n// showing the causal shape of the run.\ndependencies\n| where customDimensions[\"gen_ai.response.id\"] == \"resp_abc123...\"\n| project timestamp, name, duration, target,\n          model = tostring(customDimensions[\"gen_ai.request.model\"]),\n          inputTokens = tostring(customDimensions[\"gen_ai.usage.input_tokens\"]),\n          outputTokens = tostring(customDimensions[\"gen_ai.usage.output_tokens\"]),\n          toolName = tostring(customDimensions[\"gen_ai.tool.name\"])\n| order by timestamp asc\n// Aggregate latency by span type over the last 24 hours to spot\n// which stage of the pipeline is driving a regression.\ndependencies\n| where timestamp > ago(24h)\n| where customDimensions has \"gen_ai\"\n| extend spanKind = tostring(customDimensions[\"gen_ai.operation.name\"])\n| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), count() by spanKind\n| order by p95 desc\n```\n\nOnce telemetry lands in Application Insights, the Foundry portal's **Traces** tab renders it as a waterfall — a horizontal timeline where nested bars represent the parent/child span hierarchy:\n\nThis view answers the debugging question directly: in the pricing example from the introduction, you'd see the root span (the full run, ~4.2s), a planning LLM call, a `get_price` tool-call span with its actual returned arguments and result, and a final synthesis LLM call. If the tool span shows a 0.6s duration but the *displayed* answer is wrong, you immediately know the bug isn't latency-related — it's either in the tool's data or in how the model interpreted the tool's result. If instead you see an 800ms gap between a tool call finishing and the next span starting, that's a retry or a queuing delay, not a model problem. This is the entire value proposition of structured tracing over flat logs: **the shape of the trace itself is diagnostic**, before you've read a single attribute value.\n\nYou can also pivot from a trace to its **Conversation** view, which shows the response ID, ordered run steps, and full input/output payloads between user and agent — useful when the question isn't \"what was slow\" but \"what did the model actually see.\"\n\nThis is where Foundry's observability stack stops being \"a nicer log viewer\" and becomes an actual quality-control system. Because every agent response is a structured event (`response.completed`) with an attached trace, Foundry lets you attach **evaluators** — the same built-in quality/safety/RAG-specific evaluators used in offline evaluation — to a **live sampling rule** that runs continuously against production traffic.\n\nThere are two flavors:\n\n`max_hourly_runs` throttle so you don't accidentally run (and pay for) an evaluator on every single production call.\n\n```\nfrom azure.ai.projects.models import (\n    EvaluationRule,\n    ContinuousEvaluationRuleAction,\n    EvaluationRuleFilter,\n    EvaluationRuleEventType,\n)\n\n# 1. Define what \"good\" means: an evaluator config, here checking for violent content.\ndata_source_config = {\"type\": \"azure_ai_source\", \"scenario\": \"responses\"}\ntesting_criteria = [\n    {\"type\": \"azure_ai_evaluator\", \"name\": \"violence_detection\", \"evaluator_name\": \"builtin.violence\"}\n]\neval_object = openai_client.evals.create(\n    name=\"Continuous Evaluation\",\n    data_source_config=data_source_config,\n    testing_criteria=testing_criteria,\n)\n\n# 2. Wire that evaluator to a live sampling rule: run it on every\n#    response.completed event for this agent, capped at 100 runs/hour\n#    to bound evaluation cost.\ncontinuous_eval_rule = project_client.evaluation_rules.create_or_update(\n    id=\"my-continuous-eval-rule\",\n    evaluation_rule=EvaluationRule(\n        display_name=\"My Continuous Eval Rule\",\n        description=\"Runs a safety evaluator on live agent responses\",\n        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),\n        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,\n        filter=EvaluationRuleFilter(agent_name=\"pricing-assistant\"),\n        enabled=True,\n    ),\n)\n```\n\nThe architectural point worth internalizing: **the evaluation rule doesn't re-run the agent** — it consumes the already-captured trace and response as the input to the evaluator, meaning it adds evaluator inference cost but not agent re-execution cost. This is a materially cheaper design than \"shadow-run every production request through an offline eval pipeline,\" and it's why continuous evaluation is viable at meaningful sample rates in production, not just in staging.\n\nSetting this up requires the project's managed identity to hold the **Foundry User** role (recently renamed from Azure AI User) on the project — a detail that trips people up because the *evaluation rule* runs under the project's identity, not the caller's, so RBAC has to be granted ahead of time or the rule silently fails to execute.\n\nThe Monitor tab in the Foundry portal turns the raw trace stream into the four numbers you actually check daily:\n\n| Metric | What a bad number means | \n|---|---|\n| Token usage | Verbose prompts/responses; a candidate for prompt or context-window optimization | \n| Latency (p50/p95) | Above ~10s often indicates model throttling, heavy tool calls, or network issues | \n| Run success rate | Below ~95% warrants investigating failed runs — this is your first-line SLO | \n| Evaluation scores | Built-in and custom evaluator scores sampled from continuous evaluation rules | \n\nIt also surfaces **red team scan** results (adversarial testing for risks like data leakage or prohibited actions) and lets you configure **alerts** on latency, token usage, evaluation-score thresholds, or red-team findings — turning what would otherwise be a manual \"check the traces tab\" habit into an actual paging/notification system. All of this is still marked preview at the time of writing, which matters for anyone deciding whether to build a hard production dependency on the dashboard UI itself versus querying Application Insights directly (the latter is GA and stable; the dashboard is a convenience layer on top).\n\nSingle-agent tracing is a solved problem in most GenAI observability tooling at this point. Multi-agent tracing is not, and it's an area Microsoft is actively investing standards effort into. Foundry, in collaboration with Cisco Outshift, contributes to semantic conventions for multi-agent systems that extend the base OpenTelemetry GenAI agent/framework spans — the goal being a standard way to represent things like \"which agent delegated to which sub-agent,\" \"which agent owns a given tool call,\" and \"how did a task hand off across an A2A boundary\" as first-class span attributes rather than framework-specific ad hoc fields.\n\nThis matters because as you move from single-agent Prompt Agents toward orchestrated multi-agent systems (Sequential/Concurrent/Handoff/GroupChat/Magentic patterns via the Microsoft Agent Framework — see Day 7 of this series on the Workflows-to-Agent-Framework migration), the trace tree gets a lot deeper and a lot wider, and without standardized attribution, you end up with an opaque blob of nested LLM calls with no way to answer \"which *agent* introduced this error,\" only \"which *span*.\" Standardized multi-agent semantic conventions are what let a trace viewer render an agent-boundary-aware view (grouping spans by owning agent) instead of a flat operation tree.\n\nTraces capture **exactly what makes them useful for debugging** — full inputs, outputs, and tool arguments — which is also exactly what makes them a data-exfiltration and compliance risk if mishandled:\n\nTracing cost is **not a Foundry line item** — it's an Application Insights / Log Analytics ingestion and retention cost, billed per GB ingested and per GB-month retained (verify current rates before budgeting; pricing changes over time — verify this stat before publishing). This has two practical implications:\n\nIf you're not deep in the Foundry ecosystem, or you need a single pane of glass across non-Foundry services too, you have real alternatives, because Foundry's tracing is standard OTel underneath:\n\nThe trade-off in choosing Foundry's native path is mostly about **lock-in vs. leverage**: you get tight integration with evaluation and monitoring at the cost of your telemetry backend being Application Insights specifically (rather than a vendor-neutral OTel backend of your choice) for the highest-value features like continuous evaluation.\n\nObservability in Microsoft Foundry isn't a dashboard bolted on top of an agent platform — it's a data-plane decision: emit OpenTelemetry GenAI-convention spans from the runtime, store them in Application Insights, and let three different consumers (a human debugging in the Traces tab, an aggregation layer in the Monitoring dashboard, and an automated evaluator sampling live traffic) read from the same stream. That single-source-of-truth design is what makes it possible to go from \"a customer says the agent was wrong\" to \"here is the exact span, with the exact tool arguments, that produced that answer\" — and, increasingly, to catch that class of error automatically before a customer ever notices, via continuous evaluation.\n\nIf you're running Foundry agents in anything beyond a demo, connecting Application Insights and enabling server-side tracing is not optional infrastructure — it's the difference between debugging with a flashlight and debugging with a floor plan.\n\n**Call to action**: If you haven't connected an Application Insights resource to your Foundry project yet, do it before your next deploy — it's a five-minute portal action that will save you hours the first time an agent misbehaves in production. Then come back tomorrow for Day 11 of this series.\n\n*This is Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series — one deep technical dive into a different corner of the Foundry ecosystem every day. Previous entries covered long-running agent resilience, the Responses vs. Invocations protocols, Autopilot identity, Foundry Local, the Agent Optimizer, MCP toolbox governance, the Workflows-to-Agent-Framework migration, Code Interpreter internals, and Voice Agents.*", "url": "https://wpnews.pro/news/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and", "canonical_source": "https://dev.to/monuminu/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and-the-4gp", "published_at": "2026-09-22 05:39:35+00:00", "updated_at": "2026-09-22 05:52:38.015546+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-infrastructure", "developer-tools", "ai-tools"], "entities": ["Microsoft Foundry", "OpenTelemetry", "Azure Monitor Application Insights", "CNCF", "Grafana Tempo", "Honeycomb", "MCP"], "alternates": {"html": "https://wpnews.pro/news/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and", "markdown": "https://wpnews.pro/news/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and.md", "text": "https://wpnews.pro/news/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and.txt", "jsonld": "https://wpnews.pro/news/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and.jsonld"}}