Observability in Microsoft Foundry: Tracing Agent Runs, Continuous Evaluation, and the OpenTelemetry Data Plane

Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series.

You shipped an agent. It calls a model, invokes two tools, retrieves a few documents, and returns an answer. It works in your dev loop. Three weeks later, a support ticket lands on your desk: "the assistant gave a wrong price for SKU-4471." You have no idea which of the six internal steps produced that number, whether the tool returned stale data, whether the model hallucinated over a truncated retrieval result, or whether a retry silently doubled a side effect. You have logs, but logs are flat — they don't tell you which LLM call belongs to which tool result, nested inside which user turn.

This is the problem agent tracing exists to solve, and it's the problem this article is about: how Microsoft Foundry captures, stores, and lets you query the execution anatomy of an agent run — not as a marketing feature, but as a distributed-systems observability pipeline built on OpenTelemetry (OTel) semantic conventions, backed by Azure Monitor Application Insights, and wired into a continuous evaluation loop that can automatically grade production traffic.

Agents are not stateless request/response functions. A single "agent run" can fan out into a tree of operations: a planning call to the model, a tool call to an MCP server, a retrieval call to a vector index, a second model call to synthesize the tool result, and possibly a handoff to another agent. Each of those steps has its own latency, its own token cost, its own failure mode, and its own opportunity to introduce an error that only becomes visible several hops later at the top of the tree.

Without structured tracing, debugging an agent regresses to grep-ing logs and guessing. With structured tracing, you get:

Foundry treats this as a first-class capability area, not an afterthought bolted onto logging. It sits on three pillars: evaluation, monitoring, and tracing — and this article focuses primarily on tracing and its downstream monitoring/evaluation consumers, because that's where the architecturally interesting decisions live.

Foundry's tracing model is not a proprietary format — it's built directly on OpenTelemetry, the CNCF standard for distributed tracing, metrics, and logs. If you've instrumented a microservice with OTel before, the mental model transfers almost directly:

trace_id. gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.tool.name so that tooling built for one GenAI framework can render traces from another. The reason semantic conventions matter architecturally: they decouple the producer of telemetry (your agent code, or Foundry's own hosted runtime) from the consumer (the Foundry portal's trace viewer, Application Insights, or a third-party OTel-compatible tool like Grafana Tempo or Honeycomb). As long as both sides speak the same attribute vocabulary, you can swap the visualization layer without touching instrumentation.

At a high level, the data plane looks like this:

Three things are worth calling out about this architecture:

response.completed events) rather than requiring a second instrumentation pass. This is the architectural insight that makes the whole system compose well: instrument once, consume three ways (debug, dashboard, auto-grade). When tracing is enabled and an agent runs, the sequence is roughly:

Because server-side tracing is enabled by connecting an Application Insights resource to the project — not by changing agent code — this works uniformly for both Prompt Agents (defined via PromptAgentDefinition) and Hosted Agents (custom runtimes deployed behind the Responses/Invocations protocols), which is a meaningfully different design point from "add an SDK decorator to every function," the model most bespoke agent frameworks use.

Foundry gives you two complementary instrumentation paths, and the recommended sequence is deliberate:

This requires zero code changes. You connect an Application Insights resource to your Foundry project (via the Agents → Traces → Connect flow, or Manage → Project details → Connected resources), and Foundry automatically starts logging traces for any Prompt Agent, Hosted Agent, or workflow running in that project. You get 90 days of out-of-the-box trace history the moment it's wired up.

az monitor app-insights component show \
  --app my-foundry-project-insights \
  --resource-group rg-foundry-prod \
  --query "{name:name, connectionString:connectionString}" \
  -o table

If your application wraps the Foundry SDK with custom orchestration logic — retries, pre/post-processing, business rule branching — you'll want spans for that code too, not just what happens inside the agent runtime. This is standard OpenTelemetry instrumentation layered on top of the Azure SDK's tracing plugin:

pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter

provider = TracerProvider()
exporter = AzureMonitorTraceExporter(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    with tracer.start_as_current_span("pricing_lookup_orchestration") as span:
        span.set_attribute("customer.tier", "enterprise")
        span.set_attribute("sku.id", "SKU-4471")

        response = project_client.get_openai_client().responses.create(
            model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
            input="What is the current price for SKU-4471?",
        )

        span.set_attribute("response.id", response.id)
        print(response.output_text)

The important architectural detail: because the Azure SDK's tracing plugin (azure-core-tracing-opentelemetry) and your manual span both register against the same global TracerProvider, the resulting trace has your custom span as a parent (or sibling) of the SDK's auto-generated GenAI spans — you get one coherent tree, not two disconnected traces you have to mentally stitch together.

There's also a Foundry Toolkit for VS Code extension that spins up a local OTLP collector so you can view traces during development without needing an Application Insights resource at all — useful for the inner dev loop before you've provisioned cloud infrastructure.

Here's a more complete example — a Prompt Agent with a tool call, instrumented end-to-end, followed by a script that queries the resulting trace back out of Application Insights using KQL.

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
    project_client.get_openai_client() as openai_client,
):
    agent = project_client.agents.create_version(
        agent_name="pricing-assistant",
        definition=PromptAgentDefinition(
            model=model,
            instructions=(
                "You are a pricing assistant. Use the get_price tool "
                "to answer questions about SKU pricing. Never guess a price."
            ),
            tools=[
                {
                    "type": "function",
                    "name": "get_price",
                    "description": "Look up the current price for a SKU.",
                    "parameters": {
                        "type": "object",
                        "properties": {"sku_id": {"type": "string"}},
                        "required": ["sku_id"],
                    },
                }
            ],
        ),
    )

    response = openai_client.responses.create(
        model=agent.name,
        input="What is the current price for SKU-4471?",
        extra_body={"agent": {"name": agent.name, "type": "agent_reference"}},
    )

    print(f"Response ID (search this in Traces tab): {response.id}")

To pull the resulting trace back out programmatically (useful for CI gates that assert "no run in this test suite exceeded 5 seconds of model latency"), query Application Insights with KQL:

// Find all GenAI spans for a given response/run, ordered by start time,
// showing the causal shape of the run.
dependencies
| where customDimensions["gen_ai.response.id"] == "resp_abc123..."
| project timestamp, name, duration, target,
          model = tostring(customDimensions["gen_ai.request.model"]),
          inputTokens = tostring(customDimensions["gen_ai.usage.input_tokens"]),
          outputTokens = tostring(customDimensions["gen_ai.usage.output_tokens"]),
          toolName = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc
// Aggregate latency by span type over the last 24 hours to spot
// which stage of the pipeline is driving a regression.
dependencies
| where timestamp > ago(24h)
| where customDimensions has "gen_ai"
| extend spanKind = tostring(customDimensions["gen_ai.operation.name"])
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), count() by spanKind
| order by p95 desc

Once telemetry lands in Application Insights, the Foundry portal's Traces tab renders it as a waterfall — a horizontal timeline where nested bars represent the parent/child span hierarchy:

This view answers the debugging question directly: in the pricing example from the introduction, you'd see the root span (the full run, ~4.2s), a planning LLM call, a get_price tool-call span with its actual returned arguments and result, and a final synthesis LLM call. If the tool span shows a 0.6s duration but the displayed answer is wrong, you immediately know the bug isn't latency-related — it's either in the tool's data or in how the model interpreted the tool's result. If instead you see an 800ms gap between a tool call finishing and the next span starting, that's a retry or a queuing delay, not a model problem. This is the entire value proposition of structured tracing over flat logs: the shape of the trace itself is diagnostic, before you've read a single attribute value.

You can also pivot from a trace to its Conversation view, which shows the response ID, ordered run steps, and full input/output payloads between user and agent — useful when the question isn't "what was slow" but "what did the model actually see."

This is where Foundry's observability stack stops being "a nicer log viewer" and becomes an actual quality-control system. Because every agent response is a structured event (response.completed) with an attached trace, Foundry lets you attach evaluators — the same built-in quality/safety/RAG-specific evaluators used in offline evaluation — to a live sampling rule that runs continuously against production traffic.

There are two flavors:

max_hourly_runs throttle so you don't accidentally run (and pay for) an evaluator on every single production call.

from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType,
)

data_source_config = {"type": "azure_ai_source", "scenario": "responses"}
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence_detection", "evaluator_name": "builtin.violence"}
]
eval_object = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Continuous Eval Rule",
        description="Runs a safety evaluator on live agent responses",
        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="pricing-assistant"),
        enabled=True,
    ),
)

The architectural point worth internalizing: the evaluation rule doesn't re-run the agent — it consumes the already-captured trace and response as the input to the evaluator, meaning it adds evaluator inference cost but not agent re-execution cost. This is a materially cheaper design than "shadow-run every production request through an offline eval pipeline," and it's why continuous evaluation is viable at meaningful sample rates in production, not just in staging.

Setting this up requires the project's managed identity to hold the Foundry User role (recently renamed from Azure AI User) on the project — a detail that trips people up because the evaluation rule runs under the project's identity, not the caller's, so RBAC has to be granted ahead of time or the rule silently fails to execute.

The Monitor tab in the Foundry portal turns the raw trace stream into the four numbers you actually check daily:

It also surfaces red team scan results (adversarial testing for risks like data leakage or prohibited actions) and lets you configure alerts on latency, token usage, evaluation-score thresholds, or red-team findings — turning what would otherwise be a manual "check the traces tab" habit into an actual paging/notification system. All of this is still marked preview at the time of writing, which matters for anyone deciding whether to build a hard production dependency on the dashboard UI itself versus querying Application Insights directly (the latter is GA and stable; the dashboard is a convenience layer on top).

Single-agent tracing is a solved problem in most GenAI observability tooling at this point. Multi-agent tracing is not, and it's an area Microsoft is actively investing standards effort into. Foundry, in collaboration with Cisco Outshift, contributes to semantic conventions for multi-agent systems that extend the base OpenTelemetry GenAI agent/framework spans — the goal being a standard way to represent things like "which agent delegated to which sub-agent," "which agent owns a given tool call," and "how did a task hand off across an A2A boundary" as first-class span attributes rather than framework-specific ad hoc fields.

This matters because as you move from single-agent Prompt Agents toward orchestrated multi-agent systems (Sequential/Concurrent/Handoff/GroupChat/Magentic patterns via the Microsoft Agent Framework — see Day 7 of this series on the Workflows-to-Agent-Framework migration), the trace tree gets a lot deeper and a lot wider, and without standardized attribution, you end up with an opaque blob of nested LLM calls with no way to answer "which agent introduced this error," only "which span." Standardized multi-agent semantic conventions are what let a trace viewer render an agent-boundary-aware view (grouping spans by owning agent) instead of a flat operation tree.

Traces capture exactly what makes them useful for debugging — full inputs, outputs, and tool arguments — which is also exactly what makes them a data-exfiltration and compliance risk if mishandled:

Tracing cost is not a Foundry line item — it's an Application Insights / Log Analytics ingestion and retention cost, billed per GB ingested and per GB-month retained (verify current rates before budgeting; pricing changes over time — verify this stat before publishing). This has two practical implications:

If you're not deep in the Foundry ecosystem, or you need a single pane of glass across non-Foundry services too, you have real alternatives, because Foundry's tracing is standard OTel underneath:

The trade-off in choosing Foundry's native path is mostly about lock-in vs. leverage: you get tight integration with evaluation and monitoring at the cost of your telemetry backend being Application Insights specifically (rather than a vendor-neutral OTel backend of your choice) for the highest-value features like continuous evaluation.

Observability in Microsoft Foundry isn't a dashboard bolted on top of an agent platform — it's a data-plane decision: emit OpenTelemetry GenAI-convention spans from the runtime, store them in Application Insights, and let three different consumers (a human debugging in the Traces tab, an aggregation layer in the Monitoring dashboard, and an automated evaluator sampling live traffic) read from the same stream. That single-source-of-truth design is what makes it possible to go from "a customer says the agent was wrong" to "here is the exact span, with the exact tool arguments, that produced that answer" — and, increasingly, to catch that class of error automatically before a customer ever notices, via continuous evaluation.

If you're running Foundry agents in anything beyond a demo, connecting Application Insights and enabling server-side tracing is not optional infrastructure — it's the difference between debugging with a flashlight and debugging with a floor plan.

Call to action: If you haven't connected an Application Insights resource to your Foundry project yet, do it before your next deploy — it's a five-minute portal action that will save you hours the first time an agent misbehaves in production. Then come back tomorrow for Day 11 of this series.

This is Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series — one deep technical dive into a different corner of the Foundry ecosystem every day. Previous entries covered long-running agent resilience, the Responses vs. Invocations protocols, Autopilot identity, Foundry Local, the Agent Optimizer, MCP toolbox governance, the Workflows-to-Agent-Framework migration, Code Interpreter internals, and Voice Agents.