# Observability in Microsoft Foundry: Tracing Agent Runs, Continuous Evaluation, and the OpenTelemetry Data Plane

> Source: <https://dev.to/monuminu/observability-in-microsoft-foundry-tracing-agent-runs-continuous-evaluation-and-the-4gp>
> Published: 2026-09-22 05:39:35+00:00

*Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series.*

You shipped an agent. It calls a model, invokes two tools, retrieves a few documents, and returns an answer. It works in your dev loop. Three weeks later, a support ticket lands on your desk: "the assistant gave a wrong price for SKU-4471." You have no idea which of the six internal steps produced that number, whether the tool returned stale data, whether the model hallucinated over a truncated retrieval result, or whether a retry silently doubled a side effect. You have logs, but logs are flat — they don't tell you *which* LLM call belongs to *which* tool result, nested inside *which* user turn.

This is the problem agent tracing exists to solve, and it's the problem this article is about: how Microsoft Foundry captures, stores, and lets you query the execution anatomy of an agent run — not as a marketing feature, but as a distributed-systems observability pipeline built on OpenTelemetry (OTel) semantic conventions, backed by Azure Monitor Application Insights, and wired into a continuous evaluation loop that can automatically grade production traffic.

Agents are not stateless request/response functions. A single "agent run" can fan out into a tree of operations: a planning call to the model, a tool call to an MCP server, a retrieval call to a vector index, a second model call to synthesize the tool result, and possibly a handoff to another agent. Each of those steps has its own latency, its own token cost, its own failure mode, and its own opportunity to introduce an error that only becomes visible several hops later at the top of the tree.

Without structured tracing, debugging an agent regresses to grep-ing logs and guessing. With structured tracing, you get:

Foundry treats this as a first-class capability area, not an afterthought bolted onto logging. It sits on three pillars: **evaluation**, **monitoring**, and **tracing** — and this article focuses primarily on tracing and its downstream monitoring/evaluation consumers, because that's where the architecturally interesting decisions live.

Foundry's tracing model is not a proprietary format — it's built directly on **OpenTelemetry**, the CNCF standard for distributed tracing, metrics, and logs. If you've instrumented a microservice with OTel before, the mental model transfers almost directly:

`trace_id`.` gen_ai.request.model`, `gen_ai.usage.input_tokens`, and `gen_ai.tool.name` so that tooling built for one GenAI framework can render traces from another.
The reason semantic conventions matter architecturally: they decouple the *producer* of telemetry (your agent code, or Foundry's own hosted runtime) from the *consumer* (the Foundry portal's trace viewer, Application Insights, or a third-party OTel-compatible tool like Grafana Tempo or Honeycomb). As long as both sides speak the same attribute vocabulary, you can swap the visualization layer without touching instrumentation.

At a high level, the data plane looks like this:

Three things are worth calling out about this architecture:

`response.completed` events) rather than requiring a second instrumentation pass. This is the architectural insight that makes the whole system compose well: instrument once, consume three ways (debug, dashboard, auto-grade).
When tracing is enabled and an agent runs, the sequence is roughly:

Because server-side tracing is enabled by *connecting* an Application Insights resource to the project — not by changing agent code — this works uniformly for both Prompt Agents (defined via `PromptAgentDefinition`) and Hosted Agents (custom runtimes deployed behind the Responses/Invocations protocols), which is a meaningfully different design point from "add an SDK decorator to every function," the model most bespoke agent frameworks use.

Foundry gives you two complementary instrumentation paths, and the recommended sequence is deliberate:

This requires zero code changes. You connect an Application Insights resource to your Foundry project (via the **Agents → Traces → Connect** flow, or **Manage → Project details → Connected resources**), and Foundry automatically starts logging traces for any Prompt Agent, Hosted Agent, or workflow running in that project. You get 90 days of out-of-the-box trace history the moment it's wired up.

```
# There's no CLI step for the connection itself (it's a portal action today),
# but you can verify the Application Insights resource exists and is linked
# via Azure CLI as part of your provisioning pipeline:
az monitor app-insights component show \
  --app my-foundry-project-insights \
  --resource-group rg-foundry-prod \
  --query "{name:name, connectionString:connectionString}" \
  -o table
```

If your application wraps the Foundry SDK with custom orchestration logic — retries, pre/post-processing, business rule branching — you'll want spans for *that* code too, not just what happens inside the agent runtime. This is standard OpenTelemetry instrumentation layered on top of the Azure SDK's tracing plugin:

```
pip install azure-ai-projects azure-identity opentelemetry-sdk azure-core-tracing-opentelemetry
# main.py — client-side tracing for custom orchestration code around a Foundry agent call
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter

# 1. Wire up an OTel TracerProvider that exports to the same
#    Application Insights resource connected to your Foundry project.
provider = TracerProvider()
exporter = AzureMonitorTraceExporter(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    # 2. Wrap your own business logic in a span. This nests correctly
    #    alongside the server-side spans Foundry emits for the agent call,
    #    because both use the same OTel trace-context propagation.
    with tracer.start_as_current_span("pricing_lookup_orchestration") as span:
        span.set_attribute("customer.tier", "enterprise")
        span.set_attribute("sku.id", "SKU-4471")

        response = project_client.get_openai_client().responses.create(
            model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
            input="What is the current price for SKU-4471?",
        )

        span.set_attribute("response.id", response.id)
        print(response.output_text)
```

The important architectural detail: because the Azure SDK's tracing plugin (`azure-core-tracing-opentelemetry`) and your manual span both register against the *same* global `TracerProvider`, the resulting trace has your custom span as a parent (or sibling) of the SDK's auto-generated GenAI spans — you get one coherent tree, not two disconnected traces you have to mentally stitch together.

There's also a **Foundry Toolkit for VS Code** extension that spins up a local OTLP collector so you can view traces during development without needing an Application Insights resource at all — useful for the inner dev loop before you've provisioned cloud infrastructure.

Here's a more complete example — a Prompt Agent with a tool call, instrumented end-to-end, followed by a script that queries the resulting trace back out of Application Insights using KQL.

```
# create_traced_agent.py
# Production-adjacent pattern: create an agent, run it, and confirm
# the run is traceable. Requires AZURE_AI_PROJECT_ENDPOINT and
# AZURE_AI_MODEL_DEPLOYMENT_NAME to already be connected to an
# Application Insights resource in the Foundry portal.
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
    project_client.get_openai_client() as openai_client,
):
    agent = project_client.agents.create_version(
        agent_name="pricing-assistant",
        definition=PromptAgentDefinition(
            model=model,
            instructions=(
                "You are a pricing assistant. Use the get_price tool "
                "to answer questions about SKU pricing. Never guess a price."
            ),
            tools=[
                {
                    "type": "function",
                    "name": "get_price",
                    "description": "Look up the current price for a SKU.",
                    "parameters": {
                        "type": "object",
                        "properties": {"sku_id": {"type": "string"}},
                        "required": ["sku_id"],
                    },
                }
            ],
        ),
    )

    response = openai_client.responses.create(
        model=agent.name,
        input="What is the current price for SKU-4471?",
        extra_body={"agent": {"name": agent.name, "type": "agent_reference"}},
    )

    # response.id is the correlation key you'll search for in
    # Foundry's Traces tab or in Application Insights.
    print(f"Response ID (search this in Traces tab): {response.id}")
```

To pull the resulting trace back out programmatically (useful for CI gates that assert "no run in this test suite exceeded 5 seconds of model latency"), query Application Insights with KQL:

```
// Find all GenAI spans for a given response/run, ordered by start time,
// showing the causal shape of the run.
dependencies
| where customDimensions["gen_ai.response.id"] == "resp_abc123..."
| project timestamp, name, duration, target,
          model = tostring(customDimensions["gen_ai.request.model"]),
          inputTokens = tostring(customDimensions["gen_ai.usage.input_tokens"]),
          outputTokens = tostring(customDimensions["gen_ai.usage.output_tokens"]),
          toolName = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc
// Aggregate latency by span type over the last 24 hours to spot
// which stage of the pipeline is driving a regression.
dependencies
| where timestamp > ago(24h)
| where customDimensions has "gen_ai"
| extend spanKind = tostring(customDimensions["gen_ai.operation.name"])
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), count() by spanKind
| order by p95 desc
```

Once telemetry lands in Application Insights, the Foundry portal's **Traces** tab renders it as a waterfall — a horizontal timeline where nested bars represent the parent/child span hierarchy:

This view answers the debugging question directly: in the pricing example from the introduction, you'd see the root span (the full run, ~4.2s), a planning LLM call, a `get_price` tool-call span with its actual returned arguments and result, and a final synthesis LLM call. If the tool span shows a 0.6s duration but the *displayed* answer is wrong, you immediately know the bug isn't latency-related — it's either in the tool's data or in how the model interpreted the tool's result. If instead you see an 800ms gap between a tool call finishing and the next span starting, that's a retry or a queuing delay, not a model problem. This is the entire value proposition of structured tracing over flat logs: **the shape of the trace itself is diagnostic**, before you've read a single attribute value.

You can also pivot from a trace to its **Conversation** view, which shows the response ID, ordered run steps, and full input/output payloads between user and agent — useful when the question isn't "what was slow" but "what did the model actually see."

This is where Foundry's observability stack stops being "a nicer log viewer" and becomes an actual quality-control system. Because every agent response is a structured event (`response.completed`) with an attached trace, Foundry lets you attach **evaluators** — the same built-in quality/safety/RAG-specific evaluators used in offline evaluation — to a **live sampling rule** that runs continuously against production traffic.

There are two flavors:

`max_hourly_runs` throttle so you don't accidentally run (and pay for) an evaluator on every single production call.

```
from azure.ai.projects.models import (
    EvaluationRule,
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
    EvaluationRuleEventType,
)

# 1. Define what "good" means: an evaluator config, here checking for violent content.
data_source_config = {"type": "azure_ai_source", "scenario": "responses"}
testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence_detection", "evaluator_name": "builtin.violence"}
]
eval_object = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

# 2. Wire that evaluator to a live sampling rule: run it on every
#    response.completed event for this agent, capped at 100 runs/hour
#    to bound evaluation cost.
continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Continuous Eval Rule",
        description="Runs a safety evaluator on live agent responses",
        action=ContinuousEvaluationRuleAction(eval_id=eval_object.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name="pricing-assistant"),
        enabled=True,
    ),
)
```

The architectural point worth internalizing: **the evaluation rule doesn't re-run the agent** — it consumes the already-captured trace and response as the input to the evaluator, meaning it adds evaluator inference cost but not agent re-execution cost. This is a materially cheaper design than "shadow-run every production request through an offline eval pipeline," and it's why continuous evaluation is viable at meaningful sample rates in production, not just in staging.

Setting this up requires the project's managed identity to hold the **Foundry User** role (recently renamed from Azure AI User) on the project — a detail that trips people up because the *evaluation rule* runs under the project's identity, not the caller's, so RBAC has to be granted ahead of time or the rule silently fails to execute.

The Monitor tab in the Foundry portal turns the raw trace stream into the four numbers you actually check daily:

| Metric | What a bad number means | 
|---|---|
| Token usage | Verbose prompts/responses; a candidate for prompt or context-window optimization | 
| Latency (p50/p95) | Above ~10s often indicates model throttling, heavy tool calls, or network issues | 
| Run success rate | Below ~95% warrants investigating failed runs — this is your first-line SLO | 
| Evaluation scores | Built-in and custom evaluator scores sampled from continuous evaluation rules | 

It also surfaces **red team scan** results (adversarial testing for risks like data leakage or prohibited actions) and lets you configure **alerts** on latency, token usage, evaluation-score thresholds, or red-team findings — turning what would otherwise be a manual "check the traces tab" habit into an actual paging/notification system. All of this is still marked preview at the time of writing, which matters for anyone deciding whether to build a hard production dependency on the dashboard UI itself versus querying Application Insights directly (the latter is GA and stable; the dashboard is a convenience layer on top).

Single-agent tracing is a solved problem in most GenAI observability tooling at this point. Multi-agent tracing is not, and it's an area Microsoft is actively investing standards effort into. Foundry, in collaboration with Cisco Outshift, contributes to semantic conventions for multi-agent systems that extend the base OpenTelemetry GenAI agent/framework spans — the goal being a standard way to represent things like "which agent delegated to which sub-agent," "which agent owns a given tool call," and "how did a task hand off across an A2A boundary" as first-class span attributes rather than framework-specific ad hoc fields.

This matters because as you move from single-agent Prompt Agents toward orchestrated multi-agent systems (Sequential/Concurrent/Handoff/GroupChat/Magentic patterns via the Microsoft Agent Framework — see Day 7 of this series on the Workflows-to-Agent-Framework migration), the trace tree gets a lot deeper and a lot wider, and without standardized attribution, you end up with an opaque blob of nested LLM calls with no way to answer "which *agent* introduced this error," only "which *span*." Standardized multi-agent semantic conventions are what let a trace viewer render an agent-boundary-aware view (grouping spans by owning agent) instead of a flat operation tree.

Traces capture **exactly what makes them useful for debugging** — full inputs, outputs, and tool arguments — which is also exactly what makes them a data-exfiltration and compliance risk if mishandled:

Tracing cost is **not a Foundry line item** — it's an Application Insights / Log Analytics ingestion and retention cost, billed per GB ingested and per GB-month retained (verify current rates before budgeting; pricing changes over time — verify this stat before publishing). This has two practical implications:

If you're not deep in the Foundry ecosystem, or you need a single pane of glass across non-Foundry services too, you have real alternatives, because Foundry's tracing is standard OTel underneath:

The trade-off in choosing Foundry's native path is mostly about **lock-in vs. leverage**: you get tight integration with evaluation and monitoring at the cost of your telemetry backend being Application Insights specifically (rather than a vendor-neutral OTel backend of your choice) for the highest-value features like continuous evaluation.

Observability in Microsoft Foundry isn't a dashboard bolted on top of an agent platform — it's a data-plane decision: emit OpenTelemetry GenAI-convention spans from the runtime, store them in Application Insights, and let three different consumers (a human debugging in the Traces tab, an aggregation layer in the Monitoring dashboard, and an automated evaluator sampling live traffic) read from the same stream. That single-source-of-truth design is what makes it possible to go from "a customer says the agent was wrong" to "here is the exact span, with the exact tool arguments, that produced that answer" — and, increasingly, to catch that class of error automatically before a customer ever notices, via continuous evaluation.

If you're running Foundry agents in anything beyond a demo, connecting Application Insights and enabling server-side tracing is not optional infrastructure — it's the difference between debugging with a flashlight and debugging with a floor plan.

**Call to action**: If you haven't connected an Application Insights resource to your Foundry project yet, do it before your next deploy — it's a five-minute portal action that will save you hours the first time an agent misbehaves in production. Then come back tomorrow for Day 11 of this series.

*This is Day 10 of the Microsoft Foundry 100 Days / 100 Blogs series — one deep technical dive into a different corner of the Foundry ecosystem every day. Previous entries covered long-running agent resilience, the Responses vs. Invocations protocols, Autopilot identity, Foundry Local, the Agent Optimizer, MCP toolbox governance, the Workflows-to-Agent-Framework migration, Code Interpreter internals, and Voice Agents.*