{"slug": "agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the", "title": "AgentCore Evaluations: How AWS Built a Framework-Agnostic Eval Layer Using OpenTelemetry as the Contract", "summary": "AWS has launched AgentCore Evaluations, a framework-agnostic evaluation layer for AI agents that uses OpenTelemetry telemetry as the contract. The service scores agents built with LangGraph, LlamaIndex, OpenAI Agents SDK, and other frameworks by analyzing spans for tool calls, LLM calls, and retrieval, without requiring framework-specific integrations. It computes metrics like task success rate, tool accuracy, response quality, latency, and cost, storing results in a time-series database for 90 days.", "body_md": "Amazon Bedrock AgentCore Evaluations solves a real problem: you build agents on LangGraph, your teammate uses LlamaIndex, and the platform team is experimenting with the OpenAI Agents SDK. Every framework has its own evaluation story, and none of them talk to each other.\n\nAWS's answer is to treat OpenTelemetry telemetry as the evaluation contract. If your agent emits the right spans and attributes, AgentCore can score it without knowing which framework you used. The service works with LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK, and Strands Agents. It also works with custom stacks, as long as you instrument them correctly.\n\nThis is the first major cloud vendor to decouple agent evaluation from framework choice using a telemetry-based contract. Here's how the plumbing works.\n\n##\nThe OpenTelemetry Contract\n\nAgentCore Evaluations expects agents to emit structured telemetry in OpenTelemetry format. The service looks for specific span types and attributes:\n\n-\n**Agent spans**: Top-level execution context for a single agent invocation\n-\n**Tool call spans**: Individual tool invocations, including input, output, and latency\n-\n**LLM spans**: Model calls with prompt, response, token counts, and model ID\n-\n**Retrieval spans**: Vector search or knowledge base queries\n\nEach span must include semantic attributes that map to evaluation dimensions. For example:\n\nThe key insight is that AgentCore doesn't care about your framework's internal state machine or graph structure. It only cares about the observable events: what tools were called, what the LLM said, how long things took, and whether they succeeded.\n\n##\nFramework Adapters and Instrumentation Gaps\n\nMost popular frameworks already emit some OpenTelemetry telemetry, but the coverage varies:\n\n| Framework |\nNative OTel Support |\nInstrumentation Gap |\nWorkaround |\n| LangGraph |\nPartial (LangSmith integration) |\nMissing tool success/failure attributes |\nManual span enrichment |\n| LlamaIndex |\nGood (built-in OTel exporter) |\nInconsistent span naming |\nSpan processor to normalize |\n| OpenAI Agents SDK |\nMinimal |\nNo tool spans by default |\nWrap tool calls with custom tracer |\n| Claude Agent SDK |\nNone |\nEverything |\nFull manual instrumentation |\n| Custom stacks |\nNone |\nEverything |\nBuild from scratch |\n\nIf your framework doesn't emit the required telemetry, you have three options:\n\n-\n**Manual instrumentation**: Wrap your agent code with OpenTelemetry API calls\n-\n**Auto-instrumentation**: Use OpenTelemetry's auto-instrumentation libraries for HTTP, database, and LLM calls\n-\n**Span processors**: Intercept and enrich spans after they're created but before they're exported\n\nAWS doesn't provide framework-specific shims. You're responsible for making sure your agent emits the right telemetry shape. The documentation includes example instrumentation for each supported framework, but you'll need to adapt it to your specific agent architecture.\n\n##\nEvaluation Metrics and Scoring\n\nOnce AgentCore receives telemetry, it computes metrics across several dimensions:\n\n-\n**Task success rate**: Percentage of agent runs that completed without errors\n-\n**Tool accuracy**: Whether the agent called the right tools in the right order\n-\n**Response quality**: LLM-as-judge scoring of final outputs against ground truth\n-\n**Latency**: P50, P95, and P99 for agent runs, tool calls, and LLM calls\n-\n**Cost**: Token usage and estimated inference cost per run\n\nThe service stores evaluation results in a time-series database (likely Amazon Timestream, though AWS doesn't specify). You can query results via the AgentCore API or view them in the AWS console.\n\nRetention is 90 days by default. After that, you need to export results to S3 if you want long-term storage.\n\n##\nDeployment Shape and Data Flow\n\nHere's the typical data flow:\n\n- Your agent runs in your AWS account (Lambda, ECS, EC2, or on-prem)\n- The OpenTelemetry SDK batches spans and exports them to the AWS Distro for OpenTelemetry (ADOT) Collector\n- The ADOT Collector forwards spans to AgentCore Evaluations via AWS PrivateLink\n- AgentCore processes spans, computes metrics, and stores results\n- You query results via the AgentCore API or console\n\nThe ADOT Collector runs as a sidecar container or daemon process. It handles batching, retries, and credential management. You configure it with a YAML file:\n\nThe `agentcore`\n\nexporter is a custom plugin that ships with ADOT. It handles authentication via IAM roles and forwards spans to the AgentCore service.\n\n##\nLatency and Cost Overhead\n\nShipping telemetry to AgentCore adds latency and cost:\n\n-\n**Latency**: 10-50ms per agent run, depending on span volume and network conditions. The ADOT Collector batches spans, so the overhead is amortized across multiple runs.\n-\n**Cost**: AgentCore charges per span ingested. AWS hasn't published pricing yet, but expect $0.10-$0.50 per million spans based on similar services (X-Ray, CloudWatch Logs Insights).\n\nIf you're running high-throughput agents (thousands of runs per second), the cost can add up. You can reduce it by:\n\n- Sampling telemetry (e.g., only export 10% of runs)\n- Filtering spans (e.g., only export tool calls and LLM calls, skip internal framework spans)\n- Running evals asynchronously (export telemetry to S3, process in batch)\n\nFor local development or low-stakes testing, you can run evals in-process using the AgentCore SDK. It computes metrics locally without shipping telemetry to AWS. This is faster and cheaper, but you lose the centralized dashboard and historical comparison.\n\n##\nSecurity Boundaries\n\nAgentCore Evaluations runs in AWS's account, not yours. Your telemetry data crosses an account boundary, which raises two questions:\n\n-\n**Data residency**: Spans are stored in the AWS region you specify, but they're not in your VPC. If you have strict data residency requirements, you'll need to run your own evaluation stack.\n-\n**Sensitive data**: Spans can include tool inputs, LLM prompts, and user queries. If these contain PII or secrets, you need to scrub them before export. The ADOT Collector supports span processors that can redact attributes, but you have to configure them yourself.\n\nAWS encrypts spans in transit (TLS) and at rest (KMS). You can use customer-managed KMS keys if you want full control over encryption.\n\n##\nFailure Modes\n\nHere are the most likely failure modes:\n\n-\n**Missing telemetry**: If your agent doesn't emit the required spans or attributes, AgentCore can't compute metrics. The service doesn't fail gracefully. It just returns empty results.\n-\n**Schema drift**: If you change your agent's tool signatures or add new tools, the evaluation metrics may become inconsistent over time. You'll need to version your evaluation datasets and re-run historical evals.\n-\n**Collector downtime**: If the ADOT Collector crashes or loses network connectivity, spans are buffered in memory. If the buffer fills up, spans are dropped. You won't know about it unless you monitor the collector's own telemetry.\n-\n**Rate limiting**: AgentCore has undocumented rate limits on span ingestion. If you exceed them, spans are rejected with HTTP 429 errors. The ADOT Collector will retry, but you may lose data if the backlog grows too large.\n\n##\nTechnical Verdict\n\nUse AgentCore Evaluations if:\n\n- You run agents on multiple frameworks and want a single evaluation dashboard\n- You already use OpenTelemetry for observability and want to reuse the same telemetry pipeline\n- You need historical comparison and trend analysis across agent versions\n- You're willing to pay for a managed service and accept the latency/cost overhead\n\nAvoid it if:\n\n- You have strict data residency requirements or can't send telemetry outside your VPC\n- You run high-throughput agents and need sub-10ms latency\n- Your framework doesn't emit OpenTelemetry telemetry and you don't want to instrument it manually\n- You need custom evaluation metrics that AgentCore doesn't support (e.g., domain-specific accuracy measures)\n\nFor teams running heterogeneous agent stacks, the framework-agnostic contract is a real win. But you're trading flexibility for operational complexity. You'll need to manage the ADOT Collector, monitor span ingestion, and handle instrumentation gaps yourself.\n\n##\nSource Links", "url": "https://wpnews.pro/news/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the", "canonical_source": "https://dev.to/mech_app_ai/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-opentelemetry-as-the-4lc1", "published_at": "2026-08-27 00:06:58+00:00", "updated_at": "2026-08-27 00:48:19.235562+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["AWS", "Amazon Bedrock AgentCore", "OpenTelemetry", "LangGraph", "LlamaIndex", "OpenAI Agents SDK", "ADOT"], "alternates": {"html": "https://wpnews.pro/news/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the", "markdown": "https://wpnews.pro/news/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the.md", "text": "https://wpnews.pro/news/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the.txt", "jsonld": "https://wpnews.pro/news/agentcore-evaluations-how-aws-built-a-framework-agnostic-eval-layer-using-as-the.jsonld"}}