Per-Agent Cost Tracking for Multi-Agent AI on AWS

A developer built a read-only "AWS Account Investigator" multi-agent crew on Amazon Nova Pro that wires real per-span cost into every trace, reproducing three silent-waste patterns where agent runs complete successfully while billing roughly 1.4x what they should. The work targets the gap AWS's Well-Architected Agentic AI Lens calls its baseline "Level 1," where agent costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. The author cites the MAST paper (arXiv:2503.13657), which hand-annotated 150 traces across 7 multi-agent systems and measured failure rates from 41% to 86.7%, noting many failures complete rather than crash.

Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard the application performance monitoring you already run: uptime, latency, error rate says 200 OK, latency fine, everything green. And you were silently billed about 1.4x what you should have been. That is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you act on them: catch the run that looks completely successful while it burns money in the middle. The paper "Why Do Multi-Agent LLM Systems Fail?" MAST, arXiv:2503.13657 hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine. In this article I build a small read-only "AWS Account Investigator" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account. If you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff. I spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week. This is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary spans, traces, OpenTelemetry is new, I define it the first time it shows up. Contents Traditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event. So agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible. This is not a fringe opinion. AWS's own Well-Architected Agentic AI Lens https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentcost05.html frames the baseline state its "Level 1" as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The whole point of what follows is to move off Level 1: to make spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level," which is AWS's own words for the target. A quick vocabulary anchor, since the rest of the article leans on it. A span is one timed step with attributes attached one LLM call, one tool call, one AWS read . A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes status, latency . Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans. At one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways: The theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more. Once you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it. There are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. Traccia https://github.com/traccia-ai/traccia-py cleared both cleanly. The SDK is open source, Apache-2.0 licensed , and built on OpenTelemetry OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend . The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it I did, and the source-grounded critique later in this article is the result . It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me. That is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over. Why not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: "cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution." Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not "Traccia beats them." It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how. Nothing exotic. Three things to run this yourself: strands-agents , strands-agents-tools , traccia , boto3 . The repo pins the exact tested versions in requirements.txt . bedrock:InvokeModel iam/read-only-policy.json ; AWS's managed SecurityAudit + ViewOnlyAccess cover the reads, but you still add us-east-1 , amazon.nova-pro-v1:0 under No Traccia account is required. With no API key it writes traces to a local file, which is the $0 path used throughout this article. Getting it running is four commands: git clone https://github.com/simplynadaf/ai-agent-observability-aws.git cd ai-agent-observability-aws python3 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt Then create the least-privilege policy once with your own admin credentials and attach it to whoever runs the crew: aws iam create-policy \ --policy-name AgentObservabilityReadOnly \ --policy-document file://iam/read-only-policy.json Before the crew, here is the smallest version of what "wire cost into a span" actually means, because that is the one non-obvious step. Traccia auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, so on those stacks you get most of this for free. It does not yet hook Strands or raw Bedrock, so you stamp the cost yourself. It is a short function, and one attribute name will bite you more on that below . python def stamp llm cost span, result, model id="amazon.nova-pro-v1:0" : usage = result.metrics.accumulated usage in tok, out tok = usage "inputTokens" , usage "outputTokens" cost = in tok / 1000 0.0008 + out tok / 1000 0.0032 Nova Pro, us-east-1 span.set attribute "llm.model", model id REQUIRED. wrong key = silently zero span.set attribute "span.type", "LLM" span.set attribute "llm.usage.prompt tokens", in tok span.set attribute "llm.usage.completion tokens", out tok span.set attribute "llm.cost.usd", round cost, 6 That is the whole idea: read the token usage the SDK already gives you, turn it into dollars against real pricing, and attach it to the span. Everything else in this article is applying this same move across a multi-agent crew and then reading the numbers back. The full wiring init, the per-agent identity, the tool spans is in src/crew.py in the repo. The crew runs on Amazon Nova Pro amazon.nova-pro-v1:0 through AWS Strands Agents using the agents-as-tools pattern. A supervisor named investigation run delegates to three specialist sub-agents. Each specialist is a real separation of concerns, owns several read-only tools, and every one of those tools opens its own live span, so in the dashboard you see the agent, then each AWS read nested under it with a real duration. Here is the full fleet and exactly what each agent does. investigation run The orchestrator. It does not touch AWS directly; it reads the user's question, decides which specialists are in scope, delegates to them, and synthesizes one report. On its trace span it records agent.delegated to which specialists it called this run , and it stamps the shared session.id that ties the whole investigation together. The delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call ONLY the specialist whose domain the user actually asked about. Ask only about cost and it delegates to the Cost Analyst alone, while Health & Ops and the Security Auditor never run. Ask only about security and only the Security Auditor fires. Only a whole-account question "what's running, any risks, and where is my spend going?" lights up all three. This matters for the trace and the bill: agent.delegated to shows exactly which specialists ran, and a scoped question costs a fraction of a full sweep because the agents you did not need never spent a token. You can watch this live in the control panel: a cost-only prompt lights up one agent and leaves the other two idle. | Task | Tool | AWS read-only API | |---|---|---| | Plan + delegate | cost analyst , health ops , security ops | none directly delegates | The supervisor's trace in Traccia: agent.delegated to records which specialists ran this run, and the shared session.id links the four agents into one investigation. cost analyst A read-only FinOps specialist. It builds a full spend picture with three tools, and all three show up as separate tool spans in its trace. | Task | Tool | AWS read-only API | |---|---|---| | Month-to-date total, month-end forecast, top 5 services | cost forecast | ce:GetCostAndUsage , ce:GetCostForecast | | Last full month's total for month-over-month change | last month cost | ce:GetCostAndUsage | | Daily cost series to catch a spike | daily cost trend | ce:GetCostAndUsage | What it reports on a real run: actual MTD spend, the account's forecasted month-end total, the top services by spend, the month-over-month direction and rough percentage, and the single most expensive day compared against the daily average a possible spike . The Cost Analyst trace: three Cost Explorer tool spans nested under the agent, each with its real read duration and the agent's own per-agent cost. This per-agent cost figure is exactly what turns a "successful" run into a caught overspend later. health ops A read-only SRE specialist. It inventories the account and reads health signals with five tools, so it is usually the heaviest agent on input tokens it chains the most reads . | Task | Tool | AWS read-only API | |---|---|---| | List running EC2 instances | running instances | ec2:DescribeInstances | | Read CPU utilization per instance | cpu utilization | cloudwatch:GetMetricStatistics | | Find unattached idle EBS volumes | list volumes | ec2:DescribeVolumes | | Inventory Lambda functions | list functions | lambda:ListFunctions | | Inventory S3 buckets | list buckets | s3:ListAllMyBuckets | What it reports: running instances with their CPU, an inventory of volumes, functions, and buckets, and any notable health finding such as an unattached EBS volume. The Health & Ops trace: five read-only tool spans and, on this run, the highest token count of the fleet 4,772 because it chains the most reads. security ops A read-only security specialist. It runs four independent checks, each its own tool span. | Task | Tool | AWS read-only API | |---|---|---| | Security groups open to the internet 0.0.0.0/0 | open security groups | ec2:DescribeSecurityGroups | | MFA gaps on the root account and IAM users | mfa findings | iam:GetAccountSummary , iam:ListUsers , iam:ListMFADevices | | S3 buckets missing a public-access block | public s3 buckets | s3:ListAllMyBuckets , s3:GetPublicAccessBlock | | Whether GuardDuty is enabled | guardduty enabled | guardduty:ListDetectors | What it reports: each finding stated plainly with its risk, and it explicitly says so when a check comes back clean. The Security Auditor trace: four independent read-only checks, each its own tool span with a real duration. Each specialist stamps its own identity onto its trace span, so from a single crew run the dashboard shows four distinct agents with their own token and cost profiles, not one agent logged four times. Each agent runs as its own top-level trace, tied to the others by a shared session.id , and carries production ownership type, owner, team from a catalog file. Every call is a describe or get. There is no create, no modify, no delete. The worst thing this agent can do is read a bit too much, which, as you will see, is exactly the waste we want to catch. Observability comes from Traccia, an OpenTelemetry-native agent-observability SDK. pip install traccia Runs $0 by default using a local file exporter. If you set TRACCIA API KEY , it pushes spans to app.traccia.ai. No key, no network, no cost. The repo pins the exact tested version in requirements.txt ; the prose stays unpinned so it does not age. Nova Pro pricing, pulled live from the AWS Price List API effective 2026-08-01, us-east-1 : | Token type | Price per 1K | |---|---| | Input | $0.0008 | | Output | $0.0032 | Every dollar figure below is computed from real token counts against these two numbers. Here is the part most tutorials skip. Traccia auto-instruments several stacks out of the box LangChain including BedrockChat, CrewAI, OpenAI Agents, and raw OpenAI/Anthropic/Gemini , and it ships a cost engine with a bundled pricing snapshot that covers Nova and Claude. But there is no Strands integration yet, and it does not hook raw Bedrock converse calls. So for this specific stack, Strands agents-as-tools calling Bedrock directly, you wire the cost in yourself. That is a fair amount of the value proposition for supported frameworks arriving for free, and real manual work for an unsupported one. Strands hands you the token usage after a run. You read it, compute the cost, and stamp it onto the span, which is exactly the stamp llm cost function from earlier. About 40 lines once you handle all four agents and the tool spans; the full version is in src/crew.py . The gotcha cost me a confused afternoon, and reading the SDK source explained exactly why. Traccia's cost-annotating processor only computes cost for a span when three things are all true: span.type is LLM or unset , an llm.model attribute is present, and both token counts are set. Miss any one and the processor simply returns, with no error and no warning. I first set llm.request.model which felt more semantically correct instead of llm.model , so the processor silently skipped every span, and the "LLM Calls" and "Total Tokens" tiles read zero while my spans clearly had tokens on them. Set llm.model , and the tiles light up. The forgiving fail is reasonable; the fact that it is invisible is the trap. A one-line debug log "skipping cost: no llm.model" would have saved the afternoon. The other question that always comes up: if the supervisor calls two sub-agents, and I sum everyone's tokens, am I counting the sub-agent tokens twice? I wrote probes/probe doublecount.py to check instead of guessing. Strands runs each sub-agent in its own event loop with its own metrics object. A supervisor's accumulated usage is exclusive of its sub-agents' tokens. So the arithmetic is clean: crew total = supervisor + sum sub-agents No subtraction, no overlap, no double-count. Verified, not assumed. Once the bridge is in, you get per-agent cost. But here is a design decision worth being explicit about, because most demos hand-wave it: how do you model a supervisor and its specialists in a trace? You have two reasonable options. You can nest everything under one trace supervisor is the root, sub-agents are child spans . Or you can give each agent its own top-level trace and tie them together with a shared session.id . I went with the second, because it is what a real production fleet looks like: the Cost Analyst, Health & Ops, and Security Auditor are independently owned, independently operated services. On the Traces page they show up as their own executions, each with its own cost, tokens, and duration; "Group by session" folds them back into one investigation when you want the whole picture. session 4f4c1ade... one investigation, four independent traces investigation run AWS Account Investigator $0.006 delegated - 3 cost analyst Cost Analyst $0.004 health ops Health & Ops $0.005 highest total tokens: 4,772 security ops Security Auditor $0.005 crew total $0.021 These are the real per-agent figures from the exported run shown in the trace screenshots above, rounded to the dashboard's own cost tiles; they shift run to run with token usage. The crew total is the investigation workflow roll-up span, which equals the supervisor's own synthesis plus the three sub-agents, no double-count. Health & Ops carries the highest token count because it chains the most reads, while the supervisor costs about the same because it writes the long final synthesis. The CLEAN and CONTEXT BLOAT numbers later in the article come from separate, labeled runs, so do not expect them to tie back to this one. Each agent's own trace still nests its tools underneath it agent - tool:running instances - the real boto3 call , so you keep the drill-down without pretending four separate services are one call stack. On the Traces page, "Group by session" folds all four agents from one run back into a single investigation, so you can move between the fleet view and the per-agent view without losing either. This is a different run from the CLEAN baseline used later; token counts and therefore dollars shift run to run. The point is the per-agent breakdown, not the absolute number. Good. Useful. Per-agent cost on its own is becoming common. The reason it matters here is not the number itself but the data model underneath it: once every step carries cost, tokens, cycle count, and an owning agent, you can build the thing that is still rare, which is catching a run that overspends while looking perfectly healthy. That is what the primitives let you build next. A few things here are easy to get subtly wrong, and I hit them in roughly this order over a couple of days before the trace design held. First, all four agents come from a single crew run in a single process. Traccia bakes the agent identity into the OpenTelemetry resource at init, which is process-level, so my first version labeled every trace with one agent name: three identical "AWS Account Investigator" rows in the dashboard. Reading the SDK's enrichment processor showed that a span-level agent.id / agent.name attribute takes precedence over the process default, so stamping each agent's span with its own identity makes it show up as its own agent. Static ownership type, owner, team, org comes from an agent config.json catalog the SDK auto-discovers, so the dashboard shows a real fleet with owners and teams, not four anonymous rows. No extra processes, no fake agents. Second, separate traces need a real correlation key or they look disconnected. Every agent stamps the run's session.id , and the supervisor additionally records agent.delegated to which specialists it called this run . That is the explicit link that makes four independent traces read as one orchestrated investigation. Third, the "each agent is its own trace" bit did not happen by wishing, and this one cost me a rebuild. Traccia's span scope parent=None still inherits the current span if one is active, so my agents silently collapsed back into one trace until I detached the OpenTelemetry context before starting each agent's span. One small helper, verified by counting distinct trace IDs in the exported spans. Fourth, the first time I looked at the timeline every tool span was 0ms, because I was reconstructing tool spans after the fact from the metrics object. The fix was to wrap the real boto3 call in a live span while it runs, so the timeline shows each AWS read's true duration. A 0ms bar is the kind of thing that makes a viewer distrust the whole trace, and it is worth chasing down. Then a subtler follow-on bit me: those live tool spans inherited the process-level default identity, so every tool bucketed under the supervisor and the specialists looked trace-thin. I had to stamp each tool span with its calling agent's identity too. Nothing about that was in the docs; I found it by parsing the exported traces.jsonl and noticing the agent.id was wrong. One more touch that reads as production, not demo: each agent records both agent.tools available the full toolset it was granted and agent.tools called what the model used this run . On this run, Cost Analyst had two tools available month to date cost and cost forecast and used one; Health & Ops had five and used all five. That gap is not a bug to hide, it is real information. "Has two, used one" is exactly the kind of thing you want visible when you are deciding whether an agent is over-provisioned. Here is my disclaimer up front. I saw versions of all three of these in real runs while building the crew, then engineered them in src/waste demo.py to trigger reliably so you can watch them on demand instead of waiting for a bad run. That reproduction is on purpose. LLM output is non-deterministic, so in production the same patterns show up on their own, just not on a schedule you can demo. And critically: detection here is delta-vs-baseline, not magic absolute thresholds. That is how real regression detection works. You capture a known-good run, then flag runs that deviate. Every number below is real Nova Pro token usage from a representative run. Your numbers will vary; the ratios are what hold. First, the clean baseline. This is the "known good" I compare everything against. CLEAN baseline one exported run : crew total ~ $0.0083 health ops: 2,274 input / 199 output / 3 cycles The agent gets stuck re-reasoning and re-reading the same things. RUNAWAY LOOP: ~1.2x baseline $0.0100 vs $0.0083 health ops ran 4 cycles baseline: 3 re-read cpu utilization twice baseline: once health ops input tokens ~1.6x 3,557 vs 2,274 Same final answer. The APM span is 200 OK. What catches it: agent.cycle count and tool.call count . The agent looped more than its baseline and called the same tool repeatedly. No single number is "wrong." The delta is wrong. Milder, sneakier. The agent calls a tool it already has the answer for. REDUNDANT TOOL CALLS: ~1.03x baseline $0.0085 vs $0.0083 running instances called 3x baseline: 1x A few percent on one run is the kind of thing you never notice. Multiply it across thousands of daily runs and it is a line item. The signal: tool.call count for running instances jumped from 1 to 3. Only visible per-tool, per-agent, and easy to miss precisely because the dollar delta is so small on a single run. The agent drags too much context into its prompts. Every extra token in gets paid for, and it cascades. CONTEXT BLOAT: ~1.4x baseline $0.0119 vs $0.0083 health ops input tokens elevated, output nearly 4x 803 vs 199 supervisor synthesis cost rises too, bloat cascades This is the meanest one because it compounds. The sub-agent's bloat feeds a bigger blob to the supervisor, whose own synthesis cost then rises too on this run the supervisor jumped from $0.0032 to $0.0044 . The signal: llm.usage.prompt tokens and llm.cost.usd per agent. You watch prompt tokens creep up where the work did not. src/compare.py puts CLEAN next to BLOAT side by side. | | CLEAN | CONTEXT BLOAT | |---|---|---| | Final answer | Correct | Correct | | Crew total cost | $0.0083 | $0.0119 | | Delta | - | +43% ~1.4x | | APM status | 200 OK | 200 OK | Two runs. Both return the right answer. Both are green in any latency-and-errors dashboard. The only place the extra 43% shows up is in the trace, on the per-agent cost attribute you stamped yourself. In the Traccia dashboard this is the moment the tool earns its place: two runs sit side by side, both "successful," and the per-agent cost column is where the bloated one gives itself away. That is the whole argument for agent-native observability in one table. The detection logic is not clever. It is a delta check: for each agent, compare this run against the baseline and flag three things. More cycles than baseline means a possible runaway loop. Prompt tokens more than 1.25x baseline means possible context bloat. Any tool called more times than baseline means a possible redundant call. That is the whole detector, about fifteen lines in src/compare.py . The intelligence is in having the baseline and the per-agent attributes to compare against. The trace is what makes those attributes exist. This detector broke once during the build, and it is a good example of how instrumentation and detection are coupled. When I switched tools to emit one live span per call the 0ms fix above , the redundant-call check stopped working, because it had been reading a call count attribute off a single reconstructed span that no longer existed. I had to change it to count span occurrences per tool name instead. The lesson that stuck: change how you record, and you can silently break how you detect. The baseline caught it, which is the whole point. This is the panel you saw in the video above. Traces are the source of truth, but a wall of span JSON is not how you show a crew to a teammate. So the repo ships a small live control panel: a single-page UI that runs the real crew and animates the investigation as it happens. You type a prompt into a command console, hit Investigate, and the view scrolls down to a graph of the crew. The supervisor sits at the top and the three specialists fan out below it, connected by wires. As the run streams, each agent lights up like a traffic signal: idle, then running with a live activity line, "Reading Cost Explorer", "Scanning security groups" , then done, and the report reveals at the bottom. Every agent card shows the AWS services it touches as small chips, so a viewer can see at a glance that Cost Analyst reads Cost Explorer and the forecast, Health & Ops reads EC2/EBS/Lambda/S3/CloudWatch, and Security Auditor checks security groups, IAM, S3, and GuardDuty. The panel has two modes. Live runs the real crew: real Nova Pro calls, real read-only AWS reads, real dollars on the trace, about thirteen seconds. Replay animates a saved run from a committed trace file, deterministically and for free, so you can rehearse the visual as many times as you want without spending a token. Both drive the exact same UI from the same event stream; the only difference is whether the events come from a fresh Bedrock run or a recorded one. The backend is a small FastAPI app that streams the crew's lifecycle as Server-Sent Events. The important part is that the UI is a thin viewer over the same telemetry the trace records; it is not a second, hand-maintained source of truth. What the graph shows is what the crew did. You do not need a paid plan or a live AWS bill to try this. The whole thing runs locally with the file exporter and read-only AWS credentials. The permission surface is deliberately small: every action is a Get , List , or Describe , across Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, and GuardDuty. There is no create, no modify, no delete anywhere in the toolset. The full policy JSON is in the repo README; if you would rather not hand-roll it, AWS's managed SecurityAudit and ViewOnlyAccess policies cover the same set. Attach it, invoke Nova Pro through Strands, and you have a crew that can look but never touch. To send traces to the hosted dashboard, set TRACCIA API KEY ; leave it unset and everything writes to a local file. Same spans either way. The read-only shape is the same for every tool: wrap the real boto3 describe / get / list call in a live span so its duration in the trace is the true AWS read time, return the fields you need, touch nothing. src/tools.py in the repo has all seven; they are all this shape. I shipped a real crew against this SDK and read its source to understand the behavior, so here is the assessment grounded in that, not in the marketing page. What is genuinely good: TRACCIA API KEY and the same spans push to the hosted dashboard. Same spans either way, which made local development and CI painless. agent.id and agent.name override the process default, which is precisely what let a single-process crew render as a four-agent fleet with real per-agent cost. That is a thoughtful design decision, not an accident. Where it made me work, and where it could be better: llm.model is missing or span.type is not LLM . That forgiving behavior is defensible, but the silence cost me an afternoon of a zeroed dashboard. A debug log on skip would fix it outright, and it is the kind of small papercut an early-stage tool usually closes fast. span scope parent=None still inherits the current context so separate agent traces silently merge unless you detach first , and span scope is not a context manager you call .end yourself . Neither is obvious from the docs today. Net: for supported frameworks you get a lot for free, and even off the beaten path the OTel foundation and the cost/identity model are solid. The capability is there; the polish that is missing is mostly documentation and a few developer-experience papercuts, which is exactly what you would expect from a product at this stage. I want to be straight about the limits, because that is the whole point of this article. src/waste demo.py versions just make them fire on cue. LLM output is non-deterministic. In real life these patterns appear on their own, just not on a schedule you can demo. session.id . That is a deliberate choice to match how a real fleet is owned and operated. If you prefer one nested trace per run, keep the supervisor as the parent instead of detaching the context. Both are valid; pick the one that matches how your team reasons about the system. The takeaway is not "buy an observability tool." It is that a correct-looking answer tells you nothing about whether the run was efficient, and the only place the truth lives is in the trace, on attributes you have to put there on purpose. What is AI agent observability, and how is it different from LLM monitoring? LLM monitoring usually watches one model call: latency, errors, maybe token count. Agent observability watches a whole reasoning session: how many cycles an agent took, which tools it called, the cost of each step, and, in a multi-agent crew, which agent did what. Agent failures show up across a multi-step chain, not on a single call, so you need the full trace to see them. How do I track per-agent cost on Amazon Bedrock? Bedrock returns token usage after each call. You multiply input and output tokens by the model's per-1K price for Nova Pro in us-east-1, $0.0008 in and $0.0032 out and attach that dollar figure to the trace span for the agent that made the call. That is the stamp llm cost function in this article. AWS's own tag-based cost allocation in Cost Explorer works at the account and tag level; per-agent, per-reasoning-cycle attribution is what the trace adds on top. Can AWS Cost Explorer show per-agent cost by itself? Not on its own. Per AWS's Well-Architected Agentic AI Lens, the default state is that costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. Tag-based allocation plus AgentCore Observability improves this, but per-agent and per-reasoning-phase attribution comes from instrumenting the trace, which is what this build does. Why does my multi-agent app cost more than I expected even when it works? Because a correct answer is not a cheap answer. Agents can loop an extra reasoning cycle, re-call a tool they already have the answer for, or drag bloated context from step to step. None of that returns an error; it just adds tokens. The overspend shows up on the bill, not in a latency-and-errors dashboard, which is the "silent waste" this article is about. Do I need a paid tool or an AWS account to try this? No. The whole build runs at $0 locally: the Traccia SDK writes traces to a local file with no API key, and the AWS reads use read-only credentials or the committed replay run, which needs no AWS access at all . You only need a Bedrock model grant if you want to run the live crew against your own account. Is Traccia open source? The SDK traccia-py https://github.com/traccia-ai/traccia-py is open source under Apache-2.0 and built on OpenTelemetry, so the spans are standard OTel and you are not locked in. The hosted dashboard at traccia.ai https://traccia.ai is the optional commercial part; you only reach for it when you want the visualization. Does Traccia support AWS Strands Agents out of the box? Not at the time of writing. It auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, but not Strands or raw Bedrock converse , so on this stack you stamp cost onto the span yourself about 40 lines . On a supported framework, most of that is automatic. The full code crew, tools, waste demos, the live control panel, the compare view, and the double-count probe is on GitHub: ai-agent-observability-aws https://github.com/simplynadaf/ai-agent-observability-aws . There is also a live replay of a run you can click through in the browser: https://simplynadaf.github.io/ai-agent-observability-aws/ https://simplynadaf.github.io/ai-agent-observability-aws/ . Clone it, run python -m src.waste demo with local export, and watch a perfect answer cost you ~1.4x. Then go instrument your own agents before your bill does the teaching for you. To put Traccia under your own agents, the on-ramp is deliberately short and free: pip install traccia . With no API key it writes traces to a local file, so you can see spans at $0 before you sign up for anything. Source and docs: stamp llm cost pattern above or get it for free if you are on LangChain, CrewAI, or the OpenAI/Anthropic/Gemini clients, which Traccia auto-instruments . If you build something with it, tell me what silent waste you found. That is the interesting part. Follow me for more on AWS architecture, DevOps, and AI Infrastructure: Portfolio https://sarvarnadaf.com | LinkedIn https://www.linkedin.com/in/sarvar04/ | Dev.to https://dev.to/sarvar 04 | YouTube https://www.youtube.com/@sarvar-nadaf | Email mailto:simplynadaf@gmail.com | AWS Builder Center https://builder.aws.com/community/@sarvar | X https://x.com/SarvarN 04