{"slug": "per-agent-cost-tracking-for-multi-agent-ai-on-aws", "title": "Per-Agent Cost Tracking for Multi-Agent AI on AWS", "summary": "A developer built a read-only \"AWS Account Investigator\" multi-agent crew on Amazon Nova Pro that wires real per-span cost into every trace, reproducing three silent-waste patterns where agent runs complete successfully while billing roughly 1.4x what they should. The work targets the gap AWS's Well-Architected Agentic AI Lens calls its baseline \"Level 1,\" where agent costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. The author cites the MAST paper (arXiv:2503.13657), which hand-annotated 150 traces across 7 multi-agent systems and measured failure rates from 41% to 86.7%, noting many failures complete rather than crash.", "body_md": "Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance monitoring you already run: uptime, latency, error rate) says 200 OK, latency fine, everything green.\n\nAnd you were silently billed about 1.4x what you should have been.\n\nThat is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you *act* on them: catch the run that looks completely successful while it burns money in the middle. The paper \"Why Do Multi-Agent LLM Systems Fail?\" (MAST, arXiv:2503.13657) hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine.\n\nIn this article I build a small read-only \"AWS Account Investigator\" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account.\n\nIf you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff.\n\nI spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week.\n\nThis is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary (spans, traces, OpenTelemetry) is new, I define it the first time it shows up.\n\n**Contents**\n\nTraditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event.\n\nSo agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible.\n\nThis is not a fringe opinion. AWS's own Well-Architected [Agentic AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentcost05.html) frames the baseline state (its \"Level 1\") as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and \"teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing.\" The whole point of what follows is to move off Level 1: to make spending \"attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level,\" which is AWS's own words for the target.\n\nA quick vocabulary anchor, since the rest of the article leans on it. A *span* is one timed step with attributes attached (one LLM call, one tool call, one AWS read). A *trace* is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes (status, latency). Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans.\n\nAt one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways:\n\nThe theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more.\n\nOnce you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it.\n\nThere are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. [Traccia](https://github.com/traccia-ai/traccia-py) cleared both cleanly. The SDK is **open source, Apache-2.0 licensed**, and built on OpenTelemetry (OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend). The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it (I did, and the source-grounded critique later in this article is the result). It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me.\n\nThat is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over.\n\nWhy not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: \"cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution.\" Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not \"Traccia beats them.\" It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how.\n\nNothing exotic. Three things to run this yourself:\n\n`strands-agents`, `strands-agents-tools`, `traccia`, `boto3`). The repo pins the exact tested versions in `requirements.txt`.` bedrock:InvokeModel``iam/read-only-policy.json`; AWS's managed `SecurityAudit` + `ViewOnlyAccess` cover the reads, but you still add `us-east-1`, `amazon.nova-pro-v1:0`) under No Traccia account is required. With no API key it writes traces to a local file, which is the **$0** path used throughout this article.\n\nGetting it running is four commands:\n\n```\ngit clone https://github.com/simplynadaf/ai-agent-observability-aws.git\ncd ai-agent-observability-aws\npython3 -m venv .venv && . .venv/bin/activate\npip install -r requirements.txt\n```\n\nThen create the least-privilege policy once (with your own admin credentials) and attach it to whoever runs the crew:\n\n```\naws iam create-policy \\\n  --policy-name AgentObservabilityReadOnly \\\n  --policy-document file://iam/read-only-policy.json\n```\n\nBefore the crew, here is the smallest version of what \"wire cost into a span\" actually means, because that is the one non-obvious step. Traccia auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, so on those stacks you get most of this for free. It does not yet hook Strands or raw Bedrock, so you stamp the cost yourself. It is a short function, and one attribute name will bite you (more on that below).\n\n``` python\ndef stamp_llm_cost(span, result, model_id=\"amazon.nova-pro-v1:0\"):\n    usage = result.metrics.accumulated_usage\n    in_tok, out_tok = usage[\"inputTokens\"], usage[\"outputTokens\"]\n    cost = (in_tok / 1000 * 0.0008) + (out_tok / 1000 * 0.0032)  # Nova Pro, us-east-1\n\n    span.set_attribute(\"llm.model\", model_id)   # REQUIRED. wrong key = silently zero\n    span.set_attribute(\"span.type\", \"LLM\")\n    span.set_attribute(\"llm.usage.prompt_tokens\", in_tok)\n    span.set_attribute(\"llm.usage.completion_tokens\", out_tok)\n    span.set_attribute(\"llm.cost.usd\", round(cost, 6))\n```\n\nThat is the whole idea: read the token usage the SDK already gives you, turn it into dollars against real pricing, and attach it to the span. Everything else in this article is applying this same move across a multi-agent crew and then reading the numbers back. The full wiring (init, the per-agent identity, the tool spans) is in `src/crew.py` in the repo.\n\nThe crew runs on Amazon Nova Pro (`amazon.nova-pro-v1:0`) through AWS Strands Agents using the agents-as-tools pattern. A supervisor named `investigation_run` delegates to three specialist sub-agents. Each specialist is a real separation of concerns, owns several read-only tools, and every one of those tools opens its own live span, so in the dashboard you see the agent, then each AWS read nested under it with a real duration. Here is the full fleet and exactly what each agent does.\n\n`investigation_run`)\nThe orchestrator. It does not touch AWS directly; it reads the user's question, decides which specialists are in scope, delegates to them, and synthesizes one report. On its trace span it records `agent.delegated_to` (which specialists it called this run), and it stamps the shared `session.id` that ties the whole investigation together.\n\nThe delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call ONLY the specialist whose domain the user actually asked about. Ask only about cost and it delegates to the Cost Analyst alone, while Health & Ops and the Security Auditor never run. Ask only about security and only the Security Auditor fires. Only a whole-account question (\"what's running, any risks, and where is my spend going?\") lights up all three. This matters for the trace and the bill: `agent.delegated_to` shows exactly which specialists ran, and a scoped question costs a fraction of a full sweep because the agents you did not need never spent a token. You can watch this live in the control panel: a cost-only prompt lights up one agent and leaves the other two idle.\n\n| Task | Tool | AWS read-only API | \n|---|---|---|\n| Plan + delegate | `cost_analyst` ,`health_ops` ,`security_ops` | none directly (delegates) | \n\n*The supervisor's trace in Traccia: `agent.delegated_to` records which specialists ran this run, and the shared `session.id` links the four agents into one investigation.*\n\n`cost_analyst`)\nA read-only FinOps specialist. It builds a full spend picture with three tools, and all three show up as separate tool spans in its trace.\n\n| Task | Tool | AWS read-only API | \n|---|---|---|\n| Month-to-date total, month-end forecast, top 5 services | `cost_forecast` | `ce:GetCostAndUsage` ,`ce:GetCostForecast` | \n| Last full month's total for month-over-month change | `last_month_cost` | `ce:GetCostAndUsage` | \n| Daily cost series to catch a spike | `daily_cost_trend` | `ce:GetCostAndUsage` | \n\nWhat it reports on a real run: actual MTD spend, the account's forecasted month-end total, the top services by spend, the month-over-month direction and rough percentage, and the single most expensive day compared against the daily average (a possible spike).\n\n*The Cost Analyst trace: three Cost Explorer tool spans nested under the agent, each with its real read duration and the agent's own per-agent cost. This per-agent cost figure is exactly what turns a \"successful\" run into a caught overspend later.*\n\n`health_ops`)\nA read-only SRE specialist. It inventories the account and reads health signals with five tools, so it is usually the heaviest agent on input tokens (it chains the most reads).\n\n| Task | Tool | AWS read-only API | \n|---|---|---|\n| List running EC2 instances | `running_instances` | `ec2:DescribeInstances` | \n| Read CPU utilization per instance | `cpu_utilization` | `cloudwatch:GetMetricStatistics` | \n| Find unattached (idle) EBS volumes | `list_volumes` | `ec2:DescribeVolumes` | \n| Inventory Lambda functions | `list_functions` | `lambda:ListFunctions` | \n| Inventory S3 buckets | `list_buckets` | `s3:ListAllMyBuckets` | \n\nWhat it reports: running instances with their CPU, an inventory of volumes, functions, and buckets, and any notable health finding such as an unattached EBS volume.\n\n*The Health & Ops trace: five read-only tool spans and, on this run, the highest token count of the fleet (4,772) because it chains the most reads.*\n\n`security_ops`)\nA read-only security specialist. It runs four independent checks, each its own tool span.\n\n| Task | Tool | AWS read-only API | \n|---|---|---|\n| Security groups open to the internet (0.0.0.0/0) | `open_security_groups` | `ec2:DescribeSecurityGroups` | \n| MFA gaps on the root account and IAM users | `mfa_findings` | `iam:GetAccountSummary` ,`iam:ListUsers` ,`iam:ListMFADevices` | \n| S3 buckets missing a public-access block | `public_s3_buckets` | `s3:ListAllMyBuckets` ,`s3:GetPublicAccessBlock` | \n| Whether GuardDuty is enabled | `guardduty_enabled` | `guardduty:ListDetectors` | \n\nWhat it reports: each finding stated plainly with its risk, and it explicitly says so when a check comes back clean.\n\n*The Security Auditor trace: four independent read-only checks, each its own tool span with a real duration.*\n\nEach specialist stamps its own identity onto its trace span, so from a single crew run the dashboard shows four distinct agents with their own token and cost profiles, not one agent logged four times. Each agent runs as its own top-level trace, tied to the others by a shared `session.id`, and carries production ownership (type, owner, team) from a catalog file. Every call is a describe or get. There is no create, no modify, no delete. The worst thing this agent can do is read a bit too much, which, as you will see, is exactly the waste we want to catch.\n\nObservability comes from Traccia, an OpenTelemetry-native agent-observability SDK.\n\n```\npip install traccia\n```\n\nRuns $0 by default using a local file exporter. If you set `TRACCIA_API_KEY`, it pushes spans to app.traccia.ai. No key, no network, no cost. (The repo pins the exact tested version in `requirements.txt`; the prose stays unpinned so it does not age.)\n\nNova Pro pricing, pulled live from the AWS Price List API (effective 2026-08-01, us-east-1):\n\n| Token type | Price per 1K | \n|---|---|\n| Input | $0.0008 | \n| Output | $0.0032 | \n\nEvery dollar figure below is computed from real token counts against these two numbers.\n\nHere is the part most tutorials skip. Traccia auto-instruments several stacks out of the box (LangChain including BedrockChat, CrewAI, OpenAI Agents, and raw OpenAI/Anthropic/Gemini), and it ships a cost engine with a bundled pricing snapshot that covers Nova and Claude. But there is no Strands integration yet, and it does not hook raw Bedrock `converse` calls. So for this specific stack, Strands agents-as-tools calling Bedrock directly, you wire the cost in yourself. That is a fair amount of the value proposition for supported frameworks arriving for free, and real manual work for an unsupported one.\n\nStrands hands you the token usage after a run. You read it, compute the cost, and stamp it onto the span, which is exactly the `stamp_llm_cost` function from earlier. About 40 lines once you handle all four agents and the tool spans; the full version is in `src/crew.py`.\n\nThe gotcha cost me a confused afternoon, and reading the SDK source explained exactly why. Traccia's cost-annotating processor only computes cost for a span when three things are all true: `span.type` is `LLM` (or unset), an `llm.model` attribute is present, and both token counts are set. Miss any one and the processor simply returns, with no error and no warning. I first set `llm.request.model` (which felt more semantically correct) instead of `llm.model`, so the processor silently skipped every span, and the \"LLM Calls\" and \"Total Tokens\" tiles read zero while my spans clearly had tokens on them. Set `llm.model`, and the tiles light up. The forgiving fail is reasonable; the fact that it is invisible is the trap. A one-line debug log (\"skipping cost: no llm.model\") would have saved the afternoon.\n\nThe other question that always comes up: if the supervisor calls two sub-agents, and I sum everyone's tokens, am I counting the sub-agent tokens twice?\n\nI wrote `probes/probe_doublecount.py` to check instead of guessing. Strands runs each sub-agent in its own event loop with its own metrics object. A supervisor's `accumulated_usage` is exclusive of its sub-agents' tokens. So the arithmetic is clean:\n\n```\ncrew total = supervisor + sum(sub-agents)\n```\n\nNo subtraction, no overlap, no double-count. Verified, not assumed.\n\nOnce the bridge is in, you get per-agent cost. But here is a design decision worth being explicit about, because most demos hand-wave it: how do you model a supervisor and its specialists in a trace?\n\nYou have two reasonable options. You can nest everything under one trace (supervisor is the root, sub-agents are child spans). Or you can give **each agent its own top-level trace** and tie them together with a shared `session.id`. I went with the second, because it is what a real production fleet looks like: the Cost Analyst, Health & Ops, and Security Auditor are independently owned, independently operated services. On the Traces page they show up as their own executions, each with its own cost, tokens, and duration; \"Group by session\" folds them back into one investigation when you want the whole picture.\n\n```\nsession 4f4c1ade...  (one investigation, four independent traces)\n\n  investigation_run   AWS Account Investigator   $0.006   delegated -> 3\n  cost_analyst        Cost Analyst               $0.004\n  health_ops          Health & Ops               $0.005   (highest total tokens: 4,772)\n  security_ops        Security Auditor           $0.005\n                                          crew total   $0.021\n```\n\n(These are the real per-agent figures from the exported run shown in the trace screenshots above, rounded to the dashboard's own cost tiles; they shift run to run with token usage. The crew total is the `investigation_workflow` roll-up span, which equals the supervisor's own synthesis plus the three sub-agents, no double-count. Health & Ops carries the highest token count because it chains the most reads, while the supervisor costs about the same because it writes the long final synthesis. The CLEAN and CONTEXT BLOAT numbers later in the article come from separate, labeled runs, so do not expect them to tie back to this one.)\n\nEach agent's own trace still nests its tools underneath it (agent -> `tool:running_instances` -> the real boto3 call), so you keep the drill-down without pretending four separate services are one call stack. On the Traces page, \"Group by session\" folds all four agents from one run back into a single investigation, so you can move between the fleet view and the per-agent view without losing either.\n\n(This is a different run from the CLEAN baseline used later; token counts and therefore dollars shift run to run. The point is the per-agent breakdown, not the absolute number.)\n\nGood. Useful. Per-agent cost on its own is becoming common. The reason it matters here is not the number itself but the data model underneath it: once every step carries cost, tokens, cycle count, and an owning agent, you can build the thing that is still rare, which is catching a run that overspends while looking perfectly healthy. That is what the primitives let you build next.\n\nA few things here are easy to get subtly wrong, and I hit them in roughly this order over a couple of days before the trace design held. First, all four agents come from a *single* crew run in a *single* process. Traccia bakes the agent identity into the OpenTelemetry resource at init, which is process-level, so my first version labeled every trace with one agent name: three identical \"AWS Account Investigator\" rows in the dashboard. Reading the SDK's enrichment processor showed that a span-level `agent.id` / `agent.name` attribute takes precedence over the process default, so stamping each agent's span with its own identity makes it show up as its own agent. Static ownership (type, owner, team, org) comes from an `agent_config.json` catalog the SDK auto-discovers, so the dashboard shows a real fleet with owners and teams, not four anonymous rows. No extra processes, no fake agents.\n\nSecond, separate traces need a real correlation key or they look disconnected. Every agent stamps the run's `session.id`, and the supervisor additionally records `agent.delegated_to` (which specialists it called this run). That is the explicit link that makes four independent traces read as one orchestrated investigation.\n\nThird, the \"each agent is its own trace\" bit did not happen by wishing, and this one cost me a rebuild. Traccia's `span_scope(parent=None)` still inherits the *current* span if one is active, so my agents silently collapsed back into one trace until I detached the OpenTelemetry context before starting each agent's span. One small helper, verified by counting distinct trace IDs in the exported spans.\n\nFourth, the first time I looked at the timeline every tool span was 0ms, because I was reconstructing tool spans after the fact from the metrics object. The fix was to wrap the real boto3 call in a live span while it runs, so the timeline shows each AWS read's true duration. A 0ms bar is the kind of thing that makes a viewer distrust the whole trace, and it is worth chasing down. Then a subtler follow-on bit me: those live tool spans inherited the process-level default identity, so every tool bucketed under the supervisor and the specialists looked trace-thin. I had to stamp each tool span with its calling agent's identity too. Nothing about that was in the docs; I found it by parsing the exported `traces.jsonl` and noticing the `agent.id` was wrong.\n\nOne more touch that reads as production, not demo: each agent records both `agent.tools_available` (the full toolset it was granted) and `agent.tools_called` (what the model used this run). On this run, Cost Analyst had two tools available (`month_to_date_cost` and `cost_forecast`) and used one; Health & Ops had five and used all five. That gap is not a bug to hide, it is real information. \"Has two, used one\" is exactly the kind of thing you want visible when you are deciding whether an agent is over-provisioned.\n\nHere is my disclaimer up front. I saw versions of all three of these in real runs while building the crew, then engineered them in `src/waste_demo.py` to trigger reliably so you can watch them on demand instead of waiting for a bad run. That reproduction is on purpose. LLM output is non-deterministic, so in production the same patterns show up on their own, just not on a schedule you can demo. And critically: detection here is delta-vs-baseline, not magic absolute thresholds. That is how real regression detection works. You capture a known-good run, then flag runs that deviate. Every number below is real Nova Pro token usage from a representative run. Your numbers will vary; the ratios are what hold.\n\nFirst, the clean baseline. This is the \"known good\" I compare everything against.\n\n```\nCLEAN baseline (one exported run): crew total ~ $0.0083\n  health_ops: 2,274 input / 199 output / 3 cycles\n```\n\nThe agent gets stuck re-reasoning and re-reading the same things.\n\n```\nRUNAWAY LOOP: ~1.2x baseline ($0.0100 vs $0.0083)\n  health_ops ran 4 cycles (baseline: 3)\n  re-read cpu_utilization twice (baseline: once)\n  health_ops input tokens ~1.6x (3,557 vs 2,274)\n```\n\nSame final answer. The APM span is 200 OK. What catches it: `agent.cycle_count` and `tool.call_count`. The agent looped more than its baseline and called the same tool repeatedly. No single number is \"wrong.\" The delta is wrong.\n\nMilder, sneakier. The agent calls a tool it already has the answer for.\n\n```\nREDUNDANT TOOL CALLS: ~1.03x baseline ($0.0085 vs $0.0083)\n  running_instances called 3x (baseline: 1x)\n```\n\nA few percent on one run is the kind of thing you never notice. Multiply it across thousands of daily runs and it is a line item. The signal: `tool.call_count` for `running_instances` jumped from 1 to 3. Only visible per-tool, per-agent, and easy to miss precisely because the dollar delta is so small on a single run.\n\nThe agent drags too much context into its prompts. Every extra token in gets paid for, and it cascades.\n\n```\nCONTEXT BLOAT: ~1.4x baseline ($0.0119 vs $0.0083)\n  health_ops input tokens elevated, output nearly 4x (803 vs 199)\n  supervisor synthesis cost rises too, bloat cascades\n```\n\nThis is the meanest one because it compounds. The sub-agent's bloat feeds a bigger blob to the supervisor, whose own synthesis cost then rises too (on this run the supervisor jumped from $0.0032 to $0.0044). The signal: `llm.usage.prompt_tokens` and `llm.cost.usd` per agent. You watch prompt tokens creep up where the work did not.\n\n`src/compare.py` puts CLEAN next to BLOAT side by side.\n\n|  | CLEAN | CONTEXT BLOAT | \n|---|---|---|\n| Final answer | Correct | Correct | \n| Crew total cost | $0.0083 | $0.0119 | \n| Delta | - | +43% (~1.4x) | \n| APM status | 200 OK | 200 OK | \n\nTwo runs. Both return the right answer. Both are green in any latency-and-errors dashboard. The only place the extra 43% shows up is in the trace, on the per-agent cost attribute you stamped yourself. In the Traccia dashboard this is the moment the tool earns its place: two runs sit side by side, both \"successful,\" and the per-agent cost column is where the bloated one gives itself away. That is the whole argument for agent-native observability in one table.\n\nThe detection logic is not clever. It is a delta check: for each agent, compare this run against the baseline and flag three things. More cycles than baseline means a possible runaway loop. Prompt tokens more than 1.25x baseline means possible context bloat. Any tool called more times than baseline means a possible redundant call. That is the whole detector, about fifteen lines in `src/compare.py`.\n\nThe intelligence is in having the baseline and the per-agent attributes to compare against. The trace is what makes those attributes exist.\n\nThis detector broke once during the build, and it is a good example of how instrumentation and detection are coupled. When I switched tools to emit one live span per call (the 0ms fix above), the redundant-call check stopped working, because it had been reading a `call_count` attribute off a single reconstructed span that no longer existed. I had to change it to count span occurrences per tool name instead. The lesson that stuck: change how you record, and you can silently break how you detect. The baseline caught it, which is the whole point.\n\nThis is the panel you saw in the video above. Traces are the source of truth, but a wall of span JSON is not how you show a crew to a teammate. So the repo ships a small live control panel: a single-page UI that runs the real crew and animates the investigation as it happens.\n\nYou type a prompt into a command console, hit Investigate, and the view scrolls down to a graph of the crew. The supervisor sits at the top and the three specialists fan out below it, connected by wires. As the run streams, each agent lights up like a traffic signal: idle, then running (with a live activity line, \"Reading Cost Explorer\", \"Scanning security groups\"), then done, and the report reveals at the bottom. Every agent card shows the AWS services it touches as small chips, so a viewer can see at a glance that Cost Analyst reads Cost Explorer and the forecast, Health & Ops reads EC2/EBS/Lambda/S3/CloudWatch, and Security Auditor checks security groups, IAM, S3, and GuardDuty.\n\nThe panel has two modes. **Live** runs the real crew: real Nova Pro calls, real read-only AWS reads, real dollars on the trace, about thirteen seconds. **Replay** animates a saved run from a committed trace file, deterministically and for free, so you can rehearse the visual as many times as you want without spending a token. Both drive the exact same UI from the same event stream; the only difference is whether the events come from a fresh Bedrock run or a recorded one.\n\nThe backend is a small FastAPI app that streams the crew's lifecycle as Server-Sent Events. The important part is that the UI is a thin viewer over the same telemetry the trace records; it is not a second, hand-maintained source of truth. What the graph shows is what the crew did.\n\nYou do not need a paid plan or a live AWS bill to try this. The whole thing runs locally with the file exporter and read-only AWS credentials.\n\nThe permission surface is deliberately small: every action is a `Get`, `List`, or `Describe`, across Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, and GuardDuty. There is no create, no modify, no delete anywhere in the toolset. The full policy JSON is in the repo README; if you would rather not hand-roll it, AWS's managed `SecurityAudit` and `ViewOnlyAccess` policies cover the same set. Attach it, invoke Nova Pro through Strands, and you have a crew that can look but never touch. To send traces to the hosted dashboard, set `TRACCIA_API_KEY`; leave it unset and everything writes to a local file. Same spans either way.\n\nThe read-only shape is the same for every tool: wrap the real boto3 `describe`/` get`/` list` call in a live span so its duration in the trace is the true AWS read time, return the fields you need, touch nothing. `src/tools.py` in the repo has all seven; they are all this shape.\n\nI shipped a real crew against this SDK and read its source to understand the behavior, so here is the assessment grounded in that, not in the marketing page.\n\nWhat is genuinely good:\n\n`TRACCIA_API_KEY` and the same spans push to the hosted dashboard. Same spans either way, which made local development and CI painless.`agent.id` and `agent.name` override the process default, which is precisely what let a single-process crew render as a four-agent fleet with real per-agent cost. That is a thoughtful design decision, not an accident.\nWhere it made me work, and where it could be better:\n\n`llm.model` is missing or `span.type` is not `LLM`. That forgiving behavior is defensible, but the silence cost me an afternoon of a zeroed dashboard. A debug log on skip would fix it outright, and it is the kind of small papercut an early-stage tool usually closes fast.`span_scope(parent=None)` still inherits the current context (so separate agent traces silently merge unless you detach first), and `span_scope` is not a context manager (you call `.end()` yourself). Neither is obvious from the docs today.\nNet: for supported frameworks you get a lot for free, and even off the beaten path the OTel foundation and the cost/identity model are solid. The capability is there; the polish that is missing is mostly documentation and a few developer-experience papercuts, which is exactly what you would expect from a product at this stage.\n\nI want to be straight about the limits, because that is the whole point of this article.\n\n`src/waste_demo.py` versions just make them fire on cue. LLM output is non-deterministic. In real life these patterns appear on their own, just not on a schedule you can demo.`session.id`. That is a deliberate choice to match how a real fleet is owned and operated. If you prefer one nested trace per run, keep the supervisor as the parent instead of detaching the context. Both are valid; pick the one that matches how your team reasons about the system.\nThe takeaway is not \"buy an observability tool.\" It is that a correct-looking answer tells you nothing about whether the run was efficient, and the only place the truth lives is in the trace, on attributes you have to put there on purpose.\n\n**What is AI agent observability, and how is it different from LLM monitoring?**\n\nLLM monitoring usually watches one model call: latency, errors, maybe token count. Agent observability watches a whole reasoning session: how many cycles an agent took, which tools it called, the cost of each step, and, in a multi-agent crew, which agent did what. Agent failures show up across a multi-step chain, not on a single call, so you need the full trace to see them.\n\n**How do I track per-agent cost on Amazon Bedrock?**\n\nBedrock returns token usage after each call. You multiply input and output tokens by the model's per-1K price (for Nova Pro in us-east-1, $0.0008 in and $0.0032 out) and attach that dollar figure to the trace span for the agent that made the call. That is the `stamp_llm_cost` function in this article. AWS's own tag-based cost allocation in Cost Explorer works at the account and tag level; per-agent, per-reasoning-cycle attribution is what the trace adds on top.\n\n**Can AWS Cost Explorer show per-agent cost by itself?**\n\nNot on its own. Per AWS's Well-Architected Agentic AI Lens, the default state is that costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. Tag-based allocation plus AgentCore Observability improves this, but per-agent and per-reasoning-phase attribution comes from instrumenting the trace, which is what this build does.\n\n**Why does my multi-agent app cost more than I expected even when it works?**\n\nBecause a correct answer is not a cheap answer. Agents can loop an extra reasoning cycle, re-call a tool they already have the answer for, or drag bloated context from step to step. None of that returns an error; it just adds tokens. The overspend shows up on the bill, not in a latency-and-errors dashboard, which is the \"silent waste\" this article is about.\n\n**Do I need a paid tool or an AWS account to try this?**\n\nNo. The whole build runs at $0 locally: the Traccia SDK writes traces to a local file with no API key, and the AWS reads use read-only credentials (or the committed replay run, which needs no AWS access at all). You only need a Bedrock model grant if you want to run the live crew against your own account.\n\n**Is Traccia open source?**\n\nThe SDK ([traccia-py](https://github.com/traccia-ai/traccia-py)) is open source under Apache-2.0 and built on OpenTelemetry, so the spans are standard OTel and you are not locked in. The hosted dashboard at [traccia.ai](https://traccia.ai) is the optional commercial part; you only reach for it when you want the visualization.\n\n**Does Traccia support AWS Strands Agents out of the box?**\n\nNot at the time of writing. It auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, but not Strands or raw Bedrock `converse`, so on this stack you stamp cost onto the span yourself (about 40 lines). On a supported framework, most of that is automatic.\n\nThe full code (crew, tools, waste demos, the live control panel, the compare view, and the double-count probe) is on GitHub: **[ai-agent-observability-aws](https://github.com/simplynadaf/ai-agent-observability-aws)**. There is also a live replay of a run you can click through in the browser: **[https://simplynadaf.github.io/ai-agent-observability-aws/](https://simplynadaf.github.io/ai-agent-observability-aws/)**. Clone it, run `python -m src.waste_demo` with local export, and watch a perfect answer cost you ~1.4x. Then go instrument your own agents before your bill does the teaching for you.\n\nTo put Traccia under your own agents, the on-ramp is deliberately short and free:\n\n`pip install traccia`. With no API key it writes traces to a local file, so you can see spans at $0 before you sign up for anything. Source and docs: `stamp_llm_cost` pattern above (or get it for free if you are on LangChain, CrewAI, or the OpenAI/Anthropic/Gemini clients, which Traccia auto-instruments).\nIf you build something with it, tell me what silent waste you found. That is the interesting part.\n\n*Follow me for more on AWS architecture, DevOps, and AI Infrastructure:*\n\n[Portfolio](https://sarvarnadaf.com) | [LinkedIn](https://www.linkedin.com/in/sarvar04/) | [Dev.to](https://dev.to/sarvar_04) | [YouTube](https://www.youtube.com/@sarvar-nadaf) | [Email](mailto:simplynadaf@gmail.com) | [AWS Builder Center](https://builder.aws.com/community/@sarvar) | [X](https://x.com/SarvarN_04)", "url": "https://wpnews.pro/news/per-agent-cost-tracking-for-multi-agent-ai-on-aws", "canonical_source": "https://dev.to/sarvar_04/per-agent-cost-tracking-for-multi-agent-ai-on-aws-10eg", "published_at": "2026-09-23 13:41:46+00:00", "updated_at": "2026-09-23 13:58:41.741897+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["AWS", "Amazon Nova Pro", "Amazon Bedrock", "AWS Well-Architected Agentic AI Lens", "Cost Explorer", "OpenTelemetry", "MAST"], "alternates": {"html": "https://wpnews.pro/news/per-agent-cost-tracking-for-multi-agent-ai-on-aws", "markdown": "https://wpnews.pro/news/per-agent-cost-tracking-for-multi-agent-ai-on-aws.md", "text": "https://wpnews.pro/news/per-agent-cost-tracking-for-multi-agent-ai-on-aws.txt", "jsonld": "https://wpnews.pro/news/per-agent-cost-tracking-for-multi-agent-ai-on-aws.jsonld"}}