{"slug": "sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent", "title": "Sentry's Span Hierarchy Exposed a Silent Retry in My 5-Agent Pipeline. One Agent Took 22.6s, the Others Took 5.", "summary": "A developer built an AWS Security Posture Agent using five specialist AI agents on CrewAI with Amazon Bedrock Nova Pro. One agent took 22.6 seconds while others averaged 5-10 seconds; Sentry's trace waterfall revealed the SecurityScanner agent was overwhelmed by a 27KB JSON payload from the IAMAnalyzer tool, causing CrewAI's retry logic to fire. The fix limited IAM role analysis to the top 20 most active roles, reducing the payload and normalizing execution times.", "body_md": "*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*\n\nI built an **AWS Security Posture Agent**: five specialist AI agents that scan your AWS account for security misconfigurations, map findings to CIS benchmarks, score risk, and generate copy-paste fix commands.\n\nThe agents run sequentially on CrewAI with Amazon Bedrock Nova Pro as the LLM:\n\n```\n1. ResourceDiscovery    → inventories EC2, S3, Lambda, IAM, SGs, API GW, DynamoDB\n2. SecurityScanner      → finds open ports, public buckets, admin roles, insecure configs\n3. ComplianceChecker    → maps to CIS AWS Foundations Benchmark\n4. RiskScorer           → severity × blast radius × exploitability\n5. RemediationPlanner   → generates AWS CLI fix commands\n```\n\nEach agent has custom boto3 tools that make real AWS API calls against a live account with 90 IAM roles, 14 S3 buckets, 9 security groups, and 7 Lambda functions. Not test data. Real findings.\n\n**Five AI agents scan your AWS account. Find misconfigurations. Score risk. Get fix commands.**\n\nMost AWS accounts accumulate security debt silently. Open SSH ports from testing. S3 buckets without encryption. IAM roles with full admin access that nobody remembers creating. Manual audits miss things. AWS Config rules cost money. SecurityHub is noisy.\n\nThis agent scans your account in 60 seconds, finds real issues, maps them to CIS benchmarks, scores risk, and gives you the exact AWS CLI command to fix each one.\n\nOn a real AWS account (90 IAM roles, 14 S3 buckets, 9 security groups, 7 Lambda functions):\n\n```\n97 security findings\n├── CRITICAL: open SSH/RDP from 0.0.0.0/0, IAM users without MFA\n├── HIGH:     admin roles, unencrypted EBS, missing public access blocks\n├── MEDIUM:   deprecated Lambda runtimes, missing versioning, stale SGs\n└──\n```\n\n…Here's the full scan running against my AWS account. The scan portion is sped up 4x, results walkthrough is at normal speed:\n\nMy first instinct was to blame Bedrock latency. Every time something is slow you blame the LLM, right?\n\nThe **SecurityScanner agent** was taking 22.6 seconds while every other agent averaged 5-10 seconds. Without visibility into what was happening inside each agent's execution, I would have added `time.time()`\n\ncalls and guessed.\n\nSentry's trace waterfall told a different story. The real problem wasn't Python execution time or network latency. It was the LLM getting a context payload it couldn't process cleanly on the first attempt.\n\nThe root cause: my `IAMAnalyzer`\n\ntool was fetching all 90 IAM roles from the account (59 after filtering service-linked roles), serializing them into a 27KB JSON blob, and handing that entire payload to the LLM as tool output. The context window got overwhelmed. CrewAI's internal retry logic kicked in, burning tokens on a second attempt with even more context.\n\nOne tool. Wrong default. The entire pipeline suffered.\n\nPR with the fix:\n\nThe IAMAnalyzer tool was fetching every single role in the account (90 of them, 59 after filtering service-linked ones) and dumping the full details into a single JSON blob. That blob hit 26,980 characters. The LLM choked on it, CrewAI retried the task, and the SecurityScanner agent ended up taking 22.6s while every other agent finished in 5-10s.\n\nAdded Sentry spans to each agent and tool. The trace waterfall made it obvious: SecurityScanner was twice as wide as everything else. Drilled into the tool spans and saw `result_length_chars: 26980`\n\non `iam_analyzer`\n\nvs ~4000 on the other tools.\n\n7x output disproportion. That was the problem.\n\nThree things in `iam_analyzer.py`\n\n:\n\n`RoleLastUsed`\n\ndate, only analyze the top 20 most active ones. The rest are stale roles nobody has touched in months.The core change lives in `src/security_posture/tools/iam_analyzer.py`\n\n. Here's the before and after:\n\n**Before (the bug):**\n\n```\n# Fetches ALL roles without pagination limit\nroles = iam.list_roles(MaxItems=100)\nrole_details = []\n\nfor role in roles[\"Roles\"]:\n    role_name = role[\"RoleName\"]\n    if role.get(\"Path\", \"\").startswith(\"/aws-service-role/\"):\n        continue\n    # Analyzes every single role...\n    attached = iam.list_attached_role_policies(RoleName=role_name)\n    # ...builds massive JSON output\n```\n\nThis produces 26,980 characters of JSON for an account with 90 roles. The LLM chokes on it.\n\n**After (the fix):**\n\n```\n# FIX: Paginate and sort by relevance\nall_roles = []\npaginator = iam.get_paginator(\"list_roles\")\nfor page in paginator.paginate():\n    all_roles.extend(page[\"Roles\"])\n\n# Filter service-linked roles (31 roles, can't modify anyway)\nauditable_roles = [\n    r for r in all_roles\n    if not r.get(\"Path\", \"\").startswith(\"/aws-service-role/\")\n]\n\n# Sort by last used date (most active first)\nauditable_roles.sort(key=_last_used_sort_key, reverse=True)\n\n# Take only top 20 roles\nroles_to_analyze = auditable_roles[:max_roles]\n```\n\nPlus a token budget guard at the end:\n\n```\n# Token budget guard: truncate if output exceeds threshold\nif len(output) > 4000:\n    result[\"role_summary\"] = [\n        {\"role_name\": r[\"role_name\"], \"policies\": r.get(\"attached_policies\", [])}\n        for r in role_details\n    ]\n    result[\"note\"] = \"Role details truncated to stay within token budget\"\n    output = json.dumps(result, indent=2, default=str)\n```\n\nThree changes. Pagination, relevance sorting, and a safety valve. The SecurityScanner stopped retrying.\n\n**Results first:**\n\n| Metric | Before | After | Improvement |\n|---|---|---|---|\n| IAM tool output | 26,980 chars | 15,532 chars | 42% smaller |\n| SecurityScanner time | 22.6s | 17.8s | 21% faster |\n| IAM API calls | 59 | 20 | 66% fewer |\n| Total pipeline | 62.0s | 57.7s | 7% faster |\n| Security findings | 97 | 97 | No coverage loss |\n\nThe 21% improvement on the SecurityScanner came from the LLM completing analysis in a single pass instead of retrying.\n\n**How I found it:**\n\nThe fix itself is simple. The interesting part is how Sentry pointed me straight to it.\n\nWithout the trace waterfall, all I would have seen is \"pipeline takes 62 seconds.\" The trace said otherwise:\n\n`gen_ai.invoke_agent`\n\nspan`gen_ai.execute_tool`\n\nspan`iam_analyzer`\n\ntool span showed a `result_length_chars`\n\nof 26,980`security_group_analyzer`\n\nat 4,200 chars and `s3_config_checker`\n\nat 3,800 charsThe disproportion was the clue. If the tool output is 7x larger than its siblings, reduce it.\n\nI used Sentry's AI Agent Monitoring to instrument a multi-agent CrewAI pipeline from scratch. This isn't a web app or API. It's five autonomous AI agents making LLM calls and executing custom tools. Standard APM wouldn't help here.\n\nEvery pipeline run creates a Sentry transaction with this span hierarchy:\n\n```\nTransaction: \"Security Posture Scan\" (57s)\n├── gen_ai.invoke_agent: ResourceDiscovery\n│   └── gen_ai.execute_tool: aws_resource_scanner\n├── gen_ai.invoke_agent: SecurityScanner\n│   ├── gen_ai.execute_tool: security_group_analyzer\n│   ├── gen_ai.execute_tool: s3_config_checker\n│   ├── gen_ai.execute_tool: iam_analyzer\n│   ├── gen_ai.execute_tool: ec2_security_checker\n│   └── gen_ai.execute_tool: lambda_security_checker\n├── gen_ai.invoke_agent: ComplianceChecker\n├── gen_ai.invoke_agent: RiskScorer\n└── gen_ai.invoke_agent: RemediationPlanner\n```\n\nFor each agent (in `main.py`\n\n):\n\n```\nwith start_span(\n    op=\"gen_ai.invoke_agent\",\n    name=f\"invoke_agent {agent_name}\",\n) as agent_span:\n    agent_span.set_data(\"gen_ai.operation.name\", \"invoke_agent\")\n    agent_span.set_data(\"gen_ai.agent.name\", agent_name)\n    agent_span.set_data(\"gen_ai.request.model\", \"amazon.nova-pro-v1:0\")\n    agent_span.set_data(\"gen_ai.pipeline.name\", \"security-posture-scan\")\n\n    task_output = task_obj.execute_sync(\n        agent=task_obj.agent,\n        context=context,\n        tools=task_obj.agent.tools,\n    )\n\n    agent_span.set_data(\"duration_seconds\", round(elapsed, 2))\n    agent_span.set_data(\"output_length_chars\", len(str(task_output)))\n```\n\nFor each tool (decorator in `monitoring.py`\n\n):\n\n``` python\ndef trace_tool(tool_name: str):\n    def decorator(func):\n        @functools.wraps(func)\n        def wrapper(*args, **kwargs):\n            with start_span(\n                op=\"gen_ai.execute_tool\",\n                name=f\"execute_tool {tool_name}\",\n            ) as span:\n                span.set_data(\"gen_ai.tool.name\", tool_name)\n                result = func(*args, **kwargs)\n                span.set_data(\"result_length_chars\", len(result))\n                try:\n                    data = json.loads(result)\n                    if \"findings_count\" in data:\n                        span.set_data(\"findings_count\", data[\"findings_count\"])\n                except (json.JSONDecodeError, KeyError):\n                    pass\n                return result\n        return wrapper\n    return decorator\n```\n\nThe trace waterfall made the bottleneck visually obvious. The SecurityScanner span was nearly twice the width of any other agent, 22.6 seconds while others averaged 5-10s.\n\nInside it, the `iam_analyzer`\n\ntool span showed `result_length_chars: 26980`\n\nwhile `s3_config_checker`\n\nshowed 3,800 and `security_group_analyzer`\n\nshowed 4,200. The granularity goes down to individual boto3 calls. Every `GetPublicAccessBlock`\n\n, `ListRoles`\n\n, and `ListAttachedRolePolicies`\n\nshows up as its own span.\n\n**Before the fix** (SecurityScanner dominates the trace at 22.6s):\n\n**After the fix** (all agents proportional, no single bottleneck):\n\nThe `iam_analyzer`\n\noutput dropped from 26,980 chars to 15,532, and the SecurityScanner stopped triggering retry logic.\n\n| Feature | How I Used It |\n|---|---|\nDistributed Tracing |\nFull pipeline trace from start to final report |\nAI Agent Monitoring |\n`gen_ai.invoke_agent` spans for all 5 agents |\nTool Execution Tracing |\n`gen_ai.execute_tool` spans for all 5 boto3 tools |\nCustom Span Data |\nToken counts, output sizes, duration, finding counts |\nError Monitoring |\nException capture with `sentry_sdk.capture_exception()`\n|\nBreadcrumbs |\nAgent completion events via task callbacks |\nTransaction Metadata |\nModel name, pipeline config, agent count |\n\nStandard logging tells you an agent \"finished.\" It doesn't tell you which tool inside which agent returned a 27KB payload that triggered a retry you never asked for.\n\nWith five agents each making their own LLM calls and tool executions, you need span-level visibility. Sentry's `gen_ai.invoke_agent`\n\nand `gen_ai.execute_tool`\n\nconventions gave me exactly that. I could see the problem in the trace waterfall before I even looked at the code.\n\nThat's the difference between \"add some logging\" and actual AI observability.\n\nIf you're running multi-agent pipelines, what's your observability setup? Curious if anyone else has hit similar tool-output-size problems with CrewAI or LangGraph.\n\n*The agent found 97 real security findings in my AWS account. The fix is in production.*", "url": "https://wpnews.pro/news/sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent", "canonical_source": "https://dev.to/sarvar_04/sentrys-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent-took-226s-the-fb4", "published_at": "2026-07-24 05:48:09+00:00", "updated_at": "2026-07-24 06:01:42.706917+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools"], "entities": ["Sentry", "CrewAI", "Amazon Bedrock", "AWS", "IAMAnalyzer"], "alternates": {"html": "https://wpnews.pro/news/sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent", "markdown": "https://wpnews.pro/news/sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent.md", "text": "https://wpnews.pro/news/sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent.txt", "jsonld": "https://wpnews.pro/news/sentry-s-span-hierarchy-exposed-a-silent-retry-in-my-5-agent-pipeline-one-agent.jsonld"}}