{"slug": "sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline", "title": "Sentry Tracing: Finding the Silent Retry in my LLM Pipeline", "summary": "A developer using Sentry tracing discovered that an LLM-powered security pipeline was silently retrying because the IAMAnalyzer tool dumped a 27KB JSON blob into the context window, overwhelming the model. By implementing pagination and a relevance filter that limits input to the top 20 most active roles, the developer achieved a 42% reduction in output volume and a 21% increase in agent speed, dropping SecurityScanner execution time from 22.6 seconds to 5-8 seconds.", "body_md": "# Sentry Tracing: Finding the Silent Retry in my LLM Pipeline\n\nThe pipeline follows this flow:\n\n1. **ResourceDiscovery**: Inventories EC2, S3, Lambda, etc.\n\n2. **SecurityScanner**: Identifies open ports and public buckets.\n\n3. **ComplianceChecker**: Maps findings to CIS benchmarks.\n\n4. **RiskScorer**: Calculates severity and blast radius.\n\n5. **RemediationPlanner**: Generates the actual fix commands.\n\nThe system runs against live accounts with actual assets (90 IAM roles, 14 S3 buckets, etc.), not sanitized test data. Everything seemed fine until I noticed a massive performance delta. While most agents wrapped up in 5-10 seconds, the **SecurityScanner** was consistently hitting 22.6 seconds.\n\nInstead of sprinkling `time.time()`\n\ncalls everywhere and guessing, I used Sentry’s span hierarchy to look at the `gen_ai.invoke_agent`\n\ntraces. The waterfall view revealed something critical: the LLM wasn't just \"slow\"—it was failing and retrying.\n\nThe culprit was my `IAMAnalyzer`\n\ntool. It was fetching every single IAM role in the account, filtering out service-linked roles, and then dumping a 27KB JSON blob directly into the context window. The LLM was getting overwhelmed by the payload size, triggering CrewAI's internal retry logic. I was essentially paying a \"token tax\" for redundant attempts because the initial tool output was too bloated.\n\nTo fix this, I implemented a pagination strategy and a relevance filter. Rather than dumping everything, I now sort roles by their last used date and only pass the most active/relevant ones to the agent.\n\nHere is the technical breakdown of the fix in `src/security_posture/tools/iam_analyzer.py`\n\n.\n\n**The Buggy Implementation:**\n\nThis version fetched up to 100 roles without a strict budget, creating a massive JSON string that choked the LLM.\n\n```\n# Fetches ALL roles without pagination limit\nroles = iam.list_roles(MaxItems=100)\nrole_details = []\n\nfor role in roles[\"Roles\"]:\n    role_name = role[\"RoleName\"]\n    if role.get(\"Path\", \"\").startswith(\"/aws-service-role/\"):\n        continue\n    # Analyzes every single role...\n    attached = iam.list_attached_role_policies(RoleName=role_name)\n    # ...builds massive JSON output\n```\n\n**The Optimized Implementation:**\n\nI switched to a paginator and added a sorting mechanism to ensure the LLM only sees the roles that actually matter for a security audit.\n\n```\n# FIX: Paginate and sort by relevance\nall_roles = []\npaginator = iam.get_paginator(\"list_roles\")\nfor page in paginator.paginate():\n    all_roles.extend(page[\"Roles\"])\n\n# Filter service-linked roles (which can't be modified anyway)\nauditable_roles = [\n    r for r in all_roles \n    if not r.get(\"Path\", \"\").startswith(\"/aws-service-role/\")\n]\n\n# Sort by last used date (most active first) to prioritize analysis\nauditable_roles.sort(key=_last_used_sort_key, reverse=True)\n# Only send the top N most relevant roles to the LLM\ntop_roles = auditable_roles[:20]\n```\n\nBy implementing this token budget guard and pagination, I saw a 42% reduction in output volume and a 21% increase in overall agent speed. The \"silent retry\" disappeared from the traces, and the SecurityScanner's execution time dropped back down to the 5-8 second range.\n\nThis was a huge lesson in AI workflow optimization: when an LLM agent slows down, don't just look at the model—look at the size of the tool outputs you're shoving into the context window.\n\nFor those interested in the full deployment, the source is available here:\n\n```\nhttps://github.com/simplynadaf/aws-security-posture-agent\n```\n\n[Next My AI Workflow Nightmare: The \"Super Individual\" Team →](/en/threads/2733/)", "url": "https://wpnews.pro/news/sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline", "canonical_source": "https://promptcube3.com/en/threads/2767/", "published_at": "2026-07-24 14:47:29+00:00", "updated_at": "2026-07-24 15:45:17.047954+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Sentry", "CrewAI", "IAMAnalyzer", "AWS", "SecurityScanner"], "alternates": {"html": "https://wpnews.pro/news/sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline", "markdown": "https://wpnews.pro/news/sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline.md", "text": "https://wpnews.pro/news/sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline.txt", "jsonld": "https://wpnews.pro/news/sentry-tracing-finding-the-silent-retry-in-my-llm-pipeline.jsonld"}}