{"slug": "how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx", "title": "How Signal found two hidden retry loops in our production agent Alyx", "summary": "Arize AI's Signal tool, a managed agent in Arize AX, identified two hidden retry loops in the production agent Alyx, including a 43-call dataset retry that appeared as valid tool activity with an OK root span. Over a 30-day period, Signal surfaced 34 issues in Alyx's production traces, with fixes involving small code changes such as treating duplicate same-state updates as a successful no-op and normalizing empty optional values to None at the tool boundary.", "body_md": "*We ran Signal on Alyx, the AI engineering agent built into Arize AX. It surfaced a duplicate task-state loop and a 43-call dataset retry that appeared as valid tool activity or an OK root span.*\n\n**Key takeaways**\n\n- In AI systems, bugs rarely show up as errors. They show up as behavior: a loop that looks like progress, a valid tool call that does the wrong thing, a root span that is still OK.\n- Finding that behavior by hand is slow. You cannot grep for it the way you grep for an exception, and inspecting one trace at a time does not scale.\n- Signal reviews production traces at volume, groups recurring behavior into ranked issues, and turns each one into an investigation with evidence, impact, and a next step.\n- We ran it on Alyx, our own production agent. It caught a todo loop and a dataset lookup retry that traditional monitoring would have treated as healthy runs.\n- The code changes were small, but Signal did the expensive part: finding the pattern and making the fix path obvious.\n\nWhen it came to our AI engineering agent Alyx, built into Arize AX, we wanted to know what its [production traces](https://arize.com/docs/ax/observe/tracing) could tell us that ordinary monitoring might miss.\n\nSo we used [Signal](https://arize.com/docs/ax/observe/signal), a new built-in managed agent in Arize AX, to examine Alyx. It quickly found one answer in a run that lasted 227 seconds, generated 192 spans, and called the same tool 43 times. The root span still reported OK. Alyx hadn’t crashed, but an empty optional field had pushed it into the wrong validation path and the resulting error message kept sending the agent back to the same tool.\n\nThat was one of two retry loops Signal surfaced in Alyx that looked, at first glance, like valid agent activity. In both cases, the eventual code fix was small. But finding the behavioral pattern hidden across production traces was something we missed before using Signal.\n\n**What is Arize Signal?**\n\n[Signal](https://arize.com/docs/ax/observe/signal) is a capability in Arize AX that reviews production traces for recurring agent behaviors, groups related patterns into ranked issues, and creates investigations with supporting evidence, estimated impact, and a suggested next step.\n\nThat next step might involve changing a prompt, code, configuration, or evaluation. Enterprise teams can also connect a GitHub repository so Signal can carry an investigation into the codebase and open a [pull request](https://arize.com/blog/from-signal-to-pr/) or scoped issue for review.\n\nWe used Signal to analyze [Alyx](https://arize.com/products/alyx/), the AI engineering agent built into Arize AX. Over a 30-day period, Signal surfaced 34 issues in Alyx’s production traces, including the two behavioral failures explored in this article.\n\n**What Signal found in Alyx**\n\nThe two Alyx findings shared the same basic shape: the agent received a tool response that suggested an action had failed, then repeated an operation that could never improve the state.\n\nIssue |\nWhat monitoring showed |\nWhat the agent was actually doing |\nFix |\n|---|---|---|---|\n| Todo state retry loop | Repeated valid tool activity | Retrying a state transition that had already succeeded | Treat duplicate same-state updates as a successful no-op |\n| Empty dataset ID retry | Root span marked OK | Repeating `get_datasets` 43 times because an empty optional field entered the wrong code path |\nNormalize empty optional values to `None` at the tool boundary |\n\nNeither issue began with an engineer searching for a known exception. Signal found the recurring behavior, grouped the relevant traces, and surfaced the evidence needed to understand the failure.\n\n**Finding 1: How a **`todo_update`\n\nerror trapped Alyx in a retry loop\n\n`todo_update`\n\nerror trapped Alyx in a retry loopAlyx uses a task list to coordinate multi-step work. It can create or restore a plan, move tasks through states such as `pending`\n\n, `completed`\n\n, and `blocked`\n\n, and call `finish()`\n\nwhen the turn is ready to close.\n\nThe normal trajectory looks roughly like this:\n\n- Restore or create the todo plan.\n- Select the current task.\n- Call the tools required to complete it.\n- Update the task state with\n`todo_update`\n\n. - Call\n`finish()`\n\nwhen the plan is complete.\n\nIn the evidence trace, `finish()`\n\nwas rejected because the plan still contained blocked work. Alyx then attempted to update the blocked task.\n\nA duplicate call to `todo_update(id=0, status=\"blocked\")`\n\nreturned a recoverable error instead of confirming the existing state. Because every tool response is appended to the model context, the agent interpreted that error as evidence that its action had failed.\n\nIt tried the transition again. Sometimes it alternated between `todo_update`\n\nand `finish()`\n\n. Neither action changed the underlying state, so the agent continued looping until the stream was cancelled.\n\nThe code change was straightforward. When the requested status already matches the current status, `todo_update`\n\nnow returns the current task list as a successful result. Errors remain in place for genuinely invalid cases, such as a missing task list or an unknown task ID.\n\nThe team also added a regression test covering the duplicate state transition. The time-to-fix was driven by discovering and reconstructing the behavior. So, once the pattern was visible, the patch was small.\n\n**Finding 2: How an empty dataset ID created 43 repeated tool calls**\n\nThe second issue occurred in `get_datasets`\n\n, a tool with two operating modes:\n\n- When\n`dataset_id`\n\nis omitted, the tool lists datasets in the current space. - When\n`dataset_id`\n\nis provided, the tool previews the selected dataset.\n\nAlyx intended to list available datasets. The production payload, however, supplied an empty string in the optional `dataset_id`\n\nfield.\n\nAt the typed tool boundary, an empty string still counted as a provided value. Validation therefore treated it as a candidate dataset ID and rejected it with the following message:\n\n`Dataset ID is invalid. Make sure you are getting the id from get_datasets().`\n\nThe recovery guidance sent Alyx back to the same tool it was already calling. With no better corrective action available, the model retried.\n\nOne affected run contained:\n\n- 192 spans\n- 227 seconds of runtime\n- 50 orchestrator iterations\n- 43 repeated\n`get_datasets`\n\ncalls - A root span whose status remained OK\n\nThe fix moved empty-string normalization into the shared tool boundary. Optional fields that permit `None`\n\nnow convert empty strings to `None`\n\nbefore entering tool-specific validation.\n\nA regression test recreates the production payload and verifies that `get_datasets`\n\nenters list mode rather than returning an invalid-ID error.\n\nAgain, the code change was limited. The difficult part was identifying that an apparently healthy trace contained a repeatable behavioral failure.\n\n**Why behavioral debugging changes the cost of fixing agents**\n\nThese incidents illustrate a common production pattern. The agent’s implementation bug may be small, while discovering it requires a broad view of runtime behavior.\n\nA human investigating either issue manually would need to:\n\n- Notice that a long trace contained little meaningful progress.\n- Reconstruct the sequence of model decisions and tool responses.\n- Determine whether the pattern appeared in other traces.\n- Identify the shared mechanism.\n- Locate the code boundary responsible for that behavior.\n- Create a reproducible test case.\n\nSignal performs much of the expensive discovery work before an engineer begins [debugging](https://arize.com/blog/ai-agent-debugging-four-lessons-from-shipping-alyx-to-production/). It groups related traces, describes the recurring behavior, and packages the evidence into a reviewable investigation.\n\nSome findings can move directly into pull requests. Others are better captured as GitHub issues with the relevant traces attached. In both cases, the team receives a scoped starting point instead of maintaining a full-time trace-review rotation.\n\nThis also changes how teams use [observability](https://arize.com/blog/ai-agent-observability-why-production-systems-need-a-reasoning-layer/) data. Traces document what the agent did. Behavioral analysis helps determine whether those actions moved the system toward the intended outcome.\n\n**How to monitor production AI agents with Signal**\n\nTeams can use Signal as a continuous production feedback loop:\n\nSignal begins reviewing new traces for recurring behavioral patterns.[Enable Signal](https://arize.com/docs/ax/get-started/get-started-managed-agent)on a production project.**Review ranked issues.** Each issue includes a description of the behavior, affected traces, impact, and supporting evidence.**Inspect the full trajectory.** Follow the model decisions, tool calls, tool responses, and state transitions that produced the issue.**Choose the appropriate intervention.** The finding may point to a prompt, code, configuration, tool-schema, or evaluation change.**Create a regression case.** Convert the production payload or trajectory into a test that prevents the behavior from returning.**Carry the investigation into GitHub.** Enterprise teams can connect a repository and turn findings into pull requests or scoped issues.\n\nTo see the workflow in practice, follow the [Signal tutorial](https://arize.com/blog/debug-production-ai-agents-with-signal-tutorial/) or watch the [webinar walkthrough](https://youtu.be/lCsNVmgp8Yg).\n\nProduction agents will continue to produce failures that look healthy at the span level. The practical advantage comes from turning those traces into a repeatable engineering loop: detect the pattern, inspect the evidence, ship a change, add a regression test, and watch production again.\n\nFor the two Alyx issues, the fixes were small. Signal handled the expensive part by finding where the agent’s behavior had gone wrong.", "url": "https://wpnews.pro/news/how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx", "canonical_source": "https://arize.com/blog/how-signal-found-two-hidden-retry-loops-in-alyx/", "published_at": "2026-08-27 19:28:05+00:00", "updated_at": "2026-08-27 19:51:59.340983+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "mlops"], "entities": ["Arize AI", "Signal", "Arize AX", "Alyx"], "alternates": {"html": "https://wpnews.pro/news/how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx", "markdown": "https://wpnews.pro/news/how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx.md", "text": "https://wpnews.pro/news/how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx.txt", "jsonld": "https://wpnews.pro/news/how-signal-found-two-hidden-retry-loops-in-our-production-agent-alyx.jsonld"}}