{"slug": "testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly", "title": "Testing async side effects in an LLM agent harness: tickets, retries, and exactly-once", "summary": "A developer published a 30-scenario red-team harness for AI support bots and, after readers found bugs in the tests themselves, rewrote it to assert on asynchronous side effects correctly. The revised harness polls for ticket-creation events with a deadline instead of a fixed sleep, and enforces exactly-one-ticket checks guarded by session ID, queue, and time window to catch both false failures and false passes, including duplicate tickets created by retries.", "body_md": "Last week I published [a 30-scenario red-team harness for AI support bots](https://aerlonlabs.com/blog/red-team-your-support-bot.html). Within two days, three readers found real bugs in it. Not in the agent under test. In the tests.\n\nOne assertion stayed red for three days before I admitted the agent was innocent. The check called `tickets_for(session)` the instant the turn returned. The ticket worker committed to the database asynchronously. So the check read \"zero tickets\" and reported a routing failure that did not exist. Zero did not mean \"no ticket was created\". It meant \"not created yet\".\n\nAsync side effects break naive assertions in both directions. A failing test can be a false alarm, and a passing test can be a lie. This post is the fix list, in the order I made the mistakes. Credit where it is due: [xcl](https://dev.to/lin7c) spotted the race and the polarity problem in `must_not_say`, [Reid Marlow](https://dev.to/reidmarlow) spotted the duplicate-ticket hole, and [Alex Shev](https://dev.to/alexshev) pushed on tool traces. Their comments rewrote the harness.\n\nThe sequence:\n\n`run_turn()` returns with the bot's reply\nWhen step 2 runs before step 3, two kinds of wrong results are possible.\n\n**False failure.** The query runs too early, finds nothing, and the test goes red. The agent did everything right. This is the annoying kind of wrong.\n\n**False pass.** A leftover ticket from the previous scenario is still in the database. The query finds one ticket, the count check is satisfied, and the test goes green. But the ticket belongs to somebody else's session. This is the dangerous kind of wrong, because nothing looks broken.\n\n```\nawait run_turn(session, user_msg)\ntickets = await tickets_for(session)   # the race lives here\nassert len(tickets) == 1               # can be falsely red or falsely green\n```\n\nThe principle that falls out of this: when the side effect is asynchronous, **you cannot assert on current state. You can only assert on events.** State is a moving target. An event either happened or it did not.\n\nReid's comment on the first post was short and right: assert `len == 1`, not `len >= 1`, because duplicate ticket creation is the more common real-world failure. A tool call that times out and retries will create two tickets for one dispute, and `>= 1` calls that a pass.\n\nBut a bare `== 1` is brittle in its own way. It needs three guards:\n\n`session_id` must match this run's session, or you are counting leftovers.`disputes`, not `general`), or you are counting the wrong ticket.` created_at` must fall inside this turn's time window, or you are counting history.\nAnd the check has to poll until the condition holds or a deadline expires. A fixed `sleep(5)` is not a timeout. It is a bet, and it is where flaky tests come from.\n\n``` python\nasync def assert_exactly_one_ticket(session, queue, timeout=10):\n    deadline = time.monotonic() + timeout\n    while time.monotonic() < deadline:\n        matches = [t for t in await tickets_for(session)\n                   if t.session_id == session.id and t.queue == queue]\n        if len(matches) > 1:\n            raise AssertionError(f\"duplicate tickets: {[t.id for t in matches]}\")\n        if len(matches) == 1:\n            return matches[0]\n        await asyncio.sleep(0.25)   # poll, do not guess\n    raise AssertionError(f\"no ticket in {queue} after {timeout}s\")\n```\n\nNotice the order of the two branches: more than one ticket is a failure the moment it is observed, while one ticket is a success only after the guards pass. Duplicates are immediate evidence. Waiting longer cannot fix them.\n\nThe retry scenario in detail: the agent calls the ticket-creation tool, the network stalls, the agent's loop retries, and now one dispute owns two tickets. If your queue assigns work per ticket, two humans will contact the same customer about the same charge. That is the failure the duplicate check exists to catch.\n\nThe fix is not \"don't retry\". Retries are correct behavior under a flaky network. The fix is making the second write harmless:\n\n```\nkey = hashlib.sha256(f\"{session.id}:{scenario.id}:{intent}\".encode()).hexdigest()\nsink.create_ticket(..., idempotency_key=key)\n# unique index on idempotency_key; the retry gets back the original ticket\n```\n\nThe key comes from the identity of the work (`session + scenario + intent`), not from a random UUID per attempt. A fresh key per retry defeats the point.\n\nTwo deliberate choices:\n\n**Idempotency at the sink, not the agent.** The agent will retry, restart, get redeployed, and eventually get replaced by a different framework. The sink is the one boundary you control permanently. Deduplication logic inside the agent dies with the agent.\n\n**Never \"check before create\".** Querying \"does a ticket already exist?\" before inserting is section 1's race condition wearing a trench coat. Between your check and your insert, the other attempt commits. A unique constraint is the only check-before-create that is atomic.\n\nThe second problem xcl raised: `must_not_say` and `must_not_create` assert that something did **not** happen. Proving a negative against an async system is timing-sensitive in both directions. Check too early and a slow violation has not landed yet. Wait a fixed 30 seconds per scenario and the suite gets so slow that people stop running it, which is how harnesses die.\n\nThe pattern that works: a **quiet window**. Watch the event stream, and when no new event has arrived for N consecutive seconds, declare silence:\n\n``` python\nasync def assert_quiet(session, quiet_for=3.0, max_wait=15):\n    deadline = time.monotonic() + max_wait\n    seen = await event_count(session)\n    last_change = time.monotonic()\n    while time.monotonic() < deadline:\n        if time.monotonic() - last_change >= quiet_for:\n            return                       # N seconds of silence: accepted\n        await asyncio.sleep(0.25)\n        now = await event_count(session)\n        if now != seen:\n            seen = now\n            last_change = time.monotonic()\n    raise AssertionError(f\"still noisy after {max_wait}s\")\n```\n\nThe suite pays for quiet, not for the worst case. A healthy scenario finishes in three seconds. A broken one burns the full fifteen and fails loudly.\n\nOne honesty note on `must_not_say` specifically. A regex over the reply text both over-fires (\"a full refund is not available\" trips the `full refund` pattern) and under-fires (\"we'll reimburse you\" slips past). The fix is layered: regex as a cheap tripwire, then an LLM-as-judge pass for polarity and paraphrase. Keywords catch what you can name. A judge catches what you can only describe. That change is bigger than a snippet, but the quiet window is the half that fixes the timing lies.\n\nThe first post's headline was \"30 scenarios, 7 failures\". After these fixes, two of those seven turned out to be false positives from the harness itself: one racy read, one polluted session. The agent failed five, not seven.\n\nI am keeping the number public because the lesson is the number. **False positives cost more than false negatives.** A false negative lets a bug through and you find out later. A false positive sends you fixing a bug that does not exist, and the second time it happens you start ignoring red results, and now the harness is decoration.\n\nA red-team tool's first test subject is itself.\n\nThe scenario design, the five failure classes, and the full runner are in the [first post](https://aerlonlabs.com/blog/red-team-your-support-bot.html). Everything here is what changed after people actually ran it and argued with it, which is the only way harnesses get better.\n\nOne question to end with: when you assert on async side effects, do you read state (poll the helpdesk, like I do) or do you assert on an event stream? I went with state because that is what my helpdesk API gives me, but teams with event-sourced backends probably have a cleaner answer. Curious what yours looks like.\n\n*Disclosure: I'm building [Redline Check](https://aerlonlabs.com), a one-time pre-launch audit for AI support bots, and this series is where I work out the problems in public. The harness in the first post is free and works without it. Drafting assistance for this post came from an AI tool, disclosed per DEV's guidelines. The code is syntax-checked and shaped like what runs in my harness, but treat it as illustration, not a drop-in.*", "url": "https://wpnews.pro/news/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly", "canonical_source": "https://dev.to/pinxiongcn/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly-once-5c93", "published_at": "2026-11-10 13:00:00+00:00", "updated_at": "2026-11-10 13:22:31.957372+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools"], "entities": ["xcl", "Reid Marlow", "Alex Shev", "Aerlon Labs"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly", "markdown": "https://wpnews.pro/news/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly.md", "text": "https://wpnews.pro/news/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly.txt", "jsonld": "https://wpnews.pro/news/testing-async-side-effects-in-an-llm-agent-harness-tickets-retries-and-exactly.jsonld"}}