Testing async side effects in an LLM agent harness: tickets, retries, and exactly-once

A developer published a 30-scenario red-team harness for AI support bots and, after readers found bugs in the tests themselves, rewrote it to assert on asynchronous side effects correctly. The revised harness polls for ticket-creation events with a deadline instead of a fixed sleep, and enforces exactly-one-ticket checks guarded by session ID, queue, and time window to catch both false failures and false passes, including duplicate tickets created by retries.

Last week I published a 30-scenario red-team harness for AI support bots https://aerlonlabs.com/blog/red-team-your-support-bot.html . Within two days, three readers found real bugs in it. Not in the agent under test. In the tests. One assertion stayed red for three days before I admitted the agent was innocent. The check called tickets for session the instant the turn returned. The ticket worker committed to the database asynchronously. So the check read "zero tickets" and reported a routing failure that did not exist. Zero did not mean "no ticket was created". It meant "not created yet". Async side effects break naive assertions in both directions. A failing test can be a false alarm, and a passing test can be a lie. This post is the fix list, in the order I made the mistakes. Credit where it is due: xcl https://dev.to/lin7c spotted the race and the polarity problem in must not say , Reid Marlow https://dev.to/reidmarlow spotted the duplicate-ticket hole, and Alex Shev https://dev.to/alexshev pushed on tool traces. Their comments rewrote the harness. The sequence: run turn returns with the bot's reply When step 2 runs before step 3, two kinds of wrong results are possible. False failure. The query runs too early, finds nothing, and the test goes red. The agent did everything right. This is the annoying kind of wrong. False pass. A leftover ticket from the previous scenario is still in the database. The query finds one ticket, the count check is satisfied, and the test goes green. But the ticket belongs to somebody else's session. This is the dangerous kind of wrong, because nothing looks broken. await run turn session, user msg tickets = await tickets for session the race lives here assert len tickets == 1 can be falsely red or falsely green The principle that falls out of this: when the side effect is asynchronous, you cannot assert on current state. You can only assert on events. State is a moving target. An event either happened or it did not. Reid's comment on the first post was short and right: assert len == 1 , not len = 1 , because duplicate ticket creation is the more common real-world failure. A tool call that times out and retries will create two tickets for one dispute, and = 1 calls that a pass. But a bare == 1 is brittle in its own way. It needs three guards: session id must match this run's session, or you are counting leftovers. disputes , not general , or you are counting the wrong ticket. created at must fall inside this turn's time window, or you are counting history. And the check has to poll until the condition holds or a deadline expires. A fixed sleep 5 is not a timeout. It is a bet, and it is where flaky tests come from. python async def assert exactly one ticket session, queue, timeout=10 : deadline = time.monotonic + timeout while time.monotonic < deadline: matches = t for t in await tickets for session if t.session id == session.id and t.queue == queue if len matches 1: raise AssertionError f"duplicate tickets: { t.id for t in matches }" if len matches == 1: return matches 0 await asyncio.sleep 0.25 poll, do not guess raise AssertionError f"no ticket in {queue} after {timeout}s" Notice the order of the two branches: more than one ticket is a failure the moment it is observed, while one ticket is a success only after the guards pass. Duplicates are immediate evidence. Waiting longer cannot fix them. The retry scenario in detail: the agent calls the ticket-creation tool, the network stalls, the agent's loop retries, and now one dispute owns two tickets. If your queue assigns work per ticket, two humans will contact the same customer about the same charge. That is the failure the duplicate check exists to catch. The fix is not "don't retry". Retries are correct behavior under a flaky network. The fix is making the second write harmless: key = hashlib.sha256 f"{session.id}:{scenario.id}:{intent}".encode .hexdigest sink.create ticket ..., idempotency key=key unique index on idempotency key; the retry gets back the original ticket The key comes from the identity of the work session + scenario + intent , not from a random UUID per attempt. A fresh key per retry defeats the point. Two deliberate choices: Idempotency at the sink, not the agent. The agent will retry, restart, get redeployed, and eventually get replaced by a different framework. The sink is the one boundary you control permanently. Deduplication logic inside the agent dies with the agent. Never "check before create". Querying "does a ticket already exist?" before inserting is section 1's race condition wearing a trench coat. Between your check and your insert, the other attempt commits. A unique constraint is the only check-before-create that is atomic. The second problem xcl raised: must not say and must not create assert that something did not happen. Proving a negative against an async system is timing-sensitive in both directions. Check too early and a slow violation has not landed yet. Wait a fixed 30 seconds per scenario and the suite gets so slow that people stop running it, which is how harnesses die. The pattern that works: a quiet window . Watch the event stream, and when no new event has arrived for N consecutive seconds, declare silence: python async def assert quiet session, quiet for=3.0, max wait=15 : deadline = time.monotonic + max wait seen = await event count session last change = time.monotonic while time.monotonic < deadline: if time.monotonic - last change = quiet for: return N seconds of silence: accepted await asyncio.sleep 0.25 now = await event count session if now = seen: seen = now last change = time.monotonic raise AssertionError f"still noisy after {max wait}s" The suite pays for quiet, not for the worst case. A healthy scenario finishes in three seconds. A broken one burns the full fifteen and fails loudly. One honesty note on must not say specifically. A regex over the reply text both over-fires "a full refund is not available" trips the full refund pattern and under-fires "we'll reimburse you" slips past . The fix is layered: regex as a cheap tripwire, then an LLM-as-judge pass for polarity and paraphrase. Keywords catch what you can name. A judge catches what you can only describe. That change is bigger than a snippet, but the quiet window is the half that fixes the timing lies. The first post's headline was "30 scenarios, 7 failures". After these fixes, two of those seven turned out to be false positives from the harness itself: one racy read, one polluted session. The agent failed five, not seven. I am keeping the number public because the lesson is the number. False positives cost more than false negatives. A false negative lets a bug through and you find out later. A false positive sends you fixing a bug that does not exist, and the second time it happens you start ignoring red results, and now the harness is decoration. A red-team tool's first test subject is itself. The scenario design, the five failure classes, and the full runner are in the first post https://aerlonlabs.com/blog/red-team-your-support-bot.html . Everything here is what changed after people actually ran it and argued with it, which is the only way harnesses get better. One question to end with: when you assert on async side effects, do you read state poll the helpdesk, like I do or do you assert on an event stream? I went with state because that is what my helpdesk API gives me, but teams with event-sourced backends probably have a cleaner answer. Curious what yours looks like. Disclosure: I'm building Redline Check https://aerlonlabs.com , a one-time pre-launch audit for AI support bots, and this series is where I work out the problems in public. The harness in the first post is free and works without it. Drafting assistance for this post came from an AI tool, disclosed per DEV's guidelines. The code is syntax-checked and shaped like what runs in my harness, but treat it as illustration, not a drop-in.