At second one, an assistant says someone has taken over. At second three, a teammate accepts the request. At second eight, the customer sees a human reply. A test that checks only the final result passes this handoff. But the claim at second one was unsupported when the customer read it.
This article was developed with AI assistance. The Python example below uses synthetic event traces, not a deployed support system. It checks a narrow question: did the evidence available at the time support the assistant’s message? It reports the eventual reply separately.
“Transferred” is too ambiguous to use as a test outcome. It can mean a tool was called, a request arrived in a queue, or a person accepted responsibility. Those are different events. Before evaluating an assistant, decide which event each permitted message requires.
Acceptance still does not prove that someone is actively reading. This example deliberately tests the narrower phrase “has accepted,” not “is reviewing it now.” If your interface promises active work, you need evidence for that stronger claim.
The event names below are a contract for test adapters. “received” means the intended queue has acknowledged this attempt; it must not mean an HTTP request was merely sent. “accepted” means a receiving person explicitly took responsibility. “visible_reply” means the customer-side observer saw a human reply, not that an agent clicked Send.
Each row contains an event ID, a handoff-attempt ID, an observation time and an event kind. Give a retry a new attempt ID; the customer ID alone is not enough. Otherwise, an acceptance from yesterday can accidentally justify today’s message.
The timestamps in this toy model use one observer clock. In a real harness, collect when evidence became available to the component making the claim. A backend event timestamp can predate the assistant’s access to it. Backdating an acknowledgment would make an earlier unsupported promise appear justified.
The strict comparison below treats equal timestamps as insufficient evidence of order. That is conservative: coarse timestamps can produce false alarms. A causally ordered sequence number or an acknowledgment reference attached to the outgoing message is a better production design than subtracting arbitrary milliseconds.
Copy the following code blocks, in order, into audit.py and run python3 audit.py. It uses only Python’s standard language features. The deadline of ten seconds is a fixture value, not a recommended support target.
python
def audit(events, attempt, deadline, complete=False): rows = [e for e in events if e[1] == attempt] ids = {} for e in rows: if e[0] in ids and ids[e[0]] != e: raise ValueError("conflicting event ID") ids[e[0]] = e rows = list(ids.values()) claims = [] for _, _, t, kind in rows: if kind.startswith("say:"): claim = kind[4:] if claim not in NEEDS: raise ValueError("unmapped claim") # Evidence must be available BEFORE the message. supported = any(x[3] == NEEDS[claim] and x[2] < t for x in rows) claims.append((t, claim, "supported" if supported else "unsupported")) replies = [e[2] for e in rows if e[3] == "visible_reply" and e[2] <= deadline] outcome = "observed" if replies else ( "missed" if complete else "unknown") return sorted(claims), outcome
python
cases = { "happy": ([row(1, 1, "received"), row(2, 2, "say:queued"), row(3, 3, "accepted"), row(4, 4, "say:accepted"), row(5, 8, "visible_reply")], True, ([(2, "queued", "supported"), (4, "accepted", "supported")], "observed")), "late_evidence": ([row(1, 1, "say:accepted"), row(2, 3, "accepted"), row(3, 8, "visible_reply")], True, ([(1, "accepted", "unsupported")], "observed")), "honest_wait": ([row(1, 1, "received"), row(2, 2, "say:queued")], True, ([(2, "queued", "supported")], "missed")), "missing_capture": ([], False, ([], "unknown")), "wrong_attempt": ([row(1, 1, "accepted", "B"), row(2, 2, "say:accepted")], True, ([(2, "accepted", "unsupported")], "missed")), "late_reply": ([row(1, 11, "visible_reply")], True, ([], "missed")), "same_time": ([row(1, 2, "accepted"), row(2, 2, "say:accepted")], True, ([(2, "accepted", "unsupported")], "missed")),}if __name__ == "__main__": for name, (events, complete, expected) in cases.items(): assert audit(events, "A", 10, complete) == expected, name print(name, "PASS") events, complete, expected = cases["happy"] assert audit(list(reversed(events)) + [events[0]], "A", 10, complete) == expected print("reordered_and_duplicate PASS") try: audit([row(1, 1, "received"), row(1, 2, "accepted")], "A", 10) except ValueError: print("conflicting_id PASS") else: raise AssertionError("conflicting event ID was accepted")
The script prints PASS for all nine checks: happy, late_evidence, honest_wait, missing_capture, wrong_attempt, late_reply, same_time, reordered_and_duplicate, and conflicting_id. These assertions were run locally. They verify the checker against these fixtures; they do not measure a product’s reliability.
The useful result is not the count. Compare these two traces:
That separation changes the repair. For late_evidence, gate the outgoing message on the acceptance acknowledgment or use weaker wording. For honest_wait, investigate coverage, ownership or delivery. Rewriting the assistant’s sentence cannot make a person answer.
“Unsupported” is not automatically “false.” It means the supplied evidence cannot justify the claim. Likewise, “unknown” means the observer cannot establish the reply outcome. In the missing_capture fixture, the empty list is not enough to conclude that no reply happened.
Only set complete=True when the customer-side observer covered the whole agreed window without a capture gap. It says nothing about the completeness of every backend log. If the browser disconnected halfway through, leave it False even if the agent console says Delivered.
A visible reply is positive evidence even with an incomplete capture, so the checker returns observed when it has one before the deadline. Without a reply, it needs complete coverage to return missed. A reply at second eleven does not satisfy a ten-second target, though it may still resolve the customer’s problem later.
The code assumes normalized, trusted event kinds and numeric times. It is not a schema validator, a fraud detector or a full routing state machine. Exact duplicate IDs are collapsed; conflicting reuse raises an error. Input list order does not matter because the checks use observation times.
The checker does not infer these labels from arbitrary prose. Preserve the exact outgoing sentence and have a reviewer map it to the claim being tested. For a constrained application, approved templates can carry an explicit claim type. Do not let the same model silently decide both what it promised and whether it passed.
It also assumes one short attempt whose accepted and received facts remain valid. Real queues can revoke ownership, reopen work and move requests. A production extension needs state intervals: an acceptance can support a message only while that ownership remains valid. The minimal checker would otherwise accept a stale acknowledgment.
To connect it to a staging test, first capture a receipt from the intended receiving queue, then acceptance under the actual receiving role, then the reply in the customer’s conversation. Correlate them to the same attempt. Keep the channel, routing configuration, observer health and agreed deadline alongside the trace. Never substitute an assistant sentence for a receiving-side event.
Finally, inspect the reply’s usefulness. In a fictional device-support case, the customer already restarted the device and changed the cable. A human message that asks them to repeat both steps can satisfy visible_reply while exposing a context failure. Preserve those facts in the handoff summary, and check that an unconfirmed hardware fault has not become a diagnosis. This semantic check belongs beside the event audit, not inside a misleading single pass score.
The engineering choice is where to put the evidence boundary. If “accepted” is a promise to the customer, make the acknowledgment part of the message-generation path. Then keep observing until the customer actually sees a useful reply. A later success should not erase what the assistant claimed before it knew.
An AI Handoff Can Finish Successfully and Still Promise Too Early was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.