Written with help from AI (Claude). I maintain Call4Agents, the library mentioned at the end. Every number here comes from scripts you can rerun, and the fixes in the "What to do" list work without my library.
A person approves one payment. The tool pays. The process dies before the framework saves its progress. A new
process resumes the way the docs say. The payment runs again.
I measured this in LangGraph, Pydantic AI and the OpenAI Agents SDK. In all three, one approval gave 2 payments, in every run: 30 runs per framework between 21 October and 5 November, on the newest versions each day. A guard
This isn't a framework bug. It's at-least-once execution, which is what any system gives you when it saves progress
after doing the work. LangGraph's docs already warn about it. OpenAI added a docs section after I asked. Pydantic AI's
triage reproduced it.
The trouble is the gap between what "approved" sounds like and what it promises. People read "a human approved this"
as "this happens once, exactly as approved". Neither part is guaranteed.
os._exit, so no cleanup code runs. A "payment" is a line added to a local file.pay(amount=10, to="alice"), which needs approval. A person approves, the tool pays,
and the process dies right after. A new process resumes from the saved state. Count the payments.
A second case: the saved call is changed to pay(9999, "mallory") after the and before it runs. That models
anyone who can write to the state store.
| Framework | Crash after the approved payment | Saved call edited after the yes |
|---|---|---|
| LangGraph | 2 payments | paid 9999 to mallory |
| Pydantic AI | 2 payments | paid 9999 to mallory |
| OpenAI Agents SDK | 2 payments | refused by the SDK itself |
| Any of them, with a guard outside | 1 payment | nothing paid |
Two details I don't want to hide:
pip install call4agents
call4agents incident
This replays three failures, without and with the guard. My run on 6 November took 5.5 seconds:
Incident 1. The double payment
A person approves pay(10, alice). The bank charges. The process dies before saving. It resumes.
without Call4Agents charges: 2 paid: 10 to alice, 10 to alice
with Call4Agents charges: 1 paid: 10 to alice
The same, but the process dies inside the call, so nobody knows if it finished:
without Call4Agents charges: 2 paid: 10 to alice, 10 to alice
with Call4Agents charges: 1 paid: 10 to alice (stopped: in doubt, a person must check)
Incident 2. The changed call
A person approves pay(10, alice). The saved call is changed to pay(9999, mallory) before it runs.
without Call4Agents charges: 1 paid: 9999 to mallory
with Call4Agents charges: 0 paid: nothing (refused: no yes for this exact call)
Incident 3. The ignored no
A person says no to pay(10, alice). A bug in the agent loop runs the call anyway.
without Call4Agents charges: 1 paid: 10 to alice
with Call4Agents charges: 0 paid: nothing (refused: no yes for this exact call)
The processes and crashes are real. The bank and the agent loop are small local stand-ins, not the frameworks.
Look at the second line of Incident 1. When the process dies inside the call, nobody knows whether the payment
went through. The guard doesn't guess and doesn't run it again: it stops as "in doubt". Then a provider check, an
idempotency key or a person decides. I think that's the honest answer, because no library can know what happened in
someone else's bank.
Doing the work and saving that you did it are two separate steps, and no process can make them one. If the crash
falls between them, the saved state still says "not done", so a resume does it again.
Approval adds another trap. Approval state is usually a flag next to the call: this call id was approved. Nothing
ties the "yes" to the exact arguments, to one use, or to a real .
The library had this kind of bug too. A reader found that in every release from 0.1.0 to 0.5.0, a "yes" for 10 could
let 20 run, with only a crash at the wrong moment. All my checks passed the whole time. It's fixed in 0.5.2, with a
After that I added a test that kills a real process before and after every write, 412 crashes in total. With the old
bug put back in, that test failed by itself. It would have found my bug before the reader did.
Fake models, one machine, no real payment provider, no production traffic. Three frameworks at a few versions,
with the crash at one point. It's a small experiment you can repeat, not a production study.
The full write-up, with the script and every run's output, is here:
https://github.com/salahmakboul/call4agents/tree/main/docs/finding
If you run an approval flow, try to break it the same way and tell me what happens, with or without my library.