This is my second entry for the DEV x Sentry Bug Smash challenge. Entry #1 was a crash with a confusing error message. This one is the opposite and it is scarier: no crash, no message, no event. Just silence.
smolagents runs LLM-generated Python in a sandboxed executor with a timeout. Issue #2473 claims that one line of model-generated math defeats it completely:
from smolagents.local_python_executor import LocalPythonExecutor
executor = LocalPythonExecutor(additional_authorized_imports=[], timeout_seconds=2)
executor.send_tools({})
executor("10 ** 10 ** 8") # 2 second timeout. Should be fine, right?
I ran this with a 2 second timeout and a faulthandler bomb set for 20 seconds. The timeout never fired. The process sat frozen until the external kill. An agent that generates this expression (and "compute this huge number" is exactly the kind of thing agents try) freezes its host process forever.
Zero comments on the issue, zero PRs. Mine now.
smolagents' timeout is thread based: a worker thread runs the code, the main thread waits in future.result(timeout=2)
. That design is fine for almost everything, because CPython switches threads between bytecode instructions.
But 10 ** 10 ** 8
is not "almost everything". CPython computes arbitrary precision **
, <<
and *
inside a single C call that holds the GIL from start to finish. No bytecode boundary, no thread switch, no timeout. The result would have about 400 million bits. The computation takes somewhere between minutes and hours. Your watchdog needs the GIL to wake up, and it never gets it.
The faulthandler dump made it concrete, and it was worse than the issue described:
Thread 0x16d1f3000 (worker):
File "local_python_executor.py", line 753 in evaluate_binop <- computing the pow
Thread 0x1f00d5e80 (main):
File "threading.py", line 999 in start
File "concurrent/futures/thread.py", line 180 in submit <- never returned!
The main thread was still stuck inside ThreadPoolExecutor.submit
. It never even reached future.result
. The 2 second timer never armed at all.
The existing MAX_OPERATIONS
guard (10 million AST operations) does not help either. This is a handful of AST nodes. The entire cost lives inside one of them.
Entry #1 was about a noisy failure. This bug is the opposite. I pointed a fresh Sentry project at a simulated agent run on the unpatched PyPI release (1.26.0) and got the most unsettling result possible: nothing. No event, no transaction, an empty project. The process was frozen mid-transaction and the SDK never got a chance to flush.
The lesson: for freeze-class bugs you need a supervisor. I added a small watchdog process that gives the worker 25 seconds, then kills it and reports what it saw, with the worker's faulthandler stack attached as evidence:
watchdog: starting worker (before) with 25s budget
worker: smolagents 1.26.0
worker: step 1 executing 'result = 10 ** 10 ** 8'...
watchdog: worker FROZE, killed it, reporting to Sentry
The resulting Sentry issue carries the whole story in one place: the frozen frame (evaluate_binop
, line 753 of local_python_executor.py
) sits right in the attached worker_faulthandler_stack
extra, environment tagged before
, handled by nothing except the supervisor.
Sentry's Seer ran root cause analysis on that issue and independently landed on the same conclusion: signal and thread interruption need the GIL, and a single C-level big-int operation never releases it.
Seer's verdict, verbatim:
CodeAgent's 2s timeout uses Python signal-based interruption, which cannot fire during uninterruptible C-level big-int operations that hold the GIL. [...] A single large big-integer arithmetic operation runs entirely as one C-level call that holds the GIL continuously without yielding. CPython cannot deliver signals or switch threads during an uninterruptible C extension call, so no timeout callback fires for the duration of that operation.
Killing the computation mid-flight is impossible from Python. But predicting the damage is O(1). Before executing **
, <<
or *
on integers, the executor now estimates the result's bit length from the operands' bit lengths:
if op == "**":
estimated_bits = left.bit_length() * right # upper bound
elif op == "<<":
estimated_bits = left.bit_length() + right
elif op == "*":
estimated_bits = left.bit_length() + right.bit_length()
Above 1 million bits (about 300k digits, still generous) it raises an informative InterpreterError
:
Operation '**' would produce an integer of around 400000000 bits, exceeding
the maximum of 1000000 bits allowed. Use smaller operands, or
pow(base, exp, mod) for modular exponentiation.
That message matters. The agent surfaces it to the model, and the model can actually act on it: use pow(base, exp, mod)
, which stays unrestricted because modular exponentiation is fast and legitimate. The agent recovers on the next step instead of hanging the host.
The after run on the patched build: the guard rejects the expression in 0.0 seconds, the error lands in Sentry as a normal actionable issue and step 2 executes fine.
watchdog: starting worker (after) with 25s budget
worker: smolagents 1.27.0.dev0
worker: step 1 error surfaced to the model: InterpreterError: ...
worker: step 2 executing '2 + 2'...
worker: step 2 ok
worker: DONE
watchdog: worker exited with code 0
Minutes after I opened the PR, OpenAI's Codex reviewer flagged a real hole: my guard checked type(x) is int
, which lets bool
and int
subclasses slip through. True << 10**9
and class BigInt(int)
still reached the uninterruptible C calls. Fixed with isinstance
, added both as regression tests. AI found the bug class, AI fixed it, AI reviewed the fix. I just steered.
**
, <<
, chained *
, all augmented forms, pow(a, b)
, bool and int subclass variants*=
, pow(7, 2**64, 97)
, float pow, 1 ** 10**9
The scariest bugs are not the ones that page you at 3am. They are the ones that make sure nothing ever pages you at all. Instrument for silence.