This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
The scariest bug I caught this month never threw a stack trace. It never paged anyone. It just quietly made an AI agent dumber, on repeat, and handed the blame to the model.
Here is the whole story, because the way I found it turned out to matter more than any single fix.
smolagents is Hugging Face's agent framework, 28k+ stars, and it has an unusual design: the agent writes Python as its reasoning, and a sandboxed interpreter (LocalPythonExecutor
) runs that code. The model thinks by coding. So the sandbox is not a side feature. It is the surface the model lives on.
I went in looking for one bug to fix for this challenge. Every obvious open issue was already claimed or had a competing PR. So instead of reading the issue tracker, I did something almost stupid: I fed the sandbox ordinary, boring, valid Python and watched what it refused to run.
The rule I gave myself: if it is valid Python and the sandbox rejects it, that is a bug. The model writes valid Python. If the sandbox chokes on it, the agent pays.
config = {**{"temperature": 0.7}, "top_p": 0.9}
Merging dicts with **
. Every LLM writes this. The sandbox's answer:
InterpreterError: NoneType is not supported.
There is no None
in that line. I stared at it, then went to the source. In Python's AST, a **spread
entry inside a dict literal has None
where the key would be, a marker that says "this is a spread." smolagents walked the keys and tried to evaluate that None
as if it were an expression. So the spread marker got blamed on a value the developer never wrote.
That is when it clicked. This was not a crash bug. It was a lying bug.
Here is the part that turned a one-off fix into an obsession.
The error was handled. smolagents catches it and feeds it back to the model as guidance: "here is what went wrong, try again." Good design, normally. But the guidance was wrong. The message said NoneType
, and the model's code had no None
, so the model could not act on it. It did the only thing a faithful agent can do with a message that is already a lie: it retried the exact same valid code.
And again. And again.
I only saw the loop because I had wired the demo to Sentry. One issue. Three events. Three identical failures on one line, each burning a real LLM call and a slot in the step budget, until the run gave up and produced a worse answer than it should have. Without something counting the events, you would never see the loop. You would just see an agent that "isn't very good," and you would go blame the model.
A red stack trace is mercy. It tells you where to look. The polite, well-handled, misleading error is the one that eats your afternoon, or in this case, eats the agent's entire budget while looking like helpful feedback.
Sentry's Seer read the same event and reached the same root cause I did, independently: None
keys in ast.Dict
fed to the evaluator, agent retries in a loop. That was the moment I trusted the pattern enough to go hunting.
Same fuzzer, same rule, more boring valid Python. The sandbox kept lying:
10 ** 10 ** 8
did not error at all. It best, *rest = scores
failed with "Cannot unpack tuple of wrong size." There was no wrong size. The sandbox simply never implemented starred unpacking, a feature Python shipped in 2008.a, b = "hi"
was rejected outright. Strings unpack fine in real Python.[a, b] = [1, 2]
silently assigned Four different corners of the language. One personality: confidently wrong, politely delivered, and invisible unless you were counting.
Each one came down to matching CPython instead of guessing:
None
key means merge a mapping. And gate it on hasattr(value, "keys")
, not the Mapping
ABC, so duck-typed mappings work too.pow(base, exp, mod)
.Four fixes, four PRs, sixty-plus new tests, all verified failing on main
first.
I opened the PRs. Minutes later, OpenAI's Codex reviewer commented on two of them, and it was right both times. On the big-int fix it caught that I had checked type(x) is int
, which lets bool
and int subclasses slip through the guard. On the dict fix it caught that my Mapping
ABC check was stricter than CPython.
So the final scoreboard was: an AI wrote the buggy code, a different AI reviewed my AI-assisted fix and found the hole, and a third AI (Seer) had already confirmed the root cause. I was the one steering, but I spent a good chunk of this project as the human in a loop of machines checking each other. Fitting, for a bug about an agent that could not tell it was stuck.
The bugs are fixed. The lesson I am keeping is simpler than any of them: the dangerous failures in an agent stack are not the loud crashes. They are the polite, confident, wrong messages that let the model fail on repeat while everyone blames the model.
Each one has its own full write-up with Sentry before/after evidence, if you want the deep dives. This was the story of how they all turned out to be the same bug wearing four different masks.