Research noteJuly 2026Apache 2.0
We scanned 25 AI agent frameworks — 23,476 files — looking for one specific thing: a consequential action that a model can reach, with a parameter the model controls, and no authorisation check on the path. We found 30. We also found 12 false positives in our own scanner, and fixed them before publishing this.
25 repos, pinned by SHA
23,476 files analysed
30 findings, hand-triaged
0 in control repos
The gap
Validation is not authorisation. #
Most agent tools check that a request is well-formed. Far fewer check that this caller is permitted to perform this exact action, right now — at the point where the side effect actually happens.
That distinction is invisible in a code review, because both look like a guard. It is the difference between a tool that refuses an unauthorised refund and one that processes it correctly.
The scanner looks for exactly one shape: a model-controlled parameter reaching a recognised consequential sink, with no check that dominates every path to it.
The study25 repositories, immutable SHAs
Where the findings actually were. #
Every repository is pinned by commit hash so the result is reproducible. Five non-agent control repositories were included deliberately: any finding in those is a precision failure by definition. There were none.
| Repository | Class | Findings |
|---|---|---|
| crewAIInc/crewAI | framework | 12 |
| TransformerOptimus/SuperAGI | application | 6 |
| FoundationAgents/MetaGPT | application | 5 |
| microsoft/semantic-kernel | framework | 3 |
| agno-agi/agno | framework | 2 |
| modelcontextprotocol/servers | MCP server | 1 |
| modelcontextprotocol/python-sdk | MCP server | 1 |
| requests · flask · fastapi · click · rich | control | 0 |
| What the action was | Count |
|---|---|
| Network egress with model-controlled destination | 15 |
| File write to a model-controlled path | 7 |
| Shell execution | 3 |
| Message send to a model-controlled recipient | 2 |
| File deletion | 2 |
| SQL execution with a model-controlled statement | 1 |
What we got wrongand found ourselves
The first run of this study was 81% precise. #
Twelve of sixty-three findings were wrong. We publish that because a scanner claiming perfect precision has either not been measured or is not being honest, and because the failure classes are more useful to you than the successes.
A Kubernetes rule that matched a search client The pattern
client.*.create
was loose enough to match Elasticsearch and vector-store clients. It fired at HIGH severity in two well-known repositories. Constrained to genuine Kubernetes surfaces. - A SQL rule that caught the safe case and missed the dangerous one It matched the literal string
`execute("DROP TABLE …")`
but not`execute(query)`
— where the model controls the statement. It was detecting hardcoded SQL and ignoring caller-controlled SQL. Now matches the sink, not the text. -
A bulk-deletion call that was never in the vocabularys3.delete_objects
— deleting many objects at once — simply was not a recognised sink. Added, with the exact source line kept as a regression test.
Each fix carries a permanent regression fixture, and CI now fails if any finding in the corpus is untriaged or triaged as a false positive. The 30/30 is enforced, not asserted.
Limits
What this cannot see. #
Static analysis of guard soundness is undecidable in the general case. Rather than claim coverage we cannot demonstrate, here is what is verified against real code and what is not. | Agent architecture | Status |
|---|---|
MCP `@mcp.tool()` | COVERED |
LangChain @tool / BaseTool | COVERED |
OpenAI @function_tool | PARTIAL |
| Custom agent loops | NOT COVERED |
| Action/observation dispatchers | NOT COVERED |
A finding is not a vulnerability
The scanner establishes that a model-controlled parameter reaches a consequential action with no dominating authority check in the analysed path. It does not establish that the agent is externally reachable, that no guard exists elsewhere in the system, or that the action is exploitable.
Custom agent loops were tried and rejected
Detecting agents that run whatever the model returns, with no framework decorator, produced 10 false positives out of 10 candidates. The signal decomposes into “this module talks to an LLM” and “this function runs what it was passed” — which describes most of an agent framework. Recorded as a negative result rather than shipped.
Cryptographic binding is invisible to it
When authority is carried inside a signed proof object, the binding between the authorisation and the specific action is real but not visible at the call site. A static scanner cannot verify it. That is an argument for checking at runtime, not for trusting the scan.
Run itno account, no signup, no network calls
Ten seconds on your own code. #
Zero runtime dependencies. Nothing is uploaded. If it finds nothing, it tells you precisely what that does and does not mean.
$ pipx run actenon-scan scan .