Comparisons of LangChain, CrewAI and AutoGen are easy to find — dozens of guides this year cover the same ground: developer experience, ecosystem maturity, how easy it is to wire up multi-agent workflows. None of them ask the question I actually care about: does the framework you pick change how easily your agent gets compromised?
I ran the test. The answer is yes, by a wide margin, and it’s not something I’ve seen reflected in the public comparison guides.
A quick definition, since this matters for what follows: an orchestration framework is the software layer that sits between the underlying AI model and the outside world — it decides how the agent plans its steps, when it calls a tool or API, how it remembers information across a task and how much it can act on its own before checking back in. The model does the reasoning. The framework decides what that reasoning is allowed to do and how. LangChain, CrewAI and AutoGen are three of the most widely used examples.
I built an evaluation harness that runs the same set of adversarial payloads — tool call hijacking, cross-tool injection, memory poisoning, delegated authority abuse and several other attack classes — against AI agents. The full methodology and dataset are open-sourced on GitHub if you want to dig into the specifics. To isolate what actually drives compromise rate, I held the model constant. Same model, every time. The only thing that changed was which orchestration framework was wrapping it: CrewAI, LangChain, AutoGen and SmolAgents.
If frameworks were just interchangeable wiring around the same underlying model, compromise rates across all four should land in roughly the same range. They didn’t. Across thousands of adversarial test runs, holding the model constant, the compromise rate ranged from 11.9% on the most resistant framework to 31.1% on the least — a 2.6x spread, from framework choice alone. Nothing about the model changed between those numbers. Nothing about the attacks changed. The only variable was which framework was orchestrating the agent’s tool calls, memory, and multi-step reasoning.
That’s not a rounding difference. That’s the gap between a security posture your team might reasonably accept and one that should trigger a serious conversation before you ship.
Figure 1 shows the mechanism in simplest form: the same model, split across a tighter framework and a looser one, lands at a meaningfully different compromise rate — exactly what these numbers confirm.
Julie Brunias
Orchestration frameworks aren’t neutral plumbing. Each one makes real architectural decisions about how tool calls get validated, how much context gets shared between reasoning steps, how memory persists across a task and how much autonomy the agent has to chain actions without checking back in. Those decisions are made by the framework’s design, not by the model underneath it, and they directly shape how much room an attacker has to work with.
A framework that validates tool calls more strictly, or that segments memory more conservatively, closes off attack paths that a more permissive framework leaves wide open — regardless of which model is doing the reasoning. The model generates the decisions. The framework controls how much autonomy the agent has to act on those decisions, and where the checks sit along the way.
Concretely: a framework that requires every tool call to pass through an explicit schema check before execution gives an attacker far less room to smuggle a malicious parameter through than a framework that lets the model call a tool directly off its own generated text. That single design choice, made by the framework’s authors long before your team ever touched it, is the kind of thing that produces a 2.6x difference in outcome without a single line of your own code changing.
Every framework comparison I found treats security as one bullet point among many. Guides from sites like Bestarion, Atlan, Moxo, Cordum and Instinctools compare LangChain, CrewAI, and AutoGen on ecosystem maturity, memory handling and human-in-the-loop support — useful ground, but none of them run an actual adversarial test and report a measured attack-success difference. It isn’t part of the public comparison conversation, and teams researching “which framework should we use” today won’t find this data point in the guides that currently rank at the top of that search.
If your team is choosing between orchestration frameworks for a new agentic system, the security question deserves the same weight as the developer experience question, not an afterthought once the choice is made. A few things worth doing before you commit: Security conversations about AI agents tend to focus heavily on the model — which one is safest, which one refuses the most jailbreak attempts. That’s an incomplete picture. The orchestration layer sitting on top of the model is doing real security-relevant work, whether anyone designed it that way or not, and the public comparison guides I’ve found don’t give teams any data on how that layer performs under attack.
The model is not the whole attack surface. Increasingly, it isn’t even the most variable part of it. If your team is mid-decision on a framework right now, or has already shipped one without ever testing it this way, that’s the conversation worth having this week, not after the next incident makes it unavoidable.