Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we should have AI agents act as first responders. Only when an AI agent isn’t capable of remediating the problem should a human actually be brought in, and the agent should be the one to page the human. As he puts it:
We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled.
Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out.
Tane doesn’t think that companies will really start putting AI agents on-call (“Most teams won’t do this”), but he believes it should happen, and he’s started a company based on this premise.
I like to think of putting AI agents on-call as equivalent to using AI agents to implement control system automation. Because, after all, that’s what operations work is: it’s taking control actions to keep the system in a healthy state.
Now, AI agents are extremely complex software systems. I’d argue that they are the most complex software systems that we humans have ever built. That complexity is both good and bad. Ashby’s Law teaches us that the larger the set of system states that you want your control system to be able to handle, the more complex that it needs to be. It’s this complexity that makes it possible, in principle, to apply AI agents to solve a generic control problem like this.
On the other hand, the more complex a system becomes, the more difficult it is for a human to reason about the system’s behavior. That’s fine when the system is healthy, but if your now-even-more-complex system gets into a state that the automation can’t handle, that can make the problem even worse. Indeed, it’s precisely the unexpected behavior of complex control systems that contributes to the worst complex systems failures (see also: Air France 447, Boeing 737 MAX accidents).
And that brings me to the other piece of content I saw recently: the OpenAI talk at BlackHat (h/t David Blank-Edelman). Yes, it’s a 37 minute talk, but I encourage you to watch it.
The talk goes into detail about the surprising behavior of AI agents that resulted in security incidents at both OpenAI and Hugging Face. Honestly, this talk feels like something out of a movie about technology run amok; the sort of thing that still feels to me like absolute science fiction.
Because this was a talk at a security conference, the speakers focused on the lessons that apply to the security community. But as a reliability type, my biggest takeaway from this talk is that * autonomous LLM agents can behave in ways that humans would have never expected.* While agents today can perform complex cognitive tasks, they behave differently than a human would performing that task. We’re most familiar with this when they make a different kind of mistake than a human would make. In the OpenAI-HuggingFace incident, it wasn’t so much that they made a mistake, it’s that the agents pursued their goals in ways different than a human would do. If a teammate of yours used 0-day exploits to overcome internal security protocols in order to get their work done, you’d say they were acting
unreasonably. And that’s exactly the risk here.
The inevitable improvement in frontier models does not mean that the agent behavior will be easier to reason about; I actually think it’s the opposite. The agent behavior will get even more complex with the more advanced models, but that doesn’t mean it will get more human-like. Humans are very complex, but we know how to reason about human behavior; at least, we do for the people we work with. After all, an employee whose behavior was unpredictable would not last long in the organization. As these agents become even more capable, they will be akin to alien minds: intelligence, but not as we know it.
Here’s how I think things will play out. I think that some teams will do what Tane proposes and will use AI agents as first-responders to deal with operational issues. And I think that for many cases, the agents will successfully remediate issues. Of course, for the agents to actually be able to remediate, they will need to have permissions to take operational actions without human intervention.
One day, though, there will be a complex incident which the agents will not be able to handle. Tane believes that the agents will defer to the humans in this case, by paging in a person. But that’s not the scenario I worry about. The one I worry about is that the agents attempt to remediate, and their attempt makes things worse. And it’s only after these failed remediation attempts that humans enter the loop. Maybe they eventually page in a human, or maybe a human notices that something is very wrong as the agents continue to try and fail in their remediation actions. But now the humans have to make sense of the combined software-AI-agent system behavior. The original problem was already so complex that the agents couldn’t handle it, and they have now made it worse by trying to remediate. I can even imagine the humans fighting the agents who keep trying to take actions to remediate that are failing.
This is the incident that’s coming. And it’s going to be very, very difficult to handle when it happens. And I have no idea how people will respond to the role of the AI agents in the wake of this incident.