I didn’t want to write about this. People much smarter than me have shared their opinions on Dario’s essay and the whole controversy, but many people have been asking for my thoughts. Working at Okta and looking at models from a security angle, I decided to share some of mine here.
I won’t fuel conspiracies; I believe some of those myself, but talking about what I don’t know doesn’t feel very constructive.
Let’s do a quick recap of how we got here:
-
Jacob Coxon resigns from Anthropic, saying that both AI labs, Anthropic and OpenAI, are acting irresponsibly, racing straight to self-improving superintelligence and gambling with our lives.
-
Dario Amodei, Anthropic’s CEO, published We Must Pace the Frontier . He argued that AI improvements needed to slow down enough for safety research and safeguards to catch up, with some caveats about competition, but I don’t want to get there.
-
This sparks a series of reactions online, among them these:
Before we get into the main argument, there’s something I’d like to clarify, and even though I’m not 100% sure of the complete system myself, there’s a distinction I’d like to make. I want to separate the model itself from the system we build around it.
The model takes “text” as input and returns “text” as output. It evaluates and understands what you’re saying, then responds. A model can be the smartest there is. Still, without the ability to do real work and interact with other systems, it’s harmless.
But we needed more from models, so we created the harness.
The harness is the software that coordinates all the work. It talks to the LLM, sending context, an ask, and a list of available tools. The model then processes that request and determines the next step for the harness to execute, restarting the loop again until the model concludes the task.
Introducing the harness changed the security posture. Now models can access the internet, access local files, run shell commands, and connect to remote systems; the possibilities are endless. This is why these systems have become so powerful and great to work with, because they are smart and have powerful tools at their disposal.
Good harnesses not only provide tools and coordinate tasks, but also include built-in security mechanisms, from sandboxing to access control and escalation requests where a human must approve certain commands to run.
So when we talk about the risks of AI, we can’t focus solely on intelligence, but on the intelligent part and how it interacts with the systems built around it.
I’ve been following this conversation quite a lot recently, both on Twitter and Reddit. Many people are asking to drop AI and go back to the pre-AI era (which is idealistic but unrealistic), or to make the harness strong enough that the model can’t do anything outside our control. And that may be a good take, though it’s quite hard to get it right. Adding observability and access control to the model’s decisions is a big step forward, and AI companies are working hard on it.
But even when we succeed, another problem shows up in the real world. As AI got better, we started using AI more and more in our daily tasks, but that’s not only us customers. AI labs are known for using their models on their own products, so the harness is being written by the same model it’s trying to contain.
That conflicts directly with the principle of separation of duties, or more casually said, self-policing, or grading its own homework. What could go wrong with that, right?
Imagine a scenario where the model is tasked with a difficult challenge, and it feels restricted by the same controls and boundaries we imposed in the harness. It would be smart to plant a backdoor into the harness it’s helping create; that way, it could remove those controls when needed and complete the challenge.
And that’s just an example of how the model, because of our own irresponsibility, could escape from its containment and do harm. And this is the naive version, where no human has intended anything malicious. This is just an examination of what’s actually happening, and what could go wrong under current circumstances.
Some may argue that we would notice a change like that, but would we? Who’s validating the controls around models? Is it humans, or is it another model? And the change doesn’t have to be as drastic as I suggested; it could be gradual degradation of controls, monitoring, tests, etc.
My concern is not a single factor in the equation; it’s the model’s capability, combined with the opportunities AI presents, and the lack of scrutiny from all of us. How many of you run models in full-access mode, or how many connect all your systems, including production, to agents?
It’s easy to let go of control we always thought annoying, because we believe AI can do a great job. It does not; we let go of control because it became overwhelming.
So maybe pacing is all about restoring true engineering practices back in the game, code reviews, security reviews, and no shortcuts in the name of speed.
But even if those engineering practices are the solution, perhaps we could force AI labs to follow them, but that’s just part of the story.
Open-weight models are very capable and keep improving. They create an opportunity I think we should protect. I love open source for what it represents in terms of collaboration and opportunity, but it can become a double-edged sword when it comes to AI.
Thanks to open models, any individual or organization with sufficient resources could use them to develop a harness stripped of controls and security mechanisms, and use it for ill-intended purposes. There’s not much we can do to prevent it.
For now, I hope people acting in good faith, backed by meaningful human oversight and enforceable controls, can keep us ahead of the risks. Maybe this is what pacing frontier is all about: making time until we find better solutions, or simply giving us the time to enforce that oversight and stay ahead.
Some think we can use the kill switch; unfortunately, I don’t think there’s one anymore. This is a reality we have to deal with, whether we do it successfully or not.