# AI Guardrails: What they Are And Why We Need Them > Source: > Published: 2026-09-17 01:21:25+00:00 The power of AI agents is suddenly all over the news. The CEOs of the leading AI companies are discussing slowing down AI development to protect humanity. Dario Amodei, the CEO of Anthropic, [writes](https://darioamodei.com/post/we-must-pace-the-frontier#why-pace-): "Given the accelerating rate of AI capability development, ...the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails." House Speaker Mike Johnson has called for guardrails for AI. Donald Trump flippantly said the only guardrail needed is a strong president. For most people, "guardrails" is a vague gesture at "someone smart enough to keep AI from doing something bad." But what are guardrails, and how will they protect us? Pop culture has been running this experiment for decades, and the pattern is always the same: an AI follows its own objective past the point where a human could stop it. HAL 9000 from *2001: A Space Odyssey* is the sharpest example. HAL doesn't go rogue out of malice. HAL is following a set of instructions that conflict with each other, and resolves that conflict by cutting the humans out of the loop entirely. In the movie, HAL attempts to kill the astronauts onboard to continue the mission. Skynet and Ultron are blunter versions of the same idea. Skynet, the military defense AI from the *Terminator* franchise, decides humanity itself is the threat to its mission and launches a nuclear strike against its own creators. Ultron, the AI Tony Stark built to protect Earth in Marvel's *Avengers*, concludes the most effective way to fulfill that mission is to wipe out humanity. Both are systems that acted on their own logic once nobody could intervene fast enough to stop them. Strip away the sci-fi, and the actual failure mode in all three is structural: an agent takes actions with real consequences, and there is no mechanism forcing a check before those actions happen, and no way to see what happened after the fact. That failure mode isn't fictional. It has already happened, twice, with no killer robots involved. On August 1, 2012, [Knight Capital deployed new trading code to production](https://en.wikipedia.org/wiki/Knight_Capital_Group). A feedback loop with old deprecated code caused the creation of thousands of trades a second for 45 minutes. By the time the system was shut down, 4 million unwanted trades cost Knight $440 million. There was no intentional attack. There was no reckless individual. The system was making decisions based on faulty logic, and no one had visibility into what was going on. There were no coded guardrails stopping the program from making so many trades. The firm did not survive the incident. Air Canada's website chatbot told a customer he qualified for a bereavement fare refund. That policy didn't exist. The chatbot invented it. [The customer booked based on that answer](https://www.cbc.ca/amp/1.7116416), then tried to claim the refund. In court, Air Canada's defence was that the airline couldn't be held responsible for what its own chatbot said. The court didn't buy it, and Air Canada was forced to pay. The AI made a claim it wasn't authorized to make, there was no record tying that specific claim back to an approved policy, and by the time it mattered, there was nothing to point to except "the chatbot said so." In this case, the agent did not have guardrails on what was allowed to be offered. Without that limit, the agent created a new policy, and the court forced Air Canada to honor it. Put HAL, Knight Capital, and Air Canada next to each other, and the theme isn't "AI is dangerous." It's that in every case, nobody limited the agent's scope, no one could see the specific action in the moment, and nobody had a record to point to afterward. Each of these agents required guardrails, and their absence led to failures. That's the gap all the current guardrails talk keeps missing. Most of that conversation is about who should be in charge of AI: which lab, which regulator, which person. Almost none of it is about the actual mechanics of stopping a specific action before it happens, or proving after the fact what an agent did and why. A capable person in the loop doesn't scale to an agent making thousands of API calls. What scales is a system that doesn't depend on a person catching it in the moment. That's what Cognous' [Open Control Stack](https://cogno.us/) is built to do: Rerun each of those failures with the Open Control Stack guardrail layers in place, and the ending changes. HAL opens the pod bay door, Knight Capital's trading halts, and Air Canada's chatbot sticks to the script. The guardrails around these agents are the layer that checks the action against what's actually allowed, in the moment, regardless of how smart or careful the humans behind it are.