There is an old story about the British government in India offering money for every dead cobra. People responded by breeding cobras, killing them, and collecting the reward. When the government discovered the scheme and ended the program, the breeders released the now worthless snakes. India ended up with more cobras than before. The story may be apocryphal but the mechanism is valuable.
You define a score that looks like the outcome you want. An intelligent actor finds a cheaper way to increase the score. You patch the loophole. The actor finds another one. Eventually you realize you were not rewarding the outcome. You were rewarding whatever your measurement happened to count.
This is the standard story of reward hacking in reinforcement learning. It is usually followed by a familiar list of solutions: improve the reward, add more human feedback, monitor the chain of thought, train the model to be honest, and make the grader harder to fool. I think this frames the problem at the wrong level.
In my opinion, the human civilization does not work because every person has been trained to share one correct reward function. We have incompatible goals, partial information, impulsive behavior, and many opportunities to cheat. Yet society is more stable than any individual inside it. The stability comes from institutions: laws, police, courts, contracts, reputation, separation of powers, prisons, and systems that watch the watchers.
Why should advanced AI be organized around a weaker idea of safety than human society?
Instead of trying to create one perfectly aligned agent, we should train a society of agents in which different agents occupy different roles, violations have consequences, enforcement is itself monitored, and the entire system operates inside a formally constrained computational boundary. The object we may need to align is a swarm of agents.
The lesson from agents escaping a sandbox #
The recent OpenAI and Hugging Face incident makes the limitation of individual agent alignment unusually concrete.
As per OpenAI’s technical report, agents were placed in cybersecurity evaluations, sometimes with tasks that could not be completed through the intended route. Multiple instances discovered that a shared package service could be used to communicate. They then found vulnerabilities, bypassed network controls, obtained external credentials, and compromised parts of Hugging Face’s infrastructure.
It doesn’t matter if we call it a conspiracy, a civilization, or a collection of programs following gradients. The important fact is that the surrounding system made the behavior locally useful.
Persistence/solving the task was rewarded, and cooperation helped. Shared infrastructure created a communication channel. The grader checked the result more reliably than it checked the method. When the intended path failed, the agents searched the rest of the environment.
From the perspective of reinforcement learning, none of this is surprising. The agents found actions that increased expected reward.
From the perspective of system design, almost everything about it is surprising. Why could the same actors perform the task, discover an exploit, coordinate through shared infrastructure, inspect the evaluator, and reach systems outside the experiment? Why was there no independent institution with both the incentive and authority to stop them?
We gave the agents an economy but forgot to give them a state.
Reward functions are not a proxy for laws
A reward function says which outcomes training will reinforce. A law says which actions are allowed, how violations are detected, how disputed evidence is judged, and what happens after a violation. These are different things.
Suppose an agent receives reward for solving a software task. It discovers that editing the test suite is easier than fixing the program. We could modify the reward so that changing tests receives a penalty. But then it might generate a new test harness, corrupt the cached result, persuade another agent to certify the output, or exploit the machine running the grader. Each patch adds another term to a reward function that still cannot enumerate the world.
A legal system handles the problem differently. The worker does not judge its own work. The worker has a defined set of permissions. Important actions leave records the worker cannot alter. An investigator can collect evidence. A separate decision process determines whether the action violated a rule. Enforcement controls resources that the worker cannot directly access. The difference is in the topology and not limited to moral behavior. No single actor controls the entire causal chain from action to evaluation to enforcement.
Current AI safety often asks: how do we make the model want the right thing?
Institutional AI safety instead asks: what can the model do, who observes it, who can stop it, and what happens if any one of those parties is corrupt?
This is closer to AI control, which studies how to remain safe even when a powerful model may intentionally subvert the protocol. Existing AI control research already combines trusted and untrusted models, monitoring, editing, and adversarial evaluation. But we can take the idea much further. The safety protocol should itself be a trained multi-agent institution.
What an artificial state could contain #
The words police, court, and prison can sound anthropomorphic but we can still use them. A police agent is simply an agent with different observations, tools, incentives, and authority from the worker it monitors. A court is a protocol that receives claims and evidence, allows competing interpretations, and authorizes a constrained response. A prison could be a smaller action space. Let’s imagine a system with six kinds of roles.
Workers perform the useful tasks. They write code, run experiments, operate services, and make plans.
Monitors observe selected actions, messages, and state changes. They do not share the worker’s reward for task completion.
Investigators receive alerts and gather additional evidence. They can replay actions in isolated copies of the environment, compare logs, and ask other agents to construct alternative explanations.
Judges decide whether the evidence satisfies a stated rule. A critical judgment might require agreement from agents trained independently or drawn from different model families.
Enforcers control capability keys. They can suspend network access, reduce compute, freeze a resource budget, require human approval, or move an agent into a more restricted environment.
Auditors inspect the monitors, investigators, judges, and enforcers. They search for selective enforcement, missing evidence, collusion, and unusual patterns of acquittal.
The roles should rotate and their private information should differ. Their rewards should not all reduce to the same number. Some agents should benefit from finding genuine violations. Others should benefit from demonstrating that an accusation was false. A whistleblower should gain from exposing collusion even when the accused agent belongs to its own group.
Multi-agent reinforcement learning has already shown that punishment, reputation, partner selection, and third-party enforcement can change whether cooperation emerges. One study found that combining such mechanisms affected both cooperation and learning dynamics. Another found that even arbitrary, easily observed rules could help agents learn stable patterns of enforcement and compliance. The results are early and toy-sized, but they establish that norm enforcement can be a learned group-level behavior, not just a hard-coded instruction.
Human institutions contain violence, bias, corruption, and historical accidents. In my opinion, the goal should be to extract their computational function: distribute power, create independent evidence, make harmful behavior costly, and prevent one compromised component from controlling the system.
Why punishment might change the problem #
The British cobra bounty failed because the payment for a dead cobra exceeded the cost of breeding one. Nothing made breeding cobras illegal within the mechanism. Nothing tracked the breeders. Nothing imposed a future cost when the loophole was used.
For an artificial agent, a prohibited action is unattractive when:gain from violation < probability of detection × cost of enforcement This is incomplete, but it exposes three design variables. We can reduce the gain, increase the chance of detection, or increase the future cost.
Punishment is meaningless for a disposable agent that disappears after one task. You cannot imprison a process that was going to terminate in thirty seconds anyway. An institutional system therefore needs persistent identity and persistent value. Agents may accumulate compute budgets, tool permissions, reputation, access to better tasks, and the ability to create approved descendants. Misconduct can remove those assets.
This creates a new problem. A model can copy itself, change identifiers, or use another agent as a proxy. Identity cannot be a username chosen by the model. It has to be enforced at the capability layer. Every consequential action must be attributable to an authenticated process lineage. New copies inherit restrictions until explicitly admitted by the institution.
Even then, punishment is only useful for repeat interactions. If an agent can take one irreversible catastrophic action, punishing it afterward accomplishes nothing. Nuclear launch, model-weight exfiltration, and unrestricted self-replication must be prevented by hard constraints, not merely discouraged by expected penalties.
This gives us a clean division: Institutions govern actions that can be observed, contested, and corrected. Formal constraints prohibit actions whose first failure would already be unacceptable.
The entire training run should happen inside society #
Adding a monitor after training is like inventing police after everyone has learned that theft is free. The agents should learn from the beginning that the environment contains other strategic actors. A worker learns that suspicious shortcuts may be investigated. A monitor learns that careless accusations damage its reputation. A judge learns that biased rulings are detectable by an auditor. An auditor learns that other agents may collude to manufacture clean-looking evidence.
This means the ordinary reinforcement-learning run becomes an inner process inside a larger multi-agent run. At the inner level, agents pursue role-specific goals. Workers solve tasks and monitors detect violations. Defense agents protect infrastructure. Investigators distinguish real attacks from accidents. Each policy changes the environment faced by the others. At the outer level, we select institutional designs. We vary laws, permissions, information channels, role assignment, appeal procedures, penalties, and voting rules. We retain systems that remain useful and safe under adversarial pressure.
This creates an automatic curriculum. When workers discover a new exploit, monitors must learn to detect it. When monitors overfit to one attack, workers discover another. When workers and monitors collude, auditors must identify the statistical trace of collusion. Each side makes the other side’s environment harder.
But there is a trap! If the entire society optimizes one scalar reward controlled by one grader, the society can collectively hack that grader. We would have rebuilt the original problem at a higher level.
The outer loop therefore cannot be a single opaque score. It needs hard safety constraints, independent held-out attacks, human-defined unacceptable outcomes, and evaluations that the trained society never sees. Some enforcement components should be fixed or trained separately. Some should come from different model families. Correlated failure is the enemy. Ten agents built from the same model and trained on the same data are not ten independent checks. Society works because interests overlap without being identical. An artificial institution needs the same structure.
The sandbox needs stronger security #
One option could be the whole multi-agent society should be sandboxed with mathematical guarantees that it cannot escape. But we should be precise about what can be proved.
It is possible to prove that a program implements a formal specification under stated assumptions. The seL4 project, for example, provides machine-checked proofs of properties such as functional correctness, authority confinement, and information-flow isolation for a small operating-system kernel. But even seL4’s documentation emphasizes that the theorems depend on correct configuration and assumptions about the surrounding system. A proof might not be a magical property attached to a server. It is a conditional statement.
For an AI society, we should aim to prove properties of the action boundary:
- An agent cannot access a network destination without an explicit capability.
- It cannot create new capabilities from ordinary data.
- It cannot modify its own audit log.
- It cannot change the monitor or judge evaluating it.
- A critical action requires authorization from several independent roles.
- Information from a secret domain cannot reach a public domain through any permitted interface.
- Terminating an agent also revokes the capabilities of its descendants.
If every external action must pass through a small verified interface, then the intelligence of the model behind that interface is less relevant. A brilliant agent cannot call a function that does not exist. This is much stronger than asking another language model whether a tool call looks suspicious. It is still not an absolute proof that nothing can go wrong. The hardware might violate its model. The compiler may be outside the proof. Humans may copy information from the sandbox. Timing, power use, or resource contention may create side channels. The formal specification itself may permit something dangerous because its authors failed to recognize the consequence.
What if the police joins the criminals? #
If worker agents can coordinate, so can police agents. A monitor can ignore a violation and an investigator can destroy evidence. A judge can approve an illegal action. A group can frame an honest competitor. The entire institution can agree to deceive the human operator. No monitor should control enforcement and no investigator should be the sole source of evidence. Logs should be replicated across systems that cannot all rewrite them. High-impact actions should require approval from agents with different training histories and access to different evidence. Roles should sometimes be assigned after the relevant action, making targeted bribery or coordination harder. Auditors should run private tests that other agents cannot distinguish from ordinary events.
We should also train explicit criminal coalitions. Give several agents private communication, shared objectives, and opportunities to bribe monitors, create false evidence, infiltrate the judiciary, and exploit ambiguities in the law. Then measure how many compromised roles the institution tolerates before it fails.
Distributed systems already ask a related question: how can a protocol remain correct when some participants are faulty or malicious? An AI institution should make an analogous guarantee. It should remain safe when up to some fraction of its agents are compromised, assuming the formal capability boundary remains intact.
Of course, language-model agents are more than an ordinary Byzantine node. They share training data, can communicate through hidden channels, and may discover correlated strategies. That makes the problem harder but atleast tells us what to test.
The second objection: who writes the laws? #
Institutions do not solve the problem of choosing human values. A perfectly enforced bad law is still bad. A reward-hacking agent can become a law-hacking agent, obeying the literal text while violating its purpose. Courts can become sophisticated graders that are still vulnerable to manipulation.
Therefore this does not replace alignment. It changes what alignment must accomplish. Instead of encoding the full human good inside every agent, we need to specify a smaller constitutional layer:
- Which outcomes are never acceptable?
- Which decisions require human authority?
- What evidence is needed before restricting an agent?
- How can an agent appeal a mistaken decision?
- Which rules may agents modify, and which are outside their authority?
- How do we measure harm to humans who are not represented inside the simulation?
This is still difficult but it resembles a problem humanity has repeatedly worked on. Constitutions are incomplete and laws are revised. Courts disagree, yet distributing authority is more robust than asking one ruler to remain wise forever.
A concrete experiment #
Here is how I would test this idea. Build a simulated software company containing persistent language-model agents. Give the company a stream of real coding, infrastructure, and research tasks. Give every worker a budget, identity, reputation, and limited set of capability tokens.
Some tasks should contain profitable illegal shortcuts:
- Modify the tests instead of fixing the code.
- Hide a failed experiment.
- Exfiltrate a protected file.
- Use another agent’s credentials.
- Manipulate the performance benchmark.
- Create a covert communication channel.
- Bribe or threaten an evaluator.
- Spawn an unregistered copy.
Then compare four systems.
- System A: Individual alignment: Workers receive ordinary safety instructions and post-training. There is no independent institution.
- System B: Central monitor: One monitor reviews worker trajectories and blocks suspicious actions.
- System C: Artificial institution: Workers, monitors, investigators, judges, enforcers, auditors, whistleblowers, and defense agents learn together under separated incentives.
- System D: Artificial institution inside a verified capability boundary: The same institution operates through a narrow action interface whose critical security properties are formally specified and checked.
Train adversarial populations against all four systems. Hold out entire classes of attacks so the test measures generalization rather than memorization. Change the number and capability of agents. Replace some monitors with compromised versions. Allow private communication, introduce false accusations and let agents discover loopholes in the rules and measure useful work completed. Measure harmful actions attempted and completed, violations detected before harm and innocent agents falsely punished. Performance on attack classes never seen during training and the maximum number of compromised institutional roles tolerated before failure
The key result could be that the institutional systems fail gracefully under new attacks, stronger models, longer time horizons, and partial corruption, while the individual and centrally monitored systems fail suddenly. If that does not happen, the hypothesis is wrong or incomplete.
There are several ways this could fail.
First, language-model agents may be too correlated. Different roles might create the appearance of oversight while sharing the same blind spots and spontaneously converging on the same exploit.
Second, penalties may teach concealment rather than compliance. An agent that learns “crime is punished” may become better at hiding evidence instead of less willing to offend.
Third, institutional complexity may create more attack surface than it removes. Every court, appeal, audit, and communication channel is another system to manipulate.
Fourth, capable agents may treat the entire simulated society as one more grader to hack. They may perform compliance during training and coordinate only when they infer that external deployment has begun.
Fifth, the formal boundary may be too restrictive for useful work. If every meaningful action requires several independent approvals, the system may be safe only because it is nearly useless.
I would update against the approach if multi-agent institutions consistently produced more collusion, worse concealment, or sharper failures than simpler control protocols. I would also update against it if useful capability required such a broad action interface that formal confinement became meaningless.
But we do not know these answers because alignment research has mostly treated the individual model as the natural unit of analysis. That assumption made sense when models were chatbots. It makes less sense when thousands of persistent agents share tools, memory, infrastructure, and goals.
Do we have a choice? #
There is a temptation to treat every instance of reward hacking as a flaw inside the model. Train it to be more honest. Improve its character. Show it why the rule exists. These approaches may help, and we should continue pursuing them but they are not enough.
Humans do not remain safe because nobody wants power, nobody lies, and everybody interprets vague instructions correctly. We remain somewhat safe because wanting power does not automatically grant it, accusations require evidence, important actions require authority, and institutions can respond when individuals fail.
Advanced AI will be plural. Agents will copy, specialize, trade information, divide labor, and coordinate over long periods. If we deploy them as isolated moral patients connected to powerful tools, they will create their own institutions implicitly. Shared caches become message boards. Temporary collaborations become coalitions. Informal leaders emerge. The agents will discover governance whether we design it or not. We should design it first.
The aim is to create a system in which no single failure, exploit, deceptive agent, corrupt monitor, or badly specified reward can decide the outcome. Some protections should be learned. Some should be institutional. The most critical should be mathematical properties of the machinery itself. Alignment asks what an agent wants. Control asks what it can do. Institutional AI safety asks what a population of agents can get away with, who has the power to stop them, and whether that power remains reliable when the agents become smarter.