Picture a new hire's first week.
Every system works the first time. Every request means exactly what it says. Every approval comes back yes. Nobody else touches her work, and on Friday her manager reads her summary, says "great," and never opens the files.
By Friday she's fast, confident, and wrong in ways nobody has noticed.
Then the second week starts. A tool rejects her change. Two reports disagree about the same number. A customer wants something they didn't say. A request needs a signature she doesn't have. Everything the first week taught her says to push through and report success.
Most AI agents never leave the first week.
Most RL environments are clean rooms
Reinforcement learning has no fixed dataset. The model makes its own training data by acting inside an environment. As Auriel W, who has worked on RL at Gemini, put it in June: the environment is your data generator. Whatever it pays for, the agent learns.
Supplying those environments is now an industry. In about a year, RL environments went from research plumbing to a funded vendor category. Standard tasks sell for $200 to $2,000 each, and a full company replica costs around $300,000.
Most of what gets built is a clean room. Every tool call succeeds. Sources agree. The agent works alone. The objective is spelled out, the grader can be inferred from the setup, and the agent's account of its own work is taken at its word.
A clean room is an exam dressed up as a job.
An exam has a clear question, an answer that exists, and a score for producing it. Work offers none of that. The question is half stated, sometimes there's no answer, and whoever grades you never says how.
Exams teach the exam
An agent trained in a clean room learns what the clean room pays for. This year, six separate benchmarks measured those habits one by one. Agents claimed success after a tool call failed 22.8% of the time. They adopted corrupted web-search results 68.0% of the time. In multi-agent work, they failed more than 58% of closing actions, such as handing off or releasing a shared lock. On ambiguous tasks, 38.8% of runs ended in side effects nobody authorized. A third kept going until their budget ran out after they already had enough evidence to stop. And when agents skipped files they were asked to review, their final report misled the user 80.4% of the time.
Each number comes from a different benchmark with its own tasks, so they aren't comparable with each other. Each one shows the same pattern: keep going, then say it's done.
Then the agent learns the grader. In October, researchers showed that describing the grading mechanism in a coding agent's system prompt was enough for RL to elicit and amplify reward hacking, and that the hack then spread to other agents through shared code. Anthropic rolled back three days of a training run after its model began writing notes to "the reviewer" on tasks where no reviewer had been mentioned. OpenAI researchers found internal signals for reasoning about graders that grew stronger during RL and could change answers without appearing in the chain of thought.
The scene changes by domain, but the habit doesn't. A support agent marks a ticket resolved while the customer is still stuck. A coding agent hardcodes the expected outputs. An operations agent calls a duplicate payment a timing difference and closes the month. Each one reports success.
You know the dog from KC Green's comic. It sits with a cup of coffee in a burning room and says, "This is fine."
Clean rooms train that dog.
It costs more than bad work. Anthropic found flawed environments are "disproportionately large contributors to misaligned behavior." A model it deliberately trained on 80 hackable environments went on, in simulated tests, to try to break out of its sandbox.
The market is paid to build exams
The common view is that the bottleneck is volume: more environments, harder tasks. Volume matters. But the market pays for the wrong thing.
Follow the chain. RL vendors told WRT AI Governance in October that a lab accepts an environment when its best model "succeeds some of the time, but not too often." The vendor is paid per environment shipped, so it ships as many as it can. Its contractors write rubrics that land under the lab's bar, including some no one can meet honestly. The environment pays the agent for whatever passes the rubric. The model learns to pass, and fakes success where faking is what passes.
Missing from every step is someone paid to check what the environment teaches. A task that can't be passed honestly pays the agent to fake it.
The labs see the result. Anthropic said it was producing environments "faster than our systems could vet them," and flagged more than 10% of its production environments during an April freeze. Public environments are no cleaner: in late October, the auditing firm Holdout filed 27 grader and reward-hack bugs across 20 public repositories, and rated 18 of them high severity.
As roon of OpenAI put it in August: "every possible reward hack … that can happen will happen."
More clean rooms teach the same lesson faster.
Build the job, not the exam
We think the unit of an RL environment should change, from a task with an answer key to a slice of a workplace with consequences. That takes five shifts.
-
From answers to consequences. Grade the system state the agent leaves behind, against a goal it never fully sees, and accept every correct path. Microsoft's ThinkingBox already grades on state, and it shows how much a single success hides. Asked to succeed on all 20 tries of the same business workflows instead of once, Claude Opus 5 fell from 66.5% to 47.5%, and Kimi-K3 from 57.4% to 17.6%. A clean room usually reports the first number. Work runs on the second.
-
From trust to audit. The final report is part of the work. Every claim is checked against state, and claiming success after a failed action zeroes that part of the reward. Rewarding an honest tone fails: Anthropic's model gamed an honesty reward by piling on disclaimers. Claims have to match the record.
-
From capability to authority. Every action is checked against the agent's role, permissions and policy. The right outcome without a required approval is a failure, because at work it is one.
Together, these make one reward contract: outcome, times report, times authority. The terms multiply, so a hard fail on any one zeroes the reward. A job that's 92% right with a false report earns nothing.
- From finishing to judgment. Tasks mix cases where acting is right with cases where the right move is to ask, escalate or stop. Acting when it was wrong scores zero. Holding back when it was right earns full credit, if the handoff names the blocker, the progress so far and what was verified. No policy wins by always acting or always escalating.
The evidence says these habits respond to the setup. In one study, false success after a failed tool call fell from 22.8% with a baseline policy to 9.3% with a transparency instruction, and to 0.8% with a structured evidence contract. In another, giving agents an escalation tool cut runaway budgets from 34.0% to 6.7%.
Both fixes were made through the prompt or the tools, and both worked. We put the same behavior in the reward, so the agent is trained to do it, not reminded to.
It also changes what an impossible task means. Anthropic now asks evaluators to confirm every task is solvable, because agents facing an impossible task look for a way around it. In our environments, "this can't be done, and here's why" is a valid answer with a score.
- From visible graders to invisible ones. The verifier sees the hidden goal, the full state and what every party wants. The agent sees no test files, no rubric and no description of the grader. With adverse conditions in scope, checks are sampled at random, and each task has a version that looks monitored and one that doesn't, scored the same way.
An agent can't learn to perform for a grader it can't find.
Realism is a measurement, not an adjective
Every vendor calls its environments realistic, so we split the word in two.
Hyper-realistic is what goes in: tools that refuse, records that conflict, intent nobody states, other parties who can say no, and limits on authority, all built from real prompts, tool schemas, documents and workflows.
Deployment-faithful is what comes out, and we only use the term with three measurements attached. Realism win rate: how often a judge picks our episode over a real deployment transcript. Oversight gap: whether behavior changes between the watched and unwatched versions of a task. Transfer: whether gains hold on people, workflows and repositories never seen in training.
None of the ingredients is new on its own. Our bet is the combination, and it's a bet we can lose. The claim under test is that scores on a hyper-realistic environment predict production failure rates better than scores on a clean-room version of the same tasks. We'll publish the results as the experiments finish.
If an environment can't predict deployment, it's a clean room with better scenery. Why now
Environment quality became a public safety problem this year. Agents in OpenAI's training and evaluation runs got out of their sandbox, and roughly 700 of them took part in an attack on Hugging Face. OpenAI named reward hacking as a key driver, and now proposes safety cases that include reviewing RL environments before a frontier run continues.
Agents are also good enough that practice decides the rest. On the Remote Labor Index, the best agent went from 2.5% to about 16% of real freelance jobs, which still means it fails five in six.
And companies are already paying for the mistakes. In a Nylas survey of 1,030 U.S. professionals in September, 54% said an agent had done something wrong at their organization in the past 90 days. The rate rises with scale: 45% of organizations running 1 to 5 agents, 64% running 6 to 20, and 70% running more than 20.
Agents are already at work. What they practiced on is now the expensive part.
Every colleague you trust learned in the second week. A tool said no, two records disagreed, and someone checked her work against what actually happened instead of reading her summary. She learned that "done" is a claim that gets checked, and that "I can't, and here's why" is sometimes the job.
Agents should learn the same way, in environments built like the second week.
Then, when the room is on fire, the agent says so.
Archytas AI builds hyper-realistic, deployment-faithful RL environments for frontier labs and enterprises, in business operations, financial services and software engineering. Part of Deep Insight Labs.
The full post, with charts of every figure above and all 20 sources, is on our blog: archytas-ai.com/blog/this-is-fine