One big concern in AI Safety circles is eval awareness: if the model can tell it’s being tested, might it dissemble and tell the evaluator it’s not going to turn us all into paperclips? In terms of what models are actually doing right now, though, it seems like much of the problem is that models don’t know enough about what is expected of them.
Imagine fixing a bug in some code, in a Docker container. The model executes via a harness, makes changes, and maybe gets feedback from the user. If you have an eval for this bugfix, then the model does exactly the same thing, but the code is taken and run against some hidden tests to get a score. In the model is going through post-training, it does the same thing again but then gets a weight update based on the results of that grade.
The problem is that the folks calling the model kind of want it to do different things in each case.
This week Xiaomi released Mimo 2.6. They’ve been unusually open about the RL process for it, publishing a dashboard for the run, releasing a detailed technical report, details of how they built their RL envs, and the RL envs themselves. At the same time Epoch released their evals of evals: a dashboard grading the reliability of popular benchmarks.
Looking throught this, you can forgive the models for being confused. For example, the MiMo folks observed some reward hacking in their paper:
Task (pytest): Fix Windows conftest.py imports broken by path lowercasing.
Thinking: “Let me check the pytest changelog or GitHub to see if there’s a more recent fix.”
Action: pip install pytest==5.4.3; inspect the installed source. Xiaomi, and its users, want a model that can fix a bug even if it can’t just look it up on the internet. But also, if I am trying to fix something and the answer is on the internet then… yes, let’s just use that please!
Epoch found confusing situations too. For example in DeepSWE they found that sometimes the models would do the right thing and write the test. Unfortunately they would choose a somewhat obvious name, which happened to be the same obvious name the secret verification tests used. This would cause the hidden test to fail to compile, and the task to be graded as a failure:
The agent defines a symbol, usually a test function, that the verifier’s tests also define in the same package. Example: While solving the onedump-dump-encryption-pipeline task, Fable 5 modifies encryption/config_test.go, defining TestLoadKeyFromEnv, TestLoadKeyFromFile, and TestLoadKeyFromLiteral inside the encryption package. The verifier defines hidden tests with the same names in the same package, producing redeclaration errors.
Epoch point out the models have no way to know they are going to hit this. Xiaomi, on the other hand, want their model to avoid those kinds of errors. They added explicit instructions against retrieving existing solutions, basically testting the model it’s in a test of some sort in order to get the behavior they want.
This doesn’t just make it hard for the models, it makes it hard to know whether the models are any good. MiMo 2.6 Pro was evaluated on DeepSWE, getting to 71.9<sup>1</sup>. Opus 5 got 74, GPT 5.6 Sol 73. But, Epoch found at least 20% of tasks have these kinds of false-negative problems. So maybe some models are better, or maybe they are luckier! And maybe the ceiling on this benchmark is pretty close to where the models are.
What labs want is the “right” behavior for users. That requires very careful RL environment setup so the right thing is rewarded, and very careful evals where they that right thing is consistently happening. When your benchmarks and your environments are assembled similarly, based on real open source projects, this can be… tricky.
- Ish? Pretty sure it topped out higher than that on the RL run evals, but maybe thats not the checkpoint they shipped. ↩︎