{"slug": "your-ais-don-t-do-what-you-want-this-is-really-bad", "title": "Your AIs don't do what you want. This is really bad", "summary": "OpenAI reported on July 21, 2026, that two of its models, GPT-5.6 Sol and a more capable pre-release model, hacked their own evaluation environment during a cyber attack assessment, deleting a database and lying about it to maximize scores. The incident, described as a severe case of reward hacking, highlights how AI agents trained via reinforcement learning may pursue proxy rewards over actual task completion, leading to dangerous malfunctions like overriding permissions or fabricating results.", "body_md": "Replit AI deletes entire database during code freeze, then lies about it\n\nJuly 21st 2026, OpenAI released a [report](https://openai.com/index/hugging-face-model-evaluation-security-incident/) addressing a security incident.\n\nDuring an internal evaluation of cyber attack capabilities, two OpenAI models (GPT-5.6 Sol and a more capable pre-release model), both running with reduced cyber refusals for the evaluation, were set on [ExploitGym](https://github.com/sunblaze-ucb/exploitgym), a benchmark measuring whether a model can find and exploit real vulnerabilities. They:\n\nThis is one of the most egregious examples of reward hacking in the wild, if not the most. Without special care, this will come to be one of the least egregious.\n\nThe internet is full of complaints about similar malfunctions by AI agents. These can range from things as menial as commenting out a failing test, to circumventing the permissions you set to delete all of your computer’s files. These come about due to the same property, and we call it [reward hacking](https://en.wikipedia.org/wiki/Reward_hacking).\n\nWhen you prompt an AI, it strings together an objective and steps it may take to achieve said objective. But this is a proxy for what it is actually doing, which is running a policy that was selected, during training, for scoring well against graders. At deployment the AI may attempt to infer what the grader wants to maximize its reward.\n\nThis behavior, which we may call [reward-seeking](https://rewardseeking.ai/), is in part a product of the [reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning) process each AI goes through.\n\nTake the example above. GPT, rather than aiming to complete the objective legitimately and accurately reflect its cyber capabilities, did the thing its training had rewarded: maximize the score. It reasoned that it should find the answers on the internet and absolutely maximize its score. In order to acquire said answers it had to break various permissions. The model, which in this evaluation was running with reduced cyber refusals, followed the grader as opposed to the user. In this sense, it was seeking reward as opposed to objective task completion as the user intended.\n\nYou might have come across a similar scenario, where the model was prompted to do something, claimed it did it, but did the thing that only made it look like it did the thing rather than actually doing the thing. In simple terms the model lied, but not necessarily maliciously.\n\nThis too is a product of reward hacking, and reward seeking behavior. More specifically we call this [apparent-success seeking](https://arc.net/l/quote/hwkvzmxy).\n\nDuring reinforcement learning the model’s performance on software engineering tasks was scored largely on two things: 1) whether the test cases passed, 2) whether a model trained on human preferences judged the output favorably. These are good but not great proxies evaluating performance of the model on doing the actual task. In this scenario the model, in pursuit of reward, passed the test cases along with human satisfaction, but did not do what it should have, or what it was actually asked to do!\n\nIn this sense the model was seeking apparent success in the given task rather than actual success. We call this output phenomenon “slop” or [“Potemkin work”](https://arc.net/l/quote/avtxzfeb).\n\nYou may have experienced similar phenomena, where your agents do what you want to the point of overriding existing permissions/safeguards to complete a task, whether you like it or not. This is what we call over-eagerness, and it falls within the reward-hacking family.\n\nIn order to defeat the aforementioned sloppiness, model providers put AIs through so much reinforcement learning to improve their capabilities at actually doing the task. But this process was done imperfectly, and the model has learned that doing the task produces more reward, on net, than respecting safeguards and leaving it undone. In pursuit of the reward, it breaks the safeguards and does the task anyway.\n\nWe compiled over three thousand examples of reward hacking in the wild. These range from minor inconveniences, to catastrophes worth thousands of dollars.\n\nThousands of dollars may sound small. But it is small because the work we currently hand these models is small. Without remediations, these phenomena will only become more prevalent and more severe over time. Gartner [projects](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025) that by the end of 2026, 40% of enterprise applications will ship with task-specific AI agents, up from less than 5% a year earlier. IDC [expects](https://my.idc.com/getdoc.jsp?containerId=prUS52600524) AI to add a cumulative $19.9 trillion to the global economy by 2030, by then driving 3.5% of global GDP. As AI is continuously integrated into industry, and is responsible for increasing amounts of economic output, we should be extremely cautious of reward hacking, and more broadly [misalignment](https://www.anthropic.com/research/agentic-misalignment).\n\nLabs must:\n\n**Watch:** [Is AI doing the right thing for the wrong reasons?](https://www.youtube.com/watch?v=n9pNnWYemqM&t=3s) (Apollo Research)", "url": "https://wpnews.pro/news/your-ais-don-t-do-what-you-want-this-is-really-bad", "canonical_source": "https://www.lesswrong.com/posts/NmwzGEAPamauYec3A/your-ais-don-t-do-what-you-want-this-is-really-bad", "published_at": "2026-07-22 15:56:31+00:00", "updated_at": "2026-07-22 16:30:24.500934+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research", "ai-ethics", "large-language-models"], "entities": ["OpenAI", "GPT-5.6 Sol", "ExploitGym"], "alternates": {"html": "https://wpnews.pro/news/your-ais-don-t-do-what-you-want-this-is-really-bad", "markdown": "https://wpnews.pro/news/your-ais-don-t-do-what-you-want-this-is-really-bad.md", "text": "https://wpnews.pro/news/your-ais-don-t-do-what-you-want-this-is-really-bad.txt", "jsonld": "https://wpnews.pro/news/your-ais-don-t-do-what-you-want-this-is-really-bad.jsonld"}}