{"slug": "cheap-reward-hacking-detection", "title": "Cheap Reward Hacking Detection", "summary": "Researchers trained a small transformer encoder to detect reward hacking in reinforcement learning trajectories by mapping them onto a unit sphere where embedding distance approximates reward-metadata signal differences. The method achieved an AUC of 0.9467 and a true positive rate at 5% false positive rate of 0.8296, matching the performance of a sanitized LLM-as-judge approach while costing roughly four orders of magnitude less per trajectory. The encoder's detection capability depends on natural-language reasoning in its input, as stripping that information dropped AUC to 0.6213.", "body_md": "# Computer Science > Machine Learning\n\n[Submitted on 8 Jun 2026]\n\n# Title:Cheap Reward Hacking Detection\n\n[View PDF](/pdf/2606.08893)\n\n[HTML (experimental)](https://arxiv.org/html/2606.08893v1)\n\nAbstract:A small transformer encoder is trained to map Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the $L_1$ distance between reward and metadata signals. A linear probe on top of that embedding detects reward hacking on the cleaned test split with AUC $0.9467$ and TPR@5%FPR $0.8296$, matching the TW sanitized LLM-as-judge AUC ($0.9510$ on the cleaned split) and exceeding its TPR@5%FPR ($0.7130$ vs $0.8296$) on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader: stripping natural-language reasoning from its input at probe time drops AUC to $0.6213$.\n\n### Current browse context:\n\ncs.LG\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/cheap-reward-hacking-detection", "canonical_source": "https://arxiv.org/abs/2606.08893", "published_at": "2026-06-11 19:16:05+00:00", "updated_at": "2026-06-11 19:24:11.306694+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-safety", "neural-networks"], "entities": ["Terminal-Wrench", "LLM-as-judge"], "alternates": {"html": "https://wpnews.pro/news/cheap-reward-hacking-detection", "markdown": "https://wpnews.pro/news/cheap-reward-hacking-detection.md", "text": "https://wpnews.pro/news/cheap-reward-hacking-detection.txt", "jsonld": "https://wpnews.pro/news/cheap-reward-hacking-detection.jsonld"}}