{"slug": "enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story", "title": "EnigmaForge – an LLM benchmark where the question is hidden in the story", "summary": "A 24 September 2026 arXiv paper titled \"EnigmaForge: The Question Is Hidden in the Story\" introduces a benchmark that gives models a stack of old documents with no question, hiding a unique logic puzzle in the letters, receipts, and logbook margins that is proved solvable by a SAT solver at generation time. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions, producing a 22x spread in the headline \"intuition\" measure versus 1.6x for fact recovery, with the second-best fact-recoverer ranking fourteenth and one model performing significantly better without the question. Several models were blocked by their own content filters before reaching the puzzle, which the authors note means any benchmark scoring refusals as failure is quietly measuring filter behavior.", "body_md": "# Computer Science > Artificial Intelligence\n\n  [Submitted on 24 Sep 2026]\n\n# Title:EnigmaForge: The Question Is Hidden in the Story\n\n[View PDF](https://arxiv.org/pdf/2609.30144)\n\n[HTML (experimental)](https://arxiv.org/html/2609.30144v1)\n\nAbstract:Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and logbook margins is a small logic puzzle whose solution is unique - proved by a SAT solver at generation time, with an ablation certificate showing every clue is load-bearing. Because instances are generated rather than collected, the corpus renews forever. The headline measure is intuition: task success when handed only the story, with world reconstruction as the secondary axis. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions. Intuition reshuffles the leaderboard: a 22x spread where fact recovery spans 1.6x, the second-best fact-recoverer ranks fourteenth, one model is indifferent to being told the question, and another is significantly better without it. Several models were blocked by their own content filters before reaching the puzzle - any benchmark scoring refusals as failure is quietly measuring filter behavior.\n    \n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story", "canonical_source": "https://arxiv.org/abs/2609.30144", "published_at": "2026-09-28 17:11:54+00:00", "updated_at": "2026-09-28 17:21:33.352732+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety"], "entities": ["EnigmaForge", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story", "markdown": "https://wpnews.pro/news/enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story.md", "text": "https://wpnews.pro/news/enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story.txt", "jsonld": "https://wpnews.pro/news/enigmaforge-an-llm-benchmark-where-the-question-is-hidden-in-the-story.jsonld"}}