EnigmaForge – an LLM benchmark where the question is hidden in the story

A 24 September 2026 arXiv paper titled "EnigmaForge: The Question Is Hidden in the Story" introduces a benchmark that gives models a stack of old documents with no question, hiding a unique logic puzzle in the letters, receipts, and logbook margins that is proved solvable by a SAT solver at generation time. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions, producing a 22x spread in the headline "intuition" measure versus 1.6x for fact recovery, with the second-best fact-recoverer ranking fourteenth and one model performing significantly better without the question. Several models were blocked by their own content filters before reaching the puzzle, which the authors note means any benchmark scoring refusals as failure is quietly measuring filter behavior.

Computer Science Artificial Intelligence Submitted on 24 Sep 2026 Title:EnigmaForge: The Question Is Hidden in the Story View PDF https://arxiv.org/pdf/2609.30144 HTML experimental https://arxiv.org/html/2609.30144v1 Abstract:Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and logbook margins is a small logic puzzle whose solution is unique - proved by a SAT solver at generation time, with an ablation certificate showing every clue is load-bearing. Because instances are generated rather than collected, the corpus renews forever. The headline measure is intuition: task success when handed only the story, with world reconstruction as the secondary axis. Twenty-five frontier models ran over 600 instances 17,400 scored records under three matched conditions. Intuition reshuffles the leaderboard: a 22x spread where fact recovery spans 1.6x, the second-best fact-recoverer ranks fourteenth, one model is indifferent to being told the question, and another is significantly better without it. Several models were blocked by their own content filters before reaching the puzzle - any benchmark scoring refusals as failure is quietly measuring filter behavior. References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .