The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest

A study by Michael Timothy Bennett, submitted to arXiv on January 30, 2023, and revised April 11, 2024, argues that the optimal hypothesis for generalization is the weakest, not the shortest, and that compression is neither necessary nor sufficient for maximizing generalization performance. In experiments comparing maximum weakness with minimum description length in binary arithmetic, the former generalized at 1.1 to 5 times the rate of the latter. The authors propose weakness as a proxy for generalization and suggest it explains the effectiveness of DeepMind's Apperception Engine.

Computer Science Artificial Intelligence Submitted on 30 Jan 2023 v1 https://arxiv.org/abs/2301.12987v1 , last revised 11 Apr 2024 this version, v4 Title:The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest View PDF /pdf/2301.12987 HTML experimental https://arxiv.org/html/2301.12987v4 Abstract:If $A$ and $B$ are sets such that $A \subset B$, generalisation may be understood as the inference from $A$ of a hypothesis sufficient to construct $B$. One might infer any number of hypotheses from $A$, yet only some of those may generalise to $B$. How can one know which are likely to generalise? One strategy is to choose the shortest, equating the ability to compress information with the ability to generalise a proxy for intelligence . We examine this in the context of a mathematical formalism of enactive cognition. We show that compression is neither necessary nor sufficient to maximise performance measured in terms of the probability of a hypothesis generalising . We formulate a proxy unrelated to length or simplicity, called weakness. We show that if tasks are uniformly distributed, then there is no choice of proxy that performs at least as well as weakness maximisation in all tasks while performing strictly better in at least one. In experiments comparing maximum weakness and minimum description length in the context of binary arithmetic, the former generalised at between $1.1$ and $5$ times the rate of the latter. We argue this demonstrates that weakness is a far better proxy, and explains why Deepmind's Apperception Engine is able to generalise effectively. Submission history From: Michael Timothy Bennett view email /show-email/39071763/2301.12987 Mon, 30 Jan 2023 15:29:40 UTC 281 KB v1 /abs/2301.12987v1 Mon, 6 Mar 2023 01:54:22 UTC 279 KB v2 /abs/2301.12987v2 Tue, 25 Apr 2023 07:23:31 UTC 58 KB v3 /abs/2301.12987v3 v4 Thu, 11 Apr 2024 05:02:10 UTC 58 KB Current browse context: cs.AI References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .