Human-Like Neural Nets by Catapulting
type: NewsArticle
description: A speculative proposal argues that training massively overparameterized neural networks with extremely high learning rates on small, diverse datasets could trigger a "catapulting" effect, producing human-like generalization. This approach would resolve anomalies like sample inefficiency and adversarial vulnerability in current AI, potentially enabling safer, more robust models that resist cloning and adversarial attacks. The theory suggests testing with multi-trillion-parameter models trained for few steps on tasks like arithmetic and image classification.
date published: 2026-06-06 23:45:36+00:00
date modified: 2026-06-07 00:16:56.779696+00:00
language: en
is accessible for free: true
article body: # Human-like Neural Nets by Catapulting Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence. There are many mysteries about deep learning and human intelligence, but we could describe the biggest anomaly this way: why are artificial neural nets smart in such stupid ways, and biological brains stupid but in smart ways? I propose a major change in deep learning scaling paradigms: the architectural differences between human brains and NNs (particularly LLMs) may be due to a bias-variance tradeoff, where LLMs minimize variance and human brains minimize bias. Human brains do this by deep double descent-style overparameterization, and adopting a scaling strategy of extremely high-learning-rate training of extremely overparameterized models on small diverse highly-filtered datasets. This approach would lead to sample-efficiently and compute-efficiently traveling (or catapulting) to a highly-generalizing human-like basin in the model loss landscape, while performing poorly up until the end and failing to memorize much data.If true, this would explain a number of odd stylized facts about how humans/NNs perform well/poorly. Such a ‘catapulted LLM’ would generalize much better than existing NNs, be immune to adversarial attacks, have better economics and be more resistant to cloning, could potentially enable extremely efficient [MLP]architectures, and by giving true generalization, provide a sturdy foundation for AI safety in the form of useful NNs which are aligned & safe for therightreasons.This could be feasibly tested by training multi-trillion-parameter models for relatively few steps at high cyclical learning rate schedules, and benchmarking adversarial and hard examples on tasks like arithmetic and sma…
publisher: Web Pulse
keywords: artificial-intelligence, machine-learning, neural-networks, large-language-models, ai-research
citation: CreativeWork