The 1991 Unnormalized Linear Transformer (ULTRA)

Jürgen Schmidhuber, a pioneer in artificial intelligence, published the first Transformer variant, the unnormalized linear Transformer (ULTRA), in March 1991, which scales linearly in input size compared to the quadratic scaling of Google's 2017 Transformer. ULTRA introduced self-supervised pre-training and fast weight programming, and its mathematical equivalence to modern linearized self-attention was later recognized. Schmidhuber's 1993 extension discussed 'internal spotlights of attention,' predating the 'attention is all you need' terminology.

The T in ChatGPT GPT3 GPT3 stands for an artificial neural network NN called Transformer . In March 1991, when compute was millions of times more expensive than today, even before the LSTM deep-learning-history.html lstm , Schmidhuber published a first Transformer variant, which is now called the unnormalised linear Transformer . ULTRA ULTRA FWP0 FWP0 It had to be more efficient than Google's 2017 quadratic Transformer: TR1 TR1 ULTRA's computational costs scale linearly in input size, rather than quadratically in 1991, no journal would have accepted an NN that scales quadratically . In 1991, Schmidhuber also introduced self-supervised pre-training very-deep-learning-1991.html to enable deep learning in sequence-processing NNs see the P in ChatGPT . UN UN0-2 UN His 1993 recurrent ULTRA extension FWP2 FWP2 talked about learning "internal spotlights of attention neural-attention-1990-1993.html ”—compare the recent attention terminology, e.g., "attention is all you need," TR1 TR1 and tweets of 2022 https://twitter.com/SchmidhuberAI/status/1576966129993797632 & 2023 https://x.com/schmidhuberai/status/1864701357107634390 . Like modern quadratic Transformers, the 1991 ULTRA is highly parallelizable. It was a by-product of more general research on NNs that learn to program the fast weights of other NNs fast-weight-programmer-1991-transformer.html . FWP FWP The 1991 experiments were similar to today's: predict some effect, given a sequence of inputs. FWP0 FWP0 FKI-147-91ocr.pdf How does the unnormalized linear Transformer work? There are two feedforward NNs FNNs called the slow net and the fast net. The slow net has a special unit for each fast net unit from which at least one fast connection is originating. In 1991, the vector of real-valued activations across these units was called FROM blue in the image ; in today's Transformer terminology it's called KEY. The slow net also has a special unit for each fast net unit to which at least one fast connection is leading. In 1991, the vector of activations across these units was called TO red in the image ; today it's called VALUE. At every time step of sequence processing, each fast weight may rapidly change in proportion to the product of the current activations of the corresponding units in KEY and VALUE generated by the slow net. This product is simply added to the fast weight which then may be normalized by a squashing function FWP0 FWP0 . The additive part by itself essentially overcomes the vanishing gradient problem. FWP FWP The current INPUT to which the fast net is applied is called the QUERY. Essentially, the QUERY is processed by the fast weight matrix , which is a sum of outer products of previously generated KEYs and VALUEs ignoring normalizations and projections . The KEYs/VALUEs/QUERIES implement READ/WRITE operations on the separate storage represented by the fast network. Since all operations of both networks are differentiable, we obtain end-to-end differentiable active control of fast weight changes through additive outer products . FWP0-3a FWP0 Hence the slow net can learn by gradient descent in some given error function to rapidly modify the fast net during sequence processing, by inventing good context-dependent KEYs and VALUEs at the right times. This is mathematically equivalent to what was later called an unnormalised "linear Transformer" with "linearized self-attention." FWP6 FWP6 TR5-6a TR5 DLH DLH ULTRA ULTRA The "quadratic" Transformers of 2017 TR1-2 TR1 are a combination of Schmidhuber's 1991 additive outer product fast weight principle FWP0-2 FWP0 and softmax : attention QUERY, KEY, VALUE ~ softmax QUERY KEY VALUE. The attention neural-attention-1990-1993.html weights in Transformers can be viewed as context-dependent weight vectors or NN-programmed fast weights . FWP FWP In the interest of efficiency, linear Transformers of 2020-21 TR5-6 TR5 abandoned the softmax, essentially resurrecting the original 1991 system, ULTRA ULTRA FWP0-1 FWP0 FWP FWP whose costs scale linearly in input size, rather than quadratically. TR1 TR1 Of course, plain outer products in NNs go back at least to Konorski's informal 1948 rule HEB48 HEB48 later often called the "Hebb rule" HEB49 HEB49 and concrete formal implementations through Steinbuch's Learning Matrix around 1960. ST61-63 ST61 AMH1-2 AMH1 KOH72 KOH72 LIT74 LIT74 PAL80 PAL80 See also Kosko's bidirectional associative memories. KOS88 KOS88 However, these authors described pre-wired rules to associate user-given patterns with each other. Their systems did not learn by gradient descent to use such rules for associating self-invented KEY/VALUE patterns, like the ULTRAs and other Transformers since 1991. ULTRA ULTRA Neither did early NNs with fast weights by Malsburg 1981 and others. FAST FAST FASTa,b FASTa DLP DLP The 1991 ULTRAs are essentially NN-programming NNs whose elementary programming instructions are additive outer product rules. What was the key novelty? Errors are backpropagated through these differentiable rules such that ULTRA can learn to minimise its objective function by invoking and using the rules wisely, generating appropriate KEYs/VALUEs at the right times to create useful changes of fast weights. Later FWPs used more complex elementary programming instructions, e.g., the delta rule FWP6 FWP6 and its extensions. LT25 LT25 LT25c LT25c This is closely related to metalearning metalearning.html META1 META1 META META with self-referential NNs that can learn to execute and modify their own weight change algorithm. FWPMETA1-10 FWPMETA1 Note that even an NN with fixed weights can still learn, COCO COCO HO1 HO1 and that an NN can learn to implement backpropagation who-invented-backpropagation.html , FWPMETA6 FWPMETA6 then improve backpropagation by backpropagating errors through the differentiable backpropagation algorithm itself. Schmidhuber offered the 1991 ULTRA ULTRA ULTRA FWP0-1 FWP0 as an alternative to sequence-processing recurrent NNs RNNs deep-learning-miraculous-year-1990-1991.html Sec.%200 , the computationally most powerful NNs of them all. UN UN MIR MIR Sec. 0 deep-learning-miraculous-year-1990-1991.html Sec.%200 Modern Transformers are also viewed as RNN alternatives, despite their limitations. TR3-4,7-8 TR3 The 1991 experiments were similar to today's: given a sequence of sensory inputs, predict some effect, without using RNNs. FWP0 FWP0 Recent work on linear Transformers and similar Fast Weight Programmers Today, many researchers want to develop faster and better alternatives to quadratic Transformers, and as of 2025, there has been lots of recent work on linear Transformers and similar Fast Weight Programmers fast-weight-programmer-1991-transformer.html .e.g., LT23-25 LT23 FWP23-25b FWP23 This is also relevant for neurobiology. FWP25c FWP25c See also: who invented transformer neural networks? who-invented-transformer-neural-networks.html TR25 TR25 Acknowledgments Thanks to several expert reviewers for useful comments. Since science is about self-correction, let me know under juergen@idsia.ch if you can spot any remaining error. The contents of this article may be used for educational and non-commercial purposes, including articles for Wikipedia and similar sites. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License http://creativecommons.org/licenses/by-nc-sa/4.0/ . References AC J. Schmidhuber AI Blog blog.html , 2021 . 3 decades of artificial curiosity & creativity artificial-curiosity-since-1990.html . Our artificial scientists not only answer given questions but also invent new questions. They achieve curiosity through: 1990 the principle of generative adversarial networks, 1991 neural nets that maximise learning progress, 1995 neural nets that maximise information gain optimally since 2011 , 1997 adversarial design of surprising computational experiments, 2006 maximizing compression progress like scientists/artists/comedians do, 2011 PowerPlay... Since 2012: applications to real robots. AC90 J. Schmidhuber. Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments. Technical Report FKI-126-90, TUM, Feb 1990, revised Nov 1990. PDF FKI-126-90ocr.pdf . The first paper on long-term planning with reinforcement learning recurrent neural networks NNs more deep-learning-miraculous-year-1990-1991.html Sec.%2011 and on generative adversarial networks where a generator NN is fighting a predictor NN in a minimax game more deep-learning-miraculous-year-1990-1991.html Sec.%205 . AMH1 S. I. Amari 1972 . Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions, C 21, 1197-1206, 1972. PDF. amari1972hopfield.pdf First publication of what was later sometimes called the Hopfield network AMH2 AMH2 NOB NOB or Amari-Hopfield Network. AMH2 J. J. Hopfield 1982 . Neural networks and physical systems with emergent collective computational abilities. Proc. of the National Academy of Sciences, vol. 79, pages 2554-2558, 1982. The Hopfield network or Amari-Hopfield Network was published in 1972 by Amari. AMH1 AMH1 NOB NOB ATT J. Schmidhuber AI Blog blog.html , 2020, updated 2025 . 30-year anniversary of end-to-end differentiable sequential neural attention. Plus goal-conditional reinforcement learning. neural-attention-1990-1993.html Schmidhuber had both hard attention for foveas 1990 and soft attention in form of Transformers with linearized self-attention 1991-93 . FWP FWP Today, both types are very popular. ATT0 J. Schmidhuber and R. Huber. Learning to generate focus trajectories for attentive vision. Technical Report FKI-128-90, Institut für Informatik, Technische Universität München, 1990. PDF. FKI-128-90ocr.pdf ATT1 J. Schmidhuber and R. Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2 1 & 2 :135-141, 1991. Based on TR FKI-128-90, TUM, 1990. PDF https://sferics.idsia.ch/pub/juergen/attention.pdf . More. attentive.html ATT2 J. Schmidhuber. Learning algorithms for networks with internal and external feedback. In D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, editors, Proc. of the 1990 Connectionist Models Summer School , pages 52-61. San Mateo, CA: Morgan Kaufmann, 1990. PS hinton.ps . PDF hinton-rev.pdf . BPA H. J. Kelley. Gradient Theory of Optimal Flight Paths. ARS Journal, Vol. 30, No. 10, pp. 947-954, 1960. Precursor of modern backpropagation who-invented-backpropagation.html . BP1-4 BP1 BP1 S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis in Finnish , Univ. Helsinki, 1970. See chapters 6-7 and FORTRAN code on pages 58-60. PDF http://www.idsia.ch/~juergen/linnainmaa1970thesis.pdf . See also BIT 16, 146-160, 1976. Link. http://link.springer.com/article/10.1007%2FBF01931367 The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation. BP2 P. J. Werbos. Applications of advances in nonlinear sensitivity analysis. In R. Drenick, F. Kozin, eds : System Modeling and Optimization: Proc. IFIP, Springer, 1982. PDF http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf . First application of backpropagation BP1 BP1 to NNs concretizing thoughts in his 1974 thesis . BP4 J. Schmidhuber AI Blog blog.html , 2014; updated 2025 . Who invented backpropagation who-invented-backpropagation.html ? See also LinkedIn post 2025 https://www.linkedin.com/feed/update/urn:li:activity:7354090939369283585/ . COCO N.E. Cotter and P.R. Conwell. Fixed-weight networks can learn. International Joint Conference on Neural Networks IJCNN , 1990. DL1 J. Schmidhuber, 2015. Deep learning in neural networks: An overview. Neural Networks, 61, 85-117. More deep-learning-overview.html . Got the first Best Paper Award ever issued by the journal Neural Networks, founded in 1988. DL2 J. Schmidhuber, 2015. Deep Learning http://www.scholarpedia.org/article/Deep Learning . Scholarpedia, 10 11 :32832. DL4 J. Schmidhuber AI Blog blog.html , 2017 . Our impact on the world's most valuable public companies: Apple, Google, Microsoft, Facebook, Amazon impact-on-most-valuable-companies.html ... By 2015-17, neural nets developed in my labs were on over 3 billion devices such as smartphones, and used many billions of times per day, consuming a significant fraction of the world's compute. Examples: greatly improved CTC https://sferics.idsia.ch/pub/juergen/icml2006.pdf -based speech recognition on all Android phones, greatly improved machine translation through Google Translate and Facebook over 4 billion LSTM-based translations per day , Apple's Siri and Quicktype on all iPhones, the answers of Amazon's Alexa, etc. Google's 2019 on-device speech recognition https://arxiv.org/pdf/1811.06621.pdf on the phone, not the server https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html is still based on LSTM rnn.html . DLH J. Schmidhuber AI Blog blog.html , 2022 . Annotated History of Modern AI and Deep Learning deep-learning-history.html . Technical Report IDSIA-22-22, IDSIA, Lugano, Switzerland, 2022. Preprint arXiv:2212.11279 https://arxiv.org/abs/2212.11279 . Tweet of 2022 https://twitter.com/SchmidhuberAI/status/1606333832956973060?cxt=HHwWiMC8gYiH7MosAAAA . DLP J. Schmidhuber. How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. ai-priority-disputes.html Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 14 Dec 2023. Tweet of 2023 https://x.com/SchmidhuberAI/status/1735313711240253567 . FAST C. v.d. Malsburg. Tech Report 81-2, Abteilung f. Neurobiologie, Max-Planck Institut f. Biophysik und Chemie, Goettingen, 1981. First paper on fast weights or dynamic links. FASTa J. A. Feldman. Dynamic connections in neural networks. Biological Cybernetics, 46 1 :27-39, 1982. 2nd paper on fast weights. FASTb G. E. Hinton, D. C. Plaut. Using fast weights to deblur old memories. Proc. 9th annual conference of the Cognitive Science Society pp. 177-186 , 1987. Two types of weights with different learning rates. FWP J. Schmidhuber AI Blog blog.html , 26 March 2021, updated 2023, 2025 . 26 March 1991: Neural nets learn to program neural nets with fast weights—like Transformer variants. 2021: New stuff fast-weight-programmer-1991-transformer.html See tweet of 2022 https://twitter.com/SchmidhuberAI/status/1576966129993797632 . FWP0 J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, 26 March 1991. PDF. FKI-147-91ocr.pdf First paper on neural fast weight programmers fast-weight-programmer-1991-transformer.html that separate storage and control: a slow net learns by gradient descent to compute weight changes of a fast net. The outer product-based version Eq. 5 is now known as the unnormalized linear Transformer 1991-unnormalized-linear-transformer.html or the "Transformer with linearized self-attention." ULTRA ULTRA FWP FWP FWP1 J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4 1 :131-139, 1992. Based on FWP0 . PDF https://sferics.idsia.ch/pub/juergen/fastweights.pdf . HTML. fastweights/ncfastweightsrev.html Pictures German . habilitation/node29.html See tweet of 2022 for 30-year anniversary https://twitter.com/SchmidhuberAI/status/1576966129993797632 . FWP2 J. Schmidhuber. Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam, pages 460-463. Springer, 1993. PDF https://sferics.idsia.ch/pub/juergen/ratio.pdf . A recurrent extension of the unnormalized linear Transformer 1991-unnormalized-linear-transformer.html , ULTRA ULTRA introducing the terminology of learning "internal spotlights of attention." First recurrent NN-based fast weight programmer fast-weight-programmer-1991-transformer.html using outer products to program weight matrix changes. FWP3 I. Schlag, J. Schmidhuber. Gated Fast Weights for On-The-Fly Neural Program Generation. Workshop on Meta-Learning, @N eur IPS 2017, Long Beach, CA, USA. FWP3a I. Schlag, J. Schmidhuber. Learning to Reason with Third Order Tensor Products. Advances in Neural Information Processing Systems N eur IPS , Montreal, 2018. Preprint: arXiv:1811.12143 https://arxiv.org/abs/1811.12143 . PDF http://papers.nips.cc/paper/8203-learning-to-reason-with-third-order-tensor-products.pdf . FWP6 I. Schlag, K. Irie, J. Schmidhuber. Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021. Preprint: arXiv:2102.11174 https://arxiv.org/abs/2102.11174 . FWP7 K. Irie, I. Schlag, R. Csordas, J. Schmidhuber. Going Beyond Linear Transformers with Recurrent Fast Weight Programmers. NeurIPS 2021. Preprint: arXiv:2106.06295 https://arxiv.org/abs/2106.06295 June 2021 . FWP8 K. Irie, F. Faccio, J. Schmidhuber. Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules. NeurIPS 2022. FWP9 K. Irie, J. Schmidhuber. Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules. ICLR 2023. FWP23 J.von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, M. Vladymyrov. Transformers learn in-context by gradient descent. ICML 2023. The core FWP principle of "NNs that learn to program the fast weight changes of other NNs" FWP0 and FWP6 provide an intuitive conception of what's now called "in-context learning." FWP24 A. Behrouz, P. Zhong, V. Mirrokni. Titans: Learning to Memorize at Test Time. Arxiv preprint 2501.00663, 2024. FWP25 J. von Oswald, N. Scherrer, S. Kobayashi, L. Versari, S. Yang, M. Schlegel, K. Maile, Y. Schimpf, O. Sieberling, A. Meulemans, R. A. Saurous, G. Lajoie, C. Frenkel, R. Pascanu, B. Aguera y Arcas, J. Sacramento. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training. Arxiv preprint 2506.05233, 2025. FWP25b Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, C. Guestrin. Learning to Learn at Test Time : RNNs with Expressive Hidden States. ICML 2025. FWP25c K. Irie, S. J. Gershman. Fast weight programming and linear Transformers: from machine learning to neurobiology. Arxiv preprint 2508.08435, 2025. FWPMETA1 J. Schmidhuber. Steps towards self-referential' learning. Technical Report CU-CS-627-92, Dept. of Comp. Sci., University of Colorado at Boulder, November 1992. PDF selfref1992.pdf . FWPMETA2 J. Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam , pages 446-451. Springer, 1993. PDF https://sferics.idsia.ch/pub/juergen/selfref.pdf . FWPMETA3 J. Schmidhuber. An introspective network that can learn to run its own weight change algorithm. https://sferics.idsia.ch/pub/juergen/iee93self.ps.gz In Proc. of the Intl. Conf. on Artificial Neural Networks, Brighton , pages 191-195. IEE, 1993. FWPMETA4 J. Schmidhuber. A neural network that embeds its own meta-levels. In Proc. of the International Conference on Neural Networks '93, San Francisco . IEEE, 1993. FWPMETA5 J. Schmidhuber. Habilitation thesis, TUM, 1993. PDF https://sferics.idsia.ch/pub/juergen/habilitation.pdf . A recurrent neural net with a self-referential, self-reading, self-modifying weight matrix can be found here habilitation/node131.html . FWPMETA6 L. Kirsch and J. Schmidhuber. Meta Learning Backpropagation & Improving It. Metalearning Workshop at NeurIPS, 2020. Preprint arXiv:2012.14905 https://arxiv.org/abs/2012.14905 cs.LG , 2020. FWPMETA8 K. Irie, I. Schlag, R. Csordas, J. Schmidhuber. A Modern Self-Referential Weight Matrix That Learns to Modify Itself. International Conference on Machine Learning ICML , 2022. Preprint: arXiv:2202.05780 https://arxiv.org/abs/2202.05780 . FWPMETA9 L. Kirsch and J. Schmidhuber. Self-Referential Meta Learning. First Conference on Automated Machine Learning Late-Breaking Workshop , 2022. FWPMETA10 K. Irie, R. Csordas, J. Schmidhuber. Metalearning Continual Learning Algorithms. TMLR 2025. GGP F. Faccio, V. Herrmann, A. Ramesh, L. Kirsch, J. Schmidhuber. Goal-Conditioned Generators of Deep Policies. Preprint arXiv/2207.01570 https://arxiv.org/abs/2207.01570 , 4 July 2022 submitted in May 2022 . GOD K. Gödel. Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I. Monatshefte für Mathematik und Physik, 38:173-198, 1931. In the early 1930s, Gödel founded theoretical computer science goedel-1931-founder-theoretical-computer-science-AI.html . He identified fundamental limits of mathematics and theorem proving and computing and Artificial Intelligence. HEB48 J. Konorski 1948 . Conditioned reflexes and neuron organization. Translation from the Polish manuscript under the author's supervision. Cambridge University Press, 1948. Konorski published the so-called "Hebb rule" before Hebb HEB49 HEB49 . HEB49 D. O. Hebb. The Organization of Behavior. Wiley, New York, 1949. Konorski HEB48 HEB48 published the so-called "Hebb rule" before Hebb. HO1 S. Hochreiter, A. S. Younger, P. R. Conwell 2001 . Learning to Learn Using Gradient Descent. ICANN 2001. Lecture Notes in Computer Science, 2130, pp. 87-94. GPT3 T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei. Language Models are Few-Shot Learners 2020 . Preprint arXiv/2005.14165 https://arxiv.org/abs/2005.14165 . KOH72 T. Kohonen, T. Correlation matrix memories. IEEE Transactions on Computers, 21 4 :353-359, 1972. KOS88 B. Kosko. Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 18 1 :49-60, 1988. LT23 K. Irie, R. Csordas, J. Schmidhuber. Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions. EMNLP 2023. LT24 S. Yang, B. Wang, Y. Zhang, Y. Shen, Y. Kim. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS 2024. LT25 S. Yang, J. Kautz, A. Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025. "Mamba2" is the 1991 ULTRA with a scalar time-decay factor on the fast weight matrix. LT25b R. Grazzi, J. Siems, A. Zela, J. K.H. Franke, F. Hutter, M. Pontil. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. ICLR 2025. Shows that the delta-rule extension FWP6 LT23 is more expressive than the quadratic transformer and other naive linear transformers e.g., it can do parity and modular arithmetics . LT25c J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, R. Grazzi. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products ICLR 2025 Workshop FM-Wild. Extending the DeltaNet FWP6 LT23 through additional "micro-steps." LIT74 W. A. Little. The existence of persistent states in the brain. Mathematical biosciences, 19 1-2 :101-120, 1974. LSTM1 S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9 8 :1735-1780, 1997. PDF https://sferics.idsia.ch/pub/juergen/lstm.pdf . More. rnn.html LSTM2 F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12 10 :2451-2471, 2000. PDF https://sferics.idsia.ch/pub/juergen/FgGates-NC.pdf . The "vanilla LSTM architecture" that everybody is using today, e.g., in Google's Tensorflow. LSTM13 F. A. Gers and J. Schmidhuber. LSTM Recurrent Networks Learn Simple Context Free and Context Sensitive Languages. IEEE Transactions on Neural Networks 12 6 :1333-1340, 2001. PDF. https://sferics.idsia.ch/pub/juergen/L-IEEE.pdf META1 J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987. Searchable PDF scan diploma1987ocr.pdf created by OCRmypdf which uses LSTM rnn.html . HTML diploma.html . For example, Genetic Programming geneticprogramming.html GP is applied to itself, to recursively evolve better GP methods through Meta-Evolution. More metalearning.html . META J. Schmidhuber AI Blog blog.html , 2020 . 1/3 century anniversary of first publication on metalearning machines that learn to learn 1987 metalearning.html . For its cover I drew a robot that bootstraps itself. 1992-: gradient descent-based neural metalearning. 1994-: Meta-Reinforcement Learning with self-modifying policies. 1997: Meta-RL plus artificial curiosity and intrinsic motivation. 2002-: asymptotically optimal metalearning for curriculum learning. 2003-: mathematically optimal Gödel Machine goedelmachine.html . 2020: new stuff METARL10 L. Kirsch, S. van Steenkiste, J. Schmidhuber. Improving Generalization in Meta Reinforcement Learning using Neural Objectives. International Conference on Learning Representations, 2020. MIR J. Schmidhuber Oct 2019, updated 2021, 2022, 2025 . Deep Learning: Our Miraculous Year 1990-1991. deep-learning-miraculous-year-1990-1991.html Preprint arXiv:2005.05744 https://arxiv.org/abs/2005.05744 . The Deep Learning Artificial Neural Networks NNs of our team have revolutionised Machine Learning & AI deep-learning-history.html . Many of the basic ideas behind this revolution were published within the 12 months of our "Annus Mirabilis" 1990-1991 at our lab in TU Munich. Back then, few people were interested. But a quarter century later, NNs based on our "Miraculous Year" were on over 3 billion devices, and used many billions of times per day, consuming a significant fraction of the world's compute impact-on-most-valuable-companies.html . In particular, in 1990-91, we laid foundations of Generative AI, publishing principles of 1 Generative Adversarial Networks artificial-curiosity-since-1990.html sec1 for Artificial Curiosity and Creativity artificial-curiosity-since-1990.html now used for deepfakes , 2 Transformers fast-weight-programmer-1991-transformer.html the T in ChatGPT—see the 1991 Unnormalized Linear Transformer 1991-unnormalized-linear-transformer.html , 3 Pre-training very-deep-learning-1991.html for deep NNs see the P in ChatGPT , 4 NN distillation very-deep-learning-1991.html key for DeepSeek https://x.com/SchmidhuberAI/status/1885357355938046382 , and 5 recurrent World Models world-models-planning-curiosity-fki-1990.html for Reinforcement Learning and Planning deep-learning-history.html rl in partially observable environments. The year 1991 also marks the emergence of the defining features of 6 LSTM deep-learning-miraculous-year-1990-1991.html Sec.%204 , the most cited AI paper of the 20th century based on constant error flow through residual NN connections , and 7 ResNet, the most cited AI paper of the 21st century, based on our LSTM-inspired Highway Net highway-networks.html that was 10 times deeper than previous feedforward NNs. MOST J. Schmidhuber AI Blog blog.html , 2021, updated 2025 . The most cited neural networks all build on work done in my labs most-cited-neural-nets.html : 1. Long Short-Term Memory deep-learning-miraculous-year-1990-1991.html Sec.%204 LSTM , the most cited AI of the 20th century. 2. ResNet open-gated Highway Net highway-networks.html , the most cited AI of the 21st century. 3. AlexNet & VGG Net the similar but earlier DanNet DanNet-triggers-deep-CNN-revolution-2011.html of 2011 won 4 image recognition challenges computer-vision-contests-won-by-gpu-cnns.html before them . 4. GAN an instance of Adversarial Artificial Curiosity artificial-curiosity-since-1990.html sec1 of 1990 . 5. Transformer variants—see the 1991 unnormalised linear Transformer 1991-unnormalized-linear-transformer.html ULTRA . Foundations of Generative AI were published in 1991: the principles of GANs deep-learning-history.html gan now used for deepfakes , Transformers fast-weight-programmer-1991-transformer.html the T in ChatGPT , Pre-training very-deep-learning-1991.html for deep NNs the P in ChatGPT , NN distillation deep-learning-miraculous-year-1990-1991.html Sec.%202 , and the famous DeepSeek—see the tweet https://x.com/SchmidhuberAI/status/1885357355938046382 . NOB J. Schmidhuber. A Nobel Prize for Plagiarism. Technical Report IDSIA-24-24. physics-nobel-2024-plagiarism.html Sadly, the Nobel Prize in Physics 2024 for Hopfield & Hinton is a Nobel Prize for plagiarism. They republished methodologies developed in Ukraine and Japan by Ivakhnenko and Amari in the 1960s & 1970s, as well as other techniques, without citing the original papers. Even in later surveys, they didn't credit the original inventors thus turning what may have been unintentional plagiarism into a deliberate form . None of the important algorithms for modern Artificial Intelligence were created by Hopfield & Hinton. See also popular tweet1 https://x.com/SchmidhuberAI/status/1844022724328394780 , tweet2 https://x.com/SchmidhuberAI/status/1865310820856393929 , and LinkedIn post https://lnkd.in/eS92dg86 . PAL80 G. Palm. On associative memory. Biological cybernetics, 36 1 :19-31, 1980. ST61 K. Steinbuch. Die Lernmatrix. Kybernetik, 1 1 :36-45, 1961. ST63 K. Steinbuch, U. A. W. Piske 1963 . Learning matrices and their applications. IEEE Transactions on Electronic Computers, vol. EC-12, no. 6, pp. 846-862, 1963. TR1 A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin 2017 . Attention is all you need. NIPS 2017, pp. 5998-6008. This paper introduced the name "Transformers" for a now widely used NN type. It did not cite the 1991 publication on what's now called unnormalized "linear Transformers" 1991-unnormalized-linear-transformer.html with "linearized self-attention." ULTRA ULTRA Schmidhuber also introduced the now popular attention terminology neural-attention-1990-1993.html in 1993. ATT ATT FWP2 FWP2 R4 R4 See tweet of 2022 for 30-year anniversary https://twitter.com/SchmidhuberAI/status/1576966129993797632 . TR2 J. Devlin, M. W. Chang, K. Lee, K. Toutanova 2018 . Bert: Pre-training of deep bidirectional Transformers for language understanding. Preprint arXiv:1810.04805. TR3 K. Tran, A. Bisazza, C. Monz. The Importance of Being Recurrent for Modeling Hierarchical Structure. EMNLP 2018, p 4731-4736. ArXiv preprint 1803.03585. TR4 M. Hahn. Theoretical Limitations of Self-Attention in Neural Sequence Models. Transactions of the Association for Computational Linguistics, Volume 8, p.156-171, 2020. TR5 A. Katharopoulos, A. Vyas, N. Pappas, F. Fleuret. Transformers are RNNs: Fast autoregressive Transformers with linear attention. In Proc. Int. Conf. on Machine Learning ICML , July 2020. TR5a Z. Shen, M. Zhang, H. Zhao, S. Yi, H. Li. Efficient Attention: Attention with Linear Complexities. WACV 2021. TR6 K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, et al. Rethinking attention with Performers. In Int. Conf. on Learning Representations ICLR , 2021. TR6a H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, L. Kong. Random Feature Attention. ICLR 2021. TR7 S. Bhattamishra, K. Ahuja, N. Goyal. On the Ability and Limitations of Transformers to Recognize Formal Languages. EMNLP 2020. TR8 W. Merrill, A. Sabharwal. The Parallelism Tradeoff: Limitations of Log-Precision Transformers. TACL 2023. TR25 J. Schmidhuber AI Blog blog.html , 2025 . Who Invented Transformer Neural Networks? who-invented-transformer-neural-networks.html Technical Note IDSIA-11-25, Nov 2025. PLAN J. Schmidhuber AI Blog blog.html , 2020 . 30-year anniversary of planning & reinforcement learning with recurrent world models and artificial curiosity 1990 . world-models-planning-curiosity-fki-1990.html This work also introduced high-dimensional reward signals, deterministic policy gradients for RNNs, the GAN principle deep-learning-miraculous-year-1990-1991.html Sec.%205 widely used today . Agents with adaptive recurrent world models even suggest a simple explanation of consciousness & self-awareness. ULTRA References on the 1991 unnormalized linear Transformer 1991-unnormalized-linear-transformer.html ULTRA : original tech report March 1991 FWP0 FWP0 . Journal publication 1992 FWP1 FWP1 . Recurrent ULTRA extension 1993 introducing the terminology of learning "internal spotlights of attention” FWP2 FWP2 . Modern "quadratic" Transformer 2017: "attention is all you need" scaling quadratically in input size TR1 TR1 . 2020 paper TR5 TR5 using the terminology "linear Transformer" for a more efficient Transformer variant that scales linearly , leveraging linearized attention TR5a TR5a . 2021 paper FWP6 FWP6 pointing out that ULTRA dates back to 1991 FWP0 FWP0 when compute was a million times more expensive. Overview of ULTRA and other Fast Weight Programmers 2021 FWP FWP . See the T in ChatGPT See also surveys DLH DLH DLP DLP , 2022 tweet for ULTRA's 30-year anniversary https://twitter.com/SchmidhuberAI/status/1576966129993797632 , and 2024 tweet https://x.com/SchmidhuberAI/status/1864701357107634390 . UN J. Schmidhuber AI Blog blog.html , 2021 . 30-year anniversary. 1991: First very deep learning with unsupervised pre-training. First neural network distillation very-deep-learning-1991.html . Unsupervised hierarchical predictive coding with self-supervised target generation finds compact internal representations of sequential data to facilitate downstream deep learning. The hierarchy can be distilled into a single deep neural network suggesting a simple model of conscious and subconscious information processing . 1993: solving problems of depth 1000. UN0 J. Schmidhuber. Neural sequence chunkers. Technical Report FKI-148-91, Institut für Informatik, Technische Universität München, April 1991. PDF. FKI-148-91ocr.pdf Unsupervised/self-supervised learning and predictive coding is used in a deep hierarchy of recurrent neural networks RNNs to find compact internal representations of long sequences of data, across multiple time scales and levels of abstraction. Each RNN tries to solve the pretext task of predicting its next input, sending only unexpected inputs to the next RNN above. The resulting compressed sequence representations greatly facilitate downstream supervised deep learning such as sequence classification. By 1993, the approach solved problems of depth 1000 UN2 UN2 requiring 1000 subsequent computational stages/layers—the more such stages, the deeper the learning . A variant collapses the hierarchy into a single deep net. It uses a so-called conscious chunker RNN which attends to unexpected events that surprise a lower-level so-called subconscious automatiser RNN. The chunker learns to understand the surprising events by predicting them. The automatiser uses a neural knowledge distillation procedure deep-learning-miraculous-year-1990-1991.html Sec.%202 to compress and absorb the formerly conscious insights and behaviours of the chunker, thus making them subconscious. The systems of 1991 allowed for much deeper learning than previous methods. More. very-deep-learning-1991.html UN1 J. Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4 2 :234-242, 1992. Based on TR FKI-148-91, TUM, 1991. UN0 UN0 PDF https://sferics.idsia.ch/pub/juergen/chunker.pdf . First working Deep Learner based on a deep RNN hierarchy with different self-organising time scales , overcoming the vanishing gradient problem through unsupervised pre-training and predictive coding with self-supervised target generation . Also: compressing or distilling a teacher net the chunker into a student net the automatizer that does not forget its old skills—such approaches are now widely used. See also this tweet https://twitter.com/SchmidhuberAI/status/1608870559609421831 . More. firstdeeplearner.html UN2 J. Schmidhuber. Habilitation thesis, TUM, 1993. PDF https://sferics.idsia.ch/pub/juergen/habilitation.pdf . An ancient experiment on "Very Deep Learning" with credit assignment across 1200 time steps or virtual layers and unsupervised / self-supervised pre-training for a stack of recurrent NN can be found here http://www.idsia.ch/~juergen/habilitation/node114.html depth 1000 . VAN1 S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TUM, 15 June 1991 advisor J. Schmidhuber http://www.idsia.ch/~juergen . PDF. http://www.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf WID Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. InProc. IRE WESCONConvention Record, pages 96-104, Los Angeles, CA, USA, August 1960. .