{"slug": "the-1991-unnormalized-linear-transformer-ultra", "title": "The 1991 Unnormalized Linear Transformer (ULTRA)", "summary": "Jürgen Schmidhuber, a pioneer in artificial intelligence, published the first Transformer variant, the unnormalized linear Transformer (ULTRA), in March 1991, which scales linearly in input size compared to the quadratic scaling of Google's 2017 Transformer. ULTRA introduced self-supervised pre-training and fast weight programming, and its mathematical equivalence to modern linearized self-attention was later recognized. Schmidhuber's 1993 extension discussed 'internal spotlights of attention,' predating the 'attention is all you need' terminology.", "body_md": "The T in ChatGPT[[GPT3]](#GPT3) stands for an artificial neural network (NN) called *Transformer*. In March 1991, when compute was millions of times more expensive than today, even before the [LSTM](deep-learning-history.html#lstm), Schmidhuber published a first Transformer variant, which is now called the *unnormalised linear Transformer*.[[ULTRA]](#ULTRA)[[FWP0]](#FWP0)\nIt had to be more efficient than Google's 2017 *quadratic* Transformer:[[TR1]](#TR1) ULTRA's computational costs scale *linearly* in input size, rather than *quadratically* (in 1991, no journal would have accepted an NN that scales quadratically).\nIn 1991, Schmidhuber also introduced [self-supervised pre-training](very-deep-learning-1991.html) to enable deep learning in sequence-processing NNs (see the P in ChatGPT).[[UN][UN0-2]](#UN)\nHis 1993 recurrent ULTRA extension[[FWP2]](#FWP2) talked about learning *\"internal spotlights of *[attention](neural-attention-1990-1993.html)”—compare the recent attention terminology, e.g., *\"attention is all you need,\"*[[TR1]](#TR1) and tweets of\n[2022](https://twitter.com/SchmidhuberAI/status/1576966129993797632) &\n[2023](https://x.com/schmidhuberai/status/1864701357107634390).\nLike modern quadratic Transformers, the 1991 ULTRA is highly parallelizable.\nIt was a by-product of more general research on\n[NNs that learn to program the fast weights of other NNs](fast-weight-programmer-1991-transformer.html).[[FWP]](#FWP)\nThe 1991 experiments were similar to today's: predict some effect, given a sequence of inputs.[[FWP0]](#FWP0)\n\n[\n](FKI-147-91ocr.pdf)\n\nHow does the unnormalized linear Transformer work?\nThere are two feedforward NNs (FNNs) called the slow net and the fast net.\nThe slow net has a special unit for each fast net\nunit from which at least one fast connection is originating. In 1991, the vector of real-valued activations across these units was called\nFROM (blue in the image); in today's Transformer terminology it's called KEY.\nThe slow net also has a special unit for each fast net unit to which at least one fast connection is leading. In 1991, the vector of activations across these units was called TO (red in the image); today it's called VALUE. At every time step of sequence processing, each fast weight may rapidly change in proportion to the product of the current activations of the corresponding units in KEY and VALUE generated by the slow net. This product is simply *added*\nto the fast weight (which then may be normalized by a squashing function[[FWP0]](#FWP0)). The\nadditive part by itself essentially\novercomes the vanishing gradient problem.[[FWP]](#FWP)\n\nThe current INPUT to which the fast net is applied is called the QUERY.\nEssentially, the QUERY is processed by the *fast weight matrix*, which is\na sum of outer products of previously generated KEYs\nand VALUEs (ignoring normalizations and projections).\nThe KEYs/VALUEs/QUERIES implement READ/WRITE operations on the separate storage represented by the fast network.\nSince all operations of both networks are differentiable, we obtain *end-to-end differentiable*\nactive control of *fast weight changes* through *additive outer products*.[[FWP0-3a]](#FWP0)\nHence the slow net can learn by gradient descent in some given error function to rapidly modify the fast net during sequence processing, by inventing *good* context-dependent KEYs and VALUEs at the right times.\n\nThis is *mathematically equivalent* to what was later called an unnormalised \"linear Transformer\" with \"linearized self-attention.\"[[FWP6]](#FWP6)[[TR5-6a]](#TR5)[[DLH]](#DLH)[[ULTRA]](#ULTRA)\n\nThe \"quadratic\" Transformers of 2017[[TR1-2]](#TR1) are a combination of Schmidhuber's 1991\n*additive outer product fast weight principle*[[FWP0-2]](#FWP0)\nand *softmax*:\n*attention* (QUERY, KEY, VALUE) ~ *softmax* (QUERY KEY) VALUE.\nThe [attention](neural-attention-1990-1993.html) weights in Transformers\ncan be viewed as context-dependent weight vectors or\n*NN-programmed fast weights*.[[FWP]](#FWP)\n\nIn the interest of efficiency,\n*linear* Transformers of 2020-21[[TR5-6]](#TR5)\nabandoned the softmax, essentially resurrecting the original 1991 system,[[ULTRA]](#ULTRA)[[FWP0-1]](#FWP0)[[FWP]](#FWP) whose costs scale linearly in input size, rather than quadratically.[[TR1]](#TR1)\n\nOf course, plain *outer products* in NNs\ngo back at least to Konorski's informal 1948 rule[[HEB48]](#HEB48) (later often called the \"Hebb rule\"[[HEB49]](#HEB49))\nand\nconcrete formal implementations through\nSteinbuch's *Learning Matrix* around 1960.[[ST61-63]](#ST61)[[AMH1-2]](#AMH1)[[KOH72]](#KOH72)[[LIT74]](#LIT74)[[PAL80]](#PAL80)\nSee also Kosko's bidirectional associative memories.[[KOS88]](#KOS88)\nHowever, these authors described *pre-wired* rules to\nassociate *user-given* patterns with each other. Their\nsystems did not *learn by gradient descent to use such rules* for\nassociating *self-invented* KEY/VALUE patterns, like the ULTRAs and other Transformers\nsince 1991.[[ULTRA]](#ULTRA)\n(Neither did early NNs with fast weights by Malsburg (1981) and others.[[FAST]](#FAST)[[FASTa,b]](#FASTa)[[DLP]](#DLP))\n\nThe 1991 ULTRAs are essentially *NN-programming NNs*\nwhose *elementary programming instructions*\nare additive outer product rules. What was the key novelty? Errors are backpropagated *through these differentiable rules* such that ULTRA can learn to minimise its objective function by invoking and using the rules wisely, generating appropriate KEYs/VALUEs at the right times to create useful changes of fast weights.\n\nLater FWPs used more complex elementary programming instructions, e.g., the *delta rule*[[FWP6]](#FWP6) and its extensions.[[LT25]](#LT25)[[LT25c]](#LT25c) This is closely related to\n[metalearning](metalearning.html)[[META1]](#META1)[[META]](#META) with self-referential NNs\nthat can learn to execute and modify their own weight change algorithm.[[FWPMETA1-10]](#FWPMETA1)\nNote that even an NN with *fixed* weights can still learn,[[COCO]](#COCO)[[HO1]](#HO1) and that an NN\ncan learn to implement [backpropagation](who-invented-backpropagation.html),[[FWPMETA6]](#FWPMETA6) then improve backpropagation by backpropagating errors through the differentiable backpropagation algorithm itself.\n\nSchmidhuber offered the 1991 ULTRA[[ULTRA]](#ULTRA)[[FWP0-1]](#FWP0) as an\nalternative to\n[sequence-processing recurrent NNs (RNNs)](deep-learning-miraculous-year-1990-1991.html#Sec.%200),\nthe computationally most powerful NNs of them all.[[UN]](#UN)[[MIR]](#MIR)([Sec. 0](deep-learning-miraculous-year-1990-1991.html#Sec.%200))\nModern Transformers are also viewed as RNN alternatives, despite their limitations.[[TR3-4,7-8]](#TR3)\nThe 1991 experiments were similar to today's: given a sequence of sensory inputs, predict some effect, without using RNNs.[[FWP0]](#FWP0)\n\nRecent work on linear Transformers and similar Fast Weight Programmers\n\nToday, many researchers want to develop faster and better alternatives to *quadratic* Transformers, and as of 2025, there has been lots of recent work on *linear* Transformers and similar [Fast Weight Programmers](fast-weight-programmer-1991-transformer.html).e.g.,[[LT23-25]](#LT23)[[FWP23-25b]](#FWP23) This is also relevant for neurobiology.[[FWP25c]](#FWP25c)\nSee also: [who invented transformer neural networks?](who-invented-transformer-neural-networks.html)[[TR25]](#TR25)\n\nAcknowledgments\n\nThanks to several expert reviewers for useful comments. Since science is about self-correction, let me know under *juergen@idsia.ch* if you can spot any remaining error.\nThe contents of this article may be used for educational and non-commercial purposes, including articles for Wikipedia and similar sites.\nThis work is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).\n\nReferences\n\n[AC]\nJ. Schmidhuber ([AI Blog](blog.html), 2021). [ 3 decades of artificial curiosity & creativity](artificial-curiosity-since-1990.html). *Our artificial scientists not only answer given questions but also invent new questions. They achieve curiosity through: (1990) the principle of generative adversarial networks, (1991) neural nets that maximise learning progress, (1995) neural nets that maximise information gain (optimally since 2011), (1997) adversarial design of surprising computational experiments, (2006) maximizing compression progress like scientists/artists/comedians do, (2011) PowerPlay... Since 2012: applications to real robots.*\n\n[AC90]\nJ. Schmidhuber.\nMaking the world differentiable: On using fully recurrent\nself-supervised neural networks for dynamic reinforcement learning and\nplanning in non-stationary environments.\nTechnical Report FKI-126-90, TUM, Feb 1990, revised Nov 1990.\n[PDF](FKI-126-90ocr.pdf).\n*The first paper on long-term planning with reinforcement learning recurrent neural networks (NNs) (*[more](deep-learning-miraculous-year-1990-1991.html#Sec.%2011)) and on generative adversarial networks\nwhere a generator NN is fighting a predictor NN in a minimax game\n([more](deep-learning-miraculous-year-1990-1991.html#Sec.%205)).\n\n[AMH1]\nS. I. Amari (1972).\nLearning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions, C 21, 1197-1206, 1972.\n[PDF.](amari1972hopfield.pdf)\n*First publication of what was later sometimes called the Hopfield network*[[AMH2]](#AMH2)[[NOB]](#NOB) or Amari-Hopfield Network.\n\n[AMH2]\nJ. J. Hopfield (1982). Neural networks and physical systems with emergent\ncollective computational abilities. Proc. of the National Academy of Sciences,\nvol. 79, pages 2554-2558, 1982.\n*The Hopfield network or Amari-Hopfield Network was published in 1972 by Amari.*[[AMH1]](#AMH1)[[NOB]](#NOB)\n\n[ATT] J. Schmidhuber ([AI Blog](blog.html), 2020, updated 2025). [30-year anniversary of end-to-end differentiable sequential neural attention. Plus goal-conditional reinforcement learning.](neural-attention-1990-1993.html) *Schmidhuber had both hard attention for foveas (1990) and soft attention in form of Transformers with linearized self-attention (1991-93).*[[FWP]](#FWP) Today, both types are very popular.\n\n[ATT0] J. Schmidhuber and R. Huber.\nLearning to generate focus trajectories for attentive vision.\nTechnical Report FKI-128-90, Institut für Informatik, Technische\nUniversität München, 1990.\n[PDF.](FKI-128-90ocr.pdf)\n\n[ATT1] J. Schmidhuber and R. Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(1 & 2):135-141, 1991. Based on TR FKI-128-90, TUM, 1990.\n[PDF](https://sferics.idsia.ch/pub/juergen/attention.pdf).\n[More.](attentive.html)\n\n[ATT2]\nJ. Schmidhuber.\nLearning algorithms for networks with internal and external feedback.\nIn D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton,\neditors, *Proc. of the 1990 Connectionist Models Summer School*, pages\n52-61. San Mateo, CA: Morgan Kaufmann, 1990.\n[PS](hinton.ps). ([PDF](hinton-rev.pdf).)\n\n[BPA]\nH. J. Kelley. Gradient Theory of Optimal Flight Paths. ARS Journal, Vol. 30, No. 10, pp. 947-954, 1960.\n*Precursor of modern *[backpropagation](who-invented-backpropagation.html).[[BP1-4]](#BP1)\n\n[BP1] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 1970.\n*See chapters 6-7 and FORTRAN code on pages 58-60.*\n[PDF](http://www.idsia.ch/~juergen/linnainmaa1970thesis.pdf).\nSee also BIT 16, 146-160, 1976.\n[Link.](http://link.springer.com/article/10.1007%2FBF01931367)\n*The first publication on \"modern\" backpropagation, also known as the reverse mode of automatic differentiation.*\n\n[BP2] P. J. Werbos. Applications of advances in nonlinear sensitivity analysis. In R. Drenick, F. Kozin, (eds): System Modeling and Optimization: Proc. IFIP,\nSpringer, 1982.\n[PDF](http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf).\n*First application of backpropagation*[[BP1]](#BP1) to NNs (concretizing thoughts in his 1974 thesis).\n\n[BP4] J. Schmidhuber ([AI Blog](blog.html), 2014; updated 2025).\n[Who invented backpropagation](who-invented-backpropagation.html)?\nSee also [LinkedIn post (2025)](https://www.linkedin.com/feed/update/urn:li:activity:7354090939369283585/).\n\n[COCO] N.E. Cotter and P.R. Conwell.\nFixed-weight networks can learn.\nInternational Joint Conference on Neural Networks (IJCNN), 1990.\n\n[DL1] J. Schmidhuber, 2015.\nDeep learning in neural networks: An overview. Neural Networks, 61, 85-117.\n[More](deep-learning-overview.html).\n*Got the first Best Paper Award ever issued by the journal Neural Networks, founded in 1988.*\n\n[DL2] J. Schmidhuber, 2015.\n[Deep Learning](http://www.scholarpedia.org/article/Deep_Learning).\nScholarpedia, 10(11):32832.\n\n[DL4] J. Schmidhuber ([AI Blog](blog.html), 2017).\n[Our impact on the world's most valuable public companies: Apple, Google, Microsoft, Facebook, Amazon](impact-on-most-valuable-companies.html)... *By 2015-17, neural nets developed in my labs were on over 3 billion devices such as smartphones, and used many billions of times per day, consuming a significant fraction of the world's compute. Examples: greatly improved (*[CTC](https://sferics.idsia.ch/pub/juergen/icml2006.pdf)-based) speech recognition on all Android phones, greatly improved machine translation through Google Translate and Facebook (over 4 billion LSTM-based translations per day), Apple's Siri and Quicktype on all iPhones, the answers of Amazon's Alexa, etc. Google's 2019\n[on-device speech recognition](https://arxiv.org/pdf/1811.06621.pdf)\n([on the phone, not the server](https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html))\nis still based on\n[LSTM](rnn.html).\n\n[DLH]\nJ. Schmidhuber ([AI Blog](blog.html), 2022).\n[Annotated History of Modern AI and Deep Learning](deep-learning-history.html). Technical Report IDSIA-22-22, IDSIA, Lugano, Switzerland, 2022.\nPreprint [arXiv:2212.11279](https://arxiv.org/abs/2212.11279).\n[Tweet of 2022](https://twitter.com/SchmidhuberAI/status/1606333832956973060?cxt=HHwWiMC8gYiH7MosAAAA).\n\n[DLP]\nJ. Schmidhuber.\n[ How 3 Turing awardees republished key methods and ideas whose creators they failed to credit.](ai-priority-disputes.html) Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 14 Dec 2023.\n[Tweet of 2023](https://x.com/SchmidhuberAI/status/1735313711240253567).\n\n[FAST] C. v.d. Malsburg. Tech Report 81-2, Abteilung f. Neurobiologie,\nMax-Planck Institut f. Biophysik und Chemie, Goettingen, 1981.\n*First paper on fast weights or dynamic links.*\n\n[FASTa]\nJ. A. Feldman. Dynamic connections in neural networks.\nBiological Cybernetics, 46(1):27-39, 1982.\n*2nd paper on fast weights.*\n\n[FASTb]\nG. E. Hinton, D. C. Plaut. Using fast weights to deblur old memories. Proc. 9th annual conference of the Cognitive Science Society (pp. 177-186), 1987.\n*Two types of weights with different learning rates.*\n\n[FWP]\nJ. Schmidhuber ([AI Blog](blog.html), 26 March 2021, updated 2023, 2025).\n[26 March 1991: Neural nets learn to program neural nets with fast weights—like Transformer variants. 2021: New stuff! ](fast-weight-programmer-1991-transformer.html)\nSee [tweet of 2022](https://twitter.com/SchmidhuberAI/status/1576966129993797632).\n\n[FWP0]\nJ. Schmidhuber.\nLearning to control fast-weight memories: An alternative to recurrent nets.\nTechnical Report FKI-147-91, Institut für Informatik, Technische\nUniversität München, 26 March 1991.\n[PDF.](FKI-147-91ocr.pdf)\n*First paper on neural *[fast weight programmers](fast-weight-programmer-1991-transformer.html) that separate storage and control: a slow net learns by gradient descent to compute weight changes of a fast net. The outer product-based version (Eq. 5) is now known as the [unnormalized linear Transformer](1991-unnormalized-linear-transformer.html) or the \"Transformer with linearized self-attention.\"[[ULTRA]](#ULTRA)[[FWP]](#FWP)\n\n[FWP1] J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992. Based on [FWP0].\n[PDF](https://sferics.idsia.ch/pub/juergen/fastweights.pdf).\n[HTML.](fastweights/ncfastweightsrev.html)\n[ Pictures (German).](habilitation/node29.html)\nSee [tweet of 2022 for 30-year anniversary](https://twitter.com/SchmidhuberAI/status/1576966129993797632).\n\n[FWP2] J. Schmidhuber. Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets. In Proceedings of the International Conference on Artificial Neural Networks, Amsterdam, pages 460-463. Springer, 1993.\n[PDF](https://sferics.idsia.ch/pub/juergen/ratio.pdf).\n*A recurrent extension of the *[unnormalized linear Transformer](1991-unnormalized-linear-transformer.html),[[ULTRA]](#ULTRA) introducing the terminology of learning \"internal spotlights of attention.\" First recurrent NN-based [fast weight programmer](fast-weight-programmer-1991-transformer.html) using outer products to program weight matrix changes.\n\n[FWP3] I. Schlag, J. Schmidhuber. Gated Fast Weights for On-The-Fly Neural Program Generation. Workshop on Meta-Learning, @N(eur)IPS 2017, Long Beach, CA, USA.\n\n[FWP3a] I. Schlag, J. Schmidhuber. Learning to Reason with Third Order Tensor Products. Advances in Neural Information Processing Systems (N(eur)IPS), Montreal, 2018.\nPreprint: [arXiv:1811.12143](https://arxiv.org/abs/1811.12143). [PDF](http://papers.nips.cc/paper/8203-learning-to-reason-with-third-order-tensor-products.pdf).\n\n[FWP6] I. Schlag, K. Irie, J. Schmidhuber.\nLinear Transformers Are Secretly Fast Weight Programmers. ICML 2021. Preprint: [arXiv:2102.11174](https://arxiv.org/abs/2102.11174).\n\n[FWP7] K. Irie, I. Schlag, R. Csordas, J. Schmidhuber.\nGoing Beyond Linear Transformers with Recurrent Fast Weight Programmers.\nNeurIPS 2021.\nPreprint: [arXiv:2106.06295](https://arxiv.org/abs/2106.06295) (June 2021).\n\n[FWP8] K. Irie, F. Faccio, J. Schmidhuber.\nNeural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules.\nNeurIPS 2022.\n\n[FWP9] K. Irie, J. Schmidhuber.\nImages as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules.\nICLR 2023.\n\n[FWP23]\nJ.von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, M. Vladymyrov.\nTransformers learn in-context by gradient descent.\nICML 2023. *The core FWP principle of \"NNs that learn to program the fast weight changes of other NNs\" [FWP0] and [FWP6] provide an intuitive conception of what's now called \"in-context learning.\"*\n\n[FWP24]\nA. Behrouz, P. Zhong, V. Mirrokni.\nTitans: Learning to Memorize at Test Time.\nArxiv preprint 2501.00663, 2024.\n\n[FWP25]\nJ. von Oswald, N. Scherrer, S. Kobayashi, L. Versari, S. Yang, M. Schlegel, K. Maile, Y. Schimpf, O. Sieberling, A. Meulemans, R. A. Saurous, G. Lajoie, C. Frenkel, R. Pascanu, B. Aguera y Arcas, J. Sacramento.\nMesaNet: Sequence Modeling by Locally Optimal Test-Time Training.\nArxiv preprint 2506.05233, 2025.\n\n[FWP25b]\nY. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, C. Guestrin.\nLearning to (Learn at Test Time): RNNs with Expressive Hidden States.\nICML 2025.\n\n[FWP25c]\nK. Irie, S. J. Gershman.\nFast weight programming and linear Transformers: from machine learning to neurobiology.\nArxiv preprint 2508.08435, 2025.\n\n[FWPMETA1] J. Schmidhuber. Steps towards `self-referential' learning. Technical Report CU-CS-627-92, Dept. of Comp. Sci., University of Colorado at Boulder, November 1992.\n[PDF](selfref1992.pdf).\n\n[FWPMETA2] J. Schmidhuber. A self-referential weight matrix.\nIn *Proceedings of the International Conference on Artificial\nNeural Networks, Amsterdam*, pages 446-451. Springer, 1993.\n[PDF](https://sferics.idsia.ch/pub/juergen/selfref.pdf).\n\n[FWPMETA3] J. Schmidhuber.\n[An introspective network that can learn to run its own weight change algorithm.](https://sferics.idsia.ch/pub/juergen/iee93self.ps.gz) In *Proc. of the Intl. Conf. on Artificial Neural Networks,\nBrighton*, pages 191-195. IEE, 1993.\n\n[FWPMETA4]\nJ. Schmidhuber.\nA neural network that embeds its own meta-levels.\nIn *Proc. of the International Conference on Neural Networks '93,\nSan Francisco*. IEEE, 1993.\n\n[FWPMETA5]\nJ. Schmidhuber. Habilitation thesis, TUM, 1993. [PDF](https://sferics.idsia.ch/pub/juergen/habilitation.pdf).\n*A recurrent neural net with a self-referential, self-reading, self-modifying weight matrix\n*[can be found here](habilitation/node131.html).\n\n[FWPMETA6]\nL. Kirsch and J. Schmidhuber. Meta Learning Backpropagation & Improving It. Metalearning Workshop at NeurIPS, 2020.\nPreprint [arXiv:2012.14905](https://arxiv.org/abs/2012.14905) [cs.LG], 2020.\n\n[FWPMETA8]\nK. Irie, I. Schlag, R. Csordas, J. Schmidhuber.\nA Modern Self-Referential Weight Matrix That Learns to Modify Itself.\nInternational Conference on Machine Learning (ICML), 2022.\nPreprint: [arXiv:2202.05780](https://arxiv.org/abs/2202.05780).\n\n[FWPMETA9]\nL. Kirsch and J. Schmidhuber.\nSelf-Referential Meta Learning.\nFirst Conference on Automated Machine Learning (Late-Breaking Workshop), 2022.\n\n[FWPMETA10] K. Irie, R. Csordas, J. Schmidhuber.\nMetalearning Continual Learning Algorithms.\nTMLR 2025.\n\n[GGP]\nF. Faccio, V. Herrmann, A. Ramesh, L. Kirsch, J. Schmidhuber.\nGoal-Conditioned Generators of Deep Policies.\nPreprint [arXiv/2207.01570](https://arxiv.org/abs/2207.01570), 4 July 2022 (submitted in May 2022).\n\n[GOD]\nK. Gödel. Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I. Monatshefte für Mathematik und Physik, 38:173-198, 1931.\n*In the early 1930s,\n*[Gödel founded theoretical computer science](goedel-1931-founder-theoretical-computer-science-AI.html). He identified fundamental limits of mathematics and theorem proving and computing and Artificial Intelligence.\n\n[HEB48]\nJ. Konorski (1948). Conditioned reflexes and neuron organization. Translation from the Polish manuscript under the author's supervision. Cambridge University Press, 1948. *Konorski published the so-called \"Hebb rule\" before Hebb *[[HEB49]](#HEB49).\n\n[HEB49]\nD. O. Hebb. The Organization of Behavior. Wiley, New York, 1949.\n*Konorski *[[HEB48]](#HEB48) published the so-called \"Hebb rule\" before Hebb.\n\n[HO1]\nS. Hochreiter, A. S. Younger, P. R. Conwell (2001). Learning to Learn Using Gradient Descent.\nICANN 2001. Lecture Notes in Computer Science, 2130, pp. 87-94.\n\n[GPT3]\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei.\nLanguage Models are Few-Shot Learners (2020).\nPreprint [arXiv/2005.14165](https://arxiv.org/abs/2005.14165).\n\n[KOH72]\nT. Kohonen, T. Correlation matrix memories. IEEE Transactions on Computers, 21(4):353-359, 1972.\n\n[KOS88]\nB. Kosko. Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 18(1):49-60, 1988.\n\n[LT23]\nK. Irie, R. Csordas, J. Schmidhuber.\nPractical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions.\nEMNLP 2023.\n\n[LT24]\nS. Yang, B. Wang, Y. Zhang, Y. Shen, Y. Kim.\nParallelizing Linear Transformers with the Delta Rule over Sequence Length.\nNeurIPS 2024.\n\n[LT25]\nS. Yang, J. Kautz, A. Hatamizadeh.\nGated Delta Networks: Improving Mamba2 with Delta Rule.\nICLR 2025. *\"Mamba2\" is the 1991 ULTRA with a scalar time-decay factor on the fast weight matrix.*\n\n[LT25b]\nR. Grazzi, J. Siems, A. Zela, J. K.H. Franke, F. Hutter, M. Pontil.\nUnlocking State-Tracking in Linear RNNs Through Negative Eigenvalues.\nICLR 2025.\n* Shows that the delta-rule extension [FWP6][LT23] is more expressive than the quadratic transformer and other naive linear transformers (e.g., it can do parity and modular arithmetics).*\n\n[LT25c]\nJ. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, R. Grazzi.\nDeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products\nICLR 2025 Workshop FM-Wild.\n* Extending the DeltaNet [FWP6][LT23] through additional \"micro-steps.\"*\n\n[LIT74]\nW. A. Little. The existence of persistent states in the brain. Mathematical biosciences, 19(1-2):101-120, 1974.\n\n[LSTM1] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. [PDF](https://sferics.idsia.ch/pub/juergen/lstm.pdf).\n[More.](rnn.html)\n\n[LSTM2] F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10):2451-2471, 2000.\n[PDF](https://sferics.idsia.ch/pub/juergen/FgGates-NC.pdf).\n*[The \"vanilla LSTM architecture\" that everybody is using today, e.g., in Google's Tensorflow.]*\n\n[LSTM13]\nF. A. Gers and J. Schmidhuber.\nLSTM Recurrent Networks Learn Simple Context Free and\nContext Sensitive Languages.\nIEEE Transactions on Neural Networks 12(6):1333-1340, 2001.\n[PDF.](https://sferics.idsia.ch/pub/juergen/L-IEEE.pdf)\n\n[META1]\nJ. Schmidhuber.\nEvolutionary principles in self-referential learning, or on learning\nhow to learn: The meta-meta-... hook. Diploma thesis,\nInstitut für Informatik, Technische Universität München, 1987.\n[Searchable PDF scan](diploma1987ocr.pdf) (created by OCRmypdf which uses\n[LSTM](rnn.html)).\n[HTML](diploma.html).\n*For example,\n*[\nGenetic Programming\n](geneticprogramming.html)\n(GP) is applied to itself, to recursively evolve\nbetter GP methods through Meta-Evolution. [More](metalearning.html).\n\n[META]\nJ. Schmidhuber ([AI Blog](blog.html), 2020). [1/3 century anniversary of\nfirst publication on metalearning machines that learn to learn (1987)](metalearning.html).\n*For its cover I drew a robot that bootstraps itself.\n1992-: gradient descent-based neural metalearning. 1994-: Meta-Reinforcement Learning with self-modifying policies. 1997: Meta-RL plus artificial curiosity and intrinsic motivation.\n2002-: asymptotically optimal metalearning for curriculum learning. 2003-: mathematically optimal *[Gödel Machine](goedelmachine.html). 2020: new stuff!\n\n[METARL10]\nL. Kirsch, S. van Steenkiste, J. Schmidhuber. Improving Generalization in Meta Reinforcement Learning using Neural Objectives. International Conference on Learning Representations, 2020.\n\n[MIR] J. Schmidhuber (Oct 2019, updated 2021, 2022, 2025). [Deep Learning: Our Miraculous Year 1990-1991.](deep-learning-miraculous-year-1990-1991.html) Preprint\n[arXiv:2005.05744](https://arxiv.org/abs/2005.05744). *The Deep Learning Artificial Neural Networks (NNs)\nof our team have\nrevolutionised\n*[Machine Learning & AI](deep-learning-history.html).\nMany of the basic ideas behind this revolution were published within the 12 months of our *\"Annus Mirabilis\"* 1990-1991 at our lab in TU Munich.\nBack then, few people were interested. But a quarter century later, NNs based on our *\"Miraculous Year\"*\n[were on over 3 billion devices,\nand used many billions of times per day,\nconsuming a significant fraction of the world's compute](impact-on-most-valuable-companies.html).\nIn particular, in 1990-91, we laid foundations of Generative AI, publishing principles of (1)\n[Generative Adversarial Networks](artificial-curiosity-since-1990.html#sec1) for [Artificial Curiosity and Creativity](artificial-curiosity-since-1990.html) (now used for deepfakes), (2) [Transformers](fast-weight-programmer-1991-transformer.html) (the T in ChatGPT—see the [1991 Unnormalized Linear Transformer](1991-unnormalized-linear-transformer.html)), (3) [Pre-training](very-deep-learning-1991.html) for deep NNs (see the P in ChatGPT), (4) [NN distillation](very-deep-learning-1991.html) (key for [DeepSeek](https://x.com/SchmidhuberAI/status/1885357355938046382)), and (5) recurrent [World Models](world-models-planning-curiosity-fki-1990.html) for\n[Reinforcement Learning and Planning](deep-learning-history.html#rl) in partially observable environments. The year 1991 also marks the emergence of the defining features of (6)\n[LSTM](deep-learning-miraculous-year-1990-1991.html#Sec.%204), the most cited AI paper of the 20th century (based on constant error flow through residual NN connections), and (7) ResNet, the most cited AI paper of the 21st century, based on our LSTM-inspired [Highway Net](highway-networks.html) that was 10 times deeper than previous feedforward NNs.\n\n[MOST]\nJ. Schmidhuber ([AI Blog](blog.html), 2021, updated 2025). [The most cited neural networks all build on work done in my labs](most-cited-neural-nets.html): * 1. *[Long Short-Term Memory](deep-learning-miraculous-year-1990-1991.html#Sec.%204) (LSTM), the most cited AI of the 20th century. 2. ResNet (open-gated [Highway Net](highway-networks.html)), the most cited AI of the 21st century. 3. AlexNet & VGG Net (the similar but earlier [DanNet](DanNet-triggers-deep-CNN-revolution-2011.html) of 2011 [won 4 image recognition challenges](computer-vision-contests-won-by-gpu-cnns.html) before them). 4. GAN (an instance of [Adversarial Artificial Curiosity](artificial-curiosity-since-1990.html#sec1) of 1990). 5. Transformer variants—see the [1991 unnormalised linear Transformer](1991-unnormalized-linear-transformer.html) (ULTRA). Foundations of Generative AI were published in 1991: the principles of [GANs](deep-learning-history.html#gan) (now used for deepfakes), [Transformers](fast-weight-programmer-1991-transformer.html) (the T in ChatGPT), [Pre-training](very-deep-learning-1991.html) for deep NNs (the P in ChatGPT), [NN distillation](deep-learning-miraculous-year-1990-1991.html#Sec.%202), and the famous DeepSeek—see the [tweet](https://x.com/SchmidhuberAI/status/1885357355938046382).\n\n[NOB] J. Schmidhuber.\nA Nobel Prize for Plagiarism.\n[Technical Report IDSIA-24-24.](physics-nobel-2024-plagiarism.html)\n*Sadly, the Nobel Prize in Physics 2024 for Hopfield & Hinton is a Nobel Prize for plagiarism. They republished methodologies developed in Ukraine and Japan by Ivakhnenko and Amari in the 1960s & 1970s, as well as other techniques, without citing the original papers. Even in later surveys, they didn't credit the original inventors (thus turning what may have been unintentional plagiarism into a deliberate form). None of the important algorithms for modern Artificial Intelligence were created by Hopfield & Hinton.\nSee also popular\n*[tweet1](https://x.com/SchmidhuberAI/status/1844022724328394780),\n[tweet2](https://x.com/SchmidhuberAI/status/1865310820856393929), and\n[LinkedIn post](https://lnkd.in/eS92dg86).\n\n[PAL80]\nG. Palm. On associative memory. Biological cybernetics, 36(1):19-31, 1980.\n\n[ST61]\nK. Steinbuch. Die Lernmatrix. Kybernetik, 1(1):36-45, 1961.\n\n[ST63]\nK. Steinbuch, U. A. W. Piske (1963). Learning matrices and their applications. IEEE Transactions on Electronic Computers, vol. EC-12, no. 6, pp. 846-862, 1963.\n\n[TR1]\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin (2017). Attention is all you need. NIPS 2017, pp. 5998-6008.\n* This paper introduced the name \"Transformers\" for a now widely used NN type. It did not cite\nthe 1991 publication on what's now called *[unnormalized \"linear Transformers\"](1991-unnormalized-linear-transformer.html) with \"linearized self-attention.\"[[ULTRA]](#ULTRA)\nSchmidhuber also introduced the now popular\n[attention terminology](neural-attention-1990-1993.html) in 1993.[[ATT]](#ATT)[[FWP2]](#FWP2)[[R4]](#R4)\nSee [tweet of 2022 for 30-year anniversary](https://twitter.com/SchmidhuberAI/status/1576966129993797632).\n\n[TR2]\nJ. Devlin, M. W. Chang, K. Lee, K. Toutanova (2018). Bert: Pre-training of deep bidirectional Transformers for language understanding. Preprint arXiv:1810.04805.\n\n[TR3] K. Tran, A. Bisazza, C. Monz. The Importance of Being Recurrent for Modeling Hierarchical Structure. EMNLP 2018, p 4731-4736. ArXiv preprint 1803.03585.\n\n[TR4]\nM. Hahn. Theoretical Limitations of Self-Attention in Neural Sequence Models. Transactions of the Association for Computational Linguistics, Volume 8, p.156-171, 2020.\n\n[TR5]\nA. Katharopoulos, A. Vyas, N. Pappas, F. Fleuret.\nTransformers are RNNs: Fast autoregressive Transformers\nwith linear attention. In Proc. Int. Conf. on Machine\nLearning (ICML), July 2020.\n\n[TR5a] Z. Shen, M. Zhang, H. Zhao, S. Yi, H. Li.\nEfficient Attention: Attention with Linear Complexities.\nWACV 2021.\n\n[TR6]\nK. Choromanski, V. Likhosherstov, D. Dohan, X. Song,\nA. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin,\nL. Kaiser, et al. Rethinking attention with Performers.\nIn Int. Conf. on Learning Representations (ICLR), 2021.\n\n[TR6a] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, L. Kong.\nRandom Feature Attention.\nICLR 2021.\n\n[TR7]\nS. Bhattamishra, K. Ahuja, N. Goyal.\nOn the Ability and Limitations of Transformers to Recognize Formal Languages.\nEMNLP 2020.\n\n[TR8]\nW. Merrill, A. Sabharwal.\nThe Parallelism Tradeoff: Limitations of Log-Precision Transformers.\nTACL 2023.\n\n[TR25]\nJ. Schmidhuber ([AI Blog](blog.html), 2025). [Who Invented Transformer Neural Networks?](who-invented-transformer-neural-networks.html) Technical Note IDSIA-11-25, Nov 2025.\n\n[PLAN]\nJ. Schmidhuber ([AI Blog](blog.html), 2020). [30-year anniversary of planning & reinforcement learning with recurrent world models and artificial curiosity (1990).](world-models-planning-curiosity-fki-1990.html) *This work also introduced high-dimensional reward signals, deterministic policy gradients for RNNs, *[the GAN principle](deep-learning-miraculous-year-1990-1991.html#Sec.%205) (widely used today). Agents with adaptive recurrent world models even suggest a simple explanation of consciousness & self-awareness.\n\n[ULTRA]\nReferences on the 1991 [unnormalized linear Transformer](1991-unnormalized-linear-transformer.html) (ULTRA): original tech report (March 1991) [[FWP0]](#FWP0). Journal publication (1992) [[FWP1]](#FWP1). Recurrent ULTRA extension (1993) introducing the terminology of learning \"internal spotlights of attention” [[FWP2]](#FWP2). Modern *\"quadratic\"* Transformer (2017: *\"attention is all you need\"*) scaling *quadratically* in input size [[TR1]](#TR1). 2020 paper [[TR5]](#TR5) using the terminology\n*\"linear Transformer\"* for a more efficient Transformer variant that scales *linearly*, leveraging *linearized attention* [[TR5a]](#TR5a).\n2021 paper [[FWP6]](#FWP6) pointing out that ULTRA dates back to 1991 [[FWP0]](#FWP0) when compute was a million times more expensive.\nOverview of ULTRA and other Fast Weight Programmers (2021) [[FWP]](#FWP).\nSee the T in ChatGPT! See also surveys [[DLH]](#DLH)[[DLP]](#DLP), [2022 tweet for ULTRA's 30-year anniversary](https://twitter.com/SchmidhuberAI/status/1576966129993797632), and [2024 tweet](https://x.com/SchmidhuberAI/status/1864701357107634390).\n\n[UN]\nJ. Schmidhuber ([AI Blog](blog.html), 2021). [30-year anniversary. 1991: First very deep learning with unsupervised pre-training. First neural network distillation](very-deep-learning-1991.html). *Unsupervised hierarchical predictive coding (with self-supervised target generation) finds compact internal representations of sequential data to facilitate downstream deep learning. The hierarchy can be distilled into a single deep neural network (suggesting a simple model of conscious and subconscious information processing). 1993: solving problems of depth >1000.*\n\n[UN0]\nJ. Schmidhuber.\nNeural sequence chunkers.\nTechnical Report FKI-148-91, Institut für Informatik, Technische\nUniversität München, April 1991.\n[PDF.](FKI-148-91ocr.pdf)\n*Unsupervised/self-supervised learning and predictive coding is used\nin a deep hierarchy of recurrent neural networks (RNNs)\nto find compact internal\nrepresentations of long sequences of data,\nacross multiple time scales and levels of abstraction.\nEach RNN tries to solve the **pretext task* of predicting its next input, sending only unexpected inputs to the next RNN above.\nThe resulting compressed sequence representations\ngreatly facilitate downstream supervised deep learning such as sequence classification.\nBy 1993, the approach solved problems of depth 1000 [[UN2]](#UN2)\n(requiring 1000 subsequent computational stages/layers—the more such stages, the deeper the learning).\nA variant collapses the hierarchy into a single deep net.\nIt uses a so-called *conscious chunker RNN*\nwhich attends to unexpected events that surprise\na lower-level so-called *subconscious automatiser RNN.*\nThe chunker learns to *understand* the surprising events by predicting them.\nThe automatiser uses a\n[neural knowledge distillation procedure](deep-learning-miraculous-year-1990-1991.html#Sec.%202)\nto compress and absorb the formerly *conscious* insights and\nbehaviours of the chunker, thus making them *subconscious.*\nThe systems of 1991 allowed for much deeper learning than previous methods. [More.](very-deep-learning-1991.html)\n\n[UN1] J. Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992. Based on TR FKI-148-91, TUM, 1991.[[UN0]](#UN0) [PDF](https://sferics.idsia.ch/pub/juergen/chunker.pdf).\n*First working Deep Learner based on a deep RNN hierarchy (with different self-organising time scales),\novercoming the vanishing gradient problem through unsupervised pre-training and predictive coding (with self-supervised target generation).\nAlso: compressing or distilling a teacher net (the chunker) into a student net (the automatizer) that does not forget its old skills—such approaches are now widely used. See also this *[tweet](https://twitter.com/SchmidhuberAI/status/1608870559609421831). [More.](firstdeeplearner.html)\n\n[UN2] J. Schmidhuber. Habilitation thesis, TUM, 1993. [PDF](https://sferics.idsia.ch/pub/juergen/habilitation.pdf).\n*An ancient experiment on \"Very Deep Learning\" with credit assignment across 1200 time steps or virtual layers and unsupervised / self-supervised pre-training for a stack of recurrent NN\n*[can be found here](http://www.idsia.ch/~juergen/habilitation/node114.html) (depth > 1000).\n\n[VAN1] S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TUM, 15 June 1991 (advisor [J. Schmidhuber](http://www.idsia.ch/~juergen)). [PDF.](http://www.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf)\n\n[WID]\nBernard Widrow and Marcian E Hoff. Adaptive switching circuits. InProc. IRE WESCONConvention Record, pages 96-104, Los Angeles, CA, USA, August 1960.\n\n.", "url": "https://wpnews.pro/news/the-1991-unnormalized-linear-transformer-ultra", "canonical_source": "https://people.idsia.ch/~juergen/1991-unnormalized-linear-transformer.html", "published_at": "2026-08-22 13:00:52+00:00", "updated_at": "2026-08-22 13:14:30.497958+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Jürgen Schmidhuber", "Google", "ChatGPT", "Transformer", "LSTM", "ULTRA"], "alternates": {"html": "https://wpnews.pro/news/the-1991-unnormalized-linear-transformer-ultra", "markdown": "https://wpnews.pro/news/the-1991-unnormalized-linear-transformer-ultra.md", "text": "https://wpnews.pro/news/the-1991-unnormalized-linear-transformer-ultra.txt", "jsonld": "https://wpnews.pro/news/the-1991-unnormalized-linear-transformer-ultra.jsonld"}}