The Continuum Memory System introduces a novel architecture that replaces the Transformer's two-rate memory with a spectrum of blocks, each updating on its own clock, potentially improving efficiency …
A developer explains that self-attention is a matrix operation, not just a mechanism for looking at important words. The post breaks down the QKV projection, scaling, softmax, and weighted sum that en…
Noam Shazeer, co-author of the Transformer paper and former Character.AI CEO, is leaving Google's Gemini team to join OpenAI in an architecture research role, shifting a key model builder from the lab…
A developer introduced DRM Language Emitter, an experimental language model that replaces the Transformer's attention mechanism with controlled latent motion through a learned relational manifold. The…
The Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need', revolutionized AI by replacing sequential RNNs with a parallelizable attention mechanism. This mechanism allows …
Researchers introduced Ghost Attractor Networks, a dynamical decoder that uses basin-structured latent representations for closed-loop sequential generation. The 2.3-million-parameter Ghost matched th…
Researchers introduced Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces pairwise query-key comparisons with routing through K learned Gaussian mixture components, ac…
Noam Shazeer, co-inventor of the transformer architecture and co-lead of Google's Gemini program, is leaving Google to join OpenAI, despite Google paying approximately $2.7 billion to retain him less …
A developer implemented a full Transformer with KV cache on an FPGA, achieving over 56,000 tokens per second at only 80 MHz, without using a GPU or CPU. The design was created gate by gate as a custom…
Researchers introduced a hierarchical model for ball action anticipation in football broadcast video, achieving 17.91% mAP on the SoccerNet benchmark. The system uses a GRU and Transformer decoder wit…
Researchers achieved 98.91% accuracy in multimodal emotion recognition from physiological signals by combining LSTM, TCN, and Transformer models with late-fusion ensemble on the WESAD dataset. Transfo…
A developer argues that balanced ternary (-1, 0, +1) could replace binary for AI hardware, citing 20× model compression, 3× inference speedup, and 8× power reduction. Microsoft's BitNet b1.58 demonstr…
Recursive's automated AI research system achieved state-of-the-art results on three benchmarks: fixed-budget language model training, small-model training speed, and GPU kernel optimization. The syste…
A developer explains how the Transformer architecture works, from self-attention to modern LLMs. The key innovation is that Transformers compare tokens directly via attention rather than processing se…
Researchers developed a Transformer-based scheduling policy for the open shop scheduling problem (OSSP) using deep reinforcement learning. The model, trained on small Taillard benchmark instances, gen…
A developer argues that the current AI revolution is fundamentally different from past waves of enthusiasm, citing the convergence of large-scale labeled data, GPU computing, and deep network architec…
Jürgen Schmidhuber, a pioneer in artificial intelligence whose lab developed foundational ideas like LSTM, world models, and artificial curiosity decades before they became mainstream, argued in a new…
NVIDIA released Nemotron 3 Ultra, a 550-billion-parameter open Mixture-of-Experts model designed for long-running AI agents that plan, use tools, and reason across many turns. The model uses a hybrid …
A developer has demonstrated that the 1982 Hopfield associative memory update rule and the 2017 Transformer scaled dot-product attention mechanism are mathematically identical operations, with one equ…
Michigan Senate candidate Mallory McMorrow unveiled a detailed artificial intelligence policy agenda last week, proposing an AI Workforce Reinvestment Fund and a "token tax" on companies using AI to f…