Machine learning news — deep learning, reinforcement learning, neural architecture search, diffusion models, and new ML frameworks and libraries.
A new AI architecture called Dendritron Transformer, designed to never forget, is explained in a video. The system aims to address catastrophic forgetting in neural networks by incorporating dendritic computation. This a…
WordPiece, Unigram, and SentencePiece are three subword tokenization algorithms that differ in how they split text, with WordPiece using a likelihood-maximizing merge criterion and a ## continuation marker, while Sentenc…
Byte-Pair Encoding (BPE), originally a data compression algorithm published by Philip Gage in 1994, was adapted in 2016 by Sennrich, Haddow, and Birch for subword tokenization in neural machine translation, solving the v…
The encoder-decoder (seq2seq) architecture, proposed by Cho et al. and Sutskever et al. in 2014, compresses the entire source sentence into a single fixed-length vector, causing performance to degrade sharply on longer s…
Recurrent neural networks (RNNs) process sequences by maintaining a hidden state that compresses the entire history into a fixed-size vector, as formalized in Elman's 1990 paper "Finding Structure in Time." The hidden st…
GloVe, FastText, and word2vec are all word embedding methods that produce dense vector representations of words, but they differ in their underlying approaches: GloVe uses a weighted least-squares regression on global co…
Google researchers led by Tomas Mikolov published word2vec in 2013, a method that made high-quality dense word embeddings cheap to train at billion-token scale on a single machine, transforming them from a research curio…
A technical blog post explains that AdamW with warmup and cosine decay is the optimizer almost everyone uses today to train large language models, tracing the evolution from vanilla SGD through momentum and Adam. The pos…
Gradient descent is the core algorithm behind deep learning, enabling neural networks with millions of parameters to minimize loss by iteratively adjusting weights in the direction opposite to the gradient. The gradient,…
Cross-entropy loss is the standard loss function for training language models like GPT, LLaMA, and BERT because it is differentiable and captures probabilistic confidence, unlike accuracy which is a step function and ign…
Language models generate text by constructing a categorical probability distribution over their vocabulary at each decoding step, converting raw neural network outputs called logits into valid probabilities using the sof…
A blog post explains that the L2 norm measures a vector's magnitude as its Euclidean distance from the origin, while the dot product quantifies similarity between vectors and is the core operation behind attention mechan…
The first post in the 'Holding the LLM Stack in Your Head' series rebuilds intuition for linear algebra, explaining that vectors and matrices are the foundation of neural network operations. The author defines a 'space' …
The AIMO Interpretability Challenge, organized by the AI Mathematical Olympiad (AIMO) and the Fields Model Initiative, aims to uncover the reasoning processes behind AI models' decision-making by testing them on olympiad…
Researchers have introduced four new metrics—activation rate, feature coverage, syntactic distance, and semantic distance—for Explainable Reinforcement Learning (XRL) to provide objective, standardized evaluations of AI …
A developer created a beginner project for schema validation using GitHub's new Issue Fields feature, which became generally available on July 2, 2026. The project demonstrates how to validate structured issue metadata b…
Researchers have introduced SteinGate, a boundary-aware distributional safety certificate for safe reinforcement learning that uses Kernelized Stein Discrepancy to detect tail-event cost violations. The method dynamicall…
A new arXiv paper (2607.13042v1) traces how PyTorch's automatic differentiation engine computes gradients for Physics-Informed Neural Network training, using a 1-3-3-1 multilayer perceptron and the initial value problem …
Researchers introduced DROPJ, a human-centred method for safely training and deploying agent policies in safety-critical environments with unknown dynamics and no reward function. The method learns a world model from pri…
Researchers propose finetuning diffusion generative models with Fréchet Distance loss (FD-loss) to improve synthetic medical image generation for heterogeneous tumors. FD-loss aligns feature statistics of real and genera…