Neural Networks

The softmax function, essential to modern AI systems like GPT, originated from Ludwig Boltzmann's 19th-century work on thermodynamics, where he derived the exponential form to maximize entropy in gas molecules. The same …

A machine learning researcher pitted 21 algorithms against each other in a regression task using a 512x512 image as the target function, but artificially inflated the input space to 500, 1000, and 2000 features, with onl…

Researchers propose an empirical test for error correction in large language models, finding that residual-stream activations are robust to small perturbations and that feature-specific directions are privileged over gen…

Researchers introduced the Swarm-Inspired Emergent Synchronizer (SIES), a graph-dynamical framework that learns local interaction rules for collective behaviors across graphs, dynamics, and tasks. SIES achieved synchroni…

Researchers propose DCQ-GNN, a spectral graph neural network using convex-concave quadratic filters to improve spectral selectivity without high-order polynomials. The model achieves top average rank on heterophilic grap…

Researchers propose SEMIR, a framework that replaces the pixel lattice with a parameterized graph minor to preserve thin-structure connectivity in segmentation tasks. The method achieves full-resolution inference without…

Researchers propose G-SPIN, a graph-based framework that corrects phonetic errors in ASR outputs by combining a graph neural network for candidate generation with a masked language model and LLM for contextual re-ranking…

A new study on holographic memory for zero-shot compositional reasoning in knowledge graphs found that while Holographic Reduced Representations (HRR) and Fourier HRR (FHRR) perform competitively on single-hop queries, t…

Researchers propose a novel neural network quantization method that learns quantization-aware linear paths in weight space to find low-loss subspaces, enabling direct quantization of the midpoint without performance degr…

Researchers introduced block-sparse featurizers (BSFs) to capture visual concept manifolds in neural network representations, showing that concepts are low-dimensional geometric structures rather than isolated directions…

Researchers introduced Pre-Warm, a zero-training-cost method for data-conditioned initialization of the first convolutional layer in CNNs. The technique clusters patches from a single training batch to initialize half of…

Researchers argue that neural network learning opacity stems from dynamical complexity, identifying sensitivity to weight initialization, feedback in gradient optimization, and sensitivity to training data as irreducible…

Researchers propose a new approach to near-sensor computing that performs lightweight Neural Architecture Search (NAS) directly on deployment devices to adapt tiny neural networks to real-time sensor data. Validated on t…

A developer completed building an LSTM cell using PyTorch and Lightning AI, implementing the forward pass and optimizer. The LSTM unit processes stock price inputs over four days, updating long-term and short-term memory…

Shrijith Venkatramana, building git-lrc, explains how scaling laws discovered by OpenAI, Google, DeepMind, and Anthropic made large language models like ChatGPT, Claude, and Gemini possible. The 2020 OpenAI paper 'Scalin…

PyTorch, JAX, and compiler-level tools like Enzyme use three different representations for automatic differentiation—runtime tape, functional trace-and-transform, and source (IR) transformation—each with distinct trade-o…

Researchers developed low-power analogue neural networks with trainable nonlinear connections, inspired by Kolmogorov-Arnold networks, achieving efficient continuous control tasks like robotic kinematics and photovoltaic…

Researchers reconstructed GRACE-like terrestrial water storage anomalies back to 1940 using a spatio-temporal graph neural network (MTGNN) applied to South America. The model achieved high correlations and low bias, outp…

Researchers developed an automated pipeline to explore heterogeneous 4-expert mixture-of-experts architectures, generating 4,463 candidate models over 28 days. The search was biased toward the AirNet family due to an enu…

Researchers propose a frequency-guided feature representation learner for small object detection that shifts from spatial to spectral processing. The method, instantiated via lightweight plug-and-play modules, achieves c…