Neural networks are classical stateful dynamical systems — I arrived at this experimentally

A researcher's causal experiments on nanoGPT, ResNet-18, and a diffusion U-Net indicate that neural networks behave as classical stateful dynamical systems, where the same update has different effects depending on the receiving state and learning leaves persistent functional reorganization. The findings, detailed in arXiv paper 2608.20965 and accompanying GitHub code, challenge the view of networks as mere parameterized input-output functions, suggesting a unified dynamics of training, learning, and inference.

I’ve been running a bunch of causal experiments on nanoGPT trying to understand what training, learning and inference are actually doing inside the model. I wasn’t thinking about dynamical systems when I started. I was asking much more basic questions. If I take the exact same realized update and apply it to different receiving states, does it have the same effect? What actually persists after learning? When the model is frozen during inference, what is it using from what was formed during training? And what happens once the consequences of inference start feeding back into later training? At some point the experiments started lining up into the same picture: state → realized update → nonlinear response → persistent functional reorganization → inference → feedback → new state The same update does different things in different receiving states. The response is nonlinear at finite amplitudes. Learning leaves persistent changes in distributed functional support. Frozen inference recruits that support, and feedback changes the state that later learning starts from. I then tried the same core tests on ResNet-18 with SGD momentum and a diffusion U-Net with AdamW. Different architecture, task, modality, optimizer. The same basic relation structure was still there. The funny part is that I only noticed afterward that I had basically experimentally arrived at a classical stateful dynamical system. I hadn’t started from that picture at all. It also seems like a pretty unusual kind of system-level dynamical organization for a computer system. So I’m curious what people here think the right object is. Is a neural network mainly a parameterized input-output function, or is that actually the wrong level of description once training, learning, inference and feedback are all part of one continuous state evolution? Paper: 2608.20965 Training, learning and inference: unified dynamics of neural systems https://arxiv.org/abs/2608.20965 Experiments/code: GitHub - wind342/gfg-training-learning-inference-experiments: Frozen executable experiments and generation-fact evidence for training, learning and inference · GitHub https://github.com/wind342/gfg-training-learning-inference-experiments