How can an autonomous and adaptive artificial reasoning agent learn human-like strategies for searching, planning, and reasoning?
This was a question I raised in a recent article on Meta-cognition for Reasoning AI:
Meta-cognition for Reasoning AI In this series of articles, I review a selection of promising ideas from the areas of Theoretical Computer Science, Reinforcement Learning, Neuroscience, Behavioral Psychology, and Behavioral Development, and I offer my own take on how we might combine those ideas together.
This is the first part of four:
This article looks at what we know about algorithmic learning. I cover the importance of inductive bias, work on Algorithmic Neural Reasoning, Graph Neural Networks, memory, and Deep Thinking neural networks.
Contents
· [Series Introduction](#4728)
· [Part Introduction](#63b4)
· [No Free Lunch and Inductive Bias](#c6ef)
· [Neural Algorithmic Reasoning](#2c44)
· [Graph Neural Networks](#4ad8)
· [Graph Neural Algorithmic Reasoning](#436a)
· [Graph Structures](#72cc)
· [GNNs, CNNs, and Transformers](#3c52)
· [Memory Augmented Neural Networks](#e108)
· [Kinds of Memory](#512b)
· [Deep Thinking Models](#8514)
· [Deep Equilibrium Models](#ac0d)
· [Not Covered](#dd1d)
· [Part Summary](#edcc)
· [Citing](#d680)
· [References](#024f)
I am interested in what it would take to build autonomous agents that learn and adapt to their environment in a way that is a closer approximation to humans than our current AI solutions. I believe a larger focus on autonomous embodied reasoning agents is equally relevant to chat-bot and “agentic” reasoning AIs as it is to full physical robots.
My particular focus is on the meta-management concerns and how an agent can not just learn to plan under a particular architected strategy, but can learn the planning algorithms themselves from its own experience and from teacher instruction.
While the questions of meta-management and learning to plan are superficially independent, I believe they are intimately linked. Intuitively, meta-management may have no cause for evolution if it were not for the need to learn to plan, and learning to plan may be nearly impossible without meta-management. To that end, my goal for this series is to bring together several lines of lesser-known research and ideas that have the potential for significant relevance to those research questions. The first three parts are primarily a review of those existing ideas, while the last focuses more on my own ideas and how all the earlier discussed ideas can be brought together. Think of this as a broad but shallow literature review of the brilliant ideas that don’t make the news.
I’m going to cover a wide range of different ideas, but I also don’t want to stray too far from the norms for a blog article. So, rather than filling this out with an exhaustive survey of existing literature or providing detailed explanations of theories, I’ll be referring to a smaller set of papers (mostly surveys themselves) that provide the additional background for anyone wanting to learn more.
The human ability to learn multiple planning strategies is in stark contrast to most RL approaches today that strictly follow one planning strategy and simply learn to use that same strategy in different contexts. Furthermore, while various aspects of the planning algorithm have been explored for learning, the algorithm itself is always hard-coded. For example, some use Monto Carlo Tree Search (MCTS) over forward models, others sample randomly selected trajectories and pick the best single action. Many so-called “learning to plan” papers simply learn a differentiable map or model of value and perform gradient descent on trajectories over that model (e.g., the Value Iteration Network, Tamar et al., 2016, and many of its descendants).
To keep the terminology simple, I use the word “planning”, but I use it in a broad sense. For example, there is an element of planning in the act of searching when you make a (conscious or unconscious) decision about what search strategy to employ. Planning is rarely a one-off exercise, but instead requires constant assessment of the continuous stream of outcomes. So the continuous monitoring and assessment of the search progress and (conscious or unconscious) decisions to either continue or give up are also attributable to the broad sense of planning. Essentially, I use planning to refer to any explicit consideration of possible actions beyond a mere application of a habituated response. And those actions may be physical or cognitive.
More generally, planning and search strategies can be identified as algorithms. Indeed, while planning is an appropriate term when thinking about navigational problems, we want the reasoning AI solution to learn its own algorithms in the more general sense.
In the last two decades, there has been growing research into connectionist models of “algorithmic learning”, focusing primarily on Graph Neural Networks (GNN), for reasons that will become clear shortly. However, there has been a thorough theoretical investigation of algorithms more broadly since the days of Turing. Beyond GNNs, other identified components include memory and a deep understanding of convergence in iterative computations. This article discusses some of those topics.
Before discussing what can be done, it’s useful to first understand what we cannot realistically hope to achieve.
At the extreme, we may wish to develop an architecture and a learning algorithm that enable the system to learn any behavioral algorithm. Here I use “behavioral algorithm” to refer to the goal of learning the algorithm — so it’s the learned algorithm underlying complex behavior. This is in contrast to the learning algorithm: the static RL algorithm that researchers would use to govern the learning.
We’d also like a single system to learn many of these unrestrained behavioral algorithms, rather than just one. Unfortunately, the No Free Lunch Theorem puts a limit on what’s possible.
The No Free Lunch Theorem considers the effectiveness of different learning algorithms across different problem domains. It says that unless a chosen algorithm incorporates a priori knowledge of the problem domain, we have no guarantees that the algorithm will be more effective than picking solutions at random (Wolpert & Macready, 1997).
Thus, for our system to be better than random, we must incorporate a priori knowledge. More practically, our architecture and learning algorithm must incorporate inductive biases toward certain structures and certain behavioral algorithms. This limits the behavioral algorithms that can be learned, with the benefit that it is at least possible to learn some behavioral algorithms. The stronger the inductive bias, the more restrictive the set of possible behavioral algorithms, but also the easier it is for the system to learn them.
Some work has focused specifically on the needs and benefits of inductive bias in artificial Neural Network (NN) design (Hamrick et al., 2018; Xu et al., 2020). Often these inductive biases are implicit within the architecture of the solution. For example, a Convolutional Neural Network (CNN) for vision implements the inductive bias that incoming sensory information is arranged on a 2D grid with Euclidean geometry and that pairwise interaction strength is proportional to the nearness of the points. Xu et al. took this further and showed that neural networks can only efficiently converge in domains where their architecture has algorithmic alignment.
They examined four classes of NN architecture and identified the kinds of problems they can efficiently learn solutions to:
Up until recently, most NN architectures employed either feed-forward or recursive MLP architectures. For the question of learning to plan, the lesson from the above is that while a strict GNN architecture will not suffice in the long term, it is good starting point for improving the reasoning abilities of AI solutions. In fact, the success of Transformers can be related to GNNs — more on that later.
Neural Algorithmic Reasoning (NAR) is a field of research that seeks to design NN architectures that efficiently and reliably learn algorithmic computation. Most work has focused on re-implementing common combinatorial optimization algorithms (Cappart et al., 2023), particularly those studied in the context of DP.
This focus partly stems from the observation that neural networks are a good solution for bridging the domain of the noisy and partially hidden real world with the clean and constrained abstract inputs that common algorithms typically require. Another reason is that there is a long history of investigation into the nature of these algorithms, with many systematic theories providing guidance on computability, efficiency, representational capacity, and more. This provides the strong mathematical foundation needed for theories of algorithmic learning.
The first significant conclusion from such research is that many real-world structures and many well-known algorithms reflect a graph structure. Across many examined problem sets within that broad class of algorithm, neural solutions based on GNNs achieve better performance and better generalization, with less training data, than equivalent solutions using simple MLP-style deep networks, Convolutional Neural Networks (CNN), or Recurrent Neural Networks (RNN) (Xu et al., 2020; Cappart et al., 2023; Veličković , 2023a).
Part of that success comes from the fact that GNNs can be seen as a generalization of deep MLP and CNN networks, and of many other architectures in use today (Bronstein et al., 2021). Another advantage comes from their natural characteristic of permutation invariance (i.e. invariance to the specific order of edges and neighbors) and equivariance (retaining relative structure under transformation) (Bronstein et al., 2021; Veličković, 2023a).
GNN NARs have successfully been applied to a growing number of algorithm domains, including Integer Linear Programs, Mixed-Integer Programs, Constrained Problems, and Satisfiable Problems. Some concrete examples include breadth-first and depth-first search, Monte-Carlo Tree Search, bipartite matching, mazes, max-flow/min-cut, model-based planning, network configuration protocols, the traveling salesman problem, and protein folding (Cappart et al., 2023; Veličković, 2023a; Qharabagh et al., 2026). Transformers are themselves an example of GNNs (discussed in a later section), and yet some further advances have been achieved by combining GNN NARs with Transformers, in the TransNAR framework (Bounsi et al., 2024), on the basis that the respective algorithm and language capabilities of NAR and Transformer are mutually beneficial.
GNN NARs show excellent scale-generalization, with one example showing success on brain blood-vessel graphs that are 180,000x larger than the synthetic graphs used during training (Veličković, 2023a). Many highly publicized recent achievements in AI have been achieved with GNNs. For example the AlphaFold system for 3D protein folding prediction (Jumper et al., 2021), and the AI-based discovery of 381,000 new stable materials (Merchant & Cubuk, 2023).
If you aren’t familiar with Graph neural networks (GNN), then allow me to briefly describe how they work, before returning to GNN NARs. GNNs are trained artificial NNs that operate over a graph structure of nodes and edges. Unlike Knowledge Graphs, GNNs typically associate all state to the nodes, with edges being simple links without associated metadata.
Nodes are associated with multi-dimensional feature vectors x₁*,…,x*ₙ for n nodes, enabling the GNN to learn latent embedding representations of objects of interest. Edges are typically represented as an n ⨉ n adjacency matrix A. In sparsely-connected graphs, the matrix has aᵢⱼ = 1 if nodes i and j are immediate neighbors, and 0 everywhere else. GNNs learn to compute new vector representations of nodes through iterative aggregation of features from neighboring nodes.
Application of a GNN over a graph computes new node embedding vectors h₁*,…,h*ₙ, with each hᵤ calculated according to one of the following formulas over its neighboring nodes Nᵤ (Bronstein et al., 2021; Veličković, 2023a):
Here, 𝛷 and 𝜓 are NNs. cᵥᵤ is a normalization factor, typically dependent only on the degree of the nodes (numbers of edges). a(xᵤ, xᵥ) is typically an additional NN that dynamically computes relative attention over neighboring nodes, based on pairwise embeddings. ⊕ is a permutation-invariant aggregator, such as sum, mean, or max, which operates only over the set of nodes within the immediate neighborhood of xᵤ. The neighborhood Nᵤ may optionally include xᵤ to enable self-attention and recurrence.
The three variants have progressively larger expressive power but also carry increasing complexity and instability during learning:
GNNs may be applied in a number of different computational paradigms. They may be stacked as layers, with the output embeddings of each layer feeding inputs to the subsequent layer, and each layer learning separate transformations. Alternatively, a single layer of GNN NNs (𝛷, 𝜓, etc.) may be used to iteratively revise node embeddings.
A whole graph can be collapsed via aggregation to a single vector or scalar, for example for the purpose of a regression or classification task. Alternatively, coarsely selected overlapping or non-overlapping subsets of a graph may be similarly collapsed in order to compute a new coarse-grained graph from a fine-grained graph. This can be seen as a form of clustering or dimensionality reduction.
GNN outputs are most commonly computed over all nodes in parallel via standard matrix operations. Node vector values are stacked into a matrix and the NNs applied against that matrix, with the adjacency matrix acting as a mask over the results. Alternatively, GNNs can be sequentially executed against individual nodes, effectively traversing the graph as it computes (Veličković et al., 2020a; Xhonneux et al., 2021). One rationale for why that could be useful is that many known algorithms operate in a sequential manner, and it has been shown that Message-passing GNNs can be Turing complete (Veličković, 2023a). Furthermore, many such algorithms have good convergence guarantees, even when computed in a decentralized fashion. For example, the Bellman backup operator in RL learning, the Bellman-Ford and Dijkstra algorithms for network routing, belief propagation networks, and coordinate descent.
The sequential or traversal approach has the advantage of performing sparse computations (computing only a small subset of data at a time), and enables significant scalability over dataset size. However, it can also be significantly less stable to train compared to parallel computation; a point that I’ll discuss further in the section on Neural Turing Machines below.
As GNNs are more general than CNNs etc., the No Free Lunch Theorem instructs us that we can expect to encounter greater problems with training convergence and stability in GNN NARs.
NAR research has yielded a number of recommendations to improve the likelihood of stable training and convergence (Veličković et al., 2020a; Cappart et al., 2023; Veličković 2023b):
GNNs operate over graph structures, and graph structures can be used to represent different kinds of things. For the purpose of the present discussion it is useful to distinguish between two broad classes of graph representation: state representations and state-space representations.
These two representations have different characteristic tendencies, and each line of research tends to focus on its preferred class.
In both cases one needs to know the graph structure before they can apply a GNN to perform operations. If the graph structure is not known a priori then it must be identified from the data through a process known as latent graph inference.
For simple cases like the objects scene above, simple machine vision techniques such as segmentation and attention have been used to identify the objects, and their relative closeness has been used as a proxy for connectedness (Burgess et al., 2019; Gref et al., 2019; Locatello, 2020). Many real world contexts are much harder for a number of reasons. The real world is only partially observable — we only observe a small part of it at any given moment, and some aspects of the underlying generative process of the world are never observed. The real world is observed through continuous senses— requiring some form of clustering in order to identify discrete nodes in the graph. The real would is dynamic — the structure of the graph continuously changes. Latent graph inference in real-world contexts is still considered extremely difficult and is an active area of research.
Ideally, we desire a sparse adjacency matrix — one with only the minimally needed edges between nodes. However, optimization of sparse adjacency matrices is difficult. For example, edge selection is a non-differentiable operation. In one approach, rather than a sparse adjacency matrix of a 0s and 1s, the graph is treated as fully-connected and adjacency values are relaxed to be anywhere within the range 0.0 to 1.0. These values represent the degree of certainty about the node relationships and are iteratively optimized. In the case of Attentional and Message-passing GNNs, the adjacency matrix may be further dropped entirely as its calculations can be rolled into those of attention or messages.
Recent work on latent graph inference has looked at nonparametric (Wang et al., 2019; Deac et al., 2022), supervised (Veličković et al., 2020b), variational (Kipf et al., 2018), reinforcement (Kazi et al., 2022) and self-supervised learning (Fatemi et al., 2021) methods.
GNNs can be seen as generalizations of many familiar network architectures.
Convolutional Neural Networks (CNN) approximate a special case of a Message-passing GNN over a regular grid, with the exception that GNNs consider node neighbors as unordered sets whereas kernels in CNNs learn orientation-dependent weights. Note that this comparison is in contrast to what may implied by the names. Convolutional GNNs do not reflect a graph-generalization of CNNs, as the node-to-node computations in Convolutional GNNs are significantly more constrained and linear.
Transformers compute query-key cross-attention between all pairs of input sequence positions (e.g., tokens), and output attention-weighted sums of values over those positions (Vaswani et al., 2017). This is an example of an Attentional GNN over a fully-connected graph (Joshi, 2020). The adjacency matrix is rolled into the calculation of attention, and thus each Transformer layer can be seen as performing latent graph inference.
Importantly, latent graph inference here is dynamically and independently calculated for each Transformer layer, and for each execution of the overall solution. The independent recalculation leads to tremendous flexibility and enables emergent behaviors such as few-shot and zero-shot learning. However, because past observations are discarded, it means that Transformers in their classic form are unable to iteratively learn and revise latent graph representations of the world.
Some of the earliest work on NN-based algorithm implementations was that of the Neural Turing Machine (Graves et al., 2014), which attempted to emulate Turing machines via neural networks.
The work of Turing and others established a formal theory of computation in which a single universal machine can simulate any computation performed by any particular machine, given an appropriate representation of that machine and its input (Turing, 1936). Together with the equivalence subsequently established between Turing machines and other formalisms of effective computation, this led to the Church–Turing thesis that any effectively computable algorithm can, in principle, be represented and executed as a Turing-machine computation (Church, 1936; Kleene, 1952; Copeland, 2026).
The Neural Turing Machine (NTM) attempted to take the idea of a Turing Machine and replace control over its memory read/write with a differentiable neural network (Graves, Wayne & Danihelka, 2014). The expectation is that such an architecture can learn any arbitrary algorithm. The earliest variants of NTM showed success in learning common low-level computer science algorithms such as repeating sequences, sorting, and similarity-based search, as well as simple relational language reasoning tasks. They used a mixture of key-value content-based addressing and location-based addressing, and included the ability to write to any attended location (Graves et al., 2016; Khosla, Zhu & He, 2023).
The key-value attention mechanism inspired the later development of query-key-value attention in the Transformer architecture, which now forms the basis of many modern AI successes. The original focus of Transformers was on a much narrower task of sequence-to-sequence transformation, which yields a number of advantages. One notable feature is they use a much simpler write strategy: always append outputs to the end of the sequence, and auto-regressively attend to the whole sequence, rather than learning control strategies for writing to arbitrary locations.
This can be understand in relation to the No Free Lunch Theorem. The freedom to learn to write to any location trades off against inductive biases that are more aligned to the specific problems that we are interested in. By restricting how writes are done, Transformers have less freedom in their optimization search and converge faster. The loss of freedom in writing location is balanced by auto-regressive attention, which must be optimized regardless.
Another perspective considers the numbers of steps needed to complete an algorithm. The original NTMs were targeted at low-level algorithms comparable to common computer algorithms like array copying, sorting, and searching. These are typically either short tasks or have a small number of unique steps that can be consistently repeated an arbitrary number of times. In contrast, the modern use of reasoning via LLMs tends to focus on high-level concepts and supports use of high-level heuristics.
This potentially enables complex problems to be translated to short horizons — a relatively small number of reasoning steps that operate over high-level representations. In contrast, attempting to solve high-level problems via low-level algorithms results in very long horizon reasoning, with associated stability issues. Unfortunately, because each LLM step only produces a single token, they are locked to a fine-grained time-step and suffer the curse of low-level long-horizons.
NTMs and Transformers are now considered part of a much larger field of study into *Memory-Augmented Neural Networks *(MANN). This focuses on how the capabilities of NNs can be extended through the addition of memory mechanisms. Memory writing has continued to prefer simple non-learned strategies, but now also includes surprisal based methods (Omidi et al., 2025).
Human memory systems have been studied for a long time, and yet exactly how the human memory systems are structured is still hotly debated. Most discussions use common terms in ways that differ, depending primarily on the kinds of research questions being asked by individual researchers (Atkinson & Shiffrin, 1968; Cowan, 2008). Still, some general themes have arisen that can be productively related to techniques in AI, illustrating both where current AI techniques can be useful, and where gaps remain (Khosla, Zhu & He, 2023; Omidi et al., 2025).
The following is a brief summary distilled primarily from the reviews by Khosla, Zhu & He (2023) and Omidi et al. (2025).
Sensory Memory:
Working Memory / Short Term:
Long Term Memory:
Long Term Memory - Episodic: Long Term Memory — Semantic Memory:
Long Term Memory — Procedural:
Long Term Memory — Emotional conditioning:
The above analysis suggests that advanced agents capable of operating in a complex dynamic world need multiple memory systems, operating over multiple granularities and timescales, with different consolidation, recall, attentional, and decay characteristics and mechanisms.
The traditional view of deep networks is that subsequent layers compute increasingly high-level abstractions over features. This came from work on deep convolutional networks, which themselves were inspired by biology. In the mammalian visual cortex, the earliest layers encode low-level features like edges and later layers appear to encode high-level features like object category. Artificial convolutional networks were found to mimic this behavior (Kriegeskorte, 2015).
A more recent view, known as Deep Thinking, has come from work on deep sequence models (e.g., LLMs and reasoning models) and suggests that successive hidden layers can be viewed as iterations of reasoning. Some work has shown that by dynamically varying the depth of hidden layers in LLMs, a smaller model can achieve similar performance as a larger model with fixed depth. This is achieved by executing repeated iterations of the same hidden layer or group of hidden layers. It is sometimes referred to as as weight-tying, because, conceptually, multiple iterations of one layer unrolls to a single iteration of multiple layers that share the same weights.
With that strategy, a 3.5B parameter model with dynamic depth was shown to achieve similar performance as a 50B parameter model with fixed depth (Geiping et al., 2025). This idea of Adaptive Computation Time (ACT) (Graves, 2016) has also been applied with significant success in various image tasks (Anil et al., 2022).
An implicit assumption behind adaptive depth is that each iteration of hidden layer encodes their activations within the same latent space, so that subsequent decoding is possible regardless of computational depth. This is particularly plausible in the context of multi-language LLMs where the pre-encoding of tokens embeds those tokens into a language-agnostic semantic latent space.
That notion is cashed-in within the recent Chain of Continuous Thought (Coconut) architecture (Shen et al., 2024). At test-time, the model is first auto-regressed in a “latent mode” where activations from the last hidden layer are added to the input sequence, instead of adding decoded token outputs. The subsequent “language mode” performs standard auto-regressive token generation, but it attends to a sequence that mixes input token embeddings, output token embeddings, and latent state embeddings. In order that the transformer can uniformly attend across both kinds of data, the hidden layers must produce activations in a latent space that is at least compatible with semantic token embeddings.
Deep Equilibrium Models (DEQ) take the notion of iterative reasoning and compatible latent spaces one step further (Bai et al., 2019). DEQ is motivated by the observation that the hidden state of many well-behaved iterative processing networks (i.e. deep thinking networks) converge to a fixed point. Mathematically, this means that a recursive function of the form of equation (1) converges towards z = z*, satisfying the fixed-point equation (2):
Here, the different zᵢ are the output activations from each layer or iteration. For an ideal function, the fixed-point z* is an attractor state that the function converges towards regardless of the initial state of z₀ (the input).
Conveniently, this optimization can be applied to any network where the main block is recursively iterated (otherwise referred to as weight-tied layers) and where its hidden state converges to a fixed point. The latter requirement can be achieved through appropriate regularization (Anil et al., 2022).
Work on Deep Thinking and DEQ have yielded two inductive biases that consistently show significant benefit: *Recall *and Path Independence.
Recall:
Path Independence:
Taken together, these ideas have enabled significant generalization in reasoning horizon — models trained on short horizon problems can be applied against significantly larger horizon problems. For example, a model trained to solve small 9x9 mazes, requiring only about 30 iterations of execution, can stably solve an unseen much larger 801x801 maze, requiring 10,000 iterations (Bansal, et al., 2022).
For me, the effectiveness of DEQ raises a deeper question about the way to conceptualize reasoning trajectories in general. Are they smooth and rapidly convergent towards an attractor state? Or are they more like the complex and meandering paths that we must take when navigating a city full of one-way streets? In the attractor case, a misstep requires no extra effort to correct. In the meandering case, a misstep may require a backtrack. My personal experience is that thought can sometimes be rapid — like falling towards the outcome. But it can also sometimes be extremely complex, involving many abrupt changes and many false turns and restarts. There are two additional features that I suspect must be added to the above discussion on DEQ in order that they are suitable for use in large real-world contexts:
In order to reproduce human-like adaptability of intelligence we need to develop fundamental theories of cognition. By viewing cognition as an algorithm, we see its relationship to theoretical work on algorithms and gain insight from that work. By viewing cognition as a system that learns the algorithm, we also gain insight from theoretical work on algorithm learning.
That’s not to say that we get a complete picture from an algorithmic perspective, but even as an impoverished perspective it is still highlights significant areas for improvement in current reasoning AI.
In this article I have discussed the limits of adaptability and the importance of inductive bias. We need more constraints, more inductive biases, less flexibility, in order for our AIs to learn faster from less data. However, we also need more flexibility and to remove constraints in order to improve their adaptability. The resolution lies not so much in striking a balance, but in the more difficult path of continual research and discovery. To find new architectures that open up doors to more advanced reasoning. To find new inductive biases that keep those more advanced architectures stable and convergent.
GNNs and graph representations of states and state-spaces are likely key components of those new architectures. New forms of memory are another set of key components. Path independence, input re-injection, and fixed point attraction are key inductive biases that will be needed in conjunction with those architectures.
The next part will review the latest achievements in RL, seeking to understand unifying principles.
Unless stated otherwise, all diagrams and images are the work of the author, with AI help for the images.
For attribution, please cite this work as: Lett, M. (2026, Aug). Can an AI Learn the Planning Algorithm? Part 1: Foundations of Algorithmic Learning. AI Advances. https://medium.com/@malcolmlett/can-an-ai-learn-the-planning-algorithm-part1-ed5b653e6b0c
Can an AI Learn the Planning Algorithm? Part 1 was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.