{"slug": "learning-without-backpopagation", "title": "Learning Without Backpopagation", "summary": "A new article explores genetic algorithms as a gradient-free alternative to backpropagation for training models on open-ended, multi-stage tasks where no single correct answer or error signal exists. The piece demonstrates the approach with a simulated \"primordial soup\" in which creatures evolve food-seeking behavior, ranking each creature by average food eaten across 4 worlds and adding movement-cost and base-metabolism penalties (0.0016 and 0.002 per step) to counter unconstrained speed. The author notes the movement penalty initially makes models lethargic because food rewards are small and unknown, requiring a balance between the two terms.", "body_md": "Out of the primordial soup\n\nBackpropagation is so ubiquitous that it's easy to forget it's not the only way to optimise a model. Or that it is not even always the best way. The problem with gradient descent and backpropagation, is that we need to have an error that we can take our gradients from to update the weights of the model. Usually this is straightforward in tasks like classification or regression where there is an obvious single prediction that resulted in a single error. When that's no longer the case, as when we have a model completing an open-ended task that requires many stages, there is no one \"right\" answer, nor one single way of getting there. Reinforcement learning offers one solution, but has a large number of complexities in part because of trying to link the outcome of the task with the gradients to update the model.\n\nThis is not the only way. Another method from the pre-deep learning era has recently started getting more attention, as people start wanting to explore more gradient free training methods. In this case, drawing on Darwinian evolution, we can evolve our model using a Genetic Algorithm. Here rather than associating a gradient with every parameter, we rely on the random variation in a population to result in models with different abilities at a particular task.\n\nTheir performance needs to be ranked, because each update step only takes forward the strongest models into the next iteration. The strong models are then replicated with small mutations to their weights, we'll get into how to do this  later, and the new population based on the previous strongest performance is let loose on the task again.\nIntuitively this feels like a natural (*haha...*) approach. And to demonstrate just how natural, we are going to look at our own primordial soup, and watch the evolution take place.\n\nSo far so trivial, the creatures drift around looking for food, and as we evolve the model over many steps we can see how behaviour starts to emerge. Through the rest of the piece we'll work through both how these emergent behaviours arise, and where this gradient free approach helps or hinders our model.\n\nIn order to rank the creatures we need a fitness function. This is how to keep score, the equivalent of the loss function in backpropagation, and its construction requires a similar consideration to the loss-shaping in RL. In the previous example this was simply just the amount of food each creature managed to eat.\n\nThis simplest version of the fitness function only considers the amount of food eaten, without any penalties for movement or other costs. In order to rank the creatures, we take the average score for each creature across multiple worlds (4 in this case) as it means we are less prone to selection purely based on random chance in any one world.\n\nHowever that's not an especially sophisticated approach, it encourages the creatures to move to the nearest food as fast as possible until they all reach the allowed limit. With no negative selective pressure to counter this the system evolves unconstrained, but if we introduce a penalty to moving at a certain speed then the model must balance dashing around for food with the cost of moving at speed, especially without a guaranteed reward. Sounds simple. Unfortunately this makes the model very lethargic, the reward from the food is both small and unknown, only by random movements will the model come across any and it takes time to learn that it must seek out the food. The movement penalty by contrast immediately disincentivises exactly the actions that would find the reward. So we need to balance it, here with a base metabolism term, if the model does nothing it will still accrue a large penalty.\n\nExtending the first equation we now have terms for the cost of movement, and the base metabolism required to stay alive.\n\nWe start here with at 0.0016 and at 0.002 per step. Click a creature to read its current metabolism and other properties in the side bar. As creatures grow their metabolism increases, reflecting the higher energy cost of maintaining a larger body.\n\nWe can already see here that the reward shaping problem is strong with genetic algorithms where even trying to model a very simple set of phenomena results in a careful tuning balance. We'll return to this fitness function through the piece as we add more complex behaviour.\n\nAs we allow the simulation to proceed, something interesting has started to happen, the creatures are still moving about pretty happily, but are much less frenetic. They will pause if there is no food nearby, and then dash towards it as it appears, and where they are moving around more generally they do so at a much more moderate pace.\n\nNow we'll turn our attention to what is actually guiding these creatures. Each has a \"brain\", which is a small neural network with two hidden layers and a non-linearity. We allow each creature to take as inputs what it can sense of the nearest food and the nearest other creature within a visibility radius. As outputs the creature gets two controls, a turn and a speed. We can see if we monitor our test creature here that there is not much to read off at the start, the weights are random and the creature wanders, and what the inspector gives us is its score, its speed, how far it can see and what is sitting inside that radius right now.\n\nThere are nine inputs in total, the first seven coming from the nearest food and nearest creature, and the last two from the creature itself. The creature can see the bearing and distance of the nearest food and the nearest other creature within its visibility radius, and we model the bearing as a sine and cosine measured from the creature's own heading rather than as an absolute angle as this is more generalisable. The nearest creature also carries a seventh input, its mass relative to the creature's own.\n\nThe network is then two hidden layers of sixteen units each, with a non-linearity on every unit, resulting in 466 parameters in total. When predation is turned off, the creature-sensing inputs are held at nothing in range the whole time for simplicity.\n\nThe creatures are clearly able to discover how to respond to the stimuli in the environment, and have a system to control their actions. So now we need to think about the mechanism that trains them without using gradients. The inspiration for this kind of genetic algorithm comes from Darwinian evolution, and the language becomes that of mutations and cloning. (tenuous but it works.) We start by selecting the top-K creatures which are going to seed the next generation based on their fitness function. Then there are three core methods to produce our new creatures.\n\nThat random walk has a direction, even though nothing in the algorithm goes looking for one. If we blur the fitness with the mutation noise and call the result , then the log-derivative trick gives us\n\nRanking a generation of the mutants gets us a noisy sample of this expectation, so selection is estimating a gradient without ever computing it, gradient free is more about how we got here rather than what is being computed.\n\nThe lab below does not pit those strategies against each other, that would need a separate run for each and we have not done it here. What it does instead is add the other half of being alive, growth and a mass that costs something to carry, and turn the saved brains loose under the new rules.\n\nNow let's add another aspect to the simulation. So far the pressures have been positive to seek out food, and an internal negative pressure to control the speed the creature is moving around at. Let's add one more aspect, danger. In the next simulation creatures will be able to eat each other if they carry more mass. And are rewarded by gaining the mass of that creature. There are a few aspects here that subtly change the dynamics, eaten creatures are not considered for the next generation (cruel perhaps, but creates interesting interactions) and consuming food becomes not just a fitness property but also a defence mechanism. This encourages creatures to become riskier, the greatest rewards are eating a creature just weaker than themselves, but opens them up to the risk that any food / prey that other creature eats might make them the aggressor instead. Let's look at how this plays out in the simulation.\n\nWe can look at this for a specific case, seeing the chasing and evading behaviour that is learnt as a result of the evolutionary pressures.\n\nUp to now the sight radius and the number of things a creature keeps track of have been evolving along with the weights, mutated like any other gene, and we have not really looked at what that buys them. What we get when we look is more food and shorter lives. Creatures that evolve their senses freely take 10.21 food on worlds they have never seen, against 8.04 for creatures held to a single food and a single rival, and they survive 43.4% of runs against 51.8%. Capping how far they can see changes neither number, so what matters is how many things a creature is tracking rather than how far off it can notice them.\n\nThe obvious worry is that all those extra inputs make the search harder, and in a sense they do, but not creature by creature. Every genome in a run is the same length, 1,474 weights and 3 sense genes here, because the brain is sized by the cap on the slots rather than by what any individual evolved, so a creature using one slot still carries and mutates the weights for all ten. The dimensional cost is paid once, when we choose the cap, and the cost of actually using the senses turns up in the survival numbers instead.\n\nSelection only ever sees the worlds the creatures actually ran in, and this run saw two of them, so it is fair to ask whether twenty-four generations of ranking has produced a brain that forages or a brain that has learned where the food is in those two worlds. Freezing the weights and moving the food answers that, and the answer is that there is no gap at all. The frozen brains take 10.19 food on the two worlds they trained in and 10.21 on twelve they have never seen, surviving 43.2% against 43.4%, which is marginally better on the worlds they have never been in.\n\nThat is less surprising once you look at what the brain is actually given, which is a bearing and a distance to the nearest food and the nearest rival, and never a position in the world. A new layout is just a new draw from the same distribution as far as the creature is concerned, so there is nothing layout-specific for it to memorise. What does break them is a change in the conditions rather than the arrangement, and halving the food takes intake from 10.21 down to 4.34 while survival only slips to 39.0%.\n\nWe can see an interesting effect through adding newborn creatures to the population after already allowing it to run for a while. A newborn is expected to generally see other creatures at around the same stage of life, so placing it among mature creatures who pose more of a threat can affect its survival and intake.\n\nThis is obviously a silly example for some fun, but the trend back towards gradient free training is starting to become more important as the application of models in complex environments with long running states increases. While RL is still by far the dominant training paradigm in this area, we can start to see how gradient free methods might have a place at the table. In the next part we'll look at a familiar problem and see how we can move past toy simulations and into real applications.", "url": "https://wpnews.pro/news/learning-without-backpopagation", "canonical_source": "https://idlemachines.co.uk/essays/learning-without-backpropagation", "published_at": "2026-10-20 23:20:43+00:00", "updated_at": "2026-10-20 23:48:33.144363+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research", "neural-networks"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/learning-without-backpopagation", "markdown": "https://wpnews.pro/news/learning-without-backpopagation.md", "text": "https://wpnews.pro/news/learning-without-backpopagation.txt", "jsonld": "https://wpnews.pro/news/learning-without-backpopagation.jsonld"}}