{"slug": "i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its", "title": "I Trained an AI Agent to Play a Game for 76 Hours (And It Learned From Its Mistakes)", "summary": "A developer trained a Double DQN agent from scratch in NumPy on a side-scrolling runner game for 76 hours and 226 million environment steps, publishing the code and raw metrics in the endless-learner GitHub repository. The agent beats a hand-written player at harder tiers, but removing either of two mechanisms built to make it learn from failure improved its performance, and the network slowly lost its ability to learn over training, matching recent research predictions. The author reports three fixes that mattered more than the failure-learning mechanisms: adding a terrain scan so the agent could see gap edges (all four runs then passed tier 1 within 230k to 340k steps, versus none in four 1.2M-step runs before), counting only greedy \"exam\" episodes for curriculum promotion, and addressing policy drift.", "body_md": "I wanted an agent that learns from its own failures and never stops training. The setup was small on purpose: a side-scrolling runner game, a laptop with no GPU, and everything written from scratch on NumPy. That meant the game, the network, backpropagation, Adam and the reinforcement learning, with no PyTorch, no Gym and no pretrained weights. Then I let it run for days and measured everything.\n\nThe agent got good. At the harder tiers it beats a hand-written player that reads the level directly. But the most interesting results were the ones that went the wrong way. I built two mechanisms specifically to make it learn from failure, and taking either one away made it better. The mechanism that mattered most was the plainest one in the system. And over 226 million steps the network slowly lost its ability to learn, exactly as recent research predicts. Fixing that kept the network healthy, but only sometimes made the agent play better.\n\nThe code, the raw metrics and every number below are in [endless-learner](https://github.com/abdulm-ai/endless-learner).\n\n**The game.** A runner with low and tall blocks, gaps, overhangs to duck under, and blocks that slide towards you. From tier 5 on, these come in tight combinations. The agent has four actions: nothing, short jump, long jump, duck. A level is a pure function of a tier and a seed. Tiers have no top: speed, density and the obstacle mix keep rising until they hit limits set by the physics. Before any level is handed out, a depth-first search over jump timings checks that it can be finished. The search runs on the same physics code as the game, so no death is ever the level’s fault.\n\n**The learner.** Double DQN (van Hasselt et al., 2016) with 3-step returns and a target network, on a 155–128–128–4 MLP. The forward pass, backward pass and Adam are written out by hand and checked against central differences, with a max relative error around 1e-8. It runs at a median of 870 to 1,180 environment steps per second on one core of an i7–1265U. That’s slow next to a GPU, but every gradient is visible.\n\n**Learning from failure.** Two mechanisms on top of plain replay:\n\n**Never stopping.** A curriculum promotes the agent when it passes 80% of fresh levels at its tier, and levels with high TD error are replayed more (prioritised level replay, Jiang et al., 2021). To avoid forgetting, 15% of episodes replay a tier it has already passed, and a reservoir sample of the whole history makes up a quarter of every training batch.\n\n**Measurement.** Every 200k steps, the agent plays 20 held-out levels per tier with no exploration. Those seeds are never trained on. One eval round is noisy, so every number here is an average over a window of rounds.\n\nBefore any of the interesting results, three bugs in my thinking had to go. Each one cost me a day, and I think each one is a general lesson.\n\n**The agent couldn’t see gap edges.** My first observation was nine rays fanning out from the agent. A fixed fan touches the ground at only four distances (0.8, 1.6, 3.1 and 8.2 units ahead), so the agent knew a gap existed but not where its edge was. In four runs of 1.2M steps, none got past tier 1, the first tier with gaps, and gaps killed the agent in 26 to 44% of its exams. Adding a terrain scan, the course sampled every half unit for 12 units ahead, fixed it: all four runs passed tier 1 within 230k to 340k steps. What the agent can observe sets the ceiling, and no amount of learning gets around it.\n\n**Exploration hid how good the policy was.** With 3% random actions per decision, a timing-critical game kills you a lot. Exploring episodes passed about 40% of tier-0 levels while the same network, played greedily, passed 100% of the held-out ones. The curriculum was promoting on the wrong signal. Now 30% of fresh levels are played as greedy “exams”, and only exams count for promotion.\n\n**The policy changed under its own feet.** This one surprised me most. With learning frozen, a snapshot of the network passed 100% of 100 random tier-0 levels. The same network playing while it trained passed about 55%. Each update flips the greedy action in some states, a well-known effect called policy churn. An episode lasts about a hundred updates, and here a jump one tick late is a death. The fix was to act with an exponential moving average of the weights (Polyak averaging, 0.5% per update) instead of the online weights. Over two seeds, that version reached frontier tier 4 on both, against tiers 2 and 4 for the online weights. Two seeds is a direction, not a measurement.\n\nWith those fixed, the main run climbed fast: tiers 1 to 7 at 0.09M, 0.25M, 0.42M, 0.60M, 0.75M, 3.3M and 6.3M steps. That’s 2.4 active hours for all seven promotions. Then it spent the remaining 220M steps, 74 more hours, between tiers 5 and 7, and never held tier 8. One other run was promoted to tier 8 at 19.8M steps and demoted again at 34.2M.\n\nFor scale, I wrote a rule-based player that reads the level directly and times each jump to centre its arc over the next obstacle. It’s perfect up to tier 4, then passes 60% of tier-5 levels and 30% at tiers 6 and 7. The learned agent gets 52 to 65% at tier 6 and 49 to 63% at tier 7, depending on the run. At tiers 6 and 7 it beats the player with perfect information, which I didn’t expect.\n\nTo find out what each piece was worth, I removed one at a time: five variants, three seeds each, 5M steps per run. The score is held-out success on tiers 0 to 5, averaged over the eval rounds from 4M to 5M steps.\n\n```\nvariant                            mean   range over 3 seedsfull config                        0.76   0.74 to 0.79no prioritised replay              0.88   0.86 to 0.92no death-bank restarts             0.88   0.84 to 0.90no curriculum (always tier 6)      0.89   0.83 to 0.91no old-tier replay                 0.21   0.14 to 0.29\n```\n\nTwo findings stand out.\n\n**Replaying old tiers is what holds everything together.** Without it, every seed reached tier 7, at 1.9M to 3.3M steps, sooner on average than any other variant. Meanwhile it forgot the easy tiers almost completely: averaged over the seeds, it passed only 5 to 23% of tier-0 to tier-3 levels, where the full config passes 61 to 96%. That is catastrophic forgetting in its plainest form (Kirkpatrick et al., 2017). I think a curriculum makes this worse by design, because it is a schedule for leaving old data behind.\n\n**The two failure mechanisms made the agent worse.** So did the curriculum. Removing any one of the three beat the full config on every seed, and the seed ranges don’t overlap.\n\nWhat makes this interesting to me is that both failure mechanisms did their local jobs well. In the main run the death bank restarted 323,444 episodes and fixed 23,117 failures, in a median of 2 attempts, without giving up on a single one. Transitions just before a death carried 5.5 times the TD error of routine ones (0.56 against 0.10), so prioritised replay was focusing exactly where it was meant to. Every local metric said “learning from failure, working”. The global metric said the agent was better off without it.\n\nMy guess is that each mechanism narrows what the agent trains on, and that narrowing costs more than the focus gains. Prioritised replay oversamples surprising transitions. The death bank spends a quarter of all episodes on fragments just before a death. The curriculum keeps the agent on one tier at a time. Each skew is small, but they stack. I haven’t tested that explanation directly, and these results are for a 5M-step budget only: with much longer training, the balance could change.\n\nThe general lesson I’d take is about metrics, not about replay. A mechanism can have excellent numbers of its own, like failures fixed or TD error concentrated where it should be, and still not move the number you care about. You only find out by taking it away.\n\nThe second question was what happens to a network that never stops training. Dohare et al. (Nature, 2024) showed that networks trained on a changing task gradually lose the ability to learn. Units die, the representation collapses, and new things get learned worse than a fresh network would. A curriculum without an end is exactly that kind of setting.\n\nIt showed up clearly. I tracked three things:\n\nAt 226M steps, 21% of the plain network’s units were dead, srank was down from about 117 to 58, and the weight norm had grown from 24 to 194.\n\nThe proposed fix is continual backprop: keep replacing a trickle of the least useful hidden units with fresh ones. A unit’s usefulness is a running average of its activation times its outgoing weights. New units start with zero outgoing weights, so a reset changes nothing until the unit has learned something. I ran it on the same seed and config. It held dead units at 0.4% and srank at 116 for the full 226M steps, and a second seed showed the same. As a fix for the network’s health, it worked completely.\n\nWhether that made the agent play better is less clear. On tier 7, the hardest tier any run held:\n\nAfter seed 0 alone, I was ready to write “continual backprop saves the agent”. Seed 1 is why that sentence isn’t in this article. What I think the evidence supports is narrower: plasticity loss is real and continual backprop removes it, but a network with a fifth of its units dead can still play well for a long time. The decay only sometimes turns into worse play. In my reading, continual backprop protects against the dips, but two seeds can’t tell me how often the dips happen.\n\nNeither version ever held tier 8. So whatever stops the agent at tier 7 is not plasticity loss alone. My guess is network capacity or the observation, but I haven’t tested either.\n\nThe open problem I’m left with is tier 7. Seven tiers took 2.4 hours, and the eighth hasn’t fallen in 74 more. I’d like to know whether it’s the 128-unit network, the 155-number observation or something about the training setup, and that’s the next experiment.\n\n[I Trained an AI Agent to Play a Game for 76 Hours (And It Learned From Its Mistakes)](https://pub.towardsai.net/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its-mistakes-7c45749a68c6) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its", "canonical_source": "https://pub.towardsai.net/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its-mistakes-7c45749a68c6?source=rss----98111c9905da---4", "published_at": "2026-10-27 03:51:35+00:00", "updated_at": "2026-10-27 04:15:48.080139+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-agents", "neural-networks"], "entities": ["endless-learner", "NumPy", "Double DQN", "Adam", "van Hasselt", "Jiang", "i7-1265U"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its", "markdown": "https://wpnews.pro/news/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its.md", "text": "https://wpnews.pro/news/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its.txt", "jsonld": "https://wpnews.pro/news/i-trained-an-ai-agent-to-play-a-game-for-76-hours-and-it-learned-from-its.jsonld"}}