{"slug": "i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules", "title": "I Improved My PPO Agent by 45%. It Still Lost to Simple Rules", "summary": "A developer's PPO-based drone navigation agent improved 45% after reward redesign but still lost to simple heuristic rules, with the best PPO policy achieving 50.8% success versus 84.3% for a hand-written heuristic. The project revealed that reward shaping redistributed failure modes rather than solving them, highlighting the importance of strong baselines in reinforcement learning.", "body_md": "I started this project thinking the problem was obvious. My PPO agent kept timing out.\n\nIt rarely crashed. It did not explode into obstacles. It did not behave randomly. Instead, it hesitated. It drifted around the environment, avoided obvious danger, and failed to commit to a path before the episode ended.\n\nNaturally, I blamed the reward function. That was only partly correct.\n\nReward redesign improved the agent. Reducing lidar resolution improved it even more. Removing one reward term destroyed the learned behavior almost completely. But the most uncomfortable result came at the end: simple heuristic controllers still outperformed every PPO policy I trained.\n\nThe best PPO policy reached 50.8% success. The best hand-written heuristic reached 84.3%. That gap changed how I interpreted the entire project.\n\nThis was not a story about “solving” drone navigation with reinforcement learning. It became a story about failure modes, observation complexity, reward dependence, and why strong baselines matter.\n\nThe task was a custom 2D drone-navigation environment. The drone received relative goal position, velocity, heading information, and lidar-style ray observations. The policy produced continuous heading and speed commands through PPO.\n\nThe objective was simple, to reach the target* without colliding with obstacles.*\n\nThe first baseline used 16 lidar rays.\n\nAcross three independent seeds, the baseline PPO achieved:\n\nAlmost half of the evaluation episodes ended in timeout. That was the first important clue. The agent was not primarily failing because it crashed. It was failing because it refused to make sustained progress.\n\nMy first intervention was reward and termination redesign.\n\nThe original hard configuration was relatively permissive:\n\nThis created a predictable failure pattern: the agent could survive for a long time without solving the task.\n\nSo I introduced a stricter configuration with explicit pressure against passive behavior. I added timeout penalties, stall penalties, low-speed penalties, stuck detection, shorter effective episode pressure, and stricter termination logic.\n\nAfter several intermediate configurations, the best balanced redesign produced the following three-seed average:\n\nAt first glance, this looked like a clear win. Success increased from 35.1% to 44.2%, a relative improvement of about 26%. Timeouts dropped from 48.2% to 7.2%. That part was genuinely useful. The agent no longer spent most episodes drifting until the clock ran out.\n\nBut the failure modes did not disappear. They moved.\n\nThe old policy mostly failed by timing out. The redesigned policy mostly failed by getting marked as stuck or by colliding more often. This distinction matters.\n\nIf I had only reported success rate, I would have said: reward redesign improved PPO. That statement is true, but incomplete.\n\nThe more accurate statement is “reward redesign improved task completion and reduced timeout failures, but it redistributed failure modes toward stuck behavior and collisions.”\n\nThat is a much more useful result.\n\nThis was the first major lesson: Reward shaping can change how an agent fails without fully solving why it fails.\n\nBefore reaching the final redesigned configuration, I tested several variants. Some looked promising for one metric while damaging others. One stricter version reduced timeout from 46.3% to 20.0%, but success dropped from 36.7% to 29.3%, and stuck episodes exploded to 45.0%.\n\nThat was not an improvement. It was failure relabeling. The old policy failed by timing out. The new policy failed by being marked stuck*.*\n\nLater configurations pushed the policy in the opposite direction. Success improved, but collisions increased sharply. The most aggressive version reached 46.0% success and the best path efficiency, but collision rate climbed to 17.0%.\n\nThat gave me three behavioral regimes:\n\nThis helped clarify the actual optimization problem.\n\nI was not simply looking for “more reward” or “stronger penalties.” I was tuning the trade-off between decisiveness, safety, and stability. The best redesigned PPO policy was not the most aggressive one. It was the one that improved completion while keeping failure modes interpretable.\n\nMore sensor information should improve navigation.\n\nAt least, that was my assumption.\n\nThe agent used lidar-like ray observations. So I tested different sensor resolutions while keeping the training setup fixed. I expected 32 rays to perform best.\n\nIt did not.\n\nUsing the redesigned reward/termination setup, the results were:\n\nThe best policy used only 8 rays. The worst used 32.\n\nThis was one of the most surprising results in the project.\n\nMore lidar resolution did not improve PPO performance under the fixed training budget and network setup. It made learning worse. A likely explanation is observation dimensionality.\n\nApproximate observation size increased like this:\n\nThe policy network and training budget stayed the same.\n\nSo the 32-ray policy had more information, but also a harder optimization problem. In this environment, extra sensory detail may have acted less like useful information and more like additional complexity. This does not mean higher-resolution perception is generally bad.\n\nIt means something narrower and more important. Under a fixed PPO setup, fixed training budget, and fixed network architecture, more observation detail did not automatically translate into better behavior.\n\nThat is the kind of result I would have missed if I had only tuned rewards.\n\nAt this point, I had improved the agent from 35.1% success to 50.8%. That is a 15.7 percentage point absolute gain, or about 45% relative improvement. But I still did not know what the policy had actually learned.\n\nWas PPO learning navigation? Or was it mostly following reward shaping?\n\nTo test this, I removed the progress reward while keeping the environment and training setup otherwise unchanged.\n\nThe result was immediate and brutal.\n\nSuccess collapsed from 50.8% to below 1%. Collision rate jumped to 99.1%. That result changed my interpretation of the agent. Progress reward was not just accelerating learning. It was providing a critical optimization signal without which PPO failed to discover a viable navigation strategy.\n\nThis was not necessarily reward hacking. The progress term was aligned with the task: moving closer to the goal is exactly what the drone should do.\n\nBut the ablation showed strong reward dependence. The learned behavior was not robust to removing that shaping signal. That made the result more honest. PPO did not simply *“learn navigation.”* It learned a navigation behavior that depended heavily on dense progress feedback.\n\nAfter all that, the best PPO policy achieved 50.8% success. That looked like progress. Then I compared it against simple hand-written baselines.\n\nThe baselines were:\n\nThe results were not even close.\n\nThis was the most uncomfortable result in the project.\n\nAfter reward redesign, sensor ablation, and progress reward analysis, a few hand-crafted navigation rules still performed dramatically better than PPO. The wall-avoiding greedy heuristic achieved 84.3% success. The best PPO policy achieved 50.8%. That is not a small gap. That is a different performance regime.\n\nThe obstacle-aware heuristic was also instructive. It almost never collided, with only 0.3% collision rate, but timed out 20.7% of the time.\n\nThat made it a useful behavioral reference. Greedy was fast but riskier. Obstacle-aware was safe but slower. Wall-avoiding greedy was both simple and strong. PPO was improved, but still was not competitive.\n\nThis changed the main conclusion. I could no longer say “PPO solved the navigation task.” The more honest conclusion was: “PPO improved substantially, but it still failed to match simple heuristic controllers.”\n\nThis project started as a reward-design problem. It ended as something broader.\n\nThe most important lessons were not:\n\nThe real lessons were:\n\nThe reward redesign helped. It increased success from 35.1% to 44.2% and reduced timeout failures from 48.2% to 7.2%. But it also shifted failures toward stuck behavior and collisions. The sensor experiment helped even more. Reducing lidar rays from 16 to 8 improved success to 50.8%, while increasing rays to 32 made performance worse. The progress reward ablation revealed how dependent the learned behavior was on dense shaping. Without progress reward, success collapsed to 0.9%. And the heuristic baselines delivered the most important result:\n\nPPO had improved, but the problem was not solved.\n\nThese results should not be read as evidence that PPO cannot solve this task.\n\nThe experiments used a fixed training budget, fixed architecture, fixed PPO setup, and a custom 2D environment. A larger policy, longer training, curriculum learning, recurrent memory, better normalization, different reward scaling, or alternative algorithms could improve performance.\n\nThe claim is narrower. In this setup, reward redesign and sensor simplification improved PPO, but simple hand-written navigation rules still outperformed the learned policy.\n\nThat is the result I trust. Not because PPO failed, but because the baselines forced me to measure what mattered.\n\nImproving success from 35.1% to 50.8% was not a victory. Discovering why that still was not enough was the real takeaway.\n\nIf I had stopped after reward redesign, I would have concluded that the project was a success. If I had stopped after the sensor experiment, I would have concluded that observation design mattered more than reward tuning. If I had stopped after reaching 50.8% success, I would have concluded that PPO had learned navigation.\n\nAll three conclusions would have been incomplete.\n\nThe heuristic baselines revealed the finding that mattered most: The agent improved, but it did not win. And that turned a reward-tuning exercise into something more valuable, an investigation of what the agent had actually learned.\n\n**Full project repository:**\n\n[I Improved My PPO Agent by 45%. It Still Lost to Simple Rules](https://pub.towardsai.net/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules-2fccde6246f8) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules", "canonical_source": "https://pub.towardsai.net/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules-2fccde6246f8?source=rss----98111c9905da---4", "published_at": "2026-09-02 17:01:04+00:00", "updated_at": "2026-09-02 17:23:26.632680+00:00", "lang": "en", "topics": ["machine-learning", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules", "markdown": "https://wpnews.pro/news/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules.md", "text": "https://wpnews.pro/news/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules.txt", "jsonld": "https://wpnews.pro/news/i-improved-my-ppo-agent-by-45-it-still-lost-to-simple-rules.jsonld"}}