I started this project thinking the problem was obvious. My PPO agent kept timing out.
It rarely crashed. It did not explode into obstacles. It did not behave randomly. Instead, it hesitated. It drifted around the environment, avoided obvious danger, and failed to commit to a path before the episode ended.
Naturally, I blamed the reward function. That was only partly correct.
Reward redesign improved the agent. Reducing lidar resolution improved it even more. Removing one reward term destroyed the learned behavior almost completely. But the most uncomfortable result came at the end: simple heuristic controllers still outperformed every PPO policy I trained.
The best PPO policy reached 50.8% success. The best hand-written heuristic reached 84.3%. That gap changed how I interpreted the entire project.
This was not a story about “solving” drone navigation with reinforcement learning. It became a story about failure modes, observation complexity, reward dependence, and why strong baselines matter.
The task was a custom 2D drone-navigation environment. The drone received relative goal position, velocity, heading information, and lidar-style ray observations. The policy produced continuous heading and speed commands through PPO.
The objective was simple, to reach the target* without colliding with obstacles.*
The first baseline used 16 lidar rays.
Across three independent seeds, the baseline PPO achieved:
Almost half of the evaluation episodes ended in timeout. That was the first important clue. The agent was not primarily failing because it crashed. It was failing because it refused to make sustained progress.
My first intervention was reward and termination redesign.
The original hard configuration was relatively permissive:
This created a predictable failure pattern: the agent could survive for a long time without solving the task.
So I introduced a stricter configuration with explicit pressure against passive behavior. I added timeout penalties, stall penalties, low-speed penalties, stuck detection, shorter effective episode pressure, and stricter termination logic.
After several intermediate configurations, the best balanced redesign produced the following three-seed average:
At first glance, this looked like a clear win. Success increased from 35.1% to 44.2%, a relative improvement of about 26%. Timeouts dropped from 48.2% to 7.2%. That part was genuinely useful. The agent no longer spent most episodes drifting until the clock ran out.
But the failure modes did not disappear. They moved.
The old policy mostly failed by timing out. The redesigned policy mostly failed by getting marked as stuck or by colliding more often. This distinction matters.
If I had only reported success rate, I would have said: reward redesign improved PPO. That statement is true, but incomplete. The more accurate statement is “reward redesign improved task completion and reduced timeout failures, but it redistributed failure modes toward stuck behavior and collisions.”
That is a much more useful result.
This was the first major lesson: Reward shaping can change how an agent fails without fully solving why it fails.
Before reaching the final redesigned configuration, I tested several variants. Some looked promising for one metric while damaging others. One stricter version reduced timeout from 46.3% to 20.0%, but success dropped from 36.7% to 29.3%, and stuck episodes exploded to 45.0%.
That was not an improvement. It was failure relabeling. The old policy failed by timing out. The new policy failed by being marked stuck*.*
Later configurations pushed the policy in the opposite direction. Success improved, but collisions increased sharply. The most aggressive version reached 46.0% success and the best path efficiency, but collision rate climbed to 17.0%.
That gave me three behavioral regimes:
This helped clarify the actual optimization problem.
I was not simply looking for “more reward” or “stronger penalties.” I was tuning the trade-off between decisiveness, safety, and stability. The best redesigned PPO policy was not the most aggressive one. It was the one that improved completion while keeping failure modes interpretable.
More sensor information should improve navigation.
At least, that was my assumption.
The agent used lidar-like ray observations. So I tested different sensor resolutions while keeping the training setup fixed. I expected 32 rays to perform best.
It did not.
Using the redesigned reward/termination setup, the results were: The best policy used only 8 rays. The worst used 32.
This was one of the most surprising results in the project.
More lidar resolution did not improve PPO performance under the fixed training budget and network setup. It made learning worse. A likely explanation is observation dimensionality.
Approximate observation size increased like this:
The policy network and training budget stayed the same.
So the 32-ray policy had more information, but also a harder optimization problem. In this environment, extra sensory detail may have acted less like useful information and more like additional complexity. This does not mean higher-resolution perception is generally bad.
It means something narrower and more important. Under a fixed PPO setup, fixed training budget, and fixed network architecture, more observation detail did not automatically translate into better behavior.
That is the kind of result I would have missed if I had only tuned rewards.
At this point, I had improved the agent from 35.1% success to 50.8%. That is a 15.7 percentage point absolute gain, or about 45% relative improvement. But I still did not know what the policy had actually learned.
Was PPO learning navigation? Or was it mostly following reward shaping?
To test this, I removed the progress reward while keeping the environment and training setup otherwise unchanged.
The result was immediate and brutal.
Success collapsed from 50.8% to below 1%. Collision rate jumped to 99.1%. That result changed my interpretation of the agent. Progress reward was not just accelerating learning. It was providing a critical optimization signal without which PPO failed to discover a viable navigation strategy.
This was not necessarily reward hacking. The progress term was aligned with the task: moving closer to the goal is exactly what the drone should do.
But the ablation showed strong reward dependence. The learned behavior was not robust to removing that shaping signal. That made the result more honest. PPO did not simply “learn navigation.” It learned a navigation behavior that depended heavily on dense progress feedback.
After all that, the best PPO policy achieved 50.8% success. That looked like progress. Then I compared it against simple hand-written baselines.
The baselines were:
The results were not even close.
This was the most uncomfortable result in the project.
After reward redesign, sensor ablation, and progress reward analysis, a few hand-crafted navigation rules still performed dramatically better than PPO. The wall-avoiding greedy heuristic achieved 84.3% success. The best PPO policy achieved 50.8%. That is not a small gap. That is a different performance regime.
The obstacle-aware heuristic was also instructive. It almost never collided, with only 0.3% collision rate, but timed out 20.7% of the time.
That made it a useful behavioral reference. Greedy was fast but riskier. Obstacle-aware was safe but slower. Wall-avoiding greedy was both simple and strong. PPO was improved, but still was not competitive.
This changed the main conclusion. I could no longer say “PPO solved the navigation task.” The more honest conclusion was: “PPO improved substantially, but it still failed to match simple heuristic controllers.”
This project started as a reward-design problem. It ended as something broader.
The most important lessons were not:
The real lessons were:
The reward redesign helped. It increased success from 35.1% to 44.2% and reduced timeout failures from 48.2% to 7.2%. But it also shifted failures toward stuck behavior and collisions. The sensor experiment helped even more. Reducing lidar rays from 16 to 8 improved success to 50.8%, while increasing rays to 32 made performance worse. The progress reward ablation revealed how dependent the learned behavior was on dense shaping. Without progress reward, success collapsed to 0.9%. And the heuristic baselines delivered the most important result:
PPO had improved, but the problem was not solved.
These results should not be read as evidence that PPO cannot solve this task.
The experiments used a fixed training budget, fixed architecture, fixed PPO setup, and a custom 2D environment. A larger policy, longer training, curriculum learning, recurrent memory, better normalization, different reward scaling, or alternative algorithms could improve performance.
The claim is narrower. In this setup, reward redesign and sensor simplification improved PPO, but simple hand-written navigation rules still outperformed the learned policy.
That is the result I trust. Not because PPO failed, but because the baselines forced me to measure what mattered.
Improving success from 35.1% to 50.8% was not a victory. Discovering why that still was not enough was the real takeaway.
If I had stopped after reward redesign, I would have concluded that the project was a success. If I had stopped after the sensor experiment, I would have concluded that observation design mattered more than reward tuning. If I had stopped after reaching 50.8% success, I would have concluded that PPO had learned navigation. All three conclusions would have been incomplete.
The heuristic baselines revealed the finding that mattered most: The agent improved, but it did not win. And that turned a reward-tuning exercise into something more valuable, an investigation of what the agent had actually learned.
Full project repository:
I Improved My PPO Agent by 45%. It Still Lost to Simple Rules was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.