I have a small cooperative multi-agent reinforcement-learning setup: eight agents on a 100×100 grid, learning to reach three goals while gathering resources and avoiding obstacles. Standard MAPPO — a shared actor, a centralized critic, per-agent advantages, the usual PPO machinery. Each agent sees the world through an 11×11 egocentric window.
That window is the whole story, and it took an ablation to make me take it seriously.
Look at the numbers for a second. The grid is 100×100. The agent sees an 11×11 patch centered on itself — reach ±5 in each direction. Goals are placed uniformly at random. So on any given step, what's the chance a goal is actually inside the patch an agent can see?
Almost never. A goal is a small target on a 10,000-cell map, and each agent perceives roughly 1% of that map at a time. For the overwhelming majority of steps, the agent's observation contains no signal about where it's supposed to go. It's a partially observed task in the most literal sense: the objective is usually outside the sensory window entirely.
There's a common fix for this, and my environment uses it: an objective beacon. Instead of hoping the goal wanders into view, you project the bearing to the nearest un-reached goal onto the border of the observation — a "go this way" arrow painted into the goal channel, always present, pointing toward the objective even when the objective itself is far outside the window. It's cheap, it's a handful of lines, and it turns an unobservable target into a directional gradient the policy can climb.
I knew, abstractly, that this kind of thing helps. What I didn't have was a sense of how much — whether the beacon was a nice-to-have that shaved some episodes off, or something more fundamental. So I ran the ablation: train with the beacon, train without it, change absolutely nothing else, and measure.
The only thing that differs between the two conditions is a beacon=True/False
flag that wraps the beacon-drawing block. Same architecture, same hyperparameters (repo defaults — LR 1e-3, γ 0.99, GAE-λ 0.95, clip 0.2, entropy 0.005), same 250 training updates, same 64 parallel envs, same evaluation protocol, same four seeds for each condition. Greedy evaluation over 50 fixed-layout episodes, plus a random-policy baseline for reference.
A quick sanity check that the toggle does what it says: summing the goal-channel activation across a batch at reset gives 490.0 with the beacon on and 1.0 with it off. With the beacon disabled, the goal channel is essentially empty — which is exactly the point. Without the beacon, the goal almost never appears in any agent's window, so the channel carries nearly no information.
| Condition | Success rate | Reach rate | Return | Steps to solve |
|---|---|---|---|---|
Beacon ON (mean ± std, 4 seeds) |
0.995 ± 0.010 | 0.998 ± 0.003 | +9.05 ± 0.22 | 43 / 200 | Beacon OFF (mean ± std, 4 seeds) | 0.000 ± 0.000 | 0.105 ± 0.053 | −75.4 ± 3.9 | 200 (timeout) | | Random policy | 0.00 | 0.10 | −72.8 | 200 |
"Success" means all three goals reached in an episode; the rate is the fraction of episodes fully solved.
With the beacon, MAPPO solves the task: 99.5% of episodes, in about 43 of the allotted 200 steps, consistent across all four seeds (per-seed success: 1.0, 0.98, 1.0, 1.0). The training return climbs from about −10 and crosses into positive territory around update 45.
Without the beacon — identical everything else — it does not learn at all. Zero percent success on every seed. Its reach rate (0.105) is statistically indistinguishable from the random policy (0.10). The training curve never rises; it drifts slightly more negative over time. This isn't "a bit worse." It's at chance.
The gap between conditions is roughly 100× the seed-to-seed noise on success rate, so it's a real effect, not variance. The one-line version is honest: 0% → 99.5%, from one channel's worth of directional signal.
It's tempting to file "add a goal beacon" under reward shaping or feature engineering — a knob you turn to speed things up. This ablation reframed it for me. In a task where the objective is almost always outside the observation window, the beacon isn't accelerating learning; it's supplying the only consistent signal about where to go. Take it away and there's nothing to climb — the policy is doing RL on observations that, most of the time, don't say anything about the goal. The agents don't learn slowly. They don't learn.
The general lesson, which I think travels beyond this toy: in a partially observed task, what you put in the observation can matter more than the algorithm you run on it. I could have spent that time tuning PPO clip ranges or critic architectures and gotten nowhere, because the bottleneck wasn't the optimizer — it was that the input didn't contain the answer. A single well-chosen feature — the bearing to the goal — was the difference between a policy that solves the map 99.5% of the time and one that never solves it at all.
A few things I want to be precise about, because the strong number invites over-reading:
This is from Kelvane, a small open-source (Apache-2.0) toolkit I've been building: a WebAssembly runtime for running untrusted, hot-swappable neural policies under a per-call compute and authority budget, paired with this compact multi-agent RL reference that trains policies and exports them to ONNX to run inside that sandbox. The gridworld, the MAPPO/QMIX trainers, and the beacon are all in kelvane-marl
, and the ablation toggle is a few lines you can flip yourself.
Repo: [https://github.com/rakib-nyc/kelvane](https://github.com/rakib-nyc/kelvane)
If you're doing partially observed multi-agent RL and your agents stubbornly refuse to learn, the first question I'd now ask isn't about the algorithm. It's: *does the observation actually contain the objective?* Mine didn't — until one channel said which way to go.