{"slug": "one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all", "title": "One channel decided whether my multi-agent RL agents learned at all", "summary": "A developer found that a single observation channel—a directional beacon pointing toward the nearest goal—determined whether multi-agent reinforcement learning agents using MAPPO learned at all. In a cooperative task with eight agents on a 100×100 grid, agents without the beacon achieved 0% success, while those with it reached 99.5% success, demonstrating that the beacon is essential for learning in this partially observable environment.", "body_md": "I have a small cooperative multi-agent reinforcement-learning setup: eight agents on a 100×100 grid, learning to reach three goals while gathering resources and avoiding obstacles. Standard MAPPO — a shared actor, a centralized critic, per-agent advantages, the usual PPO machinery. Each agent sees the world through an 11×11 egocentric window.\n\nThat window is the whole story, and it took an ablation to make me take it seriously.\n\nLook at the numbers for a second. The grid is 100×100. The agent sees an 11×11 patch centered on itself — reach ±5 in each direction. Goals are placed uniformly at random. So on any given step, what's the chance a goal is actually *inside* the patch an agent can see?\n\nAlmost never. A goal is a small target on a 10,000-cell map, and each agent perceives roughly 1% of that map at a time. For the overwhelming majority of steps, the agent's observation contains **no signal about where it's supposed to go**. It's a partially observed task in the most literal sense: the objective is usually outside the sensory window entirely.\n\nThere's a common fix for this, and my environment uses it: an **objective beacon**. Instead of hoping the goal wanders into view, you project the *bearing* to the nearest un-reached goal onto the border of the observation — a \"go this way\" arrow painted into the goal channel, always present, pointing toward the objective even when the objective itself is far outside the window. It's cheap, it's a handful of lines, and it turns an unobservable target into a directional gradient the policy can climb.\n\nI knew, abstractly, that this kind of thing helps. What I didn't have was a sense of *how much* — whether the beacon was a nice-to-have that shaved some episodes off, or something more fundamental. So I ran the ablation: train with the beacon, train without it, change absolutely nothing else, and measure.\n\nThe only thing that differs between the two conditions is a `beacon=True/False`\n\nflag that wraps the beacon-drawing block. Same architecture, same hyperparameters (repo defaults — LR 1e-3, γ 0.99, GAE-λ 0.95, clip 0.2, entropy 0.005), same 250 training updates, same 64 parallel envs, same evaluation protocol, same four seeds for each condition. Greedy evaluation over 50 fixed-layout episodes, plus a random-policy baseline for reference.\n\nA quick sanity check that the toggle does what it says: summing the goal-channel activation across a batch at reset gives **490.0 with the beacon on and 1.0 with it off**. With the beacon disabled, the goal channel is essentially empty — which is exactly the point. Without the beacon, the goal almost never appears in any agent's window, so the channel carries nearly no information.\n\n| Condition | Success rate | Reach rate | Return | Steps to solve |\n|---|---|---|---|---|\nBeacon ON (mean ± std, 4 seeds) |\n0.995 ± 0.010 |\n0.998 ± 0.003 | +9.05 ± 0.22 | 43 / 200 |\nBeacon OFF (mean ± std, 4 seeds) |\n0.000 ± 0.000 |\n0.105 ± 0.053 | −75.4 ± 3.9 | 200 (timeout) |\n| Random policy | 0.00 | 0.10 | −72.8 | 200 |\n\n\"Success\" means all three goals reached in an episode; the rate is the fraction of episodes fully solved.\n\nWith the beacon, MAPPO solves the task: **99.5% of episodes**, in about 43 of the allotted 200 steps, consistent across all four seeds (per-seed success: 1.0, 0.98, 1.0, 1.0). The training return climbs from about −10 and crosses into positive territory around update 45.\n\nWithout the beacon — identical everything else — it **does not learn at all**. Zero percent success on every seed. Its reach rate (0.105) is statistically indistinguishable from the random policy (0.10). The training curve never rises; it drifts slightly *more* negative over time. This isn't \"a bit worse.\" It's at chance.\n\nThe gap between conditions is roughly **100× the seed-to-seed noise** on success rate, so it's a real effect, not variance. The one-line version is honest: **0% → 99.5%, from one channel's worth of directional signal.**\n\nIt's tempting to file \"add a goal beacon\" under reward shaping or feature engineering — a knob you turn to speed things up. This ablation reframed it for me. In a task where the objective is almost always outside the observation window, the beacon isn't accelerating learning; it's supplying the *only* consistent signal about where to go. Take it away and there's nothing to climb — the policy is doing RL on observations that, most of the time, don't say anything about the goal. The agents don't learn slowly. They don't learn.\n\nThe general lesson, which I think travels beyond this toy: in a partially observed task, **what you put in the observation can matter more than the algorithm you run on it.** I could have spent that time tuning PPO clip ranges or critic architectures and gotten nowhere, because the bottleneck wasn't the optimizer — it was that the input didn't contain the answer. A single well-chosen feature — the bearing to the goal — was the difference between a policy that solves the map 99.5% of the time and one that never solves it at all.\n\nA few things I want to be precise about, because the strong number invites over-reading:\n\nThis is from **Kelvane**, a small open-source (Apache-2.0) toolkit I've been building: a WebAssembly runtime for running untrusted, hot-swappable neural policies under a per-call compute and authority budget, paired with this compact multi-agent RL reference that trains policies and exports them to ONNX to run inside that sandbox. The gridworld, the MAPPO/QMIX trainers, and the beacon are all in `kelvane-marl`\n\n, and the ablation toggle is a few lines you can flip yourself.\n\nRepo: [https://github.com/rakib-nyc/kelvane](https://github.com/rakib-nyc/kelvane)\n\nIf you're doing partially observed multi-agent RL and your agents stubbornly refuse to learn, the first question I'd now ask isn't about the algorithm. It's: *does the observation actually contain the objective?* Mine didn't — until one channel said which way to go.", "url": "https://wpnews.pro/news/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all", "canonical_source": "https://dev.to/rakib-nyc/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all-4051", "published_at": "2026-07-12 05:37:07+00:00", "updated_at": "2026-07-12 05:43:16.580188+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research"], "entities": ["MAPPO", "PPO"], "alternates": {"html": "https://wpnews.pro/news/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all", "markdown": "https://wpnews.pro/news/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all.md", "text": "https://wpnews.pro/news/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all.txt", "jsonld": "https://wpnews.pro/news/one-channel-decided-whether-my-multi-agent-rl-agents-learned-at-all.jsonld"}}