Is Memory the Bottleneck?

In a reproduction of Bourriez, Loizeau, and Abdin's (2023) Deep Recurrent Q-Network approach to autonomous satellite collision avoidance, a student researcher at Manipal University Jaipur found that the zero-thrust policy scored best among the policies evaluated and that the recurrent agent's greedy policy used zero fuel, under a reward the author designed. The author concludes reward design and training stability appear to be bigger problems than memory, while cautioning that one environment and one training run cannot establish that. The work addresses a real operational gap: the European Space Agency's 2024 Annual Space Environment Report estimates about 900,000 objects between 1 and 10 cm and about 128 million between 1 mm and 1 cm in orbit, and collision avoidance remains largely human-in-the-loop via Conjunction Data Messages defined in CCSDS Recommended Standard 508.0-B-1.

By-: Aarush Mahajan, B.Tech Computer Science, Manipal University Jaipur Part of my TKS Focus Project on AI for space domain awareness TL;DR: Satellites must dodge debris using tracking data that is noisy and incomplete. A recurrent agent with memory seems like the natural fix. In my reproduction, under a reward I designed, the zero-thrust policy scored best of the policies I evaluated, and the recurrent agent’s greedy policy used zero fuel. Reward design and training stability look like bigger problems than memory, but one environment and one training run cannot establish that. Imagine following another car on a foggy highway. Your dashboard shows its position, updated every few seconds, but the estimate is sometimes off by a noticeable margin. A driver who brakes or swerves on every new reading will overcorrect. A careful driver keeps a rough estimate of where the car has been and where it is heading, and reacts to that estimate. A satellite faces the same problem with debris. The debris is the car in the fog, and the satellite has to decide when to burn fuel. Bourriez, Loizeau, and Abdin 2023 argue that an agent with memory of past observations should handle this better, and they use a recurrent neural network to provide that memory. I wanted to test that claim in a simple setting, and the answer was not what I expected. I chose to focus on this problem because bridging the gap between simulated reinforcement learning rewards and the uncompromising physics of orbit is one of the most critical, yet under-discussed, safety barriers to scaling autonomous satellite constellations. Low Earth orbit holds much of the infrastructure we rely on. The European Space Agency’s Annual Space Environment Report 2024 edition estimates about 900,000 objects between 1 and 10 cm and about 128 million between 1 mm and 1 cm. Objects in the 1 to 10 cm range are too small to track reliably, and at orbital speeds even small objects can seriously damage a spacecraft European Space Agency, 2024 . Collisions are rare but costly, and each one adds debris that raises the risk for other objects. This feedback loop is known as Kessler syndrome Kessler and Cour-Palais, 1978 . Collision avoidance is still largely human-in-the-loop. Tracking networks issue Conjunction Data Messages CDMs , defined in CCSDS Recommended Standard 508.0-B-1. A review of collision avoidance methods Pavanello et al., 2025 describes CDMs as proximity alerts rather than collision warnings, which require further risk assessment. As constellations grow, this workflow becomes harder to sustain, which motivates onboard autonomy. In a Markov Decision Process MDP , an agent observes a state, chooses an action, receives a reward, and moves to a new state. The framework assumes the agent knows the state exactly. A satellite does not. Its measurements come from radar, optical sensors, and orbit predictions, and they carry two kinds of error. Measurement noise means a radar return is a cloud of possible positions, not a point. Model error means orbit predictors simplify forces such as Earth’s uneven gravity, atmospheric drag, and solar radiation pressure, so predicted positions drift over time. A Partially Observable MDP POMDP is one way to model this situation. The agent maintains a belief about the hidden state, built from its history of observations and actions. An MDP agent does not need perfect inputs to function, but without memory it cannot tell a one-off glitch from a real trend. Bourriez et al. frame autonomous collision avoidance as a POMDP and solve it with a Deep Recurrent Q-Network DRQN . A 2026 preprint by Georges and Abdin applies a Transformer-based reinforcement learning approach to autonomous orbital collision avoidance in partially observable environments. Figure 5. DRQN architecture. An LSTM carries a hidden state from step to step and feeds a Q-value layer that scores each of the five thrust options. The authors report that the agent performed well in terms of collision avoidance effectiveness, supported by training loss curves that converge to lower values in multi-environment setups. The paper does not report actual collision rates, fuel per episode, or held-out returns against a baseline. Its performance claims therefore rest on training behavior rather than task metrics. Setup. I trained agents for 400 episodes in a single environment using Clohessy-Wiltshire relative-motion dynamics, a linearized model simpler than full orbit propagation. Each run used one training seed, batch size 16, discount factor 0.99, learning rate 1e-3, and 128 hidden units. I compared three learned policies: a DRQN with an LSTM, a feed-forward DQN that stacks the last four observations, and a feed-forward DQN on the current observation alone. Each observation frame consists of seven features: 3D relative position, 3D relative velocity, and a normalized time index. Stacking four frames yields a 28-input vector. Action space. Each step has four burn slots, and each slot selects one of five options: no burn, ±0.1 m/s along y, or ±0.1 m/s along z. Thrust along x is not used. Each step applies at most 0.4 m/s. Fuel is charged on the magnitude of the net Δv summed over the slots, so opposing burns in the same step cancel. The agent could in principle expend zero net fuel while issuing opposing thrusts. Per-slot actions were not individually logged, so I cannot say whether the DRQN exploited this. Reward. I designed the reward, and it differs from the paper’s. At each step the environment charges −5.0 per m/s of net Δv and a continuous risk-pressure penalty of −1.0 × min 1, p c / 10⁻⁴ , where p c is a point-mass collision-probability proxy. The budget penalty works as follows: the 1.0 m/s budget is checked on cumulative fuel, and once cumulative fuel exceeds 1.0 m/s, a penalty of −10 is applied at every subsequent step of the episode, not once. A collision costs −100 and ends the episode. An episode that runs all 60 steps with no collision and a minimum distance above 106.7 m earns +50. Evaluation. Each policy was evaluated on a single evaluation block of 100 encounters, drawn from evaluation seeds separate from training. Training resets were not seeded, so I did not verify that no training geometry overlaps the evaluation block. “Success” means the episode ends with no collision and a minimum distance above 106.7 m. The random-action, no-maneuver, and DRQN rows come from notebook cell 20. The two feed-forward rows come from cell 33, which evaluates the same 100 encounters. Observations. First, under my reward, the zero-thrust policy scored best of the policies evaluated. Its mean return of −60.33 beats random actions −543.08 and both trained feed-forward agents −579.82 and −606.54 . This ordering is a property of the reward I designed, not evidence about the method. I hypothesize that the terminal collision penalty −100 is small relative to the per-step fuel and budget penalties, so accepting a collision is cheaper than the trained feed-forward agents’ evasive maneuvers. The table shows this ordering; it does not test the mechanism. Second, the DRQN’s greedy policy used zero fuel. Training and greedy evaluation use different procedures. During training, the agent acts with ε-greedy exploration at ε = 0.05, and the training log records one result per episode. Over episodes 1–250, when exploration was still high, the DRQN succeeded in 95.2% of training episodes with no collisions. Over episodes 301–400, at ε = 0.05, it succeeded in 60.0% and collided in 3.0%. Greedy evaluation uses no exploration and is the policy that matters for deployment. Under greedy evaluation, mean fuel was exactly 0.00 m/s. Because opposing slots can cancel, zero net Δv is consistent with zero thrust but does not prove that action 0 was chosen at every step. Per-step, per-slot actions were not logged. Its summary metrics match the no-maneuver baseline exactly on every reported statistic. Table 3. DRQN training results by episode block, from the training log. Third, the feed-forward networks’ loss diverged. The Huber loss for the stacked-frame DQN was 6.9 at episode 1, 1.9 × ¹⁰⁵ at episode 50, about 1.5 × ¹⁰⁸ at episode 200, and 1.7 × ¹⁰⁹ at episode 400. The single-frame DQN reached 7.7 × ¹⁰⁸ at episode 400. For the DRQN, the logged loss fell from 3.8 at episode 25 to 2.3 at episode 400, with values logged every 25 episodes. I hypothesize that the feed-forward agents’ fuel overspending is tied to this instability, rather than being a deliberate evasion strategy. I have not run a controlled test to establish it. Why random actions outscore the trained agents. Random actions return −543.08, and the trained feed-forward agents return −579.82 and −606.54. The random policy has 91% success and no collisions. Its advantage comes from fuel, not from avoiding collisions better: it spent 9.49 m/s, against 13.42 and 16.97 m/s for the trained agents. The reward breakdown below shows that these fuel and budget penalties dominate the return. The comparison shows that the trained agents overspend relative to this reward. It does not show that random thrust is a good policy. Reward breakdown. The table below decomposes the returns that include a non-zero collision or success term. The collision and success terms come from the reported rates in cell 33: 95% and 97% success multiplied by +50, and 5 × mean fuel for the fuel cost. The budget penalty and the unlogged residual cannot be separated from the logged totals, so the table combines them into one derived column: the reported return minus the other terms. That combined value contains the budget penalty, the continuous risk-pressure term, and anything else not logged. For the two feed-forward agents, the combined column is dominated by the budget penalty. If the residual is small, dividing it by −10 gives about 56 penalized steps per episode for the stacked-frame DQN and 57 for the single-frame DQN. These counts are inferred from summary totals, not logged per episode, and they assume the residual is negligible. I have not decomposed the random-action return, because per-term values for that policy were not logged. These results come from one simplified environment, one reward I designed, one training run per architecture, and one 100-encounter evaluation block. The conclusions apply to that setup only. Compute. The DRQN performs four gradient updates per episode, which gives about 1,600 updates over 400 episodes. The feed-forward networks update once per environment step. Their total depends on episode length: at the 60-step maximum it is about 24,000 updates over 400 episodes, and it is lower for episodes that end early through collision. The 24,000 figure is an upper bound that assumes every episode runs the full 60 steps. The ratio is roughly 15-fold at that upper bound, so the architecture comparison is not compute-matched. Dynamics. Clohessy-Wiltshire is a linearized relative-motion model. It omits nonlinear perturbations and the atmospheric density changes that matter in real low orbit. Thrust model. The paper uses discrete impulsive burns. Many modern satellites use low-thrust electric propulsion, which requires extended maneuvers that discrete impulsive action sets do not represent well. Safety. Operators need concrete guarantees, and a deep network can produce surprising actions on off-nominal inputs. Safety layers, such as those that override unsafe actions, remain an active research direction. First, the update-count confound. Run the feed-forward agents with the DRQN’s update schedule, and run the DRQN with a matched update count. If the DRQN’s passivity disappears when its update count matches the feed-forward agents, the rival explanation overtakes the reward hypothesis. Second, training stability. Investigate the feed-forward divergence by tuning the learning rate and clipping targets to stabilize the Huber loss. Until the baselines train stably, the stacked-versus-single comparison is not informative. Third, reward shaping. Replace the step-function budget penalty with a smooth term, or significantly reduce the −5.0 fuel penalty weight during early curriculum episodes, so the agent is not punished for early evasive burns. If the DRQN starts maneuvering greedily under a balanced reward, that supports the hypothesis that reward design is the bottleneck. Only after agents avoid collisions reliably under a balanced reward does it make sense to ask whether recurrent memory adds value. That test would use harder, non-Gaussian, temporally correlated tracking noise, where a recurrent agent should have an advantage if memory matters. Is Memory the Bottleneck? https://pub.towardsai.net/is-memory-the-bottleneck-e1c2967246e6 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.