PPO vs. SAC: A Comparative Study for Cooperative Motion Control of Dual-Arm Space Robots

A comparative study of reinforcement learning algorithms for dual-arm space robot motion control found Soft Actor-Critic (SAC) reached an 80% success threshold in all five random seeds (median 1.6M environment steps) while Proximal Policy Optimization (PPO) never reached it within a 4M-step budget, according to the arXiv paper 2610.32034v1. SAC posted a mean success rate of 78.6% versus PPO's 7.4%, with lower position and orientation errors and tighter bimanual synchronization, and retained most of its performance on unseen boundary targets where PPO's collapsed. The authors attribute PPO's failure to a reward-loitering local optimum under the two-phase shaped reward, highlighting on-policy learning's sensitivity to reward shaping.

arXiv:2610.32034v1 Announce Type: new Abstract: The growing population of orbital debris has intensified interest in active debris removal and on-orbit servicing, for which dual-arm space manipulators offer improved grasping stability and load distribution relative to single-arm systems. However, coordinating two manipulators under inter-arm collision avoidance, orientation alignment, and sensing uncertainty remains challenging. This paper develops a coordinated reinforcement learning framework for cooperative motion planning of a dual-arm space manipulator in a target-reaching task with inter-arm collision-avoidance constraints. A centralized state representation combines joint-space, end-effector, and target-relative information, while a multi-objective reward function promotes target-reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance. Within this framework, Proximal Policy Optimization PPO and Soft Actor-Critic SAC are compared as representative on-policy and off-policy algorithms, and the learned policies are further evaluated for generalization to previously unseen target configurations. Across five random seeds, SAC substantially outperforms PPO within the considered fixed-base simulation scenarios: it reaches an 80% success threshold in every seed median 1.6M environment steps whereas PPO does not reach it within a 4M-step budget, and it attains a higher mean success rate 78.6% versus 7.4% with lower position and orientation errors and tighter bimanual synchronization. Both algorithms degrade on unseen boundary targets, but SAC retains most of its performance while PPO's collapses. The results also expose a failure mode of PPO: under the two-phase shaped reward, it tends to converge to a reward-loitering local optimum rather than committing to the precision task. This behavior highlights the sensitivity of on-policy learning to reward shaping.