V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

Researchers introduced V-Simba, a visual reinforcement learning architecture built on Soft Actor-Critic (SAC) with data augmentation, which matches or outperforms state-of-the-art methods on the DMC, Adroit, and Meta-World benchmarks while being more computationally efficient than DrQ-v2. The architecture adds normalization layers and pointwise convolutions to stabilize training and reduce computation, addressing sample efficiency challenges in visual RL. Code is publicly available at https://github.com/DAVIAN-Robotics/V-Simba.

arXiv:2608.07870v1 Announce Type: new Abstract: Improving sample efficiency remains a core challenge in reinforcement learning RL , especially in real-world settings like robotics, where data collection is costly. This challenge is pronounced in visual RL, where high-dimensional inputs often obscure learning signals. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state-based RL show that architectural design alone can lead to significant gains in sample efficiency. This raises an important question: Can these architectural principles transfer to visual RL? In response, we introduce V-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state-based RL. Built on top of Soft Actor-Critic SAC with data augmentation, V-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation. Despite its simplicity, V-Simba matches or outperforms the state-of-the-art methods across the DMC, Adroit, and Meta-World benchmarks, while being more computationally efficient than DrQ-v2. We make our code publicly available at https://github.com/DAVIAN-Robotics/V-Simba.