FOVI-Dex: Foveation enables efficient perceptive learning of dexterous manipulation in simulation

A research team's FOVI-Dex system, described in a paper on foveated rendering and perception, trains a language-conditioned visuotactile policy that controls a dexterous arm-hand to find and grasp a target object in clutter via online reinforcement learning without demonstrations. Compared with high-resolution fixed-camera control, FOVI-Dex uses 8.9 fewer vision tokens, supports 6 more parallel environments and achieves 4.4 higher training throughput, and at matched training budgets outperforms both the high-resolution baseline and a matched uniform active-camera baseline in simulation and the real world, with an outsized advantage in the real world.

Advanced physics simulators and rendering engines promise to support the next-generation of robotic reinforcement learning RL , yet training perceptive policies remains challenging due to the high-costs of rendering and perception. We propose foveation as a solution to this challenge, coupling a novel foveated rendering solution with foveated perception, alongside a careful approach for ensuring transfer to foveated sampling of real uniform cameras. We train a language-conditioned visuotactile policy controlling a dexterous arm-hand to find and grasp a target object in clutter FOVI-Dex , via online RL without demonstrations. In comparison to a high-resolution fixed-camera control, FOVI-Dex uses 8.9 less vision tokens, supporting 6 more parallel environments and 4.4 higher training throughput. At matched training budgets, this efficiency enables FOVI-Dex to outperform both the high-resolution baseline and a matched uniform active camera baseline, in both sim and real, with an outsized advantage in the real world.