New robot policy runs on an RTX 5090, beating OpenVLA-OFT while using 0.7% of its size

Jeff Cui and collaborators announced Patch Policy, a robot-control method that beats OpenVLA-OFT with 0.7% of its parameters and trains on a single NVIDIA RTX 5090 GPU. The approach freezes a Vision Transformer pretrained on large-scale image data and uses dense patch embeddings with a lightweight transformer policy, achieving 40% higher task success over global-pooled policies and 18% over OpenVLA-OFT. The findings, detailed in arXiv:2607.18236, challenge the need for massive vision-language models in robot learning.

A lean visual policy beats a 7B‑parameter VLM Jeff Cui @jeffacce https://x.com/jeffacce announced on July 22, 2026 a new robot‑control method he calls Patch Policy . The approach freezes a state‑of‑the‑art Vision Transformer ViT pretrained on large‑scale image data, feeds every patch token to a lightweight transformer policy, and adds a simple block‑causal mask. The result, Cui writes, “beats OpenVLA‑OFT with 0.7% of its params, and trains on a 5090.” The claim is backed by an arXiv preprint arXiv:2607.18236 co‑authored with Gaoyue Zhou @GaoyueZhou https://x.com/GaoyueZhou , Ada Langford @LangfordAd91020 https://x.com/LangfordAd91020 , Bowen Tan @bowentan https://x.com/bowentan , Yann LeCun @ylecun https://x.com/ylecun and Lerrel Pinto @LerrelPinto https://x.com/LerrelPinto . The paper details experiments on a suite of simulated and real‑world manipulation tasks, comparing three families of visual policies: Global‑pooled features e.g., ResNet or ViT CLS token – the historic baseline. OpenVLA‑OFT , a 7 billion‑parameter vision‑language model fine‑tuned for control. Patch Policy , the new method using dense patch embeddings from a frozen ViT and a small transformer head. Performance gap Across the benchmark, Patch Policy improves task success by 40% over global‑pooled policies and by 18% over OpenVLA‑OFT when both are trained on the same tasks. The tiny policy contains only about 0.7 % of the total parameters of OpenVLA‑OFT, yet inference runs at roughly 10 ms latency – six times faster than the VLM baseline – without any hardware‑specific optimization. Training on consumer hardware Cui emphasizes that the entire pipeline fits on a single NVIDIA RTX 5090 GPU. “It trains on a 5090,” he notes in the first tweet of the thread. The training cost, therefore, is comparable to that of many academic labs and far below the multi‑GPU clusters typically required for 7 B‑parameter models. Why dense patches matter The thread argues that prior robot‑learning pipelines either pooled visual features globally, trained vision models from scratch, or hauled around massive VLMs. By contrast, dense patch features retain spatial detail that is crucial for precise manipulation. A side‑by‑side video in the thread shows a global‑pooled policy failing to locate a 2 mm‑tolerance cable, while the Patch Policy inserts the cable cleanly. Encoder comparison Cui’s team benchmarked five frozen encoders as control representations. DINOv2 and WebSSL consistently topped the leaderboard, suggesting that modern self‑supervised vision models provide the most useful dense features for control. The rankings remained stable across tasks and policy heads, reinforcing the importance of the visual backbone. Open‑source rollout The authors have opened a project website patch‑policy.github.io https://patch-policy.github.io/ and a GitHub repo https://github.com/gaoyuezhou/patch policy https://github.com/gaoyuezhou/patch policy where code will be released “soon.” Supplemental videos demonstrate the robot inserting a cable, unplugging mid‑rollout, and repeating the maneuver – all with the lightweight policy. Implications for robot‑learning research Patch Policy challenges the prevailing belief that scaling model size is the primary driver of better robot control. By showing that off‑the‑shelf dense representations can substitute for massive VLMs , the work may shift investment toward better pretraining of vision backbones and more efficient policy architectures. It also lowers the entry barrier for labs without access to large compute clusters, potentially accelerating experimentation in dense‑perception robotics. Sources