Robocurve finds Gemini models outperform their general AI scores on robot tasks

Robocurve reported on October 15th that Gemini models scored far above what their general AI capability rankings predicted on a set of real-robot tasks, based on a preliminary index covering five bimanual tasks and 11 model configurations. Removing Gemini from the analysis raises the reported correlation with general AI capability from 0.20 to 0.96, according to the San Francisco Public Benefit Corporation's X thread and technical report led by Drew T. Nguyen. The finding suggests general-purpose model leaderboards may not predict performance on physical tasks, though Robocurve's small task and model sample leaves the result preliminary.

Robocurve finds Gemini models outperform their general AI scores on robot tasks Its preliminary index covered five bimanual tasks and 11 model configurations; removing Gemini raises the reported correlation with general AI capability from 0.20 to 0.96. By Ryan Merket https://runtimewire.com/author/ryan-merket · Published Primary source: X https://x.com/robocurve/status/2110827995887989097 Why it matters A model's score on broad AI benchmarks may not predict its performance on physical tasks. Robocurve's pilot gives researchers a concrete reason to test that gap directly, while its small task and model sample leaves the finding preliminary. Robocurve reported that Gemini models scored far above what their general AI capability rankings predicted on a small set of real-robot tasks, an early sign that general-purpose model leaderboards may miss differences in physical performance. The San Francisco Public Benefit Corporation published the finding on October 15th in an X thread https://x.com/robocurve/status/2110827995887989097 and a technical report https://robocurve.org/rci/ led by Drew T. Nguyen.…