GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

A new arXiv paper (2611.13028v1) introduces GTAGym, an environment and framework for evaluating and training AI agents on Grand Theft Auto V tasks. Its benchmark, GTAVBench, contains 103 online tasks across nine task families plus 47 image-based shooting and sniping tasks, and the best of six frontier LLMs tested reached 70.9% overall accuracy on the GTAVBench-AGI branch, where instructions appear only in the first screenshot. LoRA reinforcement learning of Qwen3.8-27B on generated on-foot navigation tasks improved in-domain performance and transferred out-of-domain, raising native GTA V task performance from 15% to 23%.

arXiv:2611.13028v1 Announce Type: new Abstract: AAA games offer a rich testbed for evaluating agent perception, reasoning, and control. However, evaluating whether agents can complete tasks in AAA games is challenging and often human-annotated because task outcomes are often difficult to verify automatically, let alone applying it for Reinforcement Learning RL . We introduce GTAGym, an environment and framework for evaluation and RL on GTA V tasks. Its evaluation benchmark, GTAVBench, derived from GTA V main-story missions, contains 103 online tasks reconstructed across nine task families, alongside 47 image-based shooting and sniping tasks using GTA V imagery. A web server provides access to the online environment and evaluation interface, supporting parallel testing through screenshots and keyboard-mouse actions with automatic engine-state outcome checks. We evaluate six frontier LLMs on GTAVBench with textual task instructions and on its GTAVBench-AGI branch, where instructions appear only in the first screenshot and a harness automatically compresses interaction history. The best configuration achieves 70.9% overall accuracy in GTAVBench-AGI, yet aerial navigation, evasion and interception remain difficult. GTAGym also provides a scalable generator that samples task parameters within user-specified ranges and option sets under fixed templates, creating batches of 3D and image-based shooting environments for RL with automatic reward and termination signals. LoRA RL of Qwen3.8-27B on generated on-foot navigation tasks improves the in-domain performance and transfers to out-of-domain task categories, and native GTA V tasks from 15% to 23%.