Video-Coder: Reinforcing Agentic Coding for Video Generation

A paper submitted to arXiv on 6 Nov 2026 introduces Video-Coder, described by its authors as the first attempt to train a code-driven video generation agent that autonomously writes and executes code to build a dynamic 3D scene, called a Draft Video, supplying structure and motion signals to downstream video generators. The authors trained Video-Coder with supervised fine-tuning followed by harness-native agentic reinforcement learning using dual execution-based and video-based rewards, with an annealing strategy that gradually reduces the execution reward so the model relies more on the video reward later in training, and they released the Video-Coder-SFT-6k and Video-Coder-RL-2k datasets plus the Video-Coder Harness environment. Experiments show Video-Coder improves Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0, and the authors state they fully open-source their data, models, and code.

Computer Science Computer Vision and Pattern Recognition Submitted on 6 Nov 2026 Title:Video-Coder: Reinforcing Agentic Coding for Video Generation View PDF http://arxiv.org/pdf/2611.09894v1 HTML experimental https://arxiv.org/html/2611.09894v1 Abstract:Recent years have witnessed remarkable progress in video generation in terms of visual quality and temporal consistency. However, generating videos with precise spatial structure, coherent motion dynamics, and controllable scene composition remains challenging. In this paper, we introduce Video-Coder, the first attempt to train a code-driven video generation agent, which autonomously writes and executes code to construct a dynamic 3D scene Draft Video that provides structure and motion signals to downstream video generators. To support this paradigm, we design a dedicated data pipeline and curate two high-quality datasets, Video-Coder-SFT-6k and Video-Coder-RL-2k, which contain diverse prompts and ground-truth reference videos. We further develop the Video- Coder Harness, a unified interactive environment integrating code execution, 3D scene rendering, and execution feedback for agent training and inference. We then train Video-Coder with SFT fol- lowed by harness-native agentic reinforcement learning with dual reward feedback, which combines execution-based and video-based rewards. Additionally, we design an annealing training strategy that gradually reduces the execution reward over time, allowing the model to rely more on the video reward in the later training stages. Experiments show that Video-Coder achieves significant gains, improving Minimax-H3 by approximately 7 points on PhyGenBench and 24 points on VBench-2.0. We hope this work can serve as an open foundation for coding agents in video generation, and we fully open-source our data, models, and code Additional Features References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .