{"slug": "codecap-bench-can-executable-code-be-a-better-video-caption-than-words", "title": "CodeCap-Bench: Can Executable Code Be a Better Video Caption Than Words?", "summary": "A new benchmark, CodeCap-Bench, from researchers at Nanjing University, Kling AI, Tianjin University, Shanghai Jiao Tong University, CASIA and Peking University tests whether executable code can caption 2D video more precisely than natural language, and finds GPT-6-astra running in Codex leads with 73.24 of 100 on fine-grained retention and 75.37 object-level spatio-temporal IoU. The benchmark covers 323 videos averaging 19.01 seconds with about 16.8 annotated objects and 8.63 focal points each, spanning app interfaces, 2D games, visual design and animation, data and scientific visualization, and abstract motion graphics. Every model scored lower on dynamic content than static appearance, with GPT-6-astra at 68.61 dynamic versus 80.72 static, and the authors report a 0.942 Pearson correlation between the GPT-6-sol model judge and human scores on 50 videos.", "body_md": "Can a program describe a video better than a sentence? **CodeCap-Bench**, a new paper trending on Hugging Face, tests exactly that: a model watches a short 2D video, writes executable code, and the code is rendered and compared against the original. The best setup in the main table, GPT-6-astra running in Codex, keeps **73.24** of 100 on the paper's fine-grained retention score, and every model loses more on motion than on appearance.\n\nThe work comes from researchers at Nanjing University, Kling AI, Tianjin University, Shanghai Jiao Tong University, CASIA and Peking University, with the [Hugging Face paper page](https://huggingface.co/papers/2610.37341?ref=explainx) and a [project page](https://codecap-bench.github.io/?ref=explainx) carrying the results. All numbers below are the authors' own; we did not rerun them.\n\n## TL;DR\n\n| Question | Answer | \n|---|---|\n| What is tested? | Writing renderable code that reproduces a 2D video | \n| Size | 323 videos, mean 19.01 seconds, about 16.8 annotated objects and 8.63 focal points each | \n| Categories | App interfaces, 2D games, visual design and animation, data and scientific visualization, abstract motion graphics | \n| Models | GPT-6-astra, GPT-6-sol, GPT-5.6-sol, Opus-5, Qwen3.8-max, Gemini-3.8-flash, DeepSeek-v4.1-flash | \n| Best result | GPT-6-astra with Codex: 73.24 overall retention, 75.37 IoU | \n| Main finding | Motion and timing are consistently harder than static appearance | \n| Caveat | Scores rely on model judges and cover only structured 2D video | \n\n## Why code as a caption?\n\nNatural-language captions are the default way to compress a video for search, summarization or training a model. They are also vague: \"a bar chart slides left while a slider moves\" does not specify positions, easing or timing. The authors argue that a program is a more precise description because it states layout with graphical primitives and motion with time-dependent rules, and because you can render it back and check the result.\n\nThat last property is the neat part. A text caption can only be judged by another model's opinion. A code caption produces a video you can compare frame by frame. It echoes a theme we covered in [Code as Worlds](https://explainx.ai/blog/code-as-worlds-executable-physical-reasoning-august-2026), where executable programs served as a grounded representation, and in [how viral Opus 5.5 videos are made with code rather than diffusion](https://explainx.ai/blog/how-opus-5-5-viral-videos-are-made-code-not-diffusion-2026).\n\n## How the benchmark is built\n\nThe authors collected structured 2D videos from Bilibili, YouTube, Behance and Vimeo, filtered out complex 3D viewpoints, detailed textures and dense pixel-level motion, and annotated each with static and dynamic focal points and object spatio-temporal tracks. Annotations were human-reviewed.\n\nGenerated code is judged on three levels:\n\n1. **Visual similarity:** cosine similarity of video features (V-JEPA 2.1) for overall consistency.\n2. **Object-level spatio-temporal IoU:** objects tracked with SAM 3.1 and compared against reference annotations for location and timing.\n3. **Fine-grained retention:** a model judge (GPT-6-sol) scores each focal point from 0 to 5, reported separately for static and dynamic content.\n\nA video also counts as a failure, scoring zero on fidelity, if the code does not produce one. The authors report a Pearson correlation of 0.942 between the model judge and human scores on 50 videos, and show that swapping in Gemini-3.1-pro-preview as the judge lowers numbers a little but preserves the ranking of the two models they checked.\n\n## The results\n\n| Model | Harness | Video success (%) | IoU | Dynamic | Static | Overall | \n|---|---|---|---|---|---|---|\n| GPT-6-astra | Codex | 99.4 | 75.37 | 68.61 | 80.72 | 73.24 | \n| GPT-6-astra | Pi Agent | 99.7 | 74.45 | 67.66 | 81.35 | 72.87 | \n| Opus-5 | Pi Agent | 97.8 | 61.45 | 55.49 | 65.10 | 58.84 | \n| Opus-5 | Claude Code | 99.4 | 58.55 | 53.23 | 63.40 | 56.88 | \n| GPT-6-sol | Codex | 100.0 | 55.63 | 43.92 | 64.04 | 51.34 | \n| Qwen3.8-max | Pi Agent | 93.2 | 48.90 | 44.81 | 55.88 | 48.83 | \n| Gemini-3.8-flash | Pi Agent | 98.8 | 42.75 | 43.66 | 55.18 | 47.73 | \n| DeepSeek-v4.1-flash | Pi Agent | 91.6 | 45.14 | 40.45 | 51.27 | 44.29 | \n| GPT-6-astra | none (single pass) | 98.5 | 69.96 | 69.00 | 76.85 | 72.00 | \n\n*Selected rows from the project page, medium thinking effort. Higher is better. The page also lists GPT-5.6-sol rows and further single-pass results.*\n\nFour things stand out.\n\n- **Success is not fidelity.** GPT-6-sol with Codex produced a video 100% of the time yet scored 51.34 overall. A program that runs is not a program that matches.\n- **Agents help success more than quality.** Agentic generation raised success rates for the paired models, but fidelity gains varied. GPT-6-astra single-pass (72.00) sits close to its agentic scores (73.24 and 72.87).\n- **Cost does not determine quality.** Per the project page, GPT-6-astra scores above Opus-5 at lower cost, while the cheaper Qwen3.8-max and DeepSeek-v4.1-flash use more tokens and take longer than the GPT series.\n- **Thinking effort helps, with limits.** More reasoning improved GPT-6-sol steadily at a rising cost, while Qwen3.8-max gained most from low to medium effort.\n\n## Motion is the weak point\n\nIn every configuration dynamic retention is below static. For the leader, that is 68.61 against 80.72. The cause is intuitive: reproducing a static layout means measuring positions once, while reproducing motion means measuring trajectories, easing and timing over many frames and encoding them as rules. The paper's case study shows an agent reconstructing a bubble-chart video by extracting motion and timing, writing code, rendering, and then fixing a slider that was one year ahead. The rendering feedback loop caught a temporal error that a single pass would have shipped.\n\nFor anyone building agents that read video, such as UI-test replayers or tutorial generators, the practical lesson matches our look at [agentic video understanding in production](https://explainx.ai/blog/agentic-video-understanding-production-results-mosaic-ponder-revyl-2026): give the model a way to render and compare, not just to describe.\n\n## Code versus words: the supplementary evidence\n\nTwo small experiments compare code captions with natural-language ones.\n\nOn question answering over 100 videos drawn from DVBench, VideoMathQA and GUI-World, Gemini-3.8-flash answered from either the code-rendered video or the caption text. Rendered code scored higher for all three caption generators: 76.42 versus 73.58 for GPT-5.6-sol, 83.02 versus 72.64 for Opus-5, and 81.13 versus 78.30 for Qwen3.8-max. Answering from the original video scored 83.96, so Opus-5's code render nearly matches it.\n\nOn reconstruction, natural-language captions were fed to a video generator (Seedance-2.5) while code was rendered directly. Appearance similarity was roughly 0.89 to 0.91 for code against 0.66 to 0.69 for language. The authors qualify this as a result under the evaluated settings, and the language path depends on that particular generator.\n\n## How to try the idea yourself\n\nYou do not need the benchmark to test the concept. A minimal loop looks like this:\n\n1. Pick a short screen recording of a UI interaction, a chart animation or a small game clip, under 30 seconds.\n2. Sample frames at a fixed rate and give them to a multimodal model with an instruction to write a self-contained HTML and CSS or canvas file that reproduces layout and timing, using no external images.\n3. Render the file headlessly, record it, and compare it to the original, either by eye or with a feature-similarity metric.\n4. Feed the diff back to the model and let it correct the worst mismatch, ideally the timing, and repeat two or three times.\n\nThe paper's own results suggest where to spend effort. Because dynamic fidelity is the weak point, give the model tools to inspect frames and measure object positions rather than asking it to guess, and prefer a harness that lets it check its rendered output. Agentic harnesses raised success rates in every paired comparison, even when fidelity gains were uneven. If your content is mostly static dashboards, a single pass may already be close to what an agent loop gives you, as the GPT-6-astra single-pass score of 72.00 shows.\n\nOne more design question is what to store. A code caption is a small text file that can be diffed, edited and re-rendered at any resolution, which makes it attractive as a memory format for UI-testing agents. The trade-off is brittleness: a caption written in code fails loudly when a primitive is wrong, while a prose caption merely becomes vague.\n\n## Limits to keep in mind\n\n- **Scope.** Only structured 2D content. Natural footage, 3D scenes and fine textures were filtered out, so this says little about captioning a movie scene.\n- **Judges.** Scoring leans on model judges, with human validation on 50 videos and four generation models.\n- **Settings.** Main results are at medium thinking effort with a fixed frame-sampling rate.\n- **Failures.** Timeouts, incomplete HTML and rendering problems hit some models, especially the open ones.\n- **Name hygiene.** The model names are as given by the authors; treat cross-vendor comparisons as one benchmark's view. For other angles on these models see our[GPT-6 Astra launch coverage](https://explainx.ai/blog/gpt-6-astra-launch-benchmarks-pricing-2026) and[Claude Opus 5.5 launch coverage](https://explainx.ai/blog/claude-opus-5-5-launch-benchmarks-pricing-2026) .\n\n## What this means for what you build\n\nIf you compress video into text for retrieval or agent memory, test whether a structured, renderable representation beats prose for your content type, at least for UIs, charts and animations. The cost is generation time and a rendering sandbox. If you generate animation with models, benchmarks like this suggest adding a render-and-compare loop, because first-pass motion is where errors hide. And if you evaluate video models, remember that tracking-based metrics such as IoU catch timing errors that overall similarity scores miss: GPT-6-sol's 0.9563 visual similarity sits beside a 55.63 IoU.\n\nRelated robotics work uses captions differently: NVIDIA's [Flora models caption, segment and reward robot video](https://explainx.ai/blog/nvidia-flora-open-models-robot-data-engine-captioner-rewarder-segmenter-gr00t-2026), a natural-language approach worth comparing.\n\n## Related reading\n\n*Scores are from the authors' project page as of October 28, 2026 and may be revised in later paper versions.*", "url": "https://wpnews.pro/news/codecap-bench-can-executable-code-be-a-better-video-caption-than-words", "canonical_source": "https://explainx.ai/blog/codecap-bench-code-as-video-caption-gpt-6-astra-codex-73-percent-opus-5-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-10-28 00:00:00+00:00", "updated_at": "2026-10-28 07:42:30.290518+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "computer-vision", "large-language-models", "generative-ai"], "entities": ["CodeCap-Bench", "Nanjing University", "Kling AI", "Tianjin University", "Shanghai Jiao Tong University", "CASIA", "Peking University", "GPT-6-astra"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/codecap-bench-can-executable-code-be-a-better-video-caption-than-words", "markdown": "https://wpnews.pro/news/codecap-bench-can-executable-code-be-a-better-video-caption-than-words.md", "text": "https://wpnews.pro/news/codecap-bench-can-executable-code-be-a-better-video-caption-than-words.txt", "jsonld": "https://wpnews.pro/news/codecap-bench-can-executable-code-be-a-better-video-caption-than-words.jsonld"}}