CodeCap-Bench: Can Executable Code Be a Better Video Caption Than Words?

Can a program describe a video better than a sentence? CodeCap-Bench, a new paper trending on Hugging Face, tests exactly that: a model watches a short 2D video, writes executable code, and the code is rendered and compared against the original. The best setup in the main table, GPT-6-astra running in Codex, keeps 73.24 of 100 on the paper's fine-grained retention score, and every model loses more on motion than on appearance.

The work comes from researchers at Nanjing University, Kling AI, Tianjin University, Shanghai Jiao Tong University, CASIA and Peking University, with the Hugging Face paper page and a project page carrying the results. All numbers below are the authors' own; we did not rerun them.

TL;DR #

Why code as a caption? #

Natural-language captions are the default way to compress a video for search, summarization or training a model. They are also vague: "a bar chart slides left while a slider moves" does not specify positions, easing or timing. The authors argue that a program is a more precise description because it states layout with graphical primitives and motion with time-dependent rules, and because you can render it back and check the result.

That last property is the neat part. A text caption can only be judged by another model's opinion. A code caption produces a video you can compare frame by frame. It echoes a theme we covered in Code as Worlds, where executable programs served as a grounded representation, and in how viral Opus 5.5 videos are made with code rather than diffusion.

How the benchmark is built #

The authors collected structured 2D videos from Bilibili, YouTube, Behance and Vimeo, filtered out complex 3D viewpoints, detailed textures and dense pixel-level motion, and annotated each with static and dynamic focal points and object spatio-temporal tracks. Annotations were human-reviewed.

Generated code is judged on three levels:

1. Visual similarity: cosine similarity of video features (V-JEPA 2.1) for overall consistency.
2. Object-level spatio-temporal IoU: objects tracked with SAM 3.1 and compared against reference annotations for location and timing.
3. Fine-grained retention: a model judge (GPT-6-sol) scores each focal point from 0 to 5, reported separately for static and dynamic content.

A video also counts as a failure, scoring zero on fidelity, if the code does not produce one. The authors report a Pearson correlation of 0.942 between the model judge and human scores on 50 videos, and show that swapping in Gemini-3.1-pro-preview as the judge lowers numbers a little but preserves the ranking of the two models they checked.

The results #

| GPT-6-astra | none (single pass) | 98.5 | 69.96 | 69.00 | 76.85 | 72.00 | Selected rows from the project page, medium thinking effort. Higher is better. The page also lists GPT-5.6-sol rows and further single-pass results.

Four things stand out.

• Success is not fidelity. GPT-6-sol with Codex produced a video 100% of the time yet scored 51.34 overall. A program that runs is not a program that matches.
• Agents help success more than quality. Agentic generation raised success rates for the paired models, but fidelity gains varied. GPT-6-astra single-pass (72.00) sits close to its agentic scores (73.24 and 72.87).
• Cost does not determine quality. Per the project page, GPT-6-astra scores above Opus-5 at lower cost, while the cheaper Qwen3.8-max and DeepSeek-v4.1-flash use more tokens and take longer than the GPT series.
• Thinking effort helps, with limits. More reasoning improved GPT-6-sol steadily at a rising cost, while Qwen3.8-max gained most from low to medium effort.

Motion is the weak point #

In every configuration dynamic retention is below static. For the leader, that is 68.61 against 80.72. The cause is intuitive: reproducing a static layout means measuring positions once, while reproducing motion means measuring trajectories, easing and timing over many frames and encoding them as rules. The paper's case study shows an agent reconstructing a bubble-chart video by extracting motion and timing, writing code, rendering, and then fixing a slider that was one year ahead. The rendering feedback loop caught a temporal error that a single pass would have shipped.

For anyone building agents that read video, such as UI-test replayers or tutorial generators, the practical lesson matches our look at agentic video understanding in production: give the model a way to render and compare, not just to describe.

Code versus words: the supplementary evidence #

Two small experiments compare code captions with natural-language ones.

On question answering over 100 videos drawn from DVBench, VideoMathQA and GUI-World, Gemini-3.8-flash answered from either the code-rendered video or the caption text. Rendered code scored higher for all three caption generators: 76.42 versus 73.58 for GPT-5.6-sol, 83.02 versus 72.64 for Opus-5, and 81.13 versus 78.30 for Qwen3.8-max. Answering from the original video scored 83.96, so Opus-5's code render nearly matches it.

On reconstruction, natural-language captions were fed to a video generator (Seedance-2.5) while code was rendered directly. Appearance similarity was roughly 0.89 to 0.91 for code against 0.66 to 0.69 for language. The authors qualify this as a result under the evaluated settings, and the language path depends on that particular generator.

How to try the idea yourself #

You do not need the benchmark to test the concept. A minimal loop looks like this:

1. Pick a short screen recording of a UI interaction, a chart animation or a small game clip, under 30 seconds.
2. Sample frames at a fixed rate and give them to a multimodal model with an instruction to write a self-contained HTML and CSS or canvas file that reproduces layout and timing, using no external images.
3. Render the file headlessly, record it, and compare it to the original, either by eye or with a feature-similarity metric.
4. Feed the diff back to the model and let it correct the worst mismatch, ideally the timing, and repeat two or three times.

The paper's own results suggest where to spend effort. Because dynamic fidelity is the weak point, give the model tools to inspect frames and measure object positions rather than asking it to guess, and prefer a harness that lets it check its rendered output. Agentic harnesses raised success rates in every paired comparison, even when fidelity gains were uneven. If your content is mostly static dashboards, a single pass may already be close to what an agent loop gives you, as the GPT-6-astra single-pass score of 72.00 shows.

One more design question is what to store. A code caption is a small text file that can be diffed, edited and re-rendered at any resolution, which makes it attractive as a memory format for UI-testing agents. The trade-off is brittleness: a caption written in code fails loudly when a primitive is wrong, while a prose caption merely becomes vague.

Limits to keep in mind #

• Scope. Only structured 2D content. Natural footage, 3D scenes and fine textures were filtered out, so this says little about captioning a movie scene.
• Judges. Scoring leans on model judges, with human validation on 50 videos and four generation models.
• Settings. Main results are at medium thinking effort with a fixed frame-sampling rate.
• Failures. Timeouts, incomplete HTML and rendering problems hit some models, especially the open ones.
• Name hygiene. The model names are as given by the authors; treat cross-vendor comparisons as one benchmark's view. For other angles on these models see ourGPT-6 Astra launch coverage andClaude Opus 5.5 launch coverage .

What this means for what you build #

If you compress video into text for retrieval or agent memory, test whether a structured, renderable representation beats prose for your content type, at least for UIs, charts and animations. The cost is generation time and a rendering sandbox. If you generate animation with models, benchmarks like this suggest adding a render-and-compare loop, because first-pass motion is where errors hide. And if you evaluate video models, remember that tracking-based metrics such as IoU catch timing errors that overall similarity scores miss: GPT-6-sol's 0.9563 visual similarity sits beside a 55.63 IoU. Related robotics work uses captions differently: NVIDIA's Flora models caption, segment and reward robot video, a natural-language approach worth comparing.

Related reading #

Scores are from the authors' project page as of October 28, 2026 and may be revised in later paper versions.