ShotPlan: Cinematic Video Generation with Learnable Planning Token

1Institute of Artificial Intelligence (TeleAI), China Telecom 2Harbin Institute of Technology

*Equal contribution

ShotPlan is built on Wan2.1-T2V-14B and changes nothing about the backbone's treatment of video tokens — planning happens entirely through a handful of extra tokens.

Latents are patchified as usual; for a request like hard cuts at frames 21, 46, 64, three copies of the learnable planning embedding are appended with fractional temporal positions 6.00, 12.25, 16.75. They attend jointly with visual tokens through every DiT block and are removed before unpatchify.

Each transition type (hard cut, fade-in begin, fade-in end, camera move) owns a single learnable embedding F

. It is replicated once per requested event and concatenated with the visual token sequence — no attention masks, no architectural surgery.

The 3D VAE compresses time 4×, so integer latent positions can't express arbitrary frames. FRoPE gives each planning token a fractional temporal coordinate, letting it point between latent frames while video tokens keep standard RoPE.

After the final DiT block the planning tokens are sliced off; only visual tokens are unpatchified and decoded. The tokens exist purely to steer denoising — the output video has the exact same shape as the base model's.

tu

maps to fractional latent time with temporal stride st = 4

— e.g. frame 21 → position 6.00, frame 46 → 12.25, frame 64 → 16.75. All videos are generated at 832×480, 81 frames, from a single prompt with per-shot captions. Gold markers on each timeline show where each cut happens. Hover to play with sound off; videos loop.

Timeline legend — ▮ gold ticks: cut frames · ● teal dot: playback position · all clips are 81 frames.

Two planning tokens — fade-in begin and fade-in end — bracket the transition window, producing a smooth dissolve between shots instead of a hard cut.

A camera-movement planning token triggers a specific motion — circle, zoom, dolly, truck — at an exact frame inside one continuous take. The gold tick marks where the movement is scheduled to begin. For easier viewing, a blue box is drawn around the outer border of the video at the frame where the camera movement starts.

On the multi-shot generation benchmark, ShotPlan cuts transition deviation to well under one frame while also leading on consistency and narrative coherence.