ShotPlan: Cinematic Video Generation with Learnable Planning Token

China Telecom's Institute of Artificial Intelligence (TeleAI) and Harbin Institute of Technology have developed ShotPlan, a cinematic video generation method built on Wan2.1-T2V-14B that uses learnable planning tokens to control shot transitions with sub-frame accuracy. On a multi-shot generation benchmark, ShotPlan achieved a transition deviation of 0.64 frames, outperforming prior methods including CineTrans (4.73), EchoShot (6.95), HoloCine (2.71), and MultiShotMaster (1.12), while also leading in character consistency (0.46), scene consistency (0.37), and narrative coherence (0.88).

1Institute of Artificial Intelligence TeleAI , China Telecom 2Harbin Institute of Technology Equal contribution ShotPlan is built on Wan2.1-T2V-14B and changes nothing about the backbone's treatment of video tokens — planning happens entirely through a handful of extra tokens. Latents are patchified as usual; for a request like hard cuts at frames 21, 46, 64 , three copies of the learnable planning embedding are appended with fractional temporal positions 6.00, 12.25, 16.75 . They attend jointly with visual tokens through every DiT block and are removed before unpatchify. Each transition type hard cut, fade-in begin, fade-in end, camera move owns a single learnable embedding F . It is replicated once per requested event and concatenated with the visual token sequence — no attention masks, no architectural surgery. The 3D VAE compresses time 4×, so integer latent positions can't express arbitrary frames. FRoPE gives each planning token a fractional temporal coordinate, letting it point between latent frames while video tokens keep standard RoPE. After the final DiT block the planning tokens are sliced off; only visual tokens are unpatchified and decoded. The tokens exist purely to steer denoising — the output video has the exact same shape as the base model's. tu maps to fractional latent time with temporal stride st = 4 — e.g. frame 21 → position 6.00, frame 46 → 12.25, frame 64 → 16.75. All videos are generated at 832×480, 81 frames, from a single prompt with per-shot captions. Gold markers on each timeline show where each cut happens. Hover to play with sound off; videos loop. Timeline legend — ▮ gold ticks : cut frames · ● teal dot: playback position · all clips are 81 frames. Two planning tokens — fade-in begin and fade-in end — bracket the transition window, producing a smooth dissolve between shots instead of a hard cut. A camera-movement planning token triggers a specific motion — circle, zoom, dolly, truck — at an exact frame inside one continuous take. The gold tick marks where the movement is scheduled to begin. For easier viewing, a blue box is drawn around the outer border of the video at the frame where the camera movement starts. On the multi-shot generation benchmark, ShotPlan cuts transition deviation to well under one frame while also leading on consistency and narrative coherence. | Method | Trans. Dev ↓ | Text Align ↑ | Char. Cons ↑ | Scene Cons ↑ | Narrative ↑ | |---|---|---|---|---|---| | CineTrans | 4.73 | 0.25 | 0.32 | 0.21 | 0.48 | | EchoShot | 6.95 | 0.26 | 0.31 | 0.23 | 0.51 | | HoloCine | 2.71 | 0.28 | 0.39 | 0.32 | 0.85 | | MultiShotMaster | 1.12 | 0.27 | 0.36 | 0.28 | 0.83 | | ShotPlan Ours | 0.64 | 0.26 | 0.46 | 0.37 | 0.88 |