{"slug": "shotplan-cinematic-video-generation-with-learnable-planning-token", "title": "ShotPlan: Cinematic Video Generation with Learnable Planning Token", "summary": "China Telecom's Institute of Artificial Intelligence (TeleAI) and Harbin Institute of Technology have developed ShotPlan, a cinematic video generation method built on Wan2.1-T2V-14B that uses learnable planning tokens to control shot transitions with sub-frame accuracy. On a multi-shot generation benchmark, ShotPlan achieved a transition deviation of 0.64 frames, outperforming prior methods including CineTrans (4.73), EchoShot (6.95), HoloCine (2.71), and MultiShotMaster (1.12), while also leading in character consistency (0.46), scene consistency (0.37), and narrative coherence (0.88).", "body_md": "1Institute of Artificial Intelligence (TeleAI), China Telecom 2Harbin Institute of Technology\n\n*Equal contribution\n\nShotPlan is built on Wan2.1-T2V-14B and changes nothing about the backbone's treatment of video tokens — planning happens entirely through a handful of extra tokens.\n\nLatents are patchified as usual; for a request like **hard cuts at frames 21, 46, 64**, three copies of the learnable planning embedding are appended with fractional temporal positions **6.00, 12.25, 16.75**. They attend jointly with visual tokens through every DiT block and are removed before unpatchify.\n\nEach transition type (hard cut, fade-in begin, fade-in end, camera move) owns a single learnable embedding `F`\n\n. It is replicated once per requested event and concatenated with the visual token sequence — no attention masks, no architectural surgery.\n\nThe 3D VAE compresses time 4×, so integer latent positions can't express arbitrary frames. FRoPE gives each planning token a *fractional* temporal coordinate, letting it point between latent frames while video tokens keep standard RoPE.\n\nAfter the final DiT block the planning tokens are sliced off; only visual tokens are unpatchified and decoded. The tokens exist purely to steer denoising — the output video has the exact same shape as the base model's.\n\n`tu`\n\nmaps to fractional latent time with temporal stride `st = 4`\n\n— e.g. frame 21 → position 6.00, frame 46 → 12.25, frame 64 → 16.75.\nAll videos are generated at 832×480, 81 frames, from a single prompt with per-shot captions. Gold markers on each timeline show where each cut happens. Hover to play with sound off; videos loop.\n\nTimeline legend — **▮ gold ticks**: cut frames · ● teal dot: playback position · all clips are 81 frames.\n\nTwo planning tokens — fade-in begin and fade-in end — bracket the transition window, producing a smooth dissolve between shots instead of a hard cut.\n\nA camera-movement planning token triggers a specific motion — circle, zoom, dolly, truck — at an exact frame inside one continuous take. The gold tick marks where the movement is scheduled to begin. For easier viewing, a blue box is drawn around the outer border of the video at the frame where the camera movement starts.\n\nOn the multi-shot generation benchmark, ShotPlan cuts transition deviation to well under one frame while also leading on consistency and narrative coherence.\n\n| Method | Trans. Dev ↓ | Text Align ↑ | Char. Cons ↑ | Scene Cons ↑ | Narrative ↑ |\n|---|---|---|---|---|---|\n| CineTrans | 4.73 | 0.25 | 0.32 | 0.21 | 0.48 |\n| EchoShot | 6.95 | 0.26 | 0.31 | 0.23 | 0.51 |\n| HoloCine | 2.71 | 0.28 | 0.39 | 0.32 | 0.85 |\n| MultiShotMaster | 1.12 | 0.27 | 0.36 | 0.28 | 0.83 |\n| ShotPlan (Ours) | 0.64 | 0.26 | 0.46 | 0.37 | 0.88 |", "url": "https://wpnews.pro/news/shotplan-cinematic-video-generation-with-learnable-planning-token", "canonical_source": "https://pensioner-11.github.io/ShotPlan/", "published_at": "2026-07-21 04:37:58+00:00", "updated_at": "2026-07-21 04:52:50.897380+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "computer-vision"], "entities": ["China Telecom", "Institute of Artificial Intelligence (TeleAI)", "Harbin Institute of Technology", "ShotPlan", "Wan2.1-T2V-14B", "CineTrans", "EchoShot", "HoloCine"], "alternates": {"html": "https://wpnews.pro/news/shotplan-cinematic-video-generation-with-learnable-planning-token", "markdown": "https://wpnews.pro/news/shotplan-cinematic-video-generation-with-learnable-planning-token.md", "text": "https://wpnews.pro/news/shotplan-cinematic-video-generation-with-learnable-planning-token.txt", "jsonld": "https://wpnews.pro/news/shotplan-cinematic-video-generation-with-learnable-planning-token.jsonld"}}