Build a local Python pipeline that turns a long talk into six 9:16 Shorts

A developer published a local Python pipeline that converts long talks, podcasts and interviews into six vertical 9:16 Shorts, combining faster-whisper word-level transcription, a heuristic or LLM-assisted clip scorer, OpenCV face tracking and FFmpeg rendering. The pipeline scores candidate clips on seven checks, trims silence, burns in word-by-word captions and hook titles, and supports an --analyze mode that exports a timestamped transcript for an LLM to select clips. The writeup notes CPU-only runs with the base Whisper model are sufficient for a first pass and warns that only media the user has rights to should be processed.

If you cut Shorts from podcasts or interviews, most of the clock time is not creative. It is the same checklist: find a 30-second stretch that stands on its own, drop the dead air, reframe to 9:16, put the face in the middle, burn in captions, add a hook title, maybe a zoom and a sound on the punch word. That checklist is regular enough to automate on your own machine. This article walks through a local Python pipeline that does that. No cloud editor, no monthly seat. The blocks are ordinary: faster-whisper for a word-timed transcript, a small scorer or an LLM that only sees text to pick clips, OpenCV to track a face, and FFmpeg to render. I will be honest about what this replaces and what it does not. A useful default is six clips, each 20–60 seconds 30–45 is the sweet spot , already in 9:16, with silence trimmed, a vertical frame, word-by-word captions, a hook title in the first couple of seconds, light zooms, optional B-roll and sound effects, a progress bar, and a clean end that can loop. Support two inputs, because the media is different: The rest of this article is talk mode. Lifestyle is a second renderer on the same command, not a second architecture. A hard rule: only process video you or your client have the rights to. yt-dlp can fetch a URL, but downloading someone else's upload and reposting it breaks copyright and platform rules, automation or not. Keep one entry point: heuristic selection, then render six Shorts python clip.py podcast.mp4 --lang en German audio, more accurate captions slower python clip.py interview.mp4 --lang de --model medium export a transcript for an LLM to pick clips, then render that file python clip.py podcast.mp4 --analyze python clip.py podcast.mp4 --clips work/podcast/clips.json --only 2 5 Useful flags: --fast FFmpeg veryfast , --no-sfx , --no-broll , --music default for a quiet bed with ducking, and --crop for a full-bleed vertical crop. Cropping is fashionable; it also chops hands and slides. A safer default is to fit the whole landscape frame onto 9:16 and use a blurred copy of the same frame as background. You need Python 3.12, FFmpeg and ffprobe on PATH , faster-whisper , OpenCV, NumPy, Pillow, SciPy, soundfile , and librosa . A GPU helps Whisper, but a CPU-only run with --model base and --fast is enough for a first pass. Captions and clip boundaries both need word times, not paragraph times. faster-whisper the CTranslate2 port of Whisper writes {word, start, end} locally, including German with --lang de . From that list you can group sentences on punctuation, treat long gaps as silence, and later highlight the active caption word. Write work files into work/<slug / . A replayable transcript lets you re-render clip 3 without transcribing again. A Short has to survive without the rest of the episode. Score seven checks from 0–10 and skip anything with a mean under 6: Start on the first word of the hook sentence. End on the last word of the payoff keep the laugh if it comes right after . Do not let clips overlap. A first version does not need an LLM. Slide a sentence window across the 20–60 s range. Boost hook words in the first sentence, questions, and emotion or controversy terms English and German . Penalize openings that start mid-thought And , So , Und , Also . Prefer a clean sentence end and a duration near 38 s. Measure RMS energy with FFmpeg f32le mono at 8 kHz and reward variance, plus a mild words-per-second bonus. Sort by score and greedily keep non-overlapping winners. That already beats "cut every 45 seconds from the start." It still misses quiet jokes and can over-rank a segment that merely contains the word "secret." The better, still cheap path is --analyze : export a timestamped sentence list plus a short briefing, and let Claude or ChatGPT apply the seven checks and write a clips.json start , end , a hook title of at most six words, keywords to highlight, optional speaker name, optional B-roll search terms . The model never sees the pixels, only text. A 20-minute episode is roughly 5,000–8,000 tokens. Then render exactly that file. For each output frame, detect the face OpenCV's YuNet model is small and MIT-licensed , aim a 9:16 window at it, and smooth the window over time . Without a low-pass, every head nod becomes a camera bump. Two framing policies: If two faces are in frame, prefer a window where one person is clearly speaking, or start from a file that is already vertical. Once you have start, end and the word list, the renderer is bookkeeping: cut the source, drop pauses above a threshold around 0.35 s works , overlay word-by-word captions with the current word emphasized, show a hook title for about 2.5 seconds, punch-in zoom on three to eight keywords, optionally lay B-roll on those hits, add light SFX, duck a music bed, and draw a progress bar with a last-frame hold so the clip can loop. For B-roll, a local folder named by keyword is the cleanest option. If you fall back to a stock API, write a credits file and put required CC BY attribution in the description. Generating ASS or title/drawtext filters from the word timestamps is more reliable than typography frame-by-frame in Python, except for motion-graphics layers you pre-render with Pillow. Watch every file before you post. The pipeline will caption a mis-heard name, start a cut on a breath, or zoom on the wrong word. A two-minute review saves a bad upload. Lifestyle mode, briefly: skip Whisper, detect the beat with librosa , lay clips on the downbeats, and add ramps plus flash, glitch, zoom, or whip-pan transitions. This will not beat a careful CapCut or After Effects pass on tracking and music feel. Treat it as a rough cut. Use client-cleared or your own recordings. Running locally means no subscription for the core path. Optional API keys Pexels, Pixabay, or an LLM are extras. Budget about fifteen minutes on Windows for Python, FFmpeg, a venv, and pip install . Linux and macOS should work with Python 3.12 and FFmpeg; I have only treated Windows as the tested baseline. A pipeline like this can take over something like 70–80% of the repetitive cutting. It does not replace a good editor, and it will not invent a hook that is not in the tape. The win is that you spend your time on selection and a last pass, not on clicking the same five effects in a timeline. I packaged this exact flow as a paid kit the Auto-Clipping-System : the Python source, a German step-by-step guide, a Windows installer script, the clip-scoring rules, prompts for Claude, bundled CC0 sounds, OFL fonts, and demo clips. It is 29 € once, no subscription. If you want the boxed version instead of wiring the stages yourself, it is here: https://whop.com/kelvaro-creator-kits/auto-clipping-system https://whop.com/kelvaro-creator-kits/auto-clipping-system Disclosure: that kit is my own paid product. No guarantee of views or revenue. Please preview every clip before you publish it.