If you cut Shorts from podcasts or interviews, most of the clock time is not creative. It is the same checklist: find a 30-second stretch that stands on its own, drop the dead air, reframe to 9:16, put the face in the middle, burn in captions, add a hook title, maybe a zoom and a sound on the punch word. That checklist is regular enough to automate on your own machine.
This article walks through a local Python pipeline that does that. No cloud editor, no monthly seat. The blocks are ordinary: faster-whisper for a word-timed transcript, a small scorer (or an LLM that only sees text) to pick clips, OpenCV to track a face, and FFmpeg to render. I will be honest about what this replaces and what it does not.
A useful default is six clips, each 20–60 seconds (30–45 is the sweet spot), already in 9:16, with silence trimmed, a vertical frame, word-by-word captions, a hook title in the first couple of seconds, light zooms, optional B-roll and sound effects, a progress bar, and a clean end that can loop.
Support two inputs, because the media is different:
The rest of this article is talk mode. Lifestyle is a second renderer on the same command, not a second architecture.
A hard rule: only process video you or your client have the rights to. yt-dlp can fetch a URL, but down someone else's upload and reposting it breaks copyright and platform rules, automation or not.
Keep one entry point:
python clip.py podcast.mp4 --lang en
python clip.py interview.mp4 --lang de --model medium
python clip.py podcast.mp4 --analyze
python clip.py podcast.mp4 --clips work/podcast/clips.json --only 2 5
Useful flags: --fast (FFmpeg veryfast), --no-sfx, --no-broll, --music default for a quiet bed with ducking, and --crop for a full-bleed vertical crop. Cropping is fashionable; it also chops hands and slides. A safer default is to fit the whole landscape frame onto 9:16 and use a blurred copy of the same frame as background.
You need Python 3.12, FFmpeg and ffprobe on PATH, faster-whisper, OpenCV, NumPy, Pillow, SciPy, soundfile, and librosa. A GPU helps Whisper, but a CPU-only run with --model base and --fast is enough for a first pass.
Captions and clip boundaries both need word times, not paragraph times. faster-whisper (the CTranslate2 port of Whisper) writes {word, start, end} locally, including German with --lang de. From that list you can group sentences on punctuation, treat long gaps as silence, and later highlight the active caption word.
Write work files into work/<slug>/. A replayable transcript lets you re-render clip 3 without transcribing again.
A Short has to survive without the rest of the episode. Score seven checks from 0–10 and skip anything with a mean under 6:
Start on the first word of the hook sentence. End on the last word of the payoff (keep the laugh if it comes right after). Do not let clips overlap.
A first version does not need an LLM. Slide a sentence window across the 20–60 s range. Boost hook words in the first sentence, questions, and emotion or controversy terms (English and German). Penalize openings that start mid-thought (And, So, Und, Also). Prefer a clean sentence end and a duration near 38 s. Measure RMS energy with FFmpeg (f32le mono at 8 kHz) and reward variance, plus a mild words-per-second bonus. Sort by score and greedily keep non-overlapping winners.
That already beats "cut every 45 seconds from the start." It still misses quiet jokes and can over-rank a segment that merely contains the word "secret."
The better, still cheap path is --analyze: export a timestamped sentence list plus a short briefing, and let Claude or ChatGPT apply the seven checks and write a clips.json (start, end, a hook title of at most six words, keywords to highlight, optional speaker name, optional B-roll search terms). The model never sees the pixels, only text. A 20-minute episode is roughly 5,000–8,000 tokens. Then render exactly that file.
For each output frame, detect the face (OpenCV's YuNet model is small and MIT-licensed), aim a 9:16 window at it, and smooth the window over time. Without a low-pass, every head nod becomes a camera bump.
Two framing policies:
If two faces are in frame, prefer a window where one person is clearly speaking, or start from a file that is already vertical.
Once you have (start, end) and the word list, the renderer is bookkeeping: cut the source, drop s above a threshold (around 0.35 s works), overlay word-by-word captions with the current word emphasized, show a hook title for about 2.5 seconds, punch-in zoom on three to eight keywords, optionally lay B-roll on those hits, add light SFX, duck a music bed, and draw a progress bar with a last-frame hold so the clip can loop.
For B-roll, a local folder named by keyword is the cleanest option. If you fall back to a stock API, write a credits file and put required CC BY attribution in the description.
Generating ASS or title/drawtext filters from the word timestamps is more reliable than typography frame-by-frame in Python, except for motion-graphics layers you pre-render with Pillow.
Watch every file before you post. The pipeline will caption a mis-heard name, start a cut on a breath, or zoom on the wrong word. A two-minute review saves a bad upload.
Lifestyle mode, briefly: skip Whisper, detect the beat with librosa, lay clips on the downbeats, and add ramps plus flash, glitch, zoom, or whip-pan transitions. This will not beat a careful CapCut or After Effects pass on tracking and music feel. Treat it as a rough cut.
Use client-cleared or your own recordings. Running locally means no subscription for the core path. Optional API keys (Pexels, Pixabay, or an LLM) are extras. Budget about fifteen minutes on Windows for Python, FFmpeg, a venv, and pip install. Linux and macOS should work with Python 3.12 and FFmpeg; I have only treated Windows as the tested baseline.
A pipeline like this can take over something like 70–80% of the repetitive cutting. It does not replace a good editor, and it will not invent a hook that is not in the tape. The win is that you spend your time on selection and a last pass, not on clicking the same five effects in a timeline.
I packaged this exact flow as a paid kit (the Auto-Clipping-System): the Python source, a German step-by-step guide, a Windows installer script, the clip-scoring rules, prompts for Claude, bundled CC0 sounds, OFL fonts, and demo clips. It is 29 € once, no subscription. If you want the boxed version instead of wiring the stages yourself, it is here: https://whop.com/kelvaro-creator-kits/auto-clipping-system
Disclosure: that kit is my own paid product. No guarantee of views or revenue. Please preview every clip before you publish it.