Build a long-video to vertical-clips pipeline: metadata in, ranked 9:16 clips out

A developer detailed a pipeline for converting long videos into ranked vertical clips using AI metadata from FastPix, including chapter detection, scene changes, and transcripts. The approach uses boolean flags on upload to get time-aligned metadata and a ranking function to select clip boundaries, with the ranking logic being the core challenge.

TL;DR We'll upload a long recording with AI metadata flags on, catch the webhook, write a ranking function that picks clip boundaries from chapters + scene changes + transcript, and cut vertical clips. The ranking function is the part that matters; everything else is plumbing. Here is the naive version of this project: ffmpeg -ss 00:14:22 -to 00:14:52 -i webinar.mp4 -c copy clip.mp4 That's the cutting solved. Now watch the output: it starts four words into a sentence, opens on a hard cut to a slide, and ends mid-word. Six of those and you've built something nobody will use. The actual problem is choosing 00:14:22 . Let's build that. You'll need node 20.x or newer and ffmpeg 7.x or 8.x . Watch a human do this and they're checking three things: Three signals. Assemble them from three tools and you get three different clocks: Whisper timestamps drift against container PTS, PySceneDetect reports frame indices you convert with a frame rate that may be variable, and a tracker samples on its own interval. Joining them is where the week goes. So the design decision up front is: do you own the extraction, or get it pre-joined? I've been using FastPix for this because the AI capabilities are boolean flags on the upload call rather than a second pipeline you orchestrate. Same idea works with any provider that returns time-aligned metadata; the shape below is what to look for. curl -X POST 'https://api.fastpix.com/v1/on-demand' \ --user "$FASTPIX TOKEN ID:$FASTPIX SECRET" \ -H 'Content-Type: application/json' \ -d '{ "inputs": { "type": "video", "url": "https://cdn.example.com/webinar.mp4" } , "chapters": true, "accessPolicy": "private", "maxResolution": "1080p" }' { "success": true, "data": { "id": "a1d1acdd-8f4e-4add-b498-6b398cf349d9", "status": "Created", "createdAt": "2026-08-17T10:50:34.594302Z", "playbackIds": { "id": "6ta85f64-5717-4562-b3fc-2c963f66afa6", "accessPolicy": "private" } , "maxResolution": "1080p", "mediaQuality": "standard" } } Auth is Basic: Access Token ID as username, Secret Key as password. Playback is https://stream.fastpix.com/<playbackId .m3u8 , with a JWT appended as ?token= for private assets. ⚠️ data is an object, not an array. If you're porting from an older integration that did data 0 .id , that's your first bug.💡 Tip: chapters is one flag in a family. summary , namedEntities , moderation and subtitles are siblings on the same create call, and because they run inside the same processing pass they land referenced to the same timeline. That property is why I stopped maintaining my own extraction stack.⚠️ The API field is chapters , but the dashboard's custom media settings JSON calls it generateChapters . Two names for one feature. Same settings for a direct push upload, nested under pushMediaSettings : { "corsOrigin": " ", "pushMediaSettings": { "chapters": true, "accessPolicy": "private", "maxResolution": "1080p", "metadata": { "source": "weekly-webinar" } } } And for assets you uploaded before you thought of this, it's a PATCH rather than a re-upload: curl -X PATCH "https://api.fastpix.com/v1/on-demand/$MEDIA ID/chapters" \ --user "$FASTPIX TOKEN ID:$FASTPIX SECRET" \ -H 'Content-Type: application/json' \ -d '{ "chapters": true }' Results arrive asynchronously. Handle the lifecycle events and the AI event separately, because they fire at different times. python // webhook.js import express from 'express'; const app = express ; app.use express.json ; app.post '/webhooks/video', async req, res = { // ack fast, work later res.sendStatus 200 ; const { type, data, object } = req.body; switch type { case 'video.media.created': await db.assets.upsert { id: object.id, status: 'processing' } ; break; case 'video.media.ready': await db.assets.update object.id, { status: 'ready' } ; break; case 'video.media.failed': await db.assets.update object.id, { status: 'failed' } ; break; case 'video.mediaAI.chapters.ready': await onChapters object.id, data.chapters ; break; } } ; app.listen 3000 ; The chapters payload looks like this: { "type": "video.mediaAI.chapters.ready", "object": { "type": "media", "id": "f081fd53-6a9a-43ae-9d64-9974ef243dbd" }, "id": "51a61b56-0197-4127-8a61-472e4d3fa59a", "workspace": { "name": "clips-pipeline", "id": "f7a13f50-7f5c-48f4-b7b2-c901dcff61c6" }, "status": "ready", "data": { "isChaptersGenerated": true, "chapters": { "chapter": "1", "startTime": "00:00:00", "endTime": "00:03:59", "title": "The Circle Challenge Begins", "summary": "Contestants start stacking items in a circle for a chance to win." } }, "createdAt": "2026-08-17T11:52:29.526588692Z", "attempts": } Three field names worth pinning down, because they are easy to guess wrong: the sequence number is chapter and it is a string , the prose field is summary not description , and object.type is media even though the event is a mediaAI event. ⚠️ Times come back as hh:mm:ss strings. Convert once, at the boundary, and keep seconds internally. Mixing string times and float seconds in the same codebase is a bug generator. js const toSeconds = hms = { const h, m, s = hms.split ':' .map Number ; return h 3600 + m 60 + s; }; Providers will rank clips for you. FastPix's AI clipping produces ranked short clips scored on hook, pacing and narrative, and it's a fine candidate generator. But "which 30 seconds are worth posting" depends on your audience, not on the video, so treat vendor ranking as candidate generation and put your own opinion on top. js // rank.js const MIN LEN = 15; const MAX LEN = 60; const SHOT TOLERANCE = 0.4; // seconds export function rankCandidates { chapters, scenes, words } { const candidates = ; for const ch of chapters { const start = toSeconds ch.startTime ; const end = toSeconds ch.endTime ; // slide a window through the chapter, snapping to sentence starts for const w of words { if w.start < start || w.start end continue; if w.isSentenceStart continue; const closeWord = lastWordBefore words, w.start + MAX LEN ; const length = closeWord.end - w.start; if length < MIN LEN continue; candidates.push { start: w.start, end: closeWord.end, chapter: ch.title, score: score { start: w.start, end: closeWord.end, words, scenes } , } ; } } return dedupeOverlapping candidates.sort a, b = b.score - a.score ; } function score { start, end, words, scenes } { const inWindow = words.filter w = w.start = start && w.end <= end ; if inWindow.length return 0; const endsClean = inWindow.at -1 .endsSentence ? 1 : 0; const shotSafe = scenes.some s = Math.abs s - start < SHOT TOLERANCE ? 1 : 0; // words per second: dead air is not a clip, but neither is a firehose const density = inWindow.length / end - start ; const densityScore = density < 1.2 ? 0 : Math.min density / 3, 1.5 ; // single-speaker stretches survive a vertical crop; crosstalk does not const speakers = new Set inWindow.map w = w.speaker ; const speakerScore = speakers.size === 1 ? 1 : 0; return 2 + endsClean + shotSafe + densityScore + speakerScore; } The base 2 is for starting on a sentence, a hard filter rather than a score. Everything else is tunable. Tune it by watching output, not by reasoning about it. I got the density floor wrong twice before sitting down and watching twenty rejected candidates. Doing your own scene detection instead? FFmpeg gives you boundaries directly: ffmpeg -i webinar.mp4 -filter:v "select='gt scene,0.4 ',showinfo" \ -f null - 2 &1 | grep showinfo | sed -n 's/. pts time:\ 0-9. \ . /\1/p' 14.220 62.480 119.100 187.760 💡 The 0.4 threshold is a starting point, not a constant. Talking-head footage with a static camera needs it lower; a heavily-cut promo needs it higher. Cutting is the easy half, but there's one gotcha: fast, no re-encode, but snaps to the nearest keyframe ffmpeg -ss 14.22 -to 44.51 -i clip-source.mp4 -c copy clip.mp4 frame-accurate: -ss after -i, and you re-encode ffmpeg -i clip-source.mp4 -ss 14.22 -to 44.51 \ -c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k clip.mp4 Stream copy is fast and wrong for this job. Your ranking function worked hard to land on a sentence start, and -c copy will slide that to the nearest keyframe, which is up to a GOP away. Re-encode. Now the crop. Here's the version everybody writes first: ffmpeg -i clip.mp4 -vf "crop=ih 9/16:ih,scale=1080:1920" -c:a copy vertical.mp4 That's a centre crop, and it's correct exactly when your subject is centred. On a two-person interview it produces a beautiful vertical clip of the gap between them. On a stage recording where the speaker walks, it produces a lectern. Subject-tracking reframe is the fix, and it's real work to build: detect the subject per frame, smooth the crop path so it doesn't jitter, reset on cuts rather than panning across them. FastPix's auto-reframe does 16:9 to 9:16, 1:1 or 4:5 with subject tracking rather than centre-cropping, which is what made the output shippable instead of something I had to eyeball every time. Know which one you need: | Footage | Centre crop | Subject tracking | |---|---|---| | Single centred speaker, static camera | Fine | Overkill | | Screen share / slides | Fine or use 1:1 | No help | | Two-person interview | Fails | Required | | Stage recording, speaker moves | Fails | Required | Billing is per minute, encoding is free on the standard plan you pay for delivery, storage and add-ons , and signup gives you $25 in credits with no credit card for the free tier. Early-stage teams can get $600 through the Startup Program under four years old, under $10M raised , which covers running this over a real back catalogue while you decide. One honest caveat: this is API-first, not a no-code CMS. If what your team actually wants is a timeline UI where a marketer picks moments by hand, you're building that front end. That was fine here because removing the human from the loop was the point. subtitles filter away, and it's what drives silent-autoplay watch time.The documented AI flags chapters , summary , namedEntities , moderation , subtitles and their webhook payloads are in the In-Video AI docs https://fastpix.com/docs/in-video-ai/overview ; the clipping and reframe capabilities are on the product side, so check the current reference for their parameter names before you wire them in. The upload-and-webhook pattern here transfers to any managed video API, so that integration work isn't wasted if you switch. What varies a lot between providers is the AI capability set itself, so check yours against your actual list before you commit.