{"slug": "build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out", "title": "Build a long-video to vertical-clips pipeline: metadata in, ranked 9:16 clips out", "summary": "A developer detailed a pipeline for converting long videos into ranked vertical clips using AI metadata from FastPix, including chapter detection, scene changes, and transcripts. The approach uses boolean flags on upload to get time-aligned metadata and a ranking function to select clip boundaries, with the ranking logic being the core challenge.", "body_md": "## TL;DR\n\nWe'll upload a long recording with AI metadata flags on, catch the webhook, write a ranking function that picks clip boundaries from chapters + scene changes + transcript, and cut vertical clips. The ranking function is the part that matters; everything else is plumbing.\n\nHere is the naive version of this project:\n\n```\nffmpeg -ss 00:14:22 -to 00:14:52 -i webinar.mp4 -c copy clip.mp4\n```\n\nThat's the cutting solved. Now watch the output: it starts four words into a sentence, opens on a hard cut to a slide, and ends mid-word. Six of those and you've built something nobody will use.\n\nThe actual problem is choosing `00:14:22`\n\n. Let's build that. You'll need `node 20.x or newer`\n\nand `ffmpeg 7.x or 8.x`\n\n.\n\nWatch a human do this and they're checking three things:\n\nThree signals. Assemble them from three tools and you get three different clocks: Whisper timestamps drift against container PTS, PySceneDetect reports frame indices you convert with a frame rate that may be variable, and a tracker samples on its own interval. Joining them is where the week goes.\n\nSo the design decision up front is: do you own the extraction, or get it pre-joined?\n\nI've been using FastPix for this because the AI capabilities are boolean flags on the upload call rather than a second pipeline you orchestrate. Same idea works with any provider that returns time-aligned metadata; the shape below is what to look for.\n\n```\ncurl -X POST 'https://api.fastpix.com/v1/on-demand' \\\n  --user \"$FASTPIX_TOKEN_ID:$FASTPIX_SECRET\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"inputs\": [\n      { \"type\": \"video\", \"url\": \"https://cdn.example.com/webinar.mp4\" }\n    ],\n    \"chapters\": true,\n    \"accessPolicy\": \"private\",\n    \"maxResolution\": \"1080p\"\n  }'\n{\n  \"success\": true,\n  \"data\": {\n    \"id\": \"a1d1acdd-8f4e-4add-b498-6b398cf349d9\",\n    \"status\": \"Created\",\n    \"createdAt\": \"2026-08-17T10:50:34.594302Z\",\n    \"playbackIds\": [\n      { \"id\": \"6ta85f64-5717-4562-b3fc-2c963f66afa6\", \"accessPolicy\": \"private\" }\n    ],\n    \"maxResolution\": \"1080p\",\n    \"mediaQuality\": \"standard\"\n  }\n}\n```\n\nAuth is Basic: Access Token ID as username, Secret Key as password. Playback is `https://stream.fastpix.com/<playbackId>.m3u8`\n\n, with a JWT appended as `?token=`\n\nfor private assets.\n\n⚠️\n\n`data`\n\nis an object, not an array. If you're porting from an older integration that did`data[0].id`\n\n, that's your first bug.💡 Tip:\n\n`chapters`\n\nis one flag in a family.`summary`\n\n,`namedEntities`\n\n,`moderation`\n\nand`subtitles`\n\nare siblings on the same create call, and because they run inside the same processing pass they land referenced to the same timeline. That property is why I stopped maintaining my own extraction stack.⚠️ The API field is\n\n`chapters`\n\n, but the dashboard's custom media settings JSON calls it`generateChapters`\n\n. Two names for one feature.\n\nSame settings for a direct (push) upload, nested under `pushMediaSettings`\n\n:\n\n```\n{\n  \"corsOrigin\": \"*\",\n  \"pushMediaSettings\": {\n    \"chapters\": true,\n    \"accessPolicy\": \"private\",\n    \"maxResolution\": \"1080p\",\n    \"metadata\": { \"source\": \"weekly-webinar\" }\n  }\n}\n```\n\nAnd for assets you uploaded before you thought of this, it's a PATCH rather than a re-upload:\n\n```\ncurl -X PATCH \"https://api.fastpix.com/v1/on-demand/$MEDIA_ID/chapters\" \\\n  --user \"$FASTPIX_TOKEN_ID:$FASTPIX_SECRET\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{ \"chapters\": true }'\n```\n\nResults arrive asynchronously. Handle the lifecycle events and the AI event separately, because they fire at different times.\n\n``` python\n// webhook.js\nimport express from 'express';\n\nconst app = express();\napp.use(express.json());\n\napp.post('/webhooks/video', async (req, res) => {\n  // ack fast, work later\n  res.sendStatus(200);\n\n  const { type, data, object } = req.body;\n\n  switch (type) {\n    case 'video.media.created':\n      await db.assets.upsert({ id: object.id, status: 'processing' });\n      break;\n\n    case 'video.media.ready':\n      await db.assets.update(object.id, { status: 'ready' });\n      break;\n\n    case 'video.media.failed':\n      await db.assets.update(object.id, { status: 'failed' });\n      break;\n\n    case 'video.mediaAI.chapters.ready':\n      await onChapters(object.id, data.chapters);\n      break;\n  }\n});\n\napp.listen(3000);\n```\n\nThe chapters payload looks like this:\n\n```\n{\n  \"type\": \"video.mediaAI.chapters.ready\",\n  \"object\": { \"type\": \"media\", \"id\": \"f081fd53-6a9a-43ae-9d64-9974ef243dbd\" },\n  \"id\": \"51a61b56-0197-4127-8a61-472e4d3fa59a\",\n  \"workspace\": { \"name\": \"clips-pipeline\", \"id\": \"f7a13f50-7f5c-48f4-b7b2-c901dcff61c6\" },\n  \"status\": \"ready\",\n  \"data\": {\n    \"isChaptersGenerated\": true,\n    \"chapters\": [\n      {\n        \"chapter\": \"1\",\n        \"startTime\": \"00:00:00\",\n        \"endTime\": \"00:03:59\",\n        \"title\": \"The Circle Challenge Begins\",\n        \"summary\": \"Contestants start stacking items in a circle for a chance to win.\"\n      }\n    ]\n  },\n  \"createdAt\": \"2026-08-17T11:52:29.526588692Z\",\n  \"attempts\": []\n}\n```\n\nThree field names worth pinning down, because they are easy to guess wrong: the sequence number is `chapter`\n\nand it is a **string**, the prose field is `summary`\n\n(not `description`\n\n), and `object.type`\n\nis `media`\n\neven though the event is a `mediaAI`\n\nevent.\n\n⚠️ Times come back as\n\n`hh:mm:ss`\n\nstrings. Convert once, at the boundary, and keep seconds internally. Mixing string times and float seconds in the same codebase is a bug generator.\n\n``` js\nconst toSeconds = (hms) => {\n  const [h, m, s] = hms.split(':').map(Number);\n  return h * 3600 + m * 60 + s;\n};\n```\n\nProviders will rank clips for you. FastPix's AI clipping produces ranked short clips scored on hook, pacing and narrative, and it's a fine candidate generator. But \"which 30 seconds are worth posting\" depends on your audience, not on the video, so treat vendor ranking as candidate generation and put your own opinion on top.\n\n``` js\n// rank.js\nconst MIN_LEN = 15;\nconst MAX_LEN = 60;\nconst SHOT_TOLERANCE = 0.4; // seconds\n\nexport function rankCandidates({ chapters, scenes, words }) {\n  const candidates = [];\n\n  for (const ch of chapters) {\n    const start = toSeconds(ch.startTime);\n    const end = toSeconds(ch.endTime);\n\n    // slide a window through the chapter, snapping to sentence starts\n    for (const w of words) {\n      if (w.start < start || w.start > end) continue;\n      if (!w.isSentenceStart) continue;\n\n      const closeWord = lastWordBefore(words, w.start + MAX_LEN);\n      const length = closeWord.end - w.start;\n      if (length < MIN_LEN) continue;\n\n      candidates.push({\n        start: w.start,\n        end: closeWord.end,\n        chapter: ch.title,\n        score: score({ start: w.start, end: closeWord.end, words, scenes }),\n      });\n    }\n  }\n\n  return dedupeOverlapping(candidates.sort((a, b) => b.score - a.score));\n}\n\nfunction score({ start, end, words, scenes }) {\n  const inWindow = words.filter((w) => w.start >= start && w.end <= end);\n  if (!inWindow.length) return 0;\n\n  const endsClean = inWindow.at(-1).endsSentence ? 1 : 0;\n  const shotSafe = scenes.some((s) => Math.abs(s - start) < SHOT_TOLERANCE) ? 1 : 0;\n\n  // words per second: dead air is not a clip, but neither is a firehose\n  const density = inWindow.length / (end - start);\n  const densityScore = density < 1.2 ? 0 : Math.min(density / 3, 1.5);\n\n  // single-speaker stretches survive a vertical crop; crosstalk does not\n  const speakers = new Set(inWindow.map((w) => w.speaker));\n  const speakerScore = speakers.size === 1 ? 1 : 0;\n\n  return 2 + endsClean + shotSafe + densityScore + speakerScore;\n}\n```\n\nThe base `2`\n\nis for starting on a sentence, a hard filter rather than a score. Everything else is tunable. **Tune it by watching output, not by reasoning about it.** I got the density floor wrong twice before sitting down and watching twenty rejected candidates.\n\nDoing your own scene detection instead? FFmpeg gives you boundaries directly:\n\n```\nffmpeg -i webinar.mp4 -filter:v \"select='gt(scene,0.4)',showinfo\" \\\n  -f null - 2>&1 | grep showinfo | sed -n 's/.*pts_time:\\([0-9.]*\\).*/\\1/p'\n14.220\n62.480\n119.100\n187.760\n```\n\n💡 The\n\n`0.4`\n\nthreshold is a starting point, not a constant. Talking-head footage with a static camera needs it lower; a heavily-cut promo needs it higher.\n\nCutting is the easy half, but there's one gotcha:\n\n```\n# fast, no re-encode, but snaps to the nearest keyframe\nffmpeg -ss 14.22 -to 44.51 -i clip-source.mp4 -c copy clip.mp4\n\n# frame-accurate: -ss after -i, and you re-encode\nffmpeg -i clip-source.mp4 -ss 14.22 -to 44.51 \\\n  -c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k clip.mp4\n```\n\nStream copy is fast and wrong for this job. Your ranking function worked hard to land on a sentence start, and `-c copy`\n\nwill slide that to the nearest keyframe, which is up to a GOP away. Re-encode.\n\nNow the crop. Here's the version everybody writes first:\n\n```\nffmpeg -i clip.mp4 -vf \"crop=ih*9/16:ih,scale=1080:1920\" -c:a copy vertical.mp4\n```\n\nThat's a centre crop, and it's correct exactly when your subject is centred. On a two-person interview it produces a beautiful vertical clip of the gap between them. On a stage recording where the speaker walks, it produces a lectern.\n\nSubject-tracking reframe is the fix, and it's real work to build: detect the subject per frame, smooth the crop path so it doesn't jitter, reset on cuts rather than panning across them. FastPix's auto-reframe does 16:9 to 9:16, 1:1 or 4:5 with subject tracking rather than centre-cropping, which is what made the output shippable instead of something I had to eyeball every time.\n\nKnow which one you need:\n\n| Footage | Centre crop | Subject tracking |\n|---|---|---|\n| Single centred speaker, static camera | Fine | Overkill |\n| Screen share / slides | Fine (or use 1:1) | No help |\n| Two-person interview | Fails | Required |\n| Stage recording, speaker moves | Fails | Required |\n\nBilling is per minute, encoding is free on the standard plan (you pay for delivery, storage and add-ons), and signup gives you $25 in credits with no credit card for the free tier. Early-stage teams can get $600 through the Startup Program (under four years old, under $10M raised), which covers running this over a real back catalogue while you decide.\n\nOne honest caveat: this is API-first, not a no-code CMS. If what your team actually wants is a timeline UI where a marketer picks moments by hand, you're building that front end. That was fine here because removing the human from the loop was the point.\n\n`subtitles`\n\nfilter away, and it's what drives silent-autoplay watch time.The documented AI flags (`chapters`\n\n, `summary`\n\n, `namedEntities`\n\n, `moderation`\n\n, `subtitles`\n\n) and their webhook payloads are in the [In-Video AI docs](https://fastpix.com/docs/in-video-ai/overview); the clipping and reframe capabilities are on the product side, so check the current reference for their parameter names before you wire them in. The upload-and-webhook pattern here transfers to any managed video API, so that integration work isn't wasted if you switch. What varies a lot between providers is the AI capability set itself, so check yours against your actual list before you commit.", "url": "https://wpnews.pro/news/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out", "canonical_source": "https://dev.to/masonwritescode/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-916-clips-out-epo", "published_at": "2026-08-17 09:00:43+00:00", "updated_at": "2026-08-17 09:12:59.829880+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning"], "entities": ["FastPix", "Whisper", "PySceneDetect", "ffmpeg", "Node.js"], "alternates": {"html": "https://wpnews.pro/news/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out", "markdown": "https://wpnews.pro/news/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out.md", "text": "https://wpnews.pro/news/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out.txt", "jsonld": "https://wpnews.pro/news/build-a-long-video-to-vertical-clips-pipeline-metadata-in-ranked-9-16-clips-out.jsonld"}}