TL;DR #
We'll upload a long recording with AI metadata flags on, catch the webhook, write a ranking function that picks clip boundaries from chapters + scene changes + transcript, and cut vertical clips. The ranking function is the part that matters; everything else is plumbing.
Here is the naive version of this project:
ffmpeg -ss 00:14:22 -to 00:14:52 -i webinar.mp4 -c copy clip.mp4
That's the cutting solved. Now watch the output: it starts four words into a sentence, opens on a hard cut to a slide, and ends mid-word. Six of those and you've built something nobody will use.
The actual problem is choosing 00:14:22
. Let's build that. You'll need node 20.x or newer
and ffmpeg 7.x or 8.x
.
Watch a human do this and they're checking three things:
Three signals. Assemble them from three tools and you get three different clocks: Whisper timestamps drift against container PTS, PySceneDetect reports frame indices you convert with a frame rate that may be variable, and a tracker samples on its own interval. Joining them is where the week goes.
So the design decision up front is: do you own the extraction, or get it pre-joined?
I've been using FastPix for this because the AI capabilities are boolean flags on the upload call rather than a second pipeline you orchestrate. Same idea works with any provider that returns time-aligned metadata; the shape below is what to look for.
curl -X POST 'https://api.fastpix.com/v1/on-demand' \
--user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
-H 'Content-Type: application/json' \
-d '{
"inputs": [
{ "type": "video", "url": "https://cdn.example.com/webinar.mp4" }
],
"chapters": true,
"accessPolicy": "private",
"maxResolution": "1080p"
}'
{
"success": true,
"data": {
"id": "a1d1acdd-8f4e-4add-b498-6b398cf349d9",
"status": "Created",
"createdAt": "2026-08-17T10:50:34.594302Z",
"playbackIds": [
{ "id": "6ta85f64-5717-4562-b3fc-2c963f66afa6", "accessPolicy": "private" }
],
"maxResolution": "1080p",
"mediaQuality": "standard"
}
}
Auth is Basic: Access Token ID as username, Secret Key as password. Playback is https://stream.fastpix.com/<playbackId>.m3u8
, with a JWT appended as ?token=
for private assets.
⚠️
data
is an object, not an array. If you're porting from an older integration that diddata[0].id
, that's your first bug.💡 Tip:
chapters
is one flag in a family.summary
,namedEntities
,moderation
andsubtitles
are siblings on the same create call, and because they run inside the same processing pass they land referenced to the same timeline. That property is why I stopped maintaining my own extraction stack.⚠️ The API field is
chapters
, but the dashboard's custom media settings JSON calls itgenerateChapters
. Two names for one feature.
Same settings for a direct (push) upload, nested under pushMediaSettings
:
{
"corsOrigin": "*",
"pushMediaSettings": {
"chapters": true,
"accessPolicy": "private",
"maxResolution": "1080p",
"metadata": { "source": "weekly-webinar" }
}
}
And for assets you uploaded before you thought of this, it's a PATCH rather than a re-upload:
curl -X PATCH "https://api.fastpix.com/v1/on-demand/$MEDIA_ID/chapters" \
--user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
-H 'Content-Type: application/json' \
-d '{ "chapters": true }'
Results arrive asynchronously. Handle the lifecycle events and the AI event separately, because they fire at different times.
// webhook.js
import express from 'express';
const app = express();
app.use(express.json());
app.post('/webhooks/video', async (req, res) => {
// ack fast, work later
res.sendStatus(200);
const { type, data, object } = req.body;
switch (type) {
case 'video.media.created':
await db.assets.upsert({ id: object.id, status: 'processing' });
break;
case 'video.media.ready':
await db.assets.update(object.id, { status: 'ready' });
break;
case 'video.media.failed':
await db.assets.update(object.id, { status: 'failed' });
break;
case 'video.mediaAI.chapters.ready':
await onChapters(object.id, data.chapters);
break;
}
});
app.listen(3000);
The chapters payload looks like this:
{
"type": "video.mediaAI.chapters.ready",
"object": { "type": "media", "id": "f081fd53-6a9a-43ae-9d64-9974ef243dbd" },
"id": "51a61b56-0197-4127-8a61-472e4d3fa59a",
"workspace": { "name": "clips-pipeline", "id": "f7a13f50-7f5c-48f4-b7b2-c901dcff61c6" },
"status": "ready",
"data": {
"isChaptersGenerated": true,
"chapters": [
{
"chapter": "1",
"startTime": "00:00:00",
"endTime": "00:03:59",
"title": "The Circle Challenge Begins",
"summary": "Contestants start stacking items in a circle for a chance to win."
}
]
},
"createdAt": "2026-08-17T11:52:29.526588692Z",
"attempts": []
}
Three field names worth pinning down, because they are easy to guess wrong: the sequence number is chapter
and it is a string, the prose field is summary
(not description
), and object.type
is media
even though the event is a mediaAI
event.
⚠️ Times come back as
hh:mm:ss
strings. Convert once, at the boundary, and keep seconds internally. Mixing string times and float seconds in the same codebase is a bug generator.
const toSeconds = (hms) => {
const [h, m, s] = hms.split(':').map(Number);
return h * 3600 + m * 60 + s;
};
Providers will rank clips for you. FastPix's AI clipping produces ranked short clips scored on hook, pacing and narrative, and it's a fine candidate generator. But "which 30 seconds are worth posting" depends on your audience, not on the video, so treat vendor ranking as candidate generation and put your own opinion on top.
// rank.js
const MIN_LEN = 15;
const MAX_LEN = 60;
const SHOT_TOLERANCE = 0.4; // seconds
export function rankCandidates({ chapters, scenes, words }) {
const candidates = [];
for (const ch of chapters) {
const start = toSeconds(ch.startTime);
const end = toSeconds(ch.endTime);
// slide a window through the chapter, snapping to sentence starts
for (const w of words) {
if (w.start < start || w.start > end) continue;
if (!w.isSentenceStart) continue;
const closeWord = lastWordBefore(words, w.start + MAX_LEN);
const length = closeWord.end - w.start;
if (length < MIN_LEN) continue;
candidates.push({
start: w.start,
end: closeWord.end,
chapter: ch.title,
score: score({ start: w.start, end: closeWord.end, words, scenes }),
});
}
}
return dedupeOverlapping(candidates.sort((a, b) => b.score - a.score));
}
function score({ start, end, words, scenes }) {
const inWindow = words.filter((w) => w.start >= start && w.end <= end);
if (!inWindow.length) return 0;
const endsClean = inWindow.at(-1).endsSentence ? 1 : 0;
const shotSafe = scenes.some((s) => Math.abs(s - start) < SHOT_TOLERANCE) ? 1 : 0;
// words per second: dead air is not a clip, but neither is a firehose
const density = inWindow.length / (end - start);
const densityScore = density < 1.2 ? 0 : Math.min(density / 3, 1.5);
// single-speaker stretches survive a vertical crop; crosstalk does not
const speakers = new Set(inWindow.map((w) => w.speaker));
const speakerScore = speakers.size === 1 ? 1 : 0;
return 2 + endsClean + shotSafe + densityScore + speakerScore;
}
The base 2
is for starting on a sentence, a hard filter rather than a score. Everything else is tunable. Tune it by watching output, not by reasoning about it. I got the density floor wrong twice before sitting down and watching twenty rejected candidates.
Doing your own scene detection instead? FFmpeg gives you boundaries directly:
ffmpeg -i webinar.mp4 -filter:v "select='gt(scene,0.4)',showinfo" \
-f null - 2>&1 | grep showinfo | sed -n 's/.*pts_time:\([0-9.]*\).*/\1/p'
14.220
62.480
119.100
187.760
💡 The
0.4
threshold is a starting point, not a constant. Talking-head footage with a static camera needs it lower; a heavily-cut promo needs it higher.
Cutting is the easy half, but there's one gotcha:
ffmpeg -ss 14.22 -to 44.51 -i clip-source.mp4 -c copy clip.mp4
ffmpeg -i clip-source.mp4 -ss 14.22 -to 44.51 \
-c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k clip.mp4
Stream copy is fast and wrong for this job. Your ranking function worked hard to land on a sentence start, and -c copy
will slide that to the nearest keyframe, which is up to a GOP away. Re-encode.
Now the crop. Here's the version everybody writes first:
ffmpeg -i clip.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" -c:a copy vertical.mp4
That's a centre crop, and it's correct exactly when your subject is centred. On a two-person interview it produces a beautiful vertical clip of the gap between them. On a stage recording where the speaker walks, it produces a lectern.
Subject-tracking reframe is the fix, and it's real work to build: detect the subject per frame, smooth the crop path so it doesn't jitter, reset on cuts rather than panning across them. FastPix's auto-reframe does 16:9 to 9:16, 1:1 or 4:5 with subject tracking rather than centre-cropping, which is what made the output shippable instead of something I had to eyeball every time.
Know which one you need:
| Footage | Centre crop | Subject tracking |
|---|---|---|
| Single centred speaker, static camera | Fine | Overkill |
| Screen share / slides | Fine (or use 1:1) | No help |
| Two-person interview | Fails | Required |
| Stage recording, speaker moves | Fails | Required |
Billing is per minute, encoding is free on the standard plan (you pay for delivery, storage and add-ons), and signup gives you $25 in credits with no credit card for the free tier. Early-stage teams can get $600 through the Startup Program (under four years old, under $10M raised), which covers running this over a real back catalogue while you decide.
One honest caveat: this is API-first, not a no-code CMS. If what your team actually wants is a timeline UI where a marketer picks moments by hand, you're building that front end. That was fine here because removing the human from the loop was the point.
subtitles
filter away, and it's what drives silent-autoplay watch time.The documented AI flags (chapters
, summary
, namedEntities
, moderation
, subtitles
) and their webhook payloads are in the In-Video AI docs; the clipping and reframe capabilities are on the product side, so check the current reference for their parameter names before you wire them in. The upload-and-webhook pattern here transfers to any managed video API, so that integration work isn't wasted if you switch. What varies a lot between providers is the AI capability set itself, so check yours against your actual list before you commit.