Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, ort
— and you want to feed a video to a model. Step one is always the same: get the frames. And in Rust today, getting frames means one of two things. Either you shell out to Command::new("ffmpeg")
, write PNGs to disk or parse -f rawvideo
off a pipe, and read them back; or you reach for the low-level bindings and hand-write the send_packet
/ receive_frame
decode loop — forty lines before the first pixel, and you own the EAGAIN
handling and the YUV→RGB conversion yourself. The second path is hard enough that an entire cottage industry of "frame grabber" gists grew up around it: search "rust extract frame from video" and the authoritative answer is frequently someone's gist. A decade on, something this basic still has no comfortable standard answer.
This post is the third path: ez-ffmpeg
0.15's FrameExtractor
pulls frames in the same process, in one call, and hands you tightly packed RGB bytes ready for an ndarray view or a tensor. You will get runnable code, a sampling-strategy cheat sheet, and two things that actually earn their keep: why your thumbnails may have been subtly wrong on color this whole time, and how fast this path really is — with numbers you can reproduce.
Show the ugly first. The shell-out path is cheapest when the output is a file, but the instant frames have to come back into memory they cross a process boundary — PNGs to disk and read back, or a raw pipe you slice into frames yourself. The hand-written decode loop keeps everything in-process, but the boilerplate runs like this: open the demuxer, find the video stream, build the decoder, pair up send_packet
/ receive_frame
, service EAGAIN
, then run each frame through swscale from YUV to RGB — every step its own opportunity to be wrong. And that last step hides a trap: many hand-rolled paths (and some libraries) apply BT.601 conversion coefficients to HD video too, which skews saturated reds and greens visibly — with no error to tell you. More on that trap below.
Neither path is wrong. When the output is a file and the recipe exists, the CLI is still the right answer; when you need packet-level control, the raw bindings are irreplaceable. They just never made the specific thing — get a batch of frames, in-process, in one line, with the color right — comfortable.
One dependency line (it still links libav through ffmpeg-next, so FFmpeg 7.1–8.x has to be installed):
[dependencies]
ez-ffmpeg = "0.15" # needs FFmpeg 7.1-8.x installed (links libav)
Pull 32 frames to feed a VLM. Note UniformN
— it takes N frames spread uniformly across the whole clip by presentation time, the primitive fixed-budget CLIP/VLM pipelines actually want:
use ez_ffmpeg::frame_export::{FrameExtractor, Sampling};
// 32 frames evenly spread over the full duration, scaled to 224 wide
// (height derived from the aspect ratio).
let frames = FrameExtractor::new("input.mp4")
.sampling(Sampling::UniformN(32))
.width(224)
.collect_frames()?; // Vec<VideoFrame>
for f in &frames {
// f.as_bytes() is tightly packed, top-down RGB24 with no row padding —
// width*height*3 bytes, ready for an ndarray view or a tensor.
let (w, h) = (f.width(), f.height());
let rgb: &[u8] = f.as_bytes();
// your_preprocess(rgb, w, h) ...
println!("frame #{:>2} pts={:?}us {}x{}", f.index(), f.pts_us(), w, h);
}
Three things worth pointing at. new(input)
takes a path, a URL, or anything convertible into an Input
; the packed layout defaults to Rgb24
and also offers Rgba32
/ Gray8
. width(224)
sets only the width and derives the height from the source aspect ratio (internally scale=224:-2
); pin the other side with height
, or give both for an exact size. collect_frames()
gathers a Vec
; swap in frames()
for a streaming iterator (Iterator<Item = Result<VideoFrame>>
) that decodes and yields as you consume, without holding every frame in memory — reach for it when you export dense frames from a long file.
The shape of this API is not incidental — it deliberately tracks the FFmpeg CLI. The builder chain reads like a command; start_time_us
/ duration_us
map to -ss
/ -t
(seconds become microseconds); the filter graph it assembles internally is the same scale=…,format=rgb24
shape you would type on the command line, plus explicit color-management parameters. That "move the CLI into your Rust process unchanged" throughline is an argument this series' ecosystem survey makes at length; here it is enough to say your command-line muscle memory stays valid.
Half of frame extraction is deciding which frames. Sampling
lays the common strategies out:
| Strategy | Meaning | Typical use |
|---|---|---|
All |
||
| every decoded frame (default) | dense export, per-frame analysis | |
EveryNth(n) |
||
| one frame in every n | decimate by frame count | |
EverySec(k) |
||
| one frame per k seconds (float) | timeline scrubbing, preview strips | |
KeyframesOnly |
||
| keyframes only | shot/scene proxies, fast at decode time | |
UniformN(n) |
||
| exactly n frames spread by time | ||
| VLM / CLIP fixed-budget input |
Two deserve a note. KeyframesOnly
pins the decoder option skip_frame=nokey
, so non-keyframes are skipped during decode — not decoded and then discarded, not decoded at all — which is where its speed comes from. UniformN(n)
guarantees exactly n frames: an input too short to hold n distinct frames pads by repeating nearby frames (each repeat keeps its source frame's pts_us
), so the "give me exactly 32" a model pipeline was promised is always honored, with no padding logic on the caller's side.
Back to the color trap. YUV→RGB needs a coefficient set: SD video uses BT.601, HD video uses BT.709, and picking the wrong one skews saturated colors. Many "decode to RGB" shortcuts apply BT.601 to everything — fine for SD, wrong for 1080p, and silent about it; you only see it if you put the result on screen.
FrameExtractor
converts per the frame's own color tags by default (ColorPolicy::Tagged
): an HD frame tagged BT.709 converts as BT.709 — no guessing, no one-size-fits-all. This is the behavior in the module I am most willing to stand behind. For contrast: to_ndarray('rgb24')
in older PyAV releases applied BT.601 across the board (that qualifier matters — newer PyAV changed it; do not quote it as current). For untagged frames there is TaggedOrResolutionGuess
(guess by resolution: BT.709 at height ≥ 720), or Force { matrix, range }
to pin one interpretation for the whole run. Most people never touch the default tier — but knowing it gets this right for you is worth the paragraph.
The usual rule: point outward first, not just at my own crate.
ffmpeg -i in.mp4 -vf "select=..." -vsync vfr out_%03d.png
is a one-liner; pulling in a crate for it is a detour.ffmpeg-sidecar
(~131K downloads/month) wraps any video as an RGB frame iterator, links no libav, and is genuinely pleasant — at the cost of an ffmpeg binary at runtime and frames crossing a process boundary.video-rs
iterates RGB ndarrays in ten lines (~32K downloads/month); it has no audio API and self-describes as work-in-progress, but it is strong at its single subject. To be clear: Rust has FrameExtractor
's difference is doing it in-process, with sampling policies and color management, in one line.And three plain facts to close on. One: the compile-and-link pain, ez-ffmpeg removes none of it — it still links libav, and the linking gauntlet is unchanged. Two: the project is ~340 stars, still early-stage, and a "safe Rust API" is not the same as "no C CVEs" — libav is underneath. Three: the whole frame_export
module — FrameExtractor
along with its siblings SampleExtractor
(audio → 16 kHz PCM) and VideoWriter
(frame-push encoding) — is still marked experimental in the docs: 0.15 settled the behavioral semantics (default precision, sampling), but the API shape may still shift.
Frames never leave the process, one call gets them, the color follows the tags, and the conversion stays on swscale's fast path — the decade-old "render to images, stitch with the CLI" cottage industry can retire. If you are doing ML in Rust, extracting frames should not send you back out to a subprocess.
Runnable examples are all in the repo: examples/uniform_thumbnails
(UniformN into a 4×3 contact sheet), examples/extract_rgb_frames
(RGB to PPM), examples/keyframe_thumbnails
(keyframe proxies), examples/frame_sampling
(the sampling strategies side by side). The crate lives at github.com/YeautyYE/ez-ffmpeg.
An open question I would genuinely like answered: in your Rust ML pipeline today, is frame extraction a shell-out, a hand-written decode loop, or something else? And if I've gotten anything above wrong, tell me and I'll fix it.