Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop

A Rust developer introduced ez-ffmpeg 0.15's FrameExtractor, which extracts video frames in-process without shelling out to ffmpeg or writing a manual decode loop. The tool provides a one-call API that returns tightly packed RGB bytes, handles color conversion correctly, and supports uniform sampling for VLM pipelines.

Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, ort — and you want to feed a video to a model. Step one is always the same: get the frames. And in Rust today, getting frames means one of two things. Either you shell out to Command::new "ffmpeg" , write PNGs to disk or parse -f rawvideo off a pipe, and read them back; or you reach for the low-level bindings and hand-write the send packet / receive frame decode loop — forty lines before the first pixel, and you own the EAGAIN handling and the YUV→RGB conversion yourself. The second path is hard enough that an entire cottage industry of "frame grabber" gists grew up around it: search "rust extract frame from video" and the authoritative answer is frequently someone's gist. A decade on, something this basic still has no comfortable standard answer. This post is the third path: ez-ffmpeg 0.15's FrameExtractor pulls frames in the same process , in one call, and hands you tightly packed RGB bytes ready for an ndarray view or a tensor. You will get runnable code, a sampling-strategy cheat sheet, and two things that actually earn their keep: why your thumbnails may have been subtly wrong on color this whole time, and how fast this path really is — with numbers you can reproduce. Show the ugly first. The shell-out path is cheapest when the output is a file, but the instant frames have to come back into memory they cross a process boundary — PNGs to disk and read back, or a raw pipe you slice into frames yourself. The hand-written decode loop keeps everything in-process, but the boilerplate runs like this: open the demuxer, find the video stream, build the decoder, pair up send packet / receive frame , service EAGAIN , then run each frame through swscale from YUV to RGB — every step its own opportunity to be wrong. And that last step hides a trap: many hand-rolled paths and some libraries apply BT.601 conversion coefficients to HD video too, which skews saturated reds and greens visibly — with no error to tell you. More on that trap below. Neither path is wrong. When the output is a file and the recipe exists, the CLI is still the right answer; when you need packet-level control, the raw bindings are irreplaceable. They just never made the specific thing — get a batch of frames, in-process, in one line, with the color right — comfortable. One dependency line it still links libav through ffmpeg-next, so FFmpeg 7.1–8.x has to be installed : dependencies ez-ffmpeg = "0.15" needs FFmpeg 7.1-8.x installed links libav Pull 32 frames to feed a VLM. Note UniformN — it takes N frames spread uniformly across the whole clip by presentation time, the primitive fixed-budget CLIP/VLM pipelines actually want: use ez ffmpeg::frame export::{FrameExtractor, Sampling}; // 32 frames evenly spread over the full duration, scaled to 224 wide // height derived from the aspect ratio . let frames = FrameExtractor::new "input.mp4" .sampling Sampling::UniformN 32 .width 224 .collect frames ?; // Vec<VideoFrame for f in &frames { // f.as bytes is tightly packed, top-down RGB24 with no row padding — // width height 3 bytes, ready for an ndarray view or a tensor. let w, h = f.width , f.height ; let rgb: & u8 = f.as bytes ; // your preprocess rgb, w, h ... println "frame {: 2} pts={:?}us {}x{}", f.index , f.pts us , w, h ; } Three things worth pointing at. new input takes a path, a URL, or anything convertible into an Input ; the packed layout defaults to Rgb24 and also offers Rgba32 / Gray8 . width 224 sets only the width and derives the height from the source aspect ratio internally scale=224:-2 ; pin the other side with height , or give both for an exact size. collect frames gathers a Vec ; swap in frames for a streaming iterator Iterator<Item = Result<VideoFrame that decodes and yields as you consume, without holding every frame in memory — reach for it when you export dense frames from a long file. The shape of this API is not incidental — it deliberately tracks the FFmpeg CLI. The builder chain reads like a command; start time us / duration us map to -ss / -t seconds become microseconds ; the filter graph it assembles internally is the same scale=…,format=rgb24 shape you would type on the command line, plus explicit color-management parameters. That "move the CLI into your Rust process unchanged" throughline is an argument this series' ecosystem survey makes at length; here it is enough to say your command-line muscle memory stays valid. Half of frame extraction is deciding which frames. Sampling lays the common strategies out: | Strategy | Meaning | Typical use | |---|---|---| All | every decoded frame default | dense export, per-frame analysis | EveryNth n | one frame in every n | decimate by frame count | EverySec k | one frame per k seconds float | timeline scrubbing, preview strips | KeyframesOnly | keyframes only | shot/scene proxies, fast at decode time | UniformN n | exactly n frames spread by time | VLM / CLIP fixed-budget input | Two deserve a note. KeyframesOnly pins the decoder option skip frame=nokey , so non-keyframes are skipped during decode — not decoded and then discarded, not decoded at all — which is where its speed comes from. UniformN n guarantees exactly n frames : an input too short to hold n distinct frames pads by repeating nearby frames each repeat keeps its source frame's pts us , so the "give me exactly 32" a model pipeline was promised is always honored, with no padding logic on the caller's side. Back to the color trap. YUV→RGB needs a coefficient set: SD video uses BT.601, HD video uses BT.709, and picking the wrong one skews saturated colors. Many "decode to RGB" shortcuts apply BT.601 to everything — fine for SD, wrong for 1080p, and silent about it; you only see it if you put the result on screen. FrameExtractor converts per the frame's own color tags by default ColorPolicy::Tagged : an HD frame tagged BT.709 converts as BT.709 — no guessing, no one-size-fits-all. This is the behavior in the module I am most willing to stand behind. For contrast: to ndarray 'rgb24' in older PyAV releases applied BT.601 across the board that qualifier matters — newer PyAV changed it; do not quote it as current . For untagged frames there is TaggedOrResolutionGuess guess by resolution: BT.709 at height ≥ 720 , or Force { matrix, range } to pin one interpretation for the whole run. Most people never touch the default tier — but knowing it gets this right for you is worth the paragraph. The usual rule: point outward first, not just at my own crate. ffmpeg -i in.mp4 -vf "select=..." -vsync vfr out %03d.png is a one-liner; pulling in a crate for it is a detour. ffmpeg-sidecar ~131K downloads/month wraps any video as an RGB frame iterator, links no libav, and is genuinely pleasant — at the cost of an ffmpeg binary at runtime and frames crossing a process boundary. video-rs iterates RGB ndarrays in ten lines ~32K downloads/month ; it has no audio API and self-describes as work-in-progress, but it is strong at its single subject. To be clear: Rust has FrameExtractor 's difference is doing it in-process, with sampling policies and color management, in one line.And three plain facts to close on. One: the compile-and-link pain, ez-ffmpeg removes none of it — it still links libav, and the linking gauntlet is unchanged. Two: the project is ~340 stars, still early-stage, and a "safe Rust API" is not the same as "no C CVEs" — libav is underneath. Three: the whole frame export module — FrameExtractor along with its siblings SampleExtractor audio → 16 kHz PCM and VideoWriter frame-push encoding — is still marked experimental in the docs: 0.15 settled the behavioral semantics default precision, sampling , but the API shape may still shift. Frames never leave the process, one call gets them, the color follows the tags, and the conversion stays on swscale's fast path — the decade-old "render to images, stitch with the CLI" cottage industry can retire. If you are doing ML in Rust, extracting frames should not send you back out to a subprocess. Runnable examples are all in the repo: examples/uniform thumbnails UniformN into a 4×3 contact sheet , examples/extract rgb frames RGB to PPM , examples/keyframe thumbnails keyframe proxies , examples/frame sampling the sampling strategies side by side . The crate lives at github.com/YeautyYE/ez-ffmpeg https://github.com/YeautyYE/ez-ffmpeg . An open question I would genuinely like answered: in your Rust ML pipeline today, is frame extraction a shell-out, a hand-written decode loop, or something else? And if I've gotten anything above wrong, tell me and I'll fix it.