{"slug": "extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop", "title": "Extracting video frames in Rust, in-process — no CLI, no hand-written decode loop", "summary": "A Rust developer introduced ez-ffmpeg 0.15's FrameExtractor, which extracts video frames in-process without shelling out to ffmpeg or writing a manual decode loop. The tool provides a one-call API that returns tightly packed RGB bytes, handles color conversion correctly, and supports uniform sampling for VLM pipelines.", "body_md": "Here is a task that comes up constantly. You are doing inference in Rust — candle, burn, `ort`\n\n— and you want to feed a video to a model. Step one is always the same: get the frames. And in Rust today, getting frames means one of two things. Either you shell out to `Command::new(\"ffmpeg\")`\n\n, write PNGs to disk or parse `-f rawvideo`\n\noff a pipe, and read them back; or you reach for the low-level bindings and hand-write the `send_packet`\n\n/ `receive_frame`\n\ndecode loop — forty lines before the first pixel, and you own the `EAGAIN`\n\nhandling and the YUV→RGB conversion yourself. The second path is hard enough that an entire cottage industry of \"frame grabber\" gists grew up around it: search \"rust extract frame from video\" and the authoritative answer is frequently someone's gist. A decade on, something this basic still has no comfortable standard answer.\n\nThis post is the third path: `ez-ffmpeg`\n\n0.15's `FrameExtractor`\n\npulls frames **in the same process**, in one call, and hands you tightly packed RGB bytes ready for an ndarray view or a tensor. You will get runnable code, a sampling-strategy cheat sheet, and two things that actually earn their keep: why your thumbnails may have been subtly wrong on color this whole time, and how fast this path really is — with numbers you can reproduce.\n\nShow the ugly first. The shell-out path is cheapest when the output is a file, but the instant frames have to come *back* into memory they cross a process boundary — PNGs to disk and read back, or a raw pipe you slice into frames yourself. The hand-written decode loop keeps everything in-process, but the boilerplate runs like this: open the demuxer, find the video stream, build the decoder, pair up `send_packet`\n\n/ `receive_frame`\n\n, service `EAGAIN`\n\n, then run each frame through swscale from YUV to RGB — every step its own opportunity to be wrong. And that last step hides a trap: many hand-rolled paths (and some libraries) apply BT.601 conversion coefficients to HD video too, which skews saturated reds and greens visibly — with no error to tell you. More on that trap below.\n\nNeither path is wrong. When the output is a file and the recipe exists, the CLI is still the right answer; when you need packet-level control, the raw bindings are irreplaceable. They just never made the specific thing — *get a batch of frames, in-process, in one line, with the color right* — comfortable.\n\nOne dependency line (it still links libav through ffmpeg-next, so FFmpeg 7.1–8.x has to be installed):\n\n```\n[dependencies]\nez-ffmpeg = \"0.15\"   # needs FFmpeg 7.1-8.x installed (links libav)\n```\n\nPull 32 frames to feed a VLM. Note `UniformN`\n\n— it takes N frames spread uniformly across the whole clip by presentation time, the primitive fixed-budget CLIP/VLM pipelines actually want:\n\n```\nuse ez_ffmpeg::frame_export::{FrameExtractor, Sampling};\n\n// 32 frames evenly spread over the full duration, scaled to 224 wide\n// (height derived from the aspect ratio).\nlet frames = FrameExtractor::new(\"input.mp4\")\n    .sampling(Sampling::UniformN(32))\n    .width(224)\n    .collect_frames()?; // Vec<VideoFrame>\n\nfor f in &frames {\n    // f.as_bytes() is tightly packed, top-down RGB24 with no row padding —\n    // width*height*3 bytes, ready for an ndarray view or a tensor.\n    let (w, h) = (f.width(), f.height());\n    let rgb: &[u8] = f.as_bytes();\n    // your_preprocess(rgb, w, h) ...\n    println!(\"frame #{:>2} pts={:?}us {}x{}\", f.index(), f.pts_us(), w, h);\n}\n```\n\nThree things worth pointing at. `new(input)`\n\ntakes a path, a URL, or anything convertible into an `Input`\n\n; the packed layout defaults to `Rgb24`\n\nand also offers `Rgba32`\n\n/ `Gray8`\n\n. `width(224)`\n\nsets only the width and derives the height from the source aspect ratio (internally `scale=224:-2`\n\n); pin the other side with `height`\n\n, or give both for an exact size. `collect_frames()`\n\ngathers a `Vec`\n\n; swap in `frames()`\n\nfor a streaming iterator (`Iterator<Item = Result<VideoFrame>>`\n\n) that decodes and yields as you consume, without holding every frame in memory — reach for it when you export dense frames from a long file.\n\nThe shape of this API is not incidental — it deliberately tracks the FFmpeg CLI. The builder chain reads like a command; `start_time_us`\n\n/ `duration_us`\n\nmap to `-ss`\n\n/ `-t`\n\n(seconds become microseconds); the filter graph it assembles internally is the same `scale=…,format=rgb24`\n\nshape you would type on the command line, plus explicit color-management parameters. That \"move the CLI into your Rust process unchanged\" throughline is an argument this series' ecosystem survey makes at length; here it is enough to say your command-line muscle memory stays valid.\n\nHalf of frame extraction is deciding *which* frames. `Sampling`\n\nlays the common strategies out:\n\n| Strategy | Meaning | Typical use |\n|---|---|---|\n`All` |\nevery decoded frame (default) | dense export, per-frame analysis |\n`EveryNth(n)` |\none frame in every n | decimate by frame count |\n`EverySec(k)` |\none frame per k seconds (float) | timeline scrubbing, preview strips |\n`KeyframesOnly` |\nkeyframes only | shot/scene proxies, fast at decode time\n|\n`UniformN(n)` |\nexactly n frames spread by time |\nVLM / CLIP fixed-budget input |\n\nTwo deserve a note. `KeyframesOnly`\n\npins the decoder option `skip_frame=nokey`\n\n, so non-keyframes are skipped *during* decode — not decoded and then discarded, not decoded at all — which is where its speed comes from. `UniformN(n)`\n\nguarantees **exactly n frames**: an input too short to hold n distinct frames pads by repeating nearby frames (each repeat keeps its source frame's `pts_us`\n\n), so the \"give me exactly 32\" a model pipeline was promised is always honored, with no padding logic on the caller's side.\n\nBack to the color trap. YUV→RGB needs a coefficient set: SD video uses BT.601, HD video uses BT.709, and picking the wrong one skews saturated colors. Many \"decode to RGB\" shortcuts apply BT.601 to everything — fine for SD, wrong for 1080p, and silent about it; you only see it if you put the result on screen.\n\n`FrameExtractor`\n\nconverts per the frame's own color tags by default (`ColorPolicy::Tagged`\n\n): an HD frame tagged BT.709 converts as BT.709 — no guessing, no one-size-fits-all. This is the behavior in the module I am most willing to stand behind. For contrast: `to_ndarray('rgb24')`\n\nin *older* PyAV releases applied BT.601 across the board (that qualifier matters — newer PyAV changed it; do not quote it as current). For untagged frames there is `TaggedOrResolutionGuess`\n\n(guess by resolution: BT.709 at height ≥ 720), or `Force { matrix, range }`\n\nto pin one interpretation for the whole run. Most people never touch the default tier — but knowing it gets this right for you is worth the paragraph.\n\nThe usual rule: point outward first, not just at my own crate.\n\n`ffmpeg -i in.mp4 -vf \"select=...\" -vsync vfr out_%03d.png`\n\nis a one-liner; pulling in a crate for it is a detour.`ffmpeg-sidecar`\n\n(~131K downloads/month) wraps any video as an RGB frame iterator, links no libav, and is genuinely pleasant — at the cost of an ffmpeg binary at runtime and frames crossing a process boundary.`video-rs`\n\niterates RGB ndarrays in ten lines (~32K downloads/month); it has no audio API and self-describes as work-in-progress, but it is strong at its single subject. To be clear: Rust has `FrameExtractor`\n\n's difference is doing it in-process, with sampling policies and color management, in one line.And three plain facts to close on. One: the compile-and-link pain, ez-ffmpeg removes none of it — it still links libav, and the linking gauntlet is unchanged. Two: the project is ~340 stars, still early-stage, and a \"safe Rust API\" is not the same as \"no C CVEs\" — libav is underneath. Three: the whole `frame_export`\n\nmodule — `FrameExtractor`\n\nalong with its siblings `SampleExtractor`\n\n(audio → 16 kHz PCM) and `VideoWriter`\n\n(frame-push encoding) — is still marked experimental in the docs: 0.15 settled the behavioral semantics (default precision, sampling), but the API shape may still shift.\n\nFrames never leave the process, one call gets them, the color follows the tags, and the conversion stays on swscale's fast path — the decade-old \"render to images, stitch with the CLI\" cottage industry can retire. If you are doing ML in Rust, extracting frames should not send you back out to a subprocess.\n\nRunnable examples are all in the repo: `examples/uniform_thumbnails`\n\n(UniformN into a 4×3 contact sheet), `examples/extract_rgb_frames`\n\n(RGB to PPM), `examples/keyframe_thumbnails`\n\n(keyframe proxies), `examples/frame_sampling`\n\n(the sampling strategies side by side). The crate lives at [github.com/YeautyYE/ez-ffmpeg](https://github.com/YeautyYE/ez-ffmpeg).\n\nAn open question I would genuinely like answered: in your Rust ML pipeline today, is frame extraction a shell-out, a hand-written decode loop, or something else? And if I've gotten anything above wrong, tell me and I'll fix it.", "url": "https://wpnews.pro/news/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop", "canonical_source": "https://dev.to/yeauty/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop-3393", "published_at": "2026-07-25 22:52:39+00:00", "updated_at": "2026-07-25 23:31:26.580341+00:00", "lang": "en", "topics": ["developer-tools", "computer-vision", "machine-learning"], "entities": ["ez-ffmpeg", "FFmpeg", "FrameExtractor"], "alternates": {"html": "https://wpnews.pro/news/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop", "markdown": "https://wpnews.pro/news/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop.md", "text": "https://wpnews.pro/news/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop.txt", "jsonld": "https://wpnews.pro/news/extracting-video-frames-in-rust-in-process-no-cli-no-hand-written-decode-loop.jsonld"}}