{"slug": "getting-video-models-to-learn-better-faster", "title": "Getting video models to learn better, faster", "summary": "BentoML's engineering team reports that most gains in image and video model performance since Stable Diffusion 3 come from data filtering, rebalancing, annotation, and synthetic data generation, not from architectural changes. In 2024, they used CPU-based traditional computer vision methods like PySceneDetect for scene detection to filter tens of billions of images and videos, and they emphasize the importance of manually reviewing data samples to build an ontology of good and bad categories.", "body_md": "Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3. In our experience, most of the gains are directly attributable to 3 flavors of data improvements:\n\n**Data Filtering & Rebalancing**: Remove noisy data and resample your data strategically so your model learns more effectively** Data Annotation**: Gather better annotations like richer captions, bounding boxes, and font details so that it's easier for your model to disambiguate visual concepts**Synthetic Data Generation**: Finetune an ensemble of existing generative models to create training data for which there is little-to-no naturally occurring data (e.g. image editing / reference-conditioning for Nano-Banana style models)\n\nA couple of years ago, the prevailing wisdom across all generative models (be it text, image, audio) was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data (e.g. heavily compressed JPEGs) into pre-training, your model is going to waste a significant amount of its capacity learning how to mimic this slice of data. If you filter your dataset well, your model will have a lot easier time learning what you want it to learn.\n\nWe know this sounds obvious, but it's a lot harder to do in practice.\n\nToday we're going to walk you through how our approach to data filtering has evolved since 2024. And, hopefully we'll save you from a couple of headaches if you end up training your own generative models down the line.\n\n## [2024] Filtering on a budget — Traditional CV on CPUs\n\nOn the first go around, we decided to push our raw dataset through old-school computer vision algorithms. This way we could get away with a cluster of cheap CPU instances instead of an unholy number of GPUs running a multimodal LLM.\n\n### Scene detection\n\nWe need to filter down tens of billions of images and videos to create our pre-training dataset. Images don't really require any specific pre-processing, but raw videos do.\n\n[Next time you watch a television show or movie, track how often the camera cuts.](https://www.youtube.com/watch?v=kmvvA7wudKo) If you're watching something made in the last twenty years, more likely than not you'll see a cut every 5 seconds. When to cut and how to cut is an authorial decision, not something a generative video model should do arbitrarily. So, we need to slice n' dice our videos on shot boundaries into video clips before we can filter them down.\n\nWith our cheapskate CPU-only agenda, we picked up [PySceneDetect](https://www.scenedetect.com/). At a high level it maintains a rolling window of K-frames and if the K+1 frame has significantly different image statistics, it categorizes the frame as a cut. There's no underlying machine learning model. It runs really fast but struggles with common transitions like dissolves, fades, and jitter cuts (which low key is a huge issue).\n\n### Getting to know your data\n\nWhenever you get new data, you should spend a few days reviewing random samples, listing what you'd like to keep and what you'd like to throw out. Ideally, you take the time to draft an ontology of categories within \"good\" and \"bad\" and track the relative sizes of these categories.\n\nAt some point during the data filtering process, your engineer brain will take over, and you'll spend way too much time tuning the knobs of your heuristics (or LLMs), chasing that \"perfect\" decision boundary. These notes are going to save you from yourself down the line. They'll give you the facts you'll need to talk yourself out of trying \"one more idea\", when the answer is clearly \"no\".\n\nPlus, understanding the shape of the data distribution will really help with dataset rebalancing. Certain categories are overrepresented in the natural distribution of all videos. We need to subsample and suppress this signal, otherwise it will dominate training and our model will struggle to learn the long-tail of people/places/things/actions that we need in order to generate anything.\n\n### Sieving out the un-captionable\n\nGenerative video models are primarily limited by what we can describe correctly and consistently in words. Text provides a pretty good scaffold to understand the visual world, but it's by no means the correct conditioning mechanism for all aspects of video generation. Details like camera trajectories in space-time and the nuances of an actor's performance are simply indescribable in natural language.\n\nFor now, we need to filter out clips where the primary \"thing\" that makes the video clip interesting is un-captionable. Without a crystal clear text description, it's just noise to our text-to-video model.\n\n#### Text-heavy\n\nFor example, we want to filter out text-heavy videos. It's still hard for LLMs to caption motion graphics that are constantly changing on screen. We don't want to waste capacity in our 2B parameter model learning motion graphics when it could be allocated instead to learning actions.\n\nTo do this, we sampled frames from each video and ran a tiny [EAST Detector](https://docs.opencv.org/4.x/d8/ddc/classcv_1_1dnn_1_1TextDetectionModel__EAST.html) to extract bounding boxes for text. From there, we filtered out text heavy videos based on the percentage of the frames that had text and the percentage of each frame covered in text.\n\nUsing a CNN for this task was a good idea, but the specific choice was wrong. In order to run tens of billions of frames on CPUs, we had to resize the frames aggressively. So, a lot of text-heavy samples with small fonts fell through the cracks.\n\nSmall text goes undetected, after EAST image pre-processing\n\nEAST is a pretty old model from 2017. It's small and far from the state of the art on text detection. Getting it to run efficiently on CPUs without cache-thrash and thread oversubscription was a challenge. Even after performance optimizations, it was still the largest bottleneck for this version of the data pipeline.\n\n#### Indescribable actions\n\nWhen there's not much happening on the screen (e.g. close-up on a person's face), it's hard to describe the specific action taking place. If there's too much happening (e.g. extremely shaky camera, a soccer match with a bunch of folks moving across the pitch at once), LLMs struggle to caption the clip correctly. We lumped these categories of videos together as \"indescribable action\" clips to be thrown out.\n\nVideos are typically serialized on disk in a compressed format. Codecs like H.264 reduce file size by storing keyframes and motion vectors that describe how the keyframes change over time, rather than RGB values for each pixel over time.\n\nH.264 motion vectors\n\nKeep this clip. Lots of moderately sized motion vectors.\n\nWe used the motion vectors stored within the mp4 files themselves to isolate and filter out the \"indescribable action\" videos. Specifically, we used [mv-extractor](https://github.com/LukasBommes/mv-extractor) and computed two heuristics per clip:\n\n`average_frame_energy`\n\n: L2-norm of all motion vectors averaged across the video`min(sub_clip_average_frame_energy)`\n\n: Split each clip into a variable number of chunks depending on the video's length, calculate average frame energy for each chunk, and take the minimum across these L2-norms\n\nThen came the decision tree:\n\n`average_frame_energy`\n\n< 0.1: Throw the clip away. These were essentially static videos (e.g. slideshows, still frames, freeze-frames).`average_frame_energy`\n\n> 25: Throw the clip away. The footage was incredibly chaotic.`min(sub_clip_average_frame_energy)`\n\n< 0.03: Throw the clip away. A portion of the clip has nothing happening (e.g. a fade, transition to a still image in a documentary).- Keep everything else.\n\nThis works well as a cheap first filter, but it has mediocre recall (i.e., a lot of indescribable action clips are kept in the dataset).\n\n### Subsampling talking head video clips\n\nFrom our initial review of the raw data distribution, it was pretty obvious that talking head clips where folks talk straight to camera were dramatically over-represented. If we let the dataset be, it would have been significantly biased towards this sort of clip. Our model would get disproportionately good at creating them (likely at the expense of others), so we needed to find them and subsample them.\n\nLiving in an old-school CV world, we naturally burrowed deeper down the engineering tunnel and introduced *additional heuristics*. We sampled frames from each video clip and ran a [Haar-cascade face detector](https://docs.opencv.org/4.x/db/d28/tutorial_cascade_classifier.html) to extract bounding boxes for faces and calculated two numbers:\n\n`average_face_frame_energy`\n\n: L2-norm of motion vectors within face bounding boxes`average_background_frame_energy`\n\n: L2-norm of motion vectors, just in the corners of the frame (as a proxy for background motion)\n\nAnd from there, another decision tree:\n\n- Moderate\n`average_frame_energy`\n\n+`average_face_frame_energy`\n\n>=`average_background_frame_energy`\n\n: Keep the clip. Usually a really good close-up. - Low/Moderate\n`average_frame_energy`\n\n: Subsample these.\n\n## [Early 2025] `rm -rf`\n\n— Replacing hand-crafted heuristics with finetuned LLMs\n\nWe're starting to sketch a rather complicated decision tree. It's full of lossy proxies that only kind of work, and it's very incomplete.\n\nThis approach simply doesn't scale. Every time you have a new idea for a filter you have to re-examine how the new node in the decision tree impacts all the other branches. Everything is intertwined and eventually you end up with a pipeline that's both un-interpretable and uneditable.\n\nWhen we started in 2024, we were staring down the barrel of tens of billions of samples. Given our limited budget, our gut was to construct the cheapest filters possible. *This was fundamentally wrong.* Our video model struggled to learn basic motions like guitar strumming after training for several weeks on our 2024 dataset but was able to learn these very actions in less than 24 hours of training, after applying our 2025 filters. Instead of looking for the cheapest filters possible, you should optimize for the best possible filters you can afford.\n\n### Migrating to 1000s of GPUs\n\nThis brings us to our second takeaway: throw away your \"principled\" computer vision techniques and adopt black box neural networks wherever you can. There are patterns that humans simply can't describe well, no matter how hard we try. Old school CV methods were the best hand-crafted approximations of their era. They're truly impressive feats of engineering, but a well-trained neural network will learn a non-linear function that will win on precision and recall in 99% of cases.\n\nOnce you re-orient yourself around this reality, your job should shift from crafting cheap heuristics to optimizing models for GPU throughput and engineering resilient, parallelizable workloads to run on SPOT instances across providers.\n\nConcretely, we replaced PySceneDetect's heuristics with [AutoShot](https://github.com/wentaozhu/AutoShot) and [TransNetV2](https://github.com/soCzech/TransNetV2), accelerating inference with custom CUDA kernels. We stopped running the ancient EAST Detector on CPUs. Instead, we deployed PaddleOCR with TensorRT across thousands of Nvidia A10Gs and L4s. And, we ripped out the tangled web of computer vision algorithms (Haar Face Detectors, Lucas-Kanade, monocular depth estimators, etc.), replacing them with a set of fine-tuned LLMs.\n\n### Iterative dataset labeling (aka self-consistency is harder than you think)\n\nFinetuning is pretty straightforward thanks to the folks at [Unsloth](https://github.com/unslothai/unsloth?locale=en-US). This means the labeling is the work.\n\nThe hard part of training LLMs for data filtration is that the categories are always somewhat fuzzy. You'll have to answer questions like:\n\n- If a sample fits several categories to different degrees, which label do I assign it?\n- Should I simplify my categories, so I can label the data more quickly and consistently? Or, do I need to split my category into pieces to make it clearer?\n- How easy is this concept for the LLM to learn? How much data do I need for each category?\n\nMore often than not, the biggest problem you'll run into is one of self-consistency. Over the course of labeling a couple hundred samples, it's only natural that you'll relax your criteria, mislabel samples, and muddy the signal in your dataset.\n\nIn a [CMU study from 2025](https://linzhiqiu.github.io/papers/camerabench/), researchers hired cinematography experts to annotate camera motion in online video clips and train other laypeople to make similar annotations. Even with the criteria in hand, the experts disagreed with \"ground truth\" ~24% of the time. Only through repeated trials were they able to converge on 96% agreement. It's painful to spend days labeling and re-labeling a dataset, but them's the breaks. At least, we're lucky to live in an era where you only need 2-3K samples to train a good filter.\n\nIf you find yourself working on data filtration, we'd recommend you hack together a simple labeling tool like the one we show below. We just slapped together a super simple React app with Supabase to store labels and R2 to store the samples.\n\nOur internal web-app for data labeling & LLM evals\n\nHere, we load an existing dataset and thumb through a couple of examples of images graded \"1\" (ugliest on our aesthetic scale). The tool needs to make it easy to grep through your existing labels, version control your labels, and edit them, since you're going to take multiple passes on the dataset.\n\n**Use Hotkeys**: You'll want to label as fast as possible (or you'll go crazy). Make sure you can label via hotkeys and that you can edit hotkey mappings easily within the app itself.**Make Datasets Forkable**: You'll be taking several turns on your dataset, so it's helpful to have a`fork`\n\nfeature, where you seed a new dataset from your old labels. Even better if you can quickly drill down to the training samples your model misclassified. These are especially problematic. You'll need to review them to iterate on your criteria effectively. Plus, you'll want to relabel them first.**Add tools for label mapping**: As you iterate on your ontology, categories will come and go. So, you'll need to make it easy to assign samples that were labeled`A`\n\nto another category`B`\n\n, as you add, merge, and delete groupings.\n\n### Turning LLMs into categorical classifiers\n\nUltimately, we supervise-finetuned (SFT) Qwen-2-VL-2B to tag:\n\n**Image Categories**:`Ugly Product Image`\n\n,`Diagram / Screenshot`\n\n,`Collage`\n\n,`Watermarked`\n\n,`Bad Lighting`\n\n,`Pixelated`\n\n,`Drawing / Illustration`\n\n,`Keep`\n\n**Video Categories**:`Animation`\n\n,`PoV`\n\n,`Bars`\n\n,`Motion Graphics`\n\n,`Ken Burns`\n\n,`Shaky Camera`\n\n,`Little to No Motion`\n\n,`Weird Transition`\n\n,`Keep`\n\nSamples across image categories\n\nUgly Product Image: Product images with messy backgrounds or really flat gray backgrounds (think bad Amazon catalog images).\n\nWe only kept images that our filter predicted as `Drawing / Illustration`\n\nor `Keep`\n\n. For videos, we retained `Animation`\n\nand `Keep`\n\nclips wholesale, while subsampling `PoV`\n\n.\n\n## [Late 2025] Reinforcement Learning with Verifiable Rewards (RLVR) for aesthetic filtering\n\nAt the start of the data labeling process, we tried to get extremely specific about the properties of the images and videos that divvied up samples into ugly vs. pretty (e.g. overexposed lighting, muted color grades). We thought it would be easier for the LLM to learn the precise reasons why we considered an image ugly than learn an arbitrary \"ugliness score\".\n\nOnce again, our initial intuition turned out to be wrong. The properties that make a particular sample ugly tend to be correlated; you end up assigning K different aesthetic tags to the same sample. And in turn, this poses two significant challenges:\n\n**Sparse Data Signal**: The combinatorial explosion of tags makes it harder for the model to disentangle the categories, especially with a small dataset of a few thousand labels.**Slow, Inconsistent Labeling**: It takes a lot longer to label samples (and it's a lot harder to be self-consistent) when you have the cognitive load of weighing several possible tags per sample.\n\n### Fine-grained aesthetic scorers\n\nHang with us, as we work through a short history lesson.\n\nThe primary way that folks traditionally finetuned LLMs is supervised-finetuning (SFT). We used this approach when training our categorical classifiers. We fed Qwen-2-VL-2B triplets of (image/video frames, prompt `Categorize this as A, B, or C`\n\n, true label) and used the same objective from pre-training (next-word-prediction-with-cross-entropy) to update the model. With SFT, you're essentially extending pre-training, so your model learns your new task.\n\nAt the start of 2025, DeepSeek popularized a different finetuning technique called \"Reinforcement Learning with Verifiable Rewards\" (RLVR) in their [R1 paper](https://arxiv.org/abs/2501.12948). Instead of using next-word prediction, they used reinforcement learning (RL) where the model's response is graded for accuracy using a rubric.\n\nIt turns out that if your LLM is able to solve a problem even 1 in 1,000 times, we can nudge the network towards finding this solution more consistently. These network updates are much smaller than those provided during SFT, so RLVR allows us to exert much finer grained control on extending the LLM to our tasks. Bluntly put, SFT is model surgery with a butcher knife, while RLVR is model surgery with a scalpel.\n\nIngredients for aesthetic scoring (RLVR)\n\nPlease grade the image on a scale of 1 to 4, where 1 is the lowest quality and 4 is the highest quality:\n\n*1***Disgusting**(Lowest Quality)*2***Ugly**(Bad Quality)*3***Good**(Acceptable Quality)*4***Beautiful**(Highest Quality)\n\n**Disgusting** images are often blurry (out of focus) or pixelated (old image, grainy). The details are hard to see. These images are often overexposed (so much light that details are washed out) or underexposed (too dark to see details). Infographics are disgusting.\n\n**Ugly** images suffer from the same problems as disgusting images, but are qualitatively not as bad. They have low contrast, are poorly lit, or have \"flat\" looking subjects that do not stand out much from their backgrounds. Small watermarks are ugly if they go across the entire image (e.g. in center).\n\n**Good** images are well lit, have clear contrast, and clear subjects.\n\n**Beautiful** images have high dynamic range (crisp, colorful, sharp contrast, or stylized). They are often portraits or action shots.\n\nwhat the model answered (ŷ)\n\nwhat we labeled it (y)\n\n| 1 | 2 | 3 | 4 | non-numeric | |\n|---|---|---|---|---|---|\n| 1 | +1 | −1 | −2 | −3 | −4 |\n| 2 | −1 | +1 | −1 | −2 | −4 |\n| 3 | −2 | −1 | +1 | −1 | −4 |\n| 4 | −3 | −2 | −1 | +1 | −4 |\n\nBy the time we had our aesthetic dataset ready to go, RLVR was gaining adoption by the major labs; so we gave it a go for our aesthetic scorers and it smoked its SFT counterparts. Our prompt explicated the different reasons why we'd grade a sample from 1-4, while our rubric penalized the model on the absolute difference between the ground truth and predicted labels. Specifically, we used [Group Sequence Policy Optimization (GSPO)](https://arxiv.org/abs/2507.18071) as the RLVR algorithm on Qwen-2.5-VL-3B.\n\nWhen you introduce reasoning to an LLM, it's somewhat standard to first SFT the model with human-written reasoning traces, so it understands how to use the logic before learning to generate its own reasoning. We struggled to explain our aesthetic grades succinctly. So, we nixed the SFT phase altogether and jumped straight to DeepSeek-R1-Zero's techniques for training LLMs how to reason with zero human traces. That didn't work either. Turns out, it's just really hard to describe in words why something is 2 vs. 3 aesthetically.\n\n## Filtering end-to-end\n\nAfter RLVR, we adopted [WAFT](https://github.com/princeton-vl/WAFT) (a SOTA optical flow predictor) to filter out even more of the long tail of low-motion videos. Instead of relying on heuristics like Haar-cascades, we converted our video captions into tags and used these to edit our training data distribution (e.g., subsample talking heads, oversample animal videos).\n\nData Filtering Pipeline\n\n~15 billion images*01*Candidate pool**~15B** 100%keep images that match 5 target aspect-ratios*02*Aspect ratio + min size (256px+)**~7.2B** 48%drop low quality product images*03*Solid color backgrounds**~5.0B** 33%drop text-heavy images*04*PaddleOCR on TensorRT**~3.3B** 22%categorical Qwen-2-VL-2B classifiers*05*SFT Filters**~950M** 6.3%cache perceptual hashes in FAISS, drop duplicates with hamming < 5*06*P-Hash Deduplication**649M** 4.3%fine-grained aesthetic Qwen-2.5-VL-3B scorers*07*RLVR Filters**~250M** 1.7%\n\n- kept\n- thrown out\n\n~250 years of footage (~1 billion clips)*01*Candidate pool**~1B** 100%***retain 16:9 only*02*Aspect ratio**~800M** 80%***drop clips under 2s or over 10s*03*TransNetV2 & AutoShot shot detection**~700M** 70%drop low motion clips*04*H.264 motion vectors**~520M** 52%drop text-heavy clips*05*PaddleOCR on TensorRT**~375M** 37.5%categorical Qwen-2-VL-2B classifiers*06*SFT Filters**~150M** 15%fine-grained aesthetic Qwen-2.5-VL-3B scorers*07*RLVR Filters**~65M** 6.5%drop long tail low motion clips*08*WAFT optical flow**~50M** 5%\n\n- kept\n- thrown out\n\n* Extrapolated from the clip count, before we throw out clips that are too long or too short.\n\nNowadays, there is a lot of jargon being tossed around the internet when it comes to \"pre-training\", \"mid-training\", and \"post-training\". The existence of distinct pre-training and mid-training phases is just a function of the impossibility of creating the perfect filter.\n\nDuring pre-training, folks are more permissive. They're open to letting some low-quality data into the stack, so that they can guarantee that they cover all the necessary modes of their target data distribution. And by the time they get to mid-training, they're willing to tighten the distribution, even if it means throwing away some good stuff.\n\nRegardless, data filtering is the single biggest lever you can pull outside of scaling to improve the quality of your model. If you take one thing away from this journey, never skimp on your data filtering pipeline. These things are what they eat after all.\n\n## Who are we?\n\n[We're two brothers training text-to-video models from scratch, trying to make animation accessible to everyone.](/about) Stay tuned for more blogs on captioning, synthetic data generation, and our ongoing work exploring better manifolds for pixel-space generative models.\n\n### Get Field Notes\n\nTechnical deep dives on building generative video models from the ground up, plus updates on new releases from Linum.", "url": "https://wpnews.pro/news/getting-video-models-to-learn-better-faster", "canonical_source": "https://www.linum.ai/field-notes/data-filtering-gen-video", "published_at": "2026-08-27 01:53:34+00:00", "updated_at": "2026-08-27 02:18:25.491322+00:00", "lang": "en", "topics": ["machine-learning", "computer-vision", "generative-ai", "ai-research"], "entities": ["BentoML", "Stable Diffusion 3", "PySceneDetect"], "alternates": {"html": "https://wpnews.pro/news/getting-video-models-to-learn-better-faster", "markdown": "https://wpnews.pro/news/getting-video-models-to-learn-better-faster.md", "text": "https://wpnews.pro/news/getting-video-models-to-learn-better-faster.txt", "jsonld": "https://wpnews.pro/news/getting-video-models-to-learn-better-faster.jsonld"}}