Build awesome datasets for video generation
Hugging Face
Hugging Face dropped open-source tooling to help people build small, high-quality video datasets for fine-tuning AI video models. It matters because until now, good dataset tools were basically an image-only club.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Image generation had it easy. Tools like img2dataset turned scraping and curating massive image datasets into a solved problem years ago, with a thriving ecosystem of scripts and guides around it. Video never got that treatment, and anyone trying to fine-tune a video model has had to cobble together their own pipeline from scratch. Hugging Face contributors hlky and Sayak are trying to close that gap with a new open-source toolkit built specifically for video dataset curation, aimed at smaller-scale projects rather than the industrial-scale scraping that video2dataset already handles.
The pipeline runs in three stages, and it borrows heavily from the data practices behind Stable Video Diffusion and LTX-Video. Stage one uses yt-dlp to pull down source videos and a custom Video to Scenes script to chop long footage into usable clips. Stage two is where the filtering muscle lives: extracted frames get checked for watermarks using LAION-5B's detector, scored for aesthetics with an improved aesthetic predictor, and screened for NSFW content via Falconsai's classifier, while OpenCV estimates motion across the whole clip. Stage three brings in Microsoft's Florence-2 model to generate captions, detailed descriptions, region-level object detection, and OCR text extraction from frames, though the team notes you can swap in any captioner, including full-video captioners like Qwen2.5.
The real value here shows up in the messy details the post digs into. When the team built a dataset for finetrainers/crush-smol-v0, using captions from Qwen2VL and filtering to watermark scores under 0.1 and aesthetic scores above 5.5, the result was brutal: just 47 videos survived out of 1,493. Looking at the discarded frames revealed why the naive approach breaks. A toy car being crushed scored a watermark rating of 0.60 that dropped to 0.17 once the object was flattened, meaning a single frame requirement wiped out clips that were mostly fine. Their fix is to average scores across frames for watermark detection, with a looser threshold around 0.2 to 0.3, and to rely on just the first frame for aesthetic filtering rather than demanding every frame clear a high bar.
The aesthetic scoring has its own quirks worth flagging. A pink toy castle scored 5.5 fresh and 4.44 once crushed; an action figure went from 4.99 to 4.84. The pattern the team noticed is that round, colorful, rectangular objects tend to score higher, which means an aggressive aesthetic cutoff above 5.5 risks throwing out perfectly good training data just because it doesn't look like a toy commercial. Their recommendation is to treat aesthetic scoring more as a floor for cutting genuinely bad footage, somewhere around 4.25 to 4.5, rather than a ceiling for chasing only the prettiest frames.
All of this fed into real fine-tuning work. The team used their curated datasets, including ones nicknamed Cakeify and Crush, to fine-tune CogVideoX-5B with the finetrainers library, chasing the kind of satisfying object-crushing effects popularized by Pika. The example outputs, like a candle getting flattened under a descending metal cylinder, show the payoff of getting the filtering thresholds right instead of just cranking every knob to its strictest setting.
My take — AI-written commentary, not fact-checked reporting
This is the kind of unglamorous infrastructure work that actually moves open video generation forward, more than another flashy demo would. The honest admission that their own aesthetic and watermark thresholds initially discarded good data is the most useful part of the post, and it's a pattern I keep seeing across open AI tooling: the filters people ship first are almost always miscalibrated, and nobody talks about it publicly. More projects should publish their failure thresholds like this instead of just the highlight reel.
Read more about this at: Hugging Face
