Frame selection is the whole game: notes from making LLMs watch video

There's a quiet argument buried in a new open-source project called claude-real-video, and it goes like this: an article about a video is already someone else's edit. A human watched the thing, decided what mattered, and threw the rest away. Feed a model that writeup and it inherits the author's blind spots along with the author's conclusions. Feed it the actual video, and the compression decision moves to the model itself — it can catch the error message that flashed by uncredited, or notice that a "quick setup" demo actually took eleven cuts.

The catch is cost. Images eat context window fast, so once you accept a budget of roughly 100 to 150 frames per video, extraction stops mattering and selection becomes the whole problem. Most existing approaches just sample on a timer — one frame per second or per ten — which fails in both directions: it drowns the model in near-duplicate frames of someone talking, and it walks right past the one second where something happened. The tool's fix starts with ffmpeg scene detection, but with a twist: instead of a fixed threshold, it computes per-frame scores in a metadata pass and keeps a frame only when it beats a multiple of the rolling average. Slow pans and squash-and-stretch animation, which never cross a fixed line, still get caught this way, because busy footage raises its own bar and quiet footage lowers it.

Deduplication turned into three separate channels, each one built to fix a failure the last one missed. A global channel compares 16×16 RGB signatures against the last four kept frames, not just the previous one, so interview-reaction-interview cutting doesn't quietly re-admit a shot the model already saw. An action channel exists because a person taking up a sliver of a wide shot can never move enough pixels to trip a percentage threshold — that gap wasn't found in a benchmark, it turned up after someone ran the tool on 2,181 real videos, and the fix flags a frame as new if even a handful of cells on a 32×32 grid change sharply. A third "settled" channel catches things the other two are blind to entirely: a caption swap, a line of ink on a whiteboard, a UI update. It only fires when a scene is otherwise static, checks a 192×192 grid with a small pixel-shift tolerance for film grain, and raises its own gate with a cooldown after every trigger, so one flickering element can't eat the whole frame budget.

Text comes from embedded subtitles or local Whisper, and here the debugging cost showed up in the seams. Whisper occasionally hallucinates a caption sitting past the actual end of a music-only clip, so every timestamp now gets clamped to media duration plus a second of slack. And rather than trusting an LLM to line up frames against transcript lines by eyeballing timestamps — which works on short clips and quietly drifts on long ones — the tool precomputes the join itself, attaching frames to whichever transcript segment they fall inside and marking silences over 1.5 seconds as frame-only.

The output is deliberately unglamorous: a folder of JPEGs, a transcript.txt, and a manifest telling any model how to read the package. It also ships as an MCP server — pip install, then crv-mcp — so tools like Claude Desktop or Cursor can call a watch_video function on a URL and get the fused package back, with frames resized to 768px and results cached so a follow-up question doesn't re-download or re-extract anything. What it still can't capture is camera movement, cutting rhythm, or vocal tone — that's flagged as a separate, unsolved problem. But for getting a model to actually look at something before answering, this covers a lot of ground, running entirely on your own machine.