Frame selection is the whole game: notes from making LLMs watch video

leoaido.com · 3 weeks ago · 25

A developer describes techniques for efficiently extracting keyframes from videos for LLM analysis, using adaptive scene detection and three-channel deduplication to select the most informative frames within a token budget of 100–150 frames per video. Key technical details include per-frame scene scores compared against rolling averages, RGB-based global deduplication at 16×16 resolution, a separate action channel that catches small subjects via 32×32 grid analysis, and a settled channel at 192×192 resolution for UI and text changes. The result is an open-source tool (claude-real-video) that outputs JPEGs, transcripts, and a manifest file, available as an MCP server, allowing models to analyze video content directly rather than relying on human summaries.