A user opened an issue on my open-source video tool last week that named a gap I had been shipping around for months.
He runs lectures through crv so an LLM can read them. One 22-minute lecture: 1,377 candidate frames extracted, 60 kept after dedup and --max-frames
thinning. The 60 frames come out in the right order. That is all they come out with.
His complaint, in one line: the LLM can describe the slide, but it cannot tell you when the slide was on screen.
That breaks more than it sounds like:
transcript.json
segmentsThe transcript had timestamps the whole time. The frames did not.
The pipeline goes: extract with ffmpeg, drop near-identical frames, thin down to --max-frames
, rename everything to frame_001.jpg
, frame_002.jpg
.
Every one of those steps is lossy for position. Extraction writes files, dedup deletes some, thinning deletes more, renaming closes the gaps. By the time you are holding frame_012.jpg
, the only fact left in the filename is "twelfth surviving frame", and twelfth of what is no longer recoverable from the output directory.
The tempting fix is arithmetic: timestamp = frame_number / fps
. That is wrong on any variable frame rate source, which covers most screen recordings and a lot of phone video. It gives you a number that looks right and drifts.
ffmpeg already knows. The showinfo
filter prints the real PTS of every frame it passes, on the same select
pass you are already running:
-vf "select=...,showinfo"
Parse that log and you get true presentation timestamps with no second decode pass. Then you carry them: attach the PTS at extraction, keep it attached through dedup, through thinning, through the rename, and write it out next to the images as frames.json
:
{
"frames": [
{
"file": "frame_001.jpg",
"timestamp_sec": 18.42,
"timestamp": "00:00:18.420",
"selection_reason": "scene"
}
]
}
selection_reason
records which dedup channel kept the frame. That one is worth adding early: it is what you read when a frame you wanted is missing and you need to know which stage ate it.
If the showinfo
log and the extracted files ever disagree on count, the tool writes no timestamps at all rather than approximate ones.
That felt overly strict while I was writing it. It is the opposite. A missing timestamp makes the model say "I don't know when". A wrong timestamp makes it cite 00:03:41 with total confidence, and nothing downstream can catch it. In a pipeline whose entire job is handing a model verifiable evidence, a plausible wrong number is the worst thing you can emit.
The person who filed the issue re-ran his 22:12 lecture on the new build and checked it himself: 1,377 candidates down to 60 final frames, 60 entries in the mapping, all monotonic, no missing or extra image files. He replayed the full extraction pass against the original source and matched every final image back to its recorded timestamp. 60 out of 60.
I did not ask him to do that. It is the most useful thing anyone has done for this project.
If you build any extract, filter and rename pipeline that feeds an LLM, decide early where position lives. Threading an identifier through four stages is much cheaper than reconstructing it from a directory listing afterwards. And when the identifier is uncertain, emit nothing instead of something.
crv is MIT and on PyPI:
pip install -U claude-real-video
Source: https://github.com/HUANGCHIHHUNGLeo/claude-real-video
There is also a paid Pro build if you need camera motion, audio and speaker labels on top of frames and transcript: https://capafy.ai/agent/llm-real-video-pro-let-any-llm-watch-videos/5451082151?ct=devto