A local vision model that reads your footage and tells you what is in it

A folder of two hundred clips with camera-generated filenames is functionally opaque. You know roughly what is in there. You do not know which clip has the wide of the room, or where the usable take of the third setup is, and finding out means scrubbing.

Logging solves it and nobody does it, because doing it properly costs hours at exactly the moment you have none.

A local vision model closes that gap. It watches every clip, describes what it sees, and hands you a searchable list — for the cost of leaving your machine on while you do something else.

The pipeline is three stages and each one is simple:

1. Extract frames. Not every frame — one every few seconds, or better, one per detected shot change.
2. Describe each frame. Send it to a local vision model with a prompt that asks for the specific things you need.
3. Collapse to a clip summary. Several frame descriptions become one entry per clip.

ffmpeg -i clip.mov -vf "select='gt(scene,0.3)'" -vsync vfr frame_%03d.jpg

The scene threshold is the one number worth tuning. Too low and you get twenty near-identical frames from a static shot; too high and you miss a cut.

The difference between a useful log and a pile of paragraphs is entirely in how you ask. Request named fields:

Describe this frame. Answer only these fields:
SHOT: wide / medium / close
SUBJECT: what is in frame
LOCATION: interior or exterior, what kind of space
LIGHT: direction and quality
MOTION: is the camera moving
USABLE: yes / no / marginal

Now the output is parseable. You can filter for every exterior wide, or every marginal take, without reading anything.

Keep the field list short. A small model asked for twelve fields answers six of them well and invents the rest.

This is exactly the task a small local vision model handles well. It does not need to be clever; it needs to look at a picture and name what is in it, and it needs to do that the same way two hundred times.

Consistency beats capability here. A model that describes a wide shot as 'wide' every time is more useful than a better one that alternates between 'wide', 'establishing' and 'full shot' — because you are going to search this text later, and search does not know those are the same thing.

Constraining the vocabulary in the prompt, as above, is how you enforce that regardless of which model you run.

It cannot judge a performance. It will tell you a person is speaking. It will not tell you which take was good, and asking makes it confidently invent an opinion. Performance selection stays human, permanently.

It has no memory between frames. Continuity, story order and whether this is the same person as the previous clip are outside what it can see. Anything that depends on comparing across clips has to be done in your own code, on top of its output.

The description file is the interesting artefact, not the descriptions themselves. With one, you can filter a folder into a rough assembly, build a bin structure automatically, or hand an agent a shot list it can reason about without watching anything.

None of that is possible while the footage is opaque. Logging is the step everyone skips that everything else depends on — which is precisely why it is worth handing to a machine that does not mind doing it.

Run it once on an old project. The list you get back will tell you immediately whether the field prompt needs adjusting, and it costs you nothing but electricity.