How we index clips by what happens in them

Classic stock search works on metadata: a title, a description and twenty or so keywords the uploader typed at 2 a.m. That’s a decent signal for what a clip is. It says nothing about when something happens inside it — and for editors, “when” is the whole point.

Moment search, which we shipped in beta in 2025 and rebuilt as 2.0 this autumn, indexes clips by what is on screen at every half-second.

Step 1: cut the clip into shots

Most uploads are a single shot, but about a fifth contain cuts, fades or title cards. We run a lightweight shot-boundary detector first so that each shot gets its own embeddings. Without this step, a 10-second clip with a cut at 4 s would produce muddy, averaged results.

Step 2: describe every half-second

For each shot we sample two frames per second and compute:

• a visual embedding from an image–text model fine-tuned on 2.3 million editor-written captions;
• a motion descriptor from dense optical flow — direction, speed and whether the camera or the subject is moving;
• a colour signature: dominant hues, contrast and temperature;
• an audio embedding if the clip has sound.

Motion is the part people underestimate. “Push-in” and “pull-out” look identical in a single frame; you only see the difference across time. Our motion descriptors are what let queries like “waveform drifts upward” work.

Step 3: index segments, not clips

Each half-second becomes a segment in a vector index. 41 million clips at an average of 14 seconds is roughly 1.1 billion segments — too many to search naively. We quantise vectors to 64 bytes, shard by category, and keep a small in-memory graph index per shard. A query fans out to all shards and the top segments are merged and re-ranked.

Stage	p50	p95
Query encoding	6 ms	11 ms
Shard fan-out + ANN	19 ms	34 ms
Merge & re-rank	9 ms	17 ms
Total (server)	38 ms	61 ms

Step 4: rank moments, then clips

Adjacent matching segments are merged into ranges (“00:06.2 – 00:06.9”), and the clip’s score is a blend of its best range, its quality signals and how often people who ran similar searches actually used it. That last signal is aggregated and anonymous — we never build per-person search profiles.

What’s next

We’re working on query-by-clip: drop in a few seconds of your own footage and find moments that would cut well next to it. If you’d like to test it early, Team customers can ask their account manager to join the beta.