Stacking 3 Small ML Models for Powerful Video Search
The author built a video search prototype by stacking three open-source ML models - CLIP, Whisper, and ArcFace - to enable intelligent video search with precise results.
Why it matters
This demonstrates the power of combining multiple AI/ML models to create an intelligent video search system that feels 'magical' in its precision and effectiveness.
Key Points
- 1Broke video into shots and extracted representative thumbnails
- 2Used CLIP for image and text embedding, Whisper for speech-to-text, and ArcFace for face detection
- 3Stacking the models led to unreasonably effective video search results
Details
The author wanted to see how far open-source models could take intelligent video search, where you can type a query like 'outdoor scene with two people talking about robots' and get useful results. They used a 12-minute short film as test footage, and broke it into shots using shot boundary detection. Each shot was then processed by the three models - CLIP for image and text embedding, Whisper for speech-to-text transcription, and ArcFace for face detection. By stacking these models, the author found that the precision of the search results jumped dramatically, as each filter cut down the noise from the previous step. The architecture is deliberately simple, using PostgreSQL with the pgvector extension to store all the data in one place, enabling efficient SQL queries to filter, rank, and search the video content.
No comments yet
Be the first to comment