Finding the best moment in a video by listening to it
Sound often tells you where the big moment is faster than pictures do. This system reads a music video's audio track, finds its climax and turns just that moment into a preview.
The challenge
Making a preview usually means analyzing every frame of the video, which is slow and expensive on a phone or set-top box.
What we built
- The device reads only the audio track and recognizes the type of music.
- It finds the climax of the song, where the pitch peaks.
- It downloads just that short piece of video and turns it into an animated preview.
Results
- About 3.8 times less computing than the comparison method (3.76 times, tested on 16 videos on an NVIDIA Jetson TX2).
- In a viewer study, the previews were rated higher overall.
- An early step toward the audio AI behind DroSonic.
Where this applies
- Music and video platforms that need cheap, automatic previews
- Finding key moments in long recordings from sound alone
- Devices where video analysis is too heavy
Under the hood
- A CNN classifies music genre from a spectrogram of the soundtrack; pitch analysis in the climax section gives the timestamp.
- Only the matching HLS segment is fetched to make the GIF.
- Paper - Mujtaba, Lee, Kim and Ryu, Multimedia Tools and Applications, 2021.
Spectrogram CNNPitch analysisHLSNVIDIA Jetson TX2
