A personal highlight reel for every viewer, without a server farm
Each viewer's device finds the moments they care about in a full-length film or match and builds their own summary, so a service can offer personal highlights to many viewers at once without processing video for each of them.
The challenge
Making a summary on the server means decoding the whole video again for every viewer. That cost grows with every user, and it requires knowing what each person likes.
What we built
- The viewer's device scans the small preview images that come with the stream.
- A compact model recognizes events, such as a shot on goal, in those previews.
- The device downloads only the matching moments and stitches them into a personal summary.
Results
- Tested on 18 full-length videos, about 33 hours of content; much lighter to run than leading methods on the same devices.
- In a study with 56 people, viewers rated these summaries higher.
- Code and trained models are open source.
Where this applies
- Streaming services with many concurrent viewers
- Sports and event video archives
- Products that must keep viewing preferences private
Under the hood
- EfficientNet-B0 features with an attention module classify events in each thumbnail; transfer learning keeps the model small.
- Event timestamps select HLS segments; the client assembles a keyshot summary. Prototype client - NVIDIA Jetson TX2.
- Paper - Mujtaba, Malik and Ryu, IEEE Access, vol. 10, 2022 (open access).
PythonEfficientNet-B0AttentionFFmpegHLSNVIDIA Jetson TX2
