Koraiio

Thumbnails and GIF previews that match each viewer's taste

Previews sell videos. This system picks the moments a particular viewer likes in a full-length sports video and turns them into a thumbnail and a short animated preview, using a fraction of the computing of earlier methods.

Year
2023
Proof
Published in IEEE Transactions on Big Data
Runs on
NVIDIA Jetson TX2
FRC-GIF process. Input videos such as football, basketball and tennis; user selection of movie and scene; download LTC; extract and rank thumbnails; recognize events; key thumbnails ordering producing personalized thumbnails; then download segments and generate animated GIFs
Fig. 1 From the viewer's choice to personal thumbnails and GIFs, in two stages.

The challenge

A single generic thumbnail is shown to everyone, and existing ways to make highlight GIFs process the entire video, which is slow and costly on everyday devices.

What we built

  1. The viewer's device recognizes the moments they prefer from the stream's preview images.
  2. It ranks those moments and keeps the best as personal thumbnails.
  3. It downloads only those few seconds of video and turns them into animated GIFs.

Results

  • About 3.7 times less computing than the previous best method (3.73 times in the journal paper, 3.78 times in the conference version on an NVIDIA Jetson TX2).
  • Less data downloaded and stored, because only short segments are fetched.

Where this applies

  • Video platforms that want better click-through on recommendations
  • Sports apps and clip-sharing products
  • Low-power devices that generate previews locally

Under the hood

  • ConvNeXt backbone with triplet attention classifies thumbnails; frame rank pooling orders them.
  • Builds on the SIGMAP 2022 method (Xception backbone with an attention module).
  • Paper - Mujtaba, Khowaja, Jarwar, Choi and Ryu, IEEE Transactions on Big Data, 2023.

ConvNeXtTriplet attentionFrame rank poolingHLSNVIDIA Jetson TX2

CNN architecture with a ConvNeXt backbone, a triplet attention module, global average pooling and dense layers that classify scenes such as soccer shot, soccer juggling and running
Fig. 2 The vision model that recognizes scenes in each preview image.
The earlier SIGMAP 2022 method. A thumbnail container analyzer module extracts thumbnails, recognizes events and writes timestamps; a GIF generation module downloads and processes segments into artistic static thumbnails and animated GIFs
Fig. 3 The earlier conference version of the same idea.

Have a perception problem that has to run on real hardware?

Tell us what you need to detect, where it has to run and what it runs on. You'll get a reply from the founder.

Email gmujtaba@koraiio.com

Email
gmujtaba@koraiio.com
Founder
Dr. Ghulam Mujtaba
Product
DroSonic
Based in
Denver, Colorado