WEAKLY-SUPERVISED MOMENT RETRIEVAL NETWORK FOR VIDEO CORPUS MOMENT RETRIEVAL

  • Yoon, Sunjae; 
  • Kim, Dahyun; 
  • Hong, Ji Woo; 
  • Kim, Junyeong; 
  • Kim, Kookhoi; 
  • 외 1명
Citations

WEB OF SCIENCE

6
Citations

SCOPUS

7

초록

This paper proposes Weakly-supervised Moment Retrieval Network (WMRN) for Video Corpus Moment Retrieval (VCMR), which retrieves pertinent temporal moments related to natural language query in a large video corpus. Previous methods for VCMR require full supervision of temporal boundary information for training, which involves a labor-intensive process of annotating the boundaries in a large number of videos. To leverage this, the proposed WMRN performs VCMR in a weakly-supervised manner, where WMRN is learned without ground-truth labels but only with video and text queries. For weakly-supervised VCMR, WMRN addresses the following two limitations of prior methods: (1) Blurry attention over video features due to redundant video candidate proposals generation, (2) Insufficient learning due to weak supervision only with video-query pairs. To this end, WMRN is based on (1) Text Guided Proposal Generation (TGPG) that effectively generates text guided multi-scale video proposals in the prospective region related to query, and (2) Hard Negative Proposal Sampling (HNPS) that enhances video-language alignment via extracting negative video proposals in positive video sample for contrastive learning. Experimental results show that WMRN achieves state-of-the-art performance on TVR and DiDeMo benchmarks in the weakly-supervised setting. To validate the attainments of proposed components of WMRN, comprehensive ablation studies and qualitative analysis are conducted.

키워드

Multi-modal video corpus moment retrieval; Weakly-supervised learning
제목
WEAKLY-SUPERVISED MOMENT RETRIEVAL NETWORK FOR VIDEO CORPUS MOMENT RETRIEVAL
저자
Yoon, Sunjae; Kim, Dahyun; Hong, Ji Woo; Kim, Junyeong; Kim, Kookhoi; Yoo, Chang D.
DOI
10.1109/ICIP42928.2021.9506218
발행일
2021-09
유형
Proceedings Paper
저널명
2021 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP)
페이지
534 ~ 538