Video DETOX: Purifying Noisy Relevance Signals for Diverse and Long-Form Video Understanding

Citations

WEB OF SCIENCE

0
Citations

SCOPUS

0

초록

Efficiently identifying semantic keyframes in long, untrimmed videos remains a fundamental bottleneck for Video Question Answering (VideoQA). The primary challenge lies in the “Information-Sparsity Dilemma”, where critical events occupy only a small fraction of the total duration, yet are buried under massive amounts of redundant visual data. While standard uniform sampling is content-blind, naive Top-k selection based on frame-text similarity suffers from temporal clumping, resulting in redundant frames that lack narrative coverage. Furthermore, most existing methods rely on static frame budgets, failing to adapt to the varying information density of different videos. To address these issues, we propose DETOX (DEnoising and Temporal Optimization for eXtraction), a training-free and diversity-aware sampling strategy that reinterprets the frame-query relevance trajectory as a continuous 1D signal. Our method refines raw relevance scores through a three-stage pipeline: (1) Signal Restoration utilizes 1D anisotropic diffusion to suppress the semantic jitter inherent in zero-shot alignment while explicitly preserving sharp event boundaries; (2) Hybrid Adaptive Proposal synergizes Rosin’s unimodal thresholding with local adaptive filters to autonomously determine the optimal frame budget, rescuing subtle narrative shifts that global thresholds might overlook; and (3) Temporal MMR Selection formulates the final extraction as a Maximal Marginal Relevance optimization to maximize semantic intensity while strictly penalizing chronological redundancy. Experimental results on challenging benchmarks, including LongVideoBench and Video-MME, demonstrate that DETOX enhances downstream reasoning by providing a high-fidelity visual summary, maintaining competitive accuracy with significantly fewer visual tokens than fixed-budget baselines to effectively mitigate the lost-in-the-middle phenomenon.

키워드

Anisotropic Diffusion; Keyframe Sampling; Long Video Understanding; Signal Denoising; Temporal MMR
제목
Video DETOX: Purifying Noisy Relevance Signals for Diverse and Long-Form Video Understanding
저자
Han, Sungjin; Ma, Minuk; Pham, Trung X.; Kim, Junyeong
DOI
10.1007/978-3-032-31930-2_35
발행일
2027
유형
Proceedings Paper
저널명
Lecture Notes in Computer Science
권
16825
페이지
523 ~ 537