TBGA: Tripartite and Bipartite Graph Attention for Multi-modal Video Question Answering

Citations

SCOPUS

0

초록

Multi-modal Video Question Answering (MVQA) is a complex and emerging task at the intersection of computer vision and natural language processing, which requires models to comprehend and reason over multiple synchronized information streams including video, audio, and textual transcripts to correctly answer natural language questions. Despite recent progress, many existing approaches rely on late-stage fusion mechanisms or learn shallow co-occurrence patterns between modalities and questions. These limitations hinder their ability to capture nuanced, fine-grained semantic interactions, especially in challenging benchmarks such as Social-IQ, which focus on social reasoning and subtle contextual understanding. In this work, we propose Tripartite and Bipartite Graph Attention (TBGA), a novel graph-based architecture specifically designed to address the semantic alignment challenges inherent in MVQA. TVGA employs a dual-graph strategy: a tripartite graph is used to explicitly model inter-modal relationships among video, audio, and transcript representations, while bipartite graphs are constructed between question tokens and each modality to enable question-aware semantic encoding. This design facilitates structured reasoning across heterogeneous modalities and ensures that information is integrated in a semantically coherent and interpretable manner. We validate the effectiveness of our approach through experiments on Social-IQ dataset, a benchmark designed to test high-level social and emotional reasoning in videos. Our model achieves state-of-the-art performance, significantly outperforming prior methods. These results demonstrate the potential of structured graph-based reasoning in advancing multi-modal understanding and highlight the benefits of explicit cross-modal alignment in complex video reasoning tasks.

키워드

Social-IQ; Surveillance and Monitoring; Video Question Answering; Visual-Language Reasoning
제목
TBGA: Tripartite and Bipartite Graph Attention for Multi-modal Video Question Answering
저자
Kim, Junyeong
DOI
10.1109/ICECIE66637.2025.11363814
발행일
2025
유형
Conference Paper
저널명
Proceedings, International Conference on Electrical, Control and Instrumentation Engineering, ICECIE
페이지
256 ~ 262