상세 보기
TBGA: Tripartite and Bipartite Graph Attention for Multi-modal Video Question Answering
SCOPUS
0초록
Multi-modal Video Question Answering (MVQA) is a complex and emerging task at the intersection of computer vision and natural language processing, which requires models to comprehend and reason over multiple synchronized information streams including video, audio, and textual transcripts to correctly answer natural language questions. Despite recent progress, many existing approaches rely on late-stage fusion mechanisms or learn shallow co-occurrence patterns between modalities and questions. These limitations hinder their ability to capture nuanced, fine-grained semantic interactions, especially in challenging benchmarks such as Social-IQ, which focus on social reasoning and subtle contextual understanding. In this work, we propose Tripartite and Bipartite Graph Attention (TBGA), a novel graph-based architecture specifically designed to address the semantic alignment challenges inherent in MVQA. TVGA employs a dual-graph strategy: a tripartite graph is used to explicitly model inter-modal relationships among video, audio, and transcript representations, while bipartite graphs are constructed between question tokens and each modality to enable question-aware semantic encoding. This design facilitates structured reasoning across heterogeneous modalities and ensures that information is integrated in a semantically coherent and interpretable manner. We validate the effectiveness of our approach through experiments on Social-IQ dataset, a benchmark designed to test high-level social and emotional reasoning in videos. Our model achieves state-of-the-art performance, significantly outperforming prior methods. These results demonstrate the potential of structured graph-based reasoning in advancing multi-modal understanding and highlight the benefits of explicit cross-modal alignment in complex video reasoning tasks.
키워드
- 제목
- TBGA: Tripartite and Bipartite Graph Attention for Multi-modal Video Question Answering
- 저자
- Kim, Junyeong
- 발행일
- 2025
- 유형
- Conference Paper
- 저널명
- Proceedings, International Conference on Electrical, Control and Instrumentation Engineering, ICECIE
- 페이지
- 256 ~ 262
- 언어
- ENG
- 출판사
- Institute of Electrical and Electronics Engineers
- 분량
- 7 페이지
- ISSN
- E 2832-9848
P 2832-9821