Learning to See through Sound: From VggCaps to Multi2Cap for Richer Automated Audio Captioning

  • Cho, Sangyeon; 
  • Kim, Mingi; 
  • Hwang, Jinkown; 
  • Go, Jaehoon; 
  • Ma, Minuk; 
  • ... Kim, Junyeong; 
  • 외 1명
Citations

SCOPUS

0

초록

Automated Audio Captioning (AAC) aims to generate natural language descriptions of audio content, enabling machines to interpret and communicate complex acoustic scenes. However, current AAC datasets often suffer from short and simplistic captions, limiting model expressiveness and semantic depth. To address this, we introduce VggCaps, a new multimodal dataset that pairs audio with corresponding video and leverages large language models (LLMs) to generate rich, descriptive captions. VggCaps significantly outperforms existing benchmarks in caption length, lexical diversity, and human-rated quality. Furthermore, we propose Multi2Cap, a novel AAC framework that learns audio-visual representations through a AV-grounding module during pre-training and reconstructs visual semantics using audio alone at inference. This enables visually grounded captioning in audio-only scenarios. Experimental results on Clotho and AudioCaps demonstrate that Multi2Cap achieves state-of-the-art performance across multiple metrics, validating the effectiveness of cross-modal supervision and LLM-based generation in advancing AAC.

제목
Learning to See through Sound: From VggCaps to Multi2Cap for Richer Automated Audio Captioning
저자
Cho, Sangyeon; Kim, Mingi; Hwang, Jinkown; Go, Jaehoon; Ma, Minuk; Yoon, Sunjae; Kim, Junyeong
DOI
10.18653/v1/2025.emnlp-main.715
발행일
2025
유형
Conference Paper
저널명
EMNLP 2025 - 2025 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
페이지
14157 ~ 14175