Action-Concentrated Embedding Framework: This is your captain sign-tokening

Citations

SCOPUS

1

초록

Sign language is the primary communication medium for people who are deaf or have hearing loss. However, given the divergent range of sensory abilities of these individuals, there is a communication gap that needs to be addressed. In this paper, we present action-concentrated embedding (ACE), which is a novel sign token embedding framework. Additionally, to provide a more structured foundation for sign language analysis, we introduce a dedicated notation system tailored for sign language that endeavors to encapsulate the nuanced gestures and movements that are integral with sign communication. The proposed ACE approach tracks a signer's actions based on human posture estimation. Tokenizing these actions and capturing the token embedding using a short-time Fourier transform encapsulates the time-based behavioral changes. Hence, ACE offers input embedding to translate sign language into natural language sentences. When tested against a disaster sign language dataset using automated machine translation measures, ACE notably surpasses prior research in terms of translation capabilities, improving the performance by up to 5.79% for BLEU-4 and 5.46% for ROUGE-L metric. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

키워드

short-time Fourier transform; sign language token embedding framework; sign language translation
제목
Action-Concentrated Embedding Framework: This is your captain sign-tokening
저자
Yu, Hyunwook; Shin, Suhyeon; Heo, Jungu; Shin, Hyuntaek; Kim, Hyosu; Kim, Mucheol
발행일
2024
유형
Conference paper
저널명
2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings
페이지
310 ~ 320