Video-Centered Multimodal Learning for Non-Invasive Detection of Emotional Workload in Emotional Labor Settings

초록

Workers in emotional labor environments often follow display rules that require them to express emotions as directed by their boss or the company, regardless of their true feelings. These workers suppress their true feelings, which leads to ongoing mental issues and an accumulation of emotional workload. For example, in a call center situation, if a customer is shouting and making accusatory comments at the worker, the worker should respond in a positive way to the customer, even if they have negative feelings. However, it still remains a difficult issue to estimate emotional workload in real-world settings using non-invasive methods. In this study, we propose a machine learning model that automatically detects workers’ emotional workload in emotional labor situations using multimodal data, including video and physiological data. We used a dataset designed based on a call center scenario and estimated emotional workload using two labels: an objective label based on experimental conditions, and a subjective label based on workers’ self-reports. As a result, we found that geometric features derived from facial expressions captured in videos significantly contributed to model performance and that applying Multiple Kernel Learning (MKL) improved classification accuracy, especially when detecting subjective emotional workload. This study demonstrates the potential of image-based and MKL-based models for scalable and non-invasive detection of emotional workload, suggesting that they can be effectively applied in real-world work settings.

키워드