Emotion Recognition for Multimodal Physiological Signals from a Deep Learning Perspective: A Survey

Abstract

Emotion recognition constitutes a pivotal technology for enabling natural human-computer interaction and intelligent healthcare. In contrast to external behavioral cues such as facial expressions and speech, which are susceptible to subjective masking, multimodal physiological signals offer a more objective and stable reflection of genuine emotional states mediated by the autonomic nervous system. To explore mechanisms of multimodal fusion and refine related methodological systems, this review adopts a deep learning perspective, focusing on emot ion recognition driven by physiological signals. Consequently, a two-dimensional analytical framework integrating deep learning model architectures and multimodal fusion strategies is constructed. Firstly, we review the theoretical foundations of multimodal physiological signals and summarize mainstream emotion assessment standards and experimental datasets. Then, we systematically examine feature extraction techniques and multimodal fusion strategies. Specifically, for complex hybrid fusion models, this study classifies them into three distinct types - shared semantic representation, correlation-constrained representation, and deep structured fusion, and conducts an in-depth analysis of their potential mechanisms in addressing core challenges such as cross-modal semantic alignment and modality complementarity. Next, motivated by the practical challenges of data and label scarcity and insufficient interpretability in real-world scenarios, this paper systematically analyzes the key design paradigms of state-of-the-art models for alleviating data scarcity, enhancing weakly supervised representation learning, and improving decision trustworthiness. Finally, it outlines four future directions: continuous, probabilistic, and context-aware emotion supervision; adaptive learning under unknown missingness and conflict patterns; transferable and scalable multimodal physiological foundation models; and uncertainty-aware, trustworthy decision models. Compared with existing surveys, this review presents a more comprehensive categorization and synthesis of hybrid multimodal fusion models, together with a task-oriented analysis of deep learning–driven multimodal physiological emotion recognition studies under diverse practical scenarios, thereby offering a clearer and more systematic perspective on the underlying mechanisms, methodological evolution, and future research directions in this field.

Keywords

Emotion Recognition / Affective Computing / Multimodal Fusion / Deep Learning / Physiological Signals

Rights & permissions

Higher Education Press 2026

Just Accepted

This article has successfully passed peer review and final editorial review, and will soon enter typesetting, proofreading and other publishing processes. The currently displayed version is the accepted final manuscript. The officially published version will be updated with format, DOI and citation information upon launch. We recommend that you pay attention to subsequent journal notifications and preferentially cite the officially published version. Thank you for your support and cooperation.