Integrating emotion dynamics in mental health: A trimodal framework combining ecological momentary assessment, physiological measurements, and speech emotion recognition

Abstract

The complex relationship between emotions and mental health demands a more comprehensive theoretical framework that can capture its dynamic and multifaceted nature. This perspective article proposes a novel trimodal approach that conceptually integrates three complementary methodologies: Ecological Momentary Assessment, physiological measurements, and Speech Emotion Recognition. By adopting a dynamical system perspective, we argue that the convergence of these methodologies could provide unprecedented insights into emotional dynamics in mental health research and practice. We discuss how this framework could transform our understanding by simultaneously capturing subjective experiences, physiological responses, and linguistic patterns in naturalistic settings. The proposed integration offers a conceptual foundation for developing more sophisticated approaches to mental health monitoring and intervention. We explore the theoretical implications, methodological considerations, and potential future directions of this integrated perspective, highlighting its promise for advancing both research and clinical practice in mental health.

Keywords

biosensor / dynamical system / ecological momentary assessment / EMA / emotion / speech emotion recognition

References

[1] J. J. Gross, H. Uusberg, A. Uusberg, World Psychiatry 2019, 18, 130.

[2] P. Saxena, A. Dubey, R. Pandey, SIS J. Proj. Psychol. Ment. Health 2011, 18, 147.

[3] Y. Xia, P. Wang, J. Vincent, Brain-X 2024, 2, e70.

[4] P. Wang, Z. Wang, S. Qiu, Child Adolesc. Psychiatr. Ment. Health 2024, 18, 47.

[5] M. Scheffer, C. L. Bockting, D. Borsboom, R. Cools, C. Delecroix, J. A. Hartmann, K. S. Kendler, I. van de Leemput, H. L. J. van der Maas, E. van Nes, M. Mattson, P. D. McGorry, B. Nelson, JAMA Psychiatry 2024, 81, 618.

[6] P. Wang, L. Ganushchak, C. Welie, R. van Steensel, Educ. Psychol. Rev. 2024, 36, 105.

[7] M. Houben, W. Van Den Noortgate, P. Kuppens, Psychol. Bull. 2015, 141, 901.

[8] J. E. LeDoux, S. G. Hofmann, Curr. Opin. Behav. Sci. 2018, 19, 67.

[9] E. F. Pace-Schott, M. C. Amole, T. Aue, M. Balconi, L. M. Bylsma, H. Critchley, H. A. Demaree, B. H. Friedman, A. E. K. Gooding, O. Gosseries, T. Jovanovic, L. A. Kirby, K. Kozlowska, S. Laureys, L. Lowe, K. Magee, M. F. Marin, A. R. Merner, J. L. Robinson, R. C. Smith, D. P. Spangler, M. Van Overveld, M. B. VanElzakker, Neurosci. Biobehav. Rev. 2019, 103, 267.

[10] S. G. Koolagudi, K. S. Rao, Int. J. Speech Technol. 2012, 15, 99.

[11] J. O. Kephart, T. Hogg, B. A. Huberman, Phys. Rev. 1989, 40, 404.

[12] R. E. Ulanowicz, Ecol. Model. 2009, 220, 1886.

[13] B. Nelson, P. D. McGorry, M. Wichers, J. T. W. Wigman, J. A. Hartmann, JAMA Psychiatry 2017, 74, 528.

[14] M. Scheffer, C. L. Bockting, D. Borsboom, R. Cools, C. Delecroix, J. A. Hartmann, K. S. Kendler, I. van de Leemput, H. L. J. van der Maas, E. van Nes, M. Mattson, P. D. McGorry, B. Nelson, JAMA Psychiatry 2024, 81, 624.

[15] M. Chen, S.-M. Chow, Z. Hammal, D. S. Messinger, J. F. Cohn, Multivariate Behav. Res. 2021, 56, 739.

[16] M. J. Crawford, L. Thana, L. Farquharson, L. Palmer, E. Hancock, P. Bassett, J. Clarke, G. D. Parry, Br. J. Psychiatry 2016, 208, 260.

[17] S. M. Southwick, M. Vythilingam, D. S. Charney, Annu. Rev. Clin. Psychol. 2005, 1, 255.

[18] F. Etilé, P. Frijters, D. W. Johnston, M. A. Shields, J. Econ. Behav. Organ. 2021, 191, 598.

[19] C. A. Padesky, K. A. Mooney, Clin. Psychol. Psychother. 2012, 19, 283.

[20] X. Yang, N. Ram, J. P. Lougheed, P. C. M. Molenaar, T. Hollenstein, Dev. Psychol. 2019, 55, 1982.

[21] H. Pan, J. Niu, L. Feng, Y. Yin, C. Dang, Y. Lu, L. Li, J. Ji, K. Yang, L. Wang, Q. Li, Brain-X 2024, 2, e58.

[22] N. Silvis-Cividjian, J. Kenyon, E. Nazarian, S. Sluis, M. Gevonden, in Proceedings of the 2024 on Innovation and Technology in Computer Science Education V 1, Association for Computing Machinery, Milan, Italy 2024, pp. 429-435.

[23] A. Elbayoudi, A. Lotfi, C. Langensiepen, Expert Syst. Appl. 2019, 118, 493.

[24] E. J. C. de Geus, G. H. M. Willemsen, C. H. A. M. Klaver, L. J. P. van Doornen, Biol. Psychol. 1995, 41, 205.

[25] J. F. Kihlstrom, Perspect. Psychol. Sci. 2021, 16, 466.

[26] P. Wang, L. Cheng, Brain-X 2024, 2, e53.

[27] S. Shiffman, A. A. Stone, M. R. Hufford, Annu. Rev. Clin. Psychol. 2008, 4, 1.

[28] S. P. Lee, G. Ha, D. E. Wright, Y. Ma, E. Sen-Gupta, N. R. Haubrich, P. C. Branche, W. Li, G. L. Huppert, M. Johnson, H. B. Mutlu, K. Li, N. Sheth, J. A. Wright,, Y. Huang, M. Mansour, J. A. Rogers, R. Ghaffari, NPJ Digital Med. 2018, 1, 2.

[29] Y. Lin, M. Bariya, A. Javey, Adv. Funct. Mater. 2021, 31, 2008087.

[30] S. L. Rizvi, A. K. Ruork, Q. Yin, A. Yeager, M. E. Taylor, E. M. Kleiman, JMIR Ment. Health 2024, 11, e60035.

[31] V. L. Richards, N. P. Barnett, R. L. Cook, R. F. Leeman, T. Souza, S. Case, C. Prins, C. Cook, Y. Wang, Alcohol, Clin. Exp. Res. 2023, 47, 308.

[32] D. W. Murray, T. A. Ridenour, M. M. Swingler, A. Morgan, M. Hegarty-Craver, Stress Health 2023, 39, 684.

[33] M. J. Hinich, J. Time Anal. 1982, 3, 169.

[34] S. Zhang, R. Feng, Y.-N. Fu, Q. Liu, Y. He, O. Turel, Q. He, Pers. Indiv. Differ. 2022, 197, 111784.

[35] J. M. B. Haslbeck, L. F. Bringmann, L. J. Waldorp, Multivariate Behav. Res. 2021, 56, 120.

[36] L. F. Bringmann, E. Ferrer, E. L. Hamaker, D. Borsboom, F. Tuerlinckx, Multivariate Behav. Res. 2018, 53, 293.

[37] L. Han, X.-Y. Chen, H.-J. Ye, D.-C. Zhan, (Preprint) arXiv, 2404.14197, v3. 2024.

[38] T. Waaramaa-Mäki-Kulmala, in Acoustic and Perceptual Analysis of Voice Quality in the Vocal Expression of Emotions, Tampere University Press 2009.

[39] M. Malgaroli, T. D. Hull, J. M. Zech, T. Althoff, Transl. Psychiatry 2023, 13, 309.

[40] J. Deng, F. Ren, IEEE Trans. Affective Comput. 2023, 14, 49.

[41] M. Jain, S. Narayan, P. Balaji, A. Bhowmick, R. K. Muthu, (Preprint) arXiv, 2002.07590, v1. 2020.

[42] M. Vondra, R. Vích, in Multimodal Signals: Cognitive and Algorithmic Issues, Vol. 5398 (Eds: A. Esposito, A. Hussain, M. Marinaro, R. Martone), Springer, Berlin, Heidelberg 2009, pp. 256-263.

[43] J. Umamaheswari, A. Akila, in 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), IEEE, Faridabad, India 2019, pp. 177-183.

[44] S. P. Yadav, S. Zaidi, A. Mishra, V. Yadav, Arch. Comput. Methods Eng. 2022, 29, 1753.

[45] D. Li, J. Liu, Z. Yang, L. Sun, Z. Wang, Expert Syst. Appl. 2021, 173, 114683.

[46] G. Tripoli, D. Quattrone, L. Ferraro, C. Gayer-Anderson, C. La Cascia, D. La Barbera, C. Sartorio, F. Seminerio, V. Rodriguez, I. Tarricone, D. Berardi, S. Jamain, C. Arango, A. Tortelli, P. M. Llorca, L. de Haan, E. Velthorst, J. Bobes, M. Bernardo, J. Sanjuán, J. Luis Santos, M. Arrojo, C. Marta Del-Ben, P. Rossi Menezes, E. van der Ven, P. B. Jones, H. E. Jongsma, J. B. Kirkbride, S. Tosato, A. Lasalvia, A. Richards, M. O’Donovan, B. P. F. Rutten, J. van Os, C. Morgan, P. C. Sham, M. Di Forti, R. M. Murray, G. K. Murray, Schizophr. Bull. 2022, 48, 1104.

[47] K. S. Goerlich, A. Aleman, S. Martens, PLoS One 2012, 7, e36951.

[48] J. M. Murphy, J. M. Bennett, X. de la Piedad Garcia, M. L. Willis, Neuropsychol. Rev. 2022, 32, 520.

[49] A. Roefs, E. I. Fried, M. Kindt, C. Martijn, B. Elzinga, A. W. Evers, R. W. Wiers, D. Borsboom, A. Jansen, Behav. Res. Ther. 2022, 153, 104096.

[50] J. Dennison, Migration Stud. 2024, 12, 1.

[51] D. A. Adler, D. Ben-Zeev, V. W.-S. Tseng, J. M. Kane, R. Brian, A. T. Campbell, M. Hauser, E. A. Scherer, T. Choudhury, JMIR MHealth UHealth. 2020, 8, e19962.

[52] K. Alexander, P. Sanjuan, M. Terplan, Curr. Addict. Rep. 2023, 10, 366.

[53] D. D. L. Coppersmith, O. Ryan, R. G. Fortgang, A. J. Millner, E. M. Kleiman, M. K. Nock, Proc. Natl. Acad. Sci. USA 2023, 120, e2215434120.

[54] M. Mestdagh, S. Verdonck, M. Piot, K. Niemeijer, G. Kilani, F. Tuerlinckx, P. Kuppens, E. Dejonckheere, Front. Digital Health 2023, 5, 1182175.

[55] B. M. Althouse, S. V. Scarpino, L. A. Meyers, J. W. Ayers, M. Bargsten, J. Baumbach, J. S. Brownstein, L. Castro, H. Clapham, D. A. Cummings, S. Del Valle, S. Eubank, G. Fairchild, L. Finelli, N. Generous, D. George, D. R. Harper, L. Hébert-Dufresne, M. A. Johansson, K. Konty, M. Lipsitch, G. Milinovich, J. D. Miller, E. O. Nsoesie, D. R. Olson, M. Paul, P. M. Polgreen, R. Priedhorsky, J. M. Read, I. Rodríguez-Barraquer, D. J. Smith, C. Stefansen, D. L. Swerdlow, D. Thompson, A. Vespignani, A. Wesolowski, EPJ Data Sci. 2015, 4, 17.

[56] V. Tsouvalas, T. Ozcelebi, N. Meratnia, in 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), IEEE, Pisa, Italy 2022, pp. 359-364.

[57] A. Katirai, AI Ethics 2024, 4, 927.

[58] C. Xu, Y. Song, J. R. Sempionatto, S. A. Solomon, Y. Yu, H. Y. Y. Nyein, R. Y. Tay, J. Li, W. Heng, J. Min, A. Lao, T. K. Hsiai, J. A. Sumner, W. Gao, Nat. Electron 2024, 7, 168.

[59] N. Brasier, J. Wang, W. Gao, J. R. Sempionatto, C. Dincer, H. C. Ates, F. Güder, S. Olenik, I. Schauwecker, D. Schaffarczyk, E. Vayena, N. Ritz, M. Weisser, S. Mtenga, R. Ghaffari, J. A. Rogers, J. Goldhahn, Nature 2024, 636, 57.

[60] C. Chen, H. Wang, Y. Chen, Z. Yin, X. Yang, H. Ning, Q. Zhang, W. Li, R. Xiao, J. Zhao, Brain-X 2023, 1, e29.

[61] S. Cong, H. Wang, Y. Zhou, Z. Wang, X. Yao, C. Yang, Brain-X 2024, 2, e57.

RIGHTS & PERMISSIONS

2025 The Author(s). Interdisciplinary Medicine published by Wiley-VCH GmbH on behalf of Nanfang Hospital, Southern Medical University.