Fusion of visible and thermal images for facial expression recognition

Abstract

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.

Keywords

facial expression recognition / feature-level fusion / decision-level fusion / support vector machine / Bayesian network / thermal infrared images / visible spectrum images

References

[1] Zeng Z, Pantic M, Roisman G, Huang T. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39−58

[2] Wang Z, Liu F, Wang L. Survey of facial expression recognition based on computer vision. Computer Engineering, 2006, 32(11): 231−233

[3] Liu X m, Tan H c, Zhang Y j. New research advances in facial expression recogntion. Journal of Image and Graphics, 2006, 11(10): 1359−1368

[4] Xue Y L, Mao X, Guo Y, Lv S W. The research advance of facial expression recognition in human computer interaction. Journal of Image and Graphics, 2009, 14(5): 764−772

[5] Bettadapura V. Face expression recognition and analysis: The state of the art. CoRR, 2012, abs/1203.6722

[6] Wesley A, Buddharaju P, Pienta R, Pavlidis I. A comparative analysis of thermal and visual modalities for automated facial expression recognition. Advances in Visual Computing, 2012, 51−60

[7] Yoshitomi Y, Kim S, Kawano T, Kilazoe T. Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication. 2000, 178−183

[8] Wang Z, Wang S. Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images. In: Proceedings of the 2011 IEEE International Workshop on Machine Learning for Signal Processing. 2011, 1−6

[9] Wang S, He S. Spontaneous facial expression recognition by fusing thermal infrared and visible images. Intelligent Autonomous Systems, 2013, 194: 263−272

[10] Cootes T, Edwards G, Taylor C. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681−685

[11] Lv Y, Wang S. A spontaneous facial expression recognition method using head motion and AAM features. In: Proceedings of the 2nd World Congress on Nature and Biologically Inspired Computing. 2010, 334−339

[12] Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, Wang X. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Transactions on Multimedia, 2010, 12(7): 682−691

[13] Littlewort G, Whitehill J, Wu T, Butko N, Ruvolo P, Movellan J, Bartlett M. The motion in emotion − a cert based approach to the fera emotion challenge. In: Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops. 2011, 897−902

[14] Cohn J, Reed L, Ambadar Z, Xiao J, Moriyama T. Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics. 2004, 610−616

[15] Tong Y, Chen J, Ji Q. A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258−273

[16] Valstar M, Gunes H, Pantic M. How to distinguish posed from spontaneous smiles using geometric features. In: Proceedings of the 9th international conference on Multimodal interfaces. 2007, 38−45

[17] Gunes H, Pantic M. Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. In: Proceedings of the 10th International Conference on Intelligent Virtual Agents. 2010, 6356: 371−377

[18] Jarlier S, Grandjean D, Delplanque S, N’Diaye K, Cayeux I, Velazco M, Sander D, Vuilleumier P, Scherer K. Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2011, 2(1): 2−9

[19] Hernández B, Olague G, Hammoud R, Trujillo L, Romero E. Visual learning of texture descriptors for facial expression recognition in thermal imagery. Computer Vision and Image Understanding, 2007, 106(2): 258−269

[20] Khan M, Ward R, Ingleby M. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Transactions on Applied Perception, 2009, 6(1): Article 6

[21] Yoshitomi Y. Facial expression recognition for speaker using thermal image processing and speech recognition system. In: Proceedings of the 10th World Scientific and Engineering Academy and Society International Conference on Applied Computer Science. 2010, 182−186

[22] Puri C, Olson L, Pavlidis I, Levine J, Starren J. Stresscam: non-contact measurement of users’ emotional states through thermal imaging. In: Proceedings of the 2005 Conference on Human Factors in Computing Systems. 2005, 1725−1728

[23] Shen P, Wang S, Liu Z. Facial expression recognition from infrared thermal videos. Intelligent Autonomous Systems, 2013, 194: 323−333

[24] Buddharaju P, Pavlidis I, Manohar C. Face recognition beyond the visible spectrum. In: Advances in Biometrics, 157−180. Springer, 2008

[25] Pavlidis I, Levine J, Baukol P. Thermal image analysis for anxiety detection. In: Proceedings of the 2001 International Conference on Image Processing. 2001, 315−318

[26] Cootes T. am_tools.

[27] Tong Y, Wang Y, Zhu Z, Ji Q. Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition, 2007, 40(11): 3195−3208

[28] Ding C. Analysis of gene expression profiles: class discovery and leaf ordering. In: Proceedings of the 6th Annual International Conference on Computational Biology. 2002, 127−136

[29] Wu T F, Lin C J, Weng R C. Probability estimates for multi-class classification by pairwise coupling. The Journal of Machine Learning Research, 2004, 5: 975−1005

[30] Chang C C, Lin C J. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2011, 2: 27:1−27:27. Software available at

[31] Long P M, Servedio R A. Discriminative learning can succeed where generative learning fails. In: Proceedings of the 19th Annual Conference on Learning Theory. 2006, 4005: 319−334

[32] Ng A Y, Jordan A. On discriminative vs. generative classifiers: a comparison of logistic regression and naive bayes. Advances in Neural Information Processing Systems, 2002, 14: 841

[33] Hermosilla G, Ruiz-del-Solar J, Verschae R, Correa M. A comparative study of thermal face recognition methods in unconstrained environments. Pattern Recognition, 2012, 45(7): 2445−2459

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag Berlin Heidelberg