Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics

Abstract

As a typical biometric cue with great diversities, smile is a fairly influential signal in social interaction, which reveals the emotional feeling and inner state of a person. Spontaneous and posed smiles initiated by different brain systems have differences in both morphology and dynamics. Distinguishing the two types of smiles remains challenging as discriminative subtle changes need to be captured, which are also uneasily observed by human eyes. Most previous related works about spontaneous versus posed smile recognition concentrate on extracting geometric features while appearance features are not fully used, leading to the loss of texture information. In this paper, we propose a region-specific texture descriptor to represent local pattern changes of different facial regions and compensate for limitations of geometric features. The temporal phase of each facial region is divided by calculating the intensity of the corresponding facial region rather than the intensity of only the mouth region. A mid-level fusion strategy of support vector machine is employed to combine the two feature types. Experimental results show that both our proposed appearance representation and its combination with geometry-based facial dynamics achieve favorable performances on four baseline databases: BBC, SPOS, MMI, and UvA-NEMO.

Keywords

Facial landmark localization / Geometric feature / Appearance feature / Smile recognition

References

[1] Ambadar, Z., Cohn, J.F., Ian Reed, L. , 2009. All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J. Nonverb. Behav., 33(1):17–34. https://doi.org/10.1007/s10919-008-0059-5

[2] Baron-Cohen, S., Ring , H.A., Bullmore, E.T. , , 2000. The amygdala theory of autism. Neurosci. Biobehav. Rev., 24(3):355–364. https://doi.org/10.1016/S0149-7634(00)00011-7

[3] Burgos-Artizzu, X.P., Perona , P., Dollár, P. , 2013. Robust face landmark estimation under occlusion.Proc. IEEE Int. Conf. on Computer Vision, p.1513–1520. https://doi.org/10.1109/ICCV.2013.191

[4] Calvo, M.G., Nummenmaa , L., 2011. Time course of discrimination between emotional facial expressions: the role of visual saliency. Vis. Res., 51(15):1751–1759. https://doi.org/10.1016/j.visres.2011.06.001

[5] Calvo, M.G., Gutiérrez-García , A., Avero, P. , , 2013. Attentional mechanisms in judging genuine and fake smiles: eye-movement patterns. Emotion, 13(4):792–802. http://dx.doi.org/10.1037/a0032317

[6] Cao, X., Wei, Y., Wen, F., , 2014. Face alignment by explicit shape regression. Int. J. Comput. Vis., 107(2):177–190. https://doi.org/10.1007/s11263-013-0667-3

[7] Cohn, J.F., Schmidt , K.L., 2004. The timing of facial motion in posed and spontaneous smiles. Int. J. Wavel. Multiresol. Inform. Process., 2(2):121–132. https://doi.org/10.1142/S021969130400041X

[8] Cootes, T.F., Edwards , G.J., Taylor, C.J. , 2001. Active appearance models. IEEE Trans. Patt. Anal. Mach. Intell., 23(6):681–685. https://doi.org/10.1109/34.927467

[9] Dalal, N., Triggs, B., 2005. Histograms of oriented gradients for human detection.Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.886–893. https://doi.org/10.1109/CVPR.2005.177

[10] Dibeklioğlu, H. , Valenti, R., Salah, A.A., , 2010. Eyes do not lie: spontaneous versus posed smiles.Proc. Int. Conf. on Multimedia, p.703–706. https://doi.org/10.1145/1873951.1874056

[11] Dibeklioğlu, H. , Salah, A., Gevers, T., 2012. Are you really smiling at me? Spontaneous versus posed enjoyment smiles.Proc. European Conf. on Computer Vision, p.525–538. https://doi.org/10.1007/978-3-642-33712-3_38

[12] Dibeklioğlu, H. , Salah, A., Gevers, T., 2015. Recognition of genuine smiles.IEEE Trans. Multimed., 17(3):279–294. https://doi.org/10.1109/TMM.2015.2394777

[13] Dollár, P., Welinder , P., Perona, P. , 2010. Cascaded pose regression.Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1078–1085. https://doi.org/10.1109/CVPR.2010.5540094

[14] Ekman, P., 2009. Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage.W. W. Norton & Company, New York, p.140–143.

[15] Ekman, P., Rosenberg , E.L., 1997. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS).Oxford University Press.

[16] Frank, M.G., Ekman, P., 1993. Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles. Humor, 6(1):9–26. https://doi.org/10.1515/humr.1993.6.1.9

[17] Hoque, M., McDuff, D., Picard, R., 2012. Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Trans. Affect. Comput., 3(3):323–334. https://doi.org/10.1109/T-AFFC.2012.11

[18] Khokher, M.R., Bouzerdoum , A., Phung, S.L. , 2014. Crowd behavior recognition using dense trajectories.Proc. Int. Conf. on Digital lmage Computing: Techniques and Applications, p.1–7. https://doi.org/10.1109/DICTA.2014.7008098

[19] Le, V., Brandt, J., Lin, Z., , 2012. Interactive facial feature localization.Proc. European Conf. on Computer Vision, p.679–692. https://doi.org/10.1007/978-3-642-33712-3_49

[20] Li, W.S., Zhou, C.L., Xu, J.T. , 2005. A novel face recognition method with feature combination. J. Zhejiang Univ.- Sci., 6A(5):454–459. https://doi.org/10.1631/jzus.2005.A0454

[21] Liu, H., Sun, X., 2016. A partial least squares based ranker for fast and accurate age estimation.Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, p.2792–2796. https://doi.org/10.1109/ICASSP.2016.7472186

[22] Liu, H., Wu, P., 2012. Comparison of methods for smile deceit detection by training AU6 and AU12 simultaneously.Proc. IEEE Int. Conf. on Image Processing, p.1805–1808. https://doi.org/10.1109/ICIP.2012.6467232

[23] Liu, H., Gao, Y., Wang, C., 2014. Gender identification in unconstrained scenarios using self-similarity of gradients features.Proc. IEEE Int. Conf. on Image Processing, p.5911–5915. https://doi.org/10.1109/ICIP.2014.7026194

[24] Miehlke, A., Fisch, U., Eneroth, C.M. , 1973. Surgery of the Facial Nerve.Saunders, Philadelphia.

[25] Ojala, T., Pietikäinen , M., Mäenpää, T., 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Patt. Anal. Mach. Intell., 24(7):971–987. https://doi.org/10.1109/TPAMI.2002.1017623

[26] Peng, H., Long, F., Ding, C., 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Patt. Anal. Mach. Intell., 27(8):1226–1238. https://doi.org/10.1109/TPAMI.2005.159

[27] Pfister, T., Li, X., Zhao, G., , 2011a. Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework.Proc. IEEE Int. Conf. on Computer Vision Workshops, p.868–875. https://doi.org/10.1109/ICCVW.2011.6130343

[28] Pfister, T., Li, X., Zhao, G., , 2011b. Recognising spontaneous facial micro-expressions.Proc. IEEE Int. Conf. on Computer Vision, p.1449–1456. https://doi.org/10.1109/ICCV.2011.6126401

[29] Rinn, W.E., 1984. The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol. Bull., 95(1):52–77. https://doi.org/10.1037/0033-2909.95.1.52

[30] Sariyanidi, E., Gunes, H., Cavallaro, A. , 2015. Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Patt. Anal. Mach. Intell., 37(6):1113–1133. https://doi.org/10.1109/TPAMI.2014.2366127

[31] Shen, X.B., Wu, Q., Fu, X.L., 2012. Effects of the duration of expressions on the recognition of microexpressions. J. Zhejiang Univ.-Sci. B, 13(3):221–230. http://dx.doi.org/10.1631/jzus.B1100063

[32] Valstar, M., Pantic, M., 2010. Induced disgust, happiness and surprise: an addition to the MMI facial expression database.Proc. 3rd Int. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, p.65–70.

[33] Valstar, M.F., Gunes, H., Pantic, M., 2007. How to distinguish posed from spontaneous smiles using geometric features.Proc. Int. Conf. on Multimodal Interfaces, p.38–45. https://doi.org/10.1145/1322192.1322202

[34] Wang, J., Yang, J., Yu, K., , 2010. Locality-constrained linear coding for image classification.Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.3360–3367. https://doi.org/10.1109/CVPR.2010.5540018

[35] Wang, X., Wang, L., Qiao, Y., 2012. A comparative study of encoding, pooling and normalization methods for action recognition.Proc. Asian Conf. on Computer Vision, p.572–585. https://doi.org/10.1007/978-3-642-37431-9_44

[36] Whitehill, J., Bartlett , M.S., Movellan, J.R. , 2013. Automatic facial expression recognition. In: Gratch, J., Marsella, S. (Eds.), Social Emotions in Nature Artifact. Oxford Scholarship Online. https://doi.org/10.1093/acprof:oso/9780195387643.003.0007

[37] Wu, P.P., Liu, H., Zhang, X.W. , 2014. Spontaneous versus posed smile recognition using discriminative local spatio-temporal descriptors.Proc. Int. IEEE Conf. on Acoustics, Speech and Signal Processing, p.1249–1253. https://doi.org/10.1109/ICASSP.2014.6853795

[38] Wu, Q., Shen, X.B., Fu, X.L. , 2011. The machine knows what you are hiding: an automatic micro-expression recognition system.LNCS, 6975:152–162. https://doi.org/10.1007/978-3-642-24571-8_16

[39] Yang, J.C., Yu, K., Gong, Y.H., , 2009. Linear spatial pyramid matching using sparse coding for image classification.Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1794–1801. https://doi.org/10.1109/CVPR.2009.5206757

RIGHTS & PERMISSIONS

2017 Zhejiang University and Springer-Verlag Berlin Heidelberg