Pose-robust feature learning for facial expression recognition

Abstract

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods require the use of sophisticated normalization and initialization procedures. Thus head-pose invariant facial expression recognition continues to be an issue to traditional methods. In this paper, we propose a novel approach for pose-invariant FER based on pose-robust features which are learned by deep learning methods — principal component analysis network (PCANet) and convolutional neural networks (CNN) (PRP-CNN). In the first stage, unlabeled frontal face images are used to learn features by PCANet. The features, in the second stage, are used as the target of CNN to learn a feature mapping between frontal faces and non-frontal faces. We then describe the non-frontal face images using the novel descriptions generated by the maps, and get unified descriptors for arbitrary face images. Finally, the pose-robust features are used to train a single classifier for FER instead of training multiple models for each specific pose. Our method, on the whole, does not require pose/ landmark annotation and can recognize facial expression in a wide range of orientations. Extensive experiments on two public databases show that our framework yields dramatic improvements in facial expression analysis.

Keywords

facial expression recognition / pose-robust features / principal component analysis network (PCANet) / convolutional neural networks (CNN)

References

[1] Zheng W M. Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Transactions on Affective Computing, 2014, 5(1): 71–85

[2] Eleftheriadis S, Rudovic O, Pantic M. Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Transactions on Image Processing, 2015, 24(1): 189–204

[3] Liu P, Han S Z, Meng Z B, Tong Y. Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1805–1812

[4] Zeng Z, Pantic M, Roisman G I, Huang T S. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2009, 31(1): 39–58

[5] Moore S, Bowden R. Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding, 2011, 115(4): 541–558

[6] Hesse N, Gehrig T, Gao H, Ekenel H K. Multi-view facial expression recognition using local appearance features. In: Proceedings of International Conference on Pattern Recognition. 2012, 3533–3536

[7] Rudovic O, Pantic M, Patras I. Coupled Gaussian processes for poseinvariant facial expression recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1357–1369

[8] Kumano S, Otsuka K, Yamato J, Maeda E, Sato Y. Pose-invariant facial expression recognition using variable-intensity templates. International Journal of Computer Vision, 2009, 83(2): 178–194

[9] Biswas A, Ghose M K. Expression invariant face recognition using DWT sift features. International Journal of Computer Applications, 2014, 92(2): 30–32

[10] Jian S, Hu C B, Aggarwal J K. Facial expression recognition with temporal modeling of shapes. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 1642–1649

[11] Girisha H, Sreepathi B, Karibasappa K. Multi-view face recognition using local binary pattern. International Journal of Computer Science and Information Technologies, 2014, 5(3): 2978–2981

[12] Dahmane M, Meunier J. Emotion recognition using dynamic gridbased HoG features. In: Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition andWorkshops. 2011, 884–888

[13] Hu Y X, Zeng Z, Yin L J, Wei X Z, Tu J, Huang T S. A study of non-frontal-view facial expressions recognition. In: Proceedings of International Conference on Pattern Recognition. 2008, 1–4

[14] Rudovic O, Patras I, Pantic M. Regression-based multiview facial expression recognition. In: Proceedings of International Conference on Pattern Recognition. 2010, 4121–4124

[15] Hu Y X, Zeng Z H, Yin L J, Wei X Z, Zhou X, Huang T S. Multi-view facial expression recognition. In: Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition. 2008, 56–61

[16] Gupta S K, Agrwal S, Meena Y K, Nain N. A hybrid method of feature extraction for facial expression recognition. In: Proceedings of Signal-Image Technology and Internet-Based Systems. 2011, 422–425

[17] Ding C X, Tao D C. A comprehensive survey on pose-invariant face recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015

[18] Tong Y, Chen J X, Ji Q. A unified probabilistic framework for spontaneous facial action modeling and unerstanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258–273

[19] Zheng W M, Tang H, Lin Z C, Huang T S. Emotion recognition from arbitrary view facial images. In: Proceedings of European Conference on Computer Vision. 2010, 490–503

[20] Sung J, Kim D. Real-time facial expression recognition using STAAM and layered GDA classifier. Image and Vision Computing, 2009, 27(9): 1315–1325

[21] Tang H, Hasegawa-Johnson M, Huang T. Non-frontal view facial expression recognition based on ergodic hidden markov model supervectors. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2010, 1202–1207

[22] Ranzato M, Susskind J, Mnih V, Hinton G. On deep generative models with applications to recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2011, 2857–2864

[23] Rifai S, Bengio Y, Courville A, Vincent P, Mirza M. Disentangling factors of variation for facial expression recognition. In: Proceedings of European Conference on Computer Vision. 2012, 808–822

[24] Eleftheriadis S, Rudovic O, Pantic M. Shared Gaussian process latent variable model for multi-view facial expression recognition. In: Proceedings of International Symposium on Visual Computing. 2013, 527–538

[25] Liu M Y, Shan S G, Wang R P, Chen X L. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1749–1756

[26] Rifai S, Vincent P, Muller X, Glorot X, Bengio Y. Contracting autoencoders: explicit invariance during feature extraction. In: Proceedings of International Conference on Machine Learning. 2011, 833–840

[27] Saudagare P V, Chaudhari D S. Facial expression recognition using neural network — an overview. International Journal of Soft Computing and Engineering, 2012, 2(1): 224–227

[28] Li J G, Zhang Y M. Learning surf cascade for fast and accurate object detection. In: Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3468–3475

[29] Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2001, 511–518

[30] Yin L J, Wei X Z, Sun Y, Wang J, Rosato M J. A 3D facial expression database for facial behavior research. In: Proceedings of Automatic face and gesture recognition. 2006, 211–216

[31] Dhall A, Goecke R, Lucey S, Gedeon T. Static facial expressions analysis in tough conditions: data, evaluation protocol and benchmark. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 2106–2112

[32] Zheng W M, Tang H, Lin Z C, Huang T S. A novel approach to expression recognition from non-frontal face images. In: Proceedings of the 12th IEEE International Conference on Computer Vision. 2009, 1901–1908

[33] Tariq U, Yang J, Huang T. Maximum margin gmm learning for facial expression recognition. In: Proceedings of Automatic Face and Gesture Recognition. 2013, 1–6

[34] Tariq U, Yang J C, Huang T S. Supervised super-vector encoding for facial expression recognition. Pattern Recognition, 2014, 89–95

[35] Jampour M, Mauthner T, Bischof H. Multi-view facial expressions recognition using local linear regression of sparse codes. In: Proceedings of the 20th Computer Vision Winter Workshop Paul Wohlhart, 2015

[36] Tariq U, Yang J C, Huang T S. Multi-view facial expression recognition analysis with generic sparse coding feature. In: Proceedings of European Conference on Computer Vision. 2012, 578–588

[37] Kan M N, Shan S G, Zhang H H, Lao S H, Chen X L. Multi-view discriminant analysis. In: Proceedings of European Conference on Computer Vision. 2012, 808–821

[38] Sharma A, Kumar A, Daume H, Jacobs D W. Generalized multiview analysis: a discriminative latent space. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 2160–2167

RIGHTS & PERMISSIONS

2016 Higher Education Press and Springer-Verlag Berlin Heidelberg