Locality-constrained framework for face alignment

Abstract

Although the conventional active appearance model (AAM) has achieved some success for face alignment, it still suffers from the generalization problem when be applied to unseen subjects and images. To deal with the generalization problem of AAM, we first reformulate the original AAM as sparsity-regularized AAM, which can achieve more compact/better shape and appearance priors by selecting nearest neighbors as the bases of the shape and appearance model. To speed up the fitting procedure, the sparsity in sparsity-regularized AAM is approximated by using the locality (i.e., K-nearest neighbor), and thus inducing the locality-constrained active appearancemodel (LC-AAM). The LC-AAM solves a constrained AAM-like fitting problem with the K-nearest neighbors as the bases of shape and appearance model. To alleviate the adverse influence of inaccurate K-nearest neighbor results, the locality constraint is further embedded in the discriminative fitting method denoted as LC-DFM, which can find better K-nearest neighbor results by employing shape-indexed feature, and can also tolerate some inaccurate neighbors benefited from the regression model rather than the generative model in AAM. Extensive experiments on several datasets demonstrate that our methods outperform the state-of-the-arts in both detection accuracy and generalization ability.

Keywords

locality-constrained AAM / locality-constrained DFM / face alignment / sparsity-regularization

References

[1] Wiskott L, Fellous J M, Kuiger N, Malsburg C. Face recognition by elastic bunch graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 775–779

[2] Liu X, Kan M, Wu W, Shan S, Chen X. Viplfacenet: an open source deep face recognition SDK. Frontiers of Computer Science, 2017, 11(2): 208–218

[3] Jiang D, Hu Y, Yan S, Zhang L, Zhang H, Gao W. Efficient 3d reconstruction for face recognition. Pattern Recognition, 2005, 38(6): 787–798

[4] Fasel B, Luettin J. Automatic facial expression analysis: a survey. Pattern Recognition, 2003, 36(1): 259–275

[5] Zhang F, Yu Y, Mao Q, Gou J, Zhan Y. Pose-robust feature learning for facial expression recognition. Frontiers of Computer Science, 2016, 10(5): 832–844

[6] Zheng H, Geng X. Facial expression recognition via weighted group sparsity. Frontiers of Computer Science, 2017, 11(2): 266–275

[7] Cootes T, Taylor C, Cooper D, Graham J. Active shape models-their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38–59

[8] Cootes T, Edwards G, Taylor C. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681–685

[9] Gross R, Matthews I, Baker S. Generic vs. person specific active appearance models. Image and Vision Computing, 2005, 23(12): 1080–1093

[10] Liu X. Generic face alignment using boosted appearance model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

[11] Wu H, Liu X, Doretto G. Face alignment via boosted ranking model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8

[12] Saragih J, Goecke R. A nonlinear discriminative approach to AAM fitting. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8

[13] Xiong X, Torre F. Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 532–539

[14] Asthana A, Zafeiriou S, Cheng S, Pantic M. Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3444–3451

[15] Lowe D. Distinctive image features from scale-invariant key points. International Journal of Computer Vision, 2004, 60(2): 91–110

[16] Dollár P, Welinder P, Perona P. Cascaded pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1078–1085

[17] Tzimiropoulos G. Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3659–3667

[18] Lee D, Park H, Yoo C. Face alignment using cascade gaussian process regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4204–4212

[19] Kazemi V, Sullivan J. One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1867–1874

[20] Cao X, Wei Y, Wen F, Sun J. Face alignment by explicit shape regression. International Journal of Computer Vision, 2014, 107(2): 177–190

[21] Cootes T, Taylor C. A mixture model for representing shape variation. Image and Vision Computing, 1999, 17(8): 567–573

[22] Maaten L, Hendriks E. Capturing appearance variation in active appearance models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2010, 34–41

[23] Tipping M E, Bishop C M. Mixtures of probabilistic principal component analyzers. Neural Computation, 1999, 11(2): 443–482

[24] Etyngier P, Segonne F, Keriven R. Shape priors using manifold learning techniques. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8

[25] Zhang S, Zhan Y, Dewan M, Huang J, Metaxas D, Zhou X. Towards robust and effective shape modeling: sparse shape composition. Medical Image Analysis, 2012, 16(1): 265–277

[26] Ciregan D, Meier U, Schmidhuber J. Multi-column deep neural networks for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3642–3649

[27] Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. 2012, 1097–1105

[28] Szegedy C, Toshev A, Erhan D. Deep neural networks for object detection. In: Proceedings of the Advances in Neural Information Processing Systems Conference. 2013, 2553–2561

[29] Girshick R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1440–1448

[30] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3431–3440

[31] Sun Y, Wang X, Tang X. Deep convolutional network cascade for facial point detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3476–3483

[32] Wu Y, Wang Z, Ji Q. Facial feature tracking under varying facial expressions and face poses based on restricted boltzmann machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3452–3459

[33] Zhang J, Shan S, Kan M, Chen X. Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: Proceedings of the European Conference on Computer Vision. 2014, 1–16

[34] Zhang Z, Luo P, Loy C, Tang X. Learning and transferring multitask deep representation for face alignment. 2014, arXiv preprint arXiv:1408.3967

[35] Trigeorgis G, Snape P, Nicolaou M, Antonakos E, Zafeiriou S. Mnemonic descent method: a recurrent process applied for end-to-end face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4177–4187

[36] Jourabloo A, Liu X. Large-pose face alignment via CNN-based dense 3d model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4188–4196

[37] Yang Y, Ma Z, Nie F, Chang X, Hauptmann A. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 2015, 113(2): 113–127

[38] Gao N, Huang S, Chen S. Multi-label active learning by model guided distribution matching. Frontiers of Computer Science, 2016, 10(5): 845–855

[39] Matthews I, Baker S. Active appearance models revisited. International Journal of Computer Vision, 2004, 60(2): 135–164

[40] Zhao X, Shan S, Chai X, Chen X. Locality-constrained active appearance model. In: Proceedings of the Asian Conference on Computer Vision. 2013, 636–647

[41] Dalal N, Triggs B. Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 886–893

[42] Yu K, Zhang T, Gong Y. Nonlinear learning using local coordinate coding. In: Proceedings of the 22nd International Conference on Advances in Neural Information Processing Systems. 2009, 2223–2231

[43] Zhao X, Chai X, Niu Z, Heng C, Shan S. Context constrained facial landmark localization based on discontinuous haar-like feature. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 2011, 673–678

[44] Zhao X, Chai X, Niu Z, Heng C, Shan S. Context modeling for facial landmark detection based on non-adjacent rectangle (NAR) haar-like feature. Image and Vision Computing, 2012, 30(3): 136–146

[45] Sim T, Baker S, Bsat M. The CMU pose, illumination, and expression (PIE) database. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 2002, 46–51

[46] Phillips P, Flynn P, Scruggs T, Bowyer K, Chang J, Hoffman K, Marques J, Min J, Worek W. Overview of the face recognition grand challenge. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 947–954

[47] Phillips P, Wechsler H, Huang J, Rauss P. The feret database and evaluation procedure for face recognition algorithms. Image and Vision Computing, 1998, 16(5): 295–306

[48] Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X, Zhao D. The CASPEAL large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 2008, 38(1): 149–161

[49] Kumar N, Berg A, Belhumeur P, Nayar S. Attribute and simile classifiers for face verification. In: Proceedings of the IEEE International Conference on Computer Vision. 2009, 365–372

[50] Tian Y, Kanade T, Cohn J. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 97–115

[51] Gu L, Kanade T. A generative shape regularization model for robust face alignment. In: Proceedings of European Conference on Computer Vision. 2008, 413–426

[52] Milborrow S, Nicolls F. Locating facial features with an extended active shape model. In: Proceedings of European Conference on Computer Vision. 2008, 504–513

[53] Saragih J, Lucey S, Cohn J. Deformable model fitting by regularized landmark mean-shifts. International Journal of Computer Vision, 2011, 91(2): 200–215

[54] Norouzi M, Punjani A, Fleet D. Fast search in hamming space with multi-index hashing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3108–3115

[55] Liu X, Deng C, Lang B, Tao D, Li X. Query-adaptive reciprocal hash tables for nearest neighbor search. IEEE Transactions on Image Processing, 2016, 25(2): 907–919

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature