Probabilistic robust regression with adaptive weights–a case study on face recognition

Abstract

Robust regression plays an important role in many machine learning problems. A primal approach relies on the use of Huber loss and an iteratively reweighted l₂ method. However, because the Huber loss is not smooth and its corresponding distribution cannot be represented as a Gaussian scale mixture, such an approach is extremely difficult to handle using a probabilistic framework. To address those limitations, this paper proposes two novel losses and the corresponding probability functions. One is called Soft Huber, which is well suited for modeling non-Gaussian noise. Another is Nonconvex Huber, which can help produce much sparser results when imposed as a prior on regression vector. They can represent any l_q loss () with tuning parameters, which makes the regression modelmore robust. We also show that both distributions have an elegant form, which is a Gaussian scale mixture with a generalized inverse Gaussian mixing density. This enables us to devise an expectation maximization (EM) algorithm for solving the regression model.We can obtain an adaptive weight through EM, which is very useful to remove noise data or irrelevant features in regression problems. We apply our model to the face recognition problem and show that it not only reduces the impact of noise pixels but also removes more irrelevant face images. Our experiments demonstrate the promising results on two datasets.

Keywords

robust regression / nonconvex loss / face recognition

References

[1] Andersen R. Modern Methods for Robust Regression. Sage, 2008

[2] Ben-Gal I. Outlier Detection. Data Mining and Knowledge Discovery Handbook. Springer, 2010, 117–130

[3] Stigler S M. Gauss and the invention of least squares. The Annals of Statistics, 1981, 9(3): 465–474

[4] Rousseeuw P J, Hubert M. Robust statistics for outlier detection. Wiley Interdisciplinary Reviews: DataMining and Knowledge Discovery, 2011, 1(1): 73–79

[5] Huber P J. Robust regression: asymptotics, conjectures and monte carlo. The Annals of Statistics, 1973, 1(5): 799–821

[6] Huber P J, Ronchetti E M. Robust Statistics. 2nd ed. New Jersey: John Wiley & Sons, 2009

[7] Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge: Cambridge University Press, 2004

[8] Figueiredo M. Adaptive sparseness using jeffreys prior. In: Dietterich T G, Becker S, Ghahramani Z, eds. Advances in Neural Information Processing Systems. MIT Press, 2002, 697–704

[9] Kabán A. On Bayesian classification with laplace priors. Pattern Recognition Letters, 2007, 28(10): 1271–1282

[10] Lange K L, Little R J A, Taylor J M G. Robust statistical modeling using the t distribution. Journal of the American Statistical Association, 1989, 84(408): 881–896

[11] Jylänki P, Vanhatalo J, Vehtari A. Robust gaussian process regression with a student-t likelihood. Journal of Machine Learning Research, 2011, 12: 3227–3257

[12] Lange K, Sinsheimer J S. Normal/independent distributions and their applications in robust regression. Journal of Computational and Graphical Statistics, 1993, 2(2): 175–198

[13] Gao M, Wang K, He L. Probabilistic model checking and scheduling implementation of an energy router system in energy internet for green cities. IEEE Transactions on Industrial Informatics, 2018, 14(4): 1501–1510

[14] Bernardo J M, Smith A F M. Bayesian Theory. New York: JohnWilley and Sons, 1994

[15] Xu L, Jordan M I. On convergence properties of the EM algorithm for gaussian mixtures. Neural Computation, 1996, 8(1): 129–151

[16] Naseem I, Togneri R, Bennamoun M. Robust regression for face recognition. Pattern Recognition, 2012, 45(1): 104–118

[17] Yang M, Zhang L, Yang J, Zhang D. Robust sparse coding for face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 625–632

[18] Hu H, Wang K, Lv C, Wu J, Yang Z. Semi-supervised metric learningbased anchor graph hashing for large-scale image retrieval. IEEE Transactions on Image Processing, 2019, 28(2): 739–754

[19] Huber P J. Robust Estimation of a Location Parameter. Breakthroughs in Statistics. Springer, New York, 1992, 492–518

[20] Tibshirani R. Regression shrinkage and selection via the lasso: a retrospective. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2011, 73(3): 273–282

[21] Wu C F J. On the convergence properties of the EM algorithm. Annals of Statistics, 1983, 11: 95–103

[22] Wrigh t J, Yang A Y, Ganesh A, Sastry S S, Ma Y. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 31(2): 210–227

[23] Martinez A M. The AR face database. CVC Technical Report, 1998

[24] Georghiades A S, Belhumeur P N, Kriegman D J. From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 643–660

[25] Lee K C, Ho J, Kriegman D J. Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 684–698

[26] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems. 2012, 1097–1105

[27] Yang M, Zhang L, Yang J, Zhang D. Regularized robust coding for face recognition. IEEE Transactions on Image Processing, 2013, 22(5): 1753–1766

[28] Andrews D F, Mallows C L. Scale mixtures of normal distributions. Journal of the Royal Statistical Society: Series B (Methodological), 1974, 36(1): 99–102

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature