Dimensionality reduction with latent variable model

Abstract

Over the past few decades, latent variable model (LVM)-based algorithms have attracted considerable attention for the purpose of data dimensionality reduction, which plays an important role in machine learning, pattern recognition, and computer vision. LVM is an effective tool for modeling density of the observed data. It has been used in dimensionality reduction for dealing with the sparse observed samples. In this paper, two LVM-based dimensionality reduction algorithms are presented firstly, i.e., supervised Gaussian process latent variable model and semi-supervised Gaussian process latent variable model. Then, we propose an LVMbased transfer learning model to cope with the case that samples are not independent identically distributed. In the end of each part, experimental results are given to demonstrate the validity of the proposed dimensionality reduction algorithms.

Keywords

dimensionality reduction / latent variable model / pairwise constraints / Bregman divergence

References

[1] Koren Y, Carmel L. Robust linear dimensionality reduction. IEEE Transactions on Visualization and Computer Graphics, 2004, 10(4): 459-470

[2] Duda R O, Hart P E, Stork D G. Pattern Classification. 2nd ed. New York: Wiley, 2000

[3] Tao D, Li X, Wu X, HuW, Maybank S J. Supervised tensor learning. Knowledge and Information Systems, 2007, 13(1): 1-42

[4] Tao D, Li X, Wu X, Maybank S J. General averaged divergences analysis. In: Proceedings of IEEE International Conference on Data Mining. 2007, 302-311

[5] He X. Laplacian regularized D-optimal design for active learning and its application to image retrieval. IEEE Transactions on Image Processing, 2010, 19(1): 254-263

[6] Tao D, Tang X, Li X, Rui Y. Direct kernel biased discriminant analysis: A new content-based image retrieval relevance feedback algorithm. IEEE Transactions on Multimedia, 2006, 8(4): 716-727

[7] Joliffe I. Principal Component Analysis. New York: Springer, 1986

[8] Anderson T W. Asymptotic theory for principal component analysis. Annals of Mathematical Statistics, 1963, 34(1): 122-148

[9] Schölkopf B, Smola A, Müller K R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 1998, 10(5): 1299-1319

[10] Fiori S. Visualization of Riemannian-manifold-valued elements by multidimensional scaling. Neurocomputing, 2011, 74(6): 983-992

[11] Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(5500): 2323-2326

[12] Li X, Lin S, Yan S, Xu D. Discriminant locally linear embedding with high-order tensor data. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 2008, 38(2): 342-352

[13] Belkin M, Niyogi P. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 2003, 15(6): 1373-1396

[14] Tenenbaum J B, de Silva V, Langford J C. A global geometric framework for nonlinear dimensionality reduction. Science, 2000, 290(5500): 2319-2323

[15] de Silva V, Tenenbaum J B. Global versus local methods in nonlinear dimensionality reduction. Advances in Neural Information Processing Systems, 2003, 15: 705-712

[16] He X, Niyogi P. Locality preserving projections. Advances in Neural Information Processing Systems, 2003, 16: 153-160

[17] He X, Yan S, Hu Y, Niyogi P, Zhang H-J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 328-340

[18] He X, Cai D, Yan S, Zhang H-J. Neighborhood preserving embedding. In: Proceedings of the 10th International Conference on Computer Vision. 2005, 2: 1208-1213

[19] Bar-Hillel A, Hertz T, Shental N, Weinshall D. Learning a Mahalanobis metric from equivalence constraints. Journal of Machine Learning Research, 2005, 6(1): 937-965

[20] Mika S, Rätsch G,Weston J, Schölkopf B, Müller K R. Fisher discriminant analysis with kernels. In: Proceedings of IEEE Signal Processing Society Workshop (Neural Networks for Signal Processing IX). 1999, 41-48

[21] Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation, 2000, 12(10): 2385-2404

[22] Yan S, Xu D, Zhang B, Zhang H-J, Yang Q, Lin S. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(1): 40-51

[23] Sugiyama M. Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research, 2007, 8: 1027-1061

[24] Tipping M E, Bishop C M. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B: Methodological, 1999, 61(3): 611-622

[25] Bartholomew D J. Statistical Factor Analysis and Related Methods. New York: Wiley, 2004

[26] Bishop C M, Svensen M, Williams C K I. GTM: The generative topographic mapping. Neural Computation, 1998, 10(1): 215-234

[27] Lawrence N D. Gaussian process models for visualization of high dimensional data. Advances in Neural Information Processing Systems, 2004, 16: 329-336

[28] Lawrence N D. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research, 2005, 6: 1783-1816

[29] Bishop C M. Learning in Graphical Models. Cambridge: MIT Press, 1999

[30] Rabiner L R, Juang B H. An introduction to hidden Markov models. IEEE ASSP Magazine, 1986, 3(1): 4-16

[31] Blei D M, Ng A Y, Jordan M I. Latent Dirichlet allocation. Journal of Machine Learning Research, 2003, 3: 993-1022

[32] Blei D M, Jordan M I. Variational methods for the Dirichlet process. In: Proceedings of the 21st International Conference on Machine Learning. 2004

[33] Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 1989, 77(2): 257-286

[34] Baker J K. The DRAGON system — An overview. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1975, 23(1): 24-29

[35] Zhong J, Gao X, Tian C. Face sketch synthesis using E-HMM and selective ensemble. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing. 2007, 1: 485-488

[36] Gao X, Zhong J, Li J, Tian C. Face sketch synthesis algorithm based on E-HMM and selective ensemble. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(4): 487-496

[37] Fritz M, Schiele B. Decomposition, discovery and detection of visual categories using topic models. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2008, 1-8

[38] Blei D M, Griffiths T L, Jordan M I. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the Association for Computing Machinery, 2010, 57(2): 21-30

[39] Li F, Perona P. A Bayesian hierarchical model for learning natural scene categories. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2005, 2: 524-531

[40] Wang C, Blei D, Li F. Simultaneous image classification and annotation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2009, 1903-1910

[41] Niu Z, Hua G, Gao X, Tian Q. Spatial-DiscLDA for visual recognition. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2011, 1769-1776

[42] Wang J M, Fleet D J, Hertzmann A. Gaussian process dynamical models for human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 283-298

[43] Urtasun R, Fleet D J, Fua P. 3D people tracking with Gaussian process dynamical models. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2006, 1: 238-245

[44] Navaratnam R, Fitzgibbon A W, Cipolla R. The joint manifold model for semi-supervised multi-valued regression. In: Proceedings of the 11th IEEE International Conference on Computer Vision. 2007, 1-8

[45] Gupta A, Chen T, Chen F, Kimber D, Davis L S. Context and observation driven latent variable model for human pose estimation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2008, 1-8

[46] Salzmann M, Urtasun R, Fua P. Local deformation models for monocular 3D shape recovery. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2008, 1-8

[47] Gao X, Wang X, Tao D, Li X. Supervised Gaussian process latent variable model for dimensionality reduction. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 2011, 41(2): 425-434

[48] Wang X, Gao X, Yuan Y, Tao D, Li J. Semi-supervised Gaussian process latent variable model with pairwise constraints. Neurocomputing, 2010, 73(10-12): 2186-2195

[49] Gao X, Wang X, Li X, Tao D. Transfer latent variable model based on divergence analysis. Pattern Recognition, 2011, 44(10-11): 2358-2366

[50] Nock R, Nielsen F. On weighting clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(8): 1223-1235

[51] Cayton L. Fast nearest neighbor retrieval for Bregman divergences. In: Proceedings of the 25th International Conference on Machine Learning. 2008, 112-119

RIGHTS & PERMISSIONS

2014 Higher Education Press and Springer-Verlag Berlin Heidelberg