RESEARCH ARTICLE

Kernel feature extraction methods observed from the viewpoint of generating-kernels

  • Jian YANG
Expand
  • School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing 210094, China

Received date: 16 Jul 2010

Accepted date: 23 Nov 2010

Published date: 05 Mar 2011

Copyright

2014 Higher Education Press and Springer-Verlag Berlin Heidelberg

Abstract

This paper introduces an idea of generating a kernel from an arbitrary function by embedding the training samples into the function. Based on this idea, we present two nonlinear feature extraction methods: generating kernel principal component analysis (GKPCA) and generating kernel Fisher discriminant (GKFD). These two methods are shown to be equivalent to the function-mapping-space PCA (FMS-PCA) and the function-mapping-space linear discriminant analysis (FMS-LDA) methods, respectively. This equivalence reveals that the generating kernel is actually determined by the corresponding function map. From the generating kernel point of view, we can classify the current kernel Fisher discriminant (KFD) algorithms into two categories: KPCA+ LDA based algorithms and straightforward KFD (SKFD) algorithms. The KPCA+ LDA based algorithms directly work on the given kernel and are not suitable for non-kernel functions, while the SKFD algorithms essentially work on the generating kernel from a given symmetric function and are therefore suitable for non-kernels as well as kernels. Finally, we outline the tensor-based feature extraction methods and discuss ways of extending tensor-based methods to their generating kernel versions.

Cite this article

Jian YANG . Kernel feature extraction methods observed from the viewpoint of generating-kernels[J]. Frontiers of Electrical and Electronic Engineering, 2011 , 6(1) : 43 -55 . DOI: 10.1007/s11460-011-0129-z

1
Vapnik V. The Nature of Statistical Learning Theory. New York: Springer, 1995

2
Müller K R, Mika S, Rätsch G, Tsuda K, Schölkopf B. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 2001, 12(2): 181-201

DOI

3
Schölkopf B, Smola A, Muller K R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 1998, 10(5): 1299-1319

DOI

4
Mika S, Rätsch G, Weston J, Schölkopf B, Müller K R. Fisher discriminant analysis with kernels. In: Proceedings of IEEE International Workshop on Neural Networks for Signal Processing IX. 1999, 41-48

5
Mika S, Rätsch G, Schölkopf B, Smola A, Weston J, Müller K R. Invariant feature extraction and classification in kernel spaces. Advances in Neural Information Processing Systems, 1999, 12: 526-532

6
Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation, 2000, 12(10): 2385-2404

DOI

7
Roth V, Steinhage V. Nonlinear discriminant analysis using kernel functions. In: Solla S A, Leen T K, Mueller K R, eds. Advances in Neural Information Processing Systems. 2000, 12: 568-574

8
Mika S, Rätsch G, Weston J, Schölkopf B, Smola A, Müller K R. Constructing descriptive and discriminative non-linear features: Rayleigh coefficients in kernel feature spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(5): 623-628

DOI

9
Yang M H. Kernel Eigenfaces vs. kernel Fisherfaces: face recognition using kernel methods. In: Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition. 2002, 215-220

DOI

10
Lu J, Plataniotis K N, Venetsanopoulos A N. Face recognition using kernel direct discriminant analysis algorithms. IEEE Transactions on Neural Networks, 2003, 14(1): 117-126

DOI

11
Schölkopf B, Smola A. Learning with Kernels. Cambridge: MIT Press, 2002

12
Shawe-Taylor J, Cristianini N. Kernel Methods for Pattern Analysis. Cambridge: Cambridge University Press, 2004

13
Xu J, Zhang X, Li Y. Kernel MSE algorithm: a unified framework for KFD, LS-SVM, and KRR. In: Proceedings of the International Joint Conference on Neural Networks. 2001, 1486-1491

14
Billings S A, Lee K L. Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm. Neural Networks, 2002, 15(2): 263-270

15
Zheng W, Zhao L, Zou C. Foley-Sammon optimal discriminant vectors using kernel approach. IEEE Transactions on Neural Networks, 2005, 16(1): 1-9

DOI

16
Yang J, Jin Z, Yang J Y, Zhang D, Frangi A F. Essence of kernel Fisher discriminant: KPCA plus LDA. Pattern Recognition, 2004, 37(10): 2097-2100

DOI

17
Yang J, Frangi A F, Yang J Y, Zhang D, Jin Z. KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 230-244

DOI

18
Xu Y, Zhang D, Jin Z, Li M, Yang J Y. A fast kernelbased nonlinear discriminant analysis for multi-class problems. Pattern Recognition, 2006, 39(6): 1026-1033

DOI

19
Zhao J, Wang H, Ren H, Kee S C. LBP discriminant analysis for face verification. In: Proceeding of the 2005 IEEE Conference on Computer Vision and Pattern Recognition. 2005, 3: 167

20
Liu C. Capitalize on dimensionality increasing techniques for improving face recognition grand challenge performance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(5): 725-737

DOI

21
Cevikalp H, Neamtu M, Wilkes M, Barkana A. Discriminative common vector method with kernels. IEEE Transactions on Neural Networks, 2006, 17(6): 1550-1565

DOI

22
Bach F R, Jordan MI. Kernel independent component analysis. Journal of Machine Learning Research, 2002, 3(1): 1-48

DOI

23
Yang J, Gao X, Zhang D, Yang J Y. Kernel ICA: an alternative formulation and its application to face recognition. Pattern Recognition, 2005, 38(10): 1784-1787

DOI

24
Ma J. Function replacement vs. kernel trick. Neurocomputing, 2003, 50: 479-483

DOI

25
Ma J, Theiler J, Perkins S. Two realizations of a general feature extraction framework. Pattern Recognition, 2004, 37(5): 875-887

DOI

26
Lodhi H, Saunders C, Shawe-Taylor J, Cristianini N, Watkins C. Text classification using string kernels. Journal of Machine Learning Research, 2002, 2(2): 419-444

DOI

27
Chen W S, Yuen P C, Huang J, Lai J. Wavelet kernel construction for kernel discriminant analysis on face recognition. In: Proceeding of the 2006 Conference on Computer Vision and Pattern Recognition Workshop. 2006, 47-52

DOI

28
Schölkopf B. Support vector learning. Dissertation for the Doctoral Degree. Berlin: Berlin Technical University, 1997

29
Camps-Valls G, Martin-Guerrero J, Rojo-Alvarez J, Soria-Olivas E. Fuzzy sigmoid kernel for support vector classifiers. Neurocomputing, 2004, 62: 501-506

DOI

30
Ahonen T, Hadid A, Pietikäinen M. Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(12): 2037-2041

DOI

31
Mangasarian L. Generalized support vector machines. In: Smola A J, Bartlett P, Schokopf B, Schuurmans D, eds. Advances in Large Margin Classifiers. 2000, 135-146

32
Golub G H, Van Loan C F. Matrix Computations. 3rd ed. Baltimore: The Johns Hopkins University Press, 1996

33
Swets D L,Weng J. Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8): 831-836

DOI

34
Belhumeur P N, Hespanha J P, Kriengman D J. Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 711-720

DOI

35
Yang J, Yang J Y. Why can LDA be performed in PCA transformed space? Pattern Recognition, 2003, 36(2): 563-566

DOI

36
Liu C J, Wechsler H. Robust coding schemes for indexing and retrieval from large face databases. IEEE Transactions on Image Processing, 2000, 9(1): 132-137

DOI

37
Turk M, Pentland A. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 1991, 3(1): 71-86

DOI

38
Liu K, Cheng Y Q, Yang J Y. Algebraic feature extraction for image recognition based on an optimal discriminant criterion. Pattern Recognition, 1993, 26(6): 903-911

DOI

39
Yang J, Zhang D, Frangi A F, Yang J Y. Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 131-137

DOI

40
Visani M, Garcia C, Laurent C. Comparing robustness of two-dimensional PCA and eigenfaces for face recognition. Lecture Notes in Computer Science, 2004, 3212: 717-724

DOI

41
Zuo W M, Zhang D. K. Wang K. An assembled matrix distance metric for 2DPCA-based image recognition. Pattern Recognition Letters, 2006, 27(3): 210-216

42
Zhang D Q, Zhou Z H. (2D)(2)PCA: Two-directional twodimensional PCA for efficient face representation and recognition. Neurocomputing, 2005, 69(1-3): 224-231

DOI

43
Kong H,Wang L, Teoh E K, Li X, Wang J G, Venkateswarlu R. Generalized 2D principal component analysis for face image representation and recognition. Neural Networks, 2005, 18(5-6): 585-594

DOI

44
Ye J. Generalized low rank approximations of matrices. Machine Learning, 2005, 61(1-3): 167-191

DOI

45
Hyvärinen A, Oja E. Independent component analysis: algorithms and applications. Neural Networks, 2000, 13(4-5): 411-430

DOI

46
He X, Yan S, Hu Y, Niyogi P, Zhang H J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 328-340

DOI

47
Ye J, Janardan R, Li Q. Two-dimensional linear discriminant analysis. Advances in Neural Information Processing Systems, 2004, 17: 1569-1576

48
Li M, Yuan B. 2D-LDA: a novel statistical linear discriminant analysis for image matrix. Pattern Recognition Letters, 2005, 26(5): 527-532

DOI

49
Xiong H, Swamy M, Ahmad M. Two-dimensional FLD for face recognition. Pattern Recognition, 2005, 38(7): 1121-1124

DOI

50
Yang J, Zhang D, Yong X, Yang J. Two-dimensional linear discriminant transform for face recognition. Pattern Recognition, 2005, 38(7): 1125-1129

DOI

51
Zuo W, Zhang D, Yang J, Wang K. BDPCA plus LDA: A novel fast feature extraction technique for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 2006, 36(4): 946-953

52
Zou C, Sun N, Ji Z, Zhao L. 2DCCA: a novel method for small sample size face recognition. In: Proceedings of IEEE Workshop on Applications of Computer Vision. 2007, 43

53
Lee S H, Choi S. Two-dimensional canonical correlation analysis. IEEE Signal Processing Letters, 2007, 14(10): 735-738

DOI

54
Gao Q, Zhang L, Zhang D, Xu H. Independent components extraction from image matrix. Pattern Recognition Letters, 2010, 31(3): 171-178

DOI

55
Chen S B, Zhao H F, Kong M, Luo B. 2d-lpp: a twodimensional extension of locality preserving projections. Neurocomputing, 2007, 70(4-6): 912-921

56
Hu D W, Feng G Y, Zhou Z T. Two-dimensional locality preserving projections (2DLPP) with its application to palmprint recognition. Pattern Recognition, 2007, 40(1): 339-342

DOI

57
Xu D, Yan S, Zhang L, Lin S, Zhang H J, Huang T S. Reconstruction and recognition of tensor-based objects with concurrent subspaces analysis. IEEE Transactions on Circuits and Systems for Video Technology, 2008, 18(1): 36-47

DOI

58
Wang H, Ahuja N. A tensor approximation approach to dimensionality reduction. Journal of Computer Vision, 2008, 76(3): 217-229

DOI

59
Tao D, Li X, Wu X, Maybank S J. General tensor discriminant analysis and gabor features for gait recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(10): 1700-1715

DOI

60
Lu H, Plataniotis K N, Venetsanopoulos A N. MPCA: multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks, 2008, 19(1): 18-39

DOI

61
Zhang L, Gao Q, Zhang D. Directional independent component analysis with tensor representation. In: Proceedings of Computer Vision and Pattern Recognition. 2008, 1-7

62
Sun N, Wang H, Ji Z, Zou C, Zhao L. An efficient algorithm for kernel two-dimensional principal component analysis. Neural Computing & Applications, 2008, 17(1): 59-64

DOI

63
Liu J, Chen S, Zhou Z H, Tan X. Generalized low-rank approximations of matrices revisited. IEEE Transactions on Neural Networks, 2010, 21(4): 621-632

DOI

Outlines

/