An investigation of several typical model selection criteria for detecting the number of signals

Abstract

Based on the problem of detecting the number of signals, this paper provides a systematic empiricalinvestigation on model selection performances of several classical criteria and recently developed methods (including Akaike’s information criterion (AIC), Schwarz’s Bayesian information criterion, Bozdogan’s consistent AIC, Hannan-Quinn information criterion, Minka’s (MK) principal component analysis (PCA) criterion, Kritchman & Nadler’s hypothesis tests (KN), Perry & Wolfe’s minimax rank estimation thresholding algorithm (MM), and Bayesian Ying-Yang (BYY) harmony learning), by varying signal-to-noise ratio (SNR) and training sample size N. A family of model selection indifference curves is defined by the contour lines of model selection accuracies, such that we can examine the joint effect of N and SNR rather than merely the effect of either of SNR and N with the other fixed as usually done in the literature. The indifference curves visually reveal that all methods demonstrate relative advantages obviously within a region of moderate N and SNR. Moreover, the importance of studying this region is also confirmed by an alternative reference criterion by maximizing the testing likelihood. It has been shown via extensive simulations that AIC and BYY harmony learning, as well as MK, KN, and MM, are relatively more robust than the others against decreasing N and SNR, and BYY is superior for a small sample size.

Key words： number of signals; array processing; factor analysis; principal component analysis (PCA); model selection criteria

References

1 Wax M, Kailath T. Detection of signals by information theoretic criteria. IEEE Transactions on Acoustics, Speech and Signal Processing, 1985: 33(2): 387-392

2 Schmidt R. Multiple emitter location and signal parameter estimation. IEEE Transactions on Antennas and Propagation, 1986, 34(3): 276-280

3 Tu S, Xu L. Theoretical analysis and comparison of several criteria on linear model dimension reduction. In: Proceedings of the 8th International Conference on Independent Component Analysis and Signal Separation. 2009, 154-162

4 Anderson T, Rubin H. Statistical inference in factor analysis. In: Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. 1956, 5: 111-150

5 Tipping M E, Bishop C M. Mixtures of probabilistic principal component analyzers. Neural Computation, 1999, 11(2): 443-482

6 Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 1974, 19(6): 716-723

7 Minka T P. Automatic choice of dimensionality for PCA. Advances in Neural Information Processing Systems, 2001, 13: 598-604

8 Xu L. Bayesian-Kullback coupled Ying-Yang machines: unified learnings and new results on vector quantization. In: Proceedings of International Conference on Neural Information Processing. 1995, 977-988

9 Xu L. Bayesian Ying Yang learning. Scholarpedia, 2007, 2(3): 1809

10 Baik J, Silverstein J W. Eigenvalues of large sample covariance matrices of spiked population models. Journal of Multivariate Analysis, 2006, 97(6): 1382-1408

11 Johnstone I M. High dimensional statistical inference and random matrices. In: Proceedings of International Congress of Mathematicians. 2006, 1-28

12 Paul D. Asymptotics of sample eigenstruture for a large dimensional spiked covariance model. Statistica Sinica, 2007, 17(4): 1617-1642

13 Kritchman S, Nadler B. Determining the number of components in a factor model from limited noisy data. Chemometrics & Intelligent Laboratory Systems, 2008, 94(1): 19-32

14 Perry P O, Wolfe P J. Minimax rank estimation for subspace tracking. Selected Topics in Signal Proceesing, 2010, 4(3): 504-513

15 Hu X, Xu L. A comparative investigation on subspace dimension determination. Neural Networks, 2004, 17(8-9): 1051-1059

16 Chen P, Wu T J, Yang J. A comparative study of model selection criteria for the number of signals. IET Radar, Sonar and Navigation, 2008, 2(3): 180-188

17 Zhang Q T, Wong K, Yip P, Reilly J. Statistical analysis of the performance of information theoretic criteria in the detection of the number of signals in array processing. IEEE Transactions on Acoustics, Speech and Signal Processing, 1989, 37(10): 1557-1567

18 Xu W, Kaveh M. Analysis of the performance and sensitivity of eigendecomposition-based detectors. IEEE Transactions on Signal Processing, 1995, 43(6): 1413-1426

19 Liavas A, Regalia P. On the behavior of information theoretic criteria for model order selection. IEEE Transactions on Signal Processing, 2001, 49(8): 1689-1695

20 Fishler E, Grosmann M, Messer H. Detection of signals by information theoretic criteria: general asymptotic performance analysis. IEEE Transactions on Signal Processing, 2002, 50(5): 1027-1036

21 Fishler E, Poor H. Estimation of the number of sources in unbalanced arrays via information theoretic criteria. IEEE Transactions on Signal Processing, 2005, 53(9): 3543-3553

22 Nadakuditi R, Edelman A. Sample eigenvalue based detection of high-dimensional signals in white noise using relatively few samples. IEEE Transactions on Signal Processing, 2008, 56(7): 2625-2638

23 Rissanen J. Modelling by the shortest data description. Automatica, 1978, 14(5): 465-471

24 Hoyle D C. Automatic PCA dimension selection for high dimensional data and small sample sizes. Journal of Machine Learning Research, 2008, 9(12): 2733-2759

25 Bishop C M. Variational principal components. In: Proceedings of the Ninth International Conference on Artificial Neural Networks. 1999, 1: 509-514

26 Schwarz G. Estimating the dimension of a model. Annals of Statistics, 1978, 6(2): 461-464

27 Bozdogan H. Model selection and Akaike’s Information Criterion (AIC): the general theory and its analytical extensions. Psychometrika, 1987, 52(3): 345-370

28 Hannan E, Quinn B. The determination of the order of an autoregression. Journal of the Royal Statistical Society. Series B, 1979, 41(2): 190-195

29 Johnstone I M. On the distribution of the largest eigenvalue in principal component anslysis. Annals of Statistics, 2001, 29(2): 295-327

30 Xu L. Bayesian Ying-Yang system, best harmony learning, and five action circling. Frontiers of Electrical and Electronic Engineering in China, 2010, 5(3): 281-328

31 Tu S, Xu L. Parameterizations make different model selections: empirical findings from factor analysis. Frontiers of Electrical and Electronic Engineering in China (in Press)

32 Xu L. Codimensional matrix pairing perspective of BYY harmony learning: hierarchy of bilinear systems, joint decomposition of data-covariance, and applications of network biology. Frontiers of Electrical and Electronic Engineering in China, 2011, 6(1): 86-119