A Novel Flexible Kernel Density Estimator for Multimodal Probability Density Functions

Abstract

Estimating probability density functions (PDFs) is critical in data analysis, particularly for complex multimodal distributions. traditional kernel density estimator (KDE) methods often face challenges in accurately capturing multimodal structures due to their uniform weighting scheme, leading to mode loss and degraded estimation accuracy. This paper presents the fiexible kernel density estimator (F-KDE), a novel nonparametric approach designed to address these limitations. F-KDE introduces the concept of kernel unit inequivalence, assigning adaptive weights to each kernel unit, which better models local density variations in multimodal data. The method optimises an objective function that integrates estimation error and log-likelihood, using a particle swarm optimisation (PSO) algorithm that automatically determines optimal weights and bandwidths. Through extensive experiments on synthetic and real-world datasets, we demonstrated that (1) the weights and bandwidths in F-KDE stabilise as the optimisation algorithm iterates, (2) F-KDE effectively captures the multimodal characteristics and (3) F-KDE outperforms state-of-the-art density estimation methods regarding accuracy and robustness. The results confirm that F-KDE provides a valuable solution for accurately estimating multimodal PDFs.

Keywords

data analysis / learning (artificial intelligence) / machine learning / optimisation / probability

References

[1] Y. L. He, G. L. Ou, P. Fournier-Viger, J. Z. Huang, and P. N. Suganthan, “A Novel Dependency-Oriented Mixed-Attribute Data Classification Method,” Expert Systems with Applications 199 (2022): 116782, https://doi.org/10.1016/j.eswa.2022.116782.

[2] M. A. Elfarra and M. Kaya, “Estimation of Electricity Cost of Wind Energy Using Monte Carlo Simulations Based on Nonparametric and Parametric Probability Density Functions,” Alexandria Engineering Journal 60, no. 4 (2021): 3631-3640, https://doi.org/10.1016/j.aej.2021.02.027.

[3] H. Che-Ngoc, T. Nguyen-Trang, T. Nguyen-Bao, T. Nguyen-Thoi, and T. Vo-Van, “A New Approach for Face Detection Using the Maximum Function of Probability Density Functions,” Annals of Operations Research 312, no. 1 (2022): 99-119, https://doi.org/10.1007/s10479-020-03823-1.

[4] Z. Li, Y. Sun, L. Zhang, and J. Tang, “CTNet: Context-Based Tandem Network for Semantic Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 12 (2021): 9904-9917, https://doi.org/10.1109/tpami.2021.3132068.

[5] X. Wang, H. Bai, L. Yu,Y. Zhao, and J. Xiao, “Towards the Un-charted: Density-Descending Feature Perturbation for Semi-Supervised Semantic Segmentation,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024), 3303-3312, https://doi.org/10.1109/cvpr52733.2024.00318.

[6] D. Fu, Z. Zhang, J. Fan, “Dense Projection for Anomaly Detection,” in Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 38. (2024): 8398-8408, https://doi.org/10.1609/aaai.v38i8.28682.

[7] E. Parzen, “On Estimation of a Probability Density Function and Mode,” Annals of Mathematical Statistics 33, no. 3 (1962): 1065-1076, https://doi.org/10.1214/aoms/1177704472.

[8] M. Rosenblatt, “Remarks on Some Nonparametric Estimates of a Density Function,” Annals of Mathematical Statistics 27, no. 3 (1956): 832-837, https://doi.org/10.1214/aoms/1177728190.

[9] A. Rajan, Y. C. Kuang, M. P. L. Ooi, S. N. Demidenko, and H. Carstens, “Moment-Constrained Maximum Entropy Method for Expanded Uncertainty Evaluation,” IEEE Access 6 (2018): 4072-4082, https://doi.org/10.1109/access.2017.2787736.

[10] G. Li, Y. Wang, Y. Zeng, and W. He, “A New Maximum Entropy Method for Estimation of Multimodal Probability Density Function,” Applied Mathematical Modelling 102 (2022): 137-152, https://doi.org/10.1016/j.apm.2021.09.029.

[11] Z. Zhang, C. Jiang, X. Han, and X. Ruan, “A High-Precision Prob-abilistic Uncertainty Propagation Method for Problems Involving Multimodal Distributions,” Mechanical Systems and Signal Processing 126 (2019): 21-41, https://doi.org/10.1016/j.ymssp.2019.01.031.

[12] W. Zhang, Z. Zhang, H. C. Chao, and F. H. Tseng, “Kernel Mixture Model for Probability Density Estimation in Bayesian Classifiers,” Data Mining and Knowledge Discovery 32, no. 3 (2018): 675-707, https://doi.org/10.1007/s10618-018-0550-5.

[13] A. Mészáros, J. F. Schumann, J. Alonso-Mora,A. Zgonnikov, and J. Kober, “ROME: Robust Multi-Modal Density Estimator,” in Proceedings of the 33rd International Joint Conference on Artificial Intelligence, (2024), 24.

[14] T. Nagler and C. Czado, “Evading the Curse of Dimensionality in Nonparametric Density Estimation With Simplified Vine Copulas,” Journal of Multivariate Analysis 151 (2016): 69-89, https://doi.org/10.1016/j.jmva.2016.07.003.

[15] A. Tewari, “On the Estimation of Gaussian Mixture Copula Models,” in International Conference on Machine Learning (PMLR, 2023), 34090-34104.

[16] K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf, “Kernel Mean Embedding of Distributions: A Review and Beyond,” Foundations and Trends in Machine Learning 10, no. 1-2 (2017): 1-141, https://doi.org/10.1561/2200000060.

[17] Y. Chen, M. Welling, A. Smola,“Super-Samples From Kernel Herding,” arXiv preprint arXiv:1203.3472 (2010): 109-116.

[18] E. C. Cortes and C. Scott, “Sparse Approximation of a Kernel Mean,” IEEE Transactions on Signal Processing 65, no. 5 (2016): 1310-1323, https://doi.org/10.1109/tsp.2016.2628353.

[19] J. M. Phillips and W. M. Tai, “Near-Optimal Coresets of Kernel Density Estimates,” Discrete & Computational Geometry 63, no. 4 (2020): 867-887, https://doi.org/10.1007/s00454-019-00134-6.

[20] B. Coleman and A. Shrivastava,“Sub-Linear Race Sketches for Approximate Kernel Density Estimation on Streaming Data,” Pro-ceedings of The Web Conference 2020 2020): 1739-1749, https://doi.org/10.1145/3366423.3380244.

[21] C. Luo and A. Shrivastava,“Arrays of (Locality-Sensitive) Count Estimators (Ace) Anomaly Detection on the Edge,” in Proceedings of the 2018 World Wide Web Conference on World Wide Web-WWW '18 (2018): 1439-1448, https://doi.org/10.1145/3178876.3186056.

[22] M. Charikar and P. Siminelakis, “Hashing-Based-Estimators for Kernel Density in High Dimensions,” in 2017 IEEE 58th Annual Sympo-sium on Foundations of Computer Science (FOCS) IEEE, 2017), 1032-1043.

[23] A. Backurs, P. Indyk, and T. Wagner, “Space and Time Efficient Kernel Density Estimation in High Dimensions,” Advances in Neural Information Processing Systems 32 (2019), https://dl.acm.org/doi/abs/10.5555/3454287.3455702.

[24] M. Charikar, M. Kapralov, N. Nouri, and P. Siminelakis, “Kernel Density Estimation Through Density Constrained near Neighbor Search,” in 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) IEEE, 2020), 172-183.

[25] A. Andoni, T. Laarhoven, I. Razenshteyn, and E. Waingarten, “Optimal Hashing-Based Time-Space Trade-Offs for Approximate near Neighbors,” in Proceedings of the twenty-eighth Annual ACM-SIAM Symposium on Discrete Algorithms, (SIAM, 2017), 47-66.

[26] Duin, “On the Choice of Smoothing Parameters for Parzen Estimators of Probability Density Functions,” IEEE Transactions on Computers 100, no. 11 (1976): 1175-1179, https://doi.org/10.1109/TC.1976.1674577.

[27] M. Rudemo, “Empirical Choice of Histograms and Kernel Density Estimators,” Scandinavian Journal of Statistics (1982): 65-78.

[28] A. W. Bowman, “An Alternative Method of Cross-Validation for the Smoothing of Density Estimates,” Biometrika 71, no. 2 (1984): 353-360, https://doi.org/10.1093/biomet/71.2.353.

[29] D. W. Scott and G. R. Terrell, “Biased and Unbiased Cross-Validation in Density Estimation,” Journal of the American Statistical Association 82, no. 400 (1987): 1131-1146, https://doi.org/10.2307/2289391.

[30] T. Duong and M. L. Hazelton, “Cross-Validation Bandwidth Matrices for Multivariate Kernel Density Estimation,” Scandinavian Journal of Statistics 32, no. 3 (2005): 485-506, https://doi.org/10.1111/j.1467-9469.2005.00445.x.

[31] M. Jones and R. Kappenman, “On a Class of Kernel Density Esti-mate Bandwidth Selectors,” Scandinavian Journal of Statistics (1992): 337-349.

[32] W. Stute, “Modified Cross-Validation in Density Estimation,” Journal of Statistical Planning and Inference 30, no. 3 (1992): 293-305, https://doi.org/10.1016/0378-3758(92)90157-n.

[33] W. Feluch and J. Koronacki, “A Note on Modified Cross-Validation in Density Estimation,” Computational Statistics & Data Analysis 13, no. 2 (1992): 143-151, https://doi.org/10.1016/0167-9473(92)90002-w.

[34] M. D. M. Miranda,J. P. Nielsen, and S. A. Sperlich, “One-Sided Cross-Validation for Density Estimation With an Application to Oper-ational Risk,” Operational Risk toward Basel III: Best Practices and Issues in Modeling, Management, and Regulation (2009): 177-195, https://doi.org/10.1002/9781118267066.ch9.

[35] O. Y. Savchuk, J. D. Hart, and S. J. Sheather, “Indirect Cross-Validation for Density Estimation,” Journal of the American Statistical Association 105, no. 489 (2010): 415-423, https://doi.org/10.1198/jasa.2010.tm08532.

[36] C. Tenreiro, “A Weighted Least-Squares Cross-Validation Band-width Selector for Kernel Density Estimation,” Communications in Statistics—Theory and Methods 46, no. 7 (2017): 3438-3458, https://doi.org/10.1080/03610926.2015.1062108.

[37] D. W. Scott,Multivariate Density Estimation: Theory, Practice, and Visualization (John Wiley & Sons, 1992).

[38] B. W. Silverman, Density Estimation for Statistics and Data Analysis (Routledge, 1986).

[39] S. J. Sheather and M. C. Jones, “A Reliable Data-based Bandwidth Selection Method for Kernel Density Estimation,” Journal of the Royal Statistical Society: Series B 53, no. 3 (1991): 683-690, https://doi.org/10.1111/j.2517-6161.1991.tb01857.x.

[40] M. P. Wand and M. C. Jones, “Multivariate Plug-In Bandwidth Se-lection,” Computational Statistics 9, no. 2 (1994): 97-116.

[41] Z. Botev, J. Grotowski and D. Kroese, “Kernel Density Estimation via Diffusion,” Annals of Statistics 38, no. 5 (2010): 2916-2957, https://doi.org/10.1214/10-aos799.

[42] T. Duong and M. Hazelton, “Plug-In Bandwidth Matrices for Bivariate Kernel Density Estimation,” Journal of Nonparametric Statis-tics 15, no. 1 (2003): 17-30, https://doi.org/10.1080/10485250306039.

[43] C. Tenreiro, “Bandwidth Selection for Kernel Density Estimation: A Hermite Series-Based Direct Plug-In Approach,” Journal of Statistical Computation and Simulation 90, no. 18 (2020): 3433-3453, https://doi.org/10.1080/00949655.2020.1804571.

[44] C. C. Taylor, “Bootstrap Choice of the Smoothing Parameter in Kernel Density Estimation,” Biometrika 76, no. 4 (1989): 705-712, https://doi.org/10.1093/biomet/76.4.705.

[45] R. Cao, “Bootstrapping the Mean Integrated Squared Error,” Journal of Multivariate Analysis 45, no. 1 (1993): 137-160, https://doi.org/10.1006/jmva.1993.1030.

[46] J. E. Chacón, J. Montanero, and A. G. Nogales, “Bootstrap Band-width Selection Using an H-Dependent Pilot Bandwidth,” Scandinavian Journal of Statistics 35, no. 1 (2008): 139-157, https://doi.org/10.1111/j.1467-9469.2007.00565.x.

[47] E. Mammen, M. D. Martinez Miranda, J. P. Nielsen, and S. Sperlich, “Do-Validation for Kernel Density Estimation,” Journal of the American Statistical Association 106, no. 494 (2011): 651-660, https://doi.org/10.1198/jasa.2011.tm08687.

[48] J. Ćwik and J. Koronacki, “A Combined Adaptive-Mixtures/Plug-In Estimator of Multivariate Probability Densities,” Computational Statistics & Data Analysis 26, no. 2 (1997): 199-218, https://doi.org/10.1016/s0167-9473(97)00032-7.

[49] J. Kim and C. D. Scott, “Robust Kernel Density Estimation ” Journal of Machine Learning Research 13, no. 1 (2012): 2529-2565, https://dl. acm.org/doi/abs/10.5555/2503308.2503323.

[50] P. Humbert,B. Le Bars, and L. Minvielle, “Robust Kernel Density Estimation With Median-of-Means Principle,” in International Confer-ence on Machine Learning (PMLR, 2022), 9444-9465.

[51] S. Wang, A. Li, K. Wen, and X. Wu, “Robust Kernels for Kernel Density Estimation,” Economics Letters 191 (2020): 109138, https://doi.org/10.1016/j.econlet.2020.109138.

[52] X. Zhang and M. D. Pandey, “Structural Reliability Analysis Based on the Concepts of Entropy, Fractional Moment and Dimensional Reduction Method,” Structural Safety 43 (2013): 28-40, https://doi.org/10.1016/j.strusafe.2013.03.001.

[53] S. L. Yoo, J. Y. Jeong, and J. B. Yim, “Estimating Suitable Proba-bility Distribution Function for Multimodal Traffic Distribution Function,” Journal of the Korean Society of Marine Environment & Safety 21, no. 3 (2015): 253-258, https://doi.org/10.7837/kosomes.2015.21.3.253.

[54] M. Siena, C. Recalcati, A. Guadagnini, and M. Riva, “A Gaussian-Mixture Based Stochastic Framework for the Interpretation of Spatial Heterogeneity in Multimodal Fields,” Journal of Hydrology 617 (2023): 128849, https://doi.org/10.1016/j.jhydrol.2022.128849.

[55] E. Amrani, R. Ben-Ari, D. Rotman, A. Bronstein,“Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 8 (2021): 6644-6652, https://doi.org/10.1609/aaai.v35i8.16822.

[56] J. Q. Chen, Y. L. He, Y. C. Cheng, P. Fournier-Viger, and J. Z. Huang, “A Multiple Kernel-Based Kernel Density Estimator for Multimodal Probability Density Functions,” Engineering Applications of Artificial Intelligence 132 (2024): 107979, https://doi.org/10.1016/j.engappai.2024.107979.

[57] A. Tewari, M. J. Giering, and A. Raghunathan, “Parametric Char-acterization of Multimodal Distributions With Non-Gaussian Modes,” in 2011 IEEE 11th International Conference on Data Mining Workshops IEEE, 2011), 286-292.

[58] A. E. Bilgrau, P. S. Eriksen, J. G. Rasmussen, H. E. Johnsen, K. Dybkær and M. Bøgsted, “GMCM: Unsupervised Clustering and Meta-Analysis Using Gaussian Mixture Copula Models,” Journal of Statistical Software 70, no. 2 (2016): 1-23, https://doi.org/10.18637/jss.v070.i02.

[59] I. Kosmidis and D. Karlis, “Model-Based Clustering Using Copulas With Applications,” Statistics and Computing 26, no. 5 (2016): 1079-1099, https://doi.org/10.1007/s11222-015-9590-5.

[60] M. Marbac, C. Biernacki, and V. Vandewalle, “Model-Based Clus-tering of Gaussian Copulas for Mixed Data,” Communications in Statistics—Theory and Methods 46, no. 23 (2017): 11635-11656, https://doi.org/10.1080/03610926.2016.1277753.

[61] M. Clerc, “The Swarm and the Queen: Towards a Deterministic and Adaptive Particle Swarm Optimization,” in Proceedings of the 1999 Congress on Evolutionary computation-CEC99 (Cat. No. 99TH8406) (IEEE, 1999), 1951-1957.

[62] R. C. Eberhart and Y. Shi, “Comparing Inertia Weights and Constriction Factors in Particle Swarm Optimization,” in Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No. 00TH8512) (IEEE, 2000), 84-88.

Funding

Natural Science Foundation of Guangdong Province(Grant 2023A1515011667)

Science and Technology Major Project of Shenzhen(Grant KJZD20230923114809020)

Key Basic Research Foundation of Shenzhen(Grant JCYJ20220818100205012)