Robust multi-layer extreme learning machine using bias-variance tradeoff

Abstract

As a new neural network model, extreme learning machine (ELM) has a good learning rate and generalization ability. However, ELM with a single hidden layer structure often fails to achieve good results when faced with large-scale multi-featured problems. To resolve this problem, we propose a multi-layer framework for the ELM learning algorithm to improve the model’s generalization ability. Moreover, noises or abnormal points often exist in practical applications, and they result in the inability to obtain clean training data. The generalization ability of the original ELM decreases under such circumstances. To address this issue, we add model bias and variance to the loss function so that the model gains the ability to minimize model bias and model variance, thus reducing the influence of noise signals. A new robust multi-layer algorithm called ML-RELM is proposed to enhance outlier robustness in complex datasets. Simulation results show that the method has high generalization ability and strong robustness to noise.

Keywords

extreme learning machine / deep neural network / robustness / unsupervised feature learning

References

[1] HuangG-b, ZhuQ-y, SiewC K. Extreme learning machine: A new learning scheme of feedforward neural networks [C]. 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), 2004985990

[2] HuangG-b, ZhuQ-y, SiewC K. Extreme learning machine: Theory and applications [J]. Neurocomputing, 2006, 70(1–3): 489-501

[3] HuangZ-y, YuY-l, GuJ, LiuH-ping. An efficient method for traffic sign recognition based on extreme learning machine [J]. IEEE Transactions on Cybernetics, 2017, 47(4): 920-933

[4] IosifidisA, TefasA, PitasI. Approximate kernel extreme learning machine for large scale data classification [J]. Neurocomputing, 2017, 219: 210-220

[5] YangY-m, WuQ M J. Extreme learning machine with subnetwork hidden nodes for regression and classification [J]. IEEE Transactions on Cybernetics, 2016, 46(12): 2885-2898

[6] XuX-z, ShanD, LiS, SunT-f, XiaoP-c, FanJ-ping. Multi-label learning method based on ML-RBF and laplacian ELM [J]. Neurocomputing, 2019, 331: 213-219

[7] InabaF K, Teatini SallesE O, PerronS, CaporossiG. DGR-ELM-distributed generalized regularized ELM for classification [J]. Neurocomputing, 2018, 275: 1522-1530

[8] ZhangY, WangY, ZhouG-x, JinJ, WangB, WangX-y, CichockiA. Multi-kernel extreme learning machine for EEG classification in brain-computer interfaces [J]. Expert Systems with Applications, 2018, 96: 302-310

[9] DaiH-z, CaoJ-w, WangT-l, DengM-q, YangZ-xin. Multilayer one-class extreme learning machine [J]. Neural Networks, 2019, 115: 11-22

[10] ChyzhykD, SavioA, GranaM. Computer aided diagnosis of schizophrenia on resting state fMRI data by ensembles of ELM [J]. Neural Networks, 2015, 68: 23-33

[11] WongP K, YangZ-x, VongC M, ZhongJ-hua. Real-time fault diagnosis for gas turbine generator systems using extreme learning machine [J]. Neurocomputing, 2014, 128: 249-257

[12] DuF, ZhangJ-s, JiN-n, ShiG, ZhangC-xia. An effective hierarchical extreme learning machine based multimodal fusion framework [J]. Neurocomputing, 2018, 322: 141-150

[13] ChenZ-c, WuL-j, ChengS-y, LinP-j, WuY, LinW-cheng. Intelligent fault diagnosis of photovoltaic arrays based on optimized kernel extreme learning machine and I–V characteristics [J]. Applied Energy, 2017, 204: 912-931

[14] DengW-y, ZhengQ-h, ChenL, XuX-bin. Research on extreme learning of neural networks [J]. Chinese Journal of Computers, 2010, 33(2): 279-287

[15] ZhangK, LuoM-xia. Outlier-robust extreme learning machine for regression problems [J]. Neurocomputing, 2015, 151: 1519-1527

[16] LuX-j, MingL, LiuW-b, LiH-xiong. Probabilistic regularized extreme learning machine for robust modeling of noise data [J]. IEEE Transactions on Cybernetics, 2018, 48(8): 2368-2377

[17] ZhaoY-p, HuQ-k, XuJ-g, LiB, HuangG, PanY-ting. A robust extreme learning machine for modeling a small-scale turbojet engine [J]. Applied Energy, 2018, 218: 22-35

[18] HintonG E, OsinderoS, TehY W. A fast learning algorithm for deep belief nets [J]. Neural Comput, 2006, 18(7): 1527-1554

[19] KasunL L C, ZhouH-m, HuangG-b, VongC. Representational learning with extreme learning machine for big data [J]. IEEE Intelligent System, 201314

[20] TangJ-x, DengC-w, HuangG-bin. Extreme learning machine for multilayer perceptron [J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(4): 809-821

[21] ChenL-j, HoneineP, QuH, ZhaoJ-h, SunXia. Correntropy-based robust multilayer extreme learning machines [J]. Pattern Recognition, 2018, 84: 357-370

[22] Le CunY, HuangF J, BottouL. Gradient-based learning applied to document recognition [C]. Proceedings of the IEEE, 1998, 86(11): 2278-2324

[23] Le CunY, HuangF J, BottouL. Learning methods for generic object recognition with invariance to pose and lighting. Conference on Computer Vision and Pattern Recognition, 2004, Washington DC, IEEE Comp Soc, 97104

[24] LichmanMUCI machine learning repository [D], 2013, California, Irvine, CA, USA, School Inf Comput Sci Univ

[25] WiensT S, DaleB C, BoyceM S. Three-way k-fold cross-validation of resource selection functions [J]. Ecological Modelling, 2008, 212(34): 244-255

[26] SuykensJ A K, VandewalleJ. Least squares support vector machine classifiers [J]. Neural Process Lett, 1999, 9(3): 293-300

[27] ZhouH-m, HuangG-b, LinZ-p, WangH, SohY C. Stacked extreme learning machines [J]. IEEE Transactions on Cybernetics, 2014, 45(9): 2013-2025