The superior fault tolerance of artificial neural network training with a fault/noise injectionbased genetic algorithm

Abstract

Artificial neural networks (ANNs) are powerful computational tools that are designed to replicate the human brain and adopted to solve a variety of problems in many different fields. Fault tolerance (FT), an important property of ANNs, ensures their reliability when significant portions of a network are lost. In this paper, a fault/noise injection-based (FIB) genetic algorithm (GA) is proposed to construct fault-tolerant ANNs. The FT performance of an FIB-GA was compared with that of a common genetic algorithm, the back-propagation algorithm, and the modification of weights algorithm. The FIB-GA showed a slower fitting speed when solving the exclusive OR (XOR) problem and the overlapping classification problem, but it significantly reduced the errors in cases of single or multiple faults in ANN weights or nodes. Further analysis revealed that the fit weights showed no correlation with the fitting errors in the ANNs constructed with the FIB-GA, suggesting a relatively even distribution of the various fitting parameters. In contrast, the output weights in the training of ANNs implemented with the use the other three algorithms demonstrated a positive correlation with the errors. Our findings therefore indicate that a combination of the fault/noise injection-based method and a GA is capable of introducing FT to ANNs and imply that the distributed ANNs demonstrate superior FT performance.

Keywords

artificial neural networks / fault tolerance / genetic algorithm

References

[1] Almeida JS (2002) Predictive non-linear modeling of complex data by artificial neural networks. Curr Opin Biotechnol 13:72–76

[2] Arnone E, Francipane A, Noto LV, Scarbaci A, La Loggia G (2014) Strategies investigation in using artificial neural network for landslide susceptibility mapping: application to a Sicilian catchment. J Hydroinf 16:502–515

[3] Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S, Montazeri A (2015) Artificial neural networks in neurosurgery. J Neurol Neurosurg Psychiatry 86:251–256

[4] Basheer IA, Hajmeer M (2000) Artificial neural networks: fundamentals, computing, design, and application. J Microbiol Methods 43:3–31

[5] Baxt WG (1995) Application of artificial neural networks to clinical medicine. Lancet 346:1135–1138

[6] Braskie MN, Thompson PM (2014) A focus on structural brain imaging in the Alzheimer’s disease neuroimaging initiative. Biol Psychiatry 75:527–533

[7] Cavalieri S, Mirabella O (1999a) A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks. Neural Netw 12:91–106

[8] Cavalieri S, Mirabella O (1999b) A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks. Neural Netw 12:91–106

[9] Dybowski R, Gant V (1995) Artificial neural networks in pathology and medical laboratories. Lancet 346:1203–1207

[10] Emmerson MD, Damper RI (1993) Determining and improving the fault-tolerance of multilayer perceptrons in a pattern-recognition application. IEEE Trans Neural Netw 4:788–793

[11] Fayed N, Modrego PJ, Salinas GR, Gazulla J (2012) Magnetic resonance imaging based clinical research in Alzheimer’s disease. J Alzheimers Dis 31:S5–18

[12] Forrest S (1993) Genetic algorithms: principles of natural selection applied to computation. Science 261:872–878

[13] Forsstrom JJ, Dalton KJ (1995) Artificial neural networks for decision support in clinical medicine. Ann Med 27:509–517

[14] Gerlee P, Basanta D, Anderson AR (2011) Evolving homeostatic tissue using genetic algorithms. Prog Biophys Mol Biol 106:414–425

[15] Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley Pub. Co, Reading

[16] Hampson S (1991) Generalization and specialization in artificial neural networks. Prog Neurobiol 37:383–431

[17] Hampson S (1994) Problem solving in artificial neural networks. Prog Neurobiol 42:229–281

[18] Ho KI, Leung CS, Sum J (2010) Convergence and objective functions of some fault/noise-injection-based online learning algorithms for RBF networks. IEEE Trans Neural Netw 21:938–947

[19] Holland JH (1975) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. University of Michigan Press, Ann Arbor

[20] Hu X, Cammann H, Meyer HA, Miller K, Jung K, Stephan C (2013) Artificial neural networks and prostate cancer–tools for diagnosis and management. Nat Rev Urol 10:174–182

[21] Jamshidi M (2003) Tools for intelligent control: fuzzy controllers, neural networks and genetic algorithms. Philos Trans R Soc Lond A 361:1781–1808

[22] Jiang DD, Zhao ZY, Xu ZZ, Yao CP, Xu HW(2014) How to reconstruct end-to-end traffic based on time-frequency analysis and artificial neural network. Aeu-Int J Electron Commun 68:915–925

[23] Kamimura R, Konstantinov K, Stephanopoulos G (1996) Knowledge-based systems, artificial neural networks and pattern recognition: applications to biotechnological processes. Curr Opin Biotechnol 7:231–234

[24] Krogh A (2008) What are artificial neural networks? Nat Biotechnol 26:195–197

[25] Leardi R (2007) Genetic algorithms in chemistry. J Chromatogr A 1158:226–233

[26] Leung CS, Sum JP (2008) A fault-tolerant regularizer for RBF networks. IEEE Trans Neural Netw 19:493–507

[27] Li J, Pan P, Huang R, Shang H (2012) A meta-analysis of voxelbased morphometry studies of white matter volume alterations in Alzheimer’s disease. Neurosci Biobehav Rev 36:757–763

[28] Lisboa PJ (2002) A review of evidence of health benefit from artificial neural networks in medical intervention. Neural Netw 15:11–39

[29] Lisboa PJ, Taktak AF (2006) The use of artificial neural networks in decision support in cancer: a systematic review. Neural Netw 19:408–415

[30] Liu F, Wang J (2001) Genetic algorithms and its application to spectral analysis. Guang Pu Xue Yu Guang Pu Fen Xi 21:331–335

[31] Lovell BC, Bradley AP (1996) The multiscale classifier. IEEE Trans Pattern Anal Mach Intell 18:124–137

[32] Macia J, Sole RV (2009) Distributed robustness in cellular networks: insights from synthetic evolved circuits. J R Soc Interface 6:393–400

[33] Maddox J (1995) Genetics helping molecular-dynamics. Nature 376:209

[34] Mahdiani HR, Fakhraie SM, Lucas C (2012) Relaxed fault-tolerant hardware implementation of neural networks in the presence of multiple transient errors. IEEE Trans Neural Netw Learn Syst 23:1215–1228

[35] Mak SK, Sum PF, Leung CS (2011) Regularizers for fault tolerant multilayer feedforward networks. Neurocomputing 74:2028–2040

[36] Manning T, Sleator RD,Walsh P (2013) Naturally selecting solutions: the use of genetic algorithms in bioinformatics. Bioengineered 4:266–278

[37] Medler DA, Dawson MR (1994) Training redundant artificial neural networks: imposing biology on technology. Psychol Res 57:54–62

[38] Meurice N, Leherte L, Vercauteren DP (1998) Comparison of benzodiazepine-like compounds using topological analysis and genetic algorithms. SAR QSAR Environ Res 8:195–232

[39] Patel JL, Goyal RK (2007) Applications of artificial neural networks in medical science. Curr Clin Pharmacol 2:217–226

[40] Pedersen JT, Moult J (1996) Genetic algorithms for protein structure prediction. Curr Opin Struct Biol 6:227–231

[41] Pena-Malavera A, Bruno C, Fernandez E, Balzarini M (2014) Comparison of algorithms to infer genetic population structure from unlinked molecular markers. Stat Appl Genet Mol Biol 13:391–402

[42] Phatak DS, Koren I (1995a) Complete and partial fault-tolerance of feedforward neural nets. IEEE Trans Neural Netw 6:446–456

[43] Phatak DS, Koren I (1995b) Complete and partial fault tolerance of feedforward neural nets. IEEE Trans Neural Netw 6:446–456

[44] Pini L, Pievani M, Bocchetta M, Altomare D, Bosco P, Cavedo E, Galluzzi S, Marizzoni M, Frisoni GB (2016) Brain atrophy in Alzheimer’s disease and aging. Ageing Res Rev 28:30002

[45] Presnell SR, Cohen FE (1993) Artificial neural networks for pattern recognition in biochemical sequences. Annu Rev Biophys Biomol Struct 22:283–298

[46] Protzel PW, Palumbo DL, Arras MK (1993) Performance and faulttolerance of neural networks for optimization. IEEE Trans Neural Netw 4:600–614

[47] Rajan P, Tolley DA (2005) Artificial neural networks in urolithiasis. Curr Opin Urol 15:133–137

[48] Rodrigues PL, Rodrigues NF, Pinho ACM, Fonseca JC, Correia-Pinto J, Vilaca JL (2014) Automatic modeling of pectus excavatum corrective prosthesis using artificial neural networks. Med Eng Phys 36:1338–1345

[49] Rothlauf F, Goldberg DE, Heinzl A (2002) Network random keys: a tree representation scheme for genetic and evolutionary algorithms. Evol Comput 10:75–97

[50] Sasakawa T, Sawamoto J, Tsuji H (2014) Neural network to control output of hidden node according to input patterns. Am J Intell Syst 4:196–203

[51] Street ME, Buscema M, Smerieri A, Montanini L, Grossi E (2013) Artificial neural networks, and evolutionary algorithms as a systems biology approach to a data-base on fetal growth restriction. Prog Biophys Mol Biol 113:433–438

[52] Sum J, Leung ACS (2008) Prediction error of a fault tolerant neural network. Neurocomputing 72:653–658

[53] Tang W, Mao KZ, Mak LO, Ng GW (2010) Classification for overlapping classes using optimized overlapping region detection and soft decision. Paper presented at: information fusion

[54] Tchernev EB, Mulvaney RG, Phatak DS (2005) Investigating the fault tolerance of neural networks. Neural Comput 17:1646–1664

[55] Weber L (1998) Applications of genetic algorithms in molecular diversity. Curr Opin Chem Biol 2:381–385

[56] Weiner MW, Veitch DP, Aisen PS, Beckett LA, Cairns NJ, Cedarbaum J, Green RC, Harvey D, Jack CR, Jagust W (2015) 2014 update of the Alzheimer’s disease neuroimaging initiative: a review of papers published since its inception. Alzheimers Dement 11:001

[57] Willett P (1995) Genetic algorithms in molecular recognition and design. Trends Biotechnol 13:516–521

[58] Wu AH (2007) Use of genetic and nongenetic factors in warfarin dosing algorithms. Pharmacogenomics 8:851–861

[59] Xiong H, Wu J, Liu L (2010) Classification with class overlapping: a systematic study. ICEBI-10

[60] Xu C, Xu C (2013) Optimization analysis of dynamic sample number and hidden layer node number based on BP neural network. Springer, Berlin

RIGHTS & PERMISSIONS

2016 The Author(s) 2016. This article is published with open access at Springerlink.com and journal.hep.com.cn