The superior fault tolerance of artificial neural network training with a fault/noise injectionbased genetic algorithm

Abstract

Artificial neural networks (ANNs) are powerful computational tools that are designed to replicate the human brain and adopted to solve a variety of problems in many different fields. Fault tolerance (FT), an important property of ANNs, ensures their reliability when significant portions of a network are lost. In this paper, a fault/noise injection-based (FIB) genetic algorithm (GA) is proposed to construct fault-tolerant ANNs. The FT performance of an FIB-GA was compared with that of a common genetic algorithm, the back-propagation algorithm, and the modification of weights algorithm. The FIB-GA showed a slower fitting speed when solving the exclusive OR (XOR) problem and the overlapping classification problem, but it significantly reduced the errors in cases of single or multiple faults in ANN weights or nodes. Further analysis revealed that the fit weights showed no correlation with the fitting errors in the ANNs constructed with the FIB-GA, suggesting a relatively even distribution of the various fitting parameters. In contrast, the output weights in the training of ANNs implemented with the use the other three algorithms demonstrated a positive correlation with the errors. Our findings therefore indicate that a combination of the fault/noise injection-based method and a GA is capable of introducing FT to ANNs and imply that the distributed ANNs demonstrate superior FT performance.

Key words： artificial neural networks; fault tolerance; genetic algorithm

References

1 Almeida JS (2002) Predictive non-linear modeling of complex data by artificial neural networks. Curr Opin Biotechnol 13:72–76

2 Arnone E, Francipane A, Noto LV, Scarbaci A, La Loggia G (2014) Strategies investigation in using artificial neural network for landslide susceptibility mapping: application to a Sicilian catchment. J Hydroinf 16:502–515

3 Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S, Montazeri A (2015) Artificial neural networks in neurosurgery. J Neurol Neurosurg Psychiatry 86:251–256

4 Basheer IA, Hajmeer M (2000) Artificial neural networks: fundamentals, computing, design, and application. J Microbiol Methods 43:3–31

5 Baxt WG (1995) Application of artificial neural networks to clinical medicine. Lancet 346:1135–1138

6 Braskie MN, Thompson PM (2014) A focus on structural brain imaging in the Alzheimer’s disease neuroimaging initiative. Biol Psychiatry 75:527–533

7 Cavalieri S, Mirabella O (1999a) A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks. Neural Netw 12:91–106

8 Cavalieri S, Mirabella O (1999b) A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks. Neural Netw 12:91–106

9 Dybowski R, Gant V (1995) Artificial neural networks in pathology and medical laboratories. Lancet 346:1203–1207

10 Emmerson MD, Damper RI (1993) Determining and improving the fault-tolerance of multilayer perceptrons in a pattern-recognition application. IEEE Trans Neural Netw 4:788–793

11 Fayed N, Modrego PJ, Salinas GR, Gazulla J (2012) Magnetic resonance imaging based clinical research in Alzheimer’s disease. J Alzheimers Dis 31:S5–18

12 Forrest S (1993) Genetic algorithms: principles of natural selection applied to computation. Science 261:872–878

13 Forsstrom JJ, Dalton KJ (1995) Artificial neural networks for decision support in clinical medicine. Ann Med 27:509–517

14 Gerlee P, Basanta D, Anderson AR (2011) Evolving homeostatic tissue using genetic algorithms. Prog Biophys Mol Biol 106:414–425

15 Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley Pub. Co, Reading

16 Hampson S (1991) Generalization and specialization in artificial neural networks. Prog Neurobiol 37:383–431

17 Hampson S (1994) Problem solving in artificial neural networks. Prog Neurobiol 42:229–281

18 Ho KI, Leung CS, Sum J (2010) Convergence and objective functions of some fault/noise-injection-based online learning algorithms for RBF networks. IEEE Trans Neural Netw 21:938–947

19 Holland JH (1975) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. University of Michigan Press, Ann Arbor

20 Hu X, Cammann H, Meyer HA, Miller K, Jung K, Stephan C (2013) Artificial neural networks and prostate cancer–tools for diagnosis and management. Nat Rev Urol 10:174–182

21 Jamshidi M (2003) Tools for intelligent control: fuzzy controllers, neural networks and genetic algorithms. Philos Trans R Soc Lond A 361:1781–1808

22 Jiang DD, Zhao ZY, Xu ZZ, Yao CP, Xu HW(2014) How to reconstruct end-to-end traffic based on time-frequency analysis and artificial neural network. Aeu-Int J Electron Commun 68:915–925

23 Kamimura R, Konstantinov K, Stephanopoulos G (1996) Knowledge-based systems, artificial neural networks and pattern recognition: applications to biotechnological processes. Curr Opin Biotechnol 7:231–234

24 Krogh A (2008) What are artificial neural networks? Nat Biotechnol 26:195–197

25 Leardi R (2007) Genetic algorithms in chemistry. J Chromatogr A 1158:226–233

26 Leung CS, Sum JP (2008) A fault-tolerant regularizer for RBF networks. IEEE Trans Neural Netw 19:493–507

27 Li J, Pan P, Huang R, Shang H (2012) A meta-analysis of voxelbased morphometry studies of white matter volume alterations in Alzheimer’s disease. Neurosci Biobehav Rev 36:757–763

28 Lisboa PJ (2002) A review of evidence of health benefit from artificial neural networks in medical intervention. Neural Netw 15:11–39

29 Lisboa PJ, Taktak AF (2006) The use of artificial neural networks in decision support in cancer: a systematic review. Neural Netw 19:408–415

30 Liu F, Wang J (2001) Genetic algorithms and its application to spectral analysis. Guang Pu Xue Yu Guang Pu Fen Xi 21:331–335

31 Lovell BC, Bradley AP (1996) The multiscale classifier. IEEE Trans Pattern Anal Mach Intell 18:124–137

32 Macia J, Sole RV (2009) Distributed robustness in cellular networks: insights from synthetic evolved circuits. J R Soc Interface 6:393–400

33 Maddox J (1995) Genetics helping molecular-dynamics. Nature 376:209

34 Mahdiani HR, Fakhraie SM, Lucas C (2012) Relaxed fault-tolerant hardware implementation of neural networks in the presence of multiple transient errors. IEEE Trans Neural Netw Learn Syst 23:1215–1228

35 Mak SK, Sum PF, Leung CS (2011) Regularizers for fault tolerant multilayer feedforward networks. Neurocomputing 74:2028–2040

36 Manning T, Sleator RD,Walsh P (2013) Naturally selecting solutions: the use of genetic algorithms in bioinformatics. Bioengineered 4:266–278

37 Medler DA, Dawson MR (1994) Training redundant artificial neural networks: imposing biology on technology. Psychol Res 57:54–62

38 Meurice N, Leherte L, Vercauteren DP (1998) Comparison of benzodiazepine-like compounds using topological analysis and genetic algorithms. SAR QSAR Environ Res 8:195–232

39 Patel JL, Goyal RK (2007) Applications of artificial neural networks in medical science. Curr Clin Pharmacol 2:217–226

40 Pedersen JT, Moult J (1996) Genetic algorithms for protein structure prediction. Curr Opin Struct Biol 6:227–231

41 Pena-Malavera A, Bruno C, Fernandez E, Balzarini M (2014) Comparison of algorithms to infer genetic population structure from unlinked molecular markers. Stat Appl Genet Mol Biol 13:391–402

42 Phatak DS, Koren I (1995a) Complete and partial fault-tolerance of feedforward neural nets. IEEE Trans Neural Netw 6:446–456

43 Phatak DS, Koren I (1995b) Complete and partial fault tolerance of feedforward neural nets. IEEE Trans Neural Netw 6:446–456

44 Pini L, Pievani M, Bocchetta M, Altomare D, Bosco P, Cavedo E, Galluzzi S, Marizzoni M, Frisoni GB (2016) Brain atrophy in Alzheimer’s disease and aging. Ageing Res Rev 28:30002

45 Presnell SR, Cohen FE (1993) Artificial neural networks for pattern recognition in biochemical sequences. Annu Rev Biophys Biomol Struct 22:283–298

46 Protzel PW, Palumbo DL, Arras MK (1993) Performance and faulttolerance of neural networks for optimization. IEEE Trans Neural Netw 4:600–614

47 Rajan P, Tolley DA (2005) Artificial neural networks in urolithiasis. Curr Opin Urol 15:133–137

48 Rodrigues PL, Rodrigues NF, Pinho ACM, Fonseca JC, Correia-Pinto J, Vilaca JL (2014) Automatic modeling of pectus excavatum corrective prosthesis using artificial neural networks. Med Eng Phys 36:1338–1345

49 Rothlauf F, Goldberg DE, Heinzl A (2002) Network random keys: a tree representation scheme for genetic and evolutionary algorithms. Evol Comput 10:75–97

50 Sasakawa T, Sawamoto J, Tsuji H (2014) Neural network to control output of hidden node according to input patterns. Am J Intell Syst 4:196–203

51 Street ME, Buscema M, Smerieri A, Montanini L, Grossi E (2013) Artificial neural networks, and evolutionary algorithms as a systems biology approach to a data-base on fetal growth restriction. Prog Biophys Mol Biol 113:433–438

52 Sum J, Leung ACS (2008) Prediction error of a fault tolerant neural network. Neurocomputing 72:653–658

53 Tang W, Mao KZ, Mak LO, Ng GW (2010) Classification for overlapping classes using optimized overlapping region detection and soft decision. Paper presented at: information fusion

54 Tchernev EB, Mulvaney RG, Phatak DS (2005) Investigating the fault tolerance of neural networks. Neural Comput 17:1646–1664

55 Weber L (1998) Applications of genetic algorithms in molecular diversity. Curr Opin Chem Biol 2:381–385

56 Weiner MW, Veitch DP, Aisen PS, Beckett LA, Cairns NJ, Cedarbaum J, Green RC, Harvey D, Jack CR, Jagust W (2015) 2014 update of the Alzheimer’s disease neuroimaging initiative: a review of papers published since its inception. Alzheimers Dement 11:001

57 Willett P (1995) Genetic algorithms in molecular recognition and design. Trends Biotechnol 13:516–521

58 Wu AH (2007) Use of genetic and nongenetic factors in warfarin dosing algorithms. Pharmacogenomics 8:851–861

59 Xiong H, Wu J, Liu L (2010) Classification with class overlapping: a systematic study. ICEBI-10

60 Xu C, Xu C (2013) Optimization analysis of dynamic sample number and hidden layer node number based on BP neural network. Springer, Berlin