Immunity-Based Orthogonal Weights Modification Algorithm

Abstract

Recently, the catastrophic forgetting problem of neural networks in the process of continual learning (CL) has attracted more and more attention with the development of deep learning. The orthogonal weight modification (OWM) algorithm to some extent overcomes the catastrophic forgetting problem in CL. It is well-known that the mapping rule learned by the network is usually not accurate in the early stage of neural network training. Our main idea is to establish an immune mechanism in CL, which rejects unreliable mapping rules at the beginning of the training until those are reliable enough. Our algorithm showed a very good competitive advantage in the permuted and disjoint MNIST tasks and disjoint CIFAR-10 tasks. As for the more challenging task of Chinese handwriting character recognition, our algorithm showed a notable improvement compared with the OWM algorithm. In view of the context-dependent processing (CDP) module in [37], we revealed that the module may result in a loss of information and we proposed a modified CDP module to overcome this weakness. The performance of the system with the modified CDP module outperforms the original one in the CelebFaces attributed recognition task. Besides continual multi-task, we also considered a single task, where the immunity-based OWM (IOWM) algorithm was designed as an optimization solver of neural networks for low-dose computed tomography (CT) denoising task.

Keywords

Image processing / Neural network / Continual learning (CL) / Computed tomography (CT) / 68T07 / 68U10

References

[1] Abraham WC, Robins A. Memory retention-the synaptic stability versus plasticity dilemma. Trends Neurosci.. 2005, 28(2): 73-78

[2] Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., Ranzato, M.: Continual learning with tiny episodic memories. In: Workshop on Multi-Task and Lifelong Reinforcement Learning. arXiv:1902.10486 (2019)

[3] Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, Zhou J, Wang G. Low-dose ct with a residual encoder-decoder convolutional neural network. IEEE Trans. Med. Imaging. 2017, 36(12): 2524-2535

[4] Chen H, Zhang Y, Zhang W, Liao P, Wang G. Low-dose ct via convolutional neural network. Biomed. Opt. Express. 2017, 8(2): 679

[5] Coop R, Mishtal A, Arel I. Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting. IEEE Trans. Neural Netw. Learn. Syst.. 2013, 24(10): 1623-1634

[6] Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D.: PathNet: evolution channels gradient descent in super neural networks. arXiv:1701.08734 (2017)

[7] French RM. Catastrophic forgetting in connectionist networks. Trends Cogn. Sci.. 1999, 3(4): 128-135

[8] Gepperth A, Karaoguz C. A bio-inspired incremental learning architecture for applied perceptual problems. Cogn. Comput.. 2016, 8(5): 924-934

[9] Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. In: The Second International Conference on Learning Representations (2014)

[10] He, X., Jaeger, H.: Overcoming catastrophic interference using conceptor-aided backpropagation. In: The Sixth International Conference on Learning Representations (2018)

[11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE (2016)

[12] Hetherington, P.A., Seidenberg, M.S.: Is there “catastrophic interference” in connectionist networks? In: Proceedings of the 11th Annual Conference Cognitive Science Society, pp. 26–33. Psychology Press (2014)

[13] Hu, W., Lin, Z., Liu, B., Tao, C., Tao, Z., Ma, J., Zhao, D., Yan, R.: Overcoming catastrophic forgetting for continual learning via model adaptation. In: The Seventh International Conference on Learning Representations (2019)

[14] Karlik B, Olgac AV. Performance analysis of various activation functions in generalized mlp architectures of neural networks. Int. J. Artif. Intell. Expert Syst.. 2011, 1(4): 111-122

[15] Kemker, R., Kanan, C.: FearNet: brain-inspired model for incremental learning. In: The Sixth International Conference on Learning Representations. arXiv:1711.10563 (2018)

[16] Kemker, R., McClure, M., Abitino, A., Hayes, T., Kanan, C.: Measuring catastrophic forgetting in neural networks. Proc. AAAI Conf. Artif. Intell. 32(1) (2018)

[17] Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: The Third International Conference on Learning Representations. arXiv:1412.6980 (2015)

[18] Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska-Barwinska A, et al. . Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci.. 2017, 114(13): 3521-3526

[19] Kumaran D, Hassabis D, McClelland JL. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends Cogn. Sci.. 2016, 20(7): 512-534

[20] Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. Adv. Neural Inf. Process. Syst. 30, 4652–4662 (2017)

[21] Liang, X.: A study of immunity-based continual learning algorithm (in Chinese). Master’s thesis, Capital Normal University, China (2021)

[22] Liu, C.-L., Yin, F., Wang, D.-H., Wang, Q.-F.: Chinese handwriting recognition contest 2010. In: 2010 Chinese Conference on Pattern Recognition, pp. 1–5. IEEE (2010)

[23] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. IEEE (2015)

[24] Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Dec 4–9, 2017, Long Beach, CA, USA, pp. 6467–6476 (2017)

[25] Masse NY, Grant GD, Freedman DJ. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proc. Natl. Acad. Sci.. 2018, 115(44): 10467-10475

[26] McClelland JL, McNaughton BL, O’Reilly RC. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol. Rev.. 1995, 102(3): 419

[27] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989)

[28] Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, pp. 807–814 (2010)

[29] Parisi GI, Kemker R, Part JL, Kanan C, Wermter S. Continual lifelong learning with neural networks: A review. Neural Netw.. 2019, 113: 54-71

[30] Ratcliff R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol. Rev.. 1990, 97(2): 285

[31] Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45–50. ELRA, Valletta, Malta (2010)

[32] Robins A. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci.. 1995, 7(2): 123-146

[33] Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Nature. 1986, 323(6088): 533-536

[34] Saha, G., Garg, I., Roy, K.: Gradient projection memory for continual learning. In: The Ninth International Conference on Learning Representations (2021)

[35] Sharkey NE, Sharkey AJ. An analysis of catastrophic interference. Connect. Sci.. 1995, 7(3–4): 301-330

[36] Yang Q, Yan P, Zhang Y, Yu H, Shi Y, Mou X, Kalra MK, Zhang Y, Sun L, Wang G. Low-dose ct image denoising using a generative adversarial network with Wasserstein distance and perceptual loss. IEEE Trans. Med. Imaging. 2018, 37(6): 1348-1357

[37] Zeng G, Chen Y, Cui B, Yu S. Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell.. 2019, 1(8): 364-372

[38] Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995 (2017)

[39] Zhu Y, Zhao M, Zhao Y, Li H, Zhang P. Noise reduction with low dose ct data based on a modified rof model. Opt. Express. 2012, 20(16): 17987-18004

Funding

Beijing Natural Science Foundation(Z210003)

National Natural Science Foundation of China(12071313)

key research project of the Academy for Multidisciplinary Studies, Capital Normal University

National Key Research and Development Program of China(2020YFA0712200)

Major Technologies R &D Program of Shenzhen(JSGGZD20220822095600001)

RIGHTS & PERMISSIONS

Shanghai University