Immunity-Based Orthogonal Weights Modification Algorithm

Abstract

Recently, the catastrophic forgetting problem of neural networks in the process of continual learning (CL) has attracted more and more attention with the development of deep learning. The orthogonal weight modification (OWM) algorithm to some extent overcomes the catastrophic forgetting problem in CL. It is well-known that the mapping rule learned by the network is usually not accurate in the early stage of neural network training. Our main idea is to establish an immune mechanism in CL, which rejects unreliable mapping rules at the beginning of the training until those are reliable enough. Our algorithm showed a very good competitive advantage in the permuted and disjoint MNIST tasks and disjoint CIFAR-10 tasks. As for the more challenging task of Chinese handwriting character recognition, our algorithm showed a notable improvement compared with the OWM algorithm. In view of the context-dependent processing (CDP) module in [37], we revealed that the module may result in a loss of information and we proposed a modified CDP module to overcome this weakness. The performance of the system with the modified CDP module outperforms the original one in the CelebFaces attributed recognition task. Besides continual multi-task, we also considered a single task, where the immunity-based OWM (IOWM) algorithm was designed as an optimization solver of neural networks for low-dose computed tomography (CT) denoising task.

Funding

Beijing Natural Science Foundation(Z210003)

National Natural Science Foundation of China(12071313)

key research project of the Academy for Multidisciplinary Studies, Capital Normal University

National Key Research and Development Program of China(2020YFA0712200)

Major Technologies R &D Program of Shenzhen(JSGGZD20220822095600001)