Global pre-fixing, local adjusting: a simple yet effective contrastive strategy for continual learning

Abstract

Continual learning (CL) involves acquiring and accumulating knowledge from evolving tasks while alleviating catastrophic forgetting. Recently, leveraging contrastive loss to construct more transferable and less forgetful representations has been a promising direction in CL. Despite advancements, their performance is still limited due to the confusion arising from both inter-task and intra-task features. To address the problem, we propose a simple yet effective contrastive strategy named Global Pre-fixing, Local Adjusting for Supervised Contrastive learning (GPLASC). Specifically, to avoid task-level confusion, we divide the entire unit hypersphere of representations into non-overlapping regions, with the centers of the regions forming an inter-task pre-fixed Equiangular Tight Frame (ETF). Meanwhile, for individual tasks, our method helps regulate the feature structure and form intra-task adjustable ETFs within their respective allocated regions. As a result, our method simultaneously ensures discriminative feature structures both between and within tasks and can be seamlessly integrated into any existing contrastive continual learning framework. Extensive experiments validate its effectiveness.

Graphical abstract

Keywords

continual learning / contrastive learning / representation learning

References

[1] McCloskey M, Cohen N J . Catastrophic interference in connectionist networks: the sequential learning problem. Psychology of Learning and Motivation, 1989, 24: 109–165

[2] Cha H, Lee J, Shin J. Co²L: contrastive continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, 9496−9505

[3] Li J, Azizov D, Li Y, Liang S. Contrastive continual learning with importance sampling and prototype-instance relation distillation. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 13554−13562

[4] Wen Y, Tan Z, Zheng K, Xie C, Huang W. Provable contrastive continual learning. In: Proceedings of the 41st International Conference on Machine Learning. 2024

[5] Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Liu C, Krishnan D. Supervised contrastive learning. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1567

[6] Graf F, Hofer C, Niethammer M, Kwitt R. Dissecting supervised contrastive learning. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 3821−3830

[7] Ali H T, Michieli U, Moon J J, Kim D, Ozay M. Deep neural network models trained with a fixed random classifier transfer better across domains. In: Proceedings of ICASSP 2024−2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024, 5305−5309

[8] Papyan V, Han X Y, Donoho D L . Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, 2020, 117( 40): 24652–24663

[9] Saha G, Garg I, Roy K. Gradient projection memory for continual learning. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[10] Shi W, Ye M. Prospective representation learning for non-exemplar class-incremental learning. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2025, 30

[11] Liang Y S, Li W J. Loss decoupling for task-agnostic continual learning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 492

[12] Yu L, Twardowski B, Liu X, Herranz L, Wang K, Cheng Y, Jui S, van de Weijer J. Semantic drift compensation for class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 6980−6989

[13] Fickus M, Jasper J, King E J, Mixon D G . Equiangular tight frames that contain regular simplices. Linear Algebra and Its Applications, 2018, 555: 98–138

[14] Soutif-Cormerais A, Masana M, van de Weijer J, Twardowski B. On the importance of cross-task features for class-incremental learning. 2021, arXiv preprint arXiv: 2106.11930

[15] Nori M K, Kim I M. Task confusion and catastrophic forgetting in class-incremental learning: a mathematical framework for discriminative and generative modelings. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024, 1510

[16] Kim G, Xiao C, Konishi T, Ke Z, Liu B. A theoretical study on solving continual learning. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 366

[17] Bhat P S, Zonooz B, Arani E. Task-aware information routing from common representation space in lifelong learning. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[18] Sarfraz F, Arani E, Zonooz B. Error sensitivity modulation based experience replay: mitigating abrupt representation drift in continual learning. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[19] Tiwari R, Killamsetty K, Iyer R, Shenoy P. GCR: gradient coreset based replay buffer selection for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 99−108

[20] Krizhevsky A. Learning multiple layers of features from tiny images. Toronto: University of Toronto, 2009

[21] Riemer M, Cases I, Ajemian R, Liu M, Rish I, Tu Y, Tesauro G. Learning to learn without forgetting by maximizing transfer and minimizing interference. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[22] Chaudhry A, Ranzato M A, Rohrbach M, Elhoseiny M. Efficient lifelong learning with A-GEM. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[23] Rebuffi S A, Kolesnikov A, Sperl G, Lampert C H. iCaRL: Incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5533−5542

[24] Benjamin A S, Rolnick D, Kording K P. Measuring and regularizing networks in function space. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[25] Buzzega P, Boschini M, Porrello A, Abati D, Calderara S. Dark experience for general continual learning: a strong, simple baseline. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1335

[26] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., & Perona, P. Caltech-UCSD birds 200. 2010

[27] Hendrycks D, Basart S, Mu N, Kadavath S, Wang F, Dorundo E, Desai R, Zhu T, Parajuli S, Guo M, Song D, Steinhardt J, Gilmer J. The many faces of robustness: a critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, 8320−8329

[28] Hendrycks D, Zhao K, Basart S, Steinhardt J, Song D. Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 15257−15266

RIGHTS & PERMISSIONS

Higher Education Press