Enhancing Generalisation via Cascaded Inertia SGD With Learnt Hyperparameters

Yongji Guan , Shunbao Zhang , Longbang Wang , Erchao Li

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) : 1019 -1027.

PDF (1348KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) :1019 -1027. DOI: 10.1049/cit2.70111
ORIGINAL RESEARCH
research-article
Enhancing Generalisation via Cascaded Inertia SGD With Learnt Hyperparameters
Author information +
History +
PDF (1348KB)

Abstract

A central challenge in deep learning lies in achieving strong model generalisation, an area in which conventional optimisers such as stochastic gradient descent (SGD) often exhibit limitations, even though they ensure convergence. This paper introduces cascaded inertia SGD (CISGD), a novel optimisation algorithm specifically designed to address this challenge. The core mechanism of CISGD involves a hierarchical aggregation of historical gradients: it first accumulates past gradients and then compounds these accumulations to form multi-stage inertial terms. This deep integration of gradient history allows the optimiser produce more stable optimisation trajectories, thereby encouraging convergence towards flatter minima that are empirically associated with improved generalisation. To enhance practicality and robustness, we further eliminate manual hyperparameter tuning through a lightweight feed-forward neural network that automatically learns the optimal coefficients of the inertial terms, ensuring adaptability across diverse architectures and tasks. Extensive experiments on benchmark image classification and remote sensing change detection tasks demonstrate that CISGD consistently achieves superior generalisation performance compared with existing optimisers, demonstrating that CISGD achieves consistent, though moderate, improvements over standard optimisers while maintaining computational efficiency. This work offers a practical and theoretically grounded framework for automated and generalisable optimisation, promoting broader applications of adaptive deep learning in complex real-world tasks.

Keywords

deep learning / deep neural networks / optimisation

Cite this article

Download citation ▾
Yongji Guan, Shunbao Zhang, Longbang Wang, Erchao Li. Enhancing Generalisation via Cascaded Inertia SGD With Learnt Hyperparameters. CAAI Transactions on Intelligence Technology, 2026, 11 (4) : 1019-1027 DOI:10.1049/cit2.70111

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

I. Corley, C. Robinson, and A. Ortiz, “ A Change Detection Reality Check,” preprint, arXiv:2402.06994 (2024).

[2]

Y. Zhao, T. Celik, N. Liu, F. Gao, and H. C. Li, “ SSLChange: A Self-Supervised Change Detection Framework Based on Domain Adaptation,” IEEE Transactions on Geoscience and Remote Sensing 62 (2024): 1-14, https://doi.org/10.1109/tgrs.2024.3489615.

[3]

D. Yu and S. Ji, “ A New Spatial-Oriented Object Detection Framework for Remote Sensing Images,” IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-16, https://doi.org/10.1109/tgrs.2021.3127232.

[4]

X. Ma, Q. Wu, X. Zhao, X. Zhang, M. O. Pun, and B. Huang, “ SAM-Assisted Remote Sensing Imagery Semantic Segmentation With Object and Boundary Constraints,” IEEE Transactions on Geoscience and Remote Sensing 62 (2024): 1-16, https://doi.org/10.1109/tgrs.2024.3443420.

[5]

Y. Zhao, S. Li, C. H. Liu, Y. Han, H. Shi, and W. Li, “ Domain Adaptive Remote Sensing Scene Recognition via Semantic Relationship Knowledge Transfer,” IEEE Transactions on Geoscience and Remote Sensing 61 (2023): 1-13, https://doi.org/10.1109/tgrs.2023.3267149.

[6]

Z. Lv, P. Zhang, W. Sun, J. A. Benediktsson, J. Li, and W. Wang, “ Novel Adaptive Region Spectral-Spatial Features for Land Cover Classification With High Spatial Resolution Remotely Sensed Imagery,” IEEE Transactions on Geoscience and Remote Sensing 61 (2023): 1-12, https://doi.org/10.1109/tgrs.2023.3275753.

[7]

J. Yang, X. Li, I. Fatkhullin, and N. He, “ Two Sides of One Coin: The Limits of Untuned SGD and the Power of Adaptive Methods,” in Advances in Neural Information Processing Systems (2023), 74257-74288.

[8]

Z. Lin, S. Zhang, Y. Zhou, H. Wang, and S. Wang, “ Learning Rate Burst for Superior SGDM and AdamW Integration,” Journal of Intelligent and Fuzzy Systems 49, no. 2 (2024): 1-11, https://doi.org/10.3233/jifs-239157.

[9]

M. Liu, L. Chen, X. Du, L. Jin, and M. Shang, “ Activated Gradients for Deep Neural Networks,” IEEE Transactions on Neural Networks and Learning Systems 34, no. 4 (April 2023): 2156-2168, https://doi.org/10.1109/tnnls.2021.3106044.

[10]

Z. Chen, J. Zhang, Y. Kou, X. Chen, C. J. Hsieh, and Q. Gu, “ Why Does Sharpness-Aware Minimization Generalize Better than SGD?,” in Advances in Neural Information Processing Systems (2023), 72325-72376.

[11]

J. Fan, L. Jin, P. Li, J. Liu, Z. G. Wu, and W. Chen, “ Coevolutionary Neural Dynamics Considering Multiple Strategies for Nonconvex Optimization,” Tsinghua Science and Technology (2025), https://doi.org/10.26599/TST.2025.9010120.

[12]

H. Li, A. Rakhlin, and A. Jadbabaie, “ Convergence of Adam Under Relaxed Assumptions,” in Advances in Neural Information Processing Systems (2023), 52166-52196.

[13]

D. Friedman, A. Wettig, and D. Chen, “ Learning Transformer Programs,” in Advances in Neural Information Processing Systems (2023), 49044-49067.

[14]

B. Hoover, Y. Liang, B. Pham, et al., “ Energy Transformer,” in Advances in Neural Information Processing Systems (2023), 27532-27559.

[15]

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “ Roformer: Enhanced Transformer With Rotary Position Embedding,” Neurocomputing 568 (2024): 12706, https://doi.org/10.1016/j.neucom.2023.127063.

[16]

Y. Ye, M. Wang, L. Zhou, G. Lei, J. Fan, and Y. Qin, “ Adjacent-Level Feature Cross-Fusion With 3-D CNN for Remote Sensing Image Change Detection,” IEEE Transactions on Geoscience and Remote Sensing 61 (2023): 1-14, https://doi.org/10.1109/tgrs.2023.3305499.

[17]

C. Han, C. Wu, H. Guo, M. Hu, J. Li, and H. Chen, “ Change Guiding Network: Incorporating Change Prior to Guide Change Detection in Remote Sensing Imagery,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 16 (2023): 8395-8407, https://doi.org/10.1109/jstars.2023.3310208.

[18]

W. Yu, X. Zhang, R. Gloaguen, X. X. Zhu, and P. Ghamisi, “ MineNetCD: A Benchmark for Global Mining Change Detection on Remote Sensing Imagery,” IEEE Transactions on Geoscience and Remote Sensing 62 (2024): 1-16, https://doi.org/10.1109/tgrs.2024.3491715.

[19]

P. Zhou, J. Feng, C. Ma, C. Xiong, and S. C. H. Hoi, “ Towards Theoretically Understanding Why SGD Generalizes Better Than Adam in Deep Learning,” in Advances in Neural Information Processing Systems (2020), 21285-21296.

[20]

Z. Xie, X. Wang, H. Zhang, I. Sato, and M. Sugiyama, “ Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum,” in International Conference on Machine Learning, Vol. 162 (July 2022), 24430-24459.

[21]

X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan, “ Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence 46, no. 12 (December 2024): 9508-9520, https://doi.org/10.1109/tpami.2024.3423382.

[22]

L. Jin, Y. Zhang, and S. Li, “ Integration-Enhanced Zhang Neural Network for Real-Time-Varying Matrix Inversion in the Presence of Various Kinds of Noises,” IEEE Transactions on Neural Networks and Learning Systems 27, no. 12 (December 2016): 2615-2627, https://doi.org/10.1109/tnnls.2015.2497715.

[23]

L. Jin, J. Zhao, L. Chen, and S. Li, “ Collective Neural Dynamics for Sparse Motion Planning of Redundant Manipulators Without Hessian Matrix Inversion,” IEEE Transactions on Neural Networks and Learning Systems 36, no. 3 (2025): 4326-4335, https://doi.org/10.1109/tnnls.2024.3363241.

[24]

D. Su, P. S. Stanimirovic, L. B. Han, and L. Jin, “ Neural Dynamics for Improving Optimiser in Deep Learning With Noise Considered,” CAAI Transactions on Intelligence Technology 9, no. 3 (July 2024): 722-737, https://doi.org/10.1049/cit2.12263.

[25]

L. Dai, H. Xu, Y. Zhang, and B. Liao, “ Norm-Based Zeroing Neural Dynamics for Time-Variant Non-Linear Equations,” CAAI Transactions on Intelligence Technology 9, no. 6 (2024): 1561-1571, https://doi.org/10.1049/cit2.12360.

[26]

C. Qin, L. Chen, Z. Cai, M. Liu, and L. Jin, “ Long Short-Term Memory With Activation on Gradient,” Neural Networks 164, no. 4 (2023): 135-145, https://doi.org/10.1016/j.neunet.2023.04.026.

[27]

L. Chen, L. Jin, M. Shang, and F.-Y. Wang, “ Enhancing Representation Power of Deep Neural Networks With Negligible Parameter Growth for Industrial Applications,” IEEE Transactions on Systems, Man, and Cybernetics: Office Systems 54, no. 11 (2024): 6837-6848, https://doi.org/10.1109/tsmc.2024.3387408.

[28]

M. Liu, Y. Li, Y. Chen, Y. Qi, and L. Jin, “ A Distributed Competitive and Collaborative Coordination for Multirobot Systems,” IEEE Transactions on Mobile Computing 23, no. 12 (2024): 11436-11448, https://doi.org/10.1109/tmc.2024.3397242.

[29]

B. Liao, L. Han, X. Cao, S. Li, and J. Li, “ Double Integral-Enhanced Zeroing Neural Network With Linear Noise Rejection for Time-Varying Matrix Inverse,” CAAI Transactions on Intelligence Technology 9, no. 1 (2024): 197-210, https://doi.org/10.1049/cit2.12161.

[30]

L. Jin, L. Wei, and S. Li, “ Gradient-Based Differential Neural-Solution to Time-Dependent Nonlinear Optimization,” IEEE Transactions on Automatic Control 68, no. 1 (2023): 620-627, https://doi.org/10.1109/tac.2022.3144135.

[31]

E. Weinan, “ A Proposal on Machine Learning via Dynamical Systems,” Communications in Mathematics and Statistics 5, no. 1 (March 2017): 1-11, https://doi.org/10.1007/s40304-017-0103-z.

[32]

H. Huang, L. Jin, and Z. Zeng, “ A Momentum Recurrent Neural Network for Sparse Motion Planning of Redundant Manipulators With Majorization-Minimization,” IEEE Transactions on Industrial Electronics 72, no. 12 (2025): 1-10, https://doi.org/10.1109/tie.2025.3566731.

[33]

K. He, X. Zhang, S. Ren, and J. Sun, “ Deep Residual Learning for Image Recognition,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (2016), 770-778.

[34]

A. Krizhevsky and G. Hinton, “ Learning Multiple Layers of Features From Tiny Images,” Technical Reports Series 4 (2009), https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.

[35]

K. Simonyan and A. Zisserman, “ Very Deep Convolutional Networks for Large-Scale Image Recognition,” in Proceedings of International Conference on Learning Representations (2015), https://doi.org/10.48550/arXiv.1409.1556.

[36]

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “ Densely Connected Convolutional Networks,” in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017), 2261-2269.

[37]

X. Zhang, W. Yu, M. O. Pun, and W. Shi, “ Cross-Domain Landslide Mapping From Large-Scale Remote Sensing Images Using Prototype-Guided Domain-Aware Progressive Representation Learning,” ISPRS Journal of Photogrammetry and Remote Sensing 197 (2023): 1-17, https://doi.org/10.1016/j.isprsjprs.2023.01.018.

[38]

Z. Li, C. Tang, X. Liu, et al., “ Lightweight Remote Sensing Change Detection With Progressive Feature Aggregation and Supervised Attention,” IEEE Transactions on Geoscience and Remote Sensing 61 (2023): 1-12, https://doi.org/10.1109/tgrs.2023.3241436.

[39]

H. Chen, Z. Qi, and Z. Shi, “ Remote Sensing Image Change Detection With Transformers,” IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-14, https://doi.org/10.1109/tgrs.2021.3095166.

PDF (1348KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉