Vertical Deformation Mapping: Steering Optimiser Toward Flat Minima

Abstract

Standard deep learning optimisation is typically conducted on shape-fixed loss surfaces. However, shape-fixed loss surfaces may impede optimisers from reaching flat regions closely associated with strong generalisation. In this work, we propose a new paradigm named deformation mapping to deform the loss surface during optimisation. Moreover, we design various vertical deformation mappings (VDMs) and further analyse their contributions to the training process. Theoretically, we prove that deforming the loss surface enhances the optimiser's ability to filter out sharp minima in deterministic settings. Furthermore, by incorporating diffusion theory, we demonstrate that VDM exponentially reduces the escape time from sharp minima under stochastic noise and momentum. Empirically, visualisations of loss landscapes demonstrate that VDMs locate significantly flatter minima compared to standard optimisation. Furthermore, integrating VDMs into the training of various deep neural networks produces consistent accuracy gains on ImageNet, CIFAR-10, and CIFAR-100, with negligible additional computation. Notably, PreResNet-20 on CIFAR-100 achieves a 1.46% increase in top-1 accuracy. These results indicate that the deformation mapping is a promising paradigm for improving optimisation and generalisation in deep learning. The code is available at https://anonymous.4open.science/r/Vertical-Deformation-Mapping-2324.

Keywords

deep learning / flatness / generalisation / loss landscape / optimisation

Funding

This work was supported in part by the Scientific Research Project of Hunan Provincial Education Department under (Grant 25A0397), in part by the National Natural Science Foundation of China under (Grant 62506148), in part by the Fundamental Research Funds for the Central Universities under (Grant lzujbky-2025-pd05), in part by the Postdoctoral Fellowship Programme of China Postdoctoral Science Foundation under (Grant GZC20251039), and in part by the Natural Science Foundation of Gansu Province under (Grant 26JRRA247).

Conflicts of Interest

Long Jin is an Associate Editor for the journal, and was not involved in peer review process or the decision to publish this article. The authors declare that they have no conflict of interest.

Data Availability Statement

Data available on request from the authors.

References

[1] C. Tan, J. Zhang, J. Liu, and Y. Gong, “Sharpness-Aware Lookahead for Accelerating Convergence and Improving Generalization,” IEEE Transactions on Pattern Analysis and Machine Intelligence 46, no. 12 (December 2024): 10375-10388, https://doi.org/10.1109/tpami.2024.3444002.

[2] S. Chen, J. Liu, P. Wang, C. Xu, S. Cai, and J. Chu, “Accelerated Optimization in Deep Learning With a Proportional-Integral-Derivative Controller,” Nature Communications 15, no. 1 (2024): 10263, https://doi.org/10.1038/s41467-024-54451-3.

[3] A. Zharmagambetov, B. Amos, A. Ferber, T. Huang, B. Dilkina, and Y. Tian, “Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information,” in Advances in Neural Information Processing Systems (Neurips) (2023), 27332-27350.

[4] B. M. Le and S. S. Woo, “Gradient Alignment for Cross-Domain Face Anti-Spoofing,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024), 188-199.

[5] T. Wu, T. Luo, and D. C. Wunsch II, “CR-SAM: Curvature Regularized Sharpness-Aware Minimization,” AAAI Conference on Artificial Intelligence (AAAI) Vol. 38 (2024), 6144-6152.

[6] J. Yang, T. Sadiq, J. Xiong, et al., A Novel Myocarditis Detection Combining Deep Reinforcement Learning and an Improved Differential Evolution Algorithm,” CAAI Transactions on Intelligence Technology 9, no. 6 (2024): 1347-1360, https://doi.org/10.1049/cit2.12289.

[7] H. Yan and Y. Guo, “Local and Global Flatness for Federated Domain Generalization,” in European Conference on Computer Vision (ECCV) (2024), 71-87.

[8] K. Wen, Z. Li, and T. Ma, “Sharpness Minimization Algorithms Do Not Only Minimize Sharpness to Achieve Better Generalization,” in Advances in Neural Information Processing Systems (NeurIPS) (2023), 1024-1035.

[9] Z. Chen, J. Zhang, Y. Kou, X. Chen, C.-J. Hsieh, and Q. Gu, “Why Does Sharpness-Aware Minimization Generalize Better Than SGD?,” in Advances in Neural Information Processing Systems (NeurIPS) (2023), 72325-72376.

[10] T. Li, P. Zhou, Z. He, X. Cheng, and X. Huang, “Friendly Sharpness-Aware Minimization,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024), 5631-5640.

[11] M. Haddouche, P. Viallard, U. Simsekli, and B. Guedj, “A PAC-Bayesian Link Between Generalisation and Flat Minima,” in Proceedings of the 36th International Conference on Algorithmic Learning Theory, Proceedings of Machine Learning Research (PMLR, 2025), Vol. 272, 481-511.

[12] Y. Dai, K. Ahn, and S. Sra, “The Crucial Role of Normalization in Sharpness-Aware Minimization,” in Advances in Neural Information Processing Systems (NeurIPS), (2023), 67741-67770.

[13] D. Si and C. Yun, “Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima,” in Advances in Neural Information Processing Systems (NeurIPS) (2023), 26190-26228.

[14] J. Wang, S. Wang, and Y. Zhang, “Deep Learning on Medical Image Analysis,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 1-35, https://doi.org/10.1049/cit2.12356.

[15] Z.Xiangfei, Z.Qingchen, and J.Liming , “ Layer-Level Adaptive Gradient Perturbation Protecting Deep Learning Based on Differential Privacy,” CAAI Transactions on Intelligence Technology 10, no. 3 (2025): 929-944, https://doi.org/10.1049/cit2.70008.

[16] J. Pennington and Y. Bahri, “Geometry of Neural Network Loss Surfaces via Random Matrix Theory,” International Conference on Machine Learning (ICML) Vol. 70 (2017), 2798-2806.

[17] J. Shi, Y. Chen, A. A. Heidari, Z. Cai, H. Chen, and G. Liang, “Topological Search and Gradient Descent Boosted Runge-Kutta Optimiser With Application to Engineering Design and Feature Selection,” CAAI Transactions on Intelligence Technology 10, no. 2 (2025): 557-614, https://doi.org/10.1049/cit2.12387.

[18] O. Russakovsky, J. Deng, H. Su, et al., ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision 115, no. 3 (2015): 211-252, https://doi.org/10.1007/s11263-015-0816-y.

[19] A. Krizhevsky , Learning Multiple Layers of Features from Tiny Images. Master’s thesis (University of Toronto, 2009).

[20] S. Hochreiter and J. Schmidhuber, “Flat Minima,” Neural Computation 9, no. 1 (1997): 1-42, https://doi.org/10.1162/neco.1997.9.1.1.

[21] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the Loss Landscape of Neural Nets,” in Advances in Neural Information Processing Systems (NeurIPS) (2018), 6389-6399.

[22] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima,” in International Conference on Learning Representations (ICLR), 2017): [Online], https://openreview.net/forum?id=H1oyRlYgg.

[23] L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio, “Sharp Minima Can Generalize for Deep Nets,” International Conference on Machine Learning (ICML) Vol. 70 (2017), 1019-1028.

[24] H. Wang, N. S. Keskar, C. Xiong, and R. Socher, “Identifying Generalization Properties in Neural Networks,” preprint arXiv:1809.07402 (2018).

[25] B. Liao, L. Han, X. Cao, S. Li, and J. Li, “Double Integral-Enhanced Zeroing Neural Network With Linear Noise Rejection for Time-Varying Matrix Inverse,” CAAI Transactions on Intelligence Technology 9, no. 1 (2024): 197-210, https://doi.org/10.1049/cit2.12161.

[26] W. R. Huang, Z. Emam, M. Goldblum, et al., Understanding Generalization Through Visualizations,” preprint arXiv:1906.03291 (2019).

[27] W. Wen, W. Yandan, Y. Feng, et al., Smoothout: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning,” preprint arXiv:1805.07898 (2018).

[28] T. Han, L. Adilova, H. Petzka, J. Kleesiek, and M. Kamp, “Flatness Is Necessary, Neural Collapse Is Not: Rethinking Generalization via Grokking,” in Advances in Neural Information Processing Systems (NeurIPS), (2025): [Online], https://openreview.net/forum?id=lbtOctHDQ3.

[29] K. Wen, T. Ma, and Z. Li, “How sharpness-aware Minimization Minimizes Sharpness?,” in International Conference on Learning Representations (ICLR), (2023): [Online], https://openreview.net/forum?id=5spDgWmpY6x.

[30] B. Tahmasebi, A. Soleymani, D. Bahri, S. Jegelka, and P. Jaillet, “A Universal Class of Sharpness-Aware Minimization Algorithms,” in International Conference on Machine Learning (ICML), (2024), 47418-47440.

[31] X. Zhang, R. Xu, H. Yu, H. Zou, and P. Cui, “Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2023), 20247-20257.

[32] J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar, “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,” in International Conference on Learning Representations (2021): [Online], https://openreview.net/forum?id=jh-rTtvkGeM.

[33] J. Gilmer, B. Ghorbani, A. Garg, et al. , “A Loss Curvature Perspective on Training Instabilities of Deep Learning Models,” in International Conference on Learning Representations (ICLR), (2022): [Online], https://openreview.net/forum?id=OcKMT-36vUs.

[34] P. Chaudhari, A. Choromanska, S. Soatto, et al., Entropy-SGD: Biasing Gradient Descent into Wide Valleys,” in International Conference on Learning Representations (ICLR), (2017): [Online], https://openreview.net/forum?id=B1YfAfcgl.

[35] S. Gao, C. Wang, C. Gao, et al., Improving Long-Tail Classification via Decoupling and Regularisation,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 62-71, https://doi.org/10.1049/cit2.12374.

[36] C. Robertson, N. A. Tong, T. T. Nguyen, Q. V. Hung Nguyen, and J. Jo, “Resource-Adaptive and OOD-Robust Inference of Deep Neural Networks on IoT Devices,” CAAI Transactions on Intelligence Technology 10, no. 1 (2025): 115-133, https://doi.org/10.1049/cit2.12384.

[37] H. He, G. Huang, and Y. Yuan, “Asymmetric Valleys: Beyond Sharp and Flat Local Minima,” in Advances in Neural Information Processing Systems (NeurIPS) (2019), 2549-2560.

[38] Z. Xie, X. Wang, H. Zhang, I. Sato, and M. Sugiyama, “Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum,” in International Conference on Machine Learning (ICML) (PMLR, 2022), 24430-24459.

[39] M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in International Conference on Machine Learning (ICML) (2019), 6105-6114.

[40] K. He, X. Zhang, S. Ren, and J. Sun, “Identity Mappings in Deep Residual Networks,” in European Conference on Computer Vision (2016), 630-645.

[41] K. He, X. Zhang, S. Ren, and S. Jian, “Deep Residual Learning for Image Recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016), 770-778.

[42] J. Hu, L. Shen, and G. Sun, “Squeeze-and-Excitation Networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018), 7132-7141.