Stochastic three-term conjugate gradient: a tensor correction algorithmic framework for nonconvex optimization in machine learning

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

In large-scale, nonconvex optimization for machine learning, such as the training of spine large models, the balance between efficient high-order geometric features of nonconvex landscapes identification and computational stability remains challenging. This paper proposes a tensor stochastic three-term conjugate gradient (TSTCG) algorithm to address two issues under a stochastic CG framework, the insufficient curvature awareness of the stochastic gradient descent and the heavy reliance on line searches. By introducing an implicit correction mechanism based on third-order tensor information, the algorithm fuses function values and gradient differences to reconstruct evolution vectors, effectively enhancing search precision in ill-conditioned curvature regions. Simultaneously, leveraging a specially designed three-term denominator structure allows the algorithm to process sufficient descent property, line search reliance reduction, and efficient fixed step-size updates. Finally, the nonconvexity proofs and numerical experiments demonstrate that TSTCG outperforms other mainstream stochastic optimization algorithms in terms of convergence efficiency and stability.

Keywords

machine learning / nonconvex optimization / spine large models / tensor / three-term conjugate gradient / sufficient descent

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] K. J. Bergen, P. A. Johnson, M. V. de Hoop, G. C. Beroza, Machine learning for data-driven discovery in solid Earth geoscience, Science, 363 (2019), eaau0323. https://doi.org/10.1126/science.aau0323

[2] S. Malley, C. Reina, S. Nacy, J. Gilles, B. Koohbor, G. Youssef, Predictability of mechanical behavior of additively manufactured particulate composites using machine learning and data-driven approaches, Comput. Ind., 142 (2022), 103739. https://doi.org/10.1016/j.compind.2022.103739

[3] D. V. Akman, M. Malekipirbazari, Z. D. Yenice, A. Yeo, N. Adhikari, Y. K. Wong, et al., K-best feature selection and ranking via stochastic approximation, Expert Syst. Appl., 213 (2023), 118864. https://doi.org/10.1016/j.eswa.2022.118864

[4] N. Khan, M. R. Saleem, D. Lee, M. W. Park, C. Park, Utilizing safety rule correlation for mobile scaffolds monitoring leveraging deep convolution neural networks, Comput. Ind., 129 (2021), 103448. https://doi.org/10.1016/j.compind.2021.103448

[5] Y. Ma, F. Rusu, K. Wu, A. Sim, Adaptive stochastic gradient descent for deep learning on heterogeneous CPU+GPU architectures, in 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), (2021), 6-15. https://doi.org/10.1109/IPDPSW50202.2021.00012

[6] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature, 521 (2015), 436-444. https://doi.org/10.1038/nature14539

[7] S. P. Awate, R. T. Whitaker, Unsupervised, information-theoretic, adaptive image filtering for image restoration, IEEE Trans. Pattern Anal. Mach. Intell., 28 (2006), 364-376. https://doi.org/10.1109/TPAMI.2006.52

[8] Y. Zhang, M. J. Wainwright, J. C. Duchi, Communication-efficient algorithms for statistical optimization, Adv. Neural Inf. Process. Syst., 25 (2012), 1502-1510.

[9] I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning , MIT Press, Cambridge, 2016.

[10] S. Klein, M. Staring, J. P. Pluim, Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines, IEEE Trans. Image Process., 16 (2007), 2879-2890. https://doi.org/10.1109/TIP.2007.909315

[11] Z. Allen-Zhu, Y. Yuan, Improved SVRG for non-strongly-convex or sum-of-nonconvex objectives, in International Conference on Machine Learning, (2016), 1080-1089.

[12] H. Robbins, S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), 400-407. https://doi.org/10.1214/aoms/1177729586

[13] L. Bottou, Large-scale machine learning with stochastic gradient descent, in Proceedings of COMPSTAT’2010 , (2010), 177-186. https://doi.org/10.1007/978-3-7908-2604-3_16

[14] R. Johnson, T. Zhang, Accelerating stochastic gradient descent using predictive variance reduction, Adv. Neural Inf. Process. Syst., 26 (2013), 315-323.

[15] A. Defazio, F. Bach, S. Lacoste-Julien, SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives, Adv. Neural Inf. Process. Syst., 27 (2014), 1646-1654.

[16] L. M. Nguyen, J. Liu, K. Scheinberg, M. Takáč, SARAH: A novel method for machine learning problems using stochastic recursive gradient, in International Conference on Machine Learning, (2017), 2613-2621.

[17] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res., 12 (2011), 2121-2159.

[18] G. Hinton, N. Srivastava, K. Swersky, Neural Networks for Machine Learning , 2012. Available from: https://www.cs.toronto.edu/tijmen/csc321/slides/lecture_slides_lec6.pdf.

[19] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, preprint, arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980

[20] S. J. Reddi, S. Kale, S. Kumar, On the convergence of Adam and beyond, preprint, arXiv:1904.09237. https://doi.org/10.48550/arXiv.1904.09237

[21] M. Yousefi, Á. Martínez Calomardo, A stochastic modified limited memory BFGS for training deep neural networks, in Science and Information Conference , (2022), 9-28. https://doi.org/10.1007/978-3-031-10464-0_2

[22] M. R. Hestenes, E. Stiefel, Methods of conjugate gradients for solving linear systems, J. Res. Natl. Bur. Stand., 49 (1952), 409-436. https://doi.org/10.6028/jres.049.044

[23] R. Fletcher, C. M. Reeves, Function minimization by conjugate gradients, Comput. J., 7 (1964), 149-154. https://doi.org/10.1093/comjnl/7.2.149

[24] E. Polak, G. Ribiere, Note sur la convergence de méthodes de directions conjuguées, Rev. Fr. Inf. Rech. Oper., 3 (1969), 35-43. https://doi.org/10.1051/m2an/196903R100351

[25] B. T. Polyak, The conjugate gradient method in extremal problems, USSR Comput. Math. Math. Phys., 9 (1969), 94-112. https://doi.org/10.1016/0041-5553(69)90035-4

[26] N. N. Schraudolph, T. Graepel, Combining conjugate direction methods with stochastic approximation of gradients, in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, (2003), 248-253.

[27] H. Jiang, P. Wilford, A stochastic conjugate gradient method for the approximation of functions, J. Comput. Appl. Math., 236 (2012), 2529-2544. https://doi.org/10.1016/j.cam.2011.12.012

[28] X. B. Jin, X. Y. Zhang, K. Huang, G. G. Geng, Stochastic conjugate gradient algorithm with variance reduction, IEEE Trans. Neural Netw. Learn. Syst., 30 (2018), 1360-1369. https://doi.org/10.1109/TNNLS.2018.2868841

[29] C. Kou, H. Yang, A mini-batch stochastic conjugate gradient algorithm with variance reduction, J. Global Optim., 87 (2023), 1009-1025. https://doi.org/10.1007/s10898-023-01306-0

[30] J. Z. Zhang, N. Y. Deng, L. Chen, New quasi-Newton equation and related methods for unconstrained optimization, J. Optim. Theory Appl., 102 (1999), 147-167. https://doi.org/10.1023/A:1021798319692

[31] A. Bouaricha, Tensor methods for large, sparse unconstrained optimization, SIAM J. Optim., 7 (1997), 732-756. https://doi.org/10.1137/S105262349427218X

[32] J. Zhang, C. Xu, Properties and numerical performance of quasi-Newton methods with modified quasi-Newton equations, J. Comput. Appl. Math., 137 (2001), 269-278. https://doi.org/10.1016/S0377-0427(00)00713-1

[33] F. Biglari, M. A. Hassan, W. J. Leong, New quasi-Newton methods via higher order tensor models, J. Comput. Appl. Math., 235 (2011), 2412-2422. https://doi.org/10.1016/j.cam.2010.10.040

[34] L. Zhang, W. Zhou, D. Li, Some descent three-term conjugate gradient methods and their global convergence, Optim. Methods Softw., 22 (2007), 697-711. https://doi.org/10.1080/10556780600812232

[35] K. Sugiki, Y. Narushima, H. Yabe, Globally convergent three-term conjugate gradient methods that use secant conditions and generate descent search directions for unconstrained optimization, J. Optim. Theory Appl., 153 (2012), 733-757. https://doi.org/10.1007/s10957-011-9964-1

[36] M. Kelly, R. Landreville, K. Nottingham, The UCI Machine Learning Repository , 2023. Available from: https://archive.ics.uci.edu.

[37] C. C. Chang, C. J. Lin, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol., 2 (2011), 1-27. https://doi.org/10.1145/1961189.1961199