Convergence rate of Riccati-based discretization for linear quadratic optimal control problem of stochastic mixed systems

Abstract

This paper presents a closed-loop numerical algorithm for the quadratic optimal control problem of a linear mixed system that combines both deterministic and stochastic controls. The core idea is to numerically solve two Riccati equations by using linear quadratic theory. Based on these numerical solutions, a feedback-type discretization method for the original problem is developed, along with an analysis of its convergence rate. A significant advantage of the proposed method is that it avoids the computation of backward stochastic differential equations associated with Pontryagin’s maximum principle, leading to notable improvements in computational efficiency. This work builds upon the theoretical framework established by Hu and Tang (Probab. Uncertain. Quant. Risk, 4 (2019), Paper No. 1) and focuses on its numerical implementation.

Keywords

Convergence rate / Closed-loop control strategy / Riccati equation / Mixed deterministic and stochastic controls

References

[1] Ait Rami, M., Chen, X. and Zhou, X. Y., Discrete-time indefinite LQ control with state and control dependent noises, Journal of Global Optimization, 2002, 23: 245-265

[2] Archibald, R., Bao, F. and Yong, J., A stochastic gradient descent approach for stochastic optimal control, East Asian Journal of Applied Mathematics, 2020, 10: 635-658

[3] Benner, P., Stillfjord, T. and Trautwein, C., A linear implicit Euler method for the finite element discretization of a controlled stochastic heat equation, IMA Journal of Numerical Analysis, 2022, 42: 2118-2150

[4] Du, N., Shi, J. and Liu, W., An effective gradient projection method for stochastic optimal control, International Journal of Numerical Analysis and Modeling, 2013, 10: 757-774

[5] Dunst, T. and Prohl, A., The forward-backward stochastic heat equation: Numerical analysis and simulation, SIAM Journal on Scientific Computing, 2016, 38: A2725-A2755

[6] Elliott, R., Li, X. and Ni, Y.-H., Discrete time mean-field stochastic linear-quadratic optimal control problems, Automatica, 2013, 49: 3222-3233

[7] Gong, B., Liu, W., Tang, T., Zhao, W. and Zhou, T., An efficient gradient projection method for stochastic optimal control problems, SIAM Journal on Numerical Analysis, 2017, 55: 2982-3005

[8] Hinze, M., Pinnau, R., Ulbrich, M. and Ulbrich, S., Optimization with PDE constraints, Springer, New York, 2009.

[9] Hu, Y. and Tang, S., Mixed deterministic and random optimal control of linear stochastic systems with quadratic costs, Probability, Uncertainty and Quantitative Risk, 2019, 4: 1-15

[10] Kröller, M. and Kunisch, K., Convergence rates for the feedback operators arising in the linear quadratic regulator problem governed by parabolic equations, SIAM Journal on Numerical Analysis, 1991, 28: 1350-1385

[11] Li, B. and Zhou, Q., Discretization of a distributed optimal control problem with a stochastic parabolic equation driven by multiplicative noise, Journal of Scientific Computing, 2021, 87: 68

[12] Liang, X. and Xu, J., Control for networked control systems with remote and local controllers over unreliable communication channel, Automatica, 2018, 98: 86-94

[13] Prohl, A. and Wang, Y., Strong rates of convergence for a space-time discretization of the backward stochastic heat equation, and of a linear-quadratic control problem for the stochastic heat equation, ESAIM: Control, Optimisation and Calculus of Variations, 2021, 27: 54

[14] Prohl, A. and Wang, Y., Strong error estimates for a space-time discretization of the linear-quadratic control problem with the stochastic heat equation with linear noise, IMA Journal of Numerical Analysis, 2022, 42: 3386-3429

[15] Prohl, A. and Wang, Y., Convergence with rates for a Riccati-based discretization of SLQ problems with SPDEs, IMA Journal of Numerical Analysis, 2024, 44: 3393-3434

[16] Prohl, A. and Wang, Y., Numerical methods for optimal control problems with SPDEs, arXiv: 2411.11239, 2024.

[17] Sun, Y., Li, H. and Zhang, H., Decentralized LQ control and stabilization for Itô systems, Automatica, 2024, 167: 111739

[18] Wang, Y., Stochastic linear quadratic optimal control problem: From discrete to continuous time, Chinese Journal of Contemporary Mathematics, 2018, 39: 387-406

[19] Wang, Y., Error analysis of a discretization for stochastic linear quadratic control problems governed by SDEs, IMA Journal of Mathematical Control and Information, 2021, 38: 1148-1173

[20] Wang, Y., Convergence rates of a discrete feedback control arising in mean-field linear quadratic optimal control problems, Science China Mathematics, 2023, 53: 1145-1162 (in Chinese)

[21] Wang, Y., Error analysis of the feedback controls arising in the stochastic linear quadratic control problems, Journal of Systems Science and Complexity, 2023, 36: 1540-1559

[22] Wang, Y., Strong error estimates for the space-time discretization of a stochastic linear quadratic control problem with control in the diffusion, SIAM Journal on Control and Optimization, 2025, 63: 3328-3355

[23] Yong, J., Linear-quadratic optimal control problems for mean-field stochastic differential equations, SIAM Journal on Control and Optimization, 2013, 51: 2809-2838

[24] Yong, J. and Zhou, X. Y., Stochastic controls: Hamiltonian systems and HJB equations, Springer, New York, 1999.