On Loss Functionals for Physics-Informed Neural Networks for Steady-State Convection-Dominated Convection-Diffusion Problems

Abstract

Solutions of convection-dominated convection-diffusion problems usually possess layers, which are regions where the solution has a steep gradient. It is well known that many classical numerical discretization techniques face difficulties when approximating the solution to these problems. In recent years, physics-informed neural networks (PINNs) for approximating the solution to (initial-)boundary value problems ((I)BVPs) received a lot of interest. This paper studies various loss functionals for PINNs that are especially designed for convection-dominated convection-diffusion problems and that are novel in the context of PINNs. They are numerically compared to the vanilla and an hp-variational loss functional from the literature based on two steady-state benchmark problems whose solutions possess different types of layers. We observe that the best novel loss functionals reduce the

L^2(\Omega )

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$L^2(\varOmega )$$\end{document}

error by 17.3% for the first and 5.5% for the second problem compared to the methods from the literature.

Keywords

Steady-state convection-diffusion problems / Convection-dominated regime / Physics-informed neural networks (PINNs) / Loss functionals / 65N99 / 68T07

References

[1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems. Software available from https://www.tensorflow.org/ (2015)

[2] Arzani A, Cassel KW, D’Souza RM. Theory-guided physics-informed neural networks for boundary layer problems with singular perturbation. J. Comput. Phys., 2023, 473: 111768

[3] Augustin M, Caiazzo A, Fiebach A, Fuhrmann J, John V, Linke A, Umla R. An assessment of discretizations for convection-dominated convection-diffusion equations. Comput. Methods Appl. Mech. Eng., 2011, 200(47/48): 3395-3409

[4] Barrenechea, G.R., John, V., Knobloch, P.: Finite element methods respecting the discrete maximum principle for convection-diffusion equations. SIAM Rev. 66, 3–88 (2024). https://doi.org/10.1137/22M1488934

[5] Beck A, Flad D, Munz C-D. Deep neural networks for data-driven LES closure models. J. Comput. Phys., 2019, 398: 108910

[6] Bengio Y. Montavon G, Orr GB, Müller K-R. Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks of the Trade: Second Edition. Lecture Notes in Computer Science, 2012, Berlin, Springer437-478 7700

[7] Cai S, Mao Z, Wang Z, Yin M, Karniadakis GE. Physics-informed neural networks (PINNs) for fluid mechanics: a review. Acta. Mech. Sin., 2021, 37(12): 1727-1738

[8] Cuomo S, Di Cola VS, Giampaolo F, Rozza G, Raissi M, Piccialli F. Scientific machine learning through physics-informed neural networks: where we are and what’s next. J. Sci. Comput., 2022, 92(3): 88

[9] Daw, A., Bu, J., Wang, S., Perdikaris, P., Karpatne, A.: Mitigating propagation failures in physics-informed neural networks using retain-resample-release (r3) sampling. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, vol. 202, pp. 7264–7302. JMLR.org, Honolulu, Hawaii, USA (2023)

[10] De Ryck T, Mishra S, Molinaro R. wPINNs: weak physics informed neural networks for approximating entropy solutions of hyperbolic conservation laws. SIAM J. Numer. Anal., 2024, 622811-841

[11] Dissanayake G, Phan-Thien N. Neural-network-based approximations for solving partial differential equations. Commun. Numer. Methods Eng., 1994, 10(3): 195-201

[12] Doumèche, N., Biau, G., Boyer, C.: Convergence and error analysis of PINNs (2023). https://doi.org/10.48550/arXiv.2305.01240

[13] Farrell P, Rotundo N, Doan DH, Kantner M, Fuhrmann J, Koprucki T. Piprek J. Drift-diffusion models. Handbook of Optoelectronic Device Modeling and Simulation: Lasers, Modulators, Photodetectors, Solar Cells, and Numerical Methods, 20171Boca Raton, CRC Press733771 2

[14] Frerichs, D., John, V.: On reducing spurious oscillations in discontinuous Galerkin (DG) methods for steady-state convection-diffusion equations. J. Comput. Appl. Math. 393, 113487 (2021). https://doi.org/10.1016/j.cam.2021.113487

[15] Frerichs-Mihov, D.: On slope limiting and deep learning techniques for the numerical solution to convection-dominated convection-diffusion problems. Ph.D. Thesis. Free University Berlin, Berlin (2023)

[16] Frerichs-Mihov, D., Henning, L., John, V.: Data and code from the paper “On loss functionals for physics-informed neural networks for convection-dominated convection-diffusion problems” (2023). https://doi.org/10.20347/wias.data.7. https://archive.wias-berlin.de/receive/wias_mods_00007477

[17] Frerichs-Mihov D, Henning L, John V. Using deep neural networks for detecting spurious oscillations in discontinuous Galerkin solutions of convection-dominated convection-diffusion equations. J. Sci. Comput., 2023, 97(2): 36

[18] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 9, pp. 249–256. PMLR, Chia Laguna Resort, Sardinia, Italy (2010)

[19] Gomes, A.T.A., Silva, L.M., Valentin, F.: Physics-aware neural networks for boundary layer linear problems (2022). https://doi.org/10.48550/arXiv.2208.12559

[20] Gomes ATA, Silva LM, Valentin F. Navaux P, Barrios HCJ, Osthoff C, Guerrero G. Improving boundary layer predictions using parametric physics-aware neural networks. High Performance Computing. Communications in Computer and Information Science, 2022, Porto Alegre, Springer90-102

[21] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). https://www.deeplearningbook.org/. Accessed 2023-07-19

[22] He, Q., Tartakovsky, A.M.: Physics-informed neural network method for forward and backward advection-dispersion equations. Water Resour. Res. 57(7), 2020–029479 (2021). https://doi.org/10.1029/2020WR029479

[23] Hemker, P.W.: A singularly perturbed model problem for numerical computation. J. Comput. Appl. Math. 76(1/2), 277–285 (1996). https://doi.org/10.1016/S0377-0427(96)00113-6

[24] Higham CF, Higham DJ. Deep learning: an introduction for applied mathematicians. SIAM Rev., 2019, 61(4): 860-891

[25] Hou, Q., Sun, Z., He, L., Karemat, A.: Orthogonal grid physics-informed neural networks: a neural network-based simulation tool for advection-diffusion-reaction problems. Phys. Fluids 34(7), 077108 (2022). https://doi.org/10.1063/5.0095536

[26] John, V., Knobloch, P.: On spurious oscillations at layers diminishing (SOLD) methods for convection-diffusion equations: part I—a review. Comput. Methods Appl. Mech. Eng. 196(17/18/19/20), 2197–2215 (2007). https://doi.org/10.1016/j.cma.2006.11.013

[27] John V, Knobloch P. Cangiani A, Davidchack RL, Georgoulis E, Gorban AN, Levesley J, Tretyakov MV. Adaptive computation of parameters in stabilized methods for convection-diffusion problems. Numerical Mathematics and Advanced Applications 2011—Proceedings of ENUMATH 2011, 2013, Berlin, Springer275283 1

[28] John, V., Knobloch, P., Novo, J.: Finite elements for scalar convection-dominated equations and incompressible flow problems: a never ending story? Comput. Vis. Sci. 19(5/6), 47–63 (2018). https://doi.org/10.1007/s00791-018-0290-5

[29] John, V., Knobloch, P., Savescu, S.B.: A posteriori optimization of parameters in stabilized methods for convection-diffusion problems—part I. Comput. Methods Appl. Mech. Eng. 200(41/42/43/44), 2916–2929 (2011). https://doi.org/10.1016/j.cma.2011.04.016

[30] John, V., Knobloch, P., Wilbrandt, U.: A posteriori optimization of parameters in stabilized methods for convection-diffusion problems—part II. J. Comput. Appl. Math. 428, 115167–17 (2023). https://doi.org/10.1016/j.cam.2023.115167

[31] John, V., Maubach, J.M., Tobiska, L.: Nonconforming streamline-diffusion-finite-element-methods for convection-diffusion problems. Numer. Math. 78(2), 165–188 (1997). https://doi.org/10.1007/s002110050309

[32] Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nat. Rev. Phys., 2021, 3(6): 422-440

[33] Kharazmi, E., Zhang, Z., Karniadakis, G.E.: VPINNs: variational physics-informed neural networks for solving partial differential equations (2019). https://doi.org/10.48550/arXiv.1912.00873

[34] Kharazmi E, Zhang Z, Karniadakis GEM. hp\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$hp$$\end{document}-VPINNs: variational physics-informed neural networks with domain decomposition. Comput. Methods Appl. Mech. Eng., 2021, 374: 113547

[35] Khodayi-Mehr, R., Zavlanos, M.: VarNet: variational neural networks for the solution of partial differential equations. In: Proceedings of the 2nd Conference on Learning for Dynamics and Control, pp. 298–307. PMLR, Virtual, Online (2020)

[36] Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: ICLR 2015, p. 13. San Diego, California, USA (2014). https://doi.org/10.48550/ARXIV.1412.6980

[37] Knobloch P, Lukáš P, Solin P. On error indicators for optimizing parameters in stabilized methods. Adv. Comput. Math., 2019, 45(4): 1853-1862

[38] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., Mahoney, M.W.: Characterizing possible failure modes in physics-informed neural networks. In: Advances in Neural Information Processing Systems, vol. 34, pp. 26548–26560. Curran Associates, Inc., Virtual, Online (2021)

[39] Laghi, L., Schiassi, E., De Florio, M., Furfaro, R., Mostacci, D.: Physics-informed neural networks for 1-D steady-state diffusion-advection-reaction equations. Nucl. Sci. Eng. 197(9), 1–31 (2023). https://doi.org/10.1080/00295639.2022.2160604

[40] Lu L, Pestourie R, Yao W, Wang Z, Verdugo F, Johnson SG. Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput., 2021, 43(6): 1105-1132

[41] Margenberg N, Lessig C, Richter T. Structure preservation for the deep neural network multigrid solver. ETNA Electron. Trans. Numer. Anal., 2022, 56: 86-101

[42] Mishra S, Molinaro R. Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs. IMA J. Numer. Anal., 2022, 42(2): 981-1022

[43] Mishra S, Molinaro R. Estimates on the generalization error of physics-informed neural networks for approximating PDEs. IMA J. Numer. Anal., 2023, 43(1): 1-43

[44] Misra, D.: Mish: a self regularized non-monotonic activation function. In: British Machine Vision Conference (2020). https://api.semanticscholar.org/CorpusID:221113156

[45] Mojgani, R., Balajewicz, M., Hassanzadeh, P.: Lagrangian PINNs: a causality-conforming solution to failure modes of physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 404, 115810 (2023). https://doi.org/10.1016/j.cma.2022.115810. arXiv:2205.02902

[46] Pinkus A. Approximation theory of the MLP model in neural networks. Acta Numer., 1999, 8: 143-195

[47] Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 2019, 378: 686-707

[48] Ray D, Hesthaven JS. Detecting troubled-cells on two-dimensional unstructured grids using a neural network. J. Comput. Phys., 2019, 397: 108845

[49] Roos, H.-G., Stynes, M., Tobiska, L.: Robust Numerical Methods for Singularly Perturbed Differential Equations: Convection-Diffusion-Reaction and Flow Problems. Springer Series in Computational Mathematics, vol. 24, 2nd edn. Springer, Berlin (2008). https://doi.org/10.1007/978-3-540-34467-4

[50] Saadat, M.H., Gjorgiev, B., Das, L., Sansavini, G.: Neural tangent kernel analysis or PINN for advection-diffusion equation (2022). https://doi.org/10.48550/arXiv.2211.11716

[51] TensorFlow Developers: TensorFlow (v2.13.0). Zenodo (2023). https://doi.org/10.5281/zenodo.8117732

[52] von Wahl H, Richter T. Using a deep neural network to predict the motion of underresolved triangular rigid bodies in an incompressible flow. Int. J. Numer. Methods Fluids, 2021, 93(12): 3364-3383

[53] Wang, Y., Xu, C., Yang, M., Zhang, J.: Less emphasis on hard regions: curriculum learning of PINNs for singularly perturbed convection-diffusion-reaction problems. East Asian J. Appl. Math. 14(1), 104–123 (2024). https://doi.org/10.4208/eajam.2023-062.170523

[54] Wolff, T., Carrillo, H., Martí, L., Sanchez-Pi, N.: Towards optimally weighted physics-informed neural networks in ocean modelling (2021). https://doi.org/10.48550/arXiv.2106.08747

[55] Yang L, Shami A. On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing, 2020, 415: 295-316

[56] Zang Y, Bao G, Ye X, Zhou H. Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys., 2020, 411: 109409

[57] Zong Y, He Q, Tartakovsky AM. Improved training of physics-informed neural networks for parabolic differential equations with sharply perturbed initial conditions. Comput. Methods Appl. Mech. Eng., 2023, 414: 116125

Funding

Weierstraß-Institut für Angewandte Analysis und Stochastik, Leibniz-Institut im Forschungsverbund Berlin e.V. (3510)

RIGHTS & PERMISSIONS

The Author(s)