Meta-Auto-Decoder: a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations

Abstract

Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i.e., PDEs with different physical parameters, boundary conditions, shapes of computational domains, etc. Typical reduced order modeling techniques accelerate the solution of the parametric PDEs by projecting them onto a linear trial manifold constructed in the offline stage. These methods often need a predefined mesh as well as a series of precomputed solution snapshots, and may struggle to balance between the efficiency and accuracy due to the limitation of the linear ansatz. Utilizing the nonlinear representation of neural networks (NNs), we propose the Meta-Auto-Decoder (MAD) to construct a nonlinear trial manifold, whose best possible performance is measured theoretically by the decoder width. Based on the meta-learning concept, the trial manifold can be learned in a mesh-free and unsupervised way during the pre-training stage. Fast adaptation to new (possibly heterogeneous) PDE parameters is enabled by searching on this trial manifold, and optionally fine-tuning the trial manifold at the same time. Extensive numerical experiments show that the MAD method exhibits a faster convergence speed without losing the accuracy than other deep learning-based methods.

Keywords

Parametric partial differential equations (PDEs) / Meta-learning / Reduced order modeling / Neural networks (NNs) / Auto-decoder

References

[1.] Antoniou, A., Edwards, H., Storkey, A.: How to train your MAML. In: Seventh International Conference on Learning Representations (2019)

[2.] Baydin AG, Pearlmutter BA, Radul AA, Siskind JM. Automatic differentiation in machine learning: a survey. J. Mach. Learn. Res., 2018, 18: 1-43

[3.] Benner P, Ohlberger M, Cohen A, Willcox K. . Model Reduction and Approximation, 2017 Philadelphia, PA Society for Industrial and Applied Mathematics,

[4.] Bhattacharya, K., Hosseini, B., Kovachki, N.B., Stuart, A.M.: Model reduction and neural networks for parametric PDEs. arXiv:2005.03180 (2020)

[5.] Cai S, Wang Z, Wang S, Perdikaris P, Karniadakis GE. Physics-informed neural networks for heat transfer problems. J. Heat Transfer, 2021, 143(6),

[6.] Chan, E.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-GAN: periodic implicit generative adversarial networks for 3D-aware image synthesis. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021). https://doi.org/10.1109/cvpr46437.2021.00574

[7.] Chen Y, Dong B, Xu J. Meta-MgNet: meta multigrid networks for solving parameterized partial differential equations. J. Comput. Phys., 2022, 455,

[8.] Cohen A, DeVore R. Approximation of high-dimensional parametric PDEs. Acta Numerica, 2015, 24: 1-159,

[9.] Cohen A, DeVore R, Petrova G, Wojtaszczyk P. Optimal stable nonlinear approximation. Foundat. Comput. Math., 2022, 22: 607-648,

[10.] Cohen A, DeVore R, Schwab C. Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s. Anal. Appl., 2011, 9(1): 11-47,

[11.] DeVore RA, Howard R, Micchelli CA. Optimal nonlinear approximation. Manuscripta Mathematica, 1989, 63: 469-478,

[12.] Dupont, E., Kim, H., Eslami, S.M.A., Rezende, D.J., Rosenbaum, D.: From data to functa: your data point is a function and you can treat it like one. In: Proceedings of the 39th International Conference on Machine Learning (2022)

[13.] E, W., Yu, B.: The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6(1), 1–12 (2018)

[14.] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, 2017 PMLR 1126-1135

[15.] Franco NR, Manzoni A, Zunino P. A deep learning approach to reduced order modelling of parameter dependent partial differential equations. Math. Comput., 2023, 92(340): 483-524,

[16.] Fresca S, Dede’ L, Manzoni A. A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs. J. Sci. Comput., 2021, 87(2): 61,

[17.] Fresca S, Gobat G, Fedeli P, Frangi A, Manzoni A. Deep learning-based reduced order models for the real-time simulation of the nonlinear dynamics of microstructures. Int. J. Numer. Methods Eng., 2022, 123(20): 4749-4777,

[18.] Gedney SD. Introduction to the finite-difference time-domain (FDTD) method for electromagnetics. Syn. Lectures Comput. Electromag., 2011, 6(1): 1-250,

[19.] Greif C, Urban K. Decay of the Kolmogorov N-width for wave problems. Appl. Math. Lett., 2019, 96: 216-222,

[20.] Huang, X., Liu, H., Shi, B., Wang, Z., Yang, K., Li, Y., Weng, B., Wang, M., Chu, H., Zhou, J., Fan, Y., Hua, B., Chen, L., Dong, B.: Solving partial differential equations with point source based on physics-informed neural networks. arXiv:2111.01394 (2021)

[21.] Huang X, Ye Z, Liu H, Shi B, Wang Z, Yang K, Li Y, Weng B, Wang M, Chu H, Zhou J, Fan Y, Hua B, Chen L, Dong B. Meta-auto-decoder for solving parametric partial differential equations. Adv. Neural Infor. Process. Syst., 2022, 35: 23426-23438

[22.] Jiang, C.M., Esmaeilzadeh, S., Azizzadenesheli, K., Kashinath, K., Mustafa, M., Tchelepi, H.A., Marcus, P., Prabhat, M., Anandkumar, A.: MeshfreeFlownet: a physics-constrained deep continuous space-time super-resolution framework. In: International Conference for High Performance Computing, Networking, Storage, and Analysis (2020)

[23.] Khoo Y, Lu J, Ying L. Solving parametric PDE problems with artificial neural networks. Eur. J. Appl. Math., 2021, 32(3): 421-435,

[24.] Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

[25.] Kochkov, D., Smith, J., Alieva, A., Wang, Q., Brenner, M., Hoyer, S.: Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences of the United States of America. 118(21), 1 (2021)

[26.] Lee K, Carlberg KT. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. J. Comput. Phys., 2020, 404,

[27.] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.: Fourier neural operator for parametric partial differential equations. arXiv:2010.08895 (2020)

[28.] Liszka T, Orkisz J. The finite difference method at arbitrary irregular grids and its application in applied mechanics. Comput. Struct., 1980, 11(1/2): 83-95,

[29.] Liu, X., Zhang, X., Peng, W., Zhou, W., Yao, W.: A novel meta-learning initialization method for physics-informed neural networks. arXiv:2107.10991 (2021)

[30.] Liu Y, Li J, Sun S, Yu B. Advances in Gaussian random field generation: a review. Comput. Geosci., 2019, 23(5): 1011-1047,

[31.] Long Z, Lu Y, Dong B. PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys., 2019, 399,

[32.] Long Z, Lu Y, Ma X, Dong B. PDE-Net: learning PDEs from data. International Conference on Machine Learning, 2018 PMLR 3208-3216

[33.] Lu, Y., Chen, H., Lu, J., Ying, L., Blanchet, J.: Machine learning for elliptic PDEs: fast rate generalization bound, neural scaling law and minimax optimality. In: International Conference on Learning Representations (2022)

[34.] Lu, L., Jin, P., Karniadakis, G.E.: DeepONet: learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv:1910.03193 (2019)

[35.] Lu L, Meng X, Cai S, Mao Z, Goswami S, Zhang Z, Karniadakis GE. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data. Comput. Methods Appl. Mech. Eng., 2022, 393,

[36.] Lu L, Pestourie R, Yao W, Wang Z, Verdugo F, Johnson SG. Physics-informed neural networks with hard constraints for inverse design. SIAM J. Sci. Comput., 2021, 43: B1105-B1132,

[37.] Mehta, I., Gharbi, M., Barnes, C., Shechtman, E., Ramamoorthi, R., Chandraker, M.: Modulated periodic activations for generalizable local functional representations. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021). https://doi.org/10.1109/iccv48922.2021.01395

[38.] Nichol, A., Schulman, J.: Reptile: a scalable metalearning algorithm. arXiv:1803.02999 (2018)

[39.] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

[40.] Psaros AF, Kawaguchi K, Karniadakis GE. Meta-learning PINN loss functions. J. Comput. Phys., 2022, 458,

[41.] Quarteroni A, Manzoni A, Negri F. . Reduced Basis Methods for Partial Differential Equations: an Introduction, 2015 Switzerland Springer

[42.] Raissi M. Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res., 2018, 19(1): 932-955

[43.] Schneider, J.B.: Understanding the Finite-Difference Time-Domain Method. School of Electrical Engineering and Computer Science. Washington State University 28 (2010)

[44.] Sirignano J, Spiliopoulos K. DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys., 2018, 375: 1339-1364,

[45.] Sitzmann V, Martel J, Bergman A, Lindell D, Wetzstein G. Implicit neural representations with periodic activation functions. Adv. Neural Informat. Process. Syst., 2020, 33: 7462-7473

[46.] Stanziola A, Arridge SR, Cox BT, Treeby BE. A Helmholtz equation solver using unsupervised learning: application to transcranial ultrasound. J. Comput. Phys., 2021, 441,

[47.] Tran H, Webster CG, Zhang G. Analysis of quasi-optimal polynomial approximations for parameterized PDEs with deterministic and stochastic coefficients. Numerische Mathematik, 2017, 137(2): 451-493,

[48.] Treeby BE, Cox BT. k-Wave: Matlab toolbox for the simulation and reconstruction of photoacoustic wave fields. J. Biomed. Optics, 2010, 15(2),

[49.] Wang, C., Li, S., He, D., Wang, L.: Is L 2 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L^2$$\end{document} physics informed loss always suitable for training physics informed neural network? In: Advances in Neural Information Processing Systems (2022)

[50.] Wang, S., Wang, H., Perdikaris, P.: Learning the solution operator of parametric partial differential equations with physics-informed deeponets. arXiv:2103.10974 (2021)

[51.] Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., Ahn, S.: Bayesian model-agnostic meta-learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343–7353 (2018)

[52.] Zang Y, Bao G, Ye X, Zhou H. Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys., 2020, 411,

[53.] Zienkiewicz OC, Taylor RL, Nithiarasu P, Zhu J. . The Finite Element Method, 1977 London McGraw-hill

Funding

National Key R &D Program of China(2021ZD0110400)