Failure-Informed Adaptive Sampling for PINNs, Part II: Combining with Re-sampling and Subset Simulation

Abstract

This is the second part of our series works on failure-informed adaptive sampling for physic-informed neural networks (PINNs). In our previous work (SIAM J. Sci. Comput. 45: A1971–A1994), we have presented an adaptive sampling framework by using the failure probability as the posterior error indicator, where the truncated Gaussian model has been adopted for estimating the indicator. Here, we present two extensions of that work. The first extension consists in combining with a re-sampling technique, so that the new algorithm can maintain a constant training size. This is achieved through a cosine-annealing, which gradually transforms the sampling of collocation points from uniform to adaptive via the training progress. The second extension is to present the subset simulation (SS) algorithm as the posterior model (instead of the truncated Gaussian model) for estimating the error indicator, which can more effectively estimate the failure probability and generate new effective training points in the failure region. We investigate the performance of the new approach using several challenging problems, and numerical experiments demonstrate a significant improvement over the original algorithm.

Keywords

Physic-informed neural networks (PINNs) / Adaptive sampling / Failure probability

References

[1.] Au SK, Beck JL. Estimation of small failure probabilities in high dimensions by subset simulation. Probab. Eng. Mech., 2001, 16: 263-277,

[2.] Bischof, R., Kraus, M.: Multi-objective loss balancing for physics-informed deep learning. arXiv:2110.09813 (2021)

[3.] Daw, A., Bu, J., Wang, S., Perdikaris, P., Karpatne, A.: Rethinking the importance of sampling in physics-informed neural networks. arXiv:2207.02338 (2022)

[4.] Weinan E. Machine learning and computational mathematics. Commun. Comput. Phys., 2020, 28: 1639-1670,

[5.] E, W.N., Yu, B.: The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat. 6, 1–12 (2018)

[6.] Gao Z, Yan L, Zhou T. Failure-informed adaptive sampling for PINNs. SIAM J. Sci. Comput., 2023, 45: A1971-A1994,

[7.] Krishnapriyan A, Gholami A, Zhe S, Kirby R, Michael WM. Characterizing possible failure modes in physics-informed neural networks. Adv. Neural Inform. Process. Syst., 2021, 34: 26548-26560

[8.] Lu L, Meng X, Mao Z, Karniadakis GE. DeepXDE: a deep learning library for solving differential equations. SIAM Rev., 2021, 63: 208-228,

[9.] McClenny, L., Braga-Neto, U.: Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv:2009.04544 (2020)

[10.] Peng, W., Zhou, W., Zhang, X., Yao, W., Liu, Z.: RANG: a residual-based adaptive node generation method for physics-informed neural networks. arXiv:2205.01051 (2022)

[11.] Raissi M, Perdikaris P, George EK. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 2019, 378: 686-707,

[12.] Sirignano J, Spiliopoulos K. DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys., 2018, 375: 1339-1364,

[13.] Subramanian, S., Kirby, R.M., Mahoney, M.W., Gholami, A.: Adaptive self-supervision algorithms for physics-informed neural networks. arXiv:2207.04084 (2022)

[14.] Tang, K., Wan, X., Yang, C.: DAS: a deep adaptive sampling method for solving partial differential equations. arXiv:2112.14038 (2021)

[15.] Wang, S., Sankaran, S., Perdikaris, P.: Respecting causality is all you need for training physics-informed neural networks. arXiv:2203.07404 (2022)

[16.] Wang S, Teng Y, Perdikaris P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM J. Sci. Comput., 2021, 43: A3055-A3081,

[17.] Wang S, Yu XL, Perdikaris P. When and why PINNs fail to train: a neural tangent kernel perspective. J. Comput. Phys., 2022, 449,

[18.] Wu CX, Zhu M, Tan Q, Kartha Y, Lu L. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Comput. Methods Appl. Mech. Eng., 2023, 403,

[19.] Xiang Z, Wei P, Liu X, Yao W. Self-adaptive loss balanced physics-informed neural networks. Neurocomputing, 2022, 496: 11-34,

[20.] Zang Y, Bao G, Ye X, Zhou H. Weak adversarial networks for high-dimensional partial differential equations. J. Comput. Phys., 2020, 411,

[21.] Zuev, K.: Subset simulation method for rare event estimation: an introduction. arXiv:1505.03506 (2015)