Transfer Learning in Physics-Informed Neurals Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation

Abstract

AI for PDEs has garnered significant attention, particularly physics-informed neural networks (PINNs). However, PINNs are typically limited to solving specific problems, and any changes in problem conditions necessitate retraining. Therefore, we explore the generalization capability of transfer learning in the strong and energy forms of PINNs across different boundary conditions, materials, and geometries. The transfer learning methods we employ include full finetuning, lightweight finetuning, and low-rank adaptation (LoRA). Numerical experiments include the Taylor-Green Vortex in fluid mechanics and functionally graded materials with elastic properties, as well as a square plate with a circular hole in solid mechanics. The results demonstrate that full finetuning and LoRA can significantly improve convergence speed while providing a slight enhancement in accuracy. However, the overall performance of lightweight finetuning is suboptimal, as its accuracy and convergence speed are inferior to those of full finetuning and LoRA.

Keywords

AI for PDEs / AI for science / computational mechanics / PINNs / transfer learning

References

[1] E. Samaniego, C. Anitescu, S. Goswami, et al., “An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications,” Computer Methods in Applied Mechanics and Engineering 362 (2020): 112790.

[2] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-Informed Machine Learning,” Nature Reviews Physics 3, no. 6 (2021): 422–440, https://doi.org/10.1038/s42254-021-00314-5.

[3] S. Wang, H. Wang, and P. Perdikaris, “Learning the Solution Operator of Parametric Partial Differential Equations With Physics-Informed Deeponets,” Science Advances 7, no. 40 (2021): eabi8605.

[4] Y. Wang, J. Bai, Z. Lin, et al., “Artificial Intelligence for Partial Differential Equations in Computational Mechanics: A Review,” arXiv preprint arXiv:2410.19843 (2024).

[5] W. Yizheng, Z. Xiaoying, T. Rabczuk, and L. Yinghua, “AI for PDEs in Solid Mechanics: A Review,” Advances in Mechanics 54, no. 3 (2024): 1–57.

[6] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations,” Journal of Computational Physics 378 (2019): 686–707.

[7] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, “Learning Nonlinear Operators via Deeponet Based on the Universal Approximation Theorem of Operators,” Nature Machine Intelligence 3, no. 3 (2021): 218–229, https://doi.org/10.1038/s42256-021-00302-5.

[8] Z. Li, N. Kovachki, K. Azizzadenesheli, et al., “Fourier Neural Operator for Parametric Partial Differential Equations,” arXiv preprint arXiv:2010.08895 (2020).

[9] Z. Li, H. Zheng, N. Kovachki, et al., “Physics-Informed Neural Operator for Learning Partial Differential Equations,” ACM/JMS Journal of Data Science 1, no. 3 (2024): 1–27.

[10] M. S. Eshaghi, C. Anitescu, M. Thombre, Y. Wang, X. Zhuang, and T. Rabczuk, “Variational Physics-Informed Neural Operator (VINO) for Solving Partial Differential Equations,” arXiv preprint arXiv:2411.06587 (2024).

[11] L. Yang, S. Liu, T. Meng, and S. J. Osher, “In-Context Operator Learning With Data Prompts for Differential Equation Problems,” Proceedings of the National Academy of Sciences 120, no. 39 (2023): e2310142120.

[12] S. Desai, M. Mattheakis, H. Joy, P. Protopapas, and S. Roberts, “One-Shot Transfer Learning of Physics-Informed Neural Networks,” arXiv preprint arXiv:2110.11286 (2021).

[13] Y. Gao, K. C. Cheung, and M. K. Ng, “SVD-PINNS: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition,” in 2022 IEEE Symposium Series on Computational Intelligence (SSCI), eds. H. Ishibuchi, C.-K. Kwoh, Ah-H. Tan, et al. (IEEE, 2022), 1443–1450.

[14] F. Zhuang, Z. Qi, K. Duan, et al., “A Comprehensive Survey on Transfer Learning,” Proceedings of the IEEE 109, no. 1 (2020): 43–76.

[15] C. Xu, B. T. Cao, Y. Yuan, and G. Meschke, “Transfer Learning Based Physics-Informed Neural Networks for Solving Inverse Problems in Engineering Structures Under Different Loading Scenarios,” Computer Methods in Applied Mechanics and Engineering 405 (2023): 115852.

[16] H. Guo, X. Zhuang, P. Chen, N. Alajlan, and T. Rabczuk, “Analysis of Three-Dimensional Potential Problems in Non-Homogeneous Media With Physics-Informed Deep Collocation Method Using Material Transfer Learning and Sensitivity Analysis,” Engineering With Computers 38, no. 6 (2022): 5423–5444.

[17] A. Chakraborty, C. Anitescu, X. Zhuang, and T. Rabczuk, “Domain Adaptation Based Transfer Learning Approach for Solving PDEs on Complex Geometries,” Engineering With Computers 38, no. 5 (2022): 4569–4588.

[18] X. Chen, C. Gong, Q. Wan, et al., “Transfer Learning for Deep Neural Network-Based Partial Differential Equations Solving,” Advances in Aerodynamics 3 (2021): 1–14.

[19] E. Haghighat, M. Raissi, A. Moure, H. Gomez, and R. Juanes, “A Physics-Informed Deep Learning Framework for Inversion and Surrogate Modeling in Solid Mechanics,” Computer Methods in Applied Mechanics and Engineering 379 (2021): 113741, https://doi.org/10.1016/j.cma.2021.113741.

[20] S. Goswami, C. Anitescu, S. Chakraborty, and T. Rabczuk, “Transfer Learning Enhanced Physics Informed Neural Network for Phase-Field Modeling of Fracture,” Theoretical and Applied Fracture Mechanics 106 (2020): 102447.

[21] S. Chakraborty, “Transfer Learning Based Multi-Fidelity Physics Informed Deep Neural Network,” Journal of Computational Physics 426 (2021): 109942.

[22] E. J. Hu, Y. Shen, P. Wallis, et al., “Lora: Low-Rank Adaptation of Large Language Models,” arXiv preprint arXiv:2106.09685 (2021).

[23] R. Majumdar, V. Jadhav, A. Deodhar, S. Karande, L. Vig, and V. Runkana, “Hyperlora for PDEs,” arXiv preprint arXiv:2308.09290 (2023).

[24] W. Cho, K. Lee, D. Rim, and N. Park, “Hypernetwork-Based Meta-Learning for Low-Rank Physics-Informed Neural Networks,” Advances in Neural Information Processing Systems 36 (2023): 11219–11231.

[25] Y. Fung, Foundations of Solid Mechanics. 1965 (Englewood Cliffs, NJ, 2010). 436.

[26] Y. Wang, J. Sun, J. Bai, et al., “Kolmogorov Arnold Informed Neural Network: A Physics-Informed Deep Learning Framework for Solving Forward and Inverse Problems Based on Kolmogorov-Arnold Networks,” Computer Methods in Applied Mechanics and Engineering 433 (2025): 117518.

[27] X. Zhang, L. Wang, J. Helwig, et al., “Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems,” arXiv preprint arXiv:2307.08423 (2023).

[28] Y. Wang, J. Sun, W. Li, Z. Lu, and Y. Liu, “Cenn: Conservative Energy Method Based on Neural Networks With Subdomains for Solving Variational Problems Involving Heterogeneous and Complex Geometries,” Computer Methods in Applied Mechanics and Engineering 400 (2022): 115491.

[29] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a Unified View of Parameter-Efficient Transfer Learning,” arXiv preprint arXiv:2110.04366 (2021).

[30] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language Models Are Unsupervised Multitask Learners,” OpenAI Blog 1, no. 8 (2019): 9.

[31] S. V. Patankar and D. B. Spalding, “A Calculation Procedure for Heat, Mass and Momentum Transfer in Three-Dimensional Parabolic Flows,” in Numerical Prediction of Flow, Heat Transfer, Turbulence and Combustion, eds. S. V. Patankar, A. Pollard, and A. K. Singhal, (Elsevier, 1983), 54–73.

[32] S. Wang, Y. Teng, and P. Perdikaris, “Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks,” SIAM Journal on Scientific Computing 43, no. 5 (2021): A3055–A3081.

[33] S. Wang, H. Wang, and P. Perdikaris, “On the Eigenvector Bias of Fourier Feature Networks: From Regression to Solving Multi-Scale PDEs With Physics-Informed Neural Networks,” Computer Methods in Applied Mechanics and Engineering 384 (2021): 113938, https://doi.org/10.1016/j.cma.2021.113938.

[34] S. Wang, X. Yu, and P. Perdikaris, “When and Why Pinns Fail to Train: A Neural Tangent Kernel Perspective,” Journal of Computational Physics 449 (2022): 110768.

[35] M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 (Springer, 2014), 818–833.

[36] A. Paszke, S. Gross, S. Chintala, et al., “Automatic Differentiation in Pytorch,” in Proceedings of the 31st Conference on Neural Information Processing Systems, Autodiff Workshop (NeurIPS, 2017).

[37] A. M. Roy, R. Bose, V. Sundararaghavan, and R. Arróyave, “Deep Learning-Accelerated Computational Framework Based on Physics Informed Neural Network for the Solution of Linear Elasticity,” Neural Networks 162 (2023): 472–489.

[38] M. S. Eshaghi, M. Bamdad, C. Anitescu, Y. Wang, X. Zhuang, and T. Rabczuk, “Applications of Scientific Machine Learning for the Analysis of Functionally Graded Porous Beams,” Neurocomputing 619 (2025): 129119.

[39] V. M. Nguyen-Thanh, X. Zhuang, and T. Rabczuk, “A Deep Energy Method for Finite Deformation Hyperelasticity,” European Journal of Mechanics-A/Solids 80 (2020): 103874.

[40] Y. Wang, J. Sun, T. Rabczuk, and Y. Liu, “Dcem: A Deep Complementary Energy Method for Solid Mechanics,” International Journal for Numerical Methods in Engineering 125, no. 24 (2024): e7585, https://doi.org/10.1002/nme.7585.

[41] A. M. Roy and S. Guha, “A Data-Driven Physics-Constrained Deep Learning Computational Framework for Solving Von Mises Plasticity,” Engineering Applications of Artificial Intelligence 122 (2023): 106049.

[42] A. M. Roy, S. Guha, V. Sundararaghavan, and R. Arróyave, “Physics-Infused Deep Neural Network for Solution of Non-Associative Drucker-Prager Elastoplastic Constitutive Model,” Journal of the Mechanics and Physics of Solids 185 (2024): 105570.

[43] R. Bose and A. M. Roy, “Invariance Embedded Physics-Infused Deep Neural Network-Based Sub-Grid Scale Models for Turbulent Flows,” Engineering Applications of Artificial Intelligence 128 (2024): 107483.

RIGHTS & PERMISSIONS

2025 The Author(s). International Journal of Mechanical System Dynamics published by John Wiley & Sons Australia, Ltd on behalf of Nanjing University of Science and Technology.