Data-Driven Modeling of Partially Observed Biological Systems

Abstract

We present a numerical approach for modeling unknown dynamical systems using partially observed data, with a focus on biological systems with (relatively) complex dynamical behavior. As an extension of the recently developed deep neural network (DNN) learning methods, our approach is particularly suitable for practical situations when (i) measurement data are available for only a subset of the state variables, and (ii) the system parameters cannot be observed or measured at all. We demonstrate that, with a properly designed DNN structure with memory terms, effective DNN models can be learned from such partially observed data containing hidden parameters. The learned DNN model serves as an accurate predictive tool for system analysis. Through a few representative biological problems, we demonstrate that such DNN models can capture qualitative dynamical behavior changes in the system, such as bifurcations, even when the parameters controlling such behavior changes are completely unknown throughout not only the model learning process but also the system prediction process. The learned DNN model effectively creates a “closed” model involving only the observables when such a closed-form model does not exist mathematically.

Keywords

Deep neural network (DNN) / Governing equation discovery / Biological system / Partial observation

References

[1.] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/. Software available from tensorflow.org

[2.] Bongard J, Lipson H. Automated reverse engineering of nonlinear dynamical systems. Proc. Natl. Acad. Sci. USA, 2007, 104: 9943-9948,

[3.] Chan S, Elsheikh A. A machine learning approach for efficient uncertainty quantification using multiscale methods. J. Comput. Phys., 2018, 354: 494-511,

[4.] Chen Z, Churchill V, Wu K, Xiu D. Deep neural network modeling of unknown partial differential equations in nodal space. J. Comput. Phys., 2022, 449,

[5.] Chen Z, Xiu D. On generalized residual network for deep learning of unknown dynamical systems. J. Comput. Phys., 2021, 438,

[6.] Daniels, B.C., Nemenman, I.: Automated adaptive inference of phenomenological dynamical models. Nature Communications 6, 8133 (2015)

[7.] Daniels BC, Nemenman I. Efficient inference of parsimonious phenomenological models of cellular dynamics using S-systems and alternating regression. PloS One, 2015, 10,

[8.] DeAngelis, D.L., Yurek, S.: Equation-free modeling unravels the behavior of complex ecological systems. Proc. Natl. Acad. Sci. USA 112, 3856–3857 (2015).

[9.] E W, Engquist B, Huang Z. Heterogeneous multiscale method: a general methodology for multiscale modeling. Phys. Rev. B, 2003, 67: 092101,

[10.] Fall C, Marland E, Wagner J, Tyson J. . Computational Cell Biology, 2010 Berlin Springer Science+Business Media, Inc.

[11.] Fu X, Chang L-B, Xiu D. Learning reduced systems via deep neural networks with memory. J. Mach. Learn. Model. Comput., 2020, 1: 97-118,

[12.] Fu, X., Mao, W., Chang, L.-B., Xiu, D.: Modeling unknown dynamical systems with hidden parameters. J. Mach. Learn. Model. Comput. 3, 79-95 (2022)

[13.] Giannakis D, Majda AJ. Nonlinear Laplacian spectral analysis for time series with intermittency and low-frequency variability. Proc. Natl. Acad. Sci. USA, 2012, 109: 2222-2227,

[14.] Gonzalez-Garcia R, Rico-Martinez R, Kevrekidis IG. Identification of distributed parameter systems: a neural net based approach. Comput. Chem. Eng., 1998, 22: S965-S968,

[15.] Grimm, V., Railsback, S.F.: Individual-Based Modeling and Ecology. Princeton University Press, Princeton (2005)

[16.] Han J, Jentzen A, E WN. Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci., 2018, 115: 8505-8510,

[17.] Hau, D.T., Coiera, E.W.: Learning qualitative models from physiological signals internal accession date only. Technical report (1995)

[18.] Hesthaven J, Ubbiali S. Non-intrusive reduced order modeling of nonlinear problems using neural networks. J. Comput. Phys., 2018, 363: 55-78,

[19.] Hodgkin, A.L., Huxley, A.F.: A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol. 117, 500–544 (1952).

[20.] Karumuri, S., Tripathy, R., Bilionis, I., Panchal, J.: Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks, J. Comp. Phys. 404, 109120 (2019)

[21.] Kevrekidis IG, Gear CW, Hyman JM, Kevrekidid PG, Runborg O, Theodoropoulos C. Equation-free, coarse-grained multiscale computation: enabling mocroscopic simulators to perform system-level analysis. Commun. Math. Sci., 2003, 1: 715-762,

[22.] Khoo, Y., Lu, J., Ying, L.: Solving parametric PDE problems with artificial neural networks. Euro.Jnl. Appl. Math. 32, 21–435 (2018). arXiv:1707.03351

[23.] Long, Z., Lu, Y., Dong, B.: PDE-Net 2.0: learning PDEs from data with a numeric-symbolic hybrid deep network. J. Comput. Phys. 399, 108925 (2019). arXiv:1812.04426 [math.ST]

[24.] Long, Z., Lu, Y., Ma, X., Dong, B.: PDE-net: learning PDEs from data. Proceedings of the 35th International Conference on Machine Learning. PMLR 80, 3208–3216 (2018)

[25.] Maki LW, Keizer J. Mathematical analysis of a proposed mechanism for oscillatory insulin secretion in perifused hit-15 cells. Bull. Math. Biol., 1995, 57: 569-591,

[26.] Mangan NM, Brunton SL, Proctor JL, Kutz JN. Inferring biological networks by sparse identification of nonlinear dynamics. IEEE Trans. Mol. Biol. Multi-Scale Commun., 2016, 2: 52-63,

[27.] Mardt A, Pasquali L, Wu H, Noe F. VAMPnets for deep learning of molecular kinetics. Nat. Commun., 2018, 9: 5,

[28.] Mori H. Transport, collective motion, and Brownian motion. Prog. Theor. Phys., 1965, 33: 423-455,

[29.] Morris, C., Lecar, H.: Voltage oscillations in the barnacle giant muscle fiber (1981). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1327511/

[30.] Pawar S, Rahman SM, Vaddireddy H, San O, Rasheed A, Vedula P. A deep learning enabler for nonintrusive reduced order modeling of fluid flows. Phys. Fluids, 2019, 31,

[31.] Perretti CT, Munch SB, Sugihara G. Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data. Proc. Natl. Acad. Sci. USA, 2013, 110: 5253-5257,

[32.] Qin T, Chen Z, Jakeman J, Xiu D. Data-driven learning of non-autonomous systems. SIAM J. Sci. Comput., 2021, 43: A1607-A1624,

[33.] Qin T, Chen Z, Jakeman J, Xiu D. Deep learning of parameterized equations with applications to uncertainty quantification. Int. J. Uncertain. Quantif., 2021, 11: 63-82,

[34.] Qin T, Wu K, Xiu D. Data driven governing equations approximation using deep neural networks. J. Comput. Phys., 2019, 395: 620-635,

[35.] Raissi M. Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res., 2018, 19: 1-24

[36.] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (part I): data-driven solutions of nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019). arXiv:1711.10561

[37.] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (part II): data-driven discovery of nonlinear partial differential equations (2017). arXiv:1711.10566

[38.] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Multistep neural networks for data-driven discovery of nonlinear dynamical systems (2018). arXiv:1801.01236

[39.] Ray D, Hesthaven J. An artificial neural network as a troubled-cell indicator. J. Comput. Phys., 2018, 367: 166-191,

[40.] Rinzel J, Ermentrout G. Koch C, Segev I. Analysis of neural excitability and oscillations. Methods in Neuronal Modeling, 1998 2 Cambridge MIT Press 251-291

[41.] Rudy SH, Kutz JN, Brunton SL. Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. J. Comput. Phys., 2019, 396: 483-506,

[42.] Schmidt MD, Lipson H. Distilling free-form natural laws from experimental data. Science, 2009, 324: 81-85,

[43.] Schmidt MD, Vallabhajosyula RR, Jenkins JW, Hood JE, Soni AS, Wikswo JP, Lipson H. Automated refinement and inference of analytical models for metabolic networks. Phys. Biol., 2011, 8,

[44.] Sugihara G, May R, Ye H, Hsieh C, Deyle E, Fogarty M, Munch S. Detecting causality in complex ecosystems. Science, 2012, 338: 496-500,

[45.] Sun, Y., Zhang, L., Schaeffer, H.: NeuPDE: neural network based ordinary and partial differential equations for modeling time-dependent data. Proc. Machine Learning Res. 107, 352–372 (2020). arXiv:1908.03190

[46.] Tripathy R, Bilionis I. Deep UQ: learning deep neural network surrogate model for high dimensional uncertainty quantification. J. Comput. Phys., 2018, 375: 565-588,

[47.] Tsumoto K, Kitajima H, Yoshinaga T, Aihara K, Kawakami H. Bifurcations in Morris–Lecar neuron model. Neurocomputing, 2006, 69: 293-316,

[48.] Voss HU, Kolodner P, Abel M, Kurths J. Amplitude equations from spatiotemporal binary-fluid convection data. Phys. Rev. Lett., 1999, 83: 3422,

[49.] Wang Q, Ripamonti N, Hesthaven J. Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori-Zwanzig formalism. J. Comput. Phys., 2020, 410,

[50.] Wang, Y., Shen, Z., Long, Z., Dong, B.: Learning to discretize: solving 1D scalar conservation laws via deep reinforcement learning. Commun. Comput. Phys. 28, 2158–2179 (2020). arXiv:1905.11079

[51.] Wood SN, Thomas MB. Super-sensitivity to structure in biological models. Proc. R. Soc. B: Biol. Sci., 1999, 266: 565-570,

[52.] Wu K, Xiu D. Data-driven deep learning of partial differential equations in modal space. J. Comput. Phys., 2020, 408,

[53.] Ye H, Beamish RJ, Glaser SM, Grant SCH, Hsieh C, Richards LJ, Schnute JT, Sugihara G. Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proc. Natl. Acad. Sci. USA, 2015, 112: E1569-E1576,

[54.] Yodzis, P.: The indeterminacy of ecological interactions as perceived through perturbation. Technical Report (1988)

[55.] Zhu Y, Zabaras N. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. J. Comput. Phys., 2018, 366: 415-447,

[56.] Zwanzig R. Nonlinear generalized Langevin equations. J. Stat. Phys., 1973, 9: 215-220,

Funding

National Science Foundation (US)(DMS 1813071); Air Force Office of Scientific Research(FA9550-22-1-0011)