A Comparative Analysis of Performance-Based Resilience Metrics via a Quantitative-Qualitative Combined Approach: Are We Measuring the Same Thing?

Abstract

Since the proposal of the pioneering “resilience triangle” paradigm, various time-series performance-based metrics have been devised for resilience quantification. The numerous choices diversify the toolbox for measuring this compound system concept; however, this multiplicity causes intractable questions for applications, including “Do these metrics measure the same resilience?” and “Which one to pick under what circumstance?” In this study, we attempted to address these two fundamental issues using a comprehensive comparative investigation. Through a quantitative-qualitative combined approach, 12 popular performance-based resilience metrics are compared using empirical data from China’s aviation system under the disturbance of COVID-19. Quantitative results indicate that only 12 of the 66 metric pairs are strongly positively correlated and with no significant differences in quantification outcomes; qualitative results indicate that the majority of the metrics are based on different definition interpretations, basic components, and expression forms, and thus essentially measure different resilience. The advantages and disadvantages of each metric are comparatively discussed, and a “how to choose” guideline for metric users is proposed. This study is an introspective investigation of resilience quantification studies, aiming to offer a new perspective to scrutinize those benchmarking metrics.

Keywords

COVID-19 / Infrastructure system / Model comparison / Resilience metrics / Systemic risk

References

[1] Arjomandi-Nezhad A, Fotuhi-Firuzabad M, Moeini-Aghtaie M, Safdarian A, Dehghanian P, Wang F. Modeling and optimizing recovery strategies for power distribution system resilience. IEEE Systems Journal, 2020, 15(4): 4725-4734

[2] Arreguín-Sánchez F. Measuring resilience in aquatic trophic networks from supply–demand-of-energy relationships. Ecological Modelling, 2014, 272: 271-276

[3] Attoh-Okine NO, Cooper AT, Mensah SA. Formulation of resilience index of urban infrastructure using belief functions. IEEE Systems Journal, 2009, 3(2): 147-153

[4] Ayyub BM. Systems resilience for multihazard environments: Definition, metrics, and valuation for decision making. Risk Analysis, 2014, 34(2): 340-355

[5] Bruneau M, Chang SE, Eguchi RT, Lee GC, O’Rourke TD, Reinhorn AM, Shinozuka M, Tierney K A framework to quantitatively assess and enhance the seismic resilience of communities. Earthquake Spectra, 2003, 19(4): 733-752

[6] Bruyelle JL, O’Neill C, El-Koursi EM, Hamelin F, Sartori N, Khoudour L. Improving the resilience of metro vehicle and passengers for an effective emergency response to terrorist attacks. Safety Science, 2014, 62: 37-45

[7] Cai B, Xie M, Liu Y, Liu Y, Feng Q. Availability-based engineering resilience metric and its corresponding evaluation methodology. Reliability Engineering & System Safety, 2018, 172: 216-224

[8] Cai, B., Y. Zhang, H. Wang, Y. Liu, R. Ji, C. Gao, X. Kong, and J. Liu. 2021. Resilience evaluation methodology of engineering systems with dynamic-Bayesian-network-based degradation and maintenance. Reliability Engineering & System Safety. 209: Article 107464.

[9] Chen L, Miller-Hooks E. Resilience: An indicator of recovery capacity in intermodal freight transport. Transportation Science, 2012, 46(1): 109-123

[10] Civil Aviation Administration of China. 2021. Statistical bulletin on the development of the civil aviation industry in 2020. http://www.caac.gov.cn/XXGK/XXGK/TJSJ/202106/t20210610_207915.html. Accessed 9 Oct 2023 (in Chinese).

[11] Clédel T, Cuppens N, Cuppens F, Dagnas R. Resilience properties and metrics: How far have we gone?. Journal of Surveillance, Security and Safety, 2020, 1(2): 119-139.

[12] Cutter SL. Resilience to what? Resilience for whom?. The Geographical Journal, 2016, 182(2): 110-113

[13] Cutter SL, Barnes L, Berry M, Burton C, Evans E, Tate E, Webb J. A place-based model for understanding community resilience to natural disasters. Global Environmental Change, 2008, 18(4): 598-606

[14] Francis R, Bekera B. A metric and frameworks for resilience analysis of engineered and infrastructure systems. Reliability Engineering & System Safety, 2014, 121: 90-103

[15] Ghaffarpour R, Mozafari B, Ranjbar AM, Torabi T. Resilience oriented water and energy hub scheduling considering maintenance constraint. Energy, 2018, 158: 1092-1104

[16] Gheorghe AV, Vamanu DV, Katina PF, Pulfer R. Gheorghe AV, Vamanu DV, Katina PF, Pulfer R. Critical infrastructures, key resources, and key assets. Critical infrastructures, key resources, key assets: Risk, vulnerability, resilience, fragility, and perception governance, 2018, Cham: Springer 3-37

[17] Henry D, Ramirez-Marquez JE. Generic metrics and quantitative approaches for system resilience as a function of time. Reliability Engineering & System Safety, 2012, 99: 114-122

[18] Huck A, Monstadt J. Urban and infrastructure resilience: Diverging concepts and the need for cross-boundary learning. Environmental Science & Policy, 2019, 100: 211-220

[19] Jiang, Q., B. Cai, Y. Zhang, M. Xie, and C. Liu. 2023. Resilience assessment methodology of natural gas network system under random leakage. Reliability Engineering & System Safety 234: Article 109134.

[20] Liu, W., and Z. Song. 2020. Review of studies on the resilience of urban critical infrastructure networks. Reliability Engineering & System Safety 193: Article 106617.

[21] Mao Q, Li N. Assessment of the impact of interdependencies on the resilience of networked critical infrastructure systems. Natural Hazards, 2018, 93(1): 315-337

[22] Martišauskas L, Augutis J, Krikštolaitis R. Methodology for energy security assessment considering energy system resilience to disruptions. Energy Strategy Reviews, 2018, 22: 106-118

[23] Meerow S, Newell JP. Urban resilience for whom, what, when, where, and why?. Urban Geography, 2019, 40(3): 309-329

[24] Najarian M, Lim GJ. Design and assessment methodology for system resilience metrics. Risk Analysis, 2019, 39(9): 1885-1898

[25] Nan C, Sansavini G. A quantitative method for assessing resilience of interdependent infrastructures. Reliability Engineering & System Safety, 2017, 157: 35-53

[26] National Health Commission of the People’s Republic of China. 2021. The latest situation of the COVID-19 as of 24:00 on March 21. http://www.nhc.gov.cn/xcs/yqtb/202103/3309504a6f044ecaa2bc506179d52d5b.shtml. Accessed 9 Oct 2023 (in Chinese).

[27] Nishimi K, Choi KW, Cerutti J, Powers A, Bradley B, Dunn EC. Measures of adult psychological resilience following early-life adversity: How congruent are different measures?. Psychological Medicine, 2021, 51(15): 2637-2646

[28] Ouyang, M., and L. Duenas-Osorio. 2012. Time-dependent resilience assessment and improvement of urban infrastructure systems. Chaos: An Interdisciplinary Journal of Nonlinear Science 22(3): Article 033122.

[29] Panteli M, Mancarella P, Trakas DN, Kyriakides E, Hatziargyriou ND. Metrics and quantification of operational and infrastructure resilience in power systems. IEEE Transactions on Power Systems, 2017, 32(6): 4732-4742

[30] Patriarca, R., A. De Paolis, F. Costantino, and G. Di Gravio. 2021. Simulation model for simple yet robust resilience assessment metrics for engineered systems. Reliability Engineering & System Safety 209: Article 107467.

[31] Pettit TJ, Fiksel J, Croxton KL. Ensuring supply chain resilience: Development of a conceptual framework. Journal of Business Logistics, 2010, 31(1): 1-21

[32] Raoufi H, Vahidinasab V. Power system resilience assessment considering critical infrastructure resilience approaches and government policymaker criteria. IET Generation, Transmission & Distribution, 2021, 15(20): 2819-2834

[33] Serdar, M.Z., M. Koç, and S.G. Al-Ghamdi. 2022. Urban transportation networks resilience: Indicators, disturbances, and assessment methods. Sustainable Cities and Society 76: Article 103452.

[34] Sherwani RAK, Shakeel H, Awan WB, Faheem M, Aslam M. Analysis of COVID-19 data using neutrosophic Kruskal Wallis H test. BMC Medical Research Methodology, 2021, 21(1): 1-7

[35] Shirali GA, Mohammadfam I, Ebrahimipour V. A new method for quantitative assessment of resilience engineering by PCA and NT approach: A case study in a process industry. Reliability Engineering & System Safety, 2013, 119: 88-94

[36] Tang, J., H. Heinimann, K. Han, H. Luo, and B. Zhong. 2020. Evaluating resilience in urban transportation systems for sustainability: A systems-based Bayesian network model. Transportation Research Part C: Emerging Technologies 121: Article 102840.

[37] Tang J, Wei S, Li D, Li X. Optimizing systemic redundancy of traffic sensor networks while maintaining resilience: New evidence from using graph learning. IEEE Systems Journal, 2023, 17(3): 4567-4578

[38] Tang, J., P. Zhao, Z. Gong, H. Zhao, F. Huang, J. Li, Z. Chen, Y. Ling, and J. Chen. 2023. Resilience patterns of human mobility in response to extreme urban floods. National Science Review 10(8): Article nwad097.

[39] Tierney K, Bruneau M. Conceptualizing and measuring resilience: A key to disaster loss reduction. TR News, 2007, 250: 14-17.

[40] Toroghi, S.S.H., and V.M. Thomas. 2020. A framework for the resilience analysis of electric infrastructure systems including temporary generation systems. Reliability Engineering & System Safety 202: Article 107013.

[41] Vishnu N, Kameshwar S, Padgett JE. Road transportation network hazard sustainability and resilience: Correlations and comparisons. Structure and Infrastructure Engineering, 2023, 19(3): 345-365

[42] Wang J, Zhao J, Deng S, Sun T, Du Y, Li K, Xu Y. Integrated assessment for solar-assisted carbon capture and storage power plant by adopting resilience thinking on energy system. Journal of Cleaner Production, 2019, 208: 1009-1021

[43] Xu, Z., and S.S. Chopra. 2022. Network-based assessment of metro infrastructure with a spatial-temporal resilience cycle framework. Reliability Engineering & System Safety 223: Article 108434.

[44] Yarveisy, R., C. Gao, and F. Khan. 2020. A simple yet robust resilience assessment metrics. Reliability Engineering & System Safety 197: Article 106810.

[45] Yin, J., X. Ren, R. Liu, T. Tang, and S. Su. 2022. Quantitative analysis for resilience-based urban rail systems: A hybrid knowledge-based and data-driven approach. Reliability Engineering & System Safety 219: Article 108183.

[46] Zobel CW, Khansa L. Characterizing multi-event disaster resilience. Computers & Operations Research, 2014, 42: 83-94