How performance metric choice influences individual tree mortality model selection

Abstract

Understanding tree mortality is crucial to understand forest dynamics and is essential for growth models and simulators. Although factors such as competition, drought, and pathogens drive mortality, their underlying mechanisms remain difficult to model. While substantial attention has focused on selecting appropriate algorithms and covariates, evaluating individual tree mortality models also requires careful selection of performance criteria. This study compares seven different metrics to assess their impact on model evaluation and selection. Results show that candidate models exhibited varying performances across metrics and that the choice of metric significantly influences the selection of the best model. When no confusion matrix was available, the area under the precision-recall curve (AUCPR) emerged as a more reliable alternative to the area under the ROC curve (AUC), offering a more informative assessment for imbalanced datasets. When a confusion matrix was available, Cohen’s Kappa coefficient (K) and Matthews correlation coefficient (MCC) outperformed accuracy-based metrics, providing a fairer evaluation of both live and dead tree classifications. These findings emphasize the importance of choosing appropriate evaluation standards to enhance mortality model assessment and ensure reliable predictions in forestry applications.

Keywords

Forest modeling / Survival / Binary classification / Area under the precision-recall curve (AUCPR) / Mathews correlation coefficient (MCC)

References

[1] Adame P, del Río M, Cañellas I. Modeling individual-tree mortality in Pyrenean oak (Quercus pyrenaica Willd.) stands. Ann for Sci, 2010, 67(8): 810

[2] Bircher N, Cailleret M, Bugmann H. The agony of choice: different empirical mortality models lead to sharply different future forest dynamics. Ecol Appl, 2015, 2551303-1318

[3] Bravo F, Hann DW, Maguire DA. Impact of competitor species composition on predicting diameter growth and survival rates of Douglas-fir trees in southwestern Oregon. Can J for Res, 2001, 31(12): 2237-2247

[4] Bravo F, Ordóñez C, Vázquez-Veloso A, Michalakopoulos S. SIMANFOR cloud decision support system: structure, content, and applications. Ecol Model, 2025, 499: 110912

[5] Bravo F, Bravo-Núñez A (2017) Clasificación de la calidad de estación forestal mediante técnicas de aprendizaje automático (machine learning). In: Actas 7^o Congreso Forestal Español. Sociedad Española de Ciencias Forestales, Cáceres. (in Spanish). https://www.congresoforestal.es/fichero.php?t=41725&i=5686&m=2185

[6] Bravo-Oviedo A, Sterba H, del Río M, Bravo F. Competition-induced mortality for Mediterranean Pinus pinaster Ait. and P. sylvestris L. For Ecol Manage, 2006, 222(1–3): 88-98

[7] Bugmann H, Seidl R, Hartig F, Bohn F, Brůna J, Cailleret M, François L, Heinke J, Henrot AJ, Hickler T, Hülsmann L, Huth A, Jacquemin I, Kollas C, Lasch-Born P, Lexer MJ, Merganič J, Merganičová K, Mette T, Miranda BR, Nadal-Sala D, Rammer W, Rammig A, Reineking B, Roedig E, Sabaté S, Steinkamp J, Suckow F, Vacchiano G, Wild J, Xu CG, Reyer CPO. Tree mortality submodels drive simulated long-term forest dynamics: assessing 15 models from the stand to global scale. Ecosphere, 2019, 10(2 e02616

[8] Chicco D. Ten quick tips for machine learning in computational biology. BioData Min, 2017, 101): 35

[9] Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 2020, 2116

[10] Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas, 1960, 20137-46

[11] da Rocha SJSS, Torres CMME, Jacovine LAG, Leite HG, Gelcer EM, Neves KM, Schettini BLS, Villanova PH, da Silva LF, Reis LP, Zanuncio JC. Artificial neural networks: modeling tree survival and mortality in the Atlantic Forest biome in Brazil. Sci Total Environ, 2018, 645: 655-661

[12] de la Cruz Huayanay A, Bazán JL, Russo CM. Performance of evaluation metrics for classification in imbalanced data. Comput Stat, 2025, 40(3): 1447-1473

[13] Fick SE, Hijmans RJ. WorldClim 2: new 1-km spatial resolution climate surfaces for global land areas. Int J Climatol, 2017, 37(12): 4302-4315

[14] Fravolini G, Tognetti R, Lombardi F, Egli M, Ascher-Jenull J, Arfaioli P, Bardelli T, Cherubini P, Marchetti M. Quantifying decay progression of deadwood in Mediterranean mountain forests. For Ecol Manage, 2018, 408: 228-237

[15] Friend AD, Lucht W, Rademacher TT, Keribin R, Betts R, Cadule P, Ciais P, Clark DB, Dankers R, Falloon PD, Ito A, Kahana R, Kleidon A, Lomas MR, Nishina K, Ostberg S, Pavlick R, Peylin P, Schaphoff S, Vuichard N, Warszawski L, Wiltshire A, Woodward FI. Carbon residence time dominates uncertainty in terrestrial vegetation responses to future climate and atmospheric CO₂. Proc Natl Acad Sci U S A, 2014, 111(9): 3280-3285

[16] Gamer M, Lemon J, Puspendra Singh IF (2019) irr: Various Coefficients of interrater reliability and agreement. CRAN. https://doi.org/10.32614/CRAN.package.irr

[17] Harris I, Osborn TJ, Jones P, Lister D. Version 4 of the CRU TS monthly high-resolution gridded multivariate climate dataset. Sci Data, 2020, 7: 109

[18] Hartmann H, Bastos A, Das AJ, Esquivel-Muelbert A, Hammond WM, Martínez-Vilalta J, McDowell NG, Powers JS, Pugh TAM, Ruthrof KX, Allen CD. Climate change risks to global forest health: emergence of unexpected events of elevated tree mortality worldwide. Annu Rev Plant Biol, 2022, 73: 673-702

[19] Hülsmann L, Bugmann HKM, Commarmot B, Meyer P, Zimmermann S, Brang P. Does one model fit all? Patterns of beech mortality in natural forests of three European regions. Ecol Appl, 2016, 26(8): 2465-2479

[20] Hülsmann L, Bugmann H, Brang P. How to predict tree death from inventory data—lessons from a systematic assessment of European tree mortality models. Can J for Res, 2017, 47(7): 890-900

[21] Hülsmann L, Bugmann H, Cailleret M, Brang P. How to kill a tree: empirical mortality models for 18 species and their performance in a dynamic forest model. Ecol Appl, 2018, 28(2): 522-540

[22] Jeni LA, Cohn JF, De La Torre F (2013) Facing imbalanced data—recommendations for the use of performance metrics. In: 2013 humaine association conference on affective computing and intelligent interaction. Geneva, Switzerland. IEEE. https://doi.org/10.1109/acii.2013.47

[23] Kuhn M. Building predictive models in R using the caret Package. J Stat Soft, 2008

[24] Matthews BW. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta BBA Protein Struct, 1975, 405(2): 442-451

[25] McNellis BE, Smith AMS, Hudak AT, Strand EK. Tree mortality in western U.S. forests forecasted using forest inventory and random forest classification. Ecosphere, 2021, 12(3 e03419

[26] Merkl D, Hasenauer H (1998) Using neural networks to predict individual tree mortality. In: Proceedings of the int’l conference on engineering applications of neural networks, pp 10–12

[27] Naidu G, Zuva T, Sibanda EM (2023) A review of evaluation metrics in machine learning algorithms. In: Artificial intelligence application in networks and systems, Springer International Publishing, Cham, pp 15–25. https://doi.org/10.1007/978-3-031-35314-7_2

[28] Paletto A, De Meo I, Cantiani P, Ferretti F. Effects of forest management on the amount of deadwood in Mediterranean oak ecosystems. Ann for Sci, 2014, 717): 791-800

[29] Pretzsch H, Grams T, Häberle KH, Pritsch K, Bauerle T, Rötzer T. Growth and mortality of Norway spruce and European beech in monospecific and mixed-species stands under natural episodic and experimentally extended drought. Results of the KROOF throughfall exclusion experiment. Trees, 2020, 34(4): 957-970

[30] Pretzsch H, del Río M, Arcangeli C, Bielak K, Dudzinska M, Ian Forrester D, Kohnle U, Ledermann T, Matthews R, Nagel R, Ningre F, Nord-Larsen T, Szeligowski H, Biber P. Competition-based mortality and tree losses. An essential component of net primary productivity. For Ecol Manage, 2023, 544: 121204

[31] R Core Team (2021) R: a language and environment for statistical computing. R foundation for statistical computing. Vienna, Austria. https://www.R-project.org/

[32] Reis LP, de Souza AL, dos Reis PCM, Mazzei L, Soares CPB, Miquelino Eleto Torres CM, da Silva LF, Ruschel AR, Rêgo LJS, Leite HG. Estimation of mortality and survival of individual trees after harvesting wood using artificial neural networks in the Amazon rain forest. Ecol Eng, 2018, 112: 140-147

[33] Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 2015, 10(3 e0118432

[34] Salas-Eljatib C, Weiskittel AR. On studying the patterns of individual-based tree mortality in natural forests: a modelling analysis. For Ecol Manag, 2020, 475: 118369

[35] Senf C, Buras A, Zang CS, Rammig A, Seidl R. Excess forest mortality is consistently linked to drought across Europe. Nat Commun, 2020, 11: 6200

[36] Shearman TM, Varner JM, Hood SM, Cansler CA, Hiers JK. Modelling post-fire tree mortality: can random forest improve discrimination of imbalanced data?. Ecol Model, 2019, 414: 108855

[37] Shifley SR, He HS, Lischke H, Wang WJ, Jin WC, Gustafson EJ, Thompson JR, Thompson FR, Dijak WD, Yang J. The past and future of modeling forest dynamics: from growth and yield curves to forest landscape models. Landsc Ecol, 2017, 327): 1307-1325

[38] Sing T, Sander O, Beerenwinkel N, Lengauer T. ROCR: visualizing classifier performance in R. Bioinformatics, 2005, 21(20): 3940-3941

[39] Vázquez-Veloso A, Toraño Caicoya A, Bravo F, Biber P, Uhl E, Pretzsch H. Does machine learning outperform logistic regression in predicting individual tree mortality?. Ecol Inform, 2025, 88: 103140

[40] Venturas MD, Todd HN, Trugman AT, Anderegg WRL. Understanding and predicting forest mortality in the western United States using long-term forest inventory data and modeled hydraulic damage. New Phytol, 2021, 230(5): 1896-1910

[41] Vujovic ŽÐ. Classification model evaluation metrics. Int J Adv Comput Sci Appl, 2021

Funding

Universidad de Valladolid

RIGHTS & PERMISSIONS

The Author(s)