Advancing Acer phenology monitoring: fine-grained identification and analysis by deep learning RESformer

Abstract

Climate change is a global phenomenon that has profound impacts on ecological dynamics and biodiversity, shaping the interactions between species and their environment. To gain a deeper understanding of the mechanisms driving climate change, phenological monitoring is essential. Traditional methods of defining phenological phases often rely on fixed thresholds. However, with the development of technology, deep learning-based classification models are now able to more accurately delineate phenological phases from images, enabling phenological monitoring. Despite the significant advancements these models have made in phenological monitoring, they still face challenges in fully capturing the complexity of biotic-environmental interactions, which can limit the fine-grained accuracy of phenological phase identification. To address this, we propose a novel deep learning model, RESformer, designed to monitor tree phenology at a fine-grained level using PhenoCam images. RESformer features a lightweight structure, making it suitable for deployment in resource-constrained environments. It incorporates a dual-branch routing mechanism that considers both global and local information, thereby improving the accuracy of phenological monitoring. To validate the effectiveness of RESformer, we conducted a case study involving 82,118 images taken over two years from four different locations in Wisconsin, focusing on the phenology of Acer. The images were classified into seven distinct phenological stages, with RESformer achieving an overall monitoring accuracy of 96.02%. Furthermore, we compared RESformer with a phenological monitoring approach based on the Green Chromatic Coordinate (GCC) index and ten popular classification models. The results showed that RESformer excelled in fine-grained monitoring, effectively capturing and identifying changes in phenological stages. This finding not only provides strong support for monitoring the phenology of Acer species but also offers valuable insights for understanding ecological trends and developing more effective ecosystem conservation and management strategies.

Keywords

Fine-grained phenological period / Acer phenological monitoring / Green chromatic coordinate / PhenoCam

References

[1] BaumannM, OzdoganM, RichardsonAD, RadeloffVC. Phenology from landsat when data is scarce: using MODIS and dynamic time-warping to combine multi-year landsat imagery to derive annual phenology curves. Int J Appl Earth Obs Geoinf, 2017, 54: 72-83

[2] Chavana-BryantC, MalhiY, WuJ, AsnerGP, AnastasiouA, EnquistBJ, Cosio CaravasiEG, DoughtyCE, SaleskaSR, MartinRE, GerardFF. Leaf aging of Amazonian canopy trees as revealed by spectral and physiochemical measurements. New Phytol, 2017, 214(3): 1049-1063

[3] ChenJC, MaBX, JiC, ZhangJ, FengQC, LiuX, LiYJ. Apple inflorescence recognition of phenology stage in complex background based on improved YOLOv7. Comput Electron Agric, 2023, 211 108048

[4] DeyB, AhmedR, FerdousJ, HaqueMMU, KhatunR, HasanFE, UddinSN. Automated plant species identification from the stomata images using deep neural network: a study of selected mangrove and freshwater swamp forest tree species of Bangladesh. Ecol Inform, 2023, 75 102128

[5] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2020) An image is worth 16×16 Words: transformers for image recognition at scale. ArXiv, abs/2010.11929. https://doi.org/10.48550/arXiv.2010.11929

[6] FisherJI, MustardJF, VadeboncoeurMA. Green leaf phenology at landsat resolution: scaling from the field to the satellite. Remote Sens Environ, 2006, 100(2): 265-279

[7] GrünigM, RazaviE, CalancaP, MazziD, WegnerJD, PellissierL. Applying deep neural networks to predict incidence and phenology of plant pests and diseases. Ecosphere, 2021, 12(10): e03791

[8] HufkensK, FriedlMA, KeenanTF, SonnentagO, BaileyA, O’KeefeJ, RichardsonAD. Ecological impacts of a widespread frost event following early spring leaf-out. Glob Change Biol, 2012, 18(7): 2365-2377

[9] KeenanTF, GrayJ, FriedlMA, ToomeyM, BohrerG, HollingerDY, MungerJW, O’KeefeJ, SchmidHP, WingIS, YangB, RichardsonAD. Net carbon uptake has increased through warming-induced changes in temperate forest phenology. Nat Clim Change, 2014, 4(7): 598-604

[10] KlostermanST, HufkensK, GrayJM, MelaasE, SonnentagO, LavineI, MitchellL, NormanR, FriedlMA, RichardsonAD. Evaluating remote sensing of deciduous forest phenology at multiple spatial scales using PhenoCam imagery. Biogeosciences, 2014, 11(16): 4305-4320

[11] LiH, HuBX, LiQ, JingLH. CNN-based individual tree species classification using high-resolution satellite imagery and airborne LiDAR data. Forests, 2021, 12(12): 1697

[12] LiangXL, KukkoA, BalenovićI, SaarinenN, JunttilaS, KankareV, HolopainenM, MokrošM, SurovýP, KaartinenH, JurjevićL, HonkavaaraE, NäsiR, LiuJB, HollausM, TianJJ, YuXW, PanJ, CaiSS, VirtanenJP, WangYS, HyyppäJ. Close-range remote sensing of forests: the state of the art, challenges, and opportunities for systems and data acquisitions. IEEE Geosci Remote Sens Mag, 2022, 10(3): 32-71

[13] LiuQ, FuYH, ZhuZC, LiuYW, LiuZ, HuangMT, JanssensIA, PiaoSL. Delayed autumn phenology in the Northern Hemisphere is related to change in both climate and spring phenology. Glob Chang Biol, 2016, 22(11): 3702-3711

[14] Liu Z, Lin YT, Cao Y, Hu H, Wei YX, Zhang Z, Lin S, Guo BN (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF international conference on computer vision (ICCV). Montreal, QC, Canada. IEEE, pp 9992–10002

[15] LopesAP, NelsonBW, WuJ, de Alencastro GraçaPML, TavaresJV, ProhaskaN, MartinsGA, SaleskaSR. Leaf flush drives dry season green-up of the central amazon. Remote Sens Environ, 2016, 182: 90-98

[16] MagidiJ, AhmedF. Monitoring vegetation phenology using MODIS NDVI 250m in the city of Tshwane, South Africa. SA J Geomat, 2022, 11(2): 176-189

[17] Milliman T, Hufkens K, Richardson AD, Aubrecht DM, Chen M, Gray JM, Johnston MR, Keenan TF, Klosterman ST, Kosmala M, Melaas EK, Friedl MA, Frolking S (2018) PhenoCam dataset v1.0: digital camera imagery from the PhenoCam Network, 2000–2015. ORNL Distributed Active Archive Center. https://doi.org/10.3334/ORNLDAAC/1511

[18] MisraG, CawkwellF, WinglerA. Status of phenological research using sentinel-2 data: a review. Remote Sens, 2020, 12(17): 2760

[19] NeilK, WuJG. Effects of urbanization on plant flowering phenology: a review. Urban Ecosyst, 2006, 9(3): 243-257

[20] Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin ZM, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai JJ, Chintala S (2019) PyTorch: an imperative style, high-performance deep learning library. 1912.01703|. https://arxiv.org/abs/1912.01703v1

[21] RichardsonAD. PhenoCam: an evolving, open-source tool to study the temporal and spatial variability of ecosystem-scale phenology. Agric for Meteor, 2023, 342 109751

[22] RichardsonAD, BraswellBH, HollingerDY, JenkinsJP, OllingerSV. Near-surface remote sensing of spatial and temporal variation in canopy phenology. Ecol Appl, 2009, 19(6): 1417-1428

[23] RichardsonAD, HufkensK, MillimanT, AubrechtDM, ChenM, GrayJM, JohnstonMR, KeenanTF, KlostermanST, KosmalaM, MelaasEK, FriedlMA, FrolkingS. Tracking vegetation phenology across diverse North American biomes using PhenoCam imagery. Sci Data, 2018, 5 180028

[24] Sandler M, Howard A, Zhu ML, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on computer vision and pattern recognition. Salt Lake City, UT, USA. IEEE, pp 4510–4520

[25] SongGQ, WuSB, LeeCKF, SerbinSP, WolfeBT, NgMK, ElyKS, BogonovichM, WangJ, LinZY, SaleskaS, NelsonBW, RogersA, WuJ. Monitoring leaf phenology in moist tropical forests by applying a superpixel-based deep learning method to time-series images of tree canopies. ISPRS J Photogramm Remote Sens, 2022, 183: 19-33

[26] ThapaS, Garcia MillanVE, EklundhL. Assessing forest phenology: a multi-scale comparison of near-surface (UAV, spectral reflectance sensor, PhenoCam) and satellite (MODIS, sentinel-2) remote sensing. Remote Sens, 2021, 13(8): 1597

[27] Veropoulos K, Campbell C, Cristianini N (1999) Controlling the sensitivity of support vector machines. In: Proceedings of the international joint conference on AI, vol 55, p 60. https://doi.org/10.5555/1626199.1626259

[28] VisserME, BothC. Shifts in phenology due to global climate change: the need for a yardstick. Proc Biol Sci, 2005, 272(1581): 2561-2569

[29] WangX, TangJL, WhittyM. DeepPhenology: estimation of apple flower phenology distributions based on deep learning. Comput Electron Agric, 2021, 185 106123

[30] WuJ, KobayashiH, StarkSC, MengR, GuanKY, TranNN, GaoSC, YangW, Restrepo-CoupeN, MiuraT, OlivieraRC, RogersA, DyeDG, NelsonBW, SerbinSP, HueteAR, SaleskaSR. Biological processes dominate seasonality of remotely sensed canopy greenness in an Amazon evergreen forest. New Phytol, 2018, 217(4): 1507-1520

[31] ZanXL, ZhangXL, XingZY, LiuW, ZhangXD, SuW, LiuZ, ZhaoYY, LiSM. Automatic detection of maize tassels from UAV images by combining random forest classifier and VGG16. Remote Sens, 2020, 12(18): 3049

[32] ZengLL, WardlowBD, XiangDX, HuS, LiDR. A review of vegetation phenological metrics extraction using time-series, multispectral satellite data. Remote Sens Environ, 2020, 237 111511

[33] ZhangXY, FriedlMA, SchaafCB, StrahlerAH, HodgesJCF, GaoF, ReedBC, HueteA. Monitoring vegetation phenology using MODIS. Remote Sens Environ, 2003, 84(3): 471-475

RIGHTS & PERMISSIONS

Northeast Forestry University