Real-time instance segmentation of tree trunks from under-canopy images in complex forest environments

Abstract

Tree trunk instance segmentation is crucial for under-canopy unmanned aerial vehicles (UAVs) to autonomously extract standing tree stem attributes. Using cameras as sensors makes these UAVs compact and lightweight, facilitating safe and flexible navigation in dense forests. However, their limited onboard computational power makes real-time, image-based tree trunk segmentation challenging, emphasizing the urgent need for lightweight and efficient segmentation models. In this study, we present RT-Trunk, a model specifically designed for real-time tree trunk instance segmentation in complex forest environments. To ensure real-time performance, we selected SparseInst as the base framework. We incorporated ConvNeXt-T as the backbone to enhance feature extraction for tree trunks, thereby improving segmentation accuracy. We further integrate the lightweight convolutional block attention module (CBAM), enabling the model to focus on tree trunk features while suppressing irrelevant information, which leads to additional gains in segmentation accuracy. To enable RT-Trunk to operate effectively under diverse complex forest environments, we constructed a comprehensive dataset for training and testing by combining self-collected data with multiple public datasets covering different locations, seasons, weather conditions, tree species, and levels of forest clutter. Compared with the other tree trunk segmentation methods, the RT-Trunk method achieved an average precision of 91.4% and the fastest inference speed of 32.9 frames per second. Overall, the proposed RT-Trunk provides superior trunk segmentation performance that balances speed and accuracy, making it a promising solution for supporting under-canopy UAVs in the autonomous extraction of standing tree stem attributes. The code for this work is available at https://github.com/NEFU-CVRG/RT-Trunk.

References

[] Bolya D, Zhou C, Xiao FY, Lee YJ (2019) YOLACT: real-time instance segmentation. In: In: 2019 IEEE/CVF international conference on computer vision (ICCV). Seoul, Korea (South). IEEE, pp 9156–9165

[] Chen SW, Nardari GV, Lee ES, Qu C, Liu X, Romero RAF, Kumar V. SLOAM: semantic lidar odometry and mapping for forest inventory. IEEE Robot Autom Lett, 2020, 5(2): 612-619.

[] Cheng TH, Wang XG, Chen SY, Zhang WQ, Zhang Q, Huang C, Zhang ZX, Liu WY (2022) Sparse instance activation for real-time instance segmentation. In: 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). New Orleans, LA, USA. IEEE, pp 4423–4432

[] Chisholm R, Rodríguez-Ronderos M, Lin F. Estimating tree diameters from an autonomous below-canopy UAV with mounted LiDAR. Remote Sens, 2021, 13(13): 2576.

[] Czuni L, Ben Alaya K (2019) Low- and high-level methods for tree segmentation. In: 2019 10th IEEE international conference on intelligent data acquisition and advanced computing systems: technology and applications (IDAACS). Metz, France. IEEE. https://doi.org/10.1109/idaacs.2019.8924248

[] Czúni L, Kürtösi A, Ben Alaya K (2018) Color based clustering for trunk segmentation. In: 2018 25th international conference on systems, signals and image processing (IWSSIP). Maribor, Slovenia. IEEE, pp 1–4

[] da Silva DQ, Dos Santos FN, Sousa AJ, Filipe V. Visible and thermal image-based trunk detection with deep learning for forestry mobile robotics. J Imaging, 2021, 7(9): 176. 8468268

[] da Silva DQ, dos Santos FN, Filipe V, Sousa AJ, Oliveira PM. Edge AI-based tree trunk detection for forestry monitoring robotics. Robotics, 2022, 11(6): 136.

[] de Paula PR, Olofsson K, Persson HJ, Lindberg E, Holmgren J. Individual tree detection and estimation of stem attributes with mobile laser scanning along boreal forest roads. ISPRS J Photogr Remote Sens, 2022, 187: 211-224.

[] Dong WB, Roy P, Isler V. Semantic mapping for orchard environments by merging two-sides reconstructions of tree rows. J Field Robot, 2020, 37(1): 97-121.

[] Fan YX, Feng ZK, Shen CY, Khan TU, Mannan A, Gao X, Chen PP, Saeed S. A trunk-based SLAM backend for smartphones with online SLAM in large-scale forest inventories. ISPRS J Photogr Remote Sens, 2020, 162: 41-49.

[] Fortin JM, Gamache O, Grondin V, Pomerleau F, Giguère P (2022) Instance segmentation for autonomous log grasping in forestry operations. In: In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS). Kyoto, Japan. IEEE, pp 6064–6071.

[] Gogoi A, Ahirwal J, Sahoo UK. Evaluation of ecosystem carbon storage in major forest types of Eastern Himalaya: Implications for carbon sink management. J Environ Manage, 2022, 302(Pt A): 113972.

[] Grondin V, Fortin JM, Pomerleau F, Giguère P. Tree detection and diameter estimation based on deep learning. Forestry (Lond), 2023, 96(2): 264-276.

[] He KM, Zhang XY, Ren SQ, Sun J (2016) Deep residual learning for image recognition. In: In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). Las Vegas, NV, USA. IEEE, pp 770–778

[] He KM, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: 2017 IEEE international conference on computer vision (ICCV). Venice, Italy. IEEE, pp 2980–2988.

[] Hyyppä E, Yu XW, Kaartinen H, Hakala T, Kukko A, Vastaranta M, Hyyppä J. Comparison of backpack, handheld, under-canopy UAV, and above-canopy UAV laser scanning for field reference data collection in boreal forests. Remote Sens, 2020, 12(20): 3327.

[] Juman MA, Wong YW, Rajkumar RK, Goh LJ. A novel tree trunk detection method for oil-palm plantation navigation. Comput Electron Agric, 2016, 128: 172-180.

[] Karjalainen K (2023) Image segmentation for forestry scenes. Electrical Engineering

[] Ke L, Ye MQ, Danelljan M, Liu YF, Tai YW, Tang CK, Yu F, Ke L, Ye MQ, Danelljan M, Liu YF, Tai YW, Tang CK, Yu F (2024) Segment anything in high quality. In: Proceedings of the 37th international conference on neural information processing systems. New Orleans, LA, USA. ACM, pp 29914–29934

[] Earthshot Labs (2022) Tree binary segmentation. Accessed at: https://www.kaggle.com/datasets/earthshot/tree-binary-segmentation

[] Lagos J, Lempiö U, Rahtu E (2023) FinnWoodlands dataset. In: Image analysis. Springer Nature Switzerland, pp 95–110. https://doi.org/10.1007/978-3-031-31435-3_7

[] Li R, Sun GD, Wang S, Tan T, Xu F. Tree trunk detection in urban scenes using a multiscale attention-based deep learning method. Ecol Inform, 2023, 77: 102215.

[] Liang XL, Kankare V, Yu XW, Hyyppä J, Holopainen M. Automated stem curve measurement using terrestrial laser scanning. IEEE Trans Geosci Remote Sens, 2014, 52(3): 1739-1748.

[] Liang XL, Kankare V, Hyyppä J, Wang YS, Kukko A, Haggrén H, Yu XW, Kaartinen H, Jaakkola A, Guan FY, Holopainen M, Vastaranta M. Terrestrial laser scanning in forest inventories. ISPRS J Photogr Remote Sens, 2016, 115: 63-77.

[] Liang XL, Yao HY, Qi HW, Wang XC. Forest in situ observations through a fully automated under-canopy unmanned aerial vehicle. Geo Spatial Inf Sci, 2024, 27(4): 983-999.

[] Liu JZ, Wang XF, Wang T. Classification of tree species and stock volume estimation in ground forest images using deep learning. Comput Electron Agric, 2019, 166: 105012.

[] Liu X, Chen SW, Nardari GV, Qu C, Cladera F, Taylor CJ, Kumar V. Challenges and opportunities for autonomous micro-UAVs in precision agriculture. IEEE Micro, 2022, 42(1): 61-68.

[] Liu Z, Mao HZ, Wu CY, Feichtenhofer C, Darrell T, Xie SN (2022) A ConvNet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA. IEEE, pp 11966–11976

[] Lu Y, Rasmussen C (2011) Tree trunk detection using contrast templates. In: 2011 18th IEEE international conference on image processing. Brussels, Belgium. IEEE, pp 1253–1256

[] Niknejad N, Bidese-Puhl R, Bao Y, Payn KG, Zheng JY. Phenotyping of architecture traits of loblolly pine trees using stereo machine vision and deep learning: Stem diameter, branch angle, and branch diameter. Comput Electron Agric, 2023, 211: 107999.

[] Prabhu A, Liu X, Spasojevic I, Wu YW, Shao YF, Ong D, Lei JZ, Green PC, Chaudhari P, Kumar V. UAVs for forestry: Metric-semantic mapping and diameter estimation with autonomous aerial robots. Mech Syst Signal Process, 2024, 208: 111050.

[] Shi LJ, Wang GY, Mo LF, Yi XM, Wu XP, Wu P. Automatic segmentation of standing trees from forest images based on deep learning. Sensors, 2022, 22(17): 6663. 9460454

[] Tremblay JF, Béland M, Gagnon R, Pomerleau F, Giguère P. Automatic three-dimensional mapping for tree diameter measurements in inventory operations. J Field Robot, 2020, 37(8): 1328-1346.

[] Wang XL, Zhang RF, Kong T, Li L, Shen CH, Wang XL, Zhang RF, Kong T, Li L, Shen CH (2020) SOLOv2. In: Proceedings of the 34th international conference on neural information processing systems. Vancouver, BC, Canada. ACM, pp 17721–17732

[] Wells LA, Chung W. Real-time computer vision for tree stem detection and tracking. Forests, 2023, 14(2): 267.

[] Woo S, Park J, Lee JY, Kweon IS (2018) CBAM: convolutional block attention module. In: Computer vision–ECCV 2018. Springer International Publishing, pp 3–19. https://doi.org/10.1007/978-3-030-01234-2_1

[] Zhou XL, Yang MX, Liu ZL, Li P, Xie BG, Peng CH. Dynamic allometric scaling of tree biomass and size. Nat Plants, 2021, 7(1): 42-49.