Web-based multi-vision platform for earthwork productivity on construction sites using real-time model updating

Abstract

Earthwork productivity analysis is essential for successful construction projects. If productivity analysis results can be accessed anytime and anywhere, then project management can be performed more efficiently. To this end, this paper proposes an earthwork productivity monitoring framework via a real-time scene updating multi-vision platform. The framework consists of four main processes: 1) site-optimized database development; 2) real-time monitoring model updating; 3) multi-vision productivity monitoring; and 4) web-based monitoring platform for Internet-connected devices. The experimental results demonstrated satisfactory performance, with an average macro F1-score of 87.3% for continuous site-specific monitoring, an average accuracy of 86.2% for activity recognition, and the successful operation of multi-vision productivity monitoring through a web-based platform in real time. The findings can contribute to supporting site managers to understand real-time earthmoving operations while achieving better construction project and information management.

Graphical abstract

Keywords

online-active learning / site-customized monitoring / multi-vision monitoring / earthwork productivity analysis / web-based site monitoring platform

References

[1] Delgado J M D, Oyedele L, Ajayi A, Akanbi L, Akinade O, Bilal M, Owolabi H. Robotics and automated systems in construction: Understanding industry-specific challenges for adoption. Journal of Building Engineering, 2019, 26: 100868

[2] Pan W, Chen L, Zhan W. PESTEL analysis of construction productivity enhancement strategies: A case study of three economies. Journal of Management Engineering, 2019, 35(1): 05018013

[3] Ghodrati N, Yiu T W, Wilkinson S, Shahbazpour M. Role of management strategies in improving labor productivity in general construction projects in New Zealand: Managerial perspective. Journal of Management Engineering, 2018, 34(6): 04018035

[4] Bankvall L, Bygballe L E, Dubois A, Jahre M. Interdependence in supply chains and projects in construction. Supply Chain Management, 2010, 15(5): 385–393

[5] BarbosaFWoetzel JMischkeJRibeirinhoM JSridharM ParsonsMBertram NBrownS. Reinventing Construction: A Route to Higher Productivity. Brussels: McKinsey & Company, 2017

[6] KangS HSeo J WBaikK G. 3D-GIS based earthwork planning system for productivity improvement. In: Proceedings of the Construction Research Congress 2009: Building a Sustainable Future. Seattle, WA: ASCE, 2012,151–160

[7] Wong E, Swei O. New construction cost indices to improve highway management. Journal of Management Engineering, 2021, 37(4): 04021030

[8] Kisi K P, Mani N, Rojas E M, Foster E T. Optimal productivity in labor-intensive construction operations: Pilot study. Journal of Construction Engineering and Management, 2017, 143(3): 04016107

[9] Chen C, Zhu Z, Hammad A. Automated excavators activity recognition and productivity analysis from construction site surveillance videos. Automation in Construction, 2020, 110: 103045

[10] Wu H, Zhong B, Li H, Guo J, Wang Y. On-site construction quality inspection using blockchain and smart contracts. Journal of Management Engineering, 2021, 37(6): 04021065

[11] Wang S H. Engineering productivity and unit price assessment model. Journal of Management Engineering, 2022, 38(1): 04021076

[12] Pradhananga N, Teizer J. Automatic spatio-temporal analysis of construction site equipment operations using GPS data. Automation in Construction, 2013, 29: 107–122

[13] Lu W, Huang G Q, Li H. Scenarios for applying RFID technology in construction project management. Automation in Construction, 2011, 20(2): 101–106

[14] Zhang C, Shen W, Ye Z. Technical feasibility analysis on applying ultra-wide band technology in construction progress monitoring. International Journal of Construction Management, 2022, 22(15): 2951–2965

[15] Gu Y, Ai Q, Xu Z, Yao L, Wang H, Huang X, Yuan Y. Cost-effective image recognition of water leakage in metro tunnels using self-supervised learning. Automation in Construction, 2024, 167: 105678

[16] Ai Q, Yuan Y, Bi X. Acquiring sectional profile of metro tunnels using charge-coupled device cameras. Structure and Infrastructure Engineering, 2016, 12(9): 1065–1075

[17] Ai Q, Yuan Y. Rapid acquisition and identification of structural defects of metro tunnel. Sensors, 2019, 19(19): 4278

[18] Kim J, Hwang J, Jeong I, Chi S, Seo J O, Kim J. Generalized vision-based framework for construction productivity analysis using a standard classification system. Automation in Construction, 2024, 165: 105504

[19] Kim H, Kim H, Hong Y W, Byun H. Detecting construction equipment using a region-based fully convolutional network and transfer learning. Journal of Computing in Civil Engineering, 2018, 32(2): 04017082

[20] Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149

[21] HeKZhangX RenSSunJ. Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2016. New York, NY: IEEE, 2016, 770–778

[22] Li Y, Lu Y, Chen J. A deep learning approach for real-time rebar counting on the construction site based on YOLOv3 detector. Automation in Construction, 2021, 124: 103602

[23] Shin Y, Choi Y, Won J, Hong T, Koo C. A new benchmark model for the automated detection and classification of a wide range of heavy construction equipment. Journal of Management Engineering, 2024, 40(2): 04023069

[24] Nath N D, Behzadan A H. Deep convolutional networks for construction object detection under different visual conditions. Frontiers in Built Environment, 2020, 6: 97

[25] Zhu Z, Ren X, Chen Z. Integrated detection and tracking of workforce and equipment from construction jobsite videos. Automation in Construction, 2017, 81: 161–171

[26] Xiao B, Zhu Z. Two-dimensional visual tracking in construction scenarios: A comparative study. Journal of Computing in Civil Engineering, 2018, 32(3): 04018006

[27] Jeong I, Hwang J, Kim J, Chi S, Hwang B G, Kim J. Vision-based productivity monitoring of tower crane operations during curtain wall installation using a database-free approach. Journal of Computing in Civil Engineering, 2023, 37(4): 04023015

[28] Xiao B, Kang S C. Vision-based method integrating deep learning detection for tracking multiple construction machines. Journal of Computing in Civil Engineering, 2021, 35(2): 04020071

[29] Angah O, Chen A Y. Tracking multiple construction workers through deep learning and the gradient based method with re-matching based on multi-object tracking accuracy. Automation in Construction, 2020, 119: 103308

[30] Yan X, Zhang H, Gao H. Mutually coupled detection and tracking of trucks for monitoring construction material arrival delays. Automation in Construction, 2022, 142: 104491

[31] Luo H, Xiong C, Fang W, Love P E D, Zhang B, Ouyang X. Convolutional neural networks: Computer vision-based workforce activity assessment in construction. Automation in Construction, 2018, 94: 282–289

[32] Kim J, Chi S. Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles. Automation in Construction, 2019, 104: 255–264

[33] Zhang J, Zi L, Hou Y, Wang M, Jiang W, Deng D. A deep learning-based approach to enable action recognition for construction equipment. Advances in Civil Engineering, 2020, 2020(1): 8812928

[34] Luo X, Li H, Cai D, Dai F, Seo H O, Lee S H. Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks. Journal of Computing in Civil Engineering, 2018, 32(3): 04018012

[35] Roberts D, Golparvar-Fard M. End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level. Automation in Construction, 2019, 105: 102811

[36] Soltani M M, Zhu Z, Hammad A. Framework for location data fusion and pose estimation of excavators using stereo vision. Journal of Computing in Civil Engineering, 2018, 32(6): 04018045

[37] Cheng J C P, Wong P K Y, Luo H, Wang M, Leung P H. Vision-based monitoring of site safety compliance based on worker re-identification and personal protective equipment classification. Automation in Construction, 2022, 139: 104312

[38] Zhang Q, Wang Z, Yang B, Lei K, Zhang B, Liu B. Reidentification-based automated matching for 3D localization of workers in construction sites. Journal of Computing in Civil Engineering, 2021, 35(6): 04021019

[39] Wei R, Love P E D, Fang W, Luo H, Xu S. Recognizing people’s identity in construction sites with computer vision: A spatial and temporal attention pooling network. Advanced Engineering Informatics, 2019, 42: 100981

[40] Kim H, Bang S, Jeong H, Ham Y, Kim H. Analyzing context and productivity of tunnel earthmoving processes using imaging and simulation. Automation in Construction, 2018, 92: 188–198

[41] Kim H, Ham Y, Kim W, Park S, Kim H. Vision-based nonintrusive context documentation for earthmoving productivity simulation. Automation in Construction, 2019, 102: 135–147

[42] Cheng M Y, Cao M T, Nuralim C K. Computer vision-based deep learning for supervising excavator operations and measuring real-time earthwork productivity. Journal of Supercomputing, 2023, 79(4): 4468–4492

[43] Bügler M, Borrmann A, Ogunmakin G, Vela P A, Teizer J. Fusion of photogrammetry and video analysis for productivity assessment of earthwork processes. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(2): 107–123

[44] Chen C, Xiao B, Zhang Y, Zhu Z. Automatic vision-based calculation of excavator earthmoving productivity using zero-shot learning activity recognition. Automation in Construction, 2023, 146: 104702

[45] Hwang J, Kim J, Chi S, Seo J O. Development of training image database using web crawling for vision-based site monitoring. Automation in Construction, 2022, 135: 104141

[46] Lee J G, Hwang J, Chi S, Seo J. Synthetic image dataset development for vision-based construction equipment detection. Journal of Computing in Civil Engineering, 2022, 36(5): 04022020

[47] Hwang J, Kim J, Chi S. Site-optimized training image database development using web-crawled and synthetic images. Automation in Construction, 2023, 151: 104886

[48] Kim J, Hwang J, Chi S, Seo J O. Towards database-free vision-based monitoring on construction sites: A deep active learning approach. Automation in Construction, 2020, 120: 103376

[49] Meng D, Yang S, de Jesus A M P, Zhu S P. A novel Kriging-model-assisted reliability-based multidisciplinary design optimization strategy and its application in the offshore wind turbine tower. Renewable Energy, 2023, 203: 407–420

[50] Meng D, Yang S, de Jesus A M P, Fazeres-Ferradosa T, Zhu S P. A novel hybrid adaptive Kriging and water cycle algorithm for reliability-based design and optimization strategy: Application in offshore wind turbine monopile. Computer Methods in Applied Mechanics and Engineering, 2023, 412: 116083

[51] Meng D, Yang H, Yang S, Zhang Y, de Jesus A M P, Correia J, Fazeres-Ferradosa T, Macek W, Branco R, Zhu S P. Kriging-assisted hybrid reliability design and optimization of offshore wind turbine support structure based on a portfolio allocation strategy. Ocean Engineering, 2024, 295: 116842

[52] Yang S, Meng D, Wang H, Yang C. A novel learning function for adaptive surrogate-model-based reliability evaluation. Philosophical Transactions of the Royal Society A: Mathematical, Physical, and Engineering Sciences, 2024, 382(2264): 20220395

[53] Kim J, Chi S. Adaptive detector and tracker on construction sites using functional integration and online learning. Journal of Computing in Civil Engineering, 2017, 31(5): 04017026

[54] RedmonJDivvala SGirshickRFarhadiA. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, NY: IEEE, 2016, 779–788

[55] Park C, Chun H, Chi S. Multi-camera people counting using a queue-buffer algorithm for effective search and rescue in building disasters. KSCE Journal of Civil Engineering, 2024, 28(6): 2132–2146

[56] FoggI. Benchmarking the Global 5G Experience. Opensignal Limited 2023 June. 2023

[57] OccupationalSafetyHealthAdministration Standard 1926. Safety and Health Regulations for Construction. Washington, D.C.: OSHA, 1926

RIGHTS & PERMISSIONS

The Author(s). This article is published with open access at link.springer.com and journal.hep.com.cn