Computer vision-based real-time tracking and virtual simulation of construction behavior

Abstract

To improve the safety of construction workers and help workers remotely control humanoid robots in construction, this study designs and implements a computer vision-based virtual construction simulation system. For this purpose, human skeleton motion data are collected using a Kinect depth camera, and the obtained data are optimized via abnormal data elimination, smoothing, and normalization. MediaPipe extracts three-dimensional hand motion coordinates for accurate human posture tracking. Blender is used to build a virtual worker and site model, and the virtual worker motion is controlled based on the quaternion inverse kinematics algorithm while limiting the joint angle to enhance the authenticity of motion simulation. Experimental results show that the system frame rate is stable at 60 frame/s, end-to-end delay is less than 20 ms, and virtual task completion time is close to the real scene, verifying its engineering applicability. The proposed system can drive virtual workers to perform tasks and provide technical support for construction safety training.

Keywords

virtual construction / computer vision / motion control / human motion posture tracking / simulation

References

[1] ZHANG S J, TEIZER J, LEE J K, et al. Building information modeling (BIM) and safety: Automatic safety checking of construction models and schedules[J]. Automation in Construction, 2013, 29: 183-195.

[2] Health and Safety Executive. Construction statistics in Great Britain, 2021[R]. Liverpool, UK: Health and Safety Executive (HSE), 2021.

[3] Health and Safety Executive. Workplace fatal injuries in Great Britain[R]. Liverpool, UK: Health and Safety Executive (HSE), 2022.

[4] WETZEL E M, THABET W Y. The use of a BIM-based framework to support safe facility management processes[J]. Automation in Construction, 2015, 60: 12-24.

[5] KUROSU M. Human-computer interaction[C]//Advanced Interaction Modalities and Techniques:16th International Conference, HCI International 2014, Part Ⅱ. Heraklion, Crete, Greece, 2014: 1-10.

[6] JOHANSSON G. Visual perception of biological motion and a model for its analysis[J]. Perception & Psychophysics, 1973, 14(2): 201-211.

[7] LI Q, STANKOVIC J A, HANSON M A, et al. Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information[C]// 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks. Berkeley, CA, USA, 2009: 138-143.

[8] FANG Y C, DZENG R J. Accelerometer-based fall-portent detection algorithm for construction tiling operation[J]. Automation in Construction, 2017, 84: 214-230.

[9] JEBELLI H, AHN C R, STENTZ T L. Fall risk analysis of construction workers using inertial measurement units: Validating the usefulness of the postural stability metrics in construction[J]. Safety Science, 2016, 84: 161-170.

[10] BAKHSHI S, MAHOOR M H, DAVIDSON B S. Development of a body joint angle measurement system using IMU sensors[C]// 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Boston, MA, USA, 2011: 6923-6926.

[11] VALERO E, SIVANATHAN A, BOSCHÉ F, et al. Musculoskeletal disorders in construction: A review and a novel system for activity tracking with body area network[J]. Applied Ergonomics, 2016, 54: 120-130.

[12] YANG C, WANG X Y, MAO S W. RFID-based 3D human pose tracking: A subject generalization approach[J]. Digital Communications and Networks, 2022, 8(3): 278-288.

[13] ZHU Y Y, LI J H, ZHU L, et al. Bridge detection algorithm and technology based on visual enhancement[J]. Journal of Southeast University (Natural Science Edition), 2024, 54(4): 902-910. (in Chinese)

[14] DU W K, WANG H, LEI D, et al. Vibration response identification of cable based on machine vision and edge reconstruction[J]. Journal of Southeast University (Natural Science Edition), 2025, 55(5): 1467-1475. (in Chinese)

[15] ZHU L, MENG B W, HUO X J, et al. Cable force optimization of cable-stayed bridges based on the influence matrix and elite genetic algorithm[J]. Journal of Southeast University (English Edition), 2024, 40(2): 129-139.

[16] GUO J C. Real-time tracking and virtual simulation of construction behavior based on computer vision[D]. Beijing: Beijing Jiaotong University, 2024. (in Chinese)

[17] SEO J, HAN S, LEE S, et al. Computer vision techniques for construction safety and health monitoring[J]. Advanced Engineering Informatics, 2015, 29(2): 239-251.

[18] HAN S, LEE S, PEÑA-MORA F. Vision-based detection of unsafe actions of a construction worker: Case study of ladder climbing[J]. Journal of Computing in Civil Engineering, 2013, 27(6): 635-644.

[19] RAY S J, TEIZER J. Real-time construction worker posture analysis for ergonomics training[J]. Advanced Engineering Informatics, 2012, 26(2): 439-455.

[20] DZENG R J, HSUEH H H, HO C W. Automated posture assessment for construction workers[C]// 2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). Opatija, Croatia, 2017: 1027-1031.

[21] ABOBAKR A, NAHAVANDI D, ISKANDER J, et al. RGB-D human posture analysis for ergonomie studies using deep convolutional neural network[C]// 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Banff, AB, Canada, 2017: 2885-2890.

[22] OIKONOMIDIS I, KYRIAZIS N, ARGYROS A. Efficient model-based 3D tracking of hand articulations using Kinect[C]// Proceedings of the British Machine Vision Conference 2011. Dundee, UK, 2011: 1-11.

[23] KURIEN M, KIM M K, KOPSIDA M, et al. Real-time simulation of construction workers using combined human body and hand tracking for robotic construction worker system[J]. Automation in Construction, 2018, 86: 125-137.

[24] ROUGE C, SHAIKH S, OLSZEWSKA J I. HD: Efficient hand detection and tracking[C]// Proceedings of the 2016 Federated Conference on Computer Science and Information Systems. Gdansk, Poland, 2016: 291-297.

[25] GUO Z L, ZENG W Q, YU T D, et al. Vision-based finger tapping test in patients with Parkinson’s disease via spatial-temporal 3D hand pose estimation[J]. IEEE Journal of Biomedical and Health Informatics, 2022, 26(8): 3848-3859.

[26] COPA I C, RUSU D M. Innovative technology for upper limb rehabilitation using virtual reality and leap motion controller[J]. Buletin Stiintific, 2023, 28: 275.

[27] YEUNG L F, YANG Z Q, CHENG K C, et al. Effects of camera viewing angles on tracking kinematic gait patterns using Azure Kinect, Kinect v2 and Orbbec Astra Pro v2[J]. Gait & Posture, 2021, 87: 19-26.

[28] HUSSAIN M, ZHANG T L. Machine learning-based outlier detection for pipeline in-line inspection data[J]. Reliability Engineering & System Safety, 2025, 254: 110553.

[29] PAL Y. Noise cancellation using low pass FIR adaptive filter by LMS algorithm[J]. RIET-IJSET: International Journal of Science, Engineering and Technology, 2015, 2(1): 24.

[30] WU G, VAN DER HELM F C T, VEEGER H E J D J, et al. ISB recommendation on definitions of joint coordinate systems of various joints for the reporting of human joint motion: Part Ⅱ: Shoulder, elbow, wrist and hand[J]. Journal of Biomechanics, 2005, 38(5): 981-992.

Funding

The Eighth National “Ten Thousand Talents Plan for Top Young Talents” of China, the National Natural Science Foundation of China(52478117)

The Eighth National “Ten Thousand Talents Plan for Top Young Talents” of China, the National Natural Science Foundation of China(52378120)