Research on concrete structure defect repair based on three-dimensional printing

Abstract

Quality assurance and maintenance play a crucial role in engineering construction, as they have a significant impact on project safety. One common issue in concrete structures is the presence of defects. To enhance the automation level of concrete defect repairs, this study proposes a computer vision-based robotic system, which is based on three-dimensional (3D) printing technology to repair defects. This system integrates multiple sensors such as light detection and ranging (LiDAR) and camera. LiDAR is utilized to model concrete pipelines and obtain geometric parameters regarding their appearance. Additionally, a convolutional neural network (CNN) is employed with a depth camera to locate defects in concrete structures. Furthermore, a method for coordinate transformation is presented to convert the obtained coordinates into executable ones for a robotic arm. Finally, the feasibility of this concrete defect repair method is validated through simulation and experiments.

Graphical abstract

Keywords

concrete / defect detection / 3D printing / deep learning / point cloud data

1 Introduction

Concrete is a common component in the construction of infrastructure such as tunnels and bridges. Due to harsh environmental conditions and overloading, concrete structures deteriorate over time, and the appearance of defects is an early sign of concrete degradation. Concrete defects are more apparent in high-strength concrete structures, negatively impacting their stability and integrity, thereby compromising the safety and usability of the structure [1–4]. Therefore, concrete defects pose a challenge for engineers and researchers, making it necessary to explore automated methods for identification and assessment of damage [5].

Traditional methods for detecting defects in concrete structures rely on manual visual inspections typically conducted by trained operators. These inspections rely heavily on the subjective judgment and experiential knowledge of the operators, making them time-consuming and labor-intensive. The inspection results are prone to inaccuracies, errors, and omissions due to the subjectivity involved, and this can be particularly problematic in large-scale civil engineering projects [6]. In recent years, researchers have turned their attention to image processing methods driven by advancements in computer vision technology. A study pioneered the use of vision-based convolutional neural networks (CNNs) to detect concrete cracks. The method is independent of image quality, camera specifications and working distance, and has very strong robustness for detection of all kinds of surface damages in concrete and steel structures, making it a reliable method for monitoring structural health [7]. Automated inspections offer advantages such as high efficiency, low cost, and reduced potential for accidents. Since automated inspections are guided by computer algorithms, their results are objective and accurate [8–10].

Current research on the repair of concrete structure defects primarily focuses on material studies. To achieve more efficient repair of concrete structure defects, it is necessary to incorporate robotic arms into the repair process. The robotic arm has high precision and consistency, which can improve the working efficiency and adapt to the repair needs of different environments. In engineering, robotic systems are commonly used for assessing and accepting construction quality, as well as inspecting the quality of road surfaces and bridges, offering advantages in terms of safety and efficiency [11–14]. However, these robotic systems often require pre-programming and remote operation. This paper proposes a computer vision-based automated robotic system that utilizes multiple sensors such as radar and cameras for the detection and repair of concrete structure defects, thereby enhancing the automation and intelligence of the detection and repair processes.

2 Related research

2.1 Concrete defect detection

Traditional image processing techniques have been utilized for rapid scanning of concrete structure surfaces and numerous methods for automatic or semi-automatic detection of concrete defects, based on these techniques, have been proposed. Feng and Feng [15] summarized the collective experience gained from the latest developments and validations of vision-based sensors used for structural dynamic response measurement and structural health monitoring (SHM). Davoudi et al. [16] established the connection between surface observations and quantitative estimation of structural component load ratings, using machine vision, thereby developing an estimation model for quantifying internal load ratings (shear and bending moments). Noh et al. [17] applied filtering and morphological operations to image data, distinguishing candidate cracks from the rest of the image using segmentation, and they also employed three masks to filter out noise and clutter. Dinh et al. [18] extracted regions of interest from the background by processing histogram thresholds and employed line enhancement and moving average filters to remove noise from concrete surface images obtained by inspection robots. Cha et al. [19] proposed an improved cascaded feature-based method for crack detection on concrete surfaces. These methods are based on common image processing techniques such as filtering, histogram transformations, edge and boundary detection, and texture recognition [20]. While image processing methods are more effective compared to manual visual inspection, their detection performance primarily relies on the quality of manually extracted features and is limited by domain-specific knowledge. Furthermore, they heavily depend on complex classifiers and preprocessing procedures, lacking the ability to distinguish cracks from complex backgrounds in images where features are not prominent. This leads to lower detection efficiency and unsuitability for images with significant variations.

To overcome the limitations of traditional image processing techniques, deep learning has been employed for automatic detection of concrete structure defects [21–23]. Deep CNNs (DCNNs), providing one of the most effective supervised learning methods, can be trained for tasks such as image classification and object detection. These models, with multiple processing layers, automatically capture the complex structures of large-scale data and exhibit strong feature extraction and generalization capabilities [24–27]. Several recent studies have developed deep learning-based algorithms for detecting concrete structure defects, particularly in road surfaces and bridges. Gopalakrishnan et al. [28] utilized a DCNN trained on the “big data” ImageNet database, employing classifiers on composite images of hot-mix asphalt surfaces and Portland cement concrete surfaces with different surface features. Song et al. [29] employed structured light detection and quantification of concrete surface cracks using visual and two laser sensors, calculating the size of the detected cracks projected on the surface based on the intensity of the laser beam at their locations. Deep learning methods are effective for image classification, capable of distinguishing and classifying images belonging to multiple categories, and suitable for large-scale object detection problems with lower computational costs.

There are some advanced pixel level segmentation methods of deep learning for segmentation with state-of-the-art performances with fast processing of large images (1200 × 800) at more than 40 frames per second. Beckman et al. [30] proposed a faster region-based CNN (Faster R-CNN)-based concrete spalling damage detection method and performed a plane fitting implementation using the random sample consensus (RANSAC) algorithm. Lewis et al. [31] proposed a dual encoder–decoder solution named polyp segmentation network to tackle the issues associated with boundary pixel isolation and improvement, modeling of global dependencies and context, as well as model generalization issues in existing networks. Kang and Cha [32] developed a novel semantic transformer representation network for crack segmentation at the pixel level in complex scenes in a real-time manner and proposed a method for evaluating the level of complexity of image scenes. These methods used advanced operators such as depthwise separable convolution, atrous spatial pyramid pooling, and attention mechanisms, to improve the performance of pixel level segmentation with limited data.

Various sensors, in addition to cameras, have been employed in the detection of concrete structure defects. Leung et al. [33] developed a novel fiber optic sensor for crack detection and subsequent monitoring of crack opening, demonstrating the potential of this innovative sensing technology in practical applications. Ai et al. [34] conducted numerical and experimental studies to explain the influence of heating time on the use of surface-bonded piezoelectric (PZT) sensors for monitoring concrete structures. The findings of this research prompted the incorporation of a time factor when evaluating the impact of temperature on PZT-based concrete structure monitoring. Liu et al. [35] designed and implemented a durability monitoring system for high-pile wharf structures. This system utilizes anodic gradient sensors as a means of monitoring the corrosion status of steel in the wharf.

2.2 Concrete defect repair technology

Currently, the focus of concrete defect repair techniques lies in the study of the materials. Kim and Park [36] utilized epoxy resin, impregnation method, and epoxy resin/impregnation method for repairing concrete cracks, with epoxy resin used for crack injection and a topcoat applied for surface protection. Perry et al. [37] employed moisture, chloride, and damage monitoring to display conductivity data, using these novel self-sensing materials to measure the environmental metrics responsible for the deterioration of a variety of concrete structures. Martuscelli et al. [38] developed an approach to analysis of the potential use of fungi as bio-remediators in concrete structures. Fungi possess greater biomass and have mycelia that contribute to the mechanical behavior of bio-concrete. Kumar et al. [39] proposed repair and restoration strategies for reinforced concrete structures through a case study. The primary focus was on determining and correlating the results of two non-destructive testing sessions, with the aim of proposing appropriate sustainable measures for structural repair. Chen et al. [40] employed electrochemical methods for concrete repair and established a new numerical model for chloride ion migration and diffusion, simulating the distribution of free chloride ions in concrete during both the free chloride ion diffusion stage and the electrochemical repair stage. Sun and Miao [41] applied bacillus megaterium and studied calcification precipitation tests via varying experimental conditions. Through literature review, it is evident that current research on concrete defect repair techniques is predominantly focused on materials, requiring manual application of the corresponding repair materials and methods to concrete defect locations by operators, often in hazardous operational environments, leading to a relatively low level of automation [42–45].

To enhance the repair process of concrete structural defects and address issues such as low efficiency, high labor costs, and significant risks associated with the repair process, there have been several studies on autonomous control with feedback for SHM to enhance the automation of defect repair. Kang and Cha [46] proposed an autonomous unmanned aerial vehicle (UAV) method using ultrasonic beacons to replace the role of global positioning system (GPS) and used an autonomous UAV to detect concrete cracks when GPS was not available. Waqas et al. [47] proposed a framework for obstacle-avoiding autonomous UAV through fiducial marker-based UAV localization, and showed excellent performances in obstacle avoidance. Ali et al. [48] proposed an autonomous UAV system integrated with a modified Faster R-CNN for real-time damage mapping in a GPS-denied environment.

This paper proposes a computer vision-based semi-automated robotic system. This system is designed to detecting and repairing concrete structural defects, and it can be remotely operated by an operator.

3 Methodology

Our system realizes automatic repair of concrete defects. It can automatically detect concrete defects, and the robotic arm then automatically moves to the defect location and repairs it. The main steps involved in using a robotic arm for large-scale surface reinforcement and localized defect repair can be summarized as follows. The first step involves surface modeling. Our approach utilizes a combination of light detection and ranging (LiDAR) and inertial measurement unit (IMU) to model the surface. During this process, point cloud data of the tunnel is collected, and the equation representing the working surface is computed. This equation serves to generate the motion trajectory for the robotic arm during large-scale surface reinforcement and to calculate the robotic arm’s pose for localized defect repair. The second step focuses on localized defect repair. Our method involves training a YOLOv7 [49] framework using concrete defect images and acquiring depth images of concrete defects using a depth camera. By aligning the RGB images with the depth images, the three-dimensional (3D) positional coordinates of the localized defects are determined in the camera coordinate system. These coordinates are then transformed to the LiDAR coordinate system to calculate the robotic arm’s pose. The third step is to repair the defect and evaluate the repair. Our method is to send the defect location coordinates to the robotic arm through human-computer interaction, complete the repair of local defects, and evaluate the repair effectiveness from the appearance, concrete material strength, and concrete mechanical properties, as shown in Fig.1.

3.1 Modeling of tunnel cylindrical surfaces using light detection and ranging and inertial measurement unit

Modeling the tunnel cylindrical surface involves collecting point cloud data, extracting relevant information, and calculating the equation of the cylindrical surface. First, we utilize the simultaneous localization and mapping (SLAM)-based method to collect the point cloud data [50]. SLAM is a technology for self-localization and map creation in unknown environments. This technology is essential for collecting point cloud data, which is a 3D data structure used to represent the physical world. SLAM technology enables robots to map the point cloud of their environment in real time. We adopt the initial position of the LiDAR coordinate system as the world coordinate system for the entire point cloud. In the subsequent process, we transform each frame of the point cloud to the world coordinate system based on the SLAM-calculated poses, thereby obtaining the complete point cloud of the tunnel. Specifically, the LiDAR is fixed to the base of the robotic arm. Due to the use of a LiDAR based on hybrid solid-state technology, we can achieve a good angular resolution of the point cloud without rotating the LiDAR.

The crucial step in separating essential data is eliminating irrelevant data through program processing. Our approach involves using passthrough filtering for preprocessing the point cloud data. We eliminate irrelevant point cloud data based on the range of motion of the robotic arm and the approximate position of the target reinforced surface relative to the robotic arm. For example, if the maximum distance from the working surface to the LiDAR is known to be 5 m, we set the maximum value for passthrough filtering in the x-direction as 5 m. After passthrough filtering, the proportion of significant data in the point cloud increases significantly. However, the filtered point cloud data still contains some noise. This noise can complicate the calculation of local point cloud geometry features and even lead to errors, resulting in fitting failure. To address this, we employ outlier filtering, which calculates the average distance from each point to its neighboring points. We perform statistical analysis on the calculation results, assuming that they should follow a Gaussian distribution with mean and standard deviation. Points with average distances outside the defined range of mean and standard deviation are considered noise and are removed from the point cloud.

We utilize the RANSAC algorithm [51] to fit the tunnel cylinder model. RANSAC randomly samples data from the point cloud to estimate the model and assigns points that fit the model to the inlier category, while points that do not fit the model are classified as outliers. This process eventually yields a set of optimal model parameters. For the segmented tunnel cylinder model, we perform nonlinear optimization on the parameters of the cylindrical surface [52]. Our approach involves adding each location in the point cloud to a residual block to the optimization problem. The function of each residual block is to calculate the error of each point under the current cylindrical parameters, specifically the distance from the point to the cylinder surface. The cylinder parameters are a 7-dimensional vector that includes the XYZ coordinates of a point on the cylinder axis, the direction vector of the cylinder axis, and the cylinder radius. We further optimize the cylindrical surface parameter equation (Eq. (1)) using Ceres Solver on the segmented results. The coordinates of the center point of the cylinder are (x₀, y₀, z₀), and the parameters a, b, and c denote the direction vectors of the cylinder, which together describe the direction and orientation of the cylinder. The radius of the cylinder r₀ fits each parameter of the optimal cylindrical equation by minimizing the distance from each point to the existing cylindrical equation. Based on the cylindrical surface parameter equation, we down-sample the point cloud with an angular resolution of 0.25π, an inter-point spacing of 0.3 m, and an inflation radius of 0.3 m and these values can be adjusted according to the different working conditions

(1)\begin{array}{rl}& \underset{x_0,y_0,{\mathit{\text{z}}}_0,a,b,c,r_0}{\arg min}\sum _{s=0}^{s=N}{(x_s-x_0)}^2+{(y_s-y_0)}^2+{({\mathit{\text{z}}}_s-{\mathit{\text{z}}}_0)}^2\\ & -[a(x_s-x_0)+b(y_s-y_0)+c({\mathit{\text{z}}}_s-{\mathit{\text{z}}}_0){]}^2-r_0^2.\end{array}

Based on the parameter equation of the cylindrical surface, we generate down-sampled point clouds in a certain proportion and transform the point clouds into the base coordinate system of the robotic arm for subsequent trajectory generation. We fix the LiDAR on a parallel plane of the robotic arm base and attach a sharp point at the end of the robotic arm to calibrate the user coordinate system of the robotic arm. We set the LiDAR coordinate system as the calibrated user coordinate system of the robotic arm and convert the coordinates obtained in the LiDAR coordinate system to the Cartesian coordinates of the robotic arm through data processing. By means of human-computer interaction, we transmit the coordinates to the robotic arm to control its movement to the corresponding position. The whole workflow is shown in Fig.2.

3.2 Repair of local defects

For the repair of local defects, we propose a method based on depth camera and LiDAR. First, we identify and locate concrete defects. Our method is based on the YOLOv7 framework. Object detection methods based on deep learning have better robustness than traditional visual methods, and they can still meet the needs of defect recognition under strong lighting and rainy conditions. We capture images of some concrete defects and add similar concrete defects from publicly available data sets to create our data set. We train YOLOv7 and use its detection results for subsequent processing. We align the RGB image and the depth map, using the recognition results from the RGB image. We extract the corresponding depth information from the depth map to obtain the 3D coordinates of the defect in the camera coordinate system. We transform the coordinates of the object from the camera coordinate system to the tool coordinate system and then, based on the point cloud model built in the LiDAR coordinate system, we read the corresponding R_Z,R_Y,R_X values. Finally, we send the \left(x,y,\mathit{\text{z}},R_Z,R_Y,R_X\right) coordinates of the target point to the robotic arm to control its movement to the defect location.

1) Data set preparation: To create a concrete defect data set, we artificially create some concrete defects and capture images of them. We also incorporate similar defects from publicly available concrete defect data sets [53], as shown in Fig.3. To augment the data set, the framework applies mosaic data augmentation and mixup augmentation to some images in the data set. Mosaic augmentation combines four images and feeds them into the CNN for training, while mixup augmentation blends two images at a one-to-one ratio. The data set is allocated to 67% training set, 7% validation set, and 25% test set, as shown in Tab.1. In the data set, the images are of different sizes but are resized to [640,640] during the training process.

2) CNN model training: Since training a neural network model from random initialization requires a large amount of data and a long training time to achieve good training results, we use a pre-trained CNN model on the Microsoft COCO data set [54] and perform fine-tuning with new data. In this study, we use a deep learning model based on YOLOv7 for object detection. We train the model for a total of 300 epochs. During the first 50 epochs, we freeze the backbone of the model, and then we unfreeze the entire model for training to improve training speed and effectiveness. The maximum learning rate of the model is set to 1 × 10⁻², the minimum learning rate is set to 1 × 10⁻⁴, and the learning rate decay follows a cosine schedule. We use stochastic gradient descent as the optimizer with a weight decay of 5 × 10⁻⁴ to prevent overfitting.

3) Evaluation: We use a 10-fold cross validation approach to assess the quality of the model. The model we train achieves an average precision of 90.88% on the data set, demonstrating good recognition capability for concrete defects under conditions such as strong lighting and dust coverage.

4) Calculation of robotic arm motion trajectory: To control the robotic arm to move to the defect location, it is necessary to calculate the target pose of the robotic arm and convert the coordinates obtained from the depth camera and LiDAR to the defect location coordinates in the robotic arm coordinate system. First, we transform the coordinates obtained by the depth camera in the pixel coordinate system to obtain the relative distance \left(x_{\mathrm{c}},y_{\mathrm{c}},{\mathit{\text{z}}}_{\mathrm{c}}\right) between the depth camera and the defect location. We calibrate the tool coordinate system to be the end effector of the robotic arm and measure the distance between the depth camera and the robotic arm end effector to calculate the relative distance \left(\mathrm{\Delta }{x}_{\mathrm{t}},\mathrm{\Delta }{y}_{\mathrm{t}},\mathrm{\Delta }{\mathit{\text{z}}}_{\mathrm{t}}\right) between the end effector and the defect point in the tool coordinate system. We denote the initial coordinates of the robotic arm in the base coordinate system as \left(x_{{\mathrm{b}}_i},y_{{\mathrm{b}}_i},{\mathit{\text{z}}}_{{\mathrm{b}}_i}\right). The LiDAR is fixed on the ground, and its coordinate system is calibrated as the user coordinate system. Thus, we obtain the initial coordinates of the robotic arm in the user coordinate system \left(x_{{\mathrm{u}}_i},y_{{\mathrm{u}}_i},{\mathit{\text{z}}}_{{\mathrm{u}}_i}\right). Using the offset tool function of the robotic arm, we offset the initial position \left(x_{{\mathrm{u}}_i},y_{{\mathrm{u}}_i},{\mathit{\text{z}}}_{{\mathrm{u}}_i}\right) of the robotic arm by the relative distance \left(x_{\mathrm{c}},y_{\mathrm{c}},{\mathit{\text{z}}}_{\mathrm{c}}\right) in the tool coordinate system to obtain the coordinates of the defect location in the robotic arm tool coordinate system \left(x_{\mathrm{l}},y_{\mathrm{l}},{\mathit{\text{z}}}_{\mathrm{l}}\right). Then, based on the coordinates of the defect location in the robotic arm tool coordinate system and the point cloud model in the LiDAR coordinate system, we obtain the Euler angles \left(R_{{\mathrm{l}}_x},R_{{\mathrm{l}}_y},R_{{\mathrm{l}}_{\mathit{\text{z}}}}\right) of the defect location. Finally, the complete defect location coordinates \left(x_{\mathrm{l}},y_{\mathrm{l}},{\mathit{\text{z}}}_{\mathrm{l}},R_{{\mathrm{l}}_x},R_{{\mathrm{l}}_y},R_{{\mathrm{l}}_{\mathit{\text{z}}}}\right) are sent to the robotic arm to control its motion to the defect location.

4 Laboratory and experiments

4.1 Simulation

Before conducting actual experiments, it is necessary to perform simulation experiments to ensure the safety of robotic arm motion. It is necessary to plan the trajectory of the robotic arm in a simulated environment in advance to verify that the robotic arm’s motion is effective and executable. We use ER_Factory for robotic arm motion simulation. The simulation software allows offline and online programming of the robotic arm, provides existing robotic arm models, supports various types of file input and output, automatically optimizes generated trajectories, and enables simulation and export of robotic arm motion. We set up a virtual simulation environment, including the robotic arm, actuators, concrete pump, concrete mixer, and concrete pipelines. In the simulation environment, we take the six-axis robotic arm ER-210 as an example and perform a simple simulation of the robotic arm’s motion. First, we simplify the model through lightweight operations to avoid collisions between the robotic arm’s trajectory and the concrete pipeline. We perform motion trajectory planning and optimization for the Cartesian coordinates obtained from the camera and LiDAR, as shown in Fig.4.

4.2 Experimental materials and proportions

The cement uses in the experiment is P·I 42.5 ordinary Portland cement, which complies with the Chinese standard GB 8076-2008 [55]. The density of this cement is 3.16 g/cm³, and the specific surface area is 340 mm²/kg. The aggregates used include sand and gravel. The sand used is standard sand produced according to GSB 08-1337-2018 [56], and the gravel is specialized 10–20 mm gravel. Choosing the above-mentioned well-graded coarse and fine aggregates can ensure good wrapping of the concrete during pumping, and the mixture exhibits good lubrication and stability. To increase the flowability of the concrete, and to facilitate mixing and pumping, polycarboxylate superplasticizer is used. Its retarding effect is attributed to the delayed dissolution of anhydrous C₃S and inhibition of the free growth of C-S-H in the system by Polycarboxylate Superplasticizer. To improve the stability, cohesion, and robustness of the concrete, Hydroxy Propyl Methyl Cellulose with a viscosity of 200000 is used as a thickener. Its function is to improve stability, while superplasticizers are commonly used to increase flowability and reduce water-cement ratio (w/c). To accelerate the setting time of the concrete, and to enhance constructability and formability, an alkali-free liquid rapid-setting agent mainly composed of aluminum sulfate is used. Its function is to accelerate the hydration rate of C₃A and provide {\mathrm{S}\mathrm{O}}_4^{2-} ions in the solution without significantly changing the {\mathrm{A}\mathrm{l}}^{3+}/{\mathrm{S}\mathrm{O}}_4^{2-} ratio in the solution, creating a suitable sulfate system for the solution. The overall hydration rate of C₃A can be controlled, which does not hinder the subsequent hydration of C₃S or the densification process of C-S-H gel. Therefore, the addition of alkali-free rapid-setting agents containing aluminum sulfate promotes stable strength development and avoids significant rebound. Based on the experimental results, the mixed proportions are selected, as shown in Tab.2.

4.3 Experimental process

We set up a concrete structural defect repair experimental system, as shown in Fig.5, which includes a concrete mixer, concrete pump truck, robotic arm, depth camera fixed on the robotic arm, and a LiDAR fixed on the ground. We use DN80 standard hoses to connect the concrete pump truck and the robotic arm.

During the experimental process, we first determine the initial position \left(x_{{\mathrm{b}}_i},y_{{\mathrm{b}}_i},{\mathit{\text{z}}}_{{\mathrm{b}}_i}\right) of the robot. We use the depth camera to obtain the 3D coordinates \left(\mathrm{\Delta }{x}_{\mathrm{t}},\mathrm{\Delta }{y}_{\mathrm{t}},\mathrm{\Delta }{\mathit{\text{z}}}_{\mathrm{t}}\right) of the defect location relative to the initial robot position and record it, as shown in Fig.6. Through human-computer interaction, we send the coordinates of the defect location to the robotic arm, as shown in Fig.7.

After the robotic arm moved to the defect location, we use the offsettool function to obtain the 3D coordinates \left(x_{\mathrm{l}},y_{\mathrm{l}},{\mathit{\text{z}}}_{\mathrm{l}}\right) of that location in the user coordinate system of the LiDAR. Combining the coordinates and radius of the concrete pipeline obtained by the LiDAR, we calculate the angular coordinates (R_{\mathit{\text{z}}},R_y,R_x) of the robotic arm at the defect location. Finally, we adjust the robotic arm to the defect location \left(x_{\mathrm{l}},y_{\mathrm{l}},{\mathit{\text{z}}}_{\mathrm{l}},R_{\mathit{\text{z}}},R_y,R_x\right).

We set the pressure of the concrete pump truck to 15 MPa and the frequency of cylinder swing to 18 times per minute. The robot arm autonomously works to repair the detected concrete defects. The motion execution encoding is explained in Fig.8. The main program of the repair procedure is based on the programming language of the robotic arm. In this repair process, the coordinates of the defect location relative to the camera position are first received from the human-computer interaction interface by means of TCP/IP communication, and a loop is established to ensure that the coordinates are received. Next, the received coordinates are divided and transformed to obtain the coordinates pc1 that the robotic arm can execute. Pc0 describes the coordinates of the initial position of the robotic arm. Pc2 describes the coordinates of the defect location after pc0 is offset from pc1 in the tool coordinate system of the robotic arm. The robotic arm moves to the defect location by the robotic arm motion command and stops for ten seconds to complete the repair, and repeats the above actions by command and repairs the next defect. In each step of the experimental process, there is corresponding code to control the robotic arm to acquire the target point coordinates, move to the target point, adjust the pose, wait for spraying, complete the spraying, and move back to the initial position, as shown in Fig.8 and Fig.9.

4.4 Evaluation of robotic arm trajectory accuracy

To evaluate the trajectory accuracy of the robotic arm, we randomly select three different points (x₁, y₁, z₁) from the trajectory of the robotic arm’s actual movement to the defect location. We record the corresponding coordinates (x, y, z) returned by the camera at those points, as shown in the table. The precision of all coordinates is three decimal places. We use a formula to calculate the cosine similarity, which estimates the similarity between two sets of coordinates using the cosine value of the angle between the two vectors in the vector space. From the calculation results, it can be seen that the cosine similarity is close to 1, as shown in Tab.3. Therefore, we can conclude that these two sets of coordinates are highly correlated. The robotic arm exhibits a high trajectory accuracy in locating concrete structural defects, enabling precise defect repairs.

4.5 Evaluation of repair effect

We evaluate the repair effect of concrete structural defects from three aspects: appearance, concrete strength, and concrete mechanical properties. From the appearance perspective, the sprayed concrete covers the original defect locations, as shown in Fig.10. After using the depth camera to identify the concrete pipeline defects again, the defects cannot be recognized, as shown in Fig.11.

To test the strength of concrete materials, we evaluate the workability and rheology of the concrete. The slump of the freshly mixed concrete is 105 mm, which allows it to adhere well to the surface of the concrete defect. The ICAR rheometer is used to measure the development of static yield stress over time, as shown in Fig.12. The test results indicate that the static yield stress of the concrete material increases with time and can reach over 2000 Pa at 90 min, indicating that the concrete material has high strength.

We evaluate the mechanical properties of the concrete material and obtain a uniaxial compressive strength of 26.42 MPa and an elastic modulus of 23.30 GPa after hardening (28 d). The repaired concrete pipes can withstand large loads and are not prone to deformation.

5 Conclusions

This paper proposes a computer vision-based automated robot system for the repair of concrete structural defects. The system primarily utilizes camera and LiDAR sensor fusion to obtain the coordinates of the defects, and through human-machine interaction and coordinate system conversion, the robotic arm is controlled to repair defects. The results show that this method has high accuracy and can improve the automation level of concrete structural defect repairs. Future research will focus on further improving the automation level of the system, reducing human intervention, and simplifying the operational procedures of the system, aiming to develop a concrete structural defect repair system that is safe, efficient, and user-friendly.

References

[1] Cao V D, Le D A. Autonomous concrete crack detection using deep fully convolutional neural network. Automation in Construction, 2019, 99(1): 52–58

[2] Obiora C O, Ezeokoli F O, Belonwu C C, Okeke F N. Defects in concrete elements: A study of residential buildings of 30 years and above in Onitsha Metropolis, Anambra State, Nigeria. Journal of Building Construction and Planning Research, 2022, 10(3): 102–123

[3] KasaharaJ Y LYamashitaAAsamaH. Positive weak supervision quality increase by consolidation for acoustic defect detection in concrete structures. In: Proceedings of the 2021 IEEE/SICE International Symposium on System Integration. New York: IEEE, 2021

[4] Laxman K C, Tabassum N, Ai L, Cole C, Ziehl P. Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 2023, 370: 130709

[5] Ren Y P, Huang J, Hong Z Y, Lu W, Yin J, Zou L, Shen X. Image-based concrete crack detection in tunnels using deep fully convolutional networks. Construction and Building Materials, 2020, 234: 117367

[6] Gao Y, Sun H. Influence of initial defects on crack propagation of concrete under uniaxial compression. Construction and Building Materials, 2021, 277(4): 122361

[7] Cha Y J, Choi W, Büyüköztürk O. Deep learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(5): 361–378

[8] Pan Y, Zhang G, Zhang L. A spatial-channel hierarchical deep learning network for pixel-level automated crack detection. Automation in Construction, 2020, 119: 103357

[9] Liu Y, Yao J, Lu X, Xie R, Li L. DeepCrack: A deep hierarchical feature learning architecture for crack segmentation. Neurocomputing, 2019, 338(21): 139–153

[10] Li S, Zhao X, Zhou G. Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(7): 616–634

[11] Bhattacharya G, Mandal B, Puhan N B. Interleaved deep artifacts-aware attention mechanism for concrete structural defect classification. IEEE Transactions on Image Processing, 2021, 30: 6957–6969

[12] McLaughlin E, Charron N, Narasimhan S. Automated defect quantification in concrete bridges using robotics and deep learning. Journal of Computing in Civil Engineering, 2020, 34(5): 04020029

[13] Chen X, Zhao C, Chen J, Zhang D, Zhu K, Su Y. A compact Robot-based defect detection device design for silicon wafer. Journal of Physics: Conference Series, 2020, 1449(1): 012111

[14] Yang L, Li B, Li W, Brand H, Jiang B, Xian J. Concrete defects inspection and 3D mapping using CityFlyer quadrotor robot. IEEE/CAA Journal of Automatica Sinica, 2020, 7(4): 991–1002

[15] Feng D M, Feng M Q. Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection—A review. Engineering Structures, 2018, 156: 105–117

[16] Davoudi R, Miller G R, Kutz J N. Structural load estimation using machine vision and surface crack patterns for shear-critical RC beams and slabs. Journal of Computing in Civil Engineering, 2018, 32(4): 04018024

[17] NohYKooDKangY MParkDLeeD. Automatic crack detection on concrete images using segmentation via fuzzy C-means clustering. In: Proceedings of the 2017 International Conference on Applied System Innovation (ICASI). New York: IEEE, 2017

[18] DinhT HHaQ PLaH M. Computer vision-based method for concrete crack detection. In: Proceedings of the 2016 14th International Conference on Control, Automation, Robotics and Vision (ICARCV). New York: IEEE, 2016

[19] ChaY JGopalD LAliR. Vision-based concrete crack detection technique using cascade features. In: Sensors and Smart Structures Technologies for Civil, Mechanical, & Aerospace Systems 2018. Bellingham, WA: SPIE, 2018

[20] Koch C, Doycheva K, Kasireddy V, Akinci B, Fieguth P. A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Advanced Engineering Informatics, 2015, 29(2): 196–210

[21] Xu H, Su X, Wang Y, Cai H, Cui K, Chen X. Automatic bridge crack detection using a convolutional neural network. Applied Sciences, 2019, 9(14): 2867

[22] Hu L, Yang F. Runet: Convolutional networks for crack detection. Journal of Physics: Conference Series, 2022, 2171(1): 012052

[23] Ye W, Wei Y, Xu M, Ye L. Application of ground penetrating radar in concrete defect detection of sluice floor. E3S Web of Conferences, 2021, 276: 02023

[24] Cui X, Wang Q, Dai J, Xue Y, Duan Y. Intelligent crack detection based on attention mechanism in convolution neural network. Advances in Structural Engineering, 2021, 24(9): 1859–1868

[25] Wang W, Lu K, Wu Z, Long H, Zhang J, Chen P, Wang B. Surface defects classification of hot rolled strip based on improved convolutional neural network. ISIJ International, 2021, 61(5): 1579–1583

[26] Li Q, Luo Z, Chen H, Li C. An overview of deeply optimized convolutional neural networks and research in surface defect classification of workpieces. IEEE Access: Practical Innovations, Open Solutions, 2022, 10: 26443–26462

[27] Stephen O, Maduh U J, Sain M. A machine learning method for detection of surface defects on ceramic tiles using convolutional neural networks. Electronics, 2021, 11(1): 55

[28] Gopalakrishnan K, Khaitan S K, Choudhary A, Agrawal A. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Construction and Building Materials, 2017, 157(4): 322–330

[29] Song E P, Eem S H, Jeon H. Concrete crack detection and quantification using deep learning and structured light. Construction and Building Materials, 2020, 252(5): 119096

[30] Beckman G H, Polyzois D, Cha Y J. Deep learning-based automatic volumetric damage quantification using depth camera. Automation in Construction, 2019, 99(1): 114–124

[31] Lewis J, Cha Y J, Kim J. Dual encoder–decoder-based deep polyp segmentation network for colonoscopy images. Scientific Reports, 2023, 13(1): 1183

[32] Kang D H, Cha Y J. Efficient attention-based deep encoder and decoder for automatic crack segmentation. Structural Health Monitoring, 2022, 21(5): 2190–2205

[33] Leung C K, Elvin N, Olson N, Morse T F, He Y F. A novel distributed optical crack sensor for concrete structures. Engineering Fracture Mechanics, 2000, 65(2−3): 133–148

[34] Ai D, Yang Z, Li H, Zhu H. Heating-time effect on electromechanical admittance of surface-bonded PZT sensor for concrete structural monitoring. Measurement, 2021, 184: 109992

[35] Liu H, Zhang B, Liu H, Ji Z. Analysis of long-term durability monitoring data of high-piled wharf with anode-ladder sensors embedded in concrete. Frontiers in Materials, 2021, 8: 703347

[36] Kim T K, Park J S. Performance evaluation of concrete structures using crack repair methods. Sustainability, 2021, 13(6): 3217

[37] PerryMBiondiLMcAlorumJVlachakisC. Self-sensing concrete repairs based on alkali-activated materials: Recent progress. In: Proceedings of the 2021 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). New York: IEEE, 2021

[38] Martuscelli C, Soares C, Camões A, Lima N. Potential of fungi for concrete repair. Procedia Manufacturing, 2020, 46: 180–185

[39] Kumar K H, Babu N V, Lingeshwaran N. A study on repair of concrete structure using nondestructive tests. Materials Today: Proceedings, 2021, 47(15): 5439–5446

[40] Chen X, Zhang Q, Ming Y, Fu F, Rong H. A numerical study for chloride migration in concrete under electrochemical repair process. Proceedings of the Institution of Civil Engineers-Structures and Buildings, 2021, 176(7): 556–564

[41] Sun X, Miao L. Application of bio-remediation with bacillus megaterium for crack repair at low temperature. Journal of Advanced Concrete Technology, 2020, 18(5): 307–319

[42] Kan Y C, Lee M G, Lee H W. Mechanical Behavior of mode I fractured concrete repaired by polymethyl methacrylate (PMMA). Materials Science Forum, 2020, 990(5): 50–54

[43] Elhamnike S M, Abbaszadeh R, Razavinasab V, Ziaadiny H. Behavior and modeling of post-heated circular concrete specimens repaired with fiber-reinforced polymer composites. Advances in Structural Engineering, 2022, 25(3): 541–551

[44] Lenwari A, Thongchom C, Aboutaha R S. Cyclic flexural performance of fire-damaged reinforced concrete beams strengthened with carbon fiber-reinforced polymer plates. ACI Structural Journal, 2020, 117(6): 133–146

[45] Peng K D, Huang B T, Xu L Y, Hu R L, Dai J G. Flexural strengthening of reinforced concrete beams using geopolymer-bonded small-diameter CFRP bars. Engineering Structures, 2022, 256(4): 113992

[46] Kang D H, Cha Y J. Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Computer-Aided Civil and Infrastructure Engineering, 2018, 33(10): 885–902

[47] Waqas A, Kang D H, Cha Y J. Deep learning-based obstacle-avoiding autonomous UAVs with fiducial marker-based localization for structural health monitoring. Structural Health Monitoring, 2024, 23(2): 971–990

[48] Ali R, Kang D H, Suh G, Cha Y J. Real-time multiple damage mapping using autonomous UAV and deep faster region-based neural networks for GPS-denied structures. Automation in Construction, 2021, 130: 103831

[49] WangC YBochkovskiyALiaoH Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2023

[50] Xu W, Cai Y, He D, Lin J, Zhang F. Fast-Lio2: Fast direct LiDAR-inertial odometry. IEEE Transactions on Robotics, 2022, 38(4): 2053–2073

[51] Fischler M A, Bolles R C. Random sample consensus. Communications of the ACM, 1981, 24(6): 381–395

[52] LukácsGMartinRMarshallD. Faithful least-squares fitting of spheres, cylinders, cones and tori for reliable segmentation. In: Proceedings of Computer Vision—ECCV98: 5th European Conference on Computer Vision Freiburg. Berlin: Springer Berlin, 1998, 671–686

[53] MundtMMajumderSMuraliSPanetsosPRameshV. Meta-learning convolutional neural architectures for multi-target concrete defect classification with the concrete defect bridge image dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019, 11196–11205

[54] LinT YMaireMBelongieSHaysJPeronaPRamananDDollárPZitnickC L. Microsoft COCO: Common objects in context. In: Proceedings of Computer Vision—ECCV 2014: 13th European Conference. Berlin: Springer Berlin, 2014, 740–755

[55] GB8076-2008. Concrete Admixtures. Beijing: General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China, 2008

[56] GSB08-1337-2018. China ISO Standard Sand. Beijing: China National Academy of Building Materials Science and Technology Co., Ltd., 2018

RIGHTS & PERMISSIONS

Higher Education Press