Utilizing semantic-level computer vision for fracture trace characterization of hard rock pillars in underground space

Abstract

This study utilizes a semantic-level computer vision-based detection to characterize fracture traces of hard rock pillars in underground space. The trace images captured by photogrammetry are used to establish the database for training two convolutional neural network (CNN)-based models, i.e., U-Net (University of Freiburg, Germany) and DeepLabV3+ (Google, USA) models. Chain code technology, polyline approximation algorithm, and the circular window scanning approach are combined to quantify the main characteristics of fracture traces on flat and uneven surfaces, including trace length, dip angle, density, and intensity. The extraction results indicate that the CNN-based models have better performances than the edge detection methods-based Canny and Sobel operators for extracting the trace and reducing noise, especially the DeepLabV3+ model. Furthermore, the quantization results further prove the reliability of extracting the fracture trace. As a result, a case study with two types of traces (i.e., on flat and uneven surfaces) demonstrates that the applied semantic-level computer vision detection is an accurate and efficient approach for characterizing the fracture trace of hard rock pillars.

Keywords

Fracture trace / Hard rock pillar / Semantic-level computer vision / Convolutional neural network (CNN) / Underground space

CRediT authorship contribution statement

Chuanqi Li: Conceptualization, Validation, Resources, Formal analysis, Visualization, Software, Writing – original draft. Jian Zhou: Conceptualization, Methodology, Validation, Software, Investigation, Writing – original draft, Supervision. Daniel Dias: Investigation, Writing – review & editing, Funding acquisition, Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research is partially supported by the National Natural Science Foundation Project of China (Grant No. 42177164), the Outstanding Youth Project of Hunan Provincial Department of Education (Grant No. 23B0008) and the Distinguished Youth Science Foundation of Hunan Province of China (2022JJ10073). The first author was funded by China Scholarship Council (Grant No. 202106370038). Thanks to Sinosteel Maanshan General Institute of Mining Research Co.,Ltd for providing data acquisition equipment (Conon EOS-5D Mark III) and on-site guidance for this research. In addition, authors would like to thank all the experts and scholars who have provided guidance and assistance for this research.

References

A. Abbaszadeh Shahri, C. Shan, E. Zäll, S. Larsson. Spatial distribution modeling of subsurface bedrock using a developed automated intelligence deep learning procedure: A case study in Sweden. J. Rock Mech. Geotech. Eng., 13 (6) (2021), pp. 1300-1310

A. Abbaszadeh Shahri, C. Shan, S. Larsson. A novel approach to uncertainty quantification in groundwater table modeling by automated predictive deep learning. Nat. Resour. Res., 31 (3) (2022), pp. 1351-1373

A. Abellán, T. Oppikofer, M. Jaboyedoff, N.J. Rosser, M. Lim, M.J. Lato. Terrestrial laser scanning of rock slope instabilities. Earth Surf. Proc. Land., 39 (1) (2014), pp. 80-97

M. Alipour, D.K. Harris. Increasing the robustness of material-specific deep learning models for crack detection across different materials. Eng. Struct., 206 (2020), Article 110157

K. Anvari, A. Mousavi, A.R. Sayadi, E. Sellers, E.F. Salmi. Automatic detection of rock boundaries using a hybrid recurrence quantification analysis and machine learning techniques. Bull. Eng. Geol. Environ., 81 (10) (2022), pp. 1-19

R. Asheghi, S.A. Hosseini, M. Saneie, A.A. Shahri. Updating the neural network sediment load models using different sensitivity analysis methods: a regional application. J. Hydroinf., 22 (3) (2020), pp. 562-577

P. Assali, P. Grussenmeyer, T. Villemin, N. Pollet, F. Viguier. Solid images for geostructural mapping and key block modeling of rock discontinuities. Comput. Geosci., 89 (2016), pp. 21-31

Y.S. Aurelio, De Almeida, de Castro, A.P. Braga. Learning from imbalanced data sets with weighted cross-entropy function. Neural processing letters, 50 (2019), pp. 1937-1949

T. Bamford, K. Esmaeili, A.P. Schoellig. A deep learning approach for rock fragmentation analysis. Int. J. Rock Mech. Min. Sci., 145 (2021), Article 104839

R. Battulwar, M. Zare-Naghadehi, E. Emami, J. Sattarvand. A state-of-the-art review of automated extraction of rock mass discontinuity characteristics using three-dimensional surface models. J. Rock Mech. Geotech. Eng., 13 (4) (2021), pp. 920-936

E. Bribiesca. A chain code for representing 3D curves. Pattern Recogn., 33 (5) (2000), pp. 755-765

A. Buyer, S. Aichinger, W. Schubert. Applying photogrammetry and semi-automated joint mapping for rock mass characterization. Eng. Geol., 264 (2020), Article 105332

Buyer, A., Schubert, W., 2018. Joint trace detection in digital images. In: ISRM International Symposium-10th Asian Rock Mechanics Symposium. OnePetro.

H. Byun, J. Kim, D. Yoon, I.S. Kang, J.J. Song. A deep convolutional neural network for rock fracture image segmentation. Earth Sci. Inf., 14 (4) (2021), pp. 1937-1951

J. Canny. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 6 (1986), pp. 679-698

J. Chen, M. Zhou, H. Huang, D. Zhang, Z. Peng. Automated extraction and evaluation of fracture trace maps from rock tunnel face images via deep learning. Int. J. Rock Mech. Min. Sci., 142 (2021), Article 104745

J. Chen, Y. Chen, A.G. Cohn, H. Huang, J. Man, L. Wei. A novel image-based approach for interactive characterization of rock fracture spacing in a tunnel face. Journal of Rock Mechanics and Geotechnical. Engineering (2022)

J. Chen, H. Seo, C. Gao, Q. Fang, D. Zhang, H. Huang. A Comprehensive Method for Similarity Evaluation in Discrete Fracture Network Modeling of Jointed Rock Masses. Rock Mech. Rock Eng., 1–15 (2023),

Cho, Y. J., 2021. Weighted Intersection over Union (wIoU): A New Evaluation Metric for Image Segmentation. arXiv preprint arXiv:2107.09858.

B. Chudasama, N. Ovaskainen, J. Tamminen, N. Nordbäck, J. Engström, I. Aaltonen. Automated mapping of bedrock-fracture traces from UAV-acquired images using U-Net convolutional neural networks. Comput. Geosci., 105463 (2023)

Dony, R. D., 1988. Line detection on rock face images. MA Sc (Doctoral dissertation, Thesis, Univ. of Waterloo, Waterloo, Canada). Eng. 36 (2), 163–179.

K. Du, X. Li, R. Su, M. Tao, S. Lv, J. Luo, J. Zhou. Shape ratio effects on the mechanical characteristics of rectangular prism rocks and isolated pillars under uniaxial compression. Int. J. Min. Sci. Technol., 32 (2) (2022), pp. 347-362

G.S. Esterhuizen, J.L. Ellenberger. . Effects of Weak Bands on Pillar Stability in Stone Mines: Field Observations and Numerical Model Assessment, West Virginia University (2007), pp. 320-326

G.S. Esterhuizen, A.T. Iannacchione, J.L. Ellenberger, D.R. Dolinar. Pillar stability issues based on a survey of pillar performance in underground limestone mines. In: Proceedings of 25th International Conference on Ground Control in Mining (2006), pp. 354-361

G.S. Esterhuizen, P.L. Tyrna, M.M. Murphy. A case study of the collapse of slender pillars affected by through-going discontinuities at a limestone mine in Pennsylvania. Rock Mech. Rock Eng., 52 (12) (2019), pp. 4941-4952

I. Farmakis, V. Marinos, G. Papathanassiou, E. Karantanellis. Automated 3D jointed rock mass structural analysis and characterization using LiDAR terrestrial laser scanner for rockfall susceptibility assessment: Perissa area case (Santorini). Geotech. Geol. Eng., 38 (3) (2020), pp. 3007-3024

Y. Fei, K.C. Wang, A. Zhang, C. Chen, J.Q. Li, Y. Liu, G. Yang, B. Li. Pixel-level cracking detection on 3D asphalt pavement images through deep-learning-based CrackNet-V. IEEE Trans. Intell. Transp. Syst., 21 (1) (2019), pp. 273-284

Ferrero, A. M., Migliazza, M. R., Umili, G., 2014. Rock mass characterization by means of advanced survey methods. In: ISRM Regional Symposium-EUROCK 2014. OnePetro.

A.M. Ferrero, G. Forlani, R. Roncella, H.I. Voyat. Advanced geostructural survey methods applied to rock mass characterization. Rock Mech. Rock Eng., 42 (4) (2009), pp. 631-665

A. Ghaderi, A.A. Shahri, S. Larsson. A visualized hybrid intelligent model to delineate Swedish fine-grained soil layers using clay sensitivity. Catena, 214 (2022), Article 106289

G. Gigli, N. Casagli. Semi-automatic extraction of rock mass structural data from high resolution LIDAR point clouds. Int. J. Rock Mech. Min. Sci., 48 (2) (2011), pp. 187-198

D. Güllmar, N. Jacobsen, A. Deistung, D. Timmann, S. Ropele, J.R. Reichenbach. Investigation of biases in convolutional neural networks for semantic segmentation using performance sensitivity analysis. Z. Med. Phys., 32 (3) (2022), pp. 346-360

J. Guo, L. Wu, M. Zhang, S. Liu, X. Sun. Towards automatic discontinuity trace extraction from rock mass point cloud without triangulation. Int. J. Rock Mech. Min. Sci., 112 (2018), pp. 226-237

J. Guo, Y. Liu, L. Wu, S. Liu, T. Yang, W. Zhu, Z. Zhang. A geometry-and texture-based automatic discontinuity trace extraction method for rock mass point cloud. Int. J. Rock Mech. Min. Sci., 124 (2019), Article 104132

D. Healy, R.E. Rizzo, D.G. Cornwell, N.J. Farrell, H. Watkins, N.E. Timms, E. Gomez-Rivas, M. Smith. FracPaQ: A MATLAB™ toolbox for the quantification of fracture patterns. J. Struct. Geol., 95 (2017), pp. 1-16

Hu, W., Wang, W., Ai, C., Wang, J., Wang, W., Meng, X., Liu., J., Tao, H., Qiu, S., 2021. Machine vision-based surface crack analysis for transportation infrastructure. Automation in Construction, 132, 103973.

H.W. Huang, Q.T. Li, D.M. Zhang. Deep learning based image recognition for crack and leakage defects of metro shield tunnel. Tunn. Undergr. Space Technol., 77 (2018), pp. 166-176

F. Jiang, G. Wang, P. He, C. Zheng, Z. Xiao, Y. Wu. Application of canny operator threshold adaptive segmentation algorithm combined with digital image processing in tunnel face crevice extraction. J. Supercomput., 78 (9) (2022), pp. 11601-11620

C. Jin, K. Wang, T. Han, Y. Lu, A. Liu, D. Liu. Segmentation of ore and waste rocks in borehole images using the multi-module densely connected U-net. Comput. Geosci., 159 (2022), Article 105018

R.P. Johnson. Contrast based edge detection. Pattern Recogn., 23 (3–4) (1990), pp. 311-318

N. Kanopoulos, N. Vasanthavada, R.L. Baker. Design of an image edge detection filter using the Sobel operator. IEEE J. Solid State Circuits, 23 (2) (1988), pp. 358-367

S. Karimpouli, P. Tahmasebi, E.H. Saenger. Coal cleat/fracture segmentation using convolutional neural networks. Nat. Resour. Res., 29 (3) (2020), pp. 1675-1685

D. Kong, C. Saroglou, F. Wu, P. Sha, B. Li. Development and application of UAV-SfM photogrammetry for quantitative characterization of rock mass discontinuities. Int. J. Rock Mech. Min. Sci., 141 (2021), Article 104729

E. Lashgari, D. Liang, U. Maoz. Data augmentation for deep-learning-based electroencephalography. J. Neurosci. Methods, 346 (2020), Article 108885

Y.K. Lee, J. Kim, C.S. Choi, J.J. Song. Semi-automatic calculation of joint trace length from digital images based on deep learning and data structuring techniques. Int. J. Rock Mech. Min. Sci., 149 (2022), Article 104981

B. Leng, H. Yang, G. Hou, A. Lyamin. Rock mass trace line identification incorporated with grouping algorithm at tunnel faces. Tunn. Undergr. Space Technol., 110 (2021), Article 103810

V. Levytskyi, R. Sobolevskyi, D. Zawieska, J. Markiewicz. The accuracy of determination of natural stone cracks parameters based on Terrestrial Laser Scanning and dense image matching data. Int. Multidiscipl. Sci. GeoConf. SGEM, 17 (2017), pp. 255-262

X. Li, Z. Chen, J. Chen, H. Zhu. Automatic characterization of rock mass discontinuities using 3D point clouds. Eng. Geol., 259 (2019), Article 105131

X.B. Li, J. Zhou, S.F. Wang, B. Liu. Review and practice of deep mining for solid mineral resources. Chin. J. Nonferrous Met., 27 (6) (2017), pp. 1236-1262

C. Li, J. Zhou, K. Du, D. Dias. Stability prediction of hard rock pillar using support vector machine optimized by three metaheuristic algorithms. Int. J. Min. Sci. Technol., 33 (8) (2023), pp. 1019-1036

Z. Liu, Y. Cao, Y. Wang, W. Wang. Computer vision-based concrete crack detection using U-net fully convolutional networks. Autom. Constr., 104 (2019), pp. 129-139

Y. Liu, J. Chen, C. Tan, J. Zhan, S. Song, W. Xu, J. Yan, Y. Zhang, M. Zhao, Q. Wang. Intelligent scanning for optimal rock discontinuity sets considering multiple parameters based on manifold learning combined with UAV photogrammetry. Eng. Geol., 106851 (2022)

Q. Liu, J. Li, Y. Li, M. Gao. Recognition methods for coal and coal gangue based on deep learning. IEEE Access, 9 (2021), pp. 77599-77610

J. Long, E. Shelhamer, T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3431-3440

C. Lopez-Molina, B. De Baets, H. Bustince, J. Sanz, E. Barrenechea. Multiscale edge detection based on Gaussian smoothing and edge tracking. Knowl.-Based Syst., 44 (2013), pp. 101-111

F. Lu, C. Fu, G. Zhang, W. Zhang, Y. Xie, Z. Li. Convolution neural network based on fusion parallel multiscale features for segmenting fractures in coal-rock images. J. Electron. Imaging, 29 (2) (2020), Article 023008

M. Mauldon, W.M. Dunne, M.B. Rohrbaugh Jr.. Circular scanlines and circular windows: new tools for characterizing the geometry of fracture traces. Journal of structural geology, 23 (2–3) (2001), pp. 247-258

M. Mohammadpour, A. Bahroudi, M. Abedi. Automatic lineament extraction method in mineral exploration using CANNY algorithm and Hough transform. Geotectonics, 54 (3) (2020), pp. 366-382

D.L. Naik. A novel sensitivity-based method for feature selection. J. Big Data, 8 (1) (2021), pp. 1-16

C. Pham, L. Zhuang, S. Yeom, H.S. Shin. Automatic fracture characterization in CT images of rocks using an ensemble deep learning approach. Int. J. Rock Mech. Min. Sci., 170 (2023), Article 105531

R. Prabhakaran, P.O. Bruna, G. Bertotti, D. Smeulders. An automated fracture trace detection technique using the complex shearlet transform. Solid Earth, 10 (6) (2019), pp. 2137-2166

Y.G. Qiu, J. Zhou. Short-term rockburst prediction in underground project: insights from an explainable and interpretable ensemble learning model. Acta Geotech., 1–30 (2023),

Y. Ren, J. Huang, Z. Hong, W. Lu, J. Yin, L. Zou, X. Shen. Image-based concrete crack detection in tunnels using deep fully convolutional networks. Constr. Build. Mater., 234 (2020), Article 117367

T.D. Seers, D. Hodgetts. Extraction of three-dimensional fracture trace maps from calibrated image sequences. Geosphere, 12 (4) (2016), pp. 1323-1340

C. Steger. An unbiased detector of curvilinear structures. IEEE Trans. Pattern Anal. Mach. Intell., 20 (2) (1998), pp. 113-125

Y. Tang, L. He, W. Lu, X. Huang, H. Wei, H. Xiao. A novel approach for fracture skeleton extraction from rock surface images. Int. J. Rock Mech. Min. Sci., 142 (2021), Article 104732

G. Umili, A. Ferrero, H.H. Einstein. A new method for automatic discontinuity traces sampling on rock mass 3D model. Comput. Geosci., 51 (2013), pp. 182-192

Y. Vasuki, E.J. Holden, P. Kovesi, S. Micklethwaite. Semi-automatic mapping of geological Structures using UAV-based photogrammetric data: An image analysis approach. Comput. Geosci., 69 (2014), pp. 22-32

T. Vu, T. Bao, Q.V. Hoang, C. Drebenstetd, P.V. Hoa, H.H. Thang. Measuring blast fragmentation at Nui Phao open-pit mine, Vietnam using the Mask R-CNN deep learning model. Min. Technol., 130 (4) (2021), pp. 232-243

W. Wang, Y. Liang. Rock fracture centerline extraction based on Hessian matrix and Steger algorithm. KSII Trans. Internet Information Syst. (TIIS), 9 (12) (2015), pp. 5073-5086

S.H. Wang, K. Muhammad, J. Hong, A.K. Sangaiah, Y.D. Zhang. Alcoholism identification via convolutional neural network based on parametric ReLU, dropout, and batch normalization. Neural Comput. Applic., 32 (2020), pp. 665-680

W. Wang, M. Wang, H. Li, H. Zhao, K. Wang, C. He, J. Wang, S. Zheng, J. Chen. Pavement crack image acquisition methods and crack extraction algorithms: A review. J. Traffic Transp. Eng. (English Ed.), 6 (6) (2019), pp. 535-556

Z. Wang, J. Wang, K. Yang, L. Wang, F. Su, X. Chen. Semantic segmentation of high-resolution remote sensing images based on a class feature attention mechanism fused with Deeplabv3+. Comput. Geosci., 158 (2022), Article 104969

L. Wang, C. Wu, Z. Yang, L. Wang. Deep learning methods for time-dependent reliability analysis of reservoir slopes in spatially variable soils. Comput. Geotech., 159 (2023), Article 105413

J. Xiao, J. He, H. Yang, C. Wu, J. Xu, Y. Li. Geochemical Characteristics and Genetic Significance of Datangpo-Type Manganese Ore Deposits during the Cryogenian Period. Resour. Geol., 69 (3) (2019), pp. 227-248

C. Xie, H. Nguyen, X.N. Bui, V.T. Nguyen, J. Zhou. Predicting roof displacement of roadways in underground coal mines using adaptive neuro-fuzzy inference system optimized by various physics-based optimization algorithms. J. Rock Mech. Geotech. Eng., 13 (6) (2021), pp. 1452-1465

Z. Xu, W. Ma, P. Lin, H. Shi, D. Pan, T. Liu. Deep learning of rock images for intelligent lithology identification. Comput. Geosci., 154 (2021), Article 104799

W. Xu, Y. Zhang, X. Li, X. Wang, F. Ma, J. Zhao, Y. Zhang. Extraction and statistics of discontinuity orientation and trace length from typical fractured rock mass: A case study of the Xinchang underground research laboratory site, China. Eng. Geol., 269 (2020), Article 105553

C.M. Yeum, S.J. Dyke. Vision-based automated crack detection for bridge inspection. Comput. Aided Civ. Inf. Eng., 30 (10) (2015), pp. 759-770

B. Yu, Z. Zhang, T. Kuang, J. Liu. Stress changes and deformation monitoring of longwall coal pillars located in weak ground. Rock Mech. Rock Eng., 49 (8) (2016), pp. 3293-3305

P. Zhang. A novel feature selection method based on global sensitivity analysis with application in machine learning-based prediction model. Appl. Soft Comput., 85 (2019), Article 105859

W. Zhang, X. Gu, L. Hong, L. Han, L. Wang. Comprehensive review of machine learning in geotechnical reliability analysis: Algorithms, applications and further challenges. Appl. Soft Comput., 110066 (2023)

C. Zhang, K. Han, D. Zhang. Face stability analysis of shallow circular tunnels in cohesive–frictional soils. Tunn. Undergr. Space Technol., 50 (2015), pp. 345-357

Y. Zhang, H. Qi, C. Li, J. Zhou. Enhancing safety, sustainability, and economics in mining through innovative pillar design: a state-of-the-art review. Journal of Safety and Sustainability, 1-21 (2023)

P. Zhang, Q. Zhao, D.D. Tannant, T. Ji, H. Zhu. 3D mapping of discontinuity traces using fusion of point cloud and image data. Bull. Eng. Geol. Environ., 78 (4) (2019), pp. 2789-2801

S. Zhao, D. Zhang, Y. Xue, M. Zhou, H. Huang. A deep learning-based approach for refined crack evaluation from shield tunnel lining images. Autom. Constr., 132 (2021), Article 103934

J. Zhou, X. Li, H.S. Mitri. Comparative performance of six supervised learning methods for the development of models of hard rock pillar stability prediction. Nat. Hazards, 79 (2015), pp. 291-316

J. Zhou, S. Huang, Y. Qiu. Optimization of random forest through the use of MVO, GWO and MFO in evaluating the stability of underground entry-type excavations. Tunn. Undergr. Space Technol., 124 (2022), Article 104494

J. Zhou, Y. Chen, H. Chen, M. Khandelwal, M. Monjezi, K. Peng. Hybridizing five neural-metaheuristic paradigms to predict the pillar stress in bord and pillar method. Front. Public Health, 11 (2023), p. 1119580

M. Zhou, M. Shadabfar, H. Huang, Y.F. Leung, S. Uchida. Meta-modelling of coupled thermo-hydro-mechanical behaviour of hydrate reservoir. Comput. Geotech., 128 (2020), Article 103848