A novel coal-rock recognition method in coal mining face based on fusing laser point cloud and images

Abstract

Rapid and accurate recognition of coal and rock is an important prerequisite for safe and efficient coal mining. In this paper, a novel coal-rock recognition method is proposed based on fusing laser point cloud and images, named Multi-Modal Frustum PointNet (MMFP). Firstly, MobileNetV3 is used as the backbone network of Mask R-CNN to reduce the network parameters and compress the model volume. The dilated convolutional block attention mechanism (Dilated CBAM) and inception structure are combined with MobileNetV3 to further enhance the detection accuracy. Subsequently, the 2D target candidate box is calculated through the improved Mask R-CNN, and the frustum point cloud in the 2D target candidate box is extracted to reduce the calculation scale and spatial search range. Then, the self-attention PointNet is constructed to segment the fused point cloud within the frustum range, and the bounding box regression network is used to predict the bounding box parameters. Finally, an experimental platform of shearer coal wall cutting is established, and multiple comparative experiments are conducted. Experimental results indicate that the proposed coal-rock recognition method is superior to other advanced models.

Keywords

Coal mining face / Coal-rock recognition / Deep learning / Laser point cloud and images fusion / Multi-Modal Frustum PointNet (MMFP)

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Nos. 52174152 and 52074271), in part by the Xuzhou Basic Research Program Project (No. KC23051), in part by the Shandong Province Technology Innova-tion Guidance Plan (Central Guidance for Local Scientific and Tech-nological Development Fund) (No. YDZX2024119), in part by the Graduate Innovation Program of China University of Mining and Technology (No. 2025WLKXJ088), and in part by the Postgraduate Research & Practice Innovation Program of Jiangsu Province (No. KYCX252830).

References

[1] Wang SB, Wang SJ. Longwall mining automation horizon control: coal seam gradient identification using piecewise linear fitting. Int J Min Sci Technol 2022; 32(4):821-9.

[2] He MC, Wang Q. Rock dynamics in deep mining. Int J Min Sci Technol 2023; 33 (9):1065-82.

[3] Zhou JP, Guo YC, Wang S, Chen W, Cheng G. Identification of coal and gangue based on R value method for dual-energy X-ray of Geant4 simulation. Environ Res 2023; 226:115650.

[4] Si L, Wang ZB, Liu XH, Tan C. A sensing identification method for shearer cutting state based on modified multi-scale fuzzy entropy and support vector machine. Eng Appl Artif Intell 2019; 78:86-101.

[5] Zhang MC, Zhao LJ, Shi BS. Analysis and construction of the coal and rock cutting state identification system in coal mine intelligent mining. Sci Rep 2023; 13:3489.

[6] Zhang Q, Gu JY, Liu JM. Research on coal and rock type recognition based on mechanical vision. Shock Vib 2021; 2021(1):6617717.

[7] Liu Y, Si L, Wang ZB, Wei D, Gu JH, Li X, Dai JB. A time-frequency analysis method of electromagnetic signal for coal and rock properties recognition while drilling based on CWT and GAPSO-ROA. Measurement 2025; 253:117447.

[8] Si L, Wang ZB, Liu P, Tan C, Chen H, Wei D. A novel coal-rock recognition method for coal mining working face based on laser point cloud data. IEEE Trans Instrum Meas 2021; 70:2514118.

[9] Xu XW, Li X, Ming WW, Chen M. A novel multi-scale CNN and attention mechanism method with multi-sensor signal for remaining useful life prediction. Comput Ind Eng 2022; 169:108204.

[10] Xiao KH, Kang XD, Liu HB, Duan PH. MOFA: A novel dataset for multi-modal image fusion applications. Inf Fusion 2023; 96:144.

[11] Kumar Singh S, Pratap Banerjee B, Raval S. A review of laser scanning for geological and geotechnical applications in underground mining. Int J Min Sci Technol 2023; 33(2):133-54.

[12] Yu T, Hu CH, Xie YN, Liu JZ, Li PP. Mature pomegranate fruit detection and location combining improved F-PointNet with 3D point cloud clustering in orchard. Comput Electron Agric 2022; 200:107233.

[13] Monsalve JJ, Baggett J, Bishop R, Ripepi N. Application of laser scanning for rock mass characterization and discrete fracture network generation in an underground limestone mine. Int J Min Sci Technol 2019; 29(1):131-7.

[14] Yin L, Luo B, Wang W, Yu H, Wang CJ, Li CY. CoMask: Corresponding mask-based end-to-end extrinsic calibration of the camera and LiDAR. Remote Sens 2020; 12(12):1925.

[15] Lee JS, Park TH. Fast road detection by CNN-based camera-lidar fusion and spherical coordinate transformation. IEEE Trans Intell Transp Syst 2021; 22 (9):5802-10.

[16] Kim JH, Starr JW, Lattimer BY. Firefighting robot stereo infrared vision and radar sensor fusion for imaging through smoke. Fire Technol 2015; 51 (4):823-45.

[17] Chen L, Hashiba K, Liu ZT, Lin FL, Mao WJ. Spatial-temporal fusion network for maximum ground surface settlement prediction during tunnel excavation. Autom Constr 2023; 147:104732.

[18] Ma JY, Xiong GM, Xu JY, Chen X. CVTNet: A cross-view transformer network for LiDAR-based place recognition in autonomous driving environments. IEEE Trans Ind Inform 2024; 20(3):4039-48.

[19] Qi CR, Liu W, Wu CX, Su H, Guibas LJ.Frustum PointNets for 3D object detection from RGB-D data. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE; 2018. p. 918-27.

[20] Xu FL, Zhao HK, Wu YF, Tao CB. F-3DNet: Extracting inner order of point cloud for 3D object detection in autonomous driving. Multimed Tools Appl 2024; 83 (3):8499-516.

[21] Li YW, Yu AW, Meng TJ, Caine B, Ngiam J, Peng DY, Shen JY, Lu YF, Zhou D, Le QV, Yuille A, Tan MX. DeepFusion: Lidar-camera deep fusion for multi-modal 3D object detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 18-24, 2022. New Orleans, LA, USA: IEEE; 2022. p. 17161-70.

[22] Zhang JM, Liu HY, Yang KL, Hu XX, Liu RP, Stiefelhagen R. CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. IEEE Trans Intell Transp Syst 2023; 24(12):14679-94.

[23] Liang M, Yang B, Wang SL, Urtasun R. Deep continuous fusion for multi-sensor 3D object detection. In: Computer Vision-ECCV 2018. Cham: Springer International Publishing; 2018. p. 663-78.

[24] Chen XZ, Ma HM, Wan J, Li B, Xia T. Multi-view 3D object detection network for autonomous driving. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Honolulu: IEEE; 2017. p. 6526-34.

[25] Ye WB, Chen HQ. Human activity classification based on micro-Doppler signatures by multiscale and multitask Fourier convolutional neural network. IEEE Sens J 2020; 20(10):5473-9.

[26] He KM, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision. Venice: IEEE; 2017. p. 2980-8.

[27] Bi XL, Hu JW, Xiao B, Li WS, Gao XB. IEMask R-CNN: Information-enhanced mask R-CNN. IEEE Trans Big Data 2023; 9(2):688-700.

[28] Howard A, Sandler M, Chen B, Wang WJ, Chen LC, Tan MX, Wang WJ, Zhu YK, Pang RM, Vasudevan V, Le QV, Adam H. Searching for MobileNetV3. In:Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul: IEEE; 2019. p. 1314-24.

[29] Siddiqui NA, Qadri MT, Akhter MO, Ali ZA. High-Precision brain tumor segmentation using a progressive layered U-Net (PLU-Net) with multi-scale data augmentation and attention mechanisms on multimodal magnetic resonance imaging. Instrumentation 2025; 12(1):2577-92.

[30] Luo JJ, Chen W, Tian ZJ, Zhang F, Liu Y. Real-time detection algorithm of underground personnel behavior based on YOLOv8-ECW. J Min Sci Technol 2025; 10(2):316-27. In Chinese.

[31] Heinrich S, Michaelis J, Reiher I, Coppers B, Lohmayer M, Fleischmann E, Kleyer A, Schett G, De Craemer A-S, Elewaut D, Leyendecker S, Liphardt A-M. Comparison and improvement of CyberGlove III calibration methods. IEEE Sens J 2024; 24(9):15283-91.

[32] Si L, Wang ZB, Li JH, Wei D, Zhao SH. A novel positioning method of anti-punching drilling robot based on the fusion of multi-IMUs and visual image. ISA Trans 2023; 137:730-46.

[33] Wang ZX, Jia K. Frustum ConvNet: Sliding Frustums to aggregate local point-wise features for amodal 3D object detection. In: Proceedings of the 2019 IEEE/ RSJ international conference on intelligent robots and systems (IROS). Macau: IEEE; 2019. p. 1742-9.

[34] Li B, Zhang YH, Sun FQ. Deep residual neural network based PointNet for 3D object part segmentation. Multimed Tools Appl 2022; 81(9):11933-47.

[35] Liu Y, Si L, Wang ZB, Wei D, Li X, Gu JH.Dual discriminator and adaptive multisource feature fusion WGAN-GP for coal-rock properties recognition under limited infrared thermal images. IEEE Trans Ind Inform 2025; PP (99):1-12.

[36] Jun TJ, Kweon J, Kim YH, Kim D. T-Net: Nested encoder-decoder architecture for the main vessel segmentation in coronary angiography. Neural Netw 2020; 128:216-33.

[37] Wang JD, Sun K, Cheng TH, Jiang BR, Deng CR, Zhao Y, Liu D, Mu YD, Tan MK, Wang XG, Liu WY, Xiao B. Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 2021; 43(10):3349-64.

[38] Wang ZG, Wang QN, Yang Y, Liu NH, Chen YM, Gao JH. Seismic facies segmentation via a segformer-based specific encoder-decoder-hypercolumns scheme. IEEE Trans Geosci Remote Sens 2023; 61:5903411.

[39] Charles RQ, Hao S, Mo KC, PointNet GLJ. Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition (CVPR). Honolulu: IEEE; 2017. p. 77-85.

[40] Shi SS, Wang XG, Li HS. PointRCNN: 3D object proposal generation and detection from point cloud. In: Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). Long Beach: IEEE; 2019. p. 770-9.

[41] Hou QB, Zhou DQ, Feng JS. Coordinate attention for efficient mobile network design. In: Proceedings of the 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR). Nashville: IEEE; 2021. p. 13708-17.

[42] Wan Y, Li JF. LGP-YOLO: An efficient convolutional neural network for surface defect detection of light guide plate. Complex Intell Syst 2024; 10 (2):2083-105.

[43] Lv QZ, Fan HX, Liu JL, Zhao YH, Xing MD, Quan YH. Multilabel deep learning-based lightweight radar compound jamming recognition method. IEEE Trans Instrum Meas 2024; 73:2521115.

[44] Tao WY, Yan XF, Wang YZ, Wei MQ. MFFDNet: Single image deraining via dual-channel mixed feature fusion. IEEE Trans Instrum Meas 2024; 73:5005813.