Coal-rock interface real-time recognition based on the improved YOLO detection and bilateral segmentation network

Abstract

To improve the accuracy and efficiency of coal-rock interface recognition, this study proposes a model built on the real-time detection algorithm, you only look once (YOLO), and the lightweight bilateral segmentation network. Simultaneously, the regional similarity transformation function and dragonfly algorithm are introduced to enhance the quality of coal-rock images. The comparison with three other models demonstrates the superior edge inference performance of the proposed model, achieving a mean Average Precision (mAP) of 90.2 at the Intersection over Union (IoU) threshold of 0.50 (mAP₅₀) and 81.4 across a range of IoU thresholds from 0.50 to 0.95 (mAP_[50,95]). Furthermore, to maintain high accuracy and real-time recognition capabilities, the proposed model is optimized using the open visual inference and neural network optimization toolkit, resulting in a 144.97% increase in the mean frames per second. Experimental results on four actual coal faces confirm the efficacy of the proposed model, showing a better balance between accuracy and efficiency in coal-rock image recognition, which supports further advancements in coal mining intelligence.

Keywords

Coal-rock real-time recognition / Grayscale enhancement / YOLO / Bilateral segmentation network / Edge inference

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

CRediT authorship contribution statement

Shuzhan Xu: Writing - original draft, Visualization, Validation, Methodology, Data curation, Conceptualization. Wanming Jiang: Writing - review & editing. Quansheng Liu: Writing - review & editing, Supervision, Funding acquisition, Conceptualization. Hongsheng Wang: Writing - review & editing. Jun Zhang: Writing - review & editing. Jinlong Li: Writing - review & editing. Xing Huang: Resources, Funding acquisition. Yin Bo: Writing - review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This research was funded by the National Natural Science Foundation of China (Grant Nos. U21A20153 and 52074258), the Key Research and Development Project of Hubei Province, China (Grant No. 2021BCA133), the Outstanding Youth Fund Program of the Natural Science Foundation of Hubei Province, China (Grant No. 2022CFA084), and the Wuhan Knowledge Innovation Supporting project (Grant No. 2022010801010162).

References

[1] Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprint ar- Xiv:2004.10934.

[2] Cao, S. G., Sun, C. M., Guo, P., Luo, F., & Liu, Y. B. (2015). Image processing and its applications of meso-crack of coal based on modified cv model. Chinese Journal of Rock Mechanics and Engineering, 34(S1), 3074-3081 (in Chinese).

[3] Carbas, S., Toktas, A., & Ustun, D. (2021). Nature-inspired metaheuristic algorithms for engineering optimization applications. Springer.

[4] Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018). Generative adversarial networks: an overview. IEEE signal processing magazine, 35(1), 53-65.

[5] Cui, Z. H., & Shi, Z. Z. (2009). Boid particle swarm optimisation. International Journal of Innovative Computing and Applications, 2(2), 77-85.

[6] Deng, C. F., Wang, M. M., Liu, L., Liu, Y., & Jiang, Y. L. (2021). Extended feature pyramid network for small object detection. IEEE Transactions on Multimedia, 24, 1968-1979.

[7] Deng, T., & Yu, Y. (2021). Research on ore identification and separation based on improved PSO-Faster R-CNN algorithm. Mining Research and Development, 41(02), 178-182 (in Chinese).

[8] Dogan, E. (2014). Solving design optimization problems via hunting search algorithm with Levy flights. Structural Engineering and Mechanics, 52(2), 351-368.

[9] Du, J. (2018). Understanding of object detection based on CNN family and YOLO. In 2nd International Conference on Machine Vision and Information Technology (CMVIT 2018) (pp. 012029). Hong Kong, China: IOP Publishing.

[10] Gao, F., Yin, X., Liu, Q. S., Huang, X., Bo, Y., Zhang, Q. T., & Wang, X. Y. (2021). Coal-rock image recognition method for mining and heading face based on spatial pyramid pooling structure. Journal of China Coal Society, 46(12), 4088-4102 (in Chinese).

[11] Gorai, A., & Ghosh, A. (2009). Gray-level image enhancement by particle swarm optimization. In 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC), Coimbatore, India (pp.72-77). IEEE.

[12] He, K. M., Zhang, X. Y., Ren, S. Q., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, USA (pp.770-778).

[13] Hua, T. X., Xing, C. E., & Zhao, L. (2019). Recognition of coal rock and positioning measurement of coal seam based on Faster R-CNN. Mining & Processing Equipment, 47(8), 4-9 (in Chinese).

[14] Huang, L., & Guo, C. Y. (2018). Texture feature extraction of coal-rock image based on variogram and local variance image. Journal of Mine Automation, 44(4), 62-68 (in Chinese).

[15] Huang, S. J. (2013). Research of coal-rock interface identification based on gray threshold method. Journal of Mine Automation, 39(05), 52-54 (in Chinese).

[16] Jiang, P. Y., Ergu, D., Liu, F. Y., Cai, Y., & Ma, B. (2022). A review of Yolo algorithm developments. Procedia Computer Science, 199, 1066-1073.

[17] Jiang, Q., Jia, M. T., Bi, L., Zhuang, Z., & Gao, K. X. (2022). Development of a core feature identification application based on the Faster R-CNN algorithm. Engineering Applications of Artificial Intelligence, 115, 105200.

[18] Jiang, T. Y., Li, C., Yang, M., & Wang, Z. L. (2022). An improved YOLOv5s algorithm for object detection with an attention mechanism. Electronics, 11(16), 2494.

[19] Jin, Z. M., & Finkel, H. (2020). Analyzing deep learning model inferences for image classification using OpenVINO. In 2020 IEEE international parallel and distributed processing symposium workshops (IPDPSW) (pp. 908-911). New Orleans, USA: IEEE.

[20] Joseph, R., Santosh, K. D., Ross, B. G., & Ali, F. (2016). You only look once: unified, real-time object detection. In 2016 IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, USA (pp.779-788).

[21] Katırcıoğlu, F., & Cingiz, Z. (2020). A novel gray image enhancement using the regional similarity transformation function and dragonfly algorithm. El-Cezeri, 7(3), 1201-1219.

[22] Koonce, B. (2021). SqueezeNet. In B. Koonce (Ed.), Convolutional neural networks with swift for tensorflow: image recognition and dataset categorization. https://doi.org/10.1007/978-1-4842-6168-2_7.

[23] Kozlov, A., Lazarevich, I., Shamporov, V., Lyalyushkin, N., & Gorbachev, Y. (2021). Neural network compression framework for fast model inference. Intelligent computing: proceedings of the 2021 computing conference ( 3, pp. 213-232). Springer International Publishing.

[24] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90.

[25] Kustikova, V., Vasiliev, E., Khvatov, A., Kumbrasiev, P., Vikhrev, I., Utkin, K., Dudchenko, A., & Gladilov, G. (2019). Intel distribution of OpenVINO toolkit: a case study of semantic segmentation. In Analysis of images, social networks and texts: 8th international conference (pp. 11-23). Springer International Publishing.

[26] Li, S. H., Jiao, J. T., Han, Y. J., & Weissman, T. (2016). Demystifying resnet. arXiv preprint arXiv:1611.01186.

[27] Li, X., Wang, X., Chen, J., & Zhao, H. (2023). The semantic segmentation method of coal rock image based on ResNeSt. In 2023 4th international seminar on artificial intelligence, networking and information technology (AINIT) (pp. 740-744). Nanjing, China: IEEE.

[28] Li, Y., & Lv, C. (2020). Ss-yolo: An object detection algorithm based on YOLOv3 and shufflenet. In 2020 IEEE 4th information technology, networking, electronic and automation control conference (ITNEC) (pp. 769-772). Chongqing, China: IEEE.

[29] Liang, F. T., Zhou, Y., Chen, X., Liu, F. Z., Zhang, C. H., & Wu, X. J. (2021). Review of target detection technology based on deep learning. In Proceedings of the 5th international conference on control engineering and artificial intelligence, Sanya, China (pp.132-135).

[30] Liu, X. Y., Jing, W., Zhou, M. X., & Li, Y. X. (2019). Multi-scale feature fusion for coal-rock recognition based on completed local binary pattern and convolution neural network. Entropy, 21(6), 622.

[31] Mahasin, M., & Dewi, I. A. (2022). Comparison of cspdarknet53, cspresnext-50, and efficientnet-b0 backbones on yolo v4 as object detector. International Journal of Engineering, Science and Information Technology, 2(3), 64-72.

[32] Mark, E., Luc, V. G., Christopher, K. I., John, W. W., & Andrew, Z. (2010). The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303-338.

[33] Mirjalili, S. (2016). Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems. Neural Computing and Applications, 27(4), 1053-1073.

[34] Qian, H. M., Wang, H. L., Feng, S., & Yan, S. Y. (2023). FESSD: SSD target detection based on feature fusion and feature enhancement. Journal of Real-Time Image Processing, 20(1), 2.

[35] Qu, Z., Shang, X., Xia, S. F., Yi, T. M., & Zhou, D. Y. (2022). A method of single-shot target detection with multi-scale feature fusion and feature enhancement. IET Image Processing, 16(6), 1752-1763.

[36] Rani, S., Ghai, D., & Kumar, S. (2022). Object detection and recognition using contour based edge detection and fast R-CNN. Multimedia Tools and Applications, 81(29), 42183-42207.

[37] Recep, D., & Ferzan, K. (2007). Segmentation of color images based on relation matrix. In 2007 IEEE 15th signal processing and communications applications, Eskisehir, Turkey (pp.1-4).

[38] Reynolds, C. W. (1987). Flocks, herds and schools:a distributed behavioral model. In Proceedings of the 14th annual conference on Computer graphics and interactive techniques (pp. 25-34). New York, USA: Association for Computing Machinery.

[39] Rezatofighi, H., Tsoi, N., Gwak, J. Y., Sadeghian, A., Reid, I., & Savarese, S. (2019). Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, USA (pp.658-666).

[40] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net:Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015 (pp. 234-241). Munich, Germany: Springer.

[41] Shi, J. T., & Li, J. (2021). lmproved algorithm for coal and rock image identification. Journal of Heilongjiang University of Science and Technology, 31(6), 830-836 (in Chinese).

[42] Si, L., Wang, Z. B., Xiong, X. X., & Tan, C. (2021). Coal-rock recognition method of fully-mechanized coal mining face based on improved UNet network model. Journal of China Coal Society, 46(S1), 578-589 (in Chinese).

[43] Siddique, N., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021). U-net and its variants for medical image segmentation: a review of theory and applications. IEEE Access, 9, 82031-82057.

[44] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[45] Sinha, D., & El-Sharkawy, M. (2019). Thin mobilenet: An enhanced mobilenet architecture. In 2019 IEEE 10th annual ubiquitous computing, electronics & mobile communication conference (UEMCON) (pp. 0280-0285). IEEE.

[46] Sultana, F., Sufian, A., & Dutta, P. (2020). A review of object detection models based on convolutional neural network. Intelligent computing: image processing based applications. Springer.

[47] Sun, C. M., Cao, S. G., & Li, Y. (2015). Investigation on meso-structure of coal and rock based on the modified LBF model. Journal of China Coal Society, 40(2), 331-341 (in Chinese).

[48] Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI conference on artificial intelligence..

[49] Targ, S., Almeida, D., & Lyman, K. (2016). Resnet in resnet: generalizing residual architectures. arXiv preprint arXiv:1603.08029.

[50] Walawalkar, D., Shen, Z. Q., Liu, Z. C., & Savvides, M. (2020). Attentive cutmix: an enhanced data augmentation approach for deep learning based image classification. In 2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), Barcelona, Spain (pp.3642-3646).

[51] Wang, J. C., Li, L. H., & Yang, S. L. (2018). Experimental study on gray and texture features extraction of coal and gangue image under different illuminance. Journal of China Coal Society, 43(11), 3051-3061 (in Chinese).

[52] Wang, J. H. (2014). Development and prospect on fully mechanized mining in Chinese coal mines. International Journal of Coal Science & Technology, 1(3), 253-260.

[53] Wang, K. F., Gou, C., Duan, Y. J., Lin, Y. L., Zheng, X. H., & Wang, F. Y. (2017). Generative adversarial networks: introduction and outlook. IEEE/CAA Journal of Automatica Sinica, 4(4), 588-598.

[54] Wang, X., Gao, F., Chen, J., Hao, P. C., & Jing, Z. J. (2021). Generative adversarial networks based sample generation of coal and rock images. Journal of China Coal Society, 46(9), 3066-3078 (in Chinese).

[55] Wu, W. T., Liu, H., Li, L. L., Long, Y. L., Wang, X. D., Wang, Z. H., Li, J. L., & Chang, Y. (2021). Application of local fully Convolutional Neural Network combined with YOLO v5 algorithm in small target detection of remote sensing image. PloS One, 16(10), e0259283.

[56] Wu, Y. X., & Meng, X. L. (2018). Locality-constrained self-taught learning for coal-rock recognition. Journal of China Coal Society, 43(9), 2639-2646 (in Chinese).

[57] Wu, Y. X., & Tian, Y. M. (2016). Method of coal-rock image feature extraction and recognition based on dictionary learning. Journal of China Coal Society, 41(12), 3190-3196 (in Chinese).

[58] Xiong, L., Ye, M., Zhang, D., Gan, Y., & Liu, Y. G. (2022). Source data-free domain adaptation for a faster R-CNN. Pattern Recognition, 124, 108436.

[59] Yan, Z. R., Wang, H. W., & Geng, Y. D. (2023). Coal-rock interface image recognition method based on improved DeeplabV3+ and transfer learning. Coal Science and Technology, 51(S1), 429-439 (in Chinese).

[60] Yang, J. J., Chang, B. S., Zhang, Y. C., Luo, W. J., Ge, S. R., & Wu, M. (2022). CNN coal and rock recognition method based on hyperspectral data. International Journal of Coal Science & Technology, 9(1), 63.

[61] Ye, Z. W., Wang, M. W., Hu, Z. B., & Liu, W. (2015). An adaptive image enhancement technique by combining cuckoo search and particle swarm optimization algorithm. Computational Intelligence and Neuroscience, 2015, 825398.

[62] Yu, C. Q., Wang, J. B., Peng, C., Gao, C. X., Yu, G., & Sang, N. (2018). Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 325-341).

[63] Yu, J. H., Jiang, Y. N., Wang, Z. Y., Cao, Z. M., & Huang, T. (2016). UnitBox: an advanced object detection network. In Proceedings of the 24th ACM international conference on multimedia (pp. 516-520). Amsterdam, The Netherlands: Association for Computing Machinery.

[64] Zhang, B., Su, X. G., Duan, Z. X., Chang, L. Z., & Wang, F. Z. (2020). Application of YOLOv2 in intelligent recognition and location of coal and rock. Journal of Mining and Strata Control Engineering, 2(2), 94-101 (in Chinese).

[65] Zhang, C. Y., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3), 107-115.

[66] Zhang, L., Li, S. B., Huang, Z. H., & Wang, X. M. (2014). Definition and realization of unmanned mining in fully-mechanized coal mining face. Coal Science and Technology, 42(9), 26-29 (in Chinese).

[67] Zhang, Q., Liu, J. M., Gu, J. Y., & Tian, Y. (2022). Study on coal-rock interface characteristics change law and recognition based on active thermal excitation. European Journal of Remote Sensing, 55(sup1), 35-45.

[68] Zhang, S. R., & Zhang, M. Y. (2018). On identification of coal and rock images. In 2018 International symposium on computer, consumer and control (IS3C) (pp. 290-293). Taichung, Taiwan, China: IEEE.

[69] Zhou, F., He, F. J., Gui, C. C., Dong, Z. Y., & Xing, M. D. (2022). SAR target detection based on improved SSD with saliency map and residual network. Remote Sensing, 14(1), 180.

[70] Zhou, Z., Zhang, J. J., Gong, C. J., & Wu, W. (2023). Automatic tunnel lining crack detection via deep learning with generative adversarial network-based data augmentation. Underground Space, 9, 140-154.

[71] Zhou, Z. W., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. M. (2018). Unet++:a nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support (pp.3-11). Granada, Spain: Springer.

[72] Zunin, V. (2021). Intel openvino toolkit for computer vision: Object detection and semantic segmentation. In 2021 International Russian Automation Conference (RusAutoCon) (pp. 847-851). Sochi Russian Federation: IEEE.