Image format pipeline and instrument diagram recognition method based on deep learning

Abstract

In this study, we proposed a recognition method based on deep artificial neural networks to identify various elements in pipelines and instrumentation diagrams (P&ID) in image formats, such as symbols, texts, and pipelines. Presently, the P&ID image format is recognized manually, and there is a problem with a high recognition error rate; therefore, automation of the above process is an important issue in the processing plant industry. The China National Offshore Petrochemical Engineering Co. provided the image set used in this study, which contains 51 P&ID drawings in the PDF. We converted the PDF P&ID drawings to PNG P&IDs with an image size of 8410 × 5940. In addition, we used labeling software to annotate the images, divided the dataset into training and test sets in a 3:1 ratio, and deployed a deep neural network for recognition. The method proposed in this study is divided into three steps. The first step segments the images and recognizes symbols using YOLOv5 + SE. The second step determines text regions using character region awareness for text detection, and performs character recognition within the text region using the optical character recognition technique. The third step is pipeline recognition using YOLOv5 + SE. The symbol recognition accuracy was 94.52%, and the recall rate was 93.27%. The recognition accuracy in the text positioning stage was 97.26% and the recall rate was 90.27%. The recognition accuracy in the character recognition stage was 90.03% and the recall rate was 91.87%. The pipeline identification accuracy was 92.9%, and the recall rate was 90.36%.

Keywords

Deep learning / Image processing / Piping and instrumentation / Object recognition / Pipeline recognition

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] E.-S. Yu, J.-M. Cha, T. Lee, J. Kim, D. Mun, Features recognition from piping and instrumentation diagrams in image format using a deep learning network, Energies 12 (23) (2019) 4425.

[2] Y. Moon, J. Lee, D. Mun, S. Lim, Deep learning-based method to rec-ognize line objects and flow arrows from image-format piping and instrumentation diagrams for digitization, Appl. Sci. 11 (21) (2021) 10054.

[3] D. Benjamin, P. Forgues, E. Gulko, J. Massicotte, C. Meubus, The use of high-level knowledge for enhanced entry of engineering drawings, in: 9th International Conference on Pattern Recognition, IEEE Computer Society, 1988, pp. 119-120.

[4] C.-S. Fahn, J.-F. Wang, J.-Y. Lee, A topology-based component extractor for understanding electronic circuit diagrams, Comput. Vis. Graph. Image Process. 44 (2) (1988) 119-138.

[5] S.-W. Lee, J.H. Kim, F.C. Groen, Translation-, rotation-and scale-invariant recognition of hand-drawn symbols in schematic diagrams, Int. J. Pattern Recognit. Artif. Intell. 4 (01) (1990) 1-25.

[6] T. Futatsumata, G. Shichino, J. Shibayama, A. Maeda, Development of an automatic recognition system for plant diagrams, in: MVA, 1990, pp. 207-210.

[7] D. Benjamin, P. Forgues, E. Gulko, J. Massicotte, C. Meubus, The use of high-level knowledge for enhanced entry of engineering drawings, in: 9th International Conference on Pattern Recognition, IEEE Computer Society, 1988, pp. 119-120.

[8] S.-O. Kang, E.-B. Lee, H.-K. Baek, A digitization and conversion tool for imaged drawings to intelligent piping and instrumentation diagrams ( P&ID), Energies 12 (13) (2019) 2593.

[9] H. Kato, S. Inokuchi,The recognition method for roughly hand-drawn log-ical diagrams based on hybrid utilization of multi-layered knowledge,in:[1990] Proceedings. 10th International Conference on Pattern Recognition, Vol. 1, IEEE, 1990, pp. 578-582.

[10] R. Maini, H. Aggarwal, Study and comparison of various image edge detection techniques, Int. J. Image Process. (IJIP) 3 (1) (2009) 1-11.

[11] L. Xu, E. Oja, P. Kultanen, A new curve detection method: Randomized hough transform (RHT), Pattern Recognit. Lett. 11 (5)(1990) 331-338.

[12] D. Wang, V. Haese-Coat, J. Ronsin, Shape decomposition and representation using a recursive morphological operation, Pattern Recognit. 28 (11) (1995) 1783-1792.

[13] W.C. Tan, I.-M. Chen, H.K. Tan, Automated identification of components in raster piping and instrumentation diagram with minimal pre-processing, in: 2016 IEEE International Conference on Automation Science and Engineering, CASE, IEEE, 2016, pp. 1301-1306.

[14] E. Arroyo, M. Hoernicke, P. Rodríguez, A. Fay, Automatic derivation of qual-itative plant simulation models from legacy piping and instrumentation diagrams, Comput. Chem. Eng. 92 (2016) 112-132.

[15] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, vol. 25, 2012.

[16] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions. In proceedings of the IEEE computer society conference on computer vision and pattern recognition, 2015.

[17] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection,in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 779-788.

[18] R. Girshick, J. Donahue, T. Darrell, J. Malik,Rich feature hierarchies for accurate object detection and semantic segmentation, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587.

[19] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A.C. Berg, SSD: Single shot multibox detector, in: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, the Netherlands,October 11-14, 2016, pp. 21-37.

[20] A. Rosebrock,Sliding windows for object detection with Python and OpenCV, 2015, PylmageSearch, Navigation.

[21] J. Long, E. Shelhamer, T. Darrell,Fully convolutional networks for semantic segmentation, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431-3440.

[22] D.-Y. Yun, S.-K. Seo, U. Zahid, C.-J. Lee, Deep neural network for automatic image recognition of engineering diagrams, Appl. Sci. 10 (11) (2020) 4005.

[23] H. Kim, W. Lee, M. Kim, Y. Moon, T. Lee, M. Cho, D. Mun, Deep-learning-based recognition of symbols and texts at an industrially applicable level from images of high-density piping and instrumentation diagrams, Expert Syst. Appl. 183 (2021) 115337.

[24] B.C. Kim, H. Kim, Y. Moon, G. Lee, D. Mun, End-to-end digitization of image format piping and instrumentation diagrams at an industrially applicable level, J. Comput. Des. Eng. 9 (4) (2022) 1298-1326.

[25] J. Illingworth, J. Kittler, A survey of the Hough transform, Comput. Vis. Graph. Image Process. 44 (1) (1988) 87-116.

[26] F.C. Akyon, S.O. Altinuc, A. Temizel, Slicing aided hyper inference and fine-tuning for small object detection, in: 2022 IEEE International Conference on Image Processing, ICIP, IEEE, 2022, pp. 966-970.

[27] G. Jocher, A. Stoken, J. Borovec, A. Chaurasia, L. Changyu, A. Hogan, J. Hajek, L. Diaconu, Y. Kwon, Y. Defretin, et al., Ultralytics/yolov5: v5. 0-YOLOv5-P6 1280 models, ly and YouTube integrations, 2021, Zenodo.

[28] Y. Baek, B. Lee, D. Han, S. Yun, H. Lee,Character region awareness for text detection, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9365-9374.

[29] S. Paliwal, A. Jain, M. Sharma, L. Vig, Digitize-PID: Automatic digitization of piping and instrumentation diagrams,in: Trends and Applications in Knowledge Discovery and Data Mining, Springer, 2021, pp. 168-180.