Automated Ship Berthing Guidance Method Based on Three-dimensional Target Measurement

Abstract

Automatic berthing guidance is an important aspect of automated ship technology to obtain the ship-shore position relationship. The current mainstream measurement methods for ship-shore position relationships are based on radar, multisensor fusion, and visual detection technologies. This paper proposes an automated ship berthing guidance method based on three-dimensional (3D) target measurement and compares it with a single-target recognition method using a binocular camera. An improved deep object pose estimation (DOPE) network is used in this method to predict the pixel coordinates of the two-dimensional (2D) keypoints of the shore target in the image. The pixel coordinates are then converted into 3D coordinates through the camera imaging principle, and an algorithm for calculating the relationship between the ship and the shore is proposed. Experiments were conducted on the improved DOPE network and the actual ship guidance performance to verify the effectiveness of the method. Results show that the proposed method with a monocular camera has high stability and accuracy and can meet the requirements of automatic berthing.

Keywords

Automated ship / Automatic berthing / Berthing guidance / 3D measurement / Neural networks / Deep learning / Position estimation

References

[1] Akbar J, Shahzad M, Malik MI, Ul-Hasan A, Shafait F (2019) Runway detection and localization in aerial images using deep learning. In 2019 IEEE Digital Image Computing: Techniques and Applications (DICTA), 1–8. https://doi.org/10.1109/DICTA47822.2019.8945889

[2] Chang JR, Chen YS (2018) Pyramid stereo matching network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5410–5418. https://doi.org/10.1109/CVPR.2018.00567

[3] Cheng Y, Li B (2021) Image segmentation technology and its application in digital image processing. In 2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), 1174–1177. https://doi.org/10.1109/IPEC51340.2021.9421206

[4] Dickmann J, Appenrodt N, Klappstein J, Bloecher HL, Muntzinger M, Sailer A, Brenk C. Making bertha see even more: Radar contribution. IEEE Access, 2015, 3: 1233-1247

[5] Dwina N, Arnia F, Munadi K (2018) Skin segmentation based on improved thresholding method. In 2018 IEEE International ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI-NCON), 95–99. https://doi.org/10.1109/ECTI-NCON.2018.8378289

[6] Fu MY, Xu YJ, Wang YH (2015) Cooperation and collision avoidance for multiple DP ships with disturbances. In 2015 IEEE 34th Chinese Control Conference (CCC), 4208–4213. https://doi.org/10.1109/ChiCC.2015.7260288

[7] Girshick R, Donahue J, Darrell T, Malik J. Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(1): 142-158

[8] Hui Y, Han Y. Advanced density peak and K-means clustering on image segmentation. In 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), 2019, 1: 2740-2744

[9] Lee SD, Tzeng CY, Kehr YZ, Huang CC, Kang CK. Autopilot system based on color recognition algorithm and internal model control scheme for controlling approaching maneuvers of a small boat. IEEE Journal of Oceanic Engineering, 2010, 35(2): 376-387

[10] Lepetit V, Moreno-Noguer F, Fua P. EP n P: An accurate O (n) solution to the P n P problem. International Journal of Computer Vision, 2009, 81: 155-166

[11] Li TY, Zhu HB (2018) Research on model control of binocular robot vision system. In 2018 IEEE Chinese Automation Congress (CAC), 1794–1797. https://doi.org/10.1109/CAC.2018.8623756

[12] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) SSD: Single shot multibox detector. In Proceedings of Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, Part I, 21–37. Springer International Publishing. https://doi.org/10.48550/arXiv.1512.02325

[13] Mazenc F, Pettersen K, Nijmeijer H. Global uniform asymptotic stabilization of an underactuated surface vessel. IEEE Transactions on Automatic Control, 2002, 47(10): 1759-1762

[14] Mi C, Huang Y, Fu C, Zhang Z, Postolache O. Vision-based measurement: actualities and developing trends in automated container terminals. IEEE Instrumentation & Measurement Magazine, 2021, 24(4): 65-76

[15] Morrical N, Tremblay J, Lin Y, Tyree S, Birchfield S, Pascucci V, Wald I (2021) NViSII: A scriptable tool for photorealistic image generation. arXiv preprint arXiv: 2105.13962. https://doi.org/10.48550/arXiv.2105.13962

[16] Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779–788. https://doi.org/10.1109/CVPR.2016.91

[17] Shao G, Ma Y, Malekian R, Yan X, Li Z. A novel cooperative platform design for coupled USV-UAV systems. IEEE Transactions on Industrial Informatics, 2019, 15(9): 4913-4922

[18] Shuo J, Yonghui Z, Wen R, Kebin T (2017) The unmanned autonomous cruise ship for water quality monitoring and sampling. In 2017 IEEE International Conference on Computer Systems, Electronics and Control (ICCSEC), 700–703. https://doi.org/10.1109/ICCSEC.2017.8447040

[19] Song Q, Lin GY, Ma JQ, Zhang HM (2016) An edge-detection method based on adaptive canny algorithm and iterative segmentation threshold. In IEEE 2016 2nd International Conference on Control Science and Systems Engineering (ICCSSE), 64–67. https://doi.org/10.1109/CCSSE.2016.7784354

[20] Sun ZY, Wang LH, Liu LQ (2020) Three-dimensional reconstruction algorithm based on inverse perspective transformation. In 2020 IEEE International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), 221–225. https://doi.org/10.1109/ICBAIE49996.2020.00053

[21] Tateno K, Tombari F, Laina I, Navab N (2017) CNN-SLAM: Real-time dense monocular slam with learned depth prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6243–6252. https://doi.org/10.1109/CVPR.2017.695

[22] Tekin B, Sinha SN, Fua P (2018) Real-time seamless single shot 6d object pose prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, 292–301. https://doi.org/10.1109/CVPR.2018.00038

[23] Tremblay J, To T, Sundaralingam B, Xiang Y, Fox D, Birchfield S (2018) Deep object pose estimation for semantic robotic grasping of household objects. arXiv preprint arXiv: 1809.10790. https://doi.org/10.48550/arXiv.1809.10790

[24] Xiang Y, Schmidt T, Narayanan V, Fox D (2017) Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv: 1711.00199. https://doi.org/10.48550/arXiv.1711.00199

[25] Yu R, Zhu Q, Xia G, Liu Z. Sliding mode tracking control of an underactuated surface vessel. IET Control Theory & Applications, 2012, 6(3): 461-466

[26] Zhao W, Zhang S, Guan Z, Zhao W, Peng J, Fan J (2020) Learning deep network for detecting 3d object keypoints and 6d poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14134–14142. https://doi.org/10.1109/CVPR42600.2020.01414