Two-stage matching based visual relocalization for indoor mobile robots

Abstract

Indoor positioning is crucial for autonomous robots, especially in indoor human-robot collaborative laboratories where moderate personnel activity, portable experimental equipment, illumination variation, and local weak-texture regions coexist. However, accurate indoor positioning remains challenging because the robot must maintain reliable perception under dynamic interference, non-structural clutter, and appearance changes. This paper presents a two-stage matching-based visual relocalization framework for indoor mobile robots operating in pre-mapped environments. First, a multi-scale variational autoencoder latent representation model is developed to evaluate scene similarity between images, enabling the robot to recognize the current scene from a pre-constructed image-based map and thereby achieve coarse localization. Then, a SuperGlue-based keypoint matching approach is utilized to calculate the relative pose between the current image and a reference image with known global pose, yielding precise global localization. Experimental results in three representative challenging scenes demonstrate that the proposed framework achieves a coarse localization accuracy of 99.00%. For precise localization, it attains mean absolute translation errors of 0.076 m, 0.065 m, and 0.033 m along the X, Y, and Z axes, respectively, as well as average rotation errors of 0.66°, 0.70°, and 0.73° in roll, pitch, and yaw, respectively.

Keywords

visual relocalization / deep learning / scene recognition / variational autoencoder / feature matching

References

[1] XIAO Z Y, YANG D G, WEN T P, et al. Monocular localization with vector HD map (MLVHM): A low-cost method for commercial IVs. Sensors, 2020, 20(7): Article 1870.

[2] LI J, WANG C Y, KANG X J, et al. Camera localization for augmented reality and indoor positioning: A vision-based 3D feature database approach. International Journal of Digital Earth, 2020, 13(6): 727-741.

[3] ROYER E, LHUILLIER M, DHOME M, et al. Monocular vision for mobile robot localization and autonomous navigation. International Journal of Computer Vision, 2007, 74(3): 237-260.

[4] WOLCOTT R W, EUSTICE R M . Visual localization within LiDAR maps for automated urban driving. Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, Sept 14-18, Chicago, IL, USA. Piscataway, NJ, USA: IEEE, 2014: 176-183.

[5] ANOOSHEH A, SATTLER T, TIMOFTE R, et al. Night-to-day image translation for retrieval-based localization. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA'19), 2019, May 20-24, Montreal, Canada. Piscataway, NJ, USA: IEEE, 2019: 5958-5964.

[6] SUN M H, YANG S W, LIU H Z . Scale-aware camera localization in 3D LiDAR maps with a monocular visual odometry. Computer Animation and Virtual Worlds, 2019, 30(3/4): Article e1879.

[7] CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. ORB-SLAM3: An accurate open-source library for visual, visual-Inertial, and multimap SLAM. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.

[8] MO J W, ISLAM M J, SATTAR J . Fast direct stereo visual SLAM. IEEE Robotics and Automation Letters, 2022, 7(2): 778-785.

[9] CASELITZ T, STEDER B, RUHNKE M, et al. Monocular camera localization in 3D LiDAR maps. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'16), 2016, Oct 9-14, Daejeon, Republic of Korea. Piscataway, NJ, USA: IEEE, 2016: 1926-1931.

[10] SHOTTON J, GLOCKER B, ZACH C, et al. Scene coordinate regression forests for camera relocalization in RGB-D images. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013, Jun 23-28, Portland, OR, USA. Piscataway, NJ, USA: IEEE, 2013: 2930-2937.

[11] ZAMIR A R, SHAH M . Accurate image localization based on google maps street view. Computer vision: Proceedings of the 11th European Conference on Computer Vision (ECCV'10), 2010, Sept 5-11, Heraklion, Crete, Greece. Berlin, Germany: Springer, 2010: 255-268.

[12] RANGANATHAN A, ILSTRUP D, WU T . Light-weight localization for vehicles using road markings. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013, Nov 3-7, Tokyo, Japan. Piscataway, NJ, USA: IEEE, 2013: 921-927.

[13] LIU L, LI H D, DAI Y C . Efficient global 2D-3D matching for camera localization in a large-scale 3D map. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV'17), 2017, Oct 22-29, Venice, Italy. Piscataway, NJ, USA: IEEE, 2017: 2372-2381.

[14] LI Y P, SNAVELY N, HUTTENLOCHER D P . Location recognition using prioritized feature matching. Computer Vision: Proceedings of the 11th European Conference on Computer Vision (ECCV'10), 2010, Sept 5-11, Heraklion, Crete, Greece. Berlin, Germany: Springer, 2010: 791-804.

[15] SATTLER T, LEIBE B, KOBBELT L . Efficient & effective prioritized matching for large-scale image-based localization. IEEE Transactions on pattern analysis and machine intelligence, 2016, 39(9): 1744-1756.

[16] LI Y P, SNAVELY N, HUTTENLOCHER D P, et al. Worldwide pose estimation using 3D point cloud. Proceedings of the 12th European Conference on Computer Vision (ECCV'12), 2012, Oct 7-13, Florence, Italy. Berlin, Germany: Springer, 2012: 147-163.

[17] GALVEZ-LOPEZ D, TARDOS J D . Real-time loop detection with bags of binary words. Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2011, Sept 25-30, San Francisco, CA, USA. Piscataway, NJ, USA: IEEE, 2011: 51-58.

[18] LASKAR Z, MELEKHOV I, KALIA S, et al. Camera relocalization by computing pairwise relative poses using convolutional neural network. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW'17), 2017, Oct 22-29, Venice, Italy. Piscataway, NJ, USA: IEEE, 2017: 929-938.

[19] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16), 2016, Jun 27-30, Las Vegas, NV, USA. Piscataway, NJ, USA: IEEE, 2016: 770-778.

[20] MA L Q, MENG D Y, ZHAO S H, et al. Visual localization with a monocular camera for unmanned aerial vehicle based on landmark detection and tracking using YOLOv5 and DeepSORT. International Journal of Advanced Robotic Systems, 2023, 20(3): doi: 10.1177/17298806231164831.

[21] VON STUMBERG L, WENZEL P, YANG N, et al. LM-reloc: Levenberg-marquardt based direct visual relocalization. Proceedings of the 2020 International Conference on 3D Vision (3DV'20), 2020, Nov 25-28, Fukuoka, Japan. Piscataway, NJ, USA: IEEE, 2020: 968-977.

[22] SAHA S, VARMA G, JAWAHAR C . Improved visual relocalization by discovering anchor points. arXiv Preprint, arXiv:1811.04370, 2018.

[23] SARLIN P E, UNAGAR A, LARSSON M, et al. Back to the feature: Learning robust camera localization from pixels to pose. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'21), 2021, Jun 20-25, Nashville, TN, USA. Piscataway, NJ, USA: IEEE, 2021: 3247-3257.

[24] ARNOLD E, WYNN J, VICENTE S, et al. Map-free visual relocalization: Metric pose relative to a single image. Computer Vision: Proceedings of the 17th European Conference on Computer Vision (ECCV'22), 2022, Oct 23-27, Tel Aviv, Israel. Berlin, Germany: Springer, 2022: 690-708.

[25] SHUBODH S, OMAMA M, ZAIDI H, et al. LIP-Loc: LiDAR image pretraining for cross-modal localization. Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW'24), 2024, Jan 1-6, Waikoloa, HI, USA. Piscataway, NJ, USA: IEEE, 2024: 948-957.

[26] WANG S Z, LASKAR Z, MELEKHOV I, et al. HSCNet++: Hierarchical scene coordinate classification and regression for visual localization with transformer. International Journal of Computer Vision, 2024, 132(7): 2530-2550.

[27] DE TONE D, MALISIEWICZ T, RABINOVICH A . Superpoint: Self-supervised interest point detection and description. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW'18), 2018, Jun 18-22, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 224-236.

[28] SARLIN P E, DE TONE D, MALISIEWICZ T, et al. Superglue: Learning feature matching with graph neural networks. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'20), 2020, Jun 13-19, Seattle, WA, USA. Piscataway, NJ, USA: IEEE, 2020: 4938-4947.

[29] MUR-ARTAL R, TARDÓS J D . ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on robotics, 2017, 33(5): 1255-1262.

[30] WEI Y C, XIAO H X, SHI H H, et al. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'18), 2018, Jun 18-23, Salt Lake City, UT, USA. Piscataway, NJ, USA: IEEE, 2018: 7268-7277.

[31] WU H Z, LIU S Y, CHENG C, et al. Multiscale variational autoencoder aided convolutional neural network for pose estimation of tunneling machine using a single monocular image. IEEE Transactions on Industrial Informatics, 2022, 18(8): 5161-5170.

[32] KINGMA D P . Auto-encoding variational Bayes. arXiv Preprint, arXiv:1312.6114, 2013.

[33] HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV'17), 2017, Oct 22-29, Venice, Italy. Piscataway, NJ, USA: IEEE, 2017: 2961-2969.

[34] CHANG A, DAI A, FUNKHOUSER T . Matterport3D: Learning from RGB-D data in indoor environments. arXiv Preprint, arXiv:1709.06158, 2017.

[35] SARWAR A, MEHMOOD Z, SABA T, et al. A novel method for content-based image retrieval to improve the effectiveness of the bag-of-words model using a support vector machine. Journal of Information Science, 2019, 45(1): 117-135.

[36] VEDALDI A, FULKERSON B. VLFeat: An open and portable library of computer vision algorithms. Proceedings of the 18th ACM International Conference on Multimedia, 2010, Oct 25-29, Firenze, Italy. New York, NY, USA: ACM, 2010: 1469-1472.

[37] MISTRY D, BANERJEE A . Comparison of feature detection and matching approaches: SIFT and SURF. Global Research and Development Journal for Engineering, 2017, 2(4): 7-13.