Boosting Embodied Visual Localisation Through Multi-Granular Semantics and Multi-Robot Consensus

Wenshuai Wang , Shiyu Liu , Peifeng Jiang , Hong Liu , Runwei Ding

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) : 1088 -1103.

PDF (3426KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) :1088 -1103. DOI: 10.1049/cit2.70152
ORIGINAL RESEARCH
research-article
Boosting Embodied Visual Localisation Through Multi-Granular Semantics and Multi-Robot Consensus
Author information +
History +
PDF (3426KB)

Abstract

Achieving high accuracy and synergy remains extremely difficult for multi-robot embodied visual localisation, which suffers from persistent real-world challenges such as viewpoint ambiguity, appearance variation and dynamic occlusion. Conventional optimisation-based methods often lead to incorrect feature matching without domain adaptation, whereas existing deep learning models are limited in both precision and generalisability for pose regression. Although semantic information can offer a cognitive-level scene representation, its potential has not been fully exploited due to inconsistent labelling and scaling. To address these issues, this paper proposes a novel semantic and synergic consensus framework (termed SSC-MR) that enhances image retrieval and pose estimation by leveraging multi-granular semantics and multi-robot consensus. Specifically, the spatial topology is attentively generated as a global scene graph for fast image retrieval across heterogeneous robots in a holistic workspace, whereas a local semantic-guided pose estimation strategy refines matched keypoints under semantic consistency constraints. To further improve the efficiency of multi-robot collaboration, an intra-robot minimum redundancy consensus module selects informative keyframes and compresses feature representations to reduce perceptual redundancy, and an inter-robot maximum correlation consensus module further establishes high-confidence state associations among interacting robots for knowledge sharing. Extensive experiments on five public datasets demonstrate that the proposed SSC-MR obtains superior effectiveness and robustness across diverse indoor and outdoor scenes, providing a reliable and scalable solution for embodied visual localisation in practical robotic applications. Notably, it achieves a 62.1% improvement in localisation accuracy with nearly 22× faster speed compared to a typical distributed architecture for 10 collaborative robots.

Keywords

image retrieval / multi-robot system / pose estimation / semantic consistency / visual localisation

Cite this article

Download citation ▾
Wenshuai Wang, Shiyu Liu, Peifeng Jiang, Hong Liu, Runwei Ding. Boosting Embodied Visual Localisation Through Multi-Granular Semantics and Multi-Robot Consensus. CAAI Transactions on Intelligence Technology, 2026, 11 (4) : 1088-1103 DOI:10.1049/cit2.70152

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

L. Chen, C. Liang, S. Yuan, M. Cao, and L. Xie, “ Relative Localizability and Localization for Multirobot Systems,” IEEE Transactions on Robotics 41, no. 1 (2025): 2931-2949, https://doi.org/10.1109/tro.2025.3544103.

[2]

D. S. Drew, “ Multi-Agent Systems for Search and Rescue Applications,” Current Robotics Reports 2, no. 2 (2021): 189-200, https://doi.org/10.1007/s43154-021-00048-3.

[3]

M. Xu, Y. Wang, B. Xu, et al., “ A Critical Analysis of Image-Based Camera Pose Estimation Techniques,” Neurocomputing 570 (2024): 127125, https://doi.org/10.1016/j.neucom.2023.127125.

[4]

C. Chen, B. Wang, C. X. Lu, N. Trigoni, and A. Markham, “ Deep Learning for Visual Localization and Mapping: A Survey,” IEEE Transactions on Neural Networks and Learning Systems 35, no. 12 (2023): 17000-17020, https://doi.org/10.1109/tnnls.2023.3309809.

[5]

J. Miao, K. Jiang, T. Wen, et al., “ A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation,” IEEE Transactions on Intelligent Vehicles 10, no. 4 (2024): 2519-2550, https://doi.org/10.1109/tiv.2024.3378716.

[6]

W. Chen, W. Chi, S. Ji, et al., “ A Survey of Autonomous Robots and multi-robot Navigation: Perception, Planning and Collaboration,” Biomimetic Intelligence and Robotics 5, no. 2 (2025): 100203, https://doi.org/10.1016/j.birob.2024.100203.

[7]

H. Wan, Y. Chen, Y. Deng, et al., “ Embodiedagent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Control,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (2025), 12140-12146.

[8]

S. Wan, W. Zou, J. Dong, and Y. Gao, “ Dual Strategies Consensus Reaching Process for Ranking Consensus Based Probabilistic Linguistic Multi-Criteria Group Decision-Making Method,” Expert Systems with Applications 262, no. 1 (2025): 125342, https://doi.org/10.1016/j.eswa.2024.125342.

[9]

W. Burger and M. J. Burge, “ Scale-Invariant Feature Transform (SIFT),” in Digital Image Processing: An Algorithmic Introduction (Springer, 2022), 709-763.

[10]

J. Shotton, B. Glocker, C. Zach, et al., “ Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2013), 2930-2937.

[11]

A. Kendall, M. Grimes, and R. Cipolla, “ POSENET: A Convolutional Network for Real-Time 6-DoF Camera Relocalization,” in Proceedings of the IEEE International Conference on Computer Vision (2015), 2938-2946.

[12]

H. Taira, M. Okutomi, T. Sattler, et al., “ INLOC: Indoor Visual Localization With Dense Matching and View Synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 7199-7209.

[13]

C. Toft, W. Maddern, A. Torii, et al., “ Long-Term Visual Localization Revisited,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 4 (2020): 2074-2088, https://doi.org/10.1109/tpami.2020.3032010.

[14]

M. A. Fischler and R. C. Bolles, “ Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography,” Communications of the ACM 24, no. 6 (1981): 381-395, https://doi.org/10.1145/358669.358692.

[15]

S. Chen, Z. Wang, and V. Prisacariu, “ Direct-Posenet: Absolute Pose Regression With Photometric Consistency,” in 2021 International Conference on 3D Vision (2021), 1175-1185.

[16]

P. E. Sarlin, A. Unagar, M. Larsson, et al., “ Back to the Feature: Learning Robust Camera Localization From Pixels to Pose,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), 3247-3257.

[17]

V. Polizzi, M. Cannici, D. Scaramuzza, et al., “ FAVOR: Features via Voxel Rendering for Camera Relocalization,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2025), 44-53.

[18]

E. Brachmann, A. Krull, S. Nowozin, et al., “ DSAC-Differentiable Ransac for Camera Localization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), 6684-6692.

[19]

E. Brachmann and C. Rother, “ Learning Less is More-6D Camera Localization via 3D Surface Regression,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 4654-4662.

[20]

E. Brachmann and C. Rother, “ Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 9 (2021): 5847-5865, https://doi.org/10.1109/TPAMI.2021.3070754.

[21]

Z. Huang, H. Zhou, Y. Li, et al., “ VS-NET: Voting With Segmentation for Visual Localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), 6101-6111.

[22]

X. Li, S. Wang, Y. Zhao, et al., “ Hierarchical Scene Coordinate Classification and Regression for Visual Localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), 11983-11992.

[23]

S. Wang, Z. Laskar, I. Melekhov, et al., “ HSCNet++: Hierarchical Scene Coordinate Classification and Regression for Visual Localization With Transformer,” International Journal of Computer Vision 132, no. 7 (2024): 2530-2550, https://doi.org/10.1007/s11263-023-01982-9.

[24]

J. Nie, L. Cai, Q. Geng, et al., “ Understanding Matters: Semantic-Structural Determined Visual Relocalization for Large Scenes,” in Proceedings of the International Joint Conference on Artificial Intelligence (2025), 8759-8767.

[25]

P. E. Sarlin, C. Cadena, R. Siegwart, et al., “ From Coarse to Fine: Robust Hierarchical Localization at Large Scale,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), 12716-12725.

[26]

F. Xue, X. Wu, S. Cai, et al., “ Learning Multi-View Camera Relocalization With Graph Neural Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), 11372-11381.

[27]

Q. Zhou, T. Sattler, M. Pollefeys, et al., “ To Learn or Not to Learn: Visual Localization From Essential Matrices,” in Proceedings of the IEEE International Conference on Robotics and Automation (2020), 3319-3326.

[28]

F. Khatib, Y. Margalit, M. Galun, et al., “ Leveraging Image Matching Toward end-to-end Relative Camera Pose Regression,” in DAGM German Conference on Pattern Recognition (Springer, 2024), 185-201.

[29]

K. T. Giang, S. Song, and S. Jo, “ Learning to Produce Semi-Dense Correspondences for Visual Localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024), 19468-19478.

[30]

A. Jaenal, P. C. Cubero, J. Araújo, et al., “ Towards Visual Localization Interoperability: Cross-Feature for Collaborative Visual Localization and Mapping,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (2025), 26783-26792.

[31]

N. Kobyshev, H. Riemenschneider, and L. Van Gool, “ Matching Features Correctly Through Semantic Understanding,” in Proceedings of the IEEE International Conference on 3D Vision (2014), 472-479.

[32]

C. Toft, E. Stenborg, L. Hammarstrand, et al., “ Semantic Match Consistency for Long-Term Visual Localization,” in Proceedings of the European Conference on Computer Vision (Springer, 2018), 383-399.

[33]

T. Wang, X. Mao, C. Zhu, et al., “ Embodiedscan: A Holistic Multi-Modal 3d Perception Suite Towards Embodied AI,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024), 19757-19767.

[34]

S. Wu, H. Fei, and T. S. Chua, “ Universal Scene Graph Generation,” in Proceedings of the Computer Vision and Pattern Recognition Conference (2025), 14158-14168.

[35]

I. Armeni, Z. Y. He, J. Gwak, et al., “ 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (2019), 5664-5673.

[36]

N. Hughes, Y. Chang, and L. Carlone, “ Hydra: A Real-Time Spatial Perception System for 3D Scene Graph Construction and Optimization,” in Proceedings of the Robotics: Science and Systems Conference, vol. 50 (2022), 1-13.

[37]

N. Hughes, Y. Chang, S. Hu, et al., “ Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-Time Systems,” International Journal of Robotics Research 43, no. 10 (2024): 1457-1505, https://doi.org/10.1177/02783649241229725.

[38]

J. Zhang, G. Zhu, S. Li, et al., “ Multiview Scene Graph,” in Proceedings of the Advances in Neural Information Processing Systems, vol. 37 (2024): 17761-17788.

[39]

X. Liu, S. Wen, J. Zhao, T. Z. Qiu, and H. Zhang, “ Edge-Assisted Multi-Robot Visual-Inertial SLAM With Efficient Communication,” IEEE Transactions on Automation Science and Engineering 22 (2024): 2186-2198, https://doi.org/10.1109/tase.2024.3376427.

[40]

T. K. Tasooji and R. Parasuraman, “ Distributed Fault-Tolerant Multi-Robot Cooperative Localization in Adversarial Environments,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (2025), 4626-4633.

[41]

Y. Zhou, J. Xiao, Y. Zhou, and G. Loianno, “ Multi-Robot Collaborative Perception With Graph Neural Networks,” IEEE Robotics and Automation Letters 7, no. 2 (2022): 2289-2296, https://doi.org/10.1109/lra.2022.3141661.

[42]

F. Wang, C. Zhang, W. Zhang, et al., “ Semantic-Geometric Triple Constraint-Based Graph Matching for Robust Multi-Robot Global Localization in Large-Scale Environments,” IEEE Robotics and Automation Letters (2025).

[43]

G. G. de Castro, T. M. Santos, F. A. Andrade, et al., “ Heterogeneous Multi-Robot Collaboration for Coverage Path Planning in Partially Known Dynamic Environments,” Machines 12, no. 3 (2024): 200, https://doi.org/10.3390/machines12030200.

[44]

M. Oquab, T. Darcet, T. Moutakanni, et al., “ DINOv2: Learning Robust Visual Features Without Supervision,” Transactions on Machine Learning Research (2024): 1-32.

[45]

S. Liu, Z. Zeng, T. Ren, et al., “ Grounding DINO: Marrying Dino With Grounded Pre-Training for Open-Set Object Detection,” in Proceedings of the European Conference on Computer Vision (Springer, 2024), 38-55.

[46]

A. Kirillov, E. Mintun, N. Ravi, et al., “ Segment Anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (2023), 4015-4026.

[47]

F. Distel, I. Bloch, and J. Atif, “ Concept Dissimilarity Based on Tree Edit Distance and Morphological Dilation,” in Proceedings of the European Conference on Artificial Intelligence (IOS Press, 2014), 249-254.

[48]

A. Kaplan, T. Avraham, and M. Lindenbaum, “ Interpreting the Ratio Criterion for Matching SIFT Descriptors,” in Proceedings of the European Conference on Computer Vision (Springer, 2016), 697-712.

[49]

R. Lu, Z. Zhu, S. Fu, et al., “ Self-Supervised Camera Relocalization With Hierarchical Fern Encoding,” IEEE Transactions on Instrumentation and Measurement 73, no. 1 (2023): 1-12, https://doi.org/10.1109/tim.2023.3347802.

[50]

G. Baruch, Z. Chen, A. Dehghan, et al., “ ARKitScenes: A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data,” in Proceedings of the Advances in Neural Information Processing Systems (2021), 1-12.

[51]

L. Gurobi Optimization, Gurobi Optimizer Reference Manual (2024), https://www.gurobi.com.

[52]

E. Rublee, V. Rabaud, K. Konolige, et al., “ ORB: An Efficient Alternative to SIFT or SURF,” in Proceedings of the IEEE International Conference on Computer Vision (2011), 2564-2571.

[53]

D. DeTone, T. Malisiewicz, and A. Rabinovich, “ Superpoint: Self-Supervised Interest Point Detection and Description,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2018), 224-236.

[54]

M. Tyszkiewicz, P. Fua, and E. Trulls, “ DISK: Learning Local Features With Policy Gradient,” in Proceedings of the Advances in Neural Information Processing Systems, vol. 33 (2020): 14254-14265.

[55]

J. Sun, Z. Shen, Y. Wang, et al., “ LoFTR: Detector-Free Local Feature Matching With Transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), 8922-8931.

PDF (3426KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉