RAGLRO: Retrieval-Augmented Generation With Large Language Models for Robotic Operations

Abstract

To enable autonomous operations in complex industrial environments, this paper proposes retrieval-augmented generation with large language models for robotic operations (RAGLRO), a robotic framework specifically designed for power switchgear operation tasks. The system integrates multimodal perception with high-level semantic reasoning and task-level action generation. A depth camera captures the environmental context, which is processed by visual modules to perform object detection and pose detection. The perception outputs are formulated into structured prompts and provided to a large language model (LLM) equipped with a retrieval-augmented generation (RAG) mechanism. The RAG component enables the LLM to dynamically access a task-specific knowledge base, including operation manuals, safety protocols and historical mission data, thereby enhancing contextual understanding and reasoning precision. Based on the retrieved knowledge and current environmental perception, the LLM selects and sequences callable action functions from a predefined robotic action library to generate executable robot control commands. A dedicated dataset for power switchgear operations is also constructed to support robust visual perception, containing annotated images for object detection and pose detection tasks. Experimental results demonstrate that RAGLRO achieves high task success rates and strong adaptability in real-world power maintenance scenarios, validating the effectiveness of integrating multimodal perception, LLM-based reasoning and RAG-grounded task planning within a unified robotic control framework.

Keywords

artificial intelligence / intelligent robots / natural languages / robotics

Acknowledgements

This work was supported in part by the National Key R&D Program of China (Grant No. 2023YFA1008500), the National Natural Science Foundation of China (NSFC) under Grant No. U22B2035 and the Jiangsu Funding Program for Excellent Postdoctoral Talent. This work was also supported by Shenzhen Science and Technology Innovation Committee (Grant No. RCBS20231211090749086).

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

[1] J. Martinez-Gomez, B. Caputo, M. Cazorla, et al., Where Are We After Five Editions?: Robot Vision Challenge, a Competition That Evaluates Solutions for the Visual Place Classification Problem,” IEEE Robotics and Automation Magazine 22, no. 4 (2015): 147-156, https://doi.org/10.1109/mra.2015.2460931.

[2] I. Ullah, D. Adhikari, H. Khan, M. Shahid Anwar, S. Ahmad, and X. Bai, “Mobile Robot Localization: Current Challenges and Future Prospective,” Computer Science Review 53 (2024): 100651, https://doi.org/10.1016/j.cosrev.2024.100651.

[3] C. Siagian and L. Itti, “Biologically Inspired Mobile Robot Vision Localization,” IEEE Transactions on Robotics 25, no. 4 (2009): 861-873, https://doi.org/10.1109/TRO.2009.2022424.

[4] C. Belta, A. Bicchi, M. Egerstedt, E. Frazzoli, E. Klavins, and G. J. Pappas, “Symbolic Planning and Control of Robot Motion [Grand Challenges of Robotics],” IEEE Robotics and Automation Magazine 14, no. 1 (2007): 61-70, https://doi.org/10.1109/mra.2007.339624.

[5] N. Simaan, R. H. Taylor, and H. Choset, “Intelligent Surgical Robots With Situational Awareness,” Mechanical Engineering 137, no. 9 (2015): S3-S6, https://doi.org/10.1115/1.2015-sep-6.

[6] J. Achiam, S. Adler, S. Agarwal, et al., Gpt-4 Technical Report,” arXiv preprint arXiv:2303.08774, (2023).

[7] Team G. L. M., A. Zeng, B. Xu, et al., Chatglm: A Family of Large Language Models From glm-130b to glm-4 all Tools,” arXiv preprint arXiv:2406.12793, (2024).

[8] H. Touvron, T. Lavril, G. Izacard, et al., Llama: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, (2023).

[9] J. Yao, W. Xu, J. Lian, X. Wang, X. Yi, and X. Xie, “Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations,” arXiv preprint arXiv:2311.10779, (2023).

[10] X. Gong, M. Liu, and X. Chen . Large Language Models With Knowledge Domain Partitioning for Specialized Domain Knowledge Concentration, (2024).

[11] H. Li, Q. Ai, J. Chen, et al., Blade: Enhancing Black-Box Large Language Models With Small Domain-Specific Models,” arXiv preprint arXiv:2403.18365, (2024).

[12] C. Wang, X. Liu, Y. Yue, et al., Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity,” arXiv preprint arXiv:2310.07521, (2023).

[13] B. L. Brumitt and A. Stentz, “Dynamic Mission Planning for Multiple Mobile Robots,” in Proceedings of IEEE International Conference on Robotics and Automation, Vol. 3 (IEEE, 1996), 2396-2401, https://doi.org/10.1109/robot.1996.506522.

[14] J. Liang, W. Huang, F. Xia, et al., Code as Policies: Language Model Programs for Embodied Control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2023), 9493-9500.

[15] K. Yamao, D. Kanaoka, K. Isomoto, A. Mizutani, Y. Tanaka, and H. Tamukoh, “Development of a Saycan-Based Task Planning System Capable of Handling Abstract Nouns,” in Proceedings of International Conference on Artificial Life & Robotics (ICAROB2024) (ALife Robotics, 2024), OS15-OS24.

[16] J. Lindley and R. Whitham, “From Prompt Engineering to Prompt Craft,” in Proceedings of the Nineteenth International Conference on Tangible, Embedded, and Embodied Interaction, (2025), 1-12.

[17] M. Arslan, H. Ghanem, S. Munawar, and C. Cruz, “A Survey on Rag With Llms,” Procedia Computer Science 246 (2024): 3781-3790, https://doi.org/10.1016/j.procs.2024.09.178.

[18] N. Wiratunga, R. Abeyratne, L. Jayawardena, et al., Cbr-Rag: Case-Based Reasoning for Retrieval Augmented Generation in Llms for Legal Question Answering,” in International Conference on Case-based Reasoning (Springer, 2024), 445-460.

[19] P. Jiang, D. Ergu, F. Liu, Y. Cai, and B. Ma, “A Review of Yolo Algorithm Developments,” Procedia Computer Science 199 (2022): 1066-1073, https://doi.org/10.1016/j.procs.2022.01.135.

[20] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE International Conference on Computer Vision, (2015), 1440-1448.

[21] W. Liu, D. Anguelov, D. Erhan, et al., Ssd: Single Shot Multibox Detector,” in Computer Vision-ECCV 2016: 14Th European Conference, Amsterdam, the Netherlands, October 11-14, 2016, Proceedings, Part I 14 (Springer, 2016), 21-37.

[22] A. Pravallika, M. F. Hashmi, and A. Gupta, “Deep Learning Frontiers in 3d Object Detection: A Comprehensive Review for Autonomous Driving,” IEEE Access (2024).

[23] S. Hoque, Md Yasir Arafat, S. Xu, A. Maiti, and Y. Wei, “A Comprehensive Review on 3d Object Detection and 6d Pose Estimation With Deep Learning,” IEEE Access 9 (2021): 143746-143770, https://doi.org/10.1109/access.2021.3114399.

[24] X. Zhao, Q. Li, Y. Chao, Q. Wang, Z. He, and D. Liang, “Rt-Less: A Multi-Scene Rgb Dataset for 6d Pose Estimation of Reflective Texture-Less Objects,” Visual Computer 40, no. 8 (2024): 5187-5200, https://doi.org/10.1007/s00371-023-03097-1.

[25] J. Duan, J. Wang, Q. Fu, et al. , “Polarization-Based Reflection Suppression Method and Its Application to Target Detection,” Photonics, 11, no. 5 (2024): 445: MDPI, https://doi.org/10.3390/photonics11050445.

[26] C. Wang, D. Xu, Y. Zhu, et al., Densefusion: 6D Object Pose Estimation by Iterative Dense Fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 3343-3352.

[27] Yu Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A Convolutional Neural Network for 6d Object Pose Estimation in Cluttered Scenes,” arXiv preprint arXiv:1711.00199, (2017).

[28] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun, “Pvn3d: A Deep Point-Wise 3d Keypoints Voting Network for 6dof Pose Estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), 11632-11641.

[29] M. Hindi, L. Mohammed, O. Maaz, and A. Alwarafy, “Enhancing the Precision and Interpretability of Retrieval-Augmented Generation (Rag) in Legal Technology: A Survey,” IEEE Access (2025).

[30] L. König, S. Mostaghim, and H. Schmeck, “Decentralized Evolution of Robotic Behavior Using Finite State Machines,” International Journal of Intelligent Computing and Cybernetics 2, no. 4 (2009): 695-723, https://doi.org/10.1108/17563780911005845.

[31] A. Marzinotto, M. Colledanchise, C. Smith, and P. Ögren, “Towards a Unified Behavior Trees Framework for Robot Control,” in 2014 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2014), 5420-5427.

[32] Y. Ding, W. Xu, Z. Liu, Z. Zhou, and D. T. Pham, “Robotic Task Oriented Knowledge Graph for Human-Robot Collaboration in Disassembly,” Procedia CIRP 83 (2019): 105-110, https://doi.org/10.1016/j.procir.2019.03.121.

[33] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation Learning: A Survey of Learning Methods,” ACM Computing Surveys 50, no. 2 (2017): 1-35, https://doi.org/10.1145/3054912.

[34] L. Pack Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement Learning: A Survey,” Journal of Artificial Intelligence Research 4 (1996): 237-285, https://doi.org/10.1613/jair.301.

[35] C. Zhou, B. Huang, and P. Fränti, “A Review of Motion Planning Algorithms for Intelligent Robots,” Journal of Intelligent Manufacturing 33, no. 2 (2022): 387-424, https://doi.org/10.1007/s10845-021-01867-z.

[36] M. Tavassoli, S. Katyara, M. Pozzi, N. Deshpande, D. G. Caldwell, and D. Prattichizzo, “Learning Skills From Demonstrations: A Trend From Motion Primitives to Experience Abstraction,” IEEE Transactions on Cognitive and Developmental Systems 16, no. 1 (2023): 57-74, https://doi.org/10.1109/tcds.2023.3296166.