DHT_xLSTM: A hybrid temporal-mapping framework for skill-oriented grasp prediction in teleoperated robots via dual-quaternion control and context-aware learning

Abstract

This paper presents a unified teleoperation framework for heterogeneous master–slave robotic systems, integrating geometric mapping with learning-based temporal modeling to enable skillful and adaptive manipulation. To address structural asymmetry and dynamic task variability, the proposed framework introduces four key components. (1) A DHT_xLSTM-based temporal modeling framework is proposed to enable context-aware skill prediction from multi-source sequential data, supporting autoregressive reproduction and long-horizon manipulation. (2) A unified master–slave mapping scheme is established by combining task-space pose alignment and joint-space transformation, enabling skill transfer across structurally asymmetric systems. (3) A unit dual quaternion-based joint-space mapping algorithm is introduced to ensure consistent directional transfer between mismatched human and robot joints, preserving motion semantics. (4) A dynamic hybrid control strategy is designed to switch between geometric mapping and learning-based prediction based on task phase, enabling seamless transition from gross to fine manipulation. Experimental results demonstrate that the proposed framework achieves high spatial fidelity, robust temporal generalization, and autonomous transition capabilities, laying a solid foundation for intelligent human–robot collaboration in complex manipulation tasks.

Keywords

Teleoperation / Manipulation skills / Heterogeneous robotic systems / Human–robot interaction

CRediT authorship contribution statement

Lihang Feng: Project administration, Methodology, Funding acquisition. Long Zhang: Writing – review & editing, Writing – original draft, Validation, Formal analysis, Data curation, Conceptualization. Shiyue Ma: Visualization, Software. Dong Wang: Validation, Supervision, Software. Aiguo Song: Validation, Supervision, Project administration.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the Jiangsu Province Science and Technology Program (BZ2024057) and the Science and Technology Project of State Grid Corporation of China (J2025059).

References

[1] T. Fu, C. Li, Y. Bai, F. Li, J. Wu, C. Wang, R. Song, A robotic skill transfer learning framework of dynamic manipulation for fabric placement, Comput. Ind. 165 (2025) 104216.

[2] K.J. Wang, D.A. Rizqi, H.P. Nguyen, Skill transfer support model based on deep learning, J. Intell. Manuf. 32 (4) (2021) 1129-1146.

[3] Y. Hu, S. Wang, L. Feng, B. Liu, Y. Xiang, C. Li, D. Wang, Design and implementation of the high-integrated smart gloves for virtual reality interaction, Sensor Rev. 44 (6) (2024) 629-637.

[4] Z. Ren, Z. Wang, X. Liu, R. Lin, Parameterization-based trajectory planning for an 8-DOF manipulator with multiple constraints, Biomim. Intell. Robot. 5 (1) (2025) 100193.

[5] Q. Wang, R. McCarthy, D.C. Bulens, F.R. Sanchez, K. McGuinness, N.E. O’Connor, S.J. Redmond, Identifying expert behavior in offline training datasets improves behavioral cloning of robotic manipulation policies, IEEE Robot. Autom. Lett. 9 (2) (2023) 1294-1301.

[6] B. Omarali, S. Javaid, M. Valle, et al., Workspace Scaling in Virtual Reality Based Robot Teleoperation[C]//Proceedings of the Augmented Humans International Conference, vol. 2023, 2023, pp. 98-104.

[7] Y. Li, C. Guo, J. Ren, B. Chen, C. Cheng, H. Zhang, H. Lu, SoftGrasp: Adaptive grasping for dexterous hand based on multimodal imitation learning, Biomim. Intell. Robot. 5 (2) (2025) 100217.

[8] S.K. Hsia, Y.H. Chuang, C.W. Chen, Auto-tuned motion scaling in teleoperation based on human reaction model identification, IEEE Robot. Autom. Lett. 7 (1) (2022) 318-325.

[9] H. Chen, B. Yang, L. Li, T. Liu, J. Zhang, Y. Zhang, Optimization-based UWB positioning with multiple tags for estimating position and rotation simultaneously, Biomim. Intell. Robot. 5 (2) (2025) 100210.

[10] C. Li, X. Lin, Z. Liao, H. Wu, Z. Xu, X. Zhou, A fast registration method for multi-view point clouds with low overlap in robotic measurement, Biomim. Intell. Robot. 5 (2) (2025) 100195.

[11] L. Chen, P. Lai, Y. Wang, Y. Dong, Fuzzy adaptive variable impedance control on deformable shield of defecation smart care robot, Biomim. Intell. Robot. 5 (2) (2025) 100214.

[12] R. Rahmatizadeh, P. Abolghasemi, A. Behal, L. Bölöni, From virtual demonstration to real-world manipulation using LSTM and MDN, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, (2018).

[13] S. Scherzinger, A. Roennau, R. Dillmann, Contact skill imitation learning for robot-independent assembly programming, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, (IROS), IEEE, 2019, pp. 4309-4316.

[14] E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguer, Context-aware human motion prediction, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6992-7001.

[15] N. Hashemi, M. Mose, L.R. Østergaard, F. Bjerrum, M. Hashemi, M.B. Svendsen, et al., Video-based robotic surgical action recognition and skills assessment on porcine models using deep learning, Surg. Endosc. 39 (3) (2025) 1709-1719.

[16] X. Liang, Y. Yan, W. Wang, T. Su, G. He, G. Li, Z.G. Hou, Adaptive human–robot interaction torque estimation with high accuracy and strong tracking ability for a lower limb rehabilitation robot, IEEE/ASME Trans. Mechatronics 29 (6) (2024) 4814-4825.

[17] K. Shi, A. Song, Y. Li, H. Li, D. Chen, L. Zhu, A cable-driven three-DOF wrist rehabilitation exoskeleton with improved performance, Front. Neurorobotics 15 (2021) 664062.

[18] X. Hu, A. Song, J. Wang, H. Zeng, W. Wei, Finger movement recognition via high-density electromyography of intrinsic and extrinsic hand muscles, Sci. Data 9 (1) (2022) 373.

[19] X. Hu, A. Song, J. Wang, H. Zeng, W. Wei, Finger movement recognition via high-density electromyography of intrinsic and extrinsic hand muscles, Sci. Data 9 (1) (2022) 373.

[20] B. Ames, J. Morgan, G. Konidaris, Ikflow: Generating diverse inverse kinematics solutions, IEEE Robot. Autom. Lett. 7 (3) (2022) 7177-7184.

[21] M. Alharthi, A. Mahmood, Xlstmtime: Long-term time series forecasting with xlstm, AI 5 (3) (2024) 1482-1495.

[22] L. Jia, M. Shi, J. Shi, D. Wang, A. Song, C. Su, L. Feng, Design and calibration of a novel multiaxis sensor for measuring the wheel–terrain interaction forces of robotic vehicles, Sensor Rev. 45 (2) (2025) 207-220.

[23] L. Jia, L. Feng, J. Shi, D. Wang, G. Zhang, C.Y. Su, An improved robotic arm constant force control method for forward terrain sensing system on planetary rovers, Int. J. Adv. Robot. Syst. 21 (6) (2024) 17298806241305883.

[24] M. Shi, L. Feng, L. Jia, A. Song, A new multi-axis force sensor for measuring the wheel-terrain interaction ahead of the robotic vehicles, International Conference on Social Robotics, Springer Nature Singapore, Singapore, 2024, pp. 61-71.