SFK: Shape- and Function-Grounded Keypoint Representation for Sequential Manipulation

Yaxin Liu , Yan Liu , Xinyu Song , Ruiqing Han , Haiyong Liu , Bo Huang , Yufeng Yao , Ming Zhong

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) : 1396 -1412.

PDF (4973KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) :1396 -1412. DOI: 10.1049/cit2.70136
ORIGINAL RESEARCH
research-article
SFK: Shape- and Function-Grounded Keypoint Representation for Sequential Manipulation
Author information +
History +
PDF (4973KB)

Abstract

Sequential manipulation is the process by which robots perform multiple interdependent steps to accomplish composite tasks, demanding tight integration of perception, planning and execution. Existing methods incorporate explicit features such as category, semantics, 6D pose or affordance to enhance consistency, yet single-feature representations limit generalisation and adaptability across diverse task stages. To overcome these limitations, we propose a shape- and function-grounded keypoint (SFK) representation that provides interpretable, task-relevant object abstractions. The SFK representation preserves topological consistency, enabling the inference of functional semantics, 6D pose, spatial constraints and task-oriented grasp configurations. Building upon this, a keypoint-driven hierarchical framework is developed to unify perception, planning and execution, thereby enabling task-level action reasoning conditioned on keypoint representations and coherent multistep action execution. For evaluation, a keypoint dataset containing over 1600 RGB images of 29 household objects is constructed, and extensive experiments demonstrate task-relevant keypoint generation with reduced computational time, as well as higher task execution success rates and broader task coverage.

Keywords

hierarchical framework / keypoint representation / sequential manipulation

Cite this article

Download citation ▾
Yaxin Liu, Yan Liu, Xinyu Song, Ruiqing Han, Haiyong Liu, Bo Huang, Yufeng Yao, Ming Zhong. SFK: Shape- and Function-Grounded Keypoint Representation for Sequential Manipulation. CAAI Transactions on Intelligence Technology, 2026, 11 (5) : 1396-1412 DOI:10.1049/cit2.70136

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

R. Pfeifer and F. Iida, Embodied Artificial Intelligence: Trends and Challenges, (2004), 1-26.

[2]

E. Triantafyllidis, F. Acero, Z. Liu, and Z. Li, “ Hybrid Hierarchical Learning for Solving Complex Sequential Tasks Using the Robotic Manipulation Network ROMAN,” Nature Machine Intelligence 5, no. 9 (2023): 991-1005, https://doi.org/10.1038/s42256-023-00709-2.

[3]

C. Tang, D. Huang, L. Meng, W. Liu, and H. Zhang, “ Task-Oriented Grasp Prediction With Visual-Language Inputs,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2023), 4881-4888.

[4]

H. Liang, X. Ma, S. Li, et al., “ PointNetGPD: Detecting Grasp Configurations From Point Sets,” in 2019 International Conference on Robotics and Automation (ICRA) (IEEE, 2019)

[5]

Z. Lu, N. Wang, and C. Yang, “ A Constrained DMPs Framework for Robot Skills Learning and Generalization From Human Demonstrations,” IEEE/ASME Transactions on Mechatronics 26, no. 6 (2021): 3265-3275, https://doi.org/10.1109/tmech.2021.3057022.

[6]

M. Guo and M. Burger, “ Geometric Task Networks: Learning Efficient and Explainable Skill Coordination for Object Manipulation,” IEEE Transactions on Robotics 38, no. 3 (2022): 1723-1734, https://doi.org/10.1109/tro.2021.3111481.

[7]

P. Englert and M. Toussaint, “ Learning Manipulation Skills From a Single Demonstration,” International Journal of Robotics Research 37, no. 1 (2018): 137-154, https://doi.org/10.1177/0278364917743795.

[8]

P. F. Lucas Manuelli and R. Tedrake, “ Keypoints Into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning,” in Conference on Robot Learning (PMLR, 2020), 693-710.

[9]

P. M. Scheikl, E. Tagliabue, B. Gyenes, et al., “ Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery,” IEEE Robotics and Automation Letters 8, no. 2 (2023): 560-567, https://doi.org/10.1109/lra.2022.3227873.

[10]

W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei, “ ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation,” preprint arXiv:2409.01652 (2024).

[11]

C. Tang, J. Yu, W. Chen, B. Xia, and H. Zhang, “ Relationship Oriented Semantic Scene Understanding for Daily Manipulation Tasks,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2022), 9926-9933.

[12]

M. Zaccaria, F. Manhardt, Y. Di, F. Tombari, J. Aleotti, and M. Giorgini, “ Self-Supervised Category-Level 6D Object Pose Estimation With Optical Flow Consistency,” IEEE Robotics and Automation Letters 8, no. 5 (2023): 2510-2517, https://doi.org/10.1109/lra.2023.3254463.

[13]

E. P. Örnek, A. K. Krishnan, S. Gayaka, et al., “ SupeRGB-D: Zero-Shot Instance Segmentation in Cluttered Indoor Environments,” IEEE Robotics and Automation Letters 8, no. 6 (2023): 3709-3716, https://doi.org/10.1109/lra.2023.3271527.

[14]

A. Delitzas, A. Takmaz, F. Tombari, R. Sumner, M. Pollefeys, and F. Engelmann, “ SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024), 14531-14542.

[15]

H. G. Cao, W. Zeng, and I. C. Wu, “ Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) (2023), 9501-9507.

[16]

Z. Y. Qin, K. Fang, Y. K. Zhu, F. F. Li, and S. Savarese, “ KETO: Learning Keypoint Representations for Tool Manipulation,” in IEEE International Conference on Robotics and Automation ICRA. IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2020), 7278-7285.

[17]

S. Lin, Z. Wang, Y. Ling, Y. Tao, and C. Yang, “ E2EK: End-to-End Regression Network Based on Keypoint for 6D Pose Estimation,” IEEE Robotics and Automation Letters 7, no. 3 (2022): 6526-6533, https://doi.org/10.1109/lra.2022.3174261.

[18]

X. Ma, D. Hsu, and W. S. Lee, “ Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects,” in 2022 International Conference on Robotics and Automation (ICRA) (2020), 8266-8273.

[19]

A. Simeonov, Y. Du, A. Tagliasacchi, et al., “ Neural Descriptor Fields: Se(3)-Equivariant Object Representations for Manipulation,” in 2022 International Conference on Robotics and Automation (ICRA) (2022), 6394-6400.

[20]

F. Zhang and Y. Demiris, “ Learning Garment Manipulation Policies Toward Robot-Assisted Dressing,” Science Robotics 7, no. 65 (2022): eabm6010, https://doi.org/10.1126/scirobotics.abm6010.

[21]

S. Bechtle, N. Das, and F. Meier, “ Multimodal Learning of Keypoint Predictive Models for Visual Object Manipulation,” IEEE Transactions on Robotics 39, no. 2 (2023): 1212-1224, https://doi.org/10.1109/tro.2022.3204509.

[22]

J. Gao, X. Jin, F. Krebs, N. Jaquier, T. Asfour, “ Bi-Kvil: Keypoints-Based Visual Imitation Learning of Bimanual Manipulation Tasks,” in IEEE International Conference on Robotics and Automation (ICRA), (2024), 16850-16857.

[23]

N. D. Palo and E. Johns, Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics, (2024).

[24]

H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao, “ Copa: General Robotic Manipulation Through Spatial Constraints of Parts With Foundation Models,” preprint arXiv:2403.08248 (2024).

[25]

M. Pan, J. Zhang, T. Wu, Y. Zhao, W. Gao, and H. Dong, “ Omnimanip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints,” in Proceedings of the Computer Vision and Pattern Recognition Conference (2025), 17359-17369.

[26]

Y. Zhang, S. Mittal, Z. Zhang, L. Ke, S. Srinivasa, and A. Gupta, ATK: Automatic Task-Driven Keypoint Selection for Robust Policy Learning, (2025).

[27]

R. Xu, F. J. Chu, C. Tang, W. Liu, and P. A. Vela, “ An Affordance Keypoint Detection Network for Robot Manipulation,” IEEE Robotics and Automation Letters 6, no. 2 (2021): 2870-2877, https://doi.org/10.1109/lra.2021.3062560.

[28]

L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “ KPAM: Keypoint Affordances for Category-Level Robotic Manipulation,” in Robotics Research (Springer International Publishing, 2019), 132-157.

[29]

J. Huang and K. W. Samuel Au, “ Task-Oriented Grasping Position Selection in Deformable Object Manipulation,” IEEE Robotics and Automation Letters 8, no. 2 (2023): 776-783, https://doi.org/10.1109/lra.2022.3231495.

[30]

S. Nasiriany, F. Xia, W. Yu, et al., “ PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs,” preprint arXiv:2402.07872 (2024).

[31]

K. F. Levine, F. Liu, P. Abbeel, and S. Moka, “ Open-World Robotic Manipulation Through Mark-Based Visual Prompting,” in First Workshop on Vision-Language Models for Navigation and Manipulation at ICRA, (2024).

[32]

T. Chao, X. Anxing, D. Yuhong, and T. Hu, “ FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulation,” arXiv (2025).

[33]

L. Yan and S. Xinyu, “ Shape- and Function-Centric Keypoint Representation,” Roboflow Universe (2025), https://universe.roboflow.com/harbin-institute-of-technology-9q8vx/feed_keypoints-2-v1frv.

[34]

J. Glenn and Q. Jing. 2024. Ultralytics YOLO11, https://github.com/ultralytics/ultralytics.

[35]

Y. Liu, Y. Liu, Y. Yao, and M. Zhong, “ A Laser Pointer-Based Interaction Method for Providing Robot Assistance to People With Upper Limb Impairments,” Expert Systems with Applications 292 (2025): 128562, https://doi.org/10.1016/j.eswa.2025.128562.

[36]

C. Warren, “ Global Path Planning Using Artificial Potential Fields,” in Proceedings, 1989 International Conference on Robotics and Automation, (1989) 316-321.

[37]

M. Oquab, T. Darcet, T. Moutakanni, et al., “ Dinov2: Learning Robust Visual Features Without Supervision,” preprint arXiv:2304.07193 (2023).

[38]

A. Radford, J. W. Kim, C. Hallacy, et al., “ Learning Transferable Visual Models From Natural Language Supervision,” in Proceedings of Machine Learning Research, (2021).

[39]

D. Alexey, B. Lucas, K. Alexander, and W. Dirk, “ An Image Is Worth 16 × 16 Words: Transformers for Image Recognition at Scale,” preprint arXiv:2010 (2020): 11929.

[40]

W. Dong, D. Huang, J. Liu, C. Tang, and H. Zhang, “ Rtagrasp: Learning Task-Oriented Grasping From Human Videos via Retrieval, Transfer, and Alignment,” in 2025 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2025), 1-7.

PDF (4973KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉