Efficient co-adaptation of humanoid robot design and locomotion control using surrogate-guided optimization

Abstract

Recent advancements in reinforcement learning (RL) and computational resources have demonstrated the efficacy of data-driven methodologies for robotic locomotion control and physical design optimization, providing a scalable alternative to traditional human-crafted design paradigms. However, existing co-design approaches face a critical challenge: the computational intractability of exploring high-dimensional design spaces, exacerbated by the resource-intensive nature of policy training and candidate design evaluations. To address this limitation, we propose an efficient co-adaptation framework for humanoid robot kinematics optimization. Building on a bi-level optimization architecture that jointly optimizes mechanical designs and control policies, our method achieves computational efficiency through two synergistic strategies: (1) a universal policy generalizable across design variations, and (2) a surrogate-assisted fitness evaluation mechanism. We implement the method with humanoid robot Kuafu, and by experimental results we demonstrate the proposed method effectively reduces the cost and the optimized design can achieve near-optimal performance.

Keywords

Humanoid robot / Design co-adaptation / Reinforcement learning

References

[1] C. Dong, Z. Yu, X. Chen, H. Chen, Y. Huang, Q. Huang, Adaptability control towards complex ground based on fuzzy logic for humanoid robots, IEEE Trans. Fuzzy Syst. 30 (6) (2022) 1574-1584.

[2] H. Chen, X. Chen, C. Dong, Z. Yu, Q. Huang, Online running pattern generation for humanoid robot with direct collocation of reference-tracking dynamics, IEEE/ASME Trans. Mechatron. 29 (3) (2024) 2091-2102.

[3] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, K. Sreenath, Real-world humanoid locomotion with reinforcement learning, Sci. Robot. 9 (89) (2024) eadi9579.

[4] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, C. Finn, HumanPlus: Humanoid shadowing and imitation from humans, in: 8th Annual Conference on Robot Learning, 2024.

[5] C. Dong, X. Chen, Z. Yu, H. Liu, F. Meng, Huang. Q., Swift running robot leg: Mechanism design and motion-guided optimization, IEEE/ASME Trans. Mechatron. 29 (3) (2023) 1702-1713.

[6] Á. Belmonte-Baeza, J. Lee, G. Valsecchi, M. Hutter, Meta reinforcement learning for optimal design of legged robots, IEEE Robot. Autom. Lett. 7 (4) (2022) 12134-12141.

[7] F. Bjelonic, J. Lee, P. Arm, D. Sako, D. Tateo, J. Peters, M. Hutter, Learning-based design and control for quadrupedal robots with parallel-elastic actuators, IEEE Robot. Autom. Lett. 8 (3) (2023) 1611-1618.

[8] K.S. Luck, R. Calandra, M. Mistry, What robot do I need? Fast co-adaptation of morphology and control using graph neural networks, 2021, arXiv preprint arXiv:2111.02371.

[9] M. Chadwick, H. Kolvenbach, F. Dubois, H.F. Lau, M. Hutter, Vitruvio: An open-source leg design optimization toolbox for walking robots, IEEE Robot. Autom. Lett. 5 (4) (2020) 6318-6325.

[10] S. Hu, Z. Yang, G. Mori, Neural fidelity warping for efficient robot morphology design, in: 2021 IEEE International Conference on Robotics and Automation, ICRA, 2021, pp. 7079-7086.

[11] X. Zhu, P. Gergondet, Z. Cai, X. Chen, Z. Yu, Q. Huang, A. Kheddar, The development of a 7-dof humanoid arm for driving using a task-driven design method, IEEE/ASME Trans. Mechatron. 29 (2) (2023) 1521-1533.

[12] K.M. Digumarti, C. Gehring, S. Coros, J. Hwangbo, R. Siegwart, Concurrent optimization of mechanical design and locomotion control of a legged robot, Mob. Serv. Robot., 2014, pp. 315-323.

[13] S. Ha, S. Coros, A. Alspach, J. Kim, K. Yamane, Task-based limb optimization for legged robots, in: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2016 pp.2062-2068, 2014.

[14] J. Won, J. Lee, Learning body shape variation in physics-based characters, ACM Trans. Graph. 38 (6) (2019) 1-12.

[15] A. Gupta, L. Fan, S. Ganguli, et al., Metamorph: learning universal controllers with transformers, 2022, arXiv preprint arXiv:2203.11931.

[16] H. Tong, C. Huang, L.L. Minku, X. Yao, Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study, Inf. Sci. 562 (2021) 414-437.

[17] Y. Wang, T. Zhang, Y. Chang, X. Wang, B. Liang, B. Yuan, A surrogate-assisted controller for expensive evolutionary reinforcement learning, Inf. Sci. 616 (2022) 539-557.

[18] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, M. Hutter, Learning agile and dynamic motor skills for legged robots, Sci. Robot. 4 (2019) 26.

[19] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, M. Hutter, Learning quadrupedal locomotion over challenging terrain, Sci. Robot. 5 (47) (2020) eabc5986.

[20] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, M. Hutter, Learning robust perceptive locomotion for quadrupedal robots in the wild, Sci. Robot. 7 (62) (2022) eabk2822.

[21] F. Jenelten, J. He, F. Farshidian, Hutter. M., Dtc: Deep tracking control, Sci. Robot. 9 (86) (2024) eadh5401.

[22] J. Siekmann, S. Valluri, J. Dao, F. Bermillo, H. Duan, A. Fern, Hurst. J., Learning memory-based control for human-scale bipedal locomotion, Robotics: Science and Systems XVI, 2020.

[23] J. Siekmann, Y. Godse, A. Fern, J. Hurst, Sim-to-real learning of all common bipedal gaits via periodic reward composition, in: 2021 IEEE International Conference on Robotics and Automation, ICRA, 2021, pp. 7309-7315.

[24] Z. Li, X.B. Peng, P. Abbeel, S. Levine, G. Berseth, K. Sreenath, Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control, Int. J. Robot. Res., 2024 02783649241285161.

[25] T. Haarnoja, B. Moran, G. Lever, S.H. Huang, D. Tirumala, J. Humplik, N. Heess, Learning agile soccer skills for a bipedal robot with deep reinforcement learning, Sci. Robot. 9 (89) (2024) eadi8022.

[26] A. Tang, T. Hiraoka, N. Hiraoka, F. Shi, K. Kawaharazuka, K. Kojima, M. Inaba, Humanmimic: Learning natural locomotion and transitions for humanoid robot via wasserstein adversarial imitation, in: 2024 IEEE International Conference on Robotics and Automation, ICRA, 2024, pp. 13107-13114.

[27] G. Fadini, T. Flayols, A. Del Prete, N. Mansard, P. Souères, Computational design of energy-efficient legged robots: Optimizing for size and actuators, in: 2021 IEEE International Conference on Robotics and Automation, ICRA, 2021, pp. 9898-9904.

[28] Y. Sun, C. Zong, F. Pancheri, T. Chen, T.C. Lueth, Design of topology optimized compliant legs for bio-inspired quadruped robots, Sci. Rep. 13 (1) (2023) 4875.

[29] J. Xu, T. Chen, L. Zlokapa, M. Foshey, W. Matusik, S. Sueda, P. Agrawal, An end-to-end differentiable framework for contact-aware robot design, Robotics: Science and Systems, 2021.

[30] D. Ha, Reinforcement learning for improving agent design, Artif. Life 25 (4) (2019) 352-365.

[31] K.S. Luck, H.B. Amor, R. Calandra, Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning, in: Conference on Robot Learning, 2020, pp. 854-869.

[32] C. Schaff, D. Yunis, A. Chakrabarti, M.R. Walter, Jointly learning to construct and control agents using deep reinforcement learning, in: 2019 International Conference on Robotics and Automation, ICRA, 2019, pp. 9798-9805.

[33] A. Zhao, J. Xu, M. Konaković-Luković, J. Hughes, A. Spielberg, D. Rus, W. Matusik, Robogrammar: Graph grammar for terrain-optimized robot design, ACM Trans. Graph. 39 (6) (2019) 1-16.

[34] A. Gupta, S. Savarese, S. Ganguli, L. Fei-Fei, Embodied intelligence via learning and evolution, Nat. Commun. 12 (1) (2021) 5721.

[35] Z. Wang, B. Benes, A.H. Qureshi, C. Mousas, Evolution-based shape and behavior co-design of virtual agents, IEEE Trans. Vis. Comput. Graphics 30 (12) (2024) 7579-7591.

[36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, 2017, arXiv preprint arXiv:1707.06347.

[37] M. Raibert, Legged Robots that Balance, MIT Press, 1986.

[38] Z. Luo, Y. Dong, X. Li, R. Huang, Z. Shu, E. Xiao, P. Lu, Moral: Learning morphologically adaptive locomotion controller for quadrupedal robots on challenging terrains, IEEE Robot. Autom. Lett. (2024).

[39] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, A. Handa, Isaac Gym: High performance GPU based physics simulation for robot learning, in: Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.

[40] N. Rudin, D. Hoeller, P. Reist, M. Hutter, Learning to walk in minutes using massively parallel deep reinforcement learning, in: Conference on Robot Learning, 2022, pp. 91-100.

[41] J. Blank, K. Deb, Pymoo: Multi-objective optimization in python, IEEE Access (8), 2020, pp. 89497-89509.