Toward multi-object nonprehensile transportation via shared teleoperation: A framework based on virtual object model predictive control

Abstract

Multi-object nonprehensile transportation in teleoperated robotic systems poses a dual control challenge: real-time trajectory tracking and simultaneous tray orientation control to satisfy object dynamic constraints. Existing approaches face limitations, including difficulty satisfying trajectory state constraints, excessive model dependency, inadequate adaptability to multi-object scenarios, and a lack of robust mechanisms for handling uncertain object parameters. To address these limitations, this work proposes a novel shared teleoperation framework for multi-object nonprehensile transportation, which enables shared control between human operators and the robotic system for object positioning; meanwhile, the robot autonomously controls object orientation to satisfy task constraints. The primary contributions are threefold: First, a theoretical analysis of dynamic constraints is developed, incorporating object position, inertial parameters, quantity, friction coefficients, and motion states. Furthermore, a virtual object-based dynamic constraint processing method is proposed for the first time, enabling simplified dynamic constraints to be directly utilized for trajectory planning. Second, a model predictive control-based trajectory smoothing algorithm with real-time dynamic constraint enforcement is designed, enabling dynamic coordination between user input tracking and orientation control. Third, simulation and experimental validation confirm that the proposed method successfully ensures dynamic constraints for all objects and achieves stable manipulation of nine different objects at accelerations up to 2.4 m/s². Compared with the baseline method, the approach achieves a 72.45% reduction in sliding distance and maintains a zero tip-over rate (compared with 13.9% for the baseline). These results demonstrate enhanced adaptability to multi-object parameters and robust performance in complex nonprehensile transportation scenarios.

Graphical abstract

Keywords

virtual object / model predictive control / multi-object nonprehensile transportation / shared teleoperation / dynamic constraint processing / trajectory smoothing

References

[1] Darvish K , Penco L , Ramos J , Cisneros R , Pratt J , Yoshida E , Ivaldi S , Pucci D . Teleoperation of humanoid robots: A survey. IEEE Transactions on Robotics, 2023, 39(3): 1706–1727

[2] Shahbazi M , Atashzar S F , Patel R V . A systematic review of multilateral teleoperation systems. IEEE Transactions on Haptics, 2018, 11(3): 338–356

[3] Lu Z , Si W , Wang N , Yang C . Dynamic movement primitives-based human action prediction and shared control for bilateral robot teleoperation. IEEE Transactions on Industrial Electronics, 2024, 71(12): 16654–16663

[4] Li G , Li Q , Yang C , Su Y , Yuan Z , Wu X . The classification and new trends of shared control strategies in telerobotic systems: A survey. IEEE Transactions on Haptics, 2023, 16(2): 118–133

[5] Chen L, Yu H, Zhang L, Naceri A, Swikir A, Haddadin S. Trajectory planning for non-prehensile object transportation. In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi: IEEE, 2024, 9939–9946

[6] Huang D , Li B , Li Y , Yang C . Cooperative manipulation of deformable objects by single-leader–dual-follower teleoperation. IEEE Transactions on Industrial Electronics, 2022, 69(12): 13162–13170

[7] Bharadhwaj H, Vakil J, Sharma M, Gupta A, Tulsiani S, Kumar V. RoboAgent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). Yokohama: IEEE, 2024, 4788–4795

[8] Fu Z, Zhao T Z, Finn C. Mobile ALOHA: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. 2024, arXiv preprint, arXiv.2401.02117

[9] Huang W, Wang C, Zhang R, Li Y, Wu J, Li F F. VoxPoser: Composable 3D value maps for robotic manipulation with language models. 2023, arXiv preprint, arXiv:2307.05973

[10] Wang J, Jin Y, Jun S, Yong A, Li D, Sun F, Luo D, Fang B. EHC-MM: Embodied holistic control for mobile manipulation. 2025, arXiv preprint, arXiv:2409.08527

[11] Zhong Y, Bai F, Cai S, Huang X, Chen Z, Zhang X, Wang Y, Guo S, Guan T, Lui K N, Qi Z, Liang Y, Chen Y, Yang Y. A survey on vision-language-action models: An action tokenization perspective. 2025, arXiv preprint, arXiv:2507.01925

[12] Ho J, Ermon S. Generative adversarial imitation learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16). Red Hook: Curran Associates Inc., 2016, 4572–4580

[13] Osa T , Pajarinen J , Neumann G , Bagnell J A , Abbeel P , Peters J . An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 2018, 7(1–2): 1–179

[14] Hussein A , Gaber M M , Elyan E , Jayne C . Imitation learning: A survey of learning methods. ACM Computing Surveys, 2018, 50(2): 1–35

[15] Lowrey K, Kolev S, Dao J, Rajeswaran A, Todorov E. Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system. In: 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR). Brisbane: IEEE, 2018, 35–42

[16] Ju H , Juan R , Gomez R , Nakamura K , Li G . Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nature Machine Intelligence, 2022, 4(12): 1077–1087

[17] Polydoros A S , Nalpantidis L . Survey of model-based reinforcement learning: Applications on robotics. Journal of Intelligent & Robotic Systems, 2017, 86(2): 153–173

[18] Nagabandi A, Kahn G, Fearing R S, Levine S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane: IEEE, 2018, 7559–7566

[19] Lambert N O, Drew D S, Yaconelli J, Calandra R, Levine S, Pister K S J. Low level control of a quadrotor with deep model-based reinforcement learning. 2019, arXiv preprint, arXiv:1901.03737

[20] Moerland T M , Broekens J , Plaat A , Jonker C M . Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 2023, 16(1): 1–118

[21] Gattringer H , Mueller A , Oberherber M , Kaserer D . Time-optimal robotic manipulation on a predefined path of loosely placed objects: Modeling and experiment. Mechatronics, 2022, 84: 102753

[22] Verscheure D , Demeulenaere B , Swevers J , De Schutter J , Diehl M . Time-optimal path tracking for robots: A convex optimization approach. IEEE Transactions on Automatic Control, 2009, 54(10): 2318–2327

[23] Csorvási G , Nagy Á , Vajk I . Near time-optimal path tracking method for waiter motion problem. IFAC-PapersOnLine, 2017, 50(1): 4929–4934

[24] Heins A , Schoellig A P . Keep it upright: Model predictive control for nonprehensile object transportation with obstacle avoidance on a mobile manipulator. IEEE Robotics and Automation Letters, 2023, 8(12): 7986–7993

[25] Heins A , Schoellig A P . Robust nonprehensile object transportation with uncertain inertial parameters. IEEE Robotics and Automation Letters, 2025, 10(5): 4492–4499

[26] Martucci G, Bimbo J, Prattichizzo D, Malvezzi M. Maintaining stable grasps during highly dynamic robot trajectories. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas: IEEE, 2020, 9198–9204

[27] Subburaman R , Selvaggio M , Ruggiero F . A non-prehensile object transportation framework with adaptive tilting based on quadratic programming. IEEE Robotics and Automation Letters, 2023, 8(6): 3581–3588

[28] Miguel Garcia-Haro J, Martinez S, Balaguer C. Balance computation of objects transported on a tray by a humanoid robot based on 3D dynamic slopes. In: 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids). Beijing: IEEE, 2018, 1–6

[29] Morlando V, Selvaggio M, Ruggiero F. Nonprehensile object transportation with a legged manipulator. In: 2022 International Conference on Robotics and Automation (ICRA). Philadelphia: IEEE, 2022, 6628–6634

[30] Zhou C, Lei M, Zhao L, Wang Z, Zheng Y. TOPP-MPC-based dual-arm dynamic collaborative manipulation for multi-object nonprehensile transportation. In: 2022 International Conference on Robotics and Automation (ICRA). Philadelphia: IEEE, 2022, 999–1005

[31] Selvaggio M , Garg A , Ruggiero F , Oriolo G , Siciliano B . Non-prehensile object transportation via model predictive non-sliding manipulation control. IEEE Transactions on Control Systems Technology, 2023, 31(5): 2231–2244

[32] Muchacho R I C, Laha R, Figueredo L F C, Haddadin S. A solution to slosh-free robot trajectory optimization. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Kyoto: IEEE, 2022, 223–230

[33] Muchacho R I C, Bien S, Laha R, Naceri A, Figueredo L F C, Haddadin S. Shared autonomy control for slosh-free teleoperation. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Detroit: IEEE, 2023, 10676–10683

[34] Biagiotti L, Chiaravalli D, Zanella R, Melchiorri C. Optimal feed-forward control for robotic transportation of solid and liquid materials via nonprehensile grasp. 2023, arXiv preprint, arXiv:2306.14212

[35] Biagiotti L, Chiaravalli D, Moriello L, Melchiorri C. A plug-in feed-forward control for sloshing suppression in robotic teleoperation tasks. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018, 5855–5860

[36] Arrizabalaga J, Pries L, Laha R, Li R, Haddadin S, Ryll M. Geometric slosh-free tracking for robotic manipulators. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). Yokohama: IEEE, 2024, 1226–1232

[37] Moriello L , Biagiotti L , Melchiorri C , Paoli A . Manipulating liquids with robots: A sloshing-free solution. Control Engineering Practice, 2018, 78: 129–141

[38] Feddema J T , Dohrmann C R , Parker G G , Robinett R D , Romero V J , Schmitt D J . Control for slosh-free motion of an open container. IEEE Control Systems, 1997, 17(1): 29–36

[39] Chen S J, Hein B, Worn H. Using acceleration compensation to reduce liquid surface oscillation during a high speed transfer. In: Proceedings of 2007 IEEE International Conference on Robotics and Automation. Rome: IEEE, 2007, 2951–2956

[40] Yano K , Terashima K . Sloshing suppression control of liquid transfer systems considering a 3-D transfer path. IEEE/ASME Transactions on Mechatronics, 2005, 10(1): 8–16

[41] Pridgen B , Bai K , Singhose W . Shaping container motion for multimode and robust slosh suppression. Journal of Spacecraft and Rockets, 2013, 50(2): 440–448

[42] Aribowo W , Yamashita T , Terashima K . Integrated trajectory planning and sloshing suppression for three-dimensional motion of liquid container transfer robot arm. Journal of Robotics, 2015, 2015: 1–15

[43] Aboel-Hassan A , Arafa M , Nassef A . Design and optimization of input shapers for liquid slosh suppression. Journal of Sound and Vibration, 2009, 320(1–2): 1–15

[44] Terashima K , Yano K . Sloshing analysis and suppression control of tilting-type automatic pouring machine. Control Engineering Practice, 2001, 9(6): 607–620

[45] Aribowo W, Yamashita T, Terashima K, Kitagawa H. Input shaping control to suppress sloshing on liquid container transfer using multi-joint robot arm. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2010, 3489–3494

[46] Hu K , Chen Z , Kang H , Tang Y . 3D vision technologies for a self-developed structural external crack damage recognition robot. Automation in Construction, 2024, 159: 105262

[47] Selvaggio M , Cacace J , Pacchierotti C , Ruggiero F , Giordano P R . A shared-control teleoperation architecture for nonprehensile object transportation. IEEE Transactions on Robotics, 2022, 38(1): 569–583

[48] Focchi M , del Prete A , Havoutis I , Featherstone R , Caldwell D G , Semini C . High-slope terrain locomotion for torque-controlled quadruped robots. Autonomous Robots, 2017, 41(1): 259–272

[49] Di Carlo J, Wensing P M, Katz B, Bledt G, Kim S. Dynamic locomotion in the MIT Cheetah 3 through convex model-predictive control. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid: IEEE, 2018, 1–9

[50] Mellinger D, Kumar V. Minimum snap trajectory generation and control for quadrotors. In: 2011 IEEE International Conference on Robotics and Automation. Shanghai: IEEE, 2011, 2520–2525

[51] Siciliano B, Slotine J J E. A general framework for managing multiple tasks in highly redundant robotic systems. In: Fifth International Conference on Advanced Robotics’ Robots in Unstructured Environments. Pisa: IEEE, 1991, 1211–1216

[52] Liu Y , Wang K , Xu Q , Zhang J , Hu Y , Ma T , Zheng Q , Luo J . Superlubricity between graphite layers in ultrahigh vacuum. ACS Applied Materials & Interfaces, 2020, 12(38): 43167–43172

RIGHTS & PERMISSIONS

Higher Education Press