Novel AR-based interface for human-robot interaction and visualization

Abstract

Intuitive and efficient interfaces for human-robot interaction (HRI) has been a challenging issue in robotics as it is essential for the prevalence of robots supporting humans in key areas of activities. This paper presents a novel augmented reality (AR) based interface to facilitate human-virtual robot interaction. A number of human-virtual robot interaction methods have been formulated and implemented with respect to the various types of operations needed in different robotic applications. A Euclidean distance-based method is developed to assist the users in the interaction with the virtual robot and the spatial entities in an AR environment. A monitor-based visualization mode is adopted as it enables the users to perceive the virtual contents associated with different interaction methods, and the virtual content augmented in the real environment is informative and useful to the users during their interaction with the virtual robot. Case researches are presented to demonstrate the successful implementation of the AR-based HRI interface in planning robot pick-and-place operations and path following operations.

Keywords

Human-robot interaction (HRI) / Human-robot interface / Augmented reality (AR)

References

[1] Meeussen W (2006) Compliant robot motion: from path planning or human demonstration to force controlled task execution. Dissertation, Katholieke Universiteit Leuven

[2] Burdea GC (1996) Virtual reality and robotics in medicine. In: IEEE International Workshop on Robot and Human Communication, Tsukuba, 11–14 Nov 1996, pp 16–25

[3] Castillo CRA, Wahrburg J. Improving robot arm control for safe and robust haptic cooperation in orthopaedic procedures. Int J Med Robot Comput Assist Surg, 2007, 3(4): 316-322.

[4] Breazeal C, Edsinger A, Fitzpatrick P, et al. Active vision for sociable robots. IEEE Trans Syst Man Cybern A, 2001, 31(5): 443-453.

[5] Rastogi A, Milgram P, Drascic D et al (1996) Tele-robotic control with stereoscopic augmented reality. In: Proceedings of the SPIE stereoscopic displays and virtual reality systems III, San Jose, CA, 30 Jan–2 Feb 1996, vol 2653. doi:10.1117/12.237424

[6] Xiong Y, Li S, Xie M. Predictive display and interaction of tele-robots based on augmented reality. Robotica, 2006, 24(4): 447-453.

[7] Green SA, Chase JG, Chen XQ, et al. Evaluating the augmented reality human-robot collaboration system. Int J Intell Syst Technol Appl, 2010, 8(1–4): 130-143.

[8] Zaeh MF, Vogl W (2006) Interactive laser-projection for programming industrial robots. In: Proceedings of the international symposium on mixed and augmented reality, Santa Barbara, 22–25 Oct 2006, pp 125–133

[9] Reinhart G, Munzert U, Vogl W. A programming system for robot-based remote-laser-welding with conventional optics. CIRP Ann, 2008, 57(1): 37-40.

[10] Collett THJ, MacDonald B (2006) Developer oriented visualization of a robot program. In: Proceedings of the conference on human–robot interaction. Salt Lake City, 2–4 Mar 2006, pp 49–55

[11] Bischoff R, Kazi A, Seyfarth M (2002) The MORPHA style guide for icon-based programming. In: Proceedings of the IEEE international workshop on robot and human interactive communication. Berlin, 25–27 Sep 2002, pp 482–491

[12] Chintamani K, Cao A, Ellis RD, et al. Improved tele-manipulator navigation during display-control misalignments using augmented reality cues. IEEE Trans Syst Man Cybern A, 2010, 40(1): 29-39.

[13] Stilman M, Michel P, Chestnutt J et al (2005) Augmented reality for robot development and experimentation. Tech Rep CMU-RI-TR-05-55. Robotics Institute, Carnegie Mellon University, Pittsburgh

[14] Mohareri O, Rad AB (2011) Autonomous humanoid robot navigation using augmented reality technique. In: Proceedings of the IEEE international conference on mechatronics, Istanbul, 13–15 April 2011, pp 463–471

[15] Parasuraman R, Sheridan TB, Wickens CD. A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern A, 2000, 30(3): 286-297.

[16] Thrun S. Toward a framework for human-robot interaction. Hum-Comput Interact, 2004, 19(1/2): 9-24.

[17] Goodrich MA, Schultz AC. Human-robot interaction: a survey. Found Trends Hum-Comput Interact, 2007, 1(3): 203-275.

[18] Prusak A, Bemshausen J, Roth H et al (2007) Applications of automated guided vehicle (AGV) and industry robots with PMD-camera. In: Proceedings of the international conference on robotics and applications, Wurzburg, 29–31 Aug 2007, pp 299–303

[19] Yanagihara Y, Kakizaki T, Arakawa K, et al. A multimodal teaching advisor for sensor-enhanced robotic systems in manufacturing. Robot Comput-Integr Manuf, 1998, 14(4): 263-273.

[20] Kock S, Bredahl J, Eriksson PJ, et al. Taming the robot—better safety without higher fences. ABB Rev, 2006, 4: 11-15.

[21] Adams JA (2002) Critical considerations for human-robot interface development. In: Proceedings of 2002 AAAI fall symposium, North Falmouth, 15–17 November 2002. Tech. Report FS-02-03, pp 1–8

[22] Steinfeld A, Fong T, Kaber D et al (2006) Common metrics for human–robot interaction. In: Proceeding of the ACM international conference on human-robot interaction. Salt Lake City, 2–3 Mar 2006, pp 33–40

[23] Endsley MR. Endsley MR, Garland DJ. Theoretical underpinnings of situation awareness: a critical review. Situation awareness analysis and measurement, 2000, Mahwah: Lawrence Erlbaum Associates 3-32.

[24] Endsley MR. Endsley MR, Garland DJ. Direct measurement of situation awareness: validity and use of SAGAT. Situation awareness analysis and measurement, 2000, Mahwah: Lawrence Erlbaum Associates 147-173.

[25] Murphy RR. Human-robot interaction in rescue robotics. IEEE Trans Syst Man Cybern C, 2004, 34(2): 138-153.

[26] Heyer C (2010) Human-robot interaction and future industrial robotics applications. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Taipei 18–22 Oct 2010, pp 4749–4754

[27] Thrun S, Bennewitz M, Burgard W et al (1999) MINERVA: a second generation mobile tour-guide robot. In: Proceedings of the IEEE international conference on robotics and automation, Detroit, 10–15 May 1999, pp 1999–2005

[28] Waldherr S, Romero R, Thrun S. A gesture based interface for human-robot interaction. Auton Robots, 2000, 9(2): 151-173.

[29] Yin X, Zhu X (2006) Hand posture recognition in gesture-based human-robot interaction. In: Proceedings of the IEEE international conference on industrial electronics and applications. Singapore, 24–26 May 2006, pp 1–6

[30] Tang X, Yamada H, Zhao D et al (2009) Haptic interaction in teleoperation control system of construction robot based on virtual reality. In: Proceedings of the IEEE international conference on mechatronics and automation, Changchun. 9–12 Aug 2009, pp 78–83

[31] Tsetserukou D, Sato K, Kawakami N et al (2009) Teleoperation system with haptic feedback for physical interaction with remote environment. In: Proceedings of the ICROS-SICE international joint conference, Fukuoka 18–21 Aug 2009, pp 3353–3358

[32] Pires JN. Robot-by-voice: experiments on commanding an industrial robot using the human voice. Industrial Robot, 2005, 32(6): 505-511.

[33] Medicherla H, Sekmen A. Human-robot interaction via voice-controllable intelligent user interface. Robotica, 2007, 25(5): 521-527.

[34] Iba S, Paredis CJJ, Khosla PK. Interactive multimodal robot Programming. Int J Robot Res, 2005, 24(1): 83-104.

[35] Marín R, Sanz PJ, Nebot P, et al. A multimodal interface to control a robot arm via the web: a case study on remote programming. IEEE Trans Ind Electron, 2005, 52(6): 1506-1520.

[36] Pires JN, Godinho T, Araujo R. Using digital pens to program welding tasks. Ind Robot, 2007, 34(6): 476-486.

[37] Schraft RD, Meyer C. The need for an intuitive teaching method for small and medium enterprises. VDI Ber, 2006, 1956: 95

[38] Chen J, Zelinsky A. Programming by demonstration: coping with suboptimal teaching actions. Int J Robot Res, 2003, 22(5): 299-319.

[39] Aleotti J, Caselli S (2005) Trajectory clustering and stochastic approximation for robot programming by demonstration. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Edmonton, 2–6 Aug 2005, pp 1029–1034

[40] Khatib O, Sentis L, Park JH, et al. Whole body dynamic behavior and control of human-like robots. Int J Humanoid Robot, 2004, 1(1): 29-43.

[41] Žlajpah L. Simulation in robotics. Math Comput Simul, 2008, 79(4): 879-897.

[42] Ehrenmann M, Ambela D, Steinaus P et al (2000) A comparison of four fast vision based object recognition methods for programming by demonstration applications. In: Proceedings of the IEEE international conference on robotics and automation, San Francisco 24–28 April 2000, pp 1862–1867

[43] Chong JWS, Ong SK, Nee AYC, et al. Robot programming using augmented reality: an interactive method for planning collision-free paths. Robot Comput-Integr Manuf, 2009, 25(3): 689-701.

[44] Fang HC, Ong SK, Nee AYC. Interactive robot trajectory planning and simulation using augmented reality. Robot Comput-Integr Manuf, 2012, 28(2): 227-237.

[45] Han BO, Kim YH, Cho K et al (2010) Museum tour guide robot with augmented reality. In: Proceedings of the international conference on virtual systems and multimedia, Seoul, 20–23 Oct 2010, pp 223–229

[46] Ong SK, Chong JWS, Nee AYC. A novel AR-based robot programming and path planning methodology. Robot Comput-Integr Manuf, 2010, 26(3): 240-249.