Dyna-QUF: Dyna-Q based univector field navigation for autonomous mobile robots in unknown environments

Abstract

A novel approach was presented to solve the navigation problem of autonomous mobile robots in unknown environments with dense obstacles based on a univector field method. In an obstacle-free environment, a robot is ensured to reach the goal position with the desired posture by following the univector field. Contrariwise, the univector field cannot guarantee that the robot will avoid obstacles in environments. In order to create an intelligent mobile robot being able to perform the obstacle avoidance task while following the univector field, Dyna-Q algorithm is developed to train the robot in learning moving directions to attain a collision-free path for its navigation. Simulations on the computer as well as experiments on the real world prove that the proposed algorithm is efficient for training the robot in reaching the goal position with the desired final orientation.

Keywords

Dyna-Q / mobile robot / reinforcement learning / univector field

References

[1] DudekG, JenkinMComputational principles of mobile robotics [M], 201080-105

[2] KimY J, KimJ H, KwonD S. Evolutionary programming-based univector field navigation method for fast mobile robots [J]. IEEE Trans on Systems, Man, and Cybernetics B, 2001, 31(3): 450-458

[3] ParkK H, KimY J, KimJ H. Modular Q-learning based multi-agent cooperation for robot soccer [J]. Robotics and Autonomous Systems, 2001, 35(2): 109-122

[4] WatkinsCLearning from delayed rewards [D], 1989LondonKing’s College

[5] HongY, KimJ HFootstep planning based on univector field method for humanoid robot [C], 2009HeidelbergSpringer-Verlag Berlin125-134

[6] VienN A, VietN H, ParkH J, LeeS G, ChungT C. Q-Learning based Univector Field Navigation Method for Mobile Robots [C]. Advances and Innovations in Systems, Computing Sciences and Software Engineering, 2007463-468

[7] KaelblingL P, LittmanM L, MooreA W. Reinforcement learning: A survey [J]. Journal of Artificial Intelligence Research, 1996, 4(1): 237-285

[8] SuttonR S, BartoA GReinforcement learning: An introduction [M], 1998CambridgeThe MIT Press66-92

[9] WatkinsC J C H, DayanP. Technical note Q-Learning [J]. Machine Learning, 1992, 8(1): 279-292

[10] KimD H, KimY J, KimK C, KimJ H, VadakkepatP. Vector field based path planning and Petri-net based role selection mechanism with Q-learning for soccer robots [J]. Intelligent Automation and Soft Computing, 2000, 6(1): 75-87

[11] KimJ H, KimD H, KimY J, SeowK TSoccer robotics [M], 200438-79

[12] SmartW D, KaelblingL P. Effective reinforcement learning for mobile robots [C]. Proceedings of IEEE International Conference on Robotics and Automation. Washington, DC, USA, 20023404-3410

[13] ZamsteinL, ArroyoA, SchwartzE, KeenS, SuttonB, GandhiGKoolio: Path planning using reinforcement learning on a real robot platform [D], 2006FloridaUniversity of Florida

[14] XieL-j, XieG-r, ChenH-w, LiX-li. Solution to reinforcement learning problems with artificial potential field [J]. Journal of Central South University of Technology, 2008, 15(4): 552-557

[15] VietH H, KyawP H, ChungT CSimulation-based evaluations of reinforcement learning algorithms for autonomous mobile robot path planning [C], 2011467-476

[16] The iRobot Create Team.iRobot Create owners guide [EB/OL], 2006

[17] BartonJ M, KoehlerO, LimJD Matlab toolbox for the create robot [EB/OL], 2011