A Sarsa reinforcement learning hybrid ensemble method for robotic battery power forecasting

Abstract

Building a rail transit workshop with efficient data interconnection has become an inevitable trend in the transformation and development of the current rail transit equipment industry. More and more diversified mobile transport robots have become a priority in the process of digital transformation of smart factories. Accurate prediction of robot battery power can guide the control center to adopt scientific and reasonable instructions in advance to ensure efficient and stable operation of the logistics transportation chain. In this study, we propose a hybrid ensemble method of multiple learners based on state-action-reward-state-action (Sarsa) reinforcement learning algorithm. Maximal overlap discrete wavelet transform (MODWT) is used to preprocess the originally measured robot power supply voltage data. This significantly reduces the non-stationarity and volatility of time series data. Gated recurrent unit (GRU), deep belief network (DBN), and long short-term memory (LSTM), are utilized for the prediction modeling of subseries after decomposition. Finally, the Sarsa reinforcement learning ensemble strategy is used to weight the three basic predictors above. The performance of the Sarsa hybrid model is verified on three real mobile robot power data sets. Experimental results elaborate that the transportation robot battery power hybrid forecasting model is competitive in robustness, accuracy, and adaptability.

Keywords

robotic power management / transportation robot / time series forecasting / deep learning / Sarsa reinforcement learning / ensemble model

References

[1] JONES J L, SEIGER B A, FLYNN A M. Mobile robots: Inspiration to implementation [M]. AK Peters/CRC Press, 1998.

[2] WakchaureM, PatleB K, MahindrakarA K. Application of AI techniques and robotics in agriculture: A review [J]. Artificial Intelligence in the Life Sciences, 2023, 3: 100057

[3] HercikR, ByrtusR, JarosR, et al. . Implementation of autonomous mobile robot in SmartFactory [J]. Applied Sciences, 2022, 12(17): 8912

[4] DaimT U, YoonB S, LindenbergJ, et al. . Strategic roadmapping of robotics technologies for the power industry: A multicriteria technology assessment [J]. Technological Forecasting and Social Change, 2018, 13149-66

[5] ZhangH-Y, LinW-M, ChenA-X. Path planning for the mobile robot: A review [J]. Symmetry, 2018, 10(10): 450

[6] GurujiA K, AgarwalH, ParsediyaD K. Time-efficient A* algorithm for robot path planning [J]. Procedia Technology, 2016, 23144-149

[7] AlexopoulosC, GriffinP M. Path planning for a mobile robot [J]. IEEE Transactions on Systems, Man, and Cybernetics, 1992, 22(2): 318-322

[8] GulF, RahimanW, AlhadyS S N. A comprehensive study for robot navigation techniques [J]. Cogent Engineering, 2019, 6(1): 1632046

[9] RavankarA, RavankarA, KobayashiY, et al. . Path smoothing techniques in robot navigation: State-of-the-art, current and future challenges [J]. Sensors, 2018, 1893170

[10] BorensteinJ, EverettH R, FengL, et al. . Mobile robot positioning: Sensors and techniques [J]. Journal of Robotic Systems, 1997, 144231-249

[11] YuJ, JiangW-S, LuoZ, et al. . Application of a vision-based single target on robot positioning system [J]. Sensors, 2021, 21(5): 1829

[12] ZhangW, ChengH-T, HaoL-N, et al. . An obstacle avoidance algorithm for robot manipulators based on decision-making force [J]. Robotics and Computer-Integrated Manufacturing, 2021, 71102114

[13] PadoyN, HagerG D. Human-machine collaborative surgery using learned models [C]. 2011 IEEE International Conference on Robotics and Automation, 2011, Shanghai, China, IEEE: 5285-5292

[14] HaesevoetsT, de CremerD, DierckxK, et al. . Human-machine collaboration in managerial decision making [J]. Computers in Human Behavior, 2021, 119106730

[15] ThurowK, ChenC, JungingerS, et al. . Transportation robot battery power forecasting based on bidirectional deep-learning method [J]. Transportation Safety and Environment, 2019, 1(3): 205-211

[16] ChenJ-D, RoP I. Human intention-oriented variable admittance control with power envelope regulation in physical human-robot interaction [J]. Mechatronics, 2022, 84102802

[17] FarooqM U, EizadA, BaeH K. Power solutions for autonomous mobile robots: A survey [J]. Robotics and Autonomous Systems, 2023, 159104285

[18] ParasuramanR, KershawK, PagalaP, et al. . Model based on-line energy prediction system for semi-autonomous mobile robots [C]. 2014 5th International Conference on Intelligent Systems, Modelling and Simulation, 2015, Langkawi, Malaysia, IEEE: 411-416

[19] AlhassanA B, ZhangX-D, ShenH-M, et al. . Power transmission line inspection robots: A review, trends and challenges for future research [J]. International Journal of Electrical Power & Energy Systems, 2020, 118: 105862

[20] QuannM, OjedaL, SmithW, et al. . Off-road ground robot path energy cost prediction through probabilistic spatial mapping [J]. Journal of Field Robotics, 2020, 37(3): 421-439

[21] HamzaA, AyanianN. Forecasting battery state of charge for robot missions [C]. Proceedings of the Symposium on Applied Computing, 2017, New York, ACM: 249-255

[22] LÜX-Q, DengR-Y, ChenC, et al. . Performance optimization of fuel cell hybrid power robot based on power demand prediction and model evaluation [J]. Applied Energy, 2022, 316: 119087

[23] ShenW X. State of available capacity estimation for lead-acid batteries in electric vehicles using neural network [J]. Energy Conversion and Management, 2007, 48(2): 433-442

[24] PENTZER J, BRENNAN S, REICHARD K. On-line estimation of vehicle motion and power model parameters for skid-steer robot energy use prediction [C]// 2014 American Control Conference. IEEE, 2014: 2786–2791.

[25] HouL-F, ZhangL, KimJ. Energy modeling and power measurement for mobile robots [J]. Energies, 2018, 12(1): 27

[26] SabareeshS U, AravindK S N, ChowdaryK B, et al. . LSTM based 24 hours ahead forecasting of solar PV system for standalone household system [J]. Procedia Computer Science, 2023, 2181304-1313

[27] ParasuramanR, MinB C, ÖgrenP. Rapid prediction of network quality in mobile robots [J]. Ad Hoc Networks, 2023, 138103014

[28] RibeiroJ, RuiL-M, EckhardtT, et al. . Robotic process automation and artificial intelligence in industry 4.0-A literature review [J]. Procedia Computer Science, 2021, 18151-58

[29] LiuH, StollN, JungingerS, et al. . A new approach to battery power tracking and predicting for mobile robot transportation using wavelet decomposition and ANFIS networks [C]. 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), 2015, Bali, Indonesia, IEEE: 253258

[30] LeutbecherM, PalmerT N. Ensemble forecasting [J]. Journal of Computational Physics, 2008, 227(7): 3515-3539

[31] PengF, ZhengL, DuanZ, et al. . Multi-objective multi-learner robot trajectory prediction method for IoT mobile robot systems [J]. Electronics, 2022, 11(13): 2094

[32] HamedY, ShafieA, MustaffaZ B, et al. . An application of k-nearest neighbor interpolation on calibrating corrosion measurements collected by two non-destructive techniques [C]. 2015 IEEE 3rd International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA), 2016, Kuala Lumpur, Malaysia, IEEE: 1-5

[33] WarsitoB, SubanarS, AbdurakhmanA. Wavelet decomposition for time series: Determining input model by using mRMR criterion [J]. Hacettepe Journal of Mathematics and Statistics, 2015, 44(1): 229-238

[34] ZhuL, WangY-X, FanQ-B. MODWT-ARMA model for time series prediction [J]. Applied Mathematical Modelling, 2014, 38(5–6): 1859-1865

[35] LiM-Y, ChenW-Z, ZhangT. Application of MODWT and log-normal distribution model for automatic epilepsy identification [J]. Biocybernetics and Biomedical Engineering, 2017, 37(4): 679-689

[36] YamakP T, LiY-J, GadoseyP K. A comparison between ARIMA, LSTM, and GRU for time series forecasting [C]. Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence, 2019, New York, ACM: 4955

[37] LiM-W, XuD-Y, GengJ, et al. . A hybrid approach for forecasting ship motion using CNN-GRU-AM and GCWOA [J]. Applied Soft Computing, 2022, 114: 108084

[38] KangK, SunH-B, ZhangC-K, et al. . Short-term electrical load forecasting method based on stacked auto-encoding and GRU neural network [J]. Evolutionary Intelligence, 2019, 12(3): 385-394

[39] KumarS, HussainL, BanarjeeS, et al. . Energy load forecasting using deep learning approach-LSTM and GRU in spark cluster [C]. 2018 Fifth International Conference on Emerging Applications of Information Technology (EAIT), 2018, Kolkata, India, IEEE: 1-4

[40] WuL-Z, KongC, HaoX-H, et al. . A short-term load forecasting method based on GRU-CNN hybrid neural network model [J]. Mathematical Problems in Engineering, 2020, 2020: 1-10

[41] ShenF-R, ChaoJ, ZhaoJ-X. Forecasting exchange rate using deep belief networks and conjugate gradient method [J]. Neurocomputing, 2015, 167: 243-253

[42] RenY-P, MaoJ-L, LiuY, et al. . A novel dbn model for time series forecasting [J]. IAENG International Journal of Computer Science, 2017, 44(1): 79-86

[43] LiuH, YangR, WangT-T, et al. . A hybrid neural network model for short-term wind speed forecasting based on decomposition, multi-learner ensemble, and adaptive multiple error corrections [J]. Renewable Energy, 2021, 165: 573-594

[44] LiuH, YangR, DuanZ. Wind speed forecasting using a new multi-factor fusion and multi-resolution ensemble model with real-time decomposition and adaptive error correction [J]. Energy Conversion and Management, 2020, 217: 112995

[45] YangR, LiuH, NikitasN, et al. . Short-term wind speed forecasting using deep reinforcement learning with improved multiple error correction approach [J]. Energy, 2022, 239122128

[46] LiuH, YangR, DuanZ, et al. . A hybrid neural network model for marine dissolved oxygen concentrations time-series forecasting based on multi-factor analysis and a multi-model ensemble [J]. Engineering, 2021, 7(12): 1751-1765

[47] JameiM, AliM, MalikA, et al. . Development of a TVF-EMD-based multi-decomposition technique integrated with Encoder-Decoder-Bidirectional-LSTM for monthly rainfall forecasting [J]. Journal of Hydrology, 2023, 617129105

[48] MaoW-F, ZhuH-M, WuH, et al. . Forecasting and trading credit default swap indices using a deep learning model integrating Merton and LSTMs [J]. Expert Systems with Applications, 2023, 213119012

[49] LiuH, YangR. A spatial multi-resolution multi-objective data-driven ensemble model for multi-step air quality index forecasting based on real-time decomposition [J]. Comput Ind, 2021, 125103387

[50] SIAMI-NAMINI S, NAMIN A. Forecasting economics and financial time series: ARIMA vs LSTM [J]. arXiv preprint arXiv:180306386, 2018.

[51] WieringM A, van HasseltH. Ensemble algorithms in reinforcement learning [J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2008, 38(4): 930-936

[52] ZhaoD-B, WangH-T, KunS, et al. . Deep reinforcement learning with experience replay based on SARSA [C]. 2016 IEEE Symposium Series on Computational Intelligence (SSCI), 2017, Athens, IEEE: 1-6

[53] FauÿerS, SchwenkerF. Neural network ensembles in reinforcement learning [J]. Neural Processing Letters, 2015, 41(1): 55-69

[54] XuZ-X, CaoL, ChenX-L, et al. . Deep reinforcement learning with sarsa and Q-learning: A hybrid approach [J]. IEICE Transactions on Information and Systems, 2018, E101(D9): 2315-2322

[55] CaneseL, CardarilliG C, Di NunzioL, et al. . Multi-agent reinforcement learning: A review of challenges and applications [J]. Applied Sciences, 2021, 11(11): 4948

[56] GoC K, LaoB, YoshimotoJ, et al. . A reinforcement learning approach to the shepherding task using SARSA [C]. 2016 International Joint Conference on Neural Networks (IJCNN), 2016, Vancouver, BC, Canada, IEEE: 3833-3836

[57] ChaiT, DraxlerR. Root mean square error (RMSE) or mean absolute error (MAE)? - Arguments against avoiding RMSE in the literature [J]. Geoscientific Model Development Discussions, 2014, 7(1): 1525-1534

[58] de MyttenaereA, GoldenB, le GrandB, et al. . Mean absolute percentage error for regression models [J]. Neurocomputing, 2016, 192: 38-48

[59] YangR, LiuH, LiY-F. Quantifying uncertainty of marine water quality forecasts for environmental management using a dynamic multi-factor analysis and multiresolution ensemble approach [J]. Chemosphere, 2023, 331138831

[60] YangR, LiuH, LiY-F. An ensemble self-learning framework combined with dynamic model selection and divide-conquer strategies for carbon emissions trading price forecasting [J]. Chaos, Solitons & Fractals, 2023, 173113692