Enhancing Transient Performance of Propulsion Engines in Actual Sea Using Reinforcement Learning-Based Proportional-Integral-Derivative Gain Tuning

Abstract

The engine speed control system is critical to marine propulsion, particularly for ships navigating in wave conditions. Fluctuations in propeller torque, caused by wave-induced variations in the ship’s speed and direction, can lead to undesirable effects such as increased fuel consumption and greenhouse gas emissions, increased engine wear and tear, and degraded performance. In the field of marine diesel engines, a proportional—integral—derivative (PID) control algorithm is the most widely used method for speed regulation. Aiming to maintain a constant engine speed, the PID governor continuously adjusts the fuel injection into the engine cylinders in response to propeller torque fluctuations. The quality of control, and thus engine performance, depends on a set of PID control gains. However, these optimal PID gains are typically determined during engine testbed commissioning and remain fixed, which leads to suboptimal performance under varying operational conditions. While modelbased tuning techniques can achieve satisfactory results in relatively simple control tasks, particularly when the number of tunable parameters is small, adaptive tuning of controller parameters, while meeting multiple, often conflicting objectives, presents a considerable challenge. This paper explores the application of a reinforcement learning (RL) algorithm to adaptively tune PID gains, aiming to optimize engine performance under varying sea conditions and operational demands. A specific reward function is designed to effectively balance variations in engine shaft rotational speed and fuel pump index, ensuring optimal engine efficiency. The performance of the trust-region policy optimization and proximal policy optimization algorithms is evaluated for their suitability in continuous-action space control, demonstrating stable convergence and robustness. Simulation results indicate that the proposed RL algorithm outperforms the reference PID controller, offering promising results for adaptive gain scheduling in propulsion system control.

Keywords

Propulsion engine / Proportional—integral—derivative speed control / Adaptive tuning / Reinforcement learning

References

[1] Alejandro SanjinesU, Maisincho JivajaA, AsanzaV, Lorente LeyvaLL, Peluffo OrdóñezDH. Adaptive PI controller based on a reinforcement learning algorithm for speed control of a DC motor. Biomimetics, 2023, 85434.

[2] Bertsekas DP (2012) Dynamic Programming and Optimal Control. Vol. II, 4^th Edition: Approximate Dynamic Programming 2012. Athena Scientic, 4th ed

[3] BlakeXJ, AphiratsakunN. Reinforcement learning for PID gain tuning in ball-and-beam systems: A comparative study. 2024 1st International Conference on Robotics, Engineering, Science, and Technology (RESTCON) 187–190, 2024

[4] BondarenkoO, KashiwagiM. Ensuring safe operation of ship propulsion plant in extreme sea condition. 11th International Marine Design Conference (IMDC), 2012, 3: 83-96

[5] BondarenkoO, FukudaT. Intelligent air management for ultimate flexibility of ship propulsion plant operation-1st Report: Experiment and Simulation. The Japan Institute of Marine Engineers, 2018, 53(3): 348-354.

[6] BondarenkoO, FukudaT. Development of a diesel engine’s digital twin for predicting propulsion system dynamics. Energy, 2020, 196117126.

[7] BondarenkoO, KitagawaY. Application of the Kalman filtering technique for nonlinear state estimation in propulsion system. Journal of Marine Science and Technology, 2021, 26: 618-631.

[8] BondarenkoO, KitagawaY. A comprehensive propulsion system analysis: an integrated approach utilizing engine simulation and free-running model experiments. J. of Mar. Eng. & Techn., 2024112

[9] Calvo RolleJL, GarcíaRF, López GarcíaH, Machón GonzálezIJ. Developed an expert system of an empirical method to choose correct expressions for PID controllers tuning in open loop. Proceedings of the 35th Annual Conference of IEEE Industrial Electronics, 2009, 1–6: 2044-2049

[10] DogruO, VelswamyK, IbrahimF, WuY, SundaramoorthyAS, HuangB, XuS, NixonM, BellN. Reinforcement learning approach to autonomous PID tuning. Computers, 2022, &ChemicalEngineering161107760

[11] HendricsE. Mean value modelling of large turbocharged twostroke diesel engines. Journal of Engines, 1989, 98: 986-998

[12] IMOFourth Greenhouse Gas Study, 2020

[13] IwanoJ, HirookaH, KondoK, ShigakiT. Onboard testing results for new fuel efficient main engine governor system—Development of NABTESCO 70X. Journal of Marine Engineering, 2009, 445: 105-110

[14] KarrayF, GueaiebW, Al-SharhanS. The hierarchical expert tuning of PID controllers using tools of soft computing. IEEE Transactions on Systems Man and Cybernetics, Part B-Cybernetics, 2002, 32(1): 77-90.

[15] KitagawaY, BondarenkoO, TsukadaY. An experimental method to identify a component of wave orbital motion in propeller effective inflow velocity and its effects on load fluctuations of a ship main engines in waves. Applied Ocean Research, 2019, 92101922.

[16] KitagawaY, KashiwagiM. A strip method including nx-related terms and its effects on propeller inflow velocity in waves. Journal of the Japan Society of Naval Architects and Ocean Engineers, 2019, 29: 39-50.

[17] LloydARJMSeakeeping, Ship Behaviour in Rough Weather, 1998, Hoboken. John Willey & Sons.

[18] RayA. Dynamic modelling of power plant turbines for controller design. Applied Mathematical Modelling, 1980, 4(2): 109-112.

[19] SchulmanJ, LevineS, MoritzP, JordanMI, AbbeelP. Trust region policy optimisation. Proceedings of the 32nd International Conference on Machine Learning, 2015, 37: 1889-1897

[20] SchulmanJ, WolskiF, DhariwalP, RadfordA, KlimovOProximal policy optimisation algorithms, 2017

[21] SyromyatnikovVF. Fundamentals of automation and complex automation of ship’s power plants. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment, 1983, 224(3): 193-205

[22] TsujimotoM, SogiharaN, KurodaM, SakuradaA. Development of a ship performance simulator in actual seas. ASME 2015 34th Interna tional Conference on Ocean, Offshore and Arctic Engineering, 2015, Newfoundland. ASME OMAE2015-41708, Vo11T12A047.

[23] van VeldhuizenVAutotuning PID control using Actor-Critic Deep Reinforcement Learning, 2020, Amsterdam. University of Amsterdam.

[24] WangH, Ricardez-SandovalLA. A deep reinforcement learning-based PID tuning strategy for nonlinear MIMO systems with time-varying uncertainty. IFAC-PapersOnline, 2024, 58(14): 887-892.

[25] XuXG, ChenXH, ZhengSG. Research on fuzzy PID control to electronic speed regulator. Proceedings of the SPIE 6794: article id. 679407, 2007

[26] XirosNRobust control of diesel ship propulsion, 2002, Berlin. Springer. .

[27] XirosN. PID marine engine speed regulation under full load conditions for sensitivity H_∞-norm specifications against propeller disturbance. Journal of Marine Engineering & Technology, 2004, 3(2): 3-11.

[28] ZhouY, ZhangQJ, ZhouQX, JiangHK. Research on nonlinear PID control of digital governor for diesel engine. Advanced Technology in Teaching. Advances in Intelligent and Soft Computing, 2011, 163: 295-303.

RIGHTS & PERMISSIONS

Harbin Engineering University and Springer-Verlag GmbH Germany, part of Springer Nature