An improved double deep Q-network algorithm for disassembly line balancing problems considering worker fatigue

Abstract

The rapid obsolescence of electronic products highlights the critical need for efficient resource recovery through disassembly to support sustainable development. However, conventional disassembly methods often fail to handle complex disassembly sequences and dynamic operational constraints. To address the challenge of disassembly line balancing under worker fatigue, this study proposes an attention-based double deep Transformer Q-network (DDTQN) for minimizing disassembly time. It develops a disassembly time optimization model that incorporates fatigue-induced efficiency decay to enable the simulation of realistic operational conditions. By integrating an attention mechanism into the DDQ framework, the proposed approach enhances the capacity of the model to capture intricate task dependencies, thereby improving state representation, exploration efficiency, and long-term decision-making. Experimental results across three disassembly cases indicate that DDTQN reduces the average disassembly time by 19.37% compared with benchmark algorithms—including DDQN, deep Q-network (DQN), and advantage actor-critic. The successful application of DDTQN to marine equipment disassembly demonstrates its broad applicability and effectiveness, offering a robust solution for both general disassembly lines and specialized contexts such as ship recycling.

Keywords

Double deep Transformer Q-learning algorithm / Deep reinforcement learning / Fatigue index / Disassembly line balance problem / Marine equipment recycling

References

[1] Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA. Deep reinforcement learning: a brief survey. IEEE Signal Proc Mag. 2017, 34(6): 26-38.

[2] Asadayoobi N, Taghipour S, Jaber MY. Optimising stochastic task allocation and scheduling plans for mission workers subject to learning-forgetting, fatigue-recovery, and stress-recovery effects. Expert Syst Appl. 2023, 229. 120524

[3] Aydemir-Karadag A, Turkbey O. Multi-objective optimization of stochastic disassembly line balancing with station paralleling. Comput Ind Eng. 2013, 65(3): 413-425.

[4] Baldé CP, Kuehr R, Yamamoto T, McDonald R, D’Angelo E, Althaf S et al (2024) The global e-waste monitor 2024. United Nations University (UNU), International Telecommunication Union (ITU) & International Solid Waste Association (ISWA), Geneva/Bonn, pp 1–147. https://www.itu.int/en/ITU-D/Environment/Pages/Publications/The-Global-E-waste-Monitor-2024.aspx

[5] Bi ZL, Guo XW, Wang JC, Qin SJ, Qi L, Zhao J (2022) A Q-learning-based selective disassembly sequence planning method. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 3216–3221. https://doi.org/10.1109/SMC53654.2022.9945073

[6] Çil ZA, Kizilay D, Li ZX, Öztop H. Two-sided disassembly line balancing problem with sequence-dependent setup time: a constraint programming model and artificial bee colony algorithm. Expert Syst Appl. 2022, 203. 117529

[7] Çil ZA, Mete S, Serin F. Robotic disassembly line balancing problem: a mathematical model and ant colony optimization approach. Appl Math Model. 2020, 86: 335-348.

[8] Ding LP, Feng YX, Tan JR, Gao YC. A new multi-objective ant colony algorithm for solving the disassembly line balancing problem. Int J Adv Manuf Technol. 2010, 48: 761-771.

[9] Dong CS, Liu PS, Guo XW, Qi L, Qin SJ, Xu GD (2021) Multi-objective ant lion optimizer for stochastic robotic disassembly line balancing problem subject to resource constraints. In: 2nd International Conference on Computer Vision and Data Mining (ICCVDM 2021). IOP Publishing, pp 1–7. https://doi.org/10.1088/1742-6596/2024/1/012014

[10] Güler E, Kalayci CB, Ilgin MA, Özceylan E, Güngör A (2024) Advances in partial disassembly line balancing: a state-of-the-art review. Comput Ind Eng 188:109898. https://doi.org/10.1016/j.cie.2024.109898

[11] Guo XW, Liu SX, Zhou MC, Tian GD (2016) Disassembly sequence optimization for large-scale products with multiresource constraints using scatter search and Petri nets. IEEE Trans Cybern 46(11):2435–2446. https://doi.org/10.1109/TCYB.2015.2478486

[12] Guo XW, Wei TT, Wang JC, Liu SX, Qin SJ, Qi L (2023) Multiobjective U-shaped disassembly line balancing problem considering human fatigue index and an efficient solution. IEEE Trans Comput Soc Syst 10(4):2061–2073. https://doi.org/10.1109/TCSS.2022.3217101

[13] Guo XW, Zhang ZW, Qi L, Liu SX, Tang Y, Zhao ZY (2022) Stochastic hybrid discrete grey wolf optimizer for multi-objective disassembly sequencing and line balancing planning in disassembling multiple products. IEEE Trans Autom Sci Eng 19(3):1744–1756. https://doi.org/10.1109/TASE.2021.3133601

[14] Igarashi K, Inoue M, Yamada T (2014) 2-stage optimal design and analysis for disassembly system with environmental and economic parts selection using the recyclability evaluation method. Ind Eng Manag Syst 13(1):52–66. https://doi.org/10.7232/iems.2014.13.1.052

[15] Iqbal KMJ, Heidegger P (2013) Pakistan ship-breaking outlook. Sustainable Development Policy Institute, and NGO Ship-Breaking Platform: Brussels, Belgium. https://scholar.google.com.hk/scholar?hl=zh-CN&as_sdt=0%2C5&q=Iqbal+KMJ%2C+Heidegger+P+%282013%29+Pakistan+ship-breaking+outlook.+Sustainable+Development+Policy+Institute%2C+and+NGO+Ship-Breaking+Platform%3A+Brussels%2C+Belgium.+&btnG=

[16] Jaber MY, Givi ZS, Neumann WP (2013) Incorporating human fatigue and recovery into the learning-forgetting process. Appl Math Model 37(12–13):7287–7299. https://doi.org/10.1016/j.apm.2013.02.028

[17] Kalayci CB, Gupta SM (2013) Artificial bee colony algorithm for solving sequence-dependent disassembly line balancing problem. Expert Syst Appl 40(18):7231–7241. https://doi.org/10.1016/j.eswa.2013.06.067

[18] Li ZX, Janardhanan MN. Modelling and solving profit-oriented U-shaped partial disassembly line balancing problem. Expert Syst Appl. 2021, 183. 115431

[19] Li ZX, Kucukkoc I, Zhang ZK. Iterated local search method and mathematical model for sequence-dependent U-shaped disassembly line balancing problem. Comput Ind Eng. 2019, 137. 106056

[20] Liang JY, Guo SS, Du BG, Li YB, Guo J, Yang ZJet al. . Minimizing energy consumption in multi-objective two-sided disassembly line balancing problem with complex execution constraints using dual-individual simulated annealing algorithm. J Clean Prod. 2021, 284. 125418

[21] Liu YZ, Zhou MC, Guo XW (2022) An improved Q-learning algorithm for human-robot collaboration two-sided disassembly line balancing problems. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 568–573. https://doi.org/10.1109/SMC53654.2022.9945263

[22] Matsumoto M, Yang SS, Martinsen K, Kainuma Y. Trends and research challenges in remanufacturing. Int J Precis Eng Manuf-Green Technol. 2016, 3: 129-142.

[23] Moncayo GA. International law on ship recycling and its interface with EU law. Mar Pollut Bull. 2016, 109(1): 301-309.

[24] Riggs RJ, Battaïa O, Hu SJ. Disassembly line balancing under high variety of end of life states using a joint precedence graph approach. J Manuf Syst. 2015, 37: 638-648.

[25] Sibal P (2001) A database for dismantling of obsolete vessels. West Virginia University. https://scholar.google.com.hk/scholar?hl=zh-CN&as_sdt=0%2C5&q=Sibal+P+%282001%29+A+database+for+dismantling+of+obsolete+vessels.+West+Virginia+University&btnG=

[26] Solakivi T, Kiiski T, Kuusinen T, Ojala L. The European ship recycling regulation and its market implications: ship-recycling capacity and market potential. J Clean Prod. 2021, 294. 126235

[27] Tian GD, Ren YP, Feng YX, Zhou MC, Zhang HH, Tan JR. Modeling and planning for dual-objective selective disassembly using AND/OR graph and discrete artificial bee colony. IEEE Trans Ind Inf. 2019, 15(4): 2456-2468.

[28] Tuo Y, Zhang Z, Wu T, Zeng Y, Zhang Y, Liu J. Multimanned disassembly line balancing optimization considering walking workers and task evaluation indicators. J Manuf Syst. 2024, 72: 263-286.

[29] Wang JC. Patient flow modeling and optimal staffing for emergency departments: a petri net approach. IEEE Trans Comput Soc Syst. 2023, 10(4): 2022-2032.

[30] Wang JC, Xi GP, Guo XW, Liu SX, Qin SJ, Han H. Reinforcement learning for hybrid disassembly line balancing problems. Neurocomputing. 2024, 569. 127145

[31] Wang KP, Li XY, Gao L. Modeling and optimization of multi-objective partial disassembly line balancing problem considering hazard and profit. J Clean Prod. 2019, 211: 115-133.

[32] Wang KP, Li YB, Guo J, Gao L, Li XY. Dynamic balancing of U-shaped robotic disassembly lines using an effective deep reinforcement learning approach. IEEE Trans Ind Inf. 2024, 20(4): 6855-6865.

[33] Wiering MA, van Otterlo M (2012) Reinforcement learning. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-27645-3

[34] Xu ZY, Han Y. Two sided disassembly line balancing problem with rest time of works: a constraint programming model and an improved NSGA II algorithm. Expert Syst Appl. 2024, 239. 122323

[35] Yu ZS, Zhang XH (2023) Actor-critic alignment for offline-to-online reinforcement learning. In: Proceedings of the 40th International Conference on Machine Learning. PMLR, pp 40452–40474. https://proceedings.mlr.press/v202/yu23k.html

[36] Zhang ZQ, Cai N, Zeng YQ, Li LK, Zou BS. Review of modeling theory and solution method for disassembly line balancing problems for remanufacturing. China Mech Eng. 2018, 29(21): 2636-2645.

[37] Zhou QJ, Du ZF, Liu JY, Liang J, Jiao YQ. Factors influencing green ship recycling: a conceptual framework and modeling. J Clean Prod. 2021, 322. 129155

[38] Zhu LX, Zhang ZQ, Wang Y, Cai N. On the end-of-life state oriented multi-objective disassembly line balancing problem. J Intell Manuf. 2020, 31: 1403-1428.

[39] Zou BS, Zhang ZQ, Li LK, Cai N. Modeling and optimization for two-sided disassembly line balancing problems. China Mech Eng. 2018, 29(9): 1090-1097.

RIGHTS & PERMISSIONS

The Author(s)