Performance metrics evaluation of multi-objective optimization for transportation

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Various multi-objective optimization methods are widely applied in the transportation field, enabling decision-makers to find solutions that balance trade-off objectives. Since comparing the performance of multi-objective optimization methods is generally difficult, many performance metrics are introduced to quantitatively evaluate the performance of multi-objective optimization methods. However, the effectiveness of the performance metrics needs to be further investigated. Thus, we first critically analysed a series of performance metrics, including number of solutions obtained (NOSO), overall nondominated solutions number (ONSN), normalized maximum spread (NMS), error ratio (ER), nearest ideal distance (NID), mean ideal distance (MID), spacing (SP), inverted generational distance (IGD), and hypervolume-based ratio (HR), which were extensively adopted to assess the performance of multi-objective optimization methods. We found that these performance metrics cannot always accurately reflect the quality of solutions obtained and may be misleading. Thereafter, two axioms were proposed to define the criteria for reliable performance metrics. Additionally, whether these performance metrics satisfied the two axioms was rigorously proved. The performance metrics that satisfied both axioms, i.e., NOSO, ONSN, NMS, ER, and HR, were considered reliable. Furthermore, a real-world cargo transportation case was investigated, indicating the unreliability of metrics MID and SP.

Keywords

multi-objective optimization / performance metrics / transportation / Pareto optimality

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] E. Demir, T. Bektaş, G. Laporte, A review of recent research on green road freight transportation, Eur. J. Oper. Res., 237 (2014), 775-793. https://doi.org/10.1016/j.ejor.2013.12.033

[2] Y. J. Hu, L. Yang, H. Cui, H. Wang, C. Li, Developing a balanced strategy: A multi-objective model for emissions reduction and development of civil aviation in China, Energy, 307 (2024), 132597. https://doi.org/10.1016/j.energy.2024.132597

[3] F. Mastroddi, S. Gemma, Analysis of Pareto frontiers for multidisciplinary design optimization of aircraft, Aerosp. Sci. Technol., 28 (2013), 40-55. https://doi.org/10.1016/j.ast.2012.10.003

[4] A. Ala, M. Yazdani, M. Ahmadi, A. Poorianasab, M. Y. N. Attari, An efficient healthcare chain design for resolving the patient scheduling problem: Queuing theory and MILP-ASA optimization approach, Ann. Oper. Res., 328 (2023), 3-33. https://doi.org/10.1007/s10479-023-05287-5

[5] Z. Fei, B. Li, S. Yang, C. Xing, H. Chen, L. Hanzo, A survey of multi-objective optimization in wireless sensor networks: Metrics, algorithms, and open problems, IEEE Commun. Surv. Tutorials, 19 (2016), 550-586. https://doi.org/10.1109/COMST.2016.2610578

[6] W. Deng, X. Zhang, Y. Zhou, Y. Liu, X. Zhou, H. Chen, et al., An enhanced fast non-dominated solution sorting genetic algorithm for multi-objective problems, Inf. Sci., 585 (2022), 441-453. https://doi.org/10.1016/j.ins.2021.11.052

[7] H. Zhang, Q. Ruan, Y. Jin, S. Wang, Bi-objective optimization for transportation: Generating near-optimal subsets of Pareto optimal solutions, Appl. Sci., 15 (2025), 2519. https://doi.org/10.3390/app15052519

[8] S. Petchrompo, D. W. Coit, A. Brintrup, A. Wannakrairot, A. K. Parlikad, A review of Pareto pruning methods for multi-objective optimization, Comput. Ind. Eng., 167 (2022), 108022. https://doi.org/10.1016/j.cie.2022.108022

[9] J. Zhao, Y. Li, J. Bai, L. Ma, C. Shi, G. Zhang, et al., Multi-objective optimization of marine nuclear power secondary circuit system based on improved multi-objective particle swarm optimization algorithm, Prog. Nucl. Energy, 161 (2023), 104740. https://doi.org/10.1016/j.pnucene.2023.104740

[10] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, V. G. da Fonseca, Performance assessment of multiobjective optimizers: An analysis and review, IEEE Trans. Evol. Comput., 7 (2003), 117-132. https://doi.org/10.1109/TEVC.2003.810758

[11] E. Demir, T. Bektaş, G. Laporte, The bi-objective pollution-routing problem, Eur. J. Oper. Res., 232 (2014), 464-478. https://doi.org/10.1016/j.ejor.2013.08.002

[12] R. H. Stewart, T. S. Palmer, B. DuPont, A survey of multi-objective optimization methods and their applications for nuclear scientists and engineers, Prog. Nucl. Energy, 138 (2021), 103830. https://doi.org/10.1016/j.pnucene.2021.103830

[13] Z. Wang, A. Ala, Z. Liu, W. Cui, H. Ding, G. Jin, et al., A hybrid equilibrium optimizer based on moth flame optimization algorithm to solve global optimization problems, J. Artif. Intell. Soft Comput. Res., 14 (2024), 207-235. https://doi.org/10.2478/jaiscr-2024-0012

[14] A. Ala, A. Goli, Incorporating machine learning and optimization techniques for assigning patients to operating rooms by considering fairness policies, Eng. Appl. Artif. Intell., 136 (2024), 108980. https://doi.org/10.1016/j.engappai.2024.108980

[15] W. Chen, Q. Zhuo, L. Zhang, Modeling and heuristically solving group train operation scheduling for heavy-haul railway transportation, Mathematics, 11 (2023), 2489. https://doi.org/10.3390/math11112489

[16] Z. Elmi, B. Li, B. Liang, Y. Y. Lau, M. Borowska-Stefańska, S. Wiśniewski, et al., An epsilon-constraint-based exact multi-objective optimization approach for the ship schedule recovery problem in liner shipping, Comput. Ind. Eng., 183 (2023), 109472. https://doi.org/10.1016/j.cie.2023.109472

[17] C. A. Nallolla, V. P, D. Chittathuru, S. Padmanaban, Multi-objective optimization algorithms for a hybrid AC/DC microgrid using RES: A comprehensive review, Electronics, 12 (2023), 1062. https://doi.org/10.3390/electronics12041062

[18] C. Zhao, J. Tang, W. Gao, Y. Zeng, Z. Li, Many-objective optimization of multi-mode public transportation under carbon emission reduction, Energy, 286 (2024), 129627. https://doi.org/10.1016/j.energy.2023.129627

[19] R. Jia, K. Gao, S. Cui, J. Chen, J. Andric, A multi-objective reinforcement learning-based velocity optimization approach for electric trucks considering battery degradation mitigation, Transp. Res. Part E: Logist. Transp. Rev., 194 (2025), 103885. https://doi.org/10.1016/j.tre.2024.103885

[20] Z. Huang, Z. Sheng, C. Ma, S. Chen, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res., 4 (2024), 100127. https://doi.org/10.1016/j.commtr.2024.100127

[21] N. Riquelme, C. Von Lücken, B. Baran, Performance metrics in multi-objective optimization, in 2015 Latin American Computing Conference (CLEI), IEEE, (2015), 1-11. https://doi.org/10.1109/CLEI.2015.7360024

[22] Y. Tian, R. Cheng, X. Zhang, M. Li, Y. Jin, Diversity assessment of multi-objective evolutionary algorithms: Performance metric and benchmark problems, IEEE Comput. Intell. Mag., 14 (2019), 61-74. https://doi.org/10.1109/MCI.2019.2919398

[23] Q. Fan, W. Wang, X. Yan, Multi-objective differential evolution with performance-metric-based self-adaptive mutation operator for chemical and biochemical dynamic optimization problems, Appl. Soft Comput., 59 (2017), 33-44. https://doi.org/10.1016/j.asoc.2017.05.044

[24] K. Lapa, Meta-optimization of multi-objective population-based algorithms using multi-objective performance metrics, Inf. Sci., 489 (2019), 193-204. https://doi.org/10.1016/j.ins.2019.03.054

[25] J. G. Falcon-Cardona, C. A. C. Coello, Indicator-based multi-objective evolutionary algorithms: A comprehensive survey, ACM Comput. Surv., 53 (2020), 1-35. https://doi.org/10.1145/3376916

[26] T. Wagner, H. Trautmann, Online convergence detection for evolutionary multi-objective algorithms revisited, in IEEE Congress on Evolutionary Computation, IEEE, (2010), 1-8. https://doi.org/10.1109/CEC.2010.5586474

[27] L. Marti, J. Garcia, A. Berlanga, J. M. Molina, A stopping criterion for multi-objective optimization evolutionary algorithms, Inf. Sci., 367 (2016), 700-718. https://doi.org/10.1016/j.ins.2016.07.025

[28] S. Mirjalili, A. Lewis, Novel performance metrics for robust multi-objective optimization algorithms, Swarm Evol. Comput., 21 (2015), 1-23. https://doi.org/10.1016/j.swevo.2014.10.005

[29] C. Audet, J. Bigeon, D. Cartier, S. Le Digabel, L. Salomon, Performance indicators in multiobjective optimization, Eur. J. Oper. Res., 292 (2021), 397-422. https://doi.org/10.1016/j.ejor.2020.11.016

[30] T. Okabe, Y. Jin, B. Sendhoff, A critical survey of performance indices for multi-objective optimisation, in 2003 Congress on Evolutionary Computation (CEC) , IEEE, (2003), 878-885. https://doi.org/10.1109/CEC.2003.1299759

[31] S. Zajac, S. Huber, Objectives and methods in multi-objective routing problems: A survey and classification scheme, Eur. J. Oper. Res., 290 (2021), 1-25. https://doi.org/10.1016/j.ejor.2020.07.005

[32] B. Qian, L. Wang, D. X. Huang, W. L. Wang, X. Wang, An effective hybrid DE-based algorithm for multi-objective flow shop scheduling with limited buffers, Comput. Oper. Res., 36 (2009), 209-233. https://doi.org/10.1016/j.cor.2007.08.007

[33] H. Amirian, R. Sahraeian, Augmented ϵ-constraint method in multi-objective flowshop problem with past sequence set-up times and a modified learning effect, Int. J. Prod. Res., 53 (2015), 5962-5976. https://doi.org/10.1080/00207543.2015.1033033

[34] M. Rashidnejad, S. Ebrahimnejad, J. Safari, A bi-objective model of preventive maintenance planning in distributed systems considering vehicle routing problem, Comput. Ind. Eng., 120 (2018), 360-381. https://doi.org/10.1016/j.cie.2018.05.001

[35] G. G. Yen, Z. He, Performance metric ensemble for multiobjective evolutionary algorithms, IEEE Trans. Evol. Comput., 18 (2014), 131-144. https://doi.org/10.1109/TEVC.2013.2240687

[36] J. Wang, X. Hu, E. Demeulemeester, Y. Zhao, A bi-objective robust resource allocation model for the RCPSP considering resource transfer costs, Int. J. Prod. Res., 59 (2019), 367-387. https://doi.org/10.1080/00207543.2019.1695168

[37] S. Mirjalili, S. Saremi, S. M. Mirjalili, L. D. S. Coelho, Multi-objective grey wolf optimizer: A novel algorithm for multi-criterion optimization, Expert Syst. Appl., 47 (2016), 106-119. https://doi.org/10.1016/j.eswa.2015.10.039

[38] M. Tadaros, A. Migdalas, Bi- and multi-objective location routing problems: Classification and literature review, Oper. Res., 22 (2022), 4641-4683. https://doi.org/10.1007/s12351-022-00734-w

[39] J. S. Neufeld, S. Schulz, U. Buscher, A systematic review of multi-objective hybrid flow shop scheduling, Eur. J. Oper. Res., 309 (2023), 1-23. https://doi.org/10.1016/j.ejor.2022.08.009

[40] Q. Liu, X. Li, H. Liu, Z. Guo, Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art, Appl. Soft Comput., 93 (2020), 106382. https://doi.org/10.1016/j.asoc.2020.106382