An evaluation framework for integrated learning and optimization by revisiting parameters and hyperparameters

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Integrated learning and optimization (ILO), an emerging paradigm for contextual optimization also known as “smart predict-then-optimize” or “end-to-end optimization”, typically presents major challenges for practical applications in the community of operations research and management science (OR/MS). This study developed a general evaluation framework that systematically assesses the difficulty of implementing ILO for contextual optimization problems. It distinguishes between computationally tractable and challenging problems, offering the corresponding exact and approximate solution strategies to obtain the final prescriptions, respectively. Our main contributions are threefold. First, we identified conceptual ambiguities in traditional definitions of parameters and hyperparameters in machine learning (ML) models, proposing new complexity- based definitions that strengthen the theoretical foundations. Second, building on this, we developed optimization frameworks to identify the optimal ML models and prescriptions under sequential learning and optimization (SLO) and ILO, respectively. The ILO evaluation framework was then established, which is structured around several key questions regarding the specific characteristics of the ML model and the downstream optimization problem. Third, we illustrated the applicability of the proposed evaluation framework through a numerical case study on a stochastic binary decision- making problem and provided a cautious discussion of the observed numerical comparison. Moreover, we analyzed and compared the efficacy and efficiency of SLO and ILO. This study provides valuable insights and practical guidance for implementing ILO in solving real-world problems in OR/MS. Furthermore, it enriches the literature on the comparative effectiveness of SLO and ILO.

Keywords

machine learning / contextual optimization / sequential learning and optimization / integrated learning and optimization / evaluation framework

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] Z. Hu, M. Xu, Q. Cheng, Multimodal large-language model empowering next- generation autonomous driving systems, J. Intell. Connected Veh., 8 (2025), 9210059. https://doi.org/10.26599/jicv.2025.9210059

[2] Z. Gao, B. Jia, D. Xie, W. Wang, J. Wu, A discussion on the complexity and transit mechanisms of urban traffic systems, Engineering, 44 (2025), 24-29. https://doi.org/10.1016/j.eng.2024.12.006

[3] U. Sadana, A. Chenreddy, E. Delage, A. Forel, E. Frejinger, T. Vidal, A survey of contextual optimization methods for decision-making under uncertainty, Eur. J. Oper. Res., 320 (2025), 271-289. https://doi.org/10.1016/j.ejor.2024.03.020

[4] D. Bertsimas, N. Kallus, From predictive to prescriptive analytics, Manage. Sci., 66 (2020), 1025-1044. https://doi.org/10.1287/mnsc.2018.3253

[5] A. N. Elmachtoub, P. Grigas, Smart “predict, then optimize”, Manage. Sci., 68 (2022), 9-26. https://doi.org/10.1287/mnsc.2020.3922

[6] Y. Deng, S. Sen, Predictive stochastic programming, Comput. Manage. Sci., 19 (2022), 65-98. https://doi.org/10.1007/s10287-021-00400-0

[7] N. Ho-Nguyen, F. Kılınç-Karzan, Risk guarantees for end-to-end prediction and optimization processes, Manage. Sci., 68 (2022), 8680-8698. https://doi.org/10.1287/mnsc.2022.4321

[8] B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, et al., Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges, Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery, 13 (2023), e1484. https://doi.org/10.1002/widm.1484

[9] T. Yu, H. Zhu, Hyper-parameter optimization: A review of algorithms and applications, preprint, arXiv:2003.05689.

[10] M. Feurer, F. Hutter, Hyperparameter Optimization , Springer International Publishing, Cham, 2019. https://doi.org/10.1007/978-3-030-05318-51

[11] J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algorithms for hyper-parameter optimization, Adv. Neural Inf. Process. Syst., 24 (2011).

[12] J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, J. Mach. Learn. Res., 13 (2012), 281-305.

[13] J. Snoek, H. Larochelle, R. P. Adams, Practical Bayesian optimization of machine learning algorithms, Adv. Inf. Process. Syst., 25 (2012).

[14] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, N. de Freitas, Taking the human out of the loop: A review of Bayesian optimization, Proc. IEEE, 104 (2015), 148-175. https://doi.org/10.1109/JPROC.2015.2494218

[15] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar, Hyperband: A novel bandit- based approach to hyperparameter optimization, J. Mach. Learn. Res., 18 (2018), 1-52.

[16] C. Thornton, F. Hutter, H. H. Hoos, K. Leyton-Brown, Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms, in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, (2013), 847-855. https://doi.org/10.1145/2487575.2487629

[17] M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, F. Hutter, Efficient and robust automated machine learning, Adv. Neural Inf. Process. Syst., 28 (2015).

[18] L. Yang, A. Shami, On hyperparameter optimization of machine learning algorithms: Theory and practice, Neurocomputing, 415 (2020), 295-316. https://doi.org/10.1016/j.neucom.2020.07.061

[19] F. Hutter, H. H. Hoos, K. Leyton-Brown, Sequential model-based optimization for general algorithm configuration, in Proceedings of the International Conference on Learning and Intelligent Optimization, Springer, (2011), 507-523. https://doi.org/10.1007/978-3-642-25566-340

[20] A. Klein, S. Falkner, S. Bartels, P. Hennig, F. Hutter, Fast Bayesian optimization of machine learning hyperparameters on large datasets, in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , PMLR, (2017), 528-536.

[21] M. Feurer, B. Letham, F. Hutter, E. Bakshy, Practical transfer learning for Bayesian optimization, preprint, arXiv:1802.02219.

[22] S. Falkner, A. Klein, F. Hutter, BOHB: Robust and efficient hyperparameter optimization at scale, in Proceedings of the 35th International Conference on Machine Learning , PMLR, (2018), 1437-1446.

[23] E. Real, A. Aggarwal, Y. Huang, Q. V. Le, Regularized evolution for image classifier architecture search, in Proceedings of the AAAI Conference on Artificial Intelligence, (2019), 4780-4789. https://doi.org/10.1609/aaai.v33i01.33014780

[24] D. Hernández-Lobato, J. M. Hernández-Lobato, A. Shah, R. P. Adams, Predictive entropy search for multi-objective Bayesian optimization, in Proceedings of the 33rd International Conference on Machine Learning , PMLR, (2016), 1492-1501.

[25] B. Paria, K. Kandasamy, B. Póczos, A flexible framework for multi-objective Bayesian optimization using random scalarizations, in Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, PMLR, (2020), 766-776.

[26] B. Zoph, Q. V. Le, Neural architecture search with reinforcement learning, in Proceedings of the International Conference on Learning Representations, 2017.

[27] T. Elsken, J. H. Metzen, F. Hutter, Neural architecture search: A survey, J. Mach. Learn. Res., 20 (2019), 1-21.

[28] M. Baratchi, C. Wang, S. Limmer, J. N. van Rijn, H. Hoos, T. Bäck, et al., Automated machine learning: Past, present and future, Artif. Intell. Rev., 57 (2024), 122. https://doi.org/10.1007/s10462-024-10726-1

[29] L. Franceschi, M. Donini, V. Perrone, A. Klein, C. Archambeau, M. Seeger, et al., Hyperparameter optimization in machine learning, Found. Trends Mach. Learn., 18 (2025), 975-1109. https://doi.org/10.1561/2200000088

[30] P. Gijsbers, M. L. P. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, et al., AMLB: An AutoML benchmark, J. Mach. Learn. Res., 25 (2024), 1-65.

[31] F. Mohr, M. Wever, Naive automated machine learning, Mach. Learn., 112 (2023), 1131-1170. https://doi.org/10.1007/s10994-022-06200-0

[32] N. Mallik, E. Bergman, C. Hvarfner, D. Stoll, M. Janowski, M. Lindauer, et al., PriorBand: Practical hyperparameter optimization in the age of deep learning, Adv. Neural Inf. Process. Syst., 36 (2023), 7377-7391. https://doi.org/10.52202/075280-0323

[33] S. Müller, M. Feurer, N. Hollmann, F. Hutter, PFNs4BO: In-context learning for Bayesian optimization, preprint, arXiv:2305.17535.

[34] M. Kuhn, K. Johnson, Applied Predictive Modeling , Springer, New York, 2013. https://doi.org/10.1007/978-1-4614-6849-3

[35] P. Probst, A. L. Boulesteix, B. Bischl, Tunability: Importance of hyperparameters of machine learning algorithms, preprint, arXiv:1802.09596.

[36] V. Kumar, M. L. Garg, Predictive analytics: A review of trends and techniques, Int. J. Comput. Appl., 182 (2018), 31-37. https://doi.org/10.5120/ijca2018917434

[37] N. Mishra, S. Silakari, Predictive analytics: A survey, trends, applications, oppurtunities & challenges, Int. J. Comput. Sci. Inf. Technol., 3 (2012), 4434-4438.

[38] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, J. Sohl-Dickstein, On the expressive power of deep neural networks, preprint, arXiv:1606.05336.

[39] V. Vapnik, The Nature of Statistical Learning Theory , Springer Science & Business Media, Berlin, 2013. https://doi.org/10.1007/978-1-4757-3264-1

[40] P. L. Bartlett, S. Mendelson, Rademacher and Gaussian complexities: Risk bounds and structural results, J. Mach. Learn. Res., 3 (2002), 463-482.

[41] H. Buhrman, R. De Wolf, Complexity measures and decision tree complexity: A survey, Theor. Comput. Sci., 288 (2002), 21-43. https://doi.org/10.1016/S0304-3975(01)00144-X

[42] N. Bulso, M. Marsili, Y. Roudi, On the complexity of logistic regression models, Neural Comput. , 31 (2019), 1592-1623. https://doi.org/10.1162/necoa 01207

[43] T. Liang, T. Poggio, A. Rakhlin, J. Stokes, Fisher-rao metric, geometry, and complexity of neural networks, preprint, arXiv:1711.01530.

[44] X. Hu, L. Chu, J. Pei, W. Liu, J. Bian, Model complexity of deep learning: A survey, Knowl. Inf. Syst., 63 (2021), 2585-2619. https://doi.org/10.1007/s10115-021-01605-0

[45] S. Liu, S. Cong, L. Yang, Survey of research on autonomous driving testing with large models, Commun. Transp. Res., 5 (2025), 100179. https://doi.org/10.1016/j.commtr.2025.100179

[46] R. Kannan, G. Bayraksan, J. R. Luedtke, Residuals-based distributionally robust optimization with covariate information, preprint, arXiv:2012.01088.

[47] D. Bertsimas, B. Van Parys, Bootstrap robust prescriptive analytics, Math. Program., 195 (2022), 39-78. https://doi.org/10.1007/s10107-021-01679-2

[48] G. Y. Ban, C. Rudin, The big data newsvendor: Practical insights from machine learning, Oper. Res., 67 (2019), 90-108. https://doi.org/10.1287/opre.2018.1757

[49] S. Liu, L. He, Z. Shen, On-time last-mile delivery: Order assignment with travel-time predictors, Manage. Sci., 67 (2021), 4095-4119. https://doi.org/10.1287/mnsc.2020.3741

[50] A. Esteban-Pérez, J. M. Morales, Distributionally robust optimal power flow with contextual information, Eur. J. Oper. Res., 306 (2023), 1047-1058. https://doi.org/10.1016/j.ejor.2022.10.024

[51] Z. Y. Chen, M. Sun, X. X. Han, Prediction-driven collaborative emergency medical resource allocation with deep learning and optimization, J. Oper. Res. Soc., 74 (2023), 590-603. https://doi.org/10.1080/01605682.2022.2101953

[52] G. Perakis, M. Sim, Q. Tang, P. Xiong, Robust pricing and production with information partitioning and adaptation, Manage. Sci., 69 (2023), 1398-1419. https://doi.org/10.1287/mnsc.2022.4446

[53] V. Monga, Y. Li, Y. C. Eldar, Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing, IEEE Signal Process Mag., 38 (2021), 18-44. https://doi.org/10.1109/MSP.2020.3016905

[54] M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-López, et al., Efficient and modular implicit differentiation, preprint, arXiv:2105.15183.

[55] Q. Berthet, M. Blondel, O. Teboul, M. Cuturi, J. P. Vert, F. Bach, Learning with differentiable pertubed optimizers, preprint, arXiv:2002.08676.

[56] C. Wang, D. Jia, W. Wang, D. Ngoduy, B. Peng, J. Wang, A knowledge-informed deep learning paradigm for generalizable and stability-optimized car-following models, Commun. Transp. Res., 5 (2025), 100211. https://doi.org/10.1016/j.commtr.2025.100211

[57] H. Chu, W. Zhang, P. Bai, Y. Chen, Data-driven optimization for last-mile delivery, Complex Intell. Syst., 9 (2023), 2271-2284. https://doi.org/10.1007/s40747-021-00293-1

[58] X. Tian, R. Yan, Y. Liu, S. Wang, A smart predict-then-optimize method for targeted and cost-effective maritime transportation, Transp. Res. Part B Methodol., 172 (2023), 32-52. https://doi.org/10.1016/j.trb.2023.03.009

[59] R. Yan, S. Wang, K. Fagerholt, A semi-“smart predict then optimize”(semi-SPO) method for efficient ship inspection, Transp. Res. Part B Methodol., 142 (2020), 100-125. https://doi.org/10.1016/j.trb.2020.09.014

[60] S. Cong, M. Zheng, W. Zhang, Y. Bie, E. Szczepański, Closed-loop coordination of connected and automated vehicles in structured road networks, Commun. Transp. Res., 5 (2025), 100182. https://doi.org/10.1016/j.commtr.2025.100182

[61] L. Han, X. Ma, Y. Wang, L. He, Y. Li, L. Zhang, et al., Safe and efficient DRL driving policies using fuzzy logic for urban lane changing scenarios, J. Intell. Connected Veh., 8 (2025), 9210054. https://doi.org/10.26599/jicv.2024.9210054

[62] C. Ma, X. Huang, Y. Zhao, T. Wang, B. Du, GRU-LSTM model based on the SSA for short-term traffic flow prediction, J. Intell. Connected Veh., 8 (2025), 9210051. https://doi.org/10.26599/jicv.2024.9210051

[63] Y. Hu, N. Kallus, X. Mao, Fast rates for contextual linear optimization, Manage. Sci., 68 (2022), 4236-4245. https://doi.org/10.1287/mnsc.2022.4383

[64] A. N. Elmachtoub, H. Lam, H. Zhang, Y. Zhao, Estimate-then-optimize versus integrated- estimation-optimization versus sample average approximation: A stochastic dominance perspective, preprint, arXiv:2304.06833.

[65] T. Vanderschueren, T. Verdonck, B. Baesens, W. Verbeke, Predict-then-optimize or predict-and- optimize? An empirical evaluation of cost-sensitive learning strategies, Inf. Sci., 594 (2022), 400-415. https://doi.org/10.1016/j.ins.2022.02.021

[66] J. Mandi, J. Kotary, S. Berden, M. Mulamba, V. Bucarey, T. Guns, et al., Decision-focused learning: Foundations, state of the art, benchmark and future opportunities, J. Artif. Intell. Res., 80 (2024), 1623-1701. https://doi.org/10.1613/jair.1.15320

[67] S. Fujita, G. Ramey, Exogenous versus endogenous separation, Am. Econ. J. Macroecon., 4 (2012), 68-93. https://doi.org/10.1257/mac.4.4.68

[68] B. Hssina, A. Merbouha, H. Ezzikouri, M. Erritali, A comparative study of decision tree ID3 and C4.5, Int. J. Adv. Comput. Sci. Appl., 4 (2014), 13-19. https://doi.org/10.14569/SpecialIssue.2014.040203

[69] J. R. Cano, Analysis of data complexity measures for classification, Expert Syst. Appl., 40 (2013), 4820-4831. https://doi.org/10.1016/j.eswa.2013.02.025

[70] E. G. Talbi, A taxonomy of metaheuristics for bi-level optimization, in Metaheuristics for Bi-level Optimization, Springer, (2013), 1-39. https://doi.org/10.1007/978-3-642-37838-61