Uncertainty estimation affects predictor selection and its calibration improves materials optimization

Abstract

Uncertainty is crucial when the available data for building a predictor are insufficient, which is ubiquitous in machine-learning-driven materials studies. However, the impact of uncertainty estimation on predictor selection and materials optimization remains incompletely understood. Here, we demonstrate that in active learning, uncertainty estimation significantly influences predictor selection, as well as that the calibration of uncertainty estimation can improve the optimization. The idea is validated on three alloy datasets (Ni-based, Fe-based, and Ti-based) using three commonly used algorithms - support vector regression (SVR), neural networks (NN), and extreme gradient boosting (XGBoost) - which yield comparable predictive accuracy. It is shown that XGBoost presents more reliable uncertainty estimation than SVR and NN. Using the directly estimated uncertainty for the three predictors with similar accuracy, we find that the optimization is quite different. This suggests that uncertainty estimation plays a role in predictor selection. The uncertainty estimation is then calibrated to improve reliability, and its effect on optimization is compared with the uncalibrated case. Among the nine cases considered (three models and three datasets), eight show improved optimization when calibrated uncertainty estimation is used. This work suggests that uncertainty estimation and its calibration deserve greater attention in active learning-driven materials discovery.

Keywords

Uncertainty estimation, uncertainty calibration, active learning / optimization

References

[1] Yuan R,Balachandran PV.Accelerated discovery of large electrostrains in BaTiO₃-based piezoelectrics using active learning.Adv Mater2018;30:1702884

[2] Yuan R,Xue D.Accelerated search for BaTiO₃-based ceramics with large energy storage at low fields using machine learning and experimental design.Adv Sci2019;6:1901395 PMCID:PMC6839636

[3] Yuan R,Xu Y.Optimizing electrocaloric effect in barium titanate-based room temperature ferroelectrics: combining landau theory, machine learning and synthesis.Acta Mater2022;235:118054

[4] Rao Z,Xie R.Machine learning-enabled high-entropy alloy discovery.Science2022;378:78-85

[5] Moon J,Siek M.Active learning guides discovery of a champion four-metal perovskite oxide for oxygen evolution electrocatalysis.Nat Mater2024;23:108-15

[6] Peng B,Qin Y.Machine learning-enabled constrained multi-objective design of architected materials.Nat Commun2023;14:6630 PMCID:PMC10587057

[7] Lookman T,Xue D.Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design.npj Comput Mater2019;5:21

[8] Schmidt J,Botti S.Recent advances and applications of machine learning in solid-state materials science.npj Comput Mater2019;5:83

[9] Kusne AG,Wu C.On-the-fly closed-loop materials discovery via Bayesian active learning.Nat Commun2020;11:5966 PMCID:PMC7686338

[10] Khatamsaz D,Singh P,Allaire D.Bayesian optimization with active learning of design constraints using an entropy-based approach.npj Comput Mater2023;9:49

[11] Ren P,Chang X.A survey of deep active learning.ACM Comput Surv2022;54:1-40

[12] Chen CT.Generative deep neural networks for inverse materials design using backpropagation and active learning.Adv Sci2020;7:1902607 PMCID:PMC7055566

[13] Strieth-Kalthoff F,Segler MHS.Machine learning the ropes: principles, applications and directions in synthetic chemistry.Chem Soc Rev2020;49:6154-68

[14] Wang Z,Yin H.Data-driven materials innovation and applications.Adv Mater2022;34:e2104113

[15] Slautin BN,Funakubo H,Ziatdinov M.Bayesian conavigation: dynamic designing of the material digital twins via active learning.ACS Nano2024;18:24898-908

[16] Kim M,Ha MY.Exploring optimal water splitting bifunctional alloy catalyst by pareto active learning.Adv Mater2023;35:e2211497

[17] Pandi A,Yazdizadeh Kharrazi A.A versatile active learning workflow for optimization of genetic and metabolic networks.Nat Commun2022;13:3876 PMCID:PMC9256728

[18] Friederich P,Proppe J.Machine-learned potentials for next-generation matter simulations.Nat Mater2021;20:750-61

[19] Wang H,Du Y.Scientific discovery in the age of artificial intelligence.Nature2023;620:47-60

[20] Raccuglia P,Adler PD.Machine-learning-assisted materials discovery using failed experiments.Nature2016;533:73-6

[21] Zhang Y.A strategy to apply machine learning to small datasets in materials science.npj Comput Mater2018;4:25

[22] Shi B,Xue D.Multi-objective optimization and its application in materials science.Mater Genome Eng Adv2023;1:e14

[23] Lin J,Li T.Machine-learning-assisted intelligent synthesis of UiO-66(Ce): balancing the trade-off between structural defects and thermal stability for efficient hydrogenation of Dicyclopentadiene.Mater Genome Eng Adv2024;2:e61

[24] Oviedo F,Sun S.Fast and interpretable classification of small X-ray diffraction datasets using data augmentation and deep neural networks.npj Comput Mater2019;5:60

[25] Goodall REA.Predicting materials properties without crystal structure: deep representation learning from stoichiometry.Nat Commun2020;11:6280 PMCID:PMC7722901

[26] Zhang Y,Zhou W,Xu Y.A multi-objective feature optimization strategy for developing high-entropy alloys with optimal strength and ductility.Mater Genome Eng Adv2025;3:e70000

[27] Tian Y,Xue D.Determining multi-component phase diagrams with desired characteristics using active learning.Adv Sci2020;8:2003165 PMCID:PMC7788591

[28] Jablonka KM,Wang S,Yoo B.Bias free multiobjective active learning for materials design and discovery.Nat Commun2021;12:2312 PMCID:PMC8055971

[29] Bassman Oftelie L,Kalia RK.Active learning for accelerated design of layered materials.npj Comput Mater2018;4:74

[30] Doan HA,Qian H.Quantum chemistry-informed active learning to accelerate the design and discovery of sustainable energy storage materials.Chem Mater2020;32:6338-46

[31] Sheng Y,Yang J,Villars P.Active learning for the power factor prediction in diamond-like thermoelectric materials.npj Comput Mater2020;6:171

[32] Tao Q,Li M.Machine learning for perovskite materials design and discovery.npj Comput Mater2021;7:23

[33] Tan AR,Goldman S,Gómez-Bombarelli R.Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles.npj Comput Mater2023;9:225

[34] Palmer G,Politowicz A.Calibration after bootstrap for accurate uncertainty quantification in regression models.npj Comput Mater2022;8:115

[35] Minaei-Bidgoli B,Alinejad-Rokny H,Punch WF.Effects of resampling method and adaptation on clustering ensemble efficacy.Artif Intell Rev2014;41:27-48

[36] Rupp AA,Zumbo BD.To bayes or not to bayes, from whether to when: applications of bayesian methodology to modeling.Struct Equ Modeling2004;11:424-51

[37] Tian Y,Xue D.Role of uncertainty estimation in accelerating materials development via active learning.J Appl Phys2020;128:014103

[38] Ward L,Choudhary A.A general-purpose machine learning framework for predicting properties of inorganic materials.npj Comput Mater2016;2:16028

[39] Wang C,Wen C.Symbolic regression in materials science via dimension-synchronous-computation.J Mater Sci Technol2022;122:77-83

[40] Huang E,Singh SS.Machine-learning and high-throughput studies for high-entropy materials.Mater Sci Eng R Rep2022;147:100645

[41] Zhao S,Wang J,Yuan R.Closed-loop inverse design of high entropy alloys using symbolic regression-oriented optimization.Mater Today2025;88:263-71

[42] Wang P,Liao W.Generalizable descriptors for automatic titanium alloys design by learning from texts via large language model.Acta Mater2025;296:121275

[43] Jones DR,Welch WJ.Efficient global optimization of expensive black-box functions.J Global Optim1998;13:455-92

[44] Yang C,Jia Y,Li M.A machine learning-based alloy design system to facilitate the rational design of high entropy alloys with enhanced hardness.Acta Mater2022;222:117431

[45] Xue D,Hogden J,Xue D.Accelerated search for materials with targeted properties by adaptive design.Nat Commun2016;7:11241 PMCID:PMC4835535

[46] Yuan R,Xu Y,Li J.Machine learning combined with feature engineering to search for BaTiO₃ based ceramics with large piezoelectric constant.J Alloys Compd2022;908:164468

[47] Zhou C,Su B.Creep rupture life prediction of high-temperature titanium alloy using cross-material transfer learning.J Mater Sci Technol2024;178:39-47

[48] Varivoda D,Omee SS.Materials property prediction with uncertainty quantification: a benchmark study.Appl Phys Rev2023;10:021409

[49] Tran K,Yoon J,Xing E.Methods for comparing uncertainty quantifications for material property predictions.Mach Learn Sci Technol2020;1:025006

[50] Balanya SA,Ramos D.Adaptive temperature scaling for Robust calibration of deep neural networks.Neural Comput Appl2024;36:8073-95