From microstructure to performance optimization: Innovative applications of computer vision in materials science

Abstract

The rapid advancements in computer vision (CV) technology have transformed the traditional approaches to material microstructure analysis. This review outlines the history of CV and explores the applications of deep-learning (DL)-driven CV in four key areas of materials science: microstructure-based performance prediction, microstructure information generation, microstructure defect detection, and crystal structure-based property prediction. The CV has significantly reduced the cost of traditional experimental methods used in material performance prediction. Moreover, recent progress made in generating microstructure images and detecting microstructural defects using CV has led to increased efficiency and reliability in material performance assessments. The DL-driven CV models can accelerate the design of new materials with optimized performance by integrating predictions based on both crystal and microstructural data, thereby allowing for the discovery and innovation of next-generation materials. Finally, the review provides insights into the rapid interdisciplinary developments in the field of materials science and future prospects.

Keywords

microstructure / deep learning / computer vision / performance prediction / image generation

References

[1] W.S. Xia, X.B. Zhao, J.W. Wang, et al., New strategy to improve the overall performance of single-crystal superalloys by designing a bimodal γ′ precipitation microstructure, Acta Mater., 257(2023), art. No. 119200.

[2] Z.X. Zhang, Y.Q. Guan, Q. Huang, et al., Insight of the microstructure evolution and performance enhancement of spinodal decomposition in (Ti, Zr)C composite carbide ceramics: Multiscale simulation and experimental investigation, Acta Mater., 282(2025), art. No. 120487.

[3] X.Y. Zhou, H.H. Wu, Y. Wu, et al., Formation and strengthening mechanism of ordered interstitial complexes in multi-principle element alloys, Acta Mater., 281(2024), art. No. 120364.

[4] Guo CY, Wang EH, Liu YS, Zheng YP, Yang T, Hou XM. Effect of Sn doping concentration on the oxidation of Al-containing MAX phase (Ti₃AlC₂) combining simulation with experiment. Fundam. Res., 2022, 2(1): 114

[5] Huang YB, Xu N, Lu HL, Ren Y, Li SL, Wang YD. Microstructures and micromechanical behaviors of high-entropy alloys investigated by synchrotron X-ray and neutron diffraction techniques: A review. Int. J. Miner. Metall. Mater., 2024, 31(6): 1333.

[6] Wei J, Chu X, Sun XY, et al. . Machine learning in materials science. InfoMat, 2019, 1(3): 338.

[7] C.Y. Guo, X.J. Duan, Z. Fang, et al., A new strategy for long-term complex oxidation of MAX phases: Database generation and oxidation kinetic model establishment with aid of machine learning, Acta Mater., 241(2022), art. No. 118378.

[8] Ouyang D, He B, Ghorbani A, et al. . Video-based AI for beat-to-beat assessment of cardiac function. Nature, 2020, 580(7802): 252

[9] Ravuri S, Lenc K, Willson M, et al. . Skilful precipitation nowcasting using deep generative models of radar. Nature, 2021, 597(7878): 672

[10] A. Agrawal and A. Choudhary, Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science, APL Mater., 4(2016), No. 5, art. No. 053208.

[11] G.R. Schleder, A.C.M. Padilha, C.M. Acosta, M. Costa, and A. Fazzio, From DFT to machine learning: Recent approaches to materials science–A review, J. Phys. Mater., 2(2019), No. 3, art. No. 032001.

[12] Fu YZ, Downey ARJ, Yuan L, Zhang TY, Pratt A, Balogun Y. Machine learning algorithms for defect detection in metal laser-based additive manufacturing: A review. J. Manuf. Process., 2022, 75: 693.

[13] Tian ZH, Shang CL, Zhang CL, et al. . Review of precipitation strengthening in ultrahigh-strength martensitic steel. Int. J. Miner. Metall. Mater., 2025, 32(2): 256.

[14] K. Choudhary, B. DeCost, C. Chen, et al., Recent advances and applications of deep learning methods in materials science, npj Comput. Mater., 8(2022), art. No. 59.

[15] T. Xie and J.C. Grossman, Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties, Phys. Rev. Lett., 120(2018), No. 14, art. No. 145301.

[16] X.Y. Shi, L.M. Zhou, Y.H. Huang, Y.J. Wu, and Z.J. Hong, A review on the applications of graph neural networks in materials science at the atomic scale, Mater. Genome Eng. Adv., 2(2024), No. 2, art. No. e50.

[17] Kirsch RA. SEAC and the start of image processing at the national bureau of standards. IEEE Ann. Hist. Comput., 1998, 20(2): 7.

[18] Roberts LG. Machine Perception of Three Dimensional Solids, 1963, Cambridge. Massachusetts Institute of Technology

[19] Papert SA. The Summer Vision Project, 1966, Cambridge, MA. MIT Artificial Intelligence Laboratory[2025-01-07]

[20] Minsky M, Papert S. An introduction to computational geometry. Cambridge tiass., HIT, 1969, 479(480): 104

[21] Szeliski R. Computer Vision: Algorithms and Applications, 2011, London. Springer.

[22] Marr D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information, 2010, Cambridge, MA. The MIT Press.

[23] Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern., 1980, 36(4): 193

[24] LeCun Y, Boser B, Denker JS, et al. . Backpropagation applied to handwritten zip code recognition. Neural Comput., 1989, 1(4): 541.

[25] Rumelhart DE, Hinton GE, Williams RJRumelhart DE, McClelland JL. Learning internal representations by error propagation. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1: Foundations, 1986, Cambridge. MIT Press318.

[26] Lowe DG. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis., 2004, 60(2): 91.

[27] Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded-up robust features (SURF). Comput. Vis. Image Underst., 2008, 110(3): 346.

[28] Canny J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 1986, PAMI-8(6): 679.

[29] Dalal N, Triggs B. Histograms of oriented gradients for human detection. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005

[30] Cootes TF, Taylor CJ, Cooper DH, Graham J. Active shape models-their training and application. Comput. Vis. Image Underst., 1995, 61(1): 38.

[31] Horn BKP, Schunck BG. Determining optical flow. Artif. Intell, 1981, 17(1–3): 185.

[32] Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 2001

[33] Lienhart R, Maydt J. An extended set of haar-like features for rapid object detection. Proceedings of International Conference on Image Processing, 2002

[34] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun. ACM, 2017, 60(6): 84.

[35] Russakovsky O, Deng J, Su H, et al. . ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 2015, 115(3): 211.

[36] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436

[37] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014

[38] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 20153431.

[39] Xie SN, Tu ZW. Holistically-nested edge detection. 2015 IEEE International Conference on Computer Vision (ICCV), 20151395.

[40] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, 201518

[41] He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016770.

[42] Goodfellow I, Abadie JP, Mirza M, et al. . Generative adversarial nets. Neural Information Processing Systems, 20142672

[43] He KM, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 20172980.

[44] Dosovitskiy A. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021

[45] Shen CG, Wang CC, Huang MH, Xu N, Zwaag SVD, Xu W. A generic high-throughput microstructure classification and quantification method for regular SEM images of complex steel microstructures combining EBSD labeling and deep learning. J. Mater. Sci. Technol., 2021, 93: 191.

[46] A. Leitherer, B.C. Yeo, C.H. Liebscher, and L.M. Ghiringhelli, Automatic identification of crystal structures and interfaces via artificial-intelligence-based electron microscopy, npj Comput. Mater., 9(2023), art. No. 179.

[47] Kondo R, Yamakawa S, Masuoka Y, Tajima S, Asahi R. Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics. Acta Mater., 2017, 141: 29.

[48] Z.R. Pei, K.A. Rozman, Ö.N. Doǧan, et al., Machine-learning microstructure for inverse material design, Adv. Sci., 8(2021), No. 23, art. No. 2101207.

[49] G. Du, L.M. Zhou, Y.H. Huang, Y.J. Wu, and Z.J. Hong, Topological data analysis assisted machine learning for polar topological structures in oxide superlattices, Acta Mater., 282(2025), art. No. 120467.

[50] Xu SH, McLeod AS, Chen XZ, et al. . Deep learning analysis of polaritonic wave images. ACS Nano, 2021, 15(11): 18182

[51] S. Ferdousi, R. Advincula, A.P. Sokolov, W. Choi, and Y.J. Jiang, Investigation of 3D printed lightweight hybrid composites via theoretical modeling and machine learning, Composites Part B, 265(2023), art. No. 110958.

[52] K.J. DeMille, R. Hall, J.R. Leigh, I. Guven, and A.D. Spear, Materials design using genetic algorithms informed by convolutional neural networks: Application to carbon nanotube bundles, Composites Part B, 286(2024), art. No. 111751.

[53] D. Ren, C.C. Wang, X.L. Wei, Q.Q. Lai, and W. Xu, Building a quantitative composition-microstructure-property relationship of dual-phase steels via multimodal data mining, Acta Mater., 252(2023), art. No. 118954.

[54] Wu JL, Yin XL, Xiao H. Seeing permeability from images: Fast prediction with convolutional neural networks. Sci. Bull., 2018, 63(18): 1215.

[55] X.Q. Jia, W. Li, Q. Lu, K. Zhang, H. Du, Y.T. Xu, and X.J. Jin, Towards accurate prediction for ultra-low carbon tempered martensite property through the cross-correlated substructures, Mater. Des., 211(2021), art. No. 110126.

[56] Külekçi G, Hacıefendioǧlu K, Başaǧa HB. Enhancing mineral processing with deep learning: Automated quartz identification using thin section images. Int. J. Miner. Metall. Mater., 2025, 32(4): 802.

[57] Z.S. Jiang, J.Z. Li, Y. Yang, et al., Machine-learning-revealed statistics of the particle-carbon/binder detachment in lithium-ion battery cathodes, Nat. Commun., 11(2020), No. 1, art. No. 2310.

[58] J. Na, S.J. Kim, H. Kim, S.H. Kang, and S. Lee, A unified microstructure segmentation approach via human-in-the-loop machine learning, Acta Mater., 255(2023), art. No. 119086.

[59] Dong XC, Li HW, Jiang ZT, et al. . 3D deep learning enables accurate layer mapping of 2D materials. ACS Nano, 2021, 15(2): 3139

[60] Chan RR, Pietryga J, Landy KM, Gibson KJ, Mirkin CA. Microcrystal growth pathways investigated with machine learning segmentation and classification in scanning electron microscopy. ACS Nano, 2024, 18(48): 33073

[61] C. Gorynski, M. Frei, F.E. Kruis, and M. Winterer, Machine learning based quantitative characterization of microstructures, Acta Mater., 256(2023), art. No. 119106.

[62] ASTM International. ASTM Standard E112–13: Standard Test Methods for Determining Average Grain Size, 2013, West Conshohocken. ASTM International

[63] N. Kulesh, A. Bolyachkin, I. Suzuki, Y.K. Takahashi, H.S. Amin, and K. Hono, Data-driven optimization of FePt heat-assisted magnetic recording media accelerated by deep learning TEM image segmentation, Acta Mater., 255(2023), art. No. 119039.

[64] P. Liu, H.Y. Huang, X. Jiang, et al., Evolution analysis of γ′ precipitate coarsening in co-based superalloys using kinetic theory and machine learning, Acta Mater., 235(2022), art. No. 118101.

[65] I. Lobato, T. Friedrich, and S. Van Aert, Deep convolutional neural networks to restore single-shot electron microscopy images, npj Comput. Mater., 10(2024), art. No. 10.

[66] Z.H. Cao, Q. Liu, Q.C. Liu, X.B. Yu, J.J. Kruzic, and X.P. Li, A machine learning method to quantitatively predict alpha phase morphology in additively manufactured Ti–6Al–4V, npj Comput. Mater., 9(2023), art. No. 195.

[67] E. Azqadan, H. Jahed, and A. Arami, Predictive microstructure image generation using denoising diffusion probabilistic models, Acta Mater., 261(2023), art. No. 119406.

[68] Im SW, Zhang D, Han JH, et al. . Investigating chiral morphogenesis of gold using generative cellular automata. Nat. Mater., 2024, 23(7): 977

[69] A. Sciazko, Y. Komatsu, T. Shimura, and N. Shikazono, Prediction of electrode microstructure evolutions with physically constrained unsupervised image-to-image translation networks, npj Comput. Mater., 10(2024), art. No. 49.

[70] S.Y. Zhu, D. Saritürk, and R. Arróyave, Accelerating CAL-PHAD-based phase diagram predictions in complex alloys using universal machine learning potentials: Opportunities and challenges, Acta Mater., 286(2025), art. No. 120747.

[71] Yang ZJ, Yabansu YC, Jha D, et al. . Establishing structure-property localization linkages for elastic deformation of three-dimensional high contrast composites using deep learning approaches. Acta Mater., 2019, 166: 335.

[72] A. Gupta, A. Bhaduri, and L.G. Brady, Accelerated multiscale mechanics modeling in a deep learning framework, Mech. Mater., 184(2023), art. No. 104709.

[73] Z.Z. Yang, C.H. Yu, and M.J. Buehler, Deep learning model to predict complex stress and strain fields in hierarchical composites, Sci. Adv., 7(2021), No. 15, art. No. eabd7416.

[74] M.M. Rashid, T. Pittie, S. Chakraborty, and N.M.A. Krishnan, Learning the stress-strain fields in digital composites using Fourier neural operator, iScience, 25(2022), No. 11, art. No. 105452.

[75] Y.L. Wu, J.M. Wang, X.L. Guo, and L.F. Zhang, Conditional generative adversarial network-based predictive method for crack initiation in a dual-phase austenite stainless weld, Corros. Sci., 240(2024), art. No. 112494.

[76] Shen YF, Pokharel R, Nizolek TJ, Kumar A, Lookman T. Convolutional neural network-based method for real-time orientation indexing of measured electron backscatter diffraction patterns. Acta Mater., 2019, 170: 118.

[77] Feng JX, Teng QZ, He XH, Wu XH. Accelerating multi-point statistics reconstruction method for porous media via deep learning. Acta Mater., 2018, 159: 296.

[78] J. Na, G. Kim, S.H. Kang, S.J. Kim, and S. Lee, Deep learning-based discriminative refocusing of scanning electron microscopy images for materials science, Acta Mater., 214(2021), art. No. 116987.

[79] A. Khan, C.H. Lee, P.Y. Huang, and B.K. Clark, Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images, npj Comput. Mater., 9(2023), art. No. 85.

[80] D.K. Jangid, N.R. Brodnik, M.G. Goebel, et al., Adaptable physics-based super-resolution for electron backscatter diffraction maps, npj Comput. Mater., 8(2022), art. No. 255.

[81] Niu T, Nasim M, Annadanam RGS, et al. . Recent studies on void shrinkage in metallic materials subjected to in situ heavy ion irradiations. JOM, 2020, 72(11): 4008.

[82] Visheratina A, Visheratin A, Kumar P, Veksler M, Kotov NA. Chirality analysis of complex microparticles using deep learning on realistic sets of microscopy images. ACS Nano, 2023, 17(8): 7431

[83] Liu ZD, Li DY, Zhu QQ, Zhang CX, Ma JY, Zhao JJ. Intelligent method to experimentally identify the fracture mechanism of red sandstone. Int. J. Miner. Metall. Mater., 2023, 30(11): 2134.

[84] J. Madsen, P. Liu, J. Kling, et al., A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images, Adv. Theory Simul., 1(2018), No. 8, art. No. 1800037.

[85] Ziatdinov M, Dyck O, Maksov A, et al. . Deep learning of atomically resolved scanning transmission electron microscopy images: Chemical identification and tracking local transformations. ACS Nano, 2017, 11(12): 12742

[86] W. Li, K.G. Field, and D. Morgan, Automated defect analysis in electron microscopic images, npj Comput. Mater., 4(2018), art. No. 36.

[87] Badmos O, Kopp A, Bernthaler T, Schneider G. Image-based defect detection in lithium-ion battery electrode using convolutional neural networks. J. Intell. Manuf., 2020, 31(4): 885.

[88] M.R. Shen, G.Z. Li, D.X. Wu, et al., Multi defect detection and analysis of electron microscopy images with deep learning, Comput. Mater. Sci., 199(2021), art. No. 110576.

[89] Dey B, Goswami D, Halder S, Khalil K, Leray P, Bayoumi MA. Deep learning-based defect classification and detection in SEM images. Metrology, Inspection, and Process Control XXXVI, 202283.

[90] A. Maksov, O. Dyck, K. Wang, et al., Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS₂, npj Comput. Mater., 5(2019), art. No. 12.

[91] Z. Maxim, S. Jesse, B.G. Sumpter, S.V. Kalinin, and O. Dyck, Tracking atomic structure evolution during directed electron beam induced Si-atom motion in graphene via deep machine learning, Nanotechnology, 32(2021), No. 3, art. No. 035703.

[92] M.R. Shen, G.Z. Li, D.X. Wu, et al., A deep learning based automatic defect analysis framework for in situ TEM ion irradiations, Comput. Mater. Sci., 197(2021), art. No. 110560.

[93] R. Sainju, W.Y. Chen, S. Schaefer, et al, DefectTrack: A deep learning-based multi-object tracking algorithm for quantitative defect analysis of in situ TEM videos in real-time, Sci. Rep., 12(2022), art. No. 15705.

[94] Lee CH, Khan A, Luo D, et al. . Deep learning enabled strain mapping of single-atom defects in two-dimensional transition metal dichalcogenides with sub-picometer precision. Nano Lett., 2020, 20(5): 3369

[95] R. Jacobs, M.R. Shen, Y.H. Liu, et al., Performance and limitations of deep learning semantic segmentation of multiple defects in transmission electron micrographs, Cell Rep. Phys. Sci., 3(2022), No. 5, art. No. 100876.

[96] W.Y. Chen, Z.G. Mei, L. Ward, et al., In-situ TEM investigation of void swelling in nickel under irradiation with analysis aided by computer vision, Acta Mater., 254(2023), art. No. 119013.

[97] P. Komninos, A.E.C. Verraest, N. Eleftheroglou, and D. Zarouchas, Intelligent fatigue damage tracking and prognostics of composite structures utilizing raw images via interpretable deep learning, Composites Part B, 287(2024), art. No. 111863.

[98] T. Bilyk, A.M. Goryaeva, M.C. Marinica, et al., Accurate quantification of dislocation loops in complex functional alloys enabled by deep learning image analysis, Sci. Rep., 14(2024), art. No. 25168.

[99] Duan XJ, Fang Z, Yang T, et al. . Maximizing the mechanical performance of Ti₃AlC₂-based MAX phases with aid of machine learning. J. Adv. Ceram., 2022, 11(8): 1307.

[100] S.S. Omee, N.H. Fu, R.Z. Dong, M. Hu, and J.J. Hu, Structure-based out-of-distribution (OOD) materials property prediction: A benchmark study, npj Comput. Mater., 10(2024), art. No. 144.

[101] C.W. Park and C. Wolverton, Developing an improved crystal graph convolutional neural network framework for accelerated materials discovery, Phys. Rev. Mater., 4(2020), No. 6, art. No. 063801.

[102] M. Karamad, R. Magar, Y.T. Shi, S. Siahrostami, I.D. Gates, and A.B. Farimani, Orbital graph convolutional neural network for material property prediction, Phys. Rev. Mater., 4(2020), No. 9, art. No. 093801.

[103] Chen C, Ye WK, Zuo YX, Zheng C, Ong SP. Graph networks as a universal machine learning framework for molecules and crystals. Chem. Mater., 2019, 31(9): 3564.

[104] Louis SY, Zhao Y, Nasiri A, et al. . Graph convolutional neural networks with global attention for improved materials property prediction. Phys. Chem. Chem. Phys., 2020, 22(32): 18141

[105] S.S. Omee, S.Y. Louis, N.H. Fu, et al., Scalable deeper graph neural networks for high-performance materials property prediction, Patterns, 3(2022), No. 5, art. No. 100491.

[106] Wen MJ, Blau SM, nSmith EWCS, Dwaraknath S, Persson KA. BonDNet: A graph neural network for the prediction of bond dissociation energies for charged molecules. Chem. Sci., 2021, 12(5): 1858.

[107] K. Choudhary and B. DeCost, Atomistic line graph neural network for improved materials property predictions, npj Comput. Mater., 7(2021), art. No. 185.

[108] H.D. Gao, X.W. Guo, G.L. Li, C. Li, and C.Q. Yang, GCPNet: An interpretable generic crystal pattern graph neural network for predicting material properties, Neural Networks, 188(2025), art. No. 107466.

RIGHTS & PERMISSIONS

University of Science and Technology Beijing