We present a framework for generating semantic embeddings of chemical elements to advance alloy inference and discovery. This framework leverages ElementBERT, a domain-specific BERT-based natural language processing model trained on 1.29 million abstracts of alloy-related scientific publications, to capture latent knowledge specific to alloys. These semantic embeddings serve as robust elemental descriptors, consistently outperforming traditional empirical descriptors across multiple downstream tasks, including predicting mechanical and transformation properties, classifying phase structures, and optimizing materials properties via Bayesian optimization. Applications to titanium alloys, high-entropy alloys, and shape memory alloys demonstrate up to 23% improvement in prediction accuracy. Notably, machine learning models based on semantic embeddings exhibit superior performance on newly published alloy data that are not included in the training corpus, indicating promising generalization capabilities for emerging alloy compositions. Our results show that ElementBERT surpasses general-purpose BERT variants by encoding specialized alloy knowledge. By bridging contextual insights from scientific literature with quantitative inference, our framework accelerates the discovery and optimization of advanced alloys, with potential for extension to other material classes.
| [1] |
S. Takamoto, C. Shinagawa, D. Motoki, et al., “Towards Universal Neural Network Potential for Material Discovery Applicable to Arbitrary Combination of 45 Elements,” Nature Communications 13, no. 1 (2022): 2991, https://doi.org/10.1038/s41467-022-30687-9.
|
| [2] |
C. Wen, Y. Zhang, C. X. Wang, et al., “Machine Learning Assisted Design of High Entropy Alloys With Desired Property,” Acta Materialia 170 (2019): 109–117, https://doi.org/10.1016/j.actamat.2019.03.010.
|
| [3] |
A. Merchant, S. Batzner, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk, “Scaling Deep Learning for Materials Discovery,” Nature 624, no. 7990 (2023): 80–85, https://doi.org/10.1038/s41586-023-06735-9.
|
| [4] |
Z. Rao, P.-Y. Tung, R. Xie, et al., “Machine Learning–Enabled High-Entropy Alloy Discovery,” Science 378, no. 6615 (2022): 78–85, https://doi.org/10.1126/science.abo4940.
|
| [5] |
P. Raccuglia, K. C. Elbert, P. D. Adler, et al., “Machine-Learning-Assisted Materials Discovery Using Failed Experiments,” Nature 533, no. 7601 (2016): 73–76, https://doi.org/10.1038/nature17439.
|
| [6] |
E. O. Pyzer-Knapp, J. W. Pitera, P. W. J. Staar, et al., “Accelerating Materials Discovery Using Artificial Intelligence, High Performance Computing and Robotics,” npj Computational Materials 8, no. 1 (2022): 84, https://doi.org/10.1038/s41524-022-00765-z.
|
| [7] |
D. Xue, P. V. Balachandran, J. Hogden, J. Theiler, D. Xue, and T. Lookman, “Accelerated Search for Materials With Targeted Properties by Adaptive Design,” Nature Communications 7, no. 1 (2016): 11241, https://doi.org/10.1038/ncomms11241.
|
| [8] |
P. Dang, J. Hu, Y. Xian, et al., “Elastocaloric Thermal Battery: Ultrahigh Heat-Storage Capacity Based on Generative Learning-Designed Phase-Change Alloys,” Advanced Materials 37, no. 8 (2025): e2412198, https://doi.org/10.1002/adma.202412198.
|
| [9] |
Y. H. Xian, P. F. Dang, Y. Tian, et al., “Compositional Design of Multicomponent Alloys Using Reinforcement Learning,” Acta Materialia 274 (2024): 120017, https://doi.org/10.1016/j.actamat.2024.120017.
|
| [10] |
M. W. Hu, Q. Y. Tan, R. Knibbe, et al., “Recent Applications of Machine Learning in Alloy Design: A Review,” Materials Science & Engineering R-Reports 155 155 (2023): 100746, https://doi.org/10.1016/j.mser.2023.100746.
|
| [11] |
B. Xiao, Y. Tang, and Y. Liu, “Integrating Materials Representations Into Feature Engineering in Machine Learning for Crystalline Materials: From Local to Global Chemistry-Structure Information Coupling,” Wiley Interdisciplinary Reviews: Computational Molecular Science 15, no. 4 (2025): e70044, https://doi.org/10.1002/wcms.70044.
|
| [12] |
J. Gao, Z. Gao, X. Fan, S. Manzhos, T. Zhang, and Y. Liu, “Transfer Learning of High-Dimensional Features via Attention-Based Embedding for Ni-Based Superalloys,” Journal of Applied Physics 138, no. 2 (2025): 024902, https://doi.org/10.1063/5.0271603.
|
| [13] |
J. Gao, Y. Tang, L. Fang, B. Xiao, Y. Liu, and Y. Liu, “Machine Learning of Energy and Structure of Multicomponent Nb-Si Alloys via Feature Dimensionality Reduction and Augmentation,” Special Casting and Nonferrous Alloys 45, no. 8 (2025): 1164–1172, https://doi.org/10.15980/j.tzzz.Z20250010.
|
| [14] |
W. Hou and Z. Ji, “Assessing GPT-4 for Cell Type Annotation in Single-Cell RNA-seq Analysis,” Nature Methods 21, no. 8 (2024): 1462–1465, https://doi.org/10.1038/s41592-024-02235-4.
|
| [15] |
D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes, “Autonomous Chemical Research With Large Language Models,” Nature 624, no. 7992 (2023): 570–578, https://doi.org/10.1038/s41586-023-06792-0.
|
| [16] |
Y. Chen and J. Zou, “Simple and Effective Embedding Model for Single-Cell Biology Built From ChatGPT,” Nature Biomedical Engineering 9, no. 4 (2025): 483–493, https://doi.org/10.1038/s41551-024-01284-6.
|
| [17] |
X. Cai, S. Liu, L. Yang, et al., “COVIDSum: A Linguistically Enriched SciBERT-Based Summarization Model for COVID-19 Scientific Papers,” Journal of Biomedical Informatics 127 (2022): 103999, https://doi.org/10.1016/j.jbi.2022.103999.
|
| [18] |
C. Kuenneth and R. Ramprasad, “polyBERT: A Chemical Language Model to Enable Fully Machine-Driven Ultrafast Polymer Informatics,” Nature Communications 14, no. 1 (2023): 4099, https://doi.org/10.1038/s41467-023-39868-6.
|
| [19] |
M. C. Ramos, C. J. Collison, and A. D. White, “A Review of Large Language Models and Autonomous Agents in Chemistry,” Chemical Science 16, no. 6 (2025): 2514–2572, https://doi.org/10.1039/d4sc03921a.
|
| [20] |
S. Yu, N. Ran, and J. Liu, “Large-Language Models: The Game-Changers for Materials Science Research,” Artificial Intelligence Chemistry 2, no. 2 (2024): 100076, https://doi.org/10.1016/j.aichem.2024.100076.
|
| [21] |
S. Y. Liu, T. Q. Wen, A. S. L. S. Pattamatta, and D. J. Srolovitz, “A Prompt-Engineered Large Language Model, Deep Learning Workflow for Materials Classification,” Materials Today 80 (2024): 240–249, https://doi.org/10.1016/j.mattod.2024.08.028.
|
| [22] |
E. Tang, B. Yang, and X. Son, “Understanding LLM Embeddings for Regression,” arXiv (2025), https://arxiv.org/abs/2411.14708.
|
| [23] |
V. Tshitoyan, J. Dagdelen, L. Weston, et al., “Unsupervised Word Embeddings Capture Latent Knowledge From Materials Science Literature,” Nature 571, no. 7763 (2019): 95–98, https://doi.org/10.1038/s41586-019-1335-8.
|
| [24] |
Z. Pei, J. Yin, P. K. Liaw, and D. Raabe, “Toward the Design of Ultrahigh-Entropy Alloys via Mining Six Million Texts,” Nature Communications 14, no. 1 (2023): 54, https://doi.org/10.1038/s41467-022-35766-5.
|
| [25] |
B. Hu, S. Liu, B. Ye, Y. Hao, and T. Wen, “A Multi-Agent Framework for Materials Laws Discovery,” arXiv (2024), https://arxiv.org/html/2411.16416v1.
|
| [26] |
T. Nguyen, Q. Zhang, B. Yang, et al., Predicting From Strings: Language Model Embeddings for Bayesian Optimization (OpenReview.net, 2024), https://arxiv.org/html/2410.10190v2.
|
| [27] |
S. Huang and J. M. Cole, “BatteryBERT: A Pretrained Language Model for Battery Database Enhancement,” Journal of Chemical Information and Modeling 62, no. 24 (2022): 6365–6377, https://doi.org/10.1021/acs.jcim.2c00035.
|
| [28] |
P. Shetty, A. C. Rajan, C. Kuenneth, et al., “A General-Purpose Material Property Data Extraction Pipeline From Large Polymer Corpora Using Natural Language Processing,” npj Computational Materials 9, no. 1 (2023): 52, https://doi.org/10.1038/s41524-023-01003-w.
|
| [29] |
J. Zhao, S. Huang, and J. M. Cole, “OpticalBERT and OpticalTable-SQA: Text- and Table-Based Language Models for the Optical-Materials Domain,” Journal of Chemical Information and Modeling 63, no. 7 (2023): 1961–1981, https://doi.org/10.1021/acs.jcim.2c01259.
|
| [30] |
Z. Zheng, O. Zhang, C. Borgs, J. T. Chayes, and O. M. Yaghi, “ChatGPT Chemistry Assistant for Text Mining and the Prediction of MOF Synthesis,” Journal of the American Chemical Society 145, no. 32 (2023): 18048–18062, https://doi.org/10.1021/jacs.3c05819.
|
| [31] |
M. P. Polak and D. Morgan, “Extracting Accurate Materials Data From Research Papers With Conversational Language Models and Prompt Engineering,” Nature Communications 15, no. 1 (2024): 1569, https://doi.org/10.1038/s41467-024-45914-8.
|
| [32] |
A. Chaudhari, C. Guntuboina, H. S. Huang, and A. B. Farimani, “AlloyBERT: Alloy Property Prediction With Large Language Models,” Computational Materials Science 244 (2024): 113256, https://doi.org/10.1016/j.commatsci.2024.113256.
|
| [33] |
D. Chen, K. Gao, D. D. Nguyen, et al., “Algebraic Graph-Assisted Bidirectional Transformers for Molecular Property Prediction,” Nature Communications 12, no. 1 (2021): 3521, https://doi.org/10.1038/s41467-021-23720-w.
|
| [34] |
A. P. O. Costa, M. R. R. Seabra, J. M. A. C. de Sá, and A. D. Santos, “Manufacturing Process Encoding Through Natural Language Processing for Prediction of Material Properties,” Computational Materials Science 237 (2024): 112896, https://doi.org/10.1016/j.commatsci.2024.112896.
|
| [35] |
K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit, “Leveraging Large Language Models for Predictive Chemistry,” Nature Machine Intelligence 6, no. 2 (2024): 122–123, https://doi.org/10.1038/s42256-023-00788-1.
|
| [36] |
P. Wang, Y. Jiang, W. J. Liao, et al., “Generalizable Descriptors for Automatic Titanium Alloys Design by Learning From Texts via Large Language Model,” Acta Materialia 296 (2025): 121275, https://doi.org/10.1016/j.actamat.2025.121275.
|
| [37] |
P. F. Liu, J. Tao, and Z. X. Ren, “A Quantitative Analysis of Knowledge-Learning Preferences in Large Language Models in Molecular Science,” Nature Machine Intelligence 7, no. 2 (2025): 315–327, https://doi.org/10.1038/s42256-024-00977-6.
|
| [38] |
N. J. Szymanski, B. Rendy, Y. Fei, et al., “An Autonomous Laboratory for the Accelerated Synthesis of Novel Materials,” Nature 624, no. 7990 (2023): 86–91, https://doi.org/10.1038/s41586-023-06734-w.
|
| [39] |
S. Tian, X. Jiang, W. Wang, et al., “Steel Design Based on a Large Language Model,” Acta Materialia 285 (2025): 120663, https://doi.org/10.1016/j.actamat.2024.120663.
|
| [40] |
K. N. Sasidhar, N. H. Siboni, J. R. Mianroodi, M. Rohwerder, J. Neugebauer, and D. Raabe, “Enhancing Corrosion-Resistant Alloy Design Through Natural Language Processing and Deep Learning,” Science Advances 9, no. 32 (2023): eadg7992, https://doi.org/10.1126/sciadv.adg7992.
|
| [41] |
P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-Enhanced Bert With Disentangled Attention,” preprint, arXiv:2006 (2020): 03654, https://arxiv.org/abs/2006.03654.
|
| [42] |
J. Devlin, “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” preprint, arXiv:1810.04805 (2018), https://arxiv.org/abs/1810.04805.
|
| [43] |
P. He, J. Gao, and W. Chen, “Debertav3: Improving Deberta Using Electra-Style Pre-Training With Gradient-Disentangled Embedding Sharing,” preprint, arXiv:2111.09543 (2021), https://arxiv.org/abs/2111.09543.
|
| [44] |
Y. H. Xian, Y. F. Wang, P. F. Dang, et al., “Leveraging Feature Gradient for Efficient Acquisition Function Maximization in Material Composition Design,” Digital Discovery 4, no. 8 (2025): 2136–2149, https://doi.org/10.1039/d5dd00080g.
|
| [45] |
J. Y. Chen, Z. L. Xiang, C. Qian, et al., “High-Yield-Strength TRIP Titanium Alloy: A Low-Cost and Short-Process Manufacturing Approach,” Materials Letters 392 (2025): 138535, https://doi.org/10.1016/j.matlet.2025.138535.
|
| [46] |
J. H. Dai, B. Tang, C. Y. Wang, et al., “Simultaneously Achieving Exceptional and Heat Treatment Insensitive Strength-Ductility Synergy in an α+β Titanium Alloy via Tailoring Silicide and Heterogeneous α Precipitates,” Journal of Materials Science & Technology 236 (2025): 51–66, https://doi.org/10.1016/j.jmst.2025.01.072.
|
| [47] |
N. Keskar, S. A. Mantri, Y. Danard, et al., “Coupling Strengthening With Local Stress Relaxation in an 800 MPa Yield Strength Strain Transformable Titanium Alloy,” Scripta Materialia 262 (2025): 116669, https://doi.org/10.1016/j.scriptamat.2025.116669.
|
| [48] |
D. F. Li, J. H. Cheng, C. G. Bai, et al., “Multi-Scale α-Phase Heterostructure Induced Transformation-Induced Plasticity (TRIP) Effect in Metastable α Plus β Titanium Alloy Showing Excellent Strength and Work-Hardening Combination,” Journal of Materials Processing Technology 341 (2025): 118880, https://doi.org/10.1016/j.jmatprotec.2025.118880.
|
| [49] |
T. L. Li, N. Xu, X. Wu, J. B. Liu, X. C. Liu, and X. F. Li, “Abnormal High Yield Strength and Strain Softening in a Metastable β Titanium Alloy at Room Temperature,” International Journal of Plasticity 188 (2025): 104310, https://doi.org/10.1016/j.ijplas.2025.104310.
|
| [50] |
Z. D. Liu, H. Y. Zhang, S. Zhang, et al., “A Machine Learning Method Approach for Designing Novel High Strength and Plasticity Metastable β Titanium Alloys,” Progress in Natural Science-Materials International 35, no. 1 (2025): 156–165, https://doi.org/10.1016/j.pnsc.2024.11.010.
|
| [51] |
J. Tang, Y. Wang, S. Wang, M. Cai, and Z. Xu, “Composition Design and Microstructure Optimization of a Metastable β-Type Titanium Alloy With Ultrahigh Strength,” Advanced Engineering Materials 27, no. 7 (2025): 2402099, https://doi.org/10.1002/adem.202402099.
|
| [52] |
H. Yang, M. Zhu, N. Chen, et al., “Novel Hierarchical α Structure Enhanced Strength-Ductility Synergy in Metastable β Titanium Alloy,” Materials Science and Engineering: A 925 (2025): 147877, https://doi.org/10.1016/j.msea.2025.147877.
|
| [53] |
J. S. Zhang, E. D. Gao, Z. B. Wang, and S. Q. Pan, “Ti-VMoCrFeZrAl β-Type Titanium Alloy With High Strength Obtained by Composition Design and Cold Rolling With Heat Treatment,” Journal of Materials Engineering and Performance 34, no. 17 (2025): 19122–19130, https://doi.org/10.1007/s11665-025-11221-6.
|
| [54] |
G. Hinton, “Distilling the Knowledge in a Neural Network,” preprint, arXiv:1503.02531 (2015), https://arxiv.org/abs/1503.02531.
|
| [55] |
A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention Is all You Need,” in 31st Annual Conference on Neural Information Processing Systems (NIPS) (Curran Associates, Inc., 2017), 6000–6010, https://arxiv.org/abs/1706.03762.
|
| [56] |
S. L. France and J. D. Carroll, “Two-Way Multidimensional Scaling: A Review,” IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews 41, no. 5 (2011): 644–661, https://doi.org/10.1109/tsmcc.2010.2078502.
|
| [57] |
S. Shen, J. Liu, L. Lin, et al., “SsciBERT: A Pre-Trained language Model for Social Science Texts,” Scientometrics 128, no. 2 (2023): 1241–1263, https://doi.org/10.1007/s11192-022-04602-4.
|
| [58] |
T. Gupta, M. Zaki, and N. M. A. Krishnan, “Mausam, MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction,” npj Computational Materials 8, no. 1 (2022): 102, https://doi.org/10.1038/s41524-022-00784-w.
|
| [59] |
T. Wolf, L. Debut, V. Sanh, et al., “Transformers: State-of-the-Art Natural Language Processing,” in Association for Computational Linguistics (2020), 38–45: Online, https://arxiv.org/abs/1910.03771.
|
| [60] |
J. Wang, H. Kwon, H. S. Kim, and B. J. Lee, “A Neural Network Model for High Entropy Alloy Design,” npj Computational Materials 9, no. 1 (2023): 60, https://doi.org/10.1038/s41524-023-01010-x.
|
| [61] |
F. Pedregosa, G. Varoquaux, A. Gramfort, et al., “Scikit-Learn: Machine Learning in Python,” Journal of Machine Learning Research 12, no. 85 (2011): 2825–2830, https://jmlr.org/papers/v12/pedregosa11a.html.
|
| [62] |
F. Nogueira, Bayesian Optimization: Open Source Constrained Global Optimization Tool for Python (2014), https://github.com/fmfn/BayesianOptimization.
|
| [63] |
Y. Zhang, C. Wen, P. F. Dang, T. Lookman, D. Z. Xue, and Y. J. Su, “Toward Ultra-High Strength High Entropy Alloys via Feature Engineering,” Journal of Materials Science & Technology 200 (2024): 243–252, https://doi.org/10.1016/j.jmst.2024.02.058.
|
Rights & permissions
2026 The Author(s). Materials Genome Engineering Advances published by Wiley-VCH GmbH on behalf of University of Science and Technology Beijing.
PDF
(3240KB)
