CrysToGraph: A Comprehensive Predictive Model for Crystal Material Properties and the Benchmark

Abstract

The bonding across the lattice and ordered structures endow crystals with unique symmetry and determine their macroscopic properties. Crystals with unique properties such as low-dimensional materials, metal-organic frameworks, and defected crystals, in particular, exhibit different structures from bulk crystals and possess exotic physical properties, making them intriguing subjects for investigation. To accurately predict the physical and chemical properties of crystals, it is crucial to consider long-range orders. While GNNs excel at capturing the local environment of atoms in crystals, they often face challenges in effectively capturing longe range interactions due to their limited depth. In this paper, we propose CrysToGraph (Crystals with Transformers on Graph), a transformer-based geometric graph network designed for unconventional crystalline systems, and UnconvBench, a benchmark to evaluate models' predictive performance on multiple categories of crystal materials. CrysToGraph effectively captures short-range interactions with transformer-based graph convolution blocks as well as long-range interactions with graph-wise transformer blocks. CrysToGraph proves its effectiveness in modelling all types of crystal materials in multiple tasks, and moreover, it outperforms most existing methods, achieving new state-of-the-art results on two benchmarks. This work has the potential to accelerate the development of novel crystal materials in various fields, including the anodes, cathodes, and solid-state electrolytes.

Keywords

AI for materials science / crystal materials / GNN / machine learning / transformer

References

[1] S. Lei, Z. Zeng, S. Cheng, and J. Xie, “Fast-Charging of Lithium-Ion Batteries: A Review of Electrolyte Design Aspects,” Battery Energy 2 (2023): 20230018.

[2] G. Song, Y. Shi, S. Jiang, and H. Pang, “Recent Progress in MOF-Derived Porous Materials as Electrodes for High-Performance Lithium-Ion Batteries,” Advanced Functional Materials 33, no. 42 (2023): 2303121.

[3] M. Wang, Z. Song, J. Bi, et al., “Probing Interfacial Electrochemistry by In Situ Atomic Force Microscope for Battery Characterization,” Battery Energy 2 (2023): 20230006.

[4] Q. Liang, M. Duan, X. Liu, et al., “Recent Advances in Effect of Crystallization Dynamics Process on the Morphology of Active Layer in Organic Solar Cells,” Battery Energy 3 (2024): 20230073.

[5] M. Aykol, P. Herring, and A. Anapolsky, “Machine Learning for Continuous Innovation in Battery Technologies,” Nature Reviews Materials 5.10 (2020): 725-727.

[6] L. Zhang, J. Han, H. Wang, R. Car, W. E, “Deep Potential Molecular Dynamics: A Scalable Model With the Accuracy of Quantum Mechanics,” Physical Review Letters 120.14 (2018): 143001.

[7] F. Wang and J. Cheng, “Understanding the Solvation Structures of Glyme-Based Electrolytes by Machine Learning Molecular Dynamics,” Chinese Journal of Structural Chemistry 42 (2023): 100061.

[8] S. Dajnowicz, G. Agarwal, J. M. Stevenson, et al., “High-Dimensional Neural Network Potential for Liquid Electrolyte Simulations,” Journal of Physical Chemistry B 126 (2022): 6271-6280.

[9] L. D. Jacobson, J. M. Stevenson, F. Ramezanghorbani, et al., “Transferable Neural Network Potential Energy Surfaces for Closed-Shell Organic Molecules: Extension to Ions,” Journal of Chemical Theory and Computation 18.4 (2022): 2354-2366.

[10] Philipp Thölke and Gianni De Fabritiis, “Torchmd-Net: Equivariant Transformers for Neural Network Based Molecular Potentials,” arXiv preprint arXiv (2022): 2202.02541.

[11] D. Zhang, H. Bi, F.-Z. Dai, W. Jiang, L. Zhang, H. Wang, “DPA-1: Pretraining of Attention-Based Deep Potential Model for Molecular Simulation,” arXiv preprint arXiv (2022): 2208.08236.

[12] S. Gong, Y. Zhang, Z. Mu, et al., “Bamboo: A Predictive and Transferable Machine Learning Force Field Framework for Liquid Electrolyte Development,” arXiv preprint arXiv (2024): 2404.07181.

[13] B. Zeng, S. Chen, X. Liu, et al., “Uni-ELF: A Multi-Level Representation Learning Framework for Electrolyte Formulation Design,” arXiv preprint arXiv (2024): 2407.06152.

[14] D. Nathani, J. Chauhan, C. Sharma, and M. Kaul, “Learning Attention-Based Embeddings for Relation Prediction in Knowledge Graphs,” arXiv preprint arXiv (2019): 1906.01195.

[15] T. Hamaguchi, H. Oiwa, M. Shimbo, and Y. Matsumoto, “Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach,” arXiv preprint arXiv (2017): 1706.05674.

[16] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling, “Modeling Relational Data With Graph Convolutional Networks,” in The Semantic Web: 15th International Conference, ESWC 2018 (Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15: Springer, 2018), 593-607.

[17] Z. Wang, Q. Lv, X. Lan, and Y. Zhang, “Cross-Lingual Knowledge Graph Alignment via Graph Convolutional Networks,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, eds. E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsuji (Association for Computational Linguistics, 2018), 349-357.

[18] M. Zhang and Y. Chen, “Link Prediction Based on Graph Neural Networks,” Advances in Neural Information Processing Systems 31 (2018): 53f0d7c5.

[19] J. Qiu, J. Tang, H. Ma, Y. Dong, K. Wang, and J. Tang, “Deepinf: Social Influence Prediction With Deep Learning,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association for Computing Machinery, 2018), 2110-2119.

[20] Y. Liu, X. Shi, L. Pierce, and X. Ren, “Characterizing and Forecasting User Engagement With In-App Action Graph: A Case Study of Snapchat,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association of Computing Machinery, 2019): 2023-2031.

[21] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph Convolutional Neural Networks for Web-Scale Recommender Systems,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association of Computing Machinery, 2018): 974-983.

[22] F. Monti, M. Bronstein, and X. Bresson, “Geometric Matrix Completion With Recurrent Multi-Graph Neural Networks,” Advances in Neural Information Processing Systems 30 (2017).

[23] R. v. d. Berg, T. N. Kipf, and M. Welling, “Graph Convolutional Matrix Completion,” arXiv preprint arXiv (2017): 1706.02263.

[24] A. Santoro, D. Raposo, D. G. Barrett, et al., “A Simple Neural Network Module for Relational Reasoning,” Advances in Neural Information Processing Systems 30 (2017).

[25] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al., “Interaction Networks for Learning About Objects, Relations and Physics,” Advances in Neural Information Processing Systems 29 (2016).

[26] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, et al., “Convolutional Networks on Graphs for Learning Molecular Fingerprints,” Advances in Neural Information Processing Systems 28 (2015).

[27] A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur, “Protein Interface Prediction Using Graph Convolutional Networks,” Advances in Neural Information Processing Systems 30 (2017).

[28] S. Kearnes, K. McCloskey, M. Berndl, V. Pande, and P. Riley, “Molecular Graph Convolutions: Moving Beyond Fingerprints,” Journal of Computer-Aided Molecular Design 30 (2016): 595-608.

[29] Y. Xu, C.-H. Chou, N. Han, J. Pei, and L. Lai, “Graph Kernel Learning for Predictive Toxicity Models,” Machine Learning and Deep Learning in Computational Toxicology (Springer, 2023), 159-182.

[30] C. Cai, H. Lin, H. Wang, et al., “Midruglikeness: Subdivisional Drug-Likeness Prediction Models Using Active Ensemble Learning Strategies,” Biomolecules 13, no. 1 (2022): 29.

[31] Z. Zhu, M. Galkin, Z. Zhang, and J. Tang, “Neural-Symbolic Models for Logical Queries on Knowledge Graphs,” in International Conference on Machine Learning, vol. 162 (PMLR, 2022), 27454-27478.

[32] S. Gong, T. Xie, Y. Shao-Horn, R. Gomez-Bombarelli, and J. C. Grossman, “Examining Graph Neural Networks for Crystal Structures: Limitations and Opportunities for Capturing Periodicity,” arXiv preprint arXiv (2022): 2208.05039.

[33] Z. Zhang, M. Xu, A. Jamasb, et al., “Protein Representation Learning by Geometric Structure Pretraining,” arXiv preprint arXiv (2022): 2203.06125.

[34] T. Xie and J. C. Grossman, “Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties,” Physical Review Letters 120, no. 14 (2018): 145301.

[35] C. W. Park and C. Wolverton, “Developing an Improved Crystal Graph Convolutional Neural Network Framework for Accelerated Materials Discovery,” Physical Review Materials 4, no. 6 (2020): 063801.

[36] D.-T. Lee, “On K-Nearest Neighbor Voronoi Diagrams in the Plane,” IEEE Transactions on Computers 100, no. 6 (1982): 478-487.

[37] J. Cheng, C. Zhang, and L. Dong, “A Geometric-Information-Enhanced Crystal Graph Network for Predicting Properties of Materials,” Communications Materials 2, no. 1 (2021): 92.

[38] C. Chen, W. Ye, Y. Zuo, C. Zheng, and S. P. Ong, “Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals,” Chemistry of Materials 31, no. 9 (2019): 3564-3572.

[39] K. Choudhary and B. DeCost, “Atomistic Line Graph Neural Network for Improved Materials Property Predictions,” npj Computational Materials 7, no. 1 (2021): 185.

[40] R. Ruff, P. Reiser, J. Stühmer, and P. Friederich, “Connectivity Optimized Nested Graph Networks for Crystal Structures,” arXiv preprint arXiv (2023): 2302.14102.

[41] K. Yan, Y. Liu, Y. Lin, and S. Ji, “Periodic Graph Transformers for Crystal Material Property Prediction,” Advances in Neural Information Processing Systems 35 (2022): 15066-15080.

[42] K. Yan, C. Fu, X. Qian, X. Qian, and S. Ji, “Complete and Efficient Graph Transformers for Crystal Material Property Prediction,” arXiv preprint arXiv (2024): 2403.11857.

[43] J. Bai, Y. Du, Y. Wang, S. Kong, J. Gregoire, and C. Gomes, “Xtal2dos: Attention-Based Crystal to Sequence Learning for Density of States Prediction,” arXiv preprint arXiv (2023): 2302.01486.

[44] S. Kong, F. Ricci, D. Guevarra, J. B. Neaton, C. P. Gomes, and J. M. Gregoire, “Density of States Prediction for Materials Discovery via Contrastive Learning From Probabilistic Embeddings,” Nature Communications 13, no. 1 (2022): 949.

[45] J. Bai, Z. Lai, R. Yang, Y. Xue, J. Gregoire, and C. Gomes, “Imitation Refinement for x-Ray Diffraction Signal Processing,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2019), 3337-3341.

[46] P.-P. De Breuck, G. Hautier, and G.-M. Rignanese, “Materials Property Prediction for Limited Datasets Enabled by Feature Selection and Joint Learning With Modnet,” npj Computational Materials 7, no. 1 (2021): 83.

[47] A. Dunn, Q. Wang, A. Ganose, D. Dopp, and A. Jain, “Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm,” npj Computational Materials 6, no. 1 (2020): 138.

[48] A. Jain, S. P. Ong, G. Hautier, et al., “Commentary: The Materials Project: A Materials Genome Approach to Accelerating Materials Innovation,” APL Materials 1, no. 1 (2013): 011002.

[49] K. Choudhary, K. F. Garrity, A. C. E. Reid, et al., “The Joint Automated Repository for Various Integrated Simulations (JARVIS) for Data-Driven Materials Design,” npj Computational Materials 6, no. 1 (2020): 173.

[50] A. F. Wells, “Structural Inorganic Chemistry,” Oxford Classic Texts in the Physical Sciences, 1 (2012): 121.

[51] X. Gao, W. Gao, W. Xiao, Z. Wang, C. Wang, and L. Xiang, “Learning Regularized Positional Encoding for Molecular Prediction,” arXiv preprint arXiv (2022): 2211.12773.

[52] X. Gao, W. Gao, W. Xiao, Z. Wang, C. Wang, and L. Xiang, “Supervised Pre-Training for Molecular Force Fields and Properties Prediction,” arXiv preprint arXiv (2022): 2211.14429.

[53] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson, “Benchmarking Graph Neural Networks,” arXiv preprint arXiv (2020): 2003.00982.

[54] V. P. Dwivedi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson, “Graph Neural Networks With Learnable Structural and Positional Representations,” arXiv preprint arXiv (2021): 2110.07875.

[55] M. N. Gjerding, A. Taghizadeh, A. Rasmussen, et al., “Recent Progress of the Computational 2D Materials Database (c2db),” 2D Materials 8, no. 4 (2021): 044002.

[56] S. Haastrup, M. Strange, M. Pandey, et al., “The Computational 2d Materials Database: High-Throughput Modeling and Discovery of Atomically Thin Crystals,” 2D Materials 5, no. 4 (2018): 042002.

[57] J. Zhou, L. Shen, M. D. Costa, et al., “2DMatPedia, an Open Computational Database of Two-Dimensional Materials From Top-Down and Bottom-Up Approaches,” Scientific Data 6, no. 1 (2019): 86.

[58] N. S. Bobbitt, J. Chen, and R. Q. Snurr, “High-Throughput Screening of Metal-Organic Frameworks for Hydrogen Storage at Cryogenic Temperature,” Journal of Physical Chemistry C 120, no. 48 (2016): 27328-27341.

[59] A. S. Rosen, S. M. Iyer, D. Ray, et al., “Machine Learning the Quantum-Chemical Properties of Metal-Organic Frameworks for Accelerated Materials Discovery,” Matter 4, no. 5 (2021): 1578-1597.

[60] V. Stanev, C. Oses, A. G. Kusne, et al., “Machine Learning Modeling of Superconducting Critical Temperature,” npj Computational Materials 4, no. 1 (2018): 29.

[61] K. Choudhary and K. Garrity, “Designing High-T_c Superconductors With BCS-Inspired Screening, Density Functional Theory, and Deep-Learning,” npj Computational Materials 8, no. 1 (2022): 244.

[62] K. Choudhary and B. G. Sumpter, “A Deep-Learning Model for Fast Prediction of Vacancy Formation in Diverse Materials,” arXiv preprint arXiv (2022): 2205.08366.

[63] P.-P. De Breuck, M. L. Evans, and G.-M. Rignanese, “Robust Model Benchmarking and Bias-Imbalance in Data-Driven Materials Science: A Case Study on Modnet,” Journal of Physics: Condensed Matter 33, no. 40 (2021): 404002.

RIGHTS & PERMISSIONS

2025 The Author(s). Battery Energy published by Xijing University and John Wiley & Sons Australia, Ltd.