LLM-driven materials knowledge extraction: multimodal parsing, ontology, and agentic systems
Shuai Yang , Yimeng Wang , Qiong Tu , Nian Ran , Ying Lu
Journal of Materials Informatics ›› 2026, Vol. 6 ›› Issue (3) : 39
Extracting reliable knowledge from unstructured materials literature remains a central bottleneck for data-driven and AI-enabled materials discovery. Large language models (LLMs) are reshaping this task by integrating multimodal document parsing, ontology-guided semantic grounding, structured extraction, and agentic verification into increasingly unified workflows. This review analyzes these developments through a Perception–Cognition–Action lens. At the perception layer, we examine how scientific document parsers, multimodal LLMs, table and chart readers, and optical chemical-structure-recognition systems convert visually rich papers into computable evidence. At the cognition layer, we discuss how ontologies and knowledge graphs constrain LLM outputs, support entity alignment, and reduce semantic ambiguity. At the action layer, we compare schema-based extraction, schema-free discovery, and agentic extraction as a control–coverage–autonomy spectrum rather than a simple succession of tools. We further argue that reliability is the decisive criterion for large-scale deployment, and synthesize failure modes, layered defenses, and evaluation protocols that connect source grounding, ontology constraints, physical verification, and human-in-the-loop review. By distinguishing demonstrated extraction capabilities from more speculative AI-scientist and self-driving-laboratory visions, this review provides a comparative and risk-aware account of how LLM-driven systems can produce evidence-linked, physically meaningful, and reusable materials knowledge.
Large language models / materials informatics / knowledge extraction / multimodal document parsing / ontology engineering / knowledge graphs / AI agents / retrieval-augmented generation
| [1] |
|
| [2] |
|
| [3] |
|
| [4] |
|
| [5] |
|
| [6] |
|
| [7] |
|
| [8] |
|
| [9] |
|
| [10] |
|
| [11] |
|
| [12] |
|
| [13] |
|
| [14] |
Hira, K.; Zaki, M.; Mausam; Krishnan, N. M. A. MatSKRAFT: a framework for large-scale materials knowledge extraction from scientific tables. arXiv 2025, arXiv:2509.10448. Available online: https://doi.org/10.48550/arXiv.2509.10448. (accessed on 29 Jul 2026) |
| [15] |
|
| [16] |
|
| [17] |
|
| [18] |
|
| [19] |
Chiang, Y.; Hsieh, E.; Chou, C. H.; Riebesell, J. LLaMP: large language model made powerful for high-fidelity materials knowledge retrieval. arXiv 2024, arXiv:2401.17244. Available online: https://doi.org/10.48550/arXiv.2401.17244. (accessed on 29 Jul 2026) |
| [20] |
|
| [21] |
Chepurova, A.; Bulatov, A.; Burtsev, M.; Kuratov, Y. Wikontic: constructing wikidata-aligned, ontology-aware knowledge graphs with large language models. arXiv 2025, arXiv:2512.00590. Available online: https://doi.org/10.48550/arXiv.2512.00590. (accessed on 29 Jul 2026) |
| [22] |
Zhang, Q.; Wang, B.; Huang, V. S. J.; et al. Document parsing unveiled: techniques, challenges, and prospects for structured information extraction. arXiv 2024, arXiv:2410.21169. Available online: https://doi.org/10.48550/arXiv.2410.21169. (accessed on 29 Jul 2026) |
| [23] |
Van, M. H.; Verma, P.; Zhao, C.; Wu, X. A survey of AI for materials science: foundation models, LLM agents, datasets, and tools. arXiv 2025, arXiv:2506.20743. Available online: https://doi.org/10.48550/arXiv.2506.20743. (accessed on 29 Jul 2026) |
| [24] |
|
| [25] |
Li, M.; Xu, Y.; Cui, L.; et al. DocBank: a benchmark dataset for document layout analysis. arXiv 2020, arXiv:2006.01038. Available online: https://doi.org/10.48550/arXiv.2006.01038. (accessed on 29 Jul 2026) |
| [26] |
Chi, Z.; Huang, H.; Xu, H. D.; Yu, H.; Yin, W.; Mao, X. L. Complicated table structure recognition. arXiv 2019, arXiv:1908.04729. Available online: https://doi.org/10.48550/arXiv.1908.04729. (accessed on 29 Jul 2026) |
| [27] |
Zhong, X.; ShafieiBavani, E.; Jimeno Yepes, A. Image-based table recognition: data, model, and evaluation. arXiv 2019, arXiv:1911.10683. Available online: https://doi.org/10.48550/arXiv.1911.10683. (accessed on 29 Jul 2026) |
| [28] |
|
| [29] |
Huang, Y.; Lv, T.; Cui, L.; Lu, Y.; Wei, F. LayoutLMv3: pre-training for document AI with unified text and image masking. arXiv 2022, arXiv:2204.08387. Available online: https://doi.org/10.48550/arXiv.2204.08387. (accessed on 29 Jul 2026) |
| [30] |
Kim, G.; Hong, T.; Yim, M.; et al. OCR-free document understanding transformer. arXiv 2021, arXiv:2111.15664. Available online: https://doi.org/10.48550/arXiv.2111.15664. (accessed on 29 Jul 2026) |
| [31] |
Lee, K.; Joshi, M.; Turc, I.; et al. Pix2Struct: screenshot parsing as pretraining for visual language understanding. arXiv 2022, arXiv:2210.03347. Available online: https://doi.org/10.48550/arXiv.2210.03347. (accessed on 29 Jul 2026) |
| [32] |
Liao, W.; Wang, J.; Li, H.; Wang, C.; Huang, J.; Jin, L. DocLayLLM: an efficient multi-modal extension of large language models for text-rich document understanding. arXiv 2024, arXiv:2408.15045. Available online: https://doi.org/10.48550/arXiv.2408.15045. (accessed on 29 Jul 2026) |
| [33] |
Shpigel Nacson, M.; Aberdam, A.; Ganz, R.; et al. DocVLM: make your VLM an efficient reader. arXiv 2024, arXiv:2412.08746. Available online: https://doi.org/10.48550/arXiv.2412.08746. (accessed on 29 Jul 2026) |
| [34] |
Huang, B.; Hua, H.; Yu, Z.; Darrell, T.; Feris, R.; Herzig, R. DAVE: a VLM vision encoder for document understanding and web agents. arXiv 2025, arXiv:2512.17221. Available online: https://doi.org/10.48550/arXiv.2512.17221. (accessed on 29 Jul 2026) |
| [35] |
|
| [36] |
Liu, F.; Eisenschlos, J. M.; Piccinno, F.; et al. DePlot: one-shot visual language reasoning by plot-to-table translation. arXiv 2022, arXiv:2212.10505. Available online: https://doi.org/10.48550/arXiv.2212.10505. (accessed on 29 Jul 2026) |
| [37] |
Khalighinejad, G.; Scott, S.; Liu, O.; et al. MatViX: multimodal information extraction from visually rich articles. arXiv 2024, arXiv:2410.20494. Available online: https://doi.org/10.48550/arXiv.2410.20494. (accessed on 29 Jul 2026) |
| [38] |
Polak, M. P.; Morgan, D. Leveraging vision capabilities of multimodal LLMs for automated data extraction from plots. arXiv 2025, arXiv:2503.12326. Available online: https://doi.org/10.48550/arXiv.2503.12326. (accessed on 29 Jul 2026) |
| [39] |
|
| [40] |
Wang, J.; He, Y.; Yang, H.; et al. GTR-CoT: graph traversal as visual chain of thought for molecular structure recognition. arXiv 2025, arXiv:2506.07553. Available online: https://doi.org/10.48550/arXiv.2506.07553. (accessed on 29 Jul 2026) |
| [41] |
Zhang, W.; Wang, X.; Feng, B.; Liu, W. MolSight: optical chemical structure recognition with SMILES pretraining, multi-granularity learning and reinforcement learning. arXiv 2025, arXiv:2511.17300. Available online: https://doi.org/10.48550/arXiv.2511.17300. (accessed on 29 Jul 2026) |
| [42] |
|
| [43] |
Xia, R.; Mao, S.; Yan, X.; et al. DocGenome: an open large-scale scientific document benchmark for training and testing multi-modal large language models. arXiv 2024, arXiv:2406.11633. Available online: https://doi.org/10.48550/arXiv.2406.11633. (accessed on 29 Jul 2026) |
| [44] |
Dong, K.; Chang, Y.; Goh, X. D.; Li, D.; Tang, R.; Liu, Y. MMDocIR: benchmarking multimodal retrieval for long documents. arXiv 2025, arXiv:2501.08828. Available online: https://doi.org/10.48550/arXiv.2501.08828. (accessed on 29 Jul 2026) |
| [45] |
Dong, K.; Huang, S.; Ye, F.; et al. Doc-Researcher: a unified system for multimodal document parsing and deep research. arXiv 2025, arXiv:2510.21603. Available online: https://doi.org/10.48550/arXiv.2510.21603. (accessed on 29 Jul 2026) |
| [46] |
|
| [47] |
Cao, S.; Liu, K. Figuring out figures: using textual references to caption scientific figures. arXiv 2024, arXiv:2407.11008. Available online: https://doi.org/10.48550/arXiv.2407.11008. (accessed on 29 Jul 2026) |
| [48] |
|
| [49] |
|
| [50] |
|
| [51] |
|
| [52] |
Garcia Trelles, E.; Schweizer, C.; Thomas, A.; von Hartrott, P.; Janka-Ramm, M. Digitalizing material knowledge: a practical framework for ontology-driven knowledge graphs in process chains.Appl Sci2024;14:11683 |
| [53] |
|
| [54] |
|
| [55] |
|
| [56] |
|
| [57] |
|
| [58] |
|
| [59] |
Sadruddin, S.; D’Souza, J.; Poupaki, E.; et al. LLMs4SchemaDiscovery: a human-in-the-loop workflow for scientific schema mining with large language models. arXiv 2025, arXiv:2504.00752. Available online: https://doi.org/10.48550/arXiv.2504.00752. (accessed on 29 Jul 2026) |
| [60] |
Sharma, K.; Kumar, P.; Li, Y. OG-RAG: ontology-grounded retrieval-augmented generation for large language models. arXiv 2024, arXiv:2412.15235. Available online: https://doi.org/10.48550/arXiv.2412.15235. (accessed on 29 Jul 2026) |
| [61] |
|
| [62] |
Ye, Y.; Ren, J.; Wang, S.; et al. Construction and application of materials knowledge graph in multidisciplinary materials science via large language model. arXiv 2024, arXiv:2404.03080. Available online: https://doi.org/10.48550/arXiv.2404.03080. (accessed on 29 Jul 2026) |
| [63] |
|
| [64] |
|
| [65] |
|
| [66] |
|
| [67] |
Lewis, P.; Perez, E.; Piktus, A.; et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv 2020, arXiv:2005.11401. Available online: https://doi.org/10.48550/arXiv.2005.11401. (accessed on 29 Jul 2026) |
| [68] |
|
| [69] |
|
| [70] |
|
| [71] |
|
| [72] |
|
| [73] |
|
| [74] |
|
| [75] |
|
| [76] |
|
| [77] |
|
| [78] |
|
| [79] |
|
| [80] |
|
| [81] |
|
| [82] |
|
| [83] |
|
| [84] |
|
| [85] |
|
| [86] |
|
| [87] |
|
| [88] |
|
| [89] |
|
| [90] |
|
| [91] |
|
| [92] |
Mo, B.; Yu, K.; Kazdan, J.; et al. KGGen: extracting knowledge graphs from plain text with language models. arXiv 2025, arXiv:2502.09956. Available online: https://doi.org/10.48550/arXiv.2502.09956. (accessed on 29 Jul 2026) |
| [93] |
Radevski, G.; Gashteovski, K.; Hung, C. C.; Lawrence, C.; Glavaš, G. Linking surface facts to large-scale knowledge graphs. arXiv 2023, arXiv:2310.14909. Available online: https://doi.org/10.48550/arXiv.2310.14909. (accessed on 29 Jul 2026) |
| [94] |
Gridach, M.; Nanavati, J.; El Abidine, K. Z.; Mendes, L.; Mack, C. Agentic AI for scientific discovery: a survey of progress, challenges, and future directions. arXiv 2025, arXiv:2503.08979. Available online: https://doi.org/10.48550/arXiv.2503.08979. (accessed on 29 Jul 2026) |
| [95] |
|
| [96] |
Wei, J.; Yang, Y.; Zhang, X.; et al. From AI for science to agentic science: a survey on autonomous scientific discovery. arXiv 2025, arXiv:2508.14111. Available online: https://doi.org/10.48550/arXiv.2508.14111. (accessed on 29 Jul 2026) |
| [97] |
Zhang, H.; Li, Y.; Huang, W.; et al. Towards agentic intelligence for materials science. arXiv 2026, arXiv:2602.00169. Available online: https://doi.org/10.48550/arXiv.2602.00169. (accessed on 29 Jul 2026) |
| [98] |
|
| [99] |
Li, Z.; Zeng, Y.; Zuo, Y.; et al. KnowCoder: coding structured knowledge into LLMs for universal information extraction. arXiv 2024, arXiv:2403.07969. Available online: https://doi.org/10.48550/arXiv.2403.07969. (accessed on 29 Jul 2026) |
| [100] |
|
| [101] |
Hu, B.; Liu, S.; Ye, B.; et al. A multi-agent framework for physical laws discovery. arXiv 2024, arXiv:2411.16416. Available online: https://doi.org/10.48550/arXiv.2411.16416. (accessed on 29 Jul 2026) |
| [102] |
|
| [103] |
|
| [104] |
Zhang, H.; Song, Y.; Hou, Z.; Miret, S.; Liu, B. HoneyComb: a flexible LLM-based agent system for materials science. arXiv 2024, arXiv:2409.00135. Available online: https://doi.org/10.48550/arXiv.2409.00135. (accessed on 29 Jul 2026) |
| [105] |
Jia, S.; Zhang, C.; Fung, V. LLMatDesign: autonomous materials discovery with large language models. arXiv 2024, arXiv:2406.13163. Available online: https://doi.org/10.48550/arXiv.2406.13163. (accessed on 29 Jul 2026) |
| [106] |
|
| [107] |
|
| [108] |
Ni, Z.; Li, Y.; Hu, K.; et al. MatPilot: an LLM-enabled AI materials scientist under the framework of human-machine collaboration. arXiv 2024, arXiv:2411.08063. Available online: https://doi.org/10.48550/arXiv.2411.08063. (accessed on 29 Jul 2026) |
| [109] |
|
| [110] |
|
| [111] |
|
| [112] |
|
| [113] |
|
| [114] |
Norouzi, E.; Waitelonis, J.; Sack, H. The landscape of ontologies in materials science and engineering: a survey and evaluation. arXiv 2024, arXiv:2408.06034. Available online: https://doi.org/10.48550/arXiv.2408.06034. (accessed on 29 Jul 2026) |
| [115] |
|
| [116] |
|
| [117] |
|
| [118] |
|
| [119] |
Wu, F.; Zheng, X.; Qu, Y.; Wang, Z.; Feng, Z.; Li H. Grounding generative planners in verifiable logic: a hybrid architecture for trustworthy embodied AI. arXiv 2026, arXiv:2602.08373. Available online: https://doi.org/10.48550/arXiv.2602.08373. (accessed on 29 Jul 2026) |
| [120] |
|
| [121] |
|
| [122] |
|
| [123] |
|
| [124] |
|
| [125] |
|
| [126] |
|
| [127] |
|
| [128] |
|
| [129] |
|
| [130] |
|
| [131] |
Cornelio, C.; Ito, T.; Cory-Wright, R.; Dash, S.; Horesh, L. The need for verification in AI-driven scientific discovery. arXiv 2025, arXiv:2509.01398. Available online: https://doi.org/10.48550/arXiv.2509.01398. (accessed on 29 Jul 2026) |
| [132] |
|
| [133] |
Wang, X.; Raj, A.; Luebbe, M.; Wen, H.; Xu, S.; Lu, K. Reliable end-to-end material information extraction from the literature with source-tracked multi-stage large language models. arXiv 2025, arXiv:2510.05142. Available online: https://doi.org/10.48550/arXiv.2510.05142. (accessed on 29 Jul 2026) |
| [134] |
|
| [135] |
Lederbauer, M.; Betala, S.; Li, X.; et al. LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature. arXiv 2025, arXiv:2510.26824. Available online: https://doi.org/10.48550/arXiv.2510.26824. (accessed on 29 Jul 2026) |
| [136] |
|
| [137] |
|
| [138] |
Song, Y.; Miret, S.; Liu, B. MatSci-NLP: evaluating scientific language models on materials science language tasks using text-to-schema modeling. arXiv 2023, arXiv:2305.08264. Available online: https://doi.org/10.48550/arXiv.2305.08264. (accessed on 29 Jul 2026) |
| [139] |
Niyongabo Rubungo, A.; Li, K.; Hattrick-Simpers, J.; Bousso Dieng, A. LLM4Mat-bench: benchmarking large language models for materials property prediction.Mach Learn Sci Technol2025;6:020501 |
| [140] |
Zhong, Z.; Zhou, K.; Mottin, D. Benchmarking large language models for molecule prediction tasks. arXiv 2024, arXiv:2403.05075. Available online: https://doi.org/10.48550/arXiv.2403.05075. (accessed on 29 Jul 2026) |
| [141] |
|
| [142] |
|
| [143] |
|
| [144] |
|
| [145] |
|
| [146] |
|
/
| 〈 |
|
〉 |