Multi-Scale Memory-Enhanced Transformer for Optical Chemical Structure Recognition

Abstract

Optical Chemical Structure Recognition (OCSR) is a key technology at the intersection of chemistry and computer science, aiming to convert molecular structure images into machine-readable sequence representations. However, existing OCSR models still fall short in building long-range dependencies within sequences, making it difficult to accurately generate long sequences with complex structures. To address this, we propose a Multi-Scale Memory-Enhanced Transformer (MSMET) model and design two modules: the Layer-wise Hierarchical Fusion (LHF) module and the Cross-Attention Memory Augmentation (CAMA) module. The LHF module employs adaptive feature recalibration across encoder blocks through learnable gating, effectively alleviating feature forgetting phenomena in deep architectures while maintaining multi-scale visual discriminability. The CAMA module equips the decoder with a learnable memory matrix that dynamically enriches cross-modal attention, significantly enhancing the model’s capacity to capture long-distance topological interactions in molecular graph structures. We evaluated the performance of MSMET, and the experimental results demonstrate that the model achieved strong overall state-of-the-art performance.

Keywords

Optical chemical structure recognition / Transformer / Attention / Deep learning

References

[1] Weininger D. J. Chem. Inf. Comput. Sci., 1988, 28: 31

[2] Heller S R, McNaught A, Pletnev I, Stein S, Tchekhovskoi D. J. Cheminform., 2015, 7: 23

[3] Krenn M, Häse F, Nigam A, Friederich P, Aspuru-Guzik A. Mach. Learn.: Sci. Technol., 2020, 1: 045024

[4] Wang R, Ji Y, Li Y, Lee S-T. J. Phys. Chem. Lett., 2025, 16: 421

[5] Gao T, Ji Y, Liu C, Li Y. Sci. China Mater., 2024, 67: 1243

[6] Wang H, Sun J, Li Y, Deng W. Sci. Data, 2025, 12: 648

[7] Chen S, Jin L, Ji Y, Li Y. J. Mater. Chem. A, 2025, 13: 9841

[8] Merchant A, Batzner S, Schoenholz S S, Aykol M, Cheon G, Cubuk E D. Nature, 2023, 624: 80

[9] Santos-Júnior C D, Torres M D T, Duan Y, Rodríguez del Río Á, Schmidt T S B, Chong H, Fullam A, Kuhn M, Zhu C, Houseman A, Somborski J, Vines A, Zhao X-M, Bork P, Huerta-Cepas J, de la Fuente-Nunez C, Coelho L P. Cell, 2024, 187: 3761

[10] McDaniel J R, Balmuth J R. J. Chem. Inf. Comput. Sci., 1992, 32: 373

[11] Filippov I V, Nicklaus M C. J. Chem. Inf. Model., 2009, 49: 740

[12] Peryea T, Katzel D, Zhao T, Southall N, Nguyen D-T. Abstracts of Papers, 258th ACS National Meeting & Exposition, 2019

[13] Valko A T, Johnson A P. J. Chem. Inf. Model., 2009, 49: 780

[14] Smolov V, Zentsev F, Rybalkin M. Proceedings of the Twentieth Text REtrieval Conference (TREC 2011), 2011

[15] Park J, Rosania G R, Shedden K A, Nguyen M, Lyu N, Saitou K. Chem. Cent. J., 2009, 3: 4

[16] Oldenhof M, Arany A, Moreau Y, Simm J. J. Chem. Inf. Model., 2020, 60: 4506

[17] Zhang X-C, Yi J-C, Yang G-P, Wu C-K, Hou T-J, Cao D-S. Brief. Bioinform., 2022, 23: bbac033

[18] LeCun Y, Bottou L, Bengio Y, Haffner P. Proc. IEEE, 1998, 86: 2278

[19] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021Virtual Event

[20] Elman J L. Cogn. Sci., 1990, 14: 179

[21] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser T, Polosukhin I. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017

[22] Yang Z, Li J. J. East China Univ. Sci. Technol., 2023, 49: 135

[23] Khokhlov I, Krasnov L, Fedorov M V, Sosnin S. Chem. Methods, 2022, 2: e202100069

[24] Rajan K, Zielesny A, Steinbeck C. J. Cheminform., 2021, 13: 61

[25] Rajan K, Brinkhaus H O, Agea M I, Zielesny A, Steinbeck C. Nat. Commun., 2023, 14: 5045

[26] Xu Z, Li J, Yang Z, Li S, Li H. J. Cheminform., 2022, 14: 41

[27] Lin F, Li J. Complex Intell. Syst., 2024, 10: 1

[28] Zhang H., Wu C., Zhang Z., Zhu Y., Lin H., Zhang Z., Sun Y., He T., Mueller J., Manmatha R., Li M., Smola A., arXiv, 2020, arXiv:2004.08955.

[29] He K, Zhang X, Ren S, Sun J. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

[30] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

[31] Tan M, Le Q V. Proceedings of the 36th International Conference on Machine Learning (ICML), 2019

[32] Tan M, Le Q V. Proceedings of the 38th International Conference on Machine Learning (ICML), 2021Virtual Event

[33] Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021Virtual Conference

[34] Lee Y, Kim J, Willette J, Hwang S J. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

[35] Kim S, Chen J, Cheng T, Gindulyte A, He J, He S, Li Q, Shoemaker B A, Thiessen P A, Yu B, Zaslavsky L, Zhang J, Bolton E E. Nucleic Acids Res., 2021, 49: D1388

[36] Willighagen E L, Mayfield J W, Alvarsson J, Berg A, Carlsson L, Jeliazkova N, Kuhn S, Pluskal T, Rojas-Chertó M, Spjuth O, Torrance G, Evelo C T, Guha R, Steinbeck C. J. Cheminform., 2017, 9: 33

[37] Landrum G. RDKit: A Software Suite for Cheminformatics, Computational Chemistry, and Predictive Modeling, 2013

[38] Zhang H, Zeng P, Gao L, Lyu X, Song J, Shen H T. IEEE Trans. Circuits Syst. Video Technol., 2024, 34: 4829

[39] Yi X, Fu Y, Liu R, Zhang H, Hua R. IEEE J. Biomed. Health Inform., 2024, 28: 2152

[40] Chen Z, Song Y, Chang T-H, Wan X. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020Online

[41] Wang R, Ji Y, Li Y, Lee S-T. J. Phys. Chem. Lett., 2025, 16: 12787

[42] OpenAI, GPT-4o System Card, OpenAI, 2024-08-08, https://cdn.openai.com/gpt-4o-system-card.pdf, accessed on 2026-07-18.

[43] OpenAI, Introducing OpenAI o3 and o4-mini, OpenAI, 2025-04-16, https://openai.com/index/introducing-o3-and-o4-mini/, accessed on 2026-05-31.

[44] OpenAI, GPT-5 System Card, OpenAI, 2025-08-07, https://openai.com/index/gpt-5-system-card/, accessed on 2026-05-31.

[45] DeepSeek-AI, DeepSeek-V3.1 Release, DeepSeek API Docs, 2025-08-21, https://api-docs.deepseek.com/news/news250821, accessed on 2026-05-31.

[46] Guo D, Yang D, Zhang H, Song J, Wang P, Zhu Q, Xu R, Zhang R, Ma S, Bi X, Zhang X, Yu X, Wu Y, Wu Z F, Gou Z, Shao Z, Li Z, Gao Z, Liu A, Xue B, Wang B, Wu B, Feng B, Lu C, Zhao C, Deng C, Ruan C, Dai D, Chen D, Ji D, Li E, Lin F, Dai F, Luo F, Hao G, Chen G, Li G, Zhang H, Xu H, Ding H, Gao H, Qu H, Li H, Guo J, Li J, Chen J, Yuan J, Tu J, Qiu J, Li J, Cai J L, Ni J, Liang J, Chen J, Dong K, Hu K, You K, Gao K, Guan K, Huang K, Yu K, Wang L, Zhang L, Zhao L, Wang L, Zhang L, Xu L, Xia L, Zhang M, Zhang M, Tang M, Zhou M, Li M, Wang M, Li M, Tian N, Huang P, Zhang P, Wang Q, Chen Q, Du Q, Ge R, Zhang R, Pan R, Wang R, Chen R J, Jin R L, Chen R, Lu S, Zhou S, Chen S, Ye S, Wang S, Yu S, Zhou S, Pan S, Li S S, Zhou S, Wu S, Yun T, Pei T, Sun T, Wang T, Zeng W, Liu W, Liang W, Gao W, Yu W, Zhang W, Xiao W L, An W, Liu X, Wang X, Chen X, Nie X, Cheng X, Liu X, Xie X, Liu X, Yang X, Li X, Su X, Lin X, Li X Q, Jin X, Shen X, Chen X, Sun X, Wang X, Song X, Zhou X, Wang X, Shan X, Li Y K, Wang Y Q, Wei Y X, Zhang Y, Xu Y, Li Y, Zhao Y, Sun Y, Wang Y, Yu Y, Zhang Y, Shi Y, Xiong Y, He Y, Piao Y, Wang Y, Tan Y, Ma Y, Liu Y, Guo Y, Ou Y, Wang Y, Gong Y, Zou Y, He Y, Xiong Y, Luo Y, You Y, Liu Y, Zhou Y, Zhu Y X, Huang Y, Li Y, Zheng Y, Zhu Y, Ma Y, Tang Y, Zha Y, Yan Y, Ren Z Z, Ren Z, Sha Z, Fu Z, Xu Z, Xie Z, Zhang Z, Hao Z, Ma Z, Yan Z, Wu Z, Gu Z, Zhu Z, Liu Z, Li Z, Xie Z, Song Z, Pan Z, Huang Z, Xu Z, Zhang Z, Zhang Z. Nature, 2025, 645: 633

Rights & permissions

Jilin University, The Editorial Department of Chemical Research in Chinese Universities and Springer-Verlag GmbH

Just Accepted

This article has successfully passed peer review and final editorial review, and will soon enter typesetting, proofreading and other publishing processes. The currently displayed version is the accepted final manuscript. The officially published version will be updated with format, DOI and citation information upon launch. We recommend that you pay attention to subsequent journal notifications and preferentially cite the officially published version. Thank you for your support and cooperation.