Computer vision for efficient object detection and segmentation in molecular image analysis

Abstract

Image recognition, classification, and analysis of large sets of high-resolution molecular images are time-consuming and labor-intensive, even for human experts, due to the lack of standardized approaches. In recent years, machine learning has emerged as a powerful tool for automating image data analysis in materials science. In this work, we developed a computer vision program for efficient object detection and instance segmentation, offering a fast alternative to manual molecular image analysis. By integrating You Only Look Once version 9 (YOLOv9) with an incremental learning strategy and hyperparameter optimization, the system enables accurate detection, classification, and segmentation of molecular species across diverse scanning tunneling microscopy datasets. Our results demonstrate robust performance and minimal forgetting rates across multiple molecular categories, enabling scalable and updatable surface image analysis workflows. We anticipate that computer vision methods will see increasing applications in image data analysis within the field of on-surface chemistry.

Keywords

YOLOv9 / object detection / instance segmentation / incremental learning / surface chemistry

References

[1] Lo S,Schrier J.Review of low-cost self-driving laboratories in chemistry and materials science: the “frugal twin” concept.Digit Discover2024;3:842-68

[2] Wang Z,Zhang P.High throughput characterization method of electrical and phonon properties by dielectric resonant spectroscopy.Mater Genome Eng Adv2025;3:e70010

[3] Xiang J,Zhang X,Sun Q.Local large language model-assisted literature mining for on-surface reactions.Mater Genome Eng Adv2025;3:e88

[4] Li Y,Wang X,Qiu D.Effect of material properties on the thermal responses of the carbonization and pyrolysis layers of polymer matrix composites for charring-ablators.J Mater Inf2025;5:31

[5] Wang WY,Li G.Artificial intelligence enabled smart design and manufacturing of advanced materials: the endless Frontier in AI⁺ era.Mater Genome Eng Adv2024;2:e56

[6] Jung TA,Gimzewski JK.Conformational identification of individual adsorbed molecules with the STM.Nature1997;386:696-8

[7] Meng T,Lei P.Self-assembly of triphenylamine macrocycles and co-assembly with guest molecules at the liquid-solid interface studied by STM: influence of different side chains on host-guest interaction.Langmuir2022;38:3568-74

[8] Wyrick J,Namboodiri P.Enhanced atomic precision fabrication by adsorption of phosphine into engineered dangling bonds on H-Si using STM and DFT.ACS Nano2022;16:19114-23

[9] Wang L,Ho W.Atomic-scale quantum sensing based on the ultrafast coherence of an H₂ molecule in an STM cavity.Science2022;376:401-5

[10] Moreno D,Urgel JI.Engineering periodic dinuclear lanthanide-directed networks featuring tunable energy level alignment and magnetic anisotropy by metal exchange.Small2022;18:e2107073

[11] Lyu CK,Gao ZA.Synthesis of single-layer two-dimensional metal–organic frameworks M₃(HAT)₂ (M=Ni, Fe, Co, HAT=1,4,5,8,9,12-hexaazatriphenylene) using an on-surface reaction.Angew Chem Int Ed Engl2022;61:e202204528

[12] Liu J,Xu Z.On-surface preparation of coordinated lanthanide-transition-metal clusters.Nat Commun2021;12:1619 PMCID:PMC7954866

[13] Di Giovannantonio, M.; Fasel, R. On-surface synthesis and atomic scale characterization of unprotected indenofluorene polymers.J Polym Sci2022;60:1814-26

[14] Wang J,Xu C.Influence of molecular configurations on the desulfonylation reactions on metal surfaces.J Am Chem Soc2022;144:21596-605

[15] Kinikar A,Urgel JI.On-surface polyarylene synthesis by cycloaromatization of isopropyl substituents.Nat Synth2022;1:289-96

[16] Liu L,Witteler MC.Polymerization of silanes through dehydrogenative Si-Si bond formation on metal surfaces.Nature Chem2021;13:350-7

[17] Mallada B,Mendieta-Moreno JI.On-surface strain-driven synthesis of nonalternant non-benzenoid aromatic compounds containing four- to eight-membered rings.J Am Chem Soc2021;143:14694-702

[18] Zhu X,Pu W.On-surface synthesis of C144 hexagonal coronoid with zigzag edges.ACS Nano2022;16:10600-7

[19] Hellerstedt J,Švec M,Hennen T.Counting molecules: python based scheme for automated enumeration and categorization of molecules in scanning tunneling microscopy images.Softw Impacts2022;12:100301

[20] Krull A,Rother C,Krull C.Artificial-intelligence-driven scanning probe microscopy.Commun Phys2020;3:54

[21] Milošević D,Galić I.Automated estimation of chronological age from panoramic dental X-ray images using deep learning.Expert Syst Appl2022;189:116038

[22] Li J,Yin J.Machine vision automated chiral molecule detection and classification in molecular imaging.J Am Chem Soc2021;143:10177-88

[23] Gordon OM,Farley SM,Moriarty PJ.Automated searching and identification of self-organized nanostructures.Nano Lett2020;20:7688-93

[24] Kang J,Park JH.DeepGT: deep learning-based quantification of nanosized bioparticles in bright-field micrographs of gires-tournois biosensor.Nano Today2023;52:101968

[25] Faraz K,Ducottet C.Deep learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies.Sci Rep2022;12:2484 PMCID:PMC8847623

[26] Newby JM,Lee PT,Lai SK.Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D.Proc Natl Acad Sci U S A2018;115:9026-31 PMCID:PMC6130393

[27] Zhu Z,Yang Q.Autonomous scanning tunneling microscopy imaging via deep learning.J Am Chem Soc2024;146:29199-206

[28] Seifert TJ,Kasten P.Chirality detection in scanning tunneling microscopy data using artificial intelligence.Small Methods2024;8:e2400549 PMCID:PMC11671859

[29] Wang CY,Liao HYM. YOLOv9: learning what you want to learn using programmable gradient information. arXiv 2024, arXiv:2402.13616. Available online: https://doi.org/10.48550/arXiv.2402.13616. (accessed 27 Mar 2026)

[30] Gao T,Zhang J,Zhang L.Development of an accurate “composition-process-properties” dataset for SLMed Al-Si-(Mg) alloys and its application in alloy design.J Mater Inf2023;3:6

[31] Zhou DW,Qi ZH,Zhan DC.Class-incremental learning: a survey.IEEE Trans Pattern Anal Mach Intell2024;46:9851-73

[32] Pan SJ.A survey on transfer learning.IEEE Trans Knowl Data Eng2010;22:1345-59

[33] Otero R,de Parga AL,Miranda R.Molecular self-assembly at solid surfaces.Adv Mater2011;23:5148-76

[34] Voigt J,Martin K,Avarvari N.An aperiodic chiral tiling by topological molecular self-assembly.Nat Commun2025;16:83 PMCID:PMC11696205