Bioinformatics and biomedical informatics with ChatGPT: Year one review

Abstract

The year 2023 marked a significant surge in the exploration of applying large language model chatbots, notably Chat Generative Pre‐trained Transformer (ChatGPT), across various disciplines. We surveyed the application of ChatGPT in bioinformatics and biomedical informatics throughout the year, covering omics, genetics, biomedical text mining, drug discovery, biomedical image understanding, bioinformatics programming, and bioinformatics education. Our survey delineates the current strengths and limitations of this chatbot in bioinformatics and offers insights into potential avenues for future developments.

Keywords

ChatGPT / bioinformatics / biomedical informatics

References

[1] Wang H, Fu T, Du Y, Gao W, Huang K, Liu Z, et al. Scientific discovery in the age of artificial intelligence. Nature. 2023; 620 (7972): 47- 60.

[2] Xu Y, Liu X, Cao X, Huang C, Liu E, Qian S, et al. Artificial intelligence: a powerful paradigm for scientific research. Innovation. 2021; 2 (4): 100179.

[3] Van Noorden R, Perkel JM. AI and science: what 1,600 researchers think. Nature. 2023; 621 (7980): 672- 5.

[4] Milano S, McGrane JA, Leonelli S. Large language models challenge the future of higher education. Nat Mach Intell. 2023; 5 (4): 333- 4.

[5] van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature. 2023; 614 (7947): 224- 6.

[6] Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023; 388 (13): 1233- 9.

[7] Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings Bioinf. 2023; 25 (1): bbad493.

[8] Liu J, Yang M, Yu Y, Xu H, Li K, and Zhou X. Large language models in bioinformatics: applications and perspectives. 2024. Preprint at arXiv:2401.04155.

[9] Xu D, Chen W, Peng W, Zhang C, Xu T, Zhao X, et al. Large language models for generative information extraction: a survey. 2023. Preprint at arXiv:2312.17617.

[10] Shue E Liu L, Li B, Feng Z, Li X, Hu G. Empowering beginners in bioinformatics with ChatGPT. Quantitative Biology. 2023; 11 (2): 105- 8.

[11] Piccolo SR, Denny P, Luxton-Reilly A, Payne SH, Ridge PG. Evaluating a large language model’s ability to solve programming exercises from an introductory bioinformatics course. PLoS Comput Biol. 2023; 19 (9): e1011511.

[12] Zhou J, Zhang B, Chen X, Li H, Xu X, Chen S, et al. An AI agent for fully automated multi‐omic analyses. 2024. Preprint at bioRxiv:2023.09.08.556814.

[13] Hu G, Liu L, Xu D. On the responsible use of chatbots in bioinformatics. Genom Proteom Bioinform. 2024; 22 (1): qzae002.

[14] Murdoch B. Privacy and artificial intelligence: challenges for protecting health information in a new era. BMC Med Ethics. 2021; 22 (1): 122.

[15] Karim MR, Islam T, Shajalal M, Beyan O, Lange C, Cochez M, et al. Explainable AI for bioinformatics: methods, tools and applications. Briefings Bioinf. 2023; 24 (5): bbad236.

[16] Hou W, Ji Z. Assessing GPT-4 for cell type annotation in single-cell rna-seq analysis. Nat Methods. 2024; 21 (8): 1462- 5.

[17] Hou W, Ji Z. Geneturing tests GPT models in genomics. 2023. Preprint at bioRxiv:2023.03.11.532238.

[18] Jin Q, Yang Y, Chen Q, Lu Z. GeneGPT: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics. 2024; 40 (2).

[19] Ahimaz P, Bergner AL, Florido ME, Harkavy N, Bhattacharyya S. Genetic counselors’ utilization of ChatGPT in professional practice: a cross‐sectional study. Am J Med Genet. 2024; 194 (4).

[20] Duong D, Solomon BD. Analysis of large-language model versus human performance for genetics questions. Eur J Hum Genet. 2024; 32 (4): 466- 8.

[21] Alkuraya IF. Is artificial intelligence getting too much credit in medical genetics? Am J Med Genet C. 2023; 193 (3): e32062.

[22] Emmert-Streib F. Can ChatGPT understand genetics? Eur J Hum Genet. 2024; 32: 371- 2.

[23] Chen Q, Sun H, Liu H, Jiang Y, Ran T, Jin X, et al. An extensive benchmark study on biomedical text generation and mining with ChatGPT. Bioinformatics. 2023; 39 (9): btad557.

[24] Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, et al. Domain‐specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthcare. 2021; 3 (Article 2): 1- 23.

[25] Chen Q, Du J, Hu Y, Kuttichi Keloth V, Peng X, Raja K, et al. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. 2023. Preprint at arXiv:2305.16326.

[26] Ateia S, Kruschwitz U. Is ChatGPT a biomedical expert? ‐‐exploring the zero‐shot performance of current GPT models in biomedical tasks. 2023. Preprint at arXiv:2306.16108.

[27] Chen S, Li Y, Lu S, Van H, Aerts H, Savova GK, et al. Evaluating the ChatGPT family of models for biomedical reasoning and classification. J Am Med Inf Assoc. 2024; 31 (4): 940- 8.

[28] Jahan I, Laskar MTR, Peng C, Huang JX. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Comput Biol Med. 2024; 171: 108189.

[29] Hou Y, Yeung J, Xu H, Su C, Wang F, Zhang R. From answers to insights: unveiling the strengths and limitations of ChatGPT and biomedical knowledge graphs. Preprint at Res Sq. 2023: rs.3.rs- 3185632.

[30] Rizvi RF, Vasilakes J, Adam TJ, Melton GB, Bishop JR, Bian J, et al. iDISK: the integrated dietary supplements knowledge base. J Am Med Inf Assoc. 2020; 27 (4): 539- 48.

[31] Zhao Q, Zhou X, Wu J, Cai J, Bao X, Tang L, et al. Biotreasury: a community-based repository enabling indexing and rating of bioinformatics tools. Sci China Life Sci. 2024; 67 (2): 221- 9.

[32] Wu X, Zeng Y, Das A, Jo S, Zhang T, Patel P, et al. ReguloGPT: harnessing GPT for knowledge graph construction of molecular regulatory pathways. 2024. Preprint at bioRxiv:2024.01.27.577521.

[33] Azam M, Chen Y, Arowolo M, Liu H, Popescu M, Xu D. A comprehensive evaluation of large language models in mining gene interactions and pathway knowledge. Quantitative Biology. 2024. In press.

[34] Rehana H, Bengisu Çam N, Basmaci M, Zheng J, Jemiyo C, He, Y, et al. Evaluation of GPT and BERT‐based models on identifying protein‐protein interactions in biomedical text. 2023. Preprint at arXiv:2303.17728.

[35] Tiwari K, Matthews L, May B, Shamovsky V, Orlic‐Milacic M, Rothfels K, et al. ChatGPT usage in the Reactome curation process. 2023. Preprint at bioRxiv:2023.11.08.566195.

[36] Zhou B, Ji B, Shen C, Zhang X, Yu X, Huang P, et al. Evlncrnas 3.0: an updated comprehensive database for manually curated functional long non‐coding RNAs validated by low‐throughput experiments. Nucleic Acids Res. 2024; 52 (D1): D98- 106.

[37] Chen X, Li C, Wang Z, Zhou Y, Chu M. Computational screening of biomarkers and potential drugs for arthrofibrosis based on combination of sequencing and large nature language model. Journal of Orthopaedic Translation. 2024; 44: 102- 13.

[38] Fo K, Chuah YS, Foo H, Davey EE, Fullwood M, Thibault G, et al. Plantconnectome: knowledge networks encompassing >100,000 plant article abstracts. 2023. Preprint at bioRxiv:2023.07.11.548541.

[39] Rawte V, Sheth A, Das A. A survey of hallucination in large foundation models. 2023. Preprint at arXiv:2309.05922.

[40] Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, et al. Siren’s song in the AI ocean: a survey on hallucination in large language models. 2023. Preprint at arXiv:2309.01219.

[41] Chen Y, Gao J, Petruc M, Hammer RD, Popescu M, Xu D. Iterative prompt refinement for mining gene relationships from ChatGPT. International Journal of Artificial Intelligence and Robotics Research. 2024. In press.

[42] Yao S, Yu D, Zhao J, Shafran I, Griffiths TL, Cao Y, et al. Tree of thoughts: deliberate problem solving with large language models. In: Neural information processing systems 36 (NeurIPS 2023) 35. Curran Associates, Inc; 2023, p. 11809- 22.

[43] Savage N. Drug discovery companies are customizing ChatGPT: here’s how. Nat Biotechnol. 2023; 41 (5): 585- 6.

[44] Chakraborty C, Bhattacharya M, Lee SS. Artificial intelligence enabled ChatGPT and large language models in drug target discovery, drug discovery, and development. Mol Ther Nucleic Acids. 2023; 33: 866- 8.

[45] Zhao A, Wu Y. Future implications of ChatGPT in pharmaceutical industry: drug discovery and development. Front Pharmacol. 2023; 14: 1194216.

[46] Hu G, Xie Z. The artificial intelligence pharma era after “chat generative pre-trained transformer”. Medical Review. 2023; 3: 198- 9.

[47] Gao Z, Li L, Ma S, Wang Q, Hemphill L, Xu R. Examining the potential of ChatGPT on biomedical information retrieval: fact-checking drug-disease associations. Ann Biomed Eng. 2023.

[48] Guo T, Guo K, Nan B, Liang Z, Guo Z, Chawla NV, et al. What can large language models do in chemistry? A comprehensive benchmark on eight tasks. In: Advances in neural information processing systems, 36. Curran Associates, Inc; 2023. p. 59662- 88.

[49] Juhi A, Pipil N, Santra S, Mondal S, Behera JK, Mondal H. The capability of ChatGPT in predicting and explaining common drug-drug interactions. Cureus. 2023; 15: e36272.

[50] Al-Ashwal FY, Zawiah M, Gharaibeh L, Abu-Farha R, Bitar AN. Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and bard against conventional drug-drug interactions clinical tools. Drug Healthc Patient Saf. 2023; 15: 137- 47.

[51] Herrero-Zazo M, Segura-Bedmar I, Martinez P, Declerck T. The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. J Biomed Inf. 2013; 46 (5): 914- 20.

[52] Wang YM, Shen HW, Chen TJ. Performance of ChatGPT on the pharmacist licensing examination in Taiwan. J Chin Med Assoc. 2023; 86 (7): 653- 8.

[53] Kunitsu Y. The potential of GPT-4 as a support tool for pharmacists: analytical study using the Japanese national examination for pharmacists. JMIR Medical Education. 2023; 9: e48452.

[54] Zong H, Li J, Wu E, Wu R, Lu J, Shen B. Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses. BMC Med Educ. 2024; 24 (1): 143.

[55] Huang X, Estau D, Liu X, Yu Y, Qin J, Li Z. Evaluating the performance of ChatGPT in clinical pharmacy: a comparative study of ChatGPT and clinical pharmacists. Br J Clin Pharmacol. 2024; 90 (1): 232- 8.

[56] Wang R, Feng H, Wei GW. ChatGPT in drug discovery: a case study on anticocaine addiction drug development with chatbots. J Chem Inf Model. 2023; 63 (22): 7189- 209.

[57] Liu S, Wang J, Yang Y, Wang C, Liu L, Guo H, et al. Conversational drug editing using retrieval and domain feedback. In: The twelfth international conference on learning representations; 2024.

[58] Liang Y, Zhang R, Zhang L, and Xie P. Drugchat: towards enabling ChatGPT‐like capabilities on drug molecule graphs. 2023. Preprint at arXiv:2309.03907.

[59] Ye G, Cai X, Lai H, Wang X, Huang J, Wang L, et al. Drugassist: a large language model for molecule optimization. 2023. Preprint at arXiv:2401.10334.

[60] Dong Q, Li L, Dai D, Zheng C, Wu Z, Chang B, et al. A survey on in‐context learning. 2022. Preprint at arXiv:2301.00234.

[61] Li J, Liu Y, Fan W, Wei X-Y, Liu H, Tang J, et al. Empowering molecule discovery for molecule-caption translation with large language models: a ChatGPT perspective. IEEE transactions on knowledge and data engineering. 2023: 1- 13. In press.

[62] Jablonka KM, Schwaller P, Ortega-Guerrero A, Smit B. Leveraging large language models for predictive chemistry. Nat Mach Intell. 2024; 6 (2): 161- 9.

[63] Cai X, Lai H, Wang X, Wang L, Liu W, Wang Y, et al. Comprehensive evaluation of molecule property prediction with ChatGPT. Methods. 2024; 222: 133- 41.

[64] Caldas Ramos M, Michtavy SS, Porosoff MD, White AD. Bayesian optimization of catalysts with in‐context learning. 2023. Preprint at arXiv:2304.05341.

[65] Zeng Z, Yin B, Wang S, Liu J, Yang C, Yao H, et al. Interactive molecular discovery with natural language. 2023. Preprint at arXiv:2306.11976.

[66] Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text‐to‐text transformer. J Mach Learn Res. 2020; 21.

[67] Hu H, Yang AJ, Deng S, Wang D, Song M, Shen S. A generative drug-drug interaction triplets extraction framework based on large language models. Proceedings of the Association for Information Science and Technology. 2023; 60 (1): 980- 2.

[68] Wei J, Bosma M, Zhao VY, Guu K, Yu AW, Lester B, et al. Fine‐tuned language models are zero‐shot learners. 2021. Preprint at arXiv:2109.01652.

[69] Fang Y, Liang X, Zhang N, Liu K, Huang R, Chen Z, et al. Mol‐instructions: a large‐scale biomolecular instruction dataset for large language models. 2023. Preprint at arXiv:2306.08018.

[70] Zhao Z, Ma D, Chen L, Sun L, Li Z, Xu H, et al. Chemdfm: dialogue foundation model for chemistry. 2024. Preprint at arXiv:2401.14818.

[71] Cao H, Liu Z, Lu X, Yao Y, Li Y. Instructmol: multi‐modal integration for building a versatile and reliable molecular assistant in drug discovery. 2023. Preprint at arXiv:2311.16208.

[72] Zhang W, Wang Q, Kong X, Xiong J, Ni S, Cao D, et al. Fine‐tuning large language models for chemical text mining. Chem Sci. 2024. In press.

[73] Acosta JN, Falcone GJ, Rajpurkar P, Topol EJ. Multimodal biomedical AI. Nat Med. 2022; 28 (9): 1773- 84.

[74] Truhn D, Weber CD, Braun BJ, Bressem K, Kather JN, Kuhl C, et al. A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports. Sci Rep-Uk. 2023; 13 (1): 20159.

[75] Liu Z, Jiang H, Zhong T, Wu Z, Ma C, Li Y, et al. Holistic evaluation of GPT‐4v for biomedical imaging. 2023. Preprint at arXiv:2312.05256.

[76] Wu C, Lei J, Zheng Q, Zhao W, Lin W, Zhang X, et al. Can GPT‐4v(ision) serve medical applications? Case studies on GPT‐4v for multimodal medical diagnosis. 2023. Preprint at arXiv:2310.09909.

[77] Yan Z, Zhang K, Zhou R, He L, Li X, Sun L. Multimodal ChatGPT for medical applications: an experimental study of GPT‐4v. 2023. Preprint at arXiv:2310.19061.

[78] Buckley T, Diao JA, Rodman A, Manrai AK. Accuracy of a vision‐language model on challenging medical cases. 2023. Preprint at arXiv:2311.05591.

[79] Yang Z, Yao Z, Tasmin M, Vashisht P, Jang WS, Ouyang F, et al. Performance of multimodal GPT‐4v on usmle with image: potential for imaging diagnostic support with explanations. 2023. Preprint at medRxiv:2023.10.26.23297629.

[80] Li Y, Liu Y, Wang Z, Liang X, Liu L, Wang L, et al. A comprehensive study of GPT‐4V’s multimodal capabilities in medical imaging. 2023. Preprint at medRxiv:2023.11.03.23298067.

[81] Hou W, Ji Z. GPT‐4V exhibits human‐like performance in biomedical image classification. 2024. Preprint at bioRxiv:2023.12.31.573796.

[82] Wang J, Ye Q, Liu L, Guo NL, Hu G. Scientific figures interpreted by ChatGPT: strengths in plot recognition and limits in color perception. NPJ Precis Oncol. 2024; 8 (1): 84.

[83] OpenAI. GPT‐4V(ision) system card. (2023) 1- 18.

[84] Nickerson RS. Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol. 1998; 2: 175- 220.

[85] Jin Q, Chen F, Zhou Y, Xu Z, Cheung JM, Chen R, et al. Hidden flaws behind expert‐level accuracy of GPT‐4 vision in medicine. 2024. Preprint at arXiv:2401.08396.

[86] Wang J, Liu Z, Zhao L, Wu Z, Ma C, Yu S, et al. Review of large vision models and visual prompt engineering. Meta-Radiology. 2023; 1 (3): 100047.

[87] Yang Z, Li L, Lin K, Wang J, Lin C, Liu Z, et al. The dawn of lmms: preliminary explorations with GPT‐4v(ision). 2023. Preprint at arXiv:2309.17421.

[88] Li Z, Wang C, Liu C, Ma P, Wu D, Wang S, et al. Vrptest: evaluating visual referring prompting in large multimodal models. 2023. Preprint at arXiv:2312.04087.

[89] Li J, Chen H, Wang Y, Chen MM, Liang H. Next-generation analytics for omics data. Cancer Cell. 2021; 39 (1): 3- 6.

[90] Xie M, Yang L, Chen G, Wang Y, Xie Z, Wang H. Ribochat: a chat‐style web interface for analysis and annotation of ribosome profiling data. Briefings Bioinf. 2022; 23 (2).

[91] Merow C, Serra-Diaz JM, Enquist BJ, Wilson AM. AI chatbots can boost scientific coding. Nat Ecol Evol. 2023; 7: 960- 2.

[92] Jansen JA, Manukyan A, Khoury NA, Akalin A. Leveraging large language models for data analysis automation. 2023. Preprint at bioRxiv:2023.12.11.571140.

[93] Dong Z, Zhong V, Lu YY. Biomania: simplifying bioinformatics data analysis through conversation. 2023. Preprint at bioRxiv:2023.10.29.564479.

[94] Liu A, Hu X, Wen L, Yu PS. A comprehensive evaluation of ChatGPT’s zero‐shot text‐to‐SQL capability. 2023. Preprint at arXiv:2303.13547.

[95] Sima A‐C, de Farias TM. On the potential of artificial intelligence chatbots for data exploration of federated bioinformatics knowledge graphs. In: SeWebMeDa’23: 6th workshop on semantic web solutions for large‐scale biomedical data analytics, 3466. CEUR‐WS.org; 2023.

[96] Rangel JC, de Farias TM, Sima AC, Kobayashi N. Sparql generation: an analysis on fine‐tuning openllama for question answering over a life science knowledge graph. In: WAT4HCLS 2024: 15th international semantic web applications and tools for health care and life Sciences conference; 2024. In press.

[97] Chen C, Stadler T. Genspectrum chat: data exploration in public health using large language models. 2023. Preprint at arXiv: 2305.13821.

[98] Wang L, Ge XJ, Liu L, Hu GQ. Code interpreter for bioinformatics: are we there yet? Ann Biomed Eng. 2024; 52 (4): 754- 6.

[99] Tang X, Qian B, Gao R, Chen J, Chen X, Gerstein M. Biocoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics. 2023. In press.

[100] Sarwal V, Munteanu V, Suhodolschi T, Ciorba D, Eskin E, Wang W, et al. Biollmbench: a comprehensive benchmarking of large language models in bioinformatics. 2023. Preprint at bioRxiv:2023.12.19.572483.

[101] Lubiana T, Lopes R, Medeiros P, Silva JC, Goncalves ANA, Maracaja-Coutinho V, et al. Ten quick tips for harnessing the power of ChatGPT in computational biology. PLoS Comput Biol. 2023; 19 (8): e1011319.

[102] Chen M, Tworek J, Jun H, Yuan Q, Ponde de Oliveira Pinto H, Kaplan J, et al. Evaluating large language models trained on code. 2021. Preprint at arXiv:2107.03374.

[103] Lehtinen T, Haaranen L, Leinonen J. Automated questionnaires about students’ Javascript programs: towards gauging novice programming processes. In: Ace’ 23: proceedings of the 25th australasian computing education conference. Association for Computing Machiner; 2023. p. 49- 58.

[104] Lin ZC. How to write effective prompts for large language models. Nat Human Behav. 2024; 8 (4): 611- 5.

[105] Denny P, Leinonen J, Prather J, Luxton‐Reilly A, Amarouche T, Becker BA, et al. Promptly: using prompt problems to teach learners how to effectively utilize Ai code generators. 2023. Preprint at arXiv:2307.16364.

[106] Soman K, Rose PW, Morris JH, Akbas RE, Smith B, Peetoom B, et al. Biomedical knowledge graph‐enhanced prompt generation for large language models. 2023. Preprint at arXiv:2311.17330.

[107] Chen L, Zaharia M, Zou J. How is ChatGPT’s behavior changing over time?. 2023. Preprint at arXiv:2307.09009.

[108] Wang G, Yang G, Du Z, Fan L, Li X. ClinicalGPT: large language models fine‐tuned with diverse medical data and comprehensive evaluation. 2023. Preprint at arXiv:2306.09968.

[109] Peng C, Yang X, Chen A, Smith KE, PourNejatian N, Costa AB, et al. A study of generative large language model for medical research and healthcare. NPJ Digit Med. 2023; 6 (1): 210.

[110] Lai T, Shi Y, Du Z, Wu, J, Fu, K, Dou Y, et al. Psy‐llm: scaling up global mental health psychological services with AI‐based large language models. 2023. Preprint at arXiv:2307.11991.

[111] Liu JM, Li D, Cao H, Ren T, Liao Z, Wu J. Chatcounselor: a large language models for mental health support. 2023. Preprint at arXiv:2309.15461.

[112] Han T, Adams LC, Papaioannou J‐M, Grundmann P, Oberhauser T, Löser A, et al. Medalpaca ‐‐ an open‐source collection of medical conversational AI models and training data. 2023. Preprint at arXiv:2304.08247.

[113] Zhang S, Fan R, Liu Y, Chen S, Liu Q, Zeng W. Applications of transformer-based language models in bioinformatics: a survey. Bioinform Adv. 2023; 3 (1): vbad001.

[114] Qiu JN, Li L, Sun JK, Peng JC, Shi PL, Zhang RY, et al. Large AI models in health informatics: applications, challenges, and the future. IEEE J Biomed Health. 2023; 27 (12): 6074- 87.

RIGHTS & PERMISSIONS

2024 2024 The Author(s). Quantitative Biology published by John Wiley & Sons Australia, Ltd on behalf of Higher Education Press.