Automatic classification of medical literature leveraging pre-trained large language models

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Background: Identifying clinical research articles can facilitate evidence synthesis and research evaluation; however, journal categories and keyword-based approaches may not accurately capture studies at the article level.
Objective: To develop and evaluate a text-based classifier for distinguishing clinical from non-clinical research articles.
Study design: We conducted a bibliographic classification study using Web of Science records published in 2023 by authors affiliated with Chinese hospitals, with temporal evaluation based on records published in 2024 from the same source.
Methods: We assembled 135,932 records published in 2023 and developed a model using a fixed subset of 20,000 records. Bidirectional Encoder Representations from Transformers (BERT), Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT), PubMedBERT, and Biomedical Language Model (BioMedLM) were evaluated using titles and keywords alone, with original abstracts, or with DeepSeek-V3-reformulated abstracts for training and validation as well as original abstracts for testing.
Results: Adding original abstracts improved the mean performance of BERT, BioBERT, and PubMedBERT. BioBERT-D3 demonstrated the highest observed accuracy (0.926), precision (0.926), F1 score (0.919), and AUC (0.972). PubMedBERT-D3 achieved the highest recall (0.922). On the 2024 temporal set, BioBERT-D3 achieved an accuracy of 0.980, a precision of 0.981, a recall of 0.970, an F1 score of 0.976, and an AUC of 0.992.
Conclusions: The framework may support first-pass literature screening and article-level bibliometric evaluation of clinical research output. Clinically ambiguous or mixed-content articles may still require human review. Although the model performed well on records published in 2024 from the same database and institutional setting, additional validation is required using other databases, datasets from different countries, and independently annotated reference labels.

Keywords

Clinical research / Medical literature / Text classification / BioBERT / Abstract reformulation / Explainable AI

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] Sayers EW, Bolton EE, Fine AM, et al. Database resources of the National Center for Biotechnology Information in 2026. Nucleic Acids Res. 2026; 54(D1):D20-D27. https://doi.org/10.1093/nar/gkaf1060

[2] Liu CL, Chen S. A review on the clinical impact identification and prediction of academic papers from the perspective of scientometrics. Libr Inf Serv. 2024; 68(5):132-148. https://doi.org/10.13266/j.issn.0252—3116.2024.05.013

[3] Wang BY, Xie QQ, Pei JH, et al. Pre—trained language models in biomedical domain: a systematic survey. ACM Comput Surv. 2024; 56(3):1-52. https://doi.org/10.1145/3611651

[4] Li XY, Liu Y, Li AH, Yang XM, Tang XL. Research on the subject mapping methods optimization of multi—sources biomedical literatures. Digit Libr Forum. 2023; 19(3):1-9. https://doi.org/10.3772/j.issn.1673—2286.2023.03.001

[5] Haynes RB, McKibbon KA, Wilczynski NL, Walter SD, Werre SR. Hedges Team. Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005; 330(7501):1179. https://doi.org/10.1136/bmj.38446.498542.8F

[6] Wilczynski NL, McKibbon KA, Haynes RB. Enhancing retrieval of best evidence for health care from bibliographic databases: calibration of the hand search of the literature. Stud Health Technol Inform. 2001; 84(pt 1):390-393. https://doi.org/10.3233/978—1—60750—928—8—390

[7] Montori VM, Wilczynski NL, Morgan D, Haynes RB. Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005; 330(7482):68. https://doi.org/10.1136/bmj.38336.804167.47

[8] Ke Q. Identifying translational science through embeddings of controlled vocabularies. J Am Med Inform Assoc. 2019; 26(6):516-523. https://doi.org/10.1093/jamia/ocy177

[9] Weber GM. Identifying translational science within the triangle of biomedicine. J Transl Med. 2013; 11:126. https://doi.org/10.1186/1479—5876—11—126

[10] Lewison G, Paraje G. The classification of biomedical journals by research level. Scientometrics. 2004; 60(2):145-157. https://doi.org/10.1023/B:SCIE.0000027677.79173.b8

[11] Hutchins BI, Yuan X, Anderson JM, Santangelo GM. Relative Citation Ratio (RCR): a new metric that uses citation rates to measure influence at the article level. PLoS Biol. 2016; 14(9):e1002541. https://doi.org/10.1371/journal.pbio.1002541

[12] Hutchins BI, Davis MT, Meseroll RA, Santangelo GM. Predicting translational progress in biomedical research. PLoS Biol. 2019; 17(10):e3000416. https://doi.org/10.1371/journal.pbio.3000416

[13] Luo PC, Wang JM, Nie L. Research on automatic classification of literature resources based on generative large language model. Inf Stud:Theory Appl. 2024; 47(12):174-182.

[14] Luo PC, Wang YB, Wang JM. Automatic discipline classification for scientific papers based on a deep pre—training language model. J China Soc Sci Tech Inf. 2020; 39(10):1046-1059.

[15] Chen Q, Du J, Allot A, Lu Z. LitMC—BERT: transformer—based multi—label classification of biomedical literature with an application on COVID—19 literature curation. IEEE/ACM Trans Comput Biol Bioinform. 2022; 19(5):2584-2595. https://doi.org/10.1109/TCBB.2022.3173562

[16] Institute of Medical Information, Chinese Academy Medical Sciences. Science Technology Evaluation Metrics (STEM). Published July 19 2024. Accessed April 29, 2026. https://stem.pumc.edu.cn/2024/index.htm.

[17] Zhou F, Saha A, Afzal M, et al. Understanding transformer—based classifications of medical text using a large language model for the attribution of feature importance: proof—of—concept algorithm development and validation study. JMIR Med Inform. 2026; 14:e81644. https://doi.org/10.2196/81644

[18] National Institutes of Health. NIH Definitions. Accessed April 27, 2025. https://orwh.od.nih.gov/toolkit/nih—policies—inclusion/definitions.

[19] van de Schoot R, de Bruin J, Schram R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021; 3(2):125-133. https://doi.org/10.1038/s42256—020—00287—7

[20] Khadhraoui M, Bellaaj H, Ammar MB, Hamam H, Jmaiel M. Survey of BERT—base models for scientific text classification: COVID—19 case study. Appl Sci (Basel). 2022; 12(6):2891. https://doi.org/10.3390/app12062891

[21] Gu Y, Tinn R, Cheng H, et al. Domain—specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc. 2022; 3(1):1-23. https://doi.org/10.1145/3458754

[22] Alsentzer E, Murphy J, Boag W, et al. Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop . Association for Computational Linguistics; 2019:72-78. https://doi.org/10.18653/v1/W19—1909

[23] Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre—training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers) . Association for Computational Linguistics; 2019:4171-4186. https://doi.org/10.18653/v1/N19—1423

[24] Lee J, Yoon W, Kim S, et al. BioBERT: a pre—trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020; 36(4):1234-1240. https://doi.org/10.1093/bioinformatics/btz682

[25] Bolton E, Venigalla A, Yasunaga M, et al. BioMedLM: a 2.7B parameter language model trained on biomedical text. arXiv. Preprint posted online March 27, 2024. https://doi.org/10.48550/arXiv.2403.18421

[26] Liu A, Feng B, Xue B, et al. DeepSeek—V3 technical report. arXiv. Preprint posted online December 27, 2024. https://doi.org/10.48550/arXiv.2412.19437

[27] Van Veen D, Van Uden C, Blankemeier L, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med. 2024; 30(4):1134-1142. https://doi.org/10.1038/s41591—024—02855—5

[28] DeepSeek Prompt Libr | DeepSeek API Docs. Accessed April 27, 2025.

[29] Wolf T, Debut L, Sanh V, et al. Transformers: state—of—the—art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Association for Computational Linguistics; 2020:38-45. https://doi.org/10.18653/v1/2020.emnlp—demos.6

[30] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI; 2019. Accessed April 27, 2026. https://cdn.openai.com/better—language—models/language—models.pdf.

[31] Tinn R, Cheng H, Gu Y, et al. Fine—tuning large neural language models for biomedical natural language processing. Patterns (N Y). 2023;4(4):100729. https://doi.org/10.1016/j.patter.2023.100729

[32] Hu EJ, Shen Y, Wallis P, et al. LoRA: low—rank adaptation of large language models. In: The Tenth International Conference on Learning Representations (ICLR 2022). OpenReview.net; 2022. Accessed April 27, 2026. https://openreview.net/forum?id=nZeVKeeFYf9

[33] Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT—4's efficacy in screening and extracting data from peer—reviewed and grey literature in multiple languages. Res Synth Methods. 2024; 15(4):616-626. https://doi.org/10.1002/jrsm.1715

[34] Lieberum JL, Toews M, Metzendorf MI, et al. Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review. J Clin Epidemiol. 2025; 181:111746. https://doi.org/10.1016/j.jclinepi.2025.111746

[35] Dennstädt F, Zink J, Putora PM, Hastings J, Cihoric N. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Syst Rev. 2024; 13(1):158. https://doi.org/10.1186/s13643—024—02575—4

[36] Emilova Doneva S, de Viragh S, Hubarava H, Schandelmaier S, Briel M, Ineichen BV. StudyTypeTeller—large language models to automatically classify research study types for systematic reviews. Res Synth Methods. 2025; 16(6):1005-1024. https://doi.org/10.1017/rsm.2025.10031

[37] Tran H, Yang Z, Yao Z, Yu H. BioInstruct: instruction tuning of large language models for biomedical natural language processing. J Am Med Inform Assoc. 2024; 31(9):1821-1832. https://doi.org/10.1093/jamia/ocae122