Confabulated references in the age of AI: contamination of the biomedical scientific literature

Abstract

Large language models (LLMs) like ChatGPT are increasingly used in drafting scientific papers. While they can improve clarity and efficiency, a troubling issue has emerged: the inclusion of fabricated references-nonexistent citations that can mislead, especially in biomedical research where evidence integrity is crucial. Studies indicate that 69% of references in ChatGPT’s medical queries are false, and only 7% of AI-generated medical articles contain accurate references. These fake citations often mimic real authors and journals, making detection difficult. Such inaccuracies can compromise research integrity, skew citation metrics, and reduce trust in scientific literature. To address this, journals are adopting policies requiring disclosure of AI use and human verification of references. Nonetheless, detecting AI-related misinformation remains challenging, and many experts believe the problem is bigger than currently known. Going forward, authors should avoid relying solely on LLMs, and reviewers must scrutinize references carefully. The scientific community needs to balance AI’s usefulness with rigorous oversight, ensuring that the pursuit of efficiency doesn't undermine credibility. Ultimately, safeguarding research from AI-generated misinformation will require combined efforts of transparency, vigilance, and adherence to ethical principles, preserving the integrity of biomedical science.

Keywords

large language models / scientific writing / systematic review / research integrity / fabricated references / ChatGPT

References

[1] Fiorillo L. Confronting the demonization of AI writing: Reevaluating its role in upholding scientific integrity. Oral Oncol Rep. 2024; 12: 100685.

[2] Gravel J, D’Amours-Gravel M, Osmanlliu E. Learning to Fake It: Limited Responses and Fabricated References Provided by ChatGPT for Medical Questions. Mayo Clin Proc Digit Health. 2023; 1: 226-34.

[3] Bai Y, Kosonocky CW, Wang JZ. How our authors are using AI tools in manuscript writing. Patterns (N Y). 2024; 5: 101075.

[4] Mac Oscar M, Boland M, Cadogan C. An assessment of generative artificial intelligence in responding to clinical queries on tapering antidepressants. Res Social Adm Pharm. 2025; 21: 924-30.

[5] Madhavan G. RETRACTED BOOK: Mastering Machine Learning: From Basics to Advanced. In: Transactions on Computer Systems and Networks. Singapore: Springer Nature; 2025.

[6] Fiorillo L. Fi-Index: A New Method to Evaluate Authors Hirsch-Index Reliability. Publ Res Q. 2022; 38: 465-74.

[7] Bhattacharyya M, Miller VM, Bhattacharyya D, Miller LE. High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content. Cureus. 2023; 15: e39238.

[8] Glynn A. Guarding against artificial intelligence--hallucinated citations: The case for full-text reference deposit. arXiv 2503.19848 [Preprint]. 2025 [cited 2025 Nov 25]. Available from: https://arxiv.org/abs/2503.19848

[9] Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023; 13: 14045.

[10] Journal taking ‘corrective actions’ after learning author used ChatGPT to update references [Internet]. Retraction Watch; [cited 2025 Sep 21]. Available from: https://retractionwatch.com/2024/05/20/journal-taking-corrective-actions-after-learning-author-used-chatgpt-to-update-references/

[11] Paper rejected for AI, fake references published elsewhere with hardly anything changed [Internet]. Retraction Watch; [cited 2025 Sep 21]. Available from: https://retractionwatch.com/2025/06/12/paper-rejected-for-ai-fake-references-published-elsewhere-with-hardly-anything-changed/

[12] Springer Nature retracts machine learning book after citations ‘reference works that don’t exist’ [Internet]. The Bookseller; [cited 2025 Sep 21]. Available from: https://www.thebookseller.com/news/springer-nature-retracts-machine-learning-book-after-citations-reference-works-that-dont-exist

[13] Kalai AT, Nachum O, Vempala SS, Zhang E. Why Language Models Hallucinate. arXiv 2509.04664 [Preprint]. 2025 [cited 2025 Nov 25]. Available from: https://arxiv.org/abs/2509.04664

[14] ChatGPT Fails to Flag Retracted and Problematic Articles [Internet]. The Scientist; [cited 2025 Sep 21]. Available from: https://www.the-scientist.com/chatgpt-fails-to-flag-retracted-and-problematic-articles-73448

[15] Wallace MB, Siersema PD. Ethics in publication. Gastrointest Endosc. 2015; 82: 439-42.

[16] Singhal S, Kalra BS. Publication ethics: Role and responsibility of authors. Indian J Gastroenterol. 2021; 40: 65-71.

[17] Ioannidis JPA, Boyack KW, Baas J. Updated science-wide author databases of standardized citation indicators. PLoS Biol. 2020; 18: e3000918.

[18] Yoo J. Defining the Boundaries of AI Use in Scientific Writing: A Comparative Review of Editorial Policies. J Korean Med Sci. 2025; 40: e187.

[19] Chetwynd E. Ethical Use of Artificial Intelligence for Scientific Writing: Current Trends. J Hum Lact. 2024; 40: 211-5.

[20] Fiorillo L, Cicciù M. The Use of Fi-Index Tool to Assess Per-manuscript Self-citations. Publ Res Q. 2022; 38: 684-92.

[21] Smith R. US calls for research into research integrity. BMJ. 2000; 321: 1369B.