Background: The rapid adoption of ChatGPT in healthcare has generated extensive literature. However, systematic analysis of this emerging field using artificial intelligence (AI)-powered tools remains challenging due to the volume and diversity of publications and the risk of AI-generated hallucinations that compromise factual accuracy.
Objective: We developed ChatCM-RAG, a deep learning pipeline integrating BERTopic with transformer-based retrieval-augmented generation to analyse ChatGPT applications in the Medicine literature.
Methods: We processed 904 peer-reviewed articles (2022–2025) using a multi-stage pipeline: BERTopic for topic modelling with UMAP dimensionality reduction and HDBSCAN clustering, Facebook AI Similarity Search for semantic retrieval, and transformer models (T5/GPT) for answer generation. The system was evaluated using representative medical queries across retrieval accuracy, generation quality, and system efficiency metrics.
Results: ChatCM-RAG identified four distinct topic clusters: general medical AI applications (46.2%), performance evaluation (13.9%), clinical applications and patient care (12.6%), and chatbot implementations (6.0%), with 21.2% unclustered documents. To reduce hallucination and ensure citation authenticity, the generation module is constrained to a curated corpus of 904 PubMed-indexed documents and only permits PMID citations that exist in the retrieval set. In our pilot evaluation on eight representative medical queries, we observed 0 fabricated PMIDs and 0 non-existent citations in generated answers. The system achieved an average response time of 1.73 s, an answer quality score of 0.81, and demonstrated topic-aware retrieval with a 0.90 relevance score, where 73% of retrieved documents originated from topically appropriate clusters. The ChatCM-RAG model has been open-sourced at https://huggingface.co/fc28/ChatCM-RAG. Additional reproducible analyses were performed using the released dataset (n = 904) and code to quantify topic interpretability and retrieval robustness without requiring external LLM calls. Using an 80/20 held-out title-query evaluation, topic-filtered lexical retrieval (TF‑IDF within predicted cluster) improved Cluster-agreement@5 from 0.599 ± 0.309 (global TF‑IDF) to 0.862 ± 0.345. Top TF-IDF terms were identified for each cluster, and retrieval performance was assessed using a held-out evaluation.
Conclusions: ChatCM-RAG effectively synthesises large-scale medical literature, revealing that ChatGPT applications in medicine are dominated by exploratory studies with an emerging focus on clinical decision support. The open-source pipeline provides researchers with powerful tools for understanding AI integration in traditional medicine.
| [1] |
Liu X, He L, Alanazi E, Liu E, Goss A, Gumireddy L. Assessing the accuracy and explainability of using ChatGPT to evaluate the quality of health news. BMC Public Health. 2025;25:2038.
|
| [2] |
Nian PP, Umesh A, Jones RH, et al. ChatGPT and Google Gemini are clinically inadequate in providing recommendations on management of developmental dysplasia of the hip compared to American Academy of Orthopaedic Surgeons Clinical Practice Guidelines. J Pediatr Soc North Am. 2025;10:100135.
|
| [3] |
Li K, Bu Z, Shahjalal M, et al. Performance of ChatGPT on Chinese Master's Degree Entrance Examination in Clinical Medicine. PLoS One. 2024;19:e0301702.
|
| [4] |
Tseng LW, Lu YC, Tseng LC, Chen YC, Chen HY. Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: cross-Sectional Study. JMIR Med Educ. 2025;11:e58897.
|
| [5] |
Pan D, Guo Y, Fan Y, Wan H. Development and application of traditional Chinese medicine using AI machine learning and deep learning strategies. Am J Chin Med. 2024;52:605-623.
|
| [6] |
Yang Q, Zuo H, Su R, et al. Dual retrieving and ranking medical large language model with retrieval augmented generation. Sci Rep. 2025;15:18062.
|
| [7] |
Kim J, Kim SJ, Ahn J, Lee S. LLM-Based Response Generation for Korean Adolescents: a Study Using the NAVER Knowledge in Q&A Dataset with RAG. Healthc Inform Res. 2025;31:136-145.
|
| [8] |
Ma L, Chen R, Ge W, et al. AI-powered topic modelling: comparing LDA and BERTopic in analysing opioid-related cardiovascular risks in women. Exp Biol Med. 2025;250:10389.
|
| [9] |
Miranda O, Kiehl SM, Qi X, et al. Enhancing post-traumatic stress disorder patient assessment: leveraging natural language processing for research of domain criteria identification using electronic medical records. BMC Med Inform Decis Mak. 2024;24:154.
|
| [10] |
González-Alemán R, Platero-Rochart D, Rodríguez-Serradet A, et al. MDSCAN: RMSD-based HDBSCAN clustering of long molecular dynamics. Bioinformatics. 2022;38:5191-5198.
|
| [11] |
Melvin RL, Xiao J, Godwin RC, Berenhaut KS. Visualizing correlated motion with HDBSCAN clustering. Protein Sci. 2018;27:62-75.
|
| [12] |
Zhu Y, Yang X, Wu Y, Zhang W. Leveraging summary guidance on medical report summarisation. IEEE J Biomed Health Inform. 2023;27:5066-5075.
|
| [13] |
Luo R, Sun L, Xia Y, et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform. 2022;23:bbac409.
|
| [14] |
Liu W, Hasikin K, Khairuddin ASM, Liu M, Zhao X. A temporal knowledge graph reasoning model based on recurrent encoding and contrastive learning. PeerJ Comput Sci. 2025;11:e2595.
|
| [15] |
Manna A, Dewan D, Sheet D. Structured hashing with deep learning for modality, organ, and disease content sensitive medical image retrieval. Sci Rep. 2025;15:8912.
|
| [16] |
Chung JH, Kim Y, Lee D, et al. Machine learning-based real-time prediction of duodenal stump leakage from gastrectomy in gastric cancer patients. Front Surg. 2025;12:1550990.
|
RIGHTS & PERMISSIONS
2026 The Author(s). Clinical and Translational Discovery published by John Wiley & Sons Australia, Ltd on behalf of Shanghai Institute of Clinical Bioinformatics.
PDF
(4302KB)
