From Manual to Machine: Revolutionizing Day Surgery Guideline and Consensus Quality Assessment With Large Language Models

Abstract

Objective: To evaluate the methodological and reporting quality of clinical practice guidelines/expert consensus for ambulatory surgery centers published since 2000, combining manual assessment with large language model (LLM) analysis, while exploring LLMs’ feasibility in quality evaluation.

Methods: We systematically searched Chinese/English databases and guideline repositories. Two researchers independently screened literature and extracted data. Quality assessments were conducted using AGREE II and RIGHT tools through both manual evaluation and GPT-4o modeling.

Results: 54 eligible documents were included. AGREE II domains showed mean compliance: Scope and purpose 25.00%, Stakeholder involvement 20.16%, Rigor of development 17.28%, Clarity of presentation 41.56%, Applicability 18.06%, Editorial independence 26.39%. RIGHT items averaged: Basic information 44.44%, Background 36.11%, Evidence 14.07%, Recommendations 34.66%, Review and quality assurance 3.70%, Funding and declaration and management of interests 24.54%, Other information 27.16%. LLMs’-evaluated documents demonstrated significantly higher scores than manual assessments in both tools. Subgroup analyses revealed superior quality in documents with evidence retrieval, conflict disclosure, funding support, and LLM integration (P < 0.05).

Conclusions: Current guidelines and consensus related to day surgery need to improve their methodological quality and quality of reporting. The study validates LLMs’ supplementary value in quality assessment while emphasizing the necessity of maintaining manual evaluation as the foundation.

Keywords

AGREE II / consensus / day surgery / guideline / LLM / quality assessment / RIGHT

References

[1] J. Luo,C. Xie, and D. Fan,“Historical Development and Experience of Day Surgery in China: From the Perspective of Anesthesiologists, ” Paediatric Anaesthesia (2025),

[2] Day Surgery Anesthesia Branch of Chinese Society of Cardiothoracic Anesthesia,“Expert Consensus on Whole-Process Management of Same-Day Surgery for Enhanced Recovery after Surgery” Task Force, and L. Peng, “Expert Consensus on Whole-Process Management of Same-Day Surgery for Enhanced Recovery After Surgery, ” Chinese Journal of Anesthesiology 43, no. 4 (2023):385–399.

[3] X. Luo,H. Lyu,Q. Shi, et al.,“The Application of Large Language Models in the Field of Evidence-Based Medicine, ” Zhongguo Xunzheng Yixue Zazhi 24, no. 4 (2024):373–377.

[4] H. Lai,L. Ge,M. Sun, et al.,“Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models, ” JAMA Network Open 7, no. 5 (2024): e2412687.

[5] M. C. Brouwers,M. E. Kho,G. P. Browman, et al.,“AGREE II: Advancing Guideline Development, Reporting and Evaluation in Health Care, ” Cmaj 182, no. 18 (2010): E839–842.

[6] Y. Chen,K. Yang,A. Marušić, et al.,“A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement, ” Annals of Internal Medicine 166, no. 2 (2017):128–132.

[7] Y. Chen,J. Zhang,T. Zhang, and F. Sun,“Systematic Review of Clinical Practice Guidelines: What, Why and How, ” Medical Journal of Peking Union Medical College Hospital 11, no. 3 (2020):320–324.

[8] N. R. Haddaway,M. J. Page,C. C. Pritchard, and L. A. McGuinness,“PRISMA2020: An R Package and Shiny App for Producing PRISMA 2020-Compliant Flow Diagrams, With Interactivity for Optimised Digital Transparency and Open Synthesis, ” Campbell Systematic Reviews 18, no. 2 (2022): e1230.

[9] J. Xu,“Expert Consensus on Postoperative Analgesia After Adult Day Surgery (2017), ” Journal of Clinical Anesthesiology 33, no. 8 (2017):812–815.

[10] Working Group of “Expert Consensus on Anesthesia Management of Enhanced Recovery after Ambulatory Surgery, “Expert Consensus on Anesthesia Management of Enhanced Recovery After Adult Ambulatory Surgery, ” Medical Journal of Peking Union Medical College Hospital 10, no. 6 (2019):562–569.

[11] Chinese Research Hospitals Association,Committee of Enhanced Recovery After Surgery; China Ambulatory Surgery Alliance,“Expert Consensus on Standardized Processes of Ambulatory Biliary Surgery (2018 Edition), ” Chinese Journal of Surgery 56, no. 5 (2018):321–327.

[12] The Hernia and Abdominal Wall Surgery Specialty Day Surgery Expert Committee of the China Day Surgery Cooperation Alliance,the Editorial Committee of the Chinese Journal of Hernia and Abdominal Wall Surgery (Electronic Edition),the Hernia and Abdominal Wall Surgery Industry and Clinical Research Branch of the National Health Industry and Enterprise Management Association,“Chinese Expert Guideline on Standardized Process and Standards of Ambulatory Surgery for Inguinal Hernia (2023 edition), ” Chinese Journal of Hernia and Abdominal Wall Surgery (Electronic Edition) 17, no. 5 (2023):497–503.

[13] W. Xiao and S. Gao,“Expert Consensus on Clinical Practice of Arthroscopy of Day Surgery, ” China Journal of Endoscopy 26, no. 6 (2020):1–7.

[14] Writing Committee of Expert Consensus on Day Surgery for Coronary Angiography,“Experts Consensus on Ambulatory Surgery of Coronary Angiography, ” Chinese Journal of Hypertension 31, no. 10 (2023):902–907.

[15] C. Zhang,Y. Gao,H. Zhang, and Y. Cheng,“Expert Consensus on the Clinical Practice of Robotic-Assisted Thoracic Day Surgery, ” China Journal of Endoscopy 27, no. 8 (2021):10–20.

[16] K. Jiangi,“Chinese Expert Consensus on Thyroid Day Surgery(2021 Edition), ” Chinese Journal of General Surgery 30, no. 5 (2021):499–509.

[17] C. Jiang,X. Jian,Z. Zhang, et al.,“Chinese Expert Consensus on Oral and Maxillofacial Ambulatory Surgery, ” China Journal of Oral and Maxillofacial Surgery 17, no. 5 (2019):385–390.

[18] Minimally Invasive Surgery Committee of the Chinese Research Hospital Association,“Expert Consensus on Day-Case Laparoscopic Cholecystectomy, ” Journal of Laparoscopic Surgery 28, no. 8 (2023):561–567+73.

[19] T. Jiali,Z. Genhai,S. Dawei, et al.,“Consensus of Chinese Experts on Hysteroscopy Day Surgery Center Set-up and Management Process, ” Chinese Journal of Obstetrics and Gynecology 57, no. 12 (2022):891–899.

[20] Day Surgery Nursing Committee of Sichuan Tianfu New Area Medical Association,“Expert Consensus on the Management Model of Integrating Emergency and Routine Medical Care in Day Surgery Wards, ” West China Medical Journal 39, no. 2 (2024):173–177.

[21] National Clinical Research Center for Geriatric Disorders (Xiangya),“Expert Consensus on the Norms for Writing Medical Records of Day Surgery (2019), ” Chinese Journal of General Surgery 28, no. 10 (2019):1171–1176.

[22] Editorial Board of Chinese Journal of Digestive Surgery,“Chinese Expert Consensus on Standardization Process of Hepatobiliary Diseases in Day Surgery (2022 Edition), ” Chinese Journal of Digestive Surgery 21, no. 2 (2022):185–190.

[23] Chinese Society of Anesthesiology,“Guidelines for Anesthesia Management for Ambulatory Surgery, ” Chinese Medical Journal 103, no. 43 (2023):3462–3471.

[24] Chinese Society of Anesthesiology,Chinese Medical Association,“Expert Consensus on Anesthesia for Day Surgery, ” Journal of Clinical Anesthesiology 32, no. 10 (2016):1017–1022.

[25] R. Ling,L. Ma,Y. Liu,Y. Liu, and J. Zhang,“Chinese Expert Consensus on the Ambulatory Surgery of Breast Surgery(2021 edition), ” Chinese Journal of Practical Surgery 41, no. 11 (2021):1201–1205.

[26] Chinese Society of Cardiothoracic and Vascular Anesthesia,Day Surgery Anesthesia Section Pediatric Anesthesia Group of the Anesthesia Society of the Chinese Medical Association,“Guidelines for Anesthesia for Pediatric Day Surgery, ” Chinese Medical Journal 99, no. 8 (2019):566–570.

[27] Endoscopic Surgery Group Chinese Society of Pediatric Surgery Chinses Medical Association,“Expert Consensus on Pediatric Ambulatory Surgery, ” Chinese Journal of Pediatric Surgery 41, no. 8 (2020):676–682.

[28] X. Chen,C. Chen,H. Duan, et al.,“Consensus of Experts on Daytime Operation for Hemorrhoiddiagnosis and Treatment (2020 Edition), ” Journal of Clinical Medicine in Practice 24, no. 10 (2020):1–8.

[29] Chinese Medical Association Orthopedic Branch Joint Group,“Expert Consensus on Perioperative Management of Day Surgery for Hip and Knee Arthroplasty in China, ” Chinese Journal of Geriatric Orthopaedics and Rehabilitation (Electronic Edition) 9, no. 6 (2023):321–332.

[30] Ophthalmological Society of China Medicine Education Association; Ophthalmology Association of PLA Medical Science Technical Committee; Ophthalmology Branch of the Chinese Academy of Geriatrics,“Chinese Expert Consensus on Management of Ophthalmic Day Surgery (2021), ” Chinese Journal of Ophthalmology 57, no. 6 (2021):406–414.

[31] China Ambulatory Surgery Alliance,Editorial Board of Chinese Journal of Hernia and Abdominal Wall Surgery (Electronic Edition),Chinese Hernia College of Surgeons,China Hernia Society,Chinese Hernia Surgery Industry and Clinical Research Society,National Association of Health Industry and Enterprise Management,Society of Hernia and Abdominal Wall Surgery,Chinese Elite Group of Digestive Surgery,C. Jie,T. Jianxiong, and S. Yingmo,“Chinese Expert Consensus on Standard Process of Ambulatory Surgery for Small and Medium Ventral Hernia (2022 edition), ” Chinese Journal of Digestive Surgery 21, no. 9 (2022):1165–1172.

[32] National Clinical Research Center for Geriatric Disorders (Xiangya Hospital),International Joint Research Center of Minimally Invasive Endoscopic Technology Equipment &Standards,X. Yingping and L. Weidong,“Chinese Expert Consensus on the Management of Day Operating Rooms in General Hospitals (2022 Edition), ” Chinese Journal of Digestive Surgery 21, no. 9 (2022):1173–1179.

[33] B. Vinson Bonnet and F. Juguet,“Ambulatory Proctologic Surgery: Recommendations of the French National Coloproctology Society (SNFCP), ” Journal of Visceral Surgery 152, no. 6 (2015):369–372.

[34] C. A. Ternent,F. Fleming,M. L. Welton,W. D. Buie,S. Steele, and J. Rafferty,“Clinical Practice Guideline for Ambulatory Anorectal Surgery, ” Diseases of the Colon and Rectum 58, no. 10 (2015):915–922.

[35] T. J. Gan,P. Diemunsch,A. S. Habib, et al.,“Consensus Guidelines for the Management of Postoperative Nausea and Vomiting, ” Anesthesia and Analgesia 118, no. 1 (2014):85–113.

[36] S. Kulkarni,S.s Harsoor,M. Chandrasekar, et al.,“Consensus Statement on Anaesthesia for Day Care Surgeries, ” Indian Journal of Anaesthesia 61, no. 2 (2017):110–124.

[37] T. Anderson,M. Walls, and R. Canelo,“Day Case Surgery Guidelines, ” Surgery (United Kingdom) 35, no. 2 (2017):85–91.

[38] L. Ng and M. Mercer-Jones,“Day Case Surgery Guidelines, ” Surgery (Oxford) 32, no. 2 (2014):73–78.

[39] M. J. Marbaniang,“Day Surgery Guidelines, ” Surgery (Oxford) 37, no. 12 (2019):706–711.

[40] L. Venclauskas,J. V. Llau,J. Y. Jenny,P. Kjaersgaard-Andersen, and Ø. Jans,“European Guidelines on Perioperative Venous Thromboembolism Prophylaxis: Day Surgery and Fast-Track Surgery, ” European Journal of Anaesthesiology 35, no. 2 (2018):134–138.

[41] L. De Gabory,E. Serrano,J. B. Lecanu, et al.,“French Otorhinolaryngology Society Guidelines for Day-Case Nasal Surgery, ” European Annals of Otorhinolaryngology Head &Neck Diseases 132, no. 1 (2015):35–40.

[42] N. N. Dephillipo,C. M. Larson,O. R. O’neill, and R. F. Laprade,“Guidelines for Ambulatory Surgery Centers for the Care of Surgically Necessary/Time-Sensitive Orthopaedic Cases During the COVID-19 Pandemic, ” Journal of Bone and Joint Surgery. American Volume 102, no. 11 (2020):933–936.

[43] C. R. Bailey,M. Ahuja,K. Bartholomew, et al.,“Guidelines for Day-Case Surgery 2019: Guidelines From the Association of Anaesthetists and the British Association of Day Surgery, ” Anaesthesia 74, no. 6 (2019):778–792.

[44] D. Maciejewski,“Guidelines for System and Anaesthesia Organisation in Short Stay Surgery (Ambulatory Anaesthesia, Anaesthesia in Day Case Surgery), ” Anaesthesiology Intensive Therapy 45, no. 4 (2013):190–199.

[45] U. De Luca,G. Mangia,S. Tesoro,A. Martino,M. Sammartino, and A. Calisti,“Guidelines on Pediatric Day Surgery of the Italian Societies of Pediatric Surgery (SICP) and Pediatric Anesthesiology (SARNePI), ” Italian Journal of Pediatrics 44, no. 1 (2018):35.

[46] L. De Gabory,L. J. Sowerby,J. M. DelGaudio,A. Al-Hussaini,C. Hopkins, and E. Serrano,“International Survey and Consensus (ICON) on Ambulatory Surgery in Rhinology, ” European Annals of Otorhinolaryngology Head &Neck Diseases 135, no. 1s (2018): S49–S53.

[47] K. Chaidas and C. Winterborn,“Oxford Guidelines for Adult Day-Case Tonsillectomy, ” Journal of Perioperative Practice 33, no. 1-2 (2023):9–14.

[48] G. P. Joshi,S. P. Ankichetty,T. J. Gan, and F. Chung,“Society for Ambulatory Anesthesia Consensus Statement on Preoperative Selection of Adult Patients With Obstructive Sleep Apnea Scheduled for Ambulatory Surgery, ” Anesthesia and Analgesia 115, no. 5 (2012):1060–1068.

[49] N. Rajan,E. W. Duggan,B. B. Abdelmalak, et al.,“Society for Ambulatory Anesthesia Updated Consensus Statement on Perioperative Blood Glucose Management in Adult Patients with Diabetes Mellitus Undergoing Ambulatory Surgery, ” Anesthesia and Analgesia 139, no. 3 (2024):459–477.

[50] P. Gandhimani and I. J. B. Jackson,“UK Guidelines for Day Surgery, ” Surgery (Oxford) 24, no. 10 (2006):346–349.

[51] Y. Gu,P. Zhang,Y. Wang, et al.,“Chinese Expert Consensus on Ambulatory Laparoscopic Sleeve Gastriectomy (2021 Edition), ” Chinese Journal of Practical Surgery 42, no. 1 (2022):17–23.

[52] Day Surgery Expert Group of National Institute of Hospital Administration,National Health Commission,Expert Working Group on Expert Consensus on the Writing and Management of Day Surgery Medical Records (2024 Edition),“Expert Consensus on the Writing and Management of Day Surgery Medical Records (2024 Edition), ” Chinese Journal of Hospital Administration 40, no. 4 (2024):254–262.

[53] Day Care Expert Database of the National Health Commission Hospital Management Institute,Expert Working Group on Expert Consensus on Nursing Management of Day Surgery,“Expert Consensus on Whole-process Nursing Management of Day Surgery.” Chinese Journal of Hospital Administration 40, no. 10 (2024):761–768.

[54] J. Zhang,Q. Jiang,X. Lai, and Day Surgery Nursing Committee of Sichuan Tianfu New Area Medical Association,“Expert Consensus on Perioperative Health Education for Patients Undergoing Thoracoscopic Pulmonary Nodule Day Surgery, ” West China Medical Journal 40, no. 2 (2025):183–187.

[55] Y. X. Shuguang Gao,“Chinese Expert Consensus on Clinical Practice of Daytime Hip Arthroplasty, ” Chinese Journal of Reparative and Reconstructive Surgery 38, no. 5 (2024):513–520.

[56] Day Surgery Expert Group of National Institute of Hospital Administration of National Health Commission,Expert Working Group of Expert Consensus on Quality Management of Perioperative Anesthesia during Day Surgery,“Expert Consensus on Quality Management of Perioperative Anesthesia During Day Surgery, ” Chinese Journal of Hospital Administration 40, no. 10 (2024):752–760.

[57] Y. Tu,Y. Liu,J. Chen,Chinese Medical Doctor Association Thoracic Surgeons Branch Hyperhidrosis Subcommittee, and Fujian Provincial Strait Medical and Health Exchange Association Hyperhidrosis Special Committee,“Chinese Expert Consensus on ETS Optimization and Surgical Quality Control of Day Surgery for Palmar Hyperhidrosis, ” Chinese Journal of Clinical Thoracic and Cardiovascular Surgery 32, no. 1 (2025):8–13.

[58] China Anti-Cancer Association Thymic Tumor Integrated Rehabilitation Special Committee,Z. Chunfang,F. Junqiang, and L. Hecheng. “Consensus of Experts on Day Surgery for Thymus Tumor, ” Chinese Journal of Thoracic and Cardiovascular Surgery 2024;(10):589–598.

[59] Consensus Group of Experts on Day Surgery Management of Lung Cancer,Hospital Management Research Institute of the National Health Commission,“Chinese Expert Consensus on Day Surgery Management of Lung Cancer (2024 Edition), ” Chinese Journal of Lung Cancer 27, no. 6 (2024):405–414.

[60] National Clinical Medical Research Center for Geriatric Diseases (Xiangya),China Day Surgery Cooperative Alliance,“Clinical Practice Guidelines for Day Surgery of the Rectum and Anus (2019 Edition), ” Chinese Journal of Gastrointestinal Surgery 22, no. 11 (2019):1001–1011.

[61] Professional Committee of Pancreatic Disease,Chinese Medical Doctor Association,Endoscopic Surgery Group,Digestive Endoscopy Branch,Chinese Medical association,Endoscopic Surgery Expert Working Group,Chinese College of Surgeons, and L. Xun,“Chinese Expert Consensus on the Management of ERCP Day Surgery (2025 Edition), ” Chinese Journal of Digestive Surgery 24, no. 1 (2025):31–40.

[62] S. M. O’shaughnessy,J. Y. Lee,L. Q. Rong, et al.,“Quality of Recent Clinical Practice Guidelines in Anaesthesia Publications Using the Appraisal of Guidelines for Research and Evaluation II Instrument, ” British Journal of Anaesthesia 128, no. 4 (2022):655–663.

[63] E. K. Alexander,E. N. Pearce,G. A. Brent, et al.,“2017 Guidelines of the American Thyroid Association for the Diagnosis and Management of Thyroid Disease During Pregnancy and the Postpartum, ” Thyroid: Official Journal of the American Thyroid Association 27, no. 3 (2017):315–389.

[64] L. Brosseau,P. Rahman,S. Poitras, et al.,“A Systematic Critical Appraisal of Non-Pharmacological Management of Rheumatoid Arthritis With Appraisal of Guidelines for Research and Evaluation II, ” PLoS ONE 9, no. 5 (2014): e95369.

[65] Y. Fu,C. Wang, and Y. Hu,“Content Analysis of Barriers and Facilitators in the Clinical Translation of Evidence, ” Journal of Nursing(China) 29, no. 5 (2022):64–68.

[66] A. M Merchan-Galvis,J. P. Caicedo,C. J. Valencia-Payán, and J. A. Calvache,“Methodological Quality and Transparency of Clinical Practice Guidelines for Difficult Airway Management Using the Appraisal of Guidelines Research &Evaluation II Instrument: A Systematic Review, ” European Journal of Anaesthesiology 37, no. 6 (2020):451–456.

[67] Y. Hoydonckx,P. Kumar,D. Flamer, et al.,“Quality of Chronic Pain Interventional Treatment Guidelines From Pain Societies: Assessment With the AGREE II Instrument, ” European Journal of Pain (London, England) 24, no. 4 (2020):704–721.

[68] Y. Chen,K. Yang,X. Wang, et al.,“Guiding Principles for the Development/Revision of Clinical Diagnosis and Treatment Guidelines in China (version 2022), ” Chinese Medical Journal 102, no. 10 (2022):697–703.

[69] A. Moore,Y. Wu,L. Kwakkenbos, et al.,“The Patient Engagement Evaluation Tool Was Valid for Clinical Practice Guideline Development, ” Journal of Clinical Epidemiology 143 (2022):61–72.

[70] D. Wang,N. Liu,Z. Shi,H. Li,H. Peng, and W. Zhao,“Evaluation of Clinical Guidelines and Consensus on the Quality of Central Venous Catheters, ” Zhongguo Xunzheng Yixue Zazhi 23, no. 8 (2023):955–962.

[71] M. Xiong,J. Zhou,N. Dai,X. Liu,X. Xue, and J. Liu,“A Guideline Quality Evaluation for the Diagnosis and Treatment of Chronic Kidney Diseases in Traditional Chinese Medicine/Integrated Chinese and Western Medicine Based on AGREE II, ” Modern Chinese Clinical Medicine 29, no. 6 (2022):50–58.

[72] L. Ke,J. Wang,H. Wang, et al.,“Reporting Quality Evaluation of Clinical Practice Guidelines Published in Journals of Mainland China in 2016, ” Chinese Journal of Evidence Based Pediatrics 13, no. 3 (2018):194–199.

[73] N. Yang,H. Liu,W. Zhao, et al.,“Development of the Scientific, Transparent and Applicable Rankings (STAR) Tool for Clinical Practice Guidelines, ” Chinese Medical Journal 136, no. 12 (2023):1430–1438.

[74] J. Ye and W. Guo,“Reflections on the Participation of Generative AI in Academic Evaluation, ” China Social Science Review no. 1 (2024):37–48. 158.

[75] Y. Wang,Q. Li,Z. Dai, and Y. Xu,“Current Status and Trends in Large Language Modeling Research, ” Chinese Journal of Engineering 46, no. 8 (2024):1411–1425.

[76] E. M. Bender,T. Gebru,A. McMillan-Major, and S. Shmitchell,“On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, ” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency; Virtual Event, Canada: Association for Computing Machinery;2021:610–623.

[77] Z. Ji,N. Lee,R. Frieske, et al.,“Survey of Hallucination in Natural Language Generation, ” ACM Computing Surveys 55, no. 12 (2023):248.

[78] A. J. Thirunavukarasu,D. S. J. Ting,K. Elangovan,L. Gutierrez,T. F. Tan, and D. S. W. Ting,“Large Language Models in Medicine, ” Nature Medicine 29, no. 8 (2023):1930–1940.

[79] Kocmi T,Federmann C, eds.,Large Language Models Are State-of-the-Art Evaluators of Translation Quality (European Association for Machine Translation Conferences/Workshops,2023).

[80] C. Rudin,“Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, ” Nature Machine Intelligence 1, no. 5 (2019):206–215.

[81] K. Singhal,S. Azizi,T. Tu, et al.,“Large Language Models Encode Clinical Knowledge, ” Nature 620, no. 7972 (2023):172–180.

[82] E. J. Topol,“High-Performance Medicine: The Convergence of Human and Artificial Intelligence, ” Nature Medicine 25, no. 1 (2019):44–56.

[83] Y. Liu,J. Zhang,Q. Shi, et al.,“Investigation and Evaluation of Chinese Clinical Practice Guidelines Published in Medical Journals in 2019: Methodological and Reporting Quality, ” Medical Journal Of Peking Union Medical College Hospital 13, no. 2 (2022):324–331.

RIGHTS & PERMISSIONS

2025 The Author(s). Journal of Evidence-Based Medicine published by Chinese Cochrane Center, West China Hospital of Sichuan University and John Wiley & Sons Australia, Ltd.