Artificial intelligence versus human researcher performance for systematic literature searches: a study focusing on the surgical management of base of thumb arthritis

Abstract

Aim: In the digital age, artificial intelligence (AI) platforms have gradually replaced traditional manual techniques for information retrieval. However, their effectiveness in conducting academic literature searches remains unclear, necessitating a comparative assessment. This study examined the efficacy of AI search engines (Elicit, Consensus, ChatGPT) vs. manual search for literature retrieval, focusing on the surgical management of trapeziometacarpal osteoarthritis.

Methods: The study was executed per the Cochrane Handbook for Systematic Reviews and PRISMA guidelines. AI platforms were given relevant keywords and prompts, while manual searches used PubMed, Cochrane CENTRAL, Web of Science, and Scopus databases from January 1901 to April 2024. The study focused on English-language randomized controlled trials (RCTs) comparing surgical management of trapeziometacarpal osteoarthritis (TMCJ OA). Two independent evaluators screened and extracted data from the studies. Primary outcomes involved the quality and relevancy of studies chosen by both search methods, evaluated by false positive rates and number of studies, including outcomes of interest.

Results: The manual search yielded the most results (6,018), followed by Elicit (4,980), Consensus (3,436), and ChatGPT (6). Elicit identified the highest number of RCTs (205) but also had the greatest false positive rate (94%). Ultimately, the manual search identified 23 suitable studies, Elicit found 10, Consensus found 9, and ChatGPT identified only 1. No additional studies were found by AI search engines that were not discovered in the manual search.

Conclusion: The findings highlight the potential advantages and drawbacks of AI search engines for literature searches. While Elicit was prone to error, Consensus and ChatGPT were less comprehensive. Significant enhancements in the precision and thoroughness of AI search engines are required before they can be effectively utilized in academia.

Keywords

Artificial intelligence / human / researcher / systematic review / searches

References

[1] Yuniarthe Y.Application of artificial intelligence (AI) in search engine optimization (SEO).IEEE2017;96-101

[2] Wagner G,Paré G.Artificial intelligence and the conduct of literature reviews.J Inf Technol2022;37:209-26

[3] Kung J.Elicit (product review).J Can Health Libr Assoc2023;44:15

[4] Schiermeier Q. Pirate research-paper sites play hide-and-seek with publishers. Nature. 2015. Available from: https://www.nature.com/articles/nature.2015.18876. [Last accessed on 6 Jan 2025].

[5] Ma J, Wu X, Huang L. The use of artificial intelligence in literature search and selection of the PubMed database. Sci Program. 2015. Available from: https://onlinelibrary.wiley.com/doi/10.1155/2022/8855307. [Last accessed on 6 Jan 2025].

[6] Cinquini M,Catanuto G.Should acellular dermal matrices be used for implant-based breast reconstruction after mastectomy?.Plast Reconstr Surg Glob Open2023;11:e4821 PMCID:PMC9946425

[7] Bowers MR,Pulos BP.Opioid-sparing pain management in upper extremity surgery: part 2: surgeon as prescriber.J Hand Surg Am2019;44:878-82

[8] Higgins JPT,Chandler J.Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2019.

[9] Belcher HJ.A comparison of trapeziectomy with and without ligament reconstruction and tendon interposition.J Hand Surg Br2000;25:350-6

[10] Belcher HJ.Adverse effect of porcine collagen interposition after trapeziectomy: a comparative study.J Hand Surg Br2001;26:159-64

[11] Brennan A,Thomson J.Simple trapeziectomy versus trapeziectomy with flexor carpi radialis suspension: a 17-year follow-up of a randomized blind trial.J Hand Surg Eur Vol2021;46:120-4

[12] Corain M,Mugnai R.Interposition arthroplasty versus hematoma and distraction for the treatment of osteoarthritis of the trapeziometacarpal joint.J Hand Surg Asian Pac Vol2016;21:85-91

[13] Smet L, Sioen W, Spaepen D, van Ransbeeck H. Treatment of basal joint arthritis of the thumb: trapeziectomy with or without tendon interposition/ligament reconstruction.Hand Surg2004;9:5-9

[14] Field J.To suspend or not to suspend: a randomised single blind trial of simple trapeziectomy versus trapeziectomy and flexor carpi radialis suspension.J Hand Surg Eur Vol2007;32:462-6

[15] Gangopadhyay S,Burke FD.Five- to 18-year follow-up for treatment of trapeziometacarpal osteoarthritis: a prospective comparison of excision, tendon interposition, and ligament reconstruction and tendon interposition.J Hand Surg Am2012;37:411-7

[16] Black BE.The cerebral palsied hip.Clin Orthop Relat Res1997;338:42-51

[17] Hansen TB.Equally good fixation of cemented and uncemented cups in total trapeziometacarpal joint prostheses. A randomized clinical RSA study with 2-year follow-up.Acta Orthop2013;84:98-105 PMCID:PMC3584612

[18] Hart R,Šiška V,Štipčák V.Interposition suspension arthroplasty according to Epping versus arthrodesis for trapeziometacarpal osteoarthritis.Eur Surg2006;38:433-8

[19] Kriegs-Au G,Fojtl E,Zachs I.Ligament reconstruction with or without tendon interposition to treat primary thumb carpometacarpal osteoarthritis. Surgical technique.J Bone Joint Surg Am2005;87 Suppl 1:78-85

[20] Marks M,Wehrli M,Schindele S.Trapeziectomy with suspension-interposition arthroplasty for thumb carpometacarpal osteoarthritis: a randomized controlled trial comparing the use of allograft versus flexor carpi radialis tendon.J Hand Surg Am2017;42:978-86

[21] Morais B,Marques N.Trapeziectomy with suture-button suspensionplasty versus ligament reconstruction and tendon interposition: a randomized controlled trial.Hand Surg Rehabil2022;41:59-64

[22] Ritchie JF.A comparison of trapeziectomy via anterior and posterior approaches.J Hand Surg Eur Vol2008;33:137-43

[23] Salem H.Six year outcome excision of the trapezium for trapeziometacarpal joint osteoarthritis: is it improved by ligament reconstruction and temporary Kirschner wire insertion?.J Hand Surg Eur Vol2012;37:211-9

[24] Salibi A,Burke FD.Prospective clinical trial comparing trapezial denervation with trapeziectomy for the surgical treatment of arthritis at the base of the thumb.J Surg Res2019;238:144-51

[25] Sánchez-Flò R,Marcano-Fernández FA,Balcells-Nolla P.Partial versus total trapeziectomy with interposition arthroplasty for trapeziometacarpal osteoarthritis grade II to III Eaton-Littler: a clinical trial.J Hand Surg Glob Online2020;2:133-7 PMCID:PMC8991873

[26] Spekreijse KR,Kedilioglu MA.Trapeziometacarpal arthrodesis or trapeziectomy with ligament reconstruction in primary trapeziometacarpal osteoarthritis: a 5-year follow-up.J Hand Surg Am2016;41:910-6

[27] Spekreijse KR,Kedilioglu MA.The effect of a bone tunnel during ligament reconstruction for trapeziometacarpal osteoarthritis: a 5-year follow-up.J Hand Surg Am2015;40:2214-22

[28] Tägil M.Swanson versus APL arthroplasty in the treatment of osteoarthritis of the trapeziometacarpal joint: a prospective and randomized study in 26 patients.J Hand Surg Br2002;27:452-6

[29] Thorkildsen RD.Trapeziectomy with LRTI or joint replacement for CMC1 arthritis, a randomised controlled trial.J Plast Surg Hand Surg2019;53:361-9

[30] Vermeulen GM,Slijper H.Trapeziometacarpal arthrodesis or trapeziectomy with ligament reconstruction in primary trapeziometacarpal osteoarthritis: a randomized controlled trial.J Bone Joint Surg Am2014;96:726-33

[31] Vermeulen GM,Slijper H,Hovius SE.Comparison of arthroplasties with or without bone tunnel creation for thumb basal joint arthritis: a randomized controlled trial.J Hand Surg Am2014;39:1692-8

[32] Xie Y,Hunter-Smith DJ,Ross R.Aesthetic surgery advice and counseling from artificial intelligence: a rhinoplasty consultation with ChatGPT.Aesthetic Plastic Surgery2023;47:1985-93 PMCID:PMC10581928

[33] Seth I,Xie Y.Evaluating Chatbot efficacy for answering frequently asked questions in plastic surgery: a ChatGPT case study focused on breast augmentation.Aesthet Surg J2023;43:1126-35

[34] Journal CME Questions. J Hand Surg. 2023;48:699. Available from: https://www.sciencedirect.com/science/article/pii/S0363502323002617. [Last accessed on 26 Dec 2024].

[35] Seth I,Xie Y,Rozen WM.Exploring the role of artificial intelligence chatbot on the management of scaphoid fractures.J Hand Surg Eur Vol2023;48:814-8

[36] Seth I,Xie Y.Comparing the efficacy of large language models ChatGPT, BARD, and Bing AI in providing information on rhinoplasty: an observational study.Aesthet Surg J Open Forum2023;5:ojad084 PMCID:PMC10547367

[37] Shridharani SM,Biesman B.Efficacy and safety of tapencarium (RZL-012) in submental fat reduction.Aesthet Surg J2023;43:NP797-806 PMCID:PMC10501747