The preliminary stage in developing an artificial intelligence algorithm: a study of the inter- and intra-individual variability of phase annotations in internal fixation of distal radius fracture videos

Abstract

Aim: As a preliminary stage in the development of an artificial intelligence (AI) algorithm for surgery, this work aimed to study the inter- and intra-individual variability of phase annotations in videos of minimally invasive plate osteosynthesis of distal radius fractures (MIPO). The main hypothesis was that the inter-individual variability was almost perfect if Cohen's kappa coefficient (k) was ≥ 81% overall; the secondary hypothesis was that the intra-individual variability was almost perfect if the F₁-score (F₁) was ≥ 81%.

Methods: The material comprised 9 annotators and three annotated MIPO videos with 5 phases and 4 sub-phases. Each video was presented 3 times to each annotator. The method involved analysing the inter-individual variability of annotations by computing k and F₁ from a reference annotator. The intra-individual variability of annotations was analysed by computing F₁.

Results: Annotation anomalies were noticed: either absences or differences in phase and sub-phase annotations. Regarding the inter-individual variability, an almost perfect agreement between annotators was observed because k ≥ 81% for the three videos. Regarding the intra-individual variability, F₁ ≥ 81% for most phases and sub-phases with the nine annotators.

Conclusion: The homogeneity of annotations must be as high as possible to develop an AI algorithm in surgery. Therefore, it is necessary to identify the least efficient annotators (measurement of the intra-individual variability) to provide them with individual training and a personalised annotation rhythm. It is also important to optimise the definition of the phases, improve the annotation protocol and choose suitable training videos.

Keywords

Algorithm / annotation / phase detection / distal radius fracture / artificial intelligence / inter-individual variability / intra-individual variability / surgical video

References

[1] Twinanda AP,Mutter D,de Mathelin M.EndoNet: a deep architecture for recognition tasks on laparoscopic videos.IEEE Trans Med Imaging2017;36:86-97

[2] Hashimoto DA,Witkowski ER.Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy.Ann Surg2019;270:414-21 PMCID:PMC7216040

[3] Kitaguchi D,Matsuzaki H.Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach.Surg Endosc2020;34:4924-31

[4] Ward TM,Ban Y.Automated operative phase identification in peroral endoscopic myotomy.Surg Endosc2021;35:4008-15 PMCID:PMC7854950

[5] Graëff C,Lampert T.Preliminary stage in the development of an artificial intelligence algorithm: variations between 100 surgeons in phase annotation in a video of internal fixation of distal radius fracture. Orthop Traumatol Surg Res 2023. In Press

[6] Tang JB.Why and how to report surgeons’ levels of expertise.J Hand Surg Eur Vol2016;41:365-6

[7] Liverneaux PA.The minimally invasive approach for distal radius fractures and malunions.J Hand Surg Eur Vol2018;43:121-30

[8] Viera AJ.Understanding interobserver agreement: the kappa statistic.Fam Med2005;37:360-3Available from: https://pubmed.ncbi.nlm.nih.gov/15883903/. [Last accessed on 30 Jun 2023]

[9] Bajpai S,Chaturvedi HK.Evaluation of inter-rater agreement and inter-rater reliability for observational data: an overview of concepts and methods.J Indian Acad Appl Psychol2015;41:20-7Available from: https://www.researchgate.net/publication/273451591_Evaluation_of_Inter. [Last accessed on 30 Jun 2023]

[10] Hripcsak G.Agreement, the f-measure, and reliability in information retrieval.J Am Med Inform Assoc2005;12:296-8 PMCID:PMC1090460

[11] Lavanchy JL,Kassem H,Mutter D.Proposal and multicentric validation of a laparoscopic Roux-en-Y gastric bypass surgery ontology.Surg Endosc2023;37:2070-7 PMCID:PMC10017621

[12] Hallgren KA.Computing inter-rater reliability for observational data: an overview and tutorial.Tutor Quant Methods Psychol2012;8:23-34 PMCID:PMC3402032

[13] Pryor AD,Jones A,Pugh C.An American board of surgery pilot of video assessment of surgeon technical performance in surgery.Ann Surg2023;277:591-5

[14] Vondrick C,Patterson D.Efficiently scaling up video annotation with crowdsourced marketplaces. In: Computer Vision - ECCV 2010. Berlin: Springer Berlin Heidelberg; 2010. p. 610-23.