Surgical gestures—An emerging field for surgical assessment and training

Abstract

As surgical training shifts from a traditional method to a more standardized approach, objective analysis and assessment of surgeon performance has become a key focus. Surgical gestures, defined as the smallest independent units of instrument-tissue interaction, offer a quantifiable way to analyze surgical performance. Standardizing the terminology for describing surgical gestures can enhance communication during surgical training in the operating room. More importantly, surgical gesture usage has been linked to surgeon expertise and has been shown to be associated with patient outcomes. This review examines current classification systems for surgical gestures in dissection and suturing tasks, across open, laparoscopic, and robotic procedures, which serve as an armamentarium for training surgeons. It also explores how surgical gestures can complement conventional surgical assessment tools. Finally, it reviews current artificial intelligent models on gesture recognition and automation, envisions a future where gesture recognition forms the foundation for intelligent assistance in the operating room.

Keywords

artificial intelligence / clinical research / robotic surgery / urology

References

[1] Nasca TJ , Philibert I , Brigham T , Flynn TC . The Next GME Accreditation System-Rationale and Benefits. N Engl J Med. 2012; 366: 1051- 6.

[2] Cameron JL . William Stewart Halsted . Our surgical heritage. Ann Surg. 1997; 225: 445- 58.

[3] Ma R , Reddy S , Vanstrum EB , Hung AJ . Innovations in urologic surgical training. Curr Urol Rep. 2021; 22: 26.

[4] Goh AC , Goldfarb DW , Sander JC , Miles BJ , Dunkin BJ . Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol. 2012; 187: 247- 52.

[5] Martin JA , Regehr G , Reznick R , Macrae H , Murnaghan J , Hutchison C , et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997; 84: 273- 8.

[6] Raza SJ , Field E , Jay C , et al. Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: development and validation of the robotic anastomosis competency evaluation. Urology. 2015; 85: 27- 32.

[7] Haque TF , Hui A , You J , Ma R , Nguyen JH , Lei X , et al. An assessment tool to provide targeted feedback to robotic surgical trainees: development and validation of the End-to-End Assessment of Suturing Expertise (EASE). Urol Pract. 2022; 9: 532- 9.

[8] Vanstrum EB , Ma R , Maya-Silva J , Sanford D , Nguyen JH , Lei X , et al. Development and validation of an objective scoring tool to evaluate surgical dissection: Dissection Assessment for Robotic Technique (DART). Urology Practice. 2021; 8: 596- 604.

[9] Conroy LM , Slovacek H , Blum KA , Canfield SE , Mann P . Current landscape of surgical assessment models in urology residency training. J Urol. 2023; 209: 474- 84.

[10] Boal MWE , Anastasiou D , Tesfai F , et al. Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg. 2024; 111: znad331.

[11] Hung AJ , Chen J , Jarc A , Hatcher D , Djaladat H , Gill IS . Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol. 2018; 199: 296- 304.

[12] Hung AJ , Ma R , Cen S , Nguyen JH , Lei X , Wagner C . Surgeon automated performance metrics as predictors of early urinary continence recovery after robotic radical prostatectomy-a prospective bi-institutional study. Eur Urol Open Sci. 2021; 27: 65- 72.

[13] Chadebecq F , Lovat LB , Stoyanov D . Artificial intelligence and automation in endoscopy and surgery. Nat Rev Gastroenterol Hepatol. 2023; 20: 171- 82.

[14] Ma R , Vanstrum EB , Nguyen JH , Chen A , Chen J , Hung AJ . A novel dissection gesture classification to characterize robotic dissection technique for renal hilar dissection. J Urol. 2021; 205: 271- 5.

[15] Hung AJ , Bottyan T , Clifford TG , Serang S , Nakhoda ZK , Shah SH , et al. Structured learning for robotic surgery utilizing a proficiency score: a pilot study. World J Urol. 2017; 35: 27- 34.

[16] Ma R , Ramaswamy A , Xu J , Trinh L , Kiyasseh D , Chu TN , et al. Surgical gestures as a method to quantify surgical performance and predict patient outcomes. npj Digit Med. 2022; 5: 187.

[17] Chen Z , An J , Wu S , Cheng K , You J , Liu J , et al. Surgesture: a novel instrument based on surgical actions for objective skill assessment. Surg Endosc. 2022; 36: 6113- 21.

[18] Olsen RG , Svendsen MBS , Tolsgaard MG , Konge L , Røder A , Bjerrum F . Surgical gestures can be used to assess surgical competence in robot-assisted surgery: a validity investigating study of simulated RARP. J Robot Surg. 2024; 18: 47.

[19] Cui Z , Ma R , Yang CH , Malpani A , Chu TN , Ghazi A , et al. Capturing relationships between suturing sub-skills to improve automatic suturing assessment. npj Digit Med. 2024; 7: 152.

[20] van Amsterdam B , Clarkson MJ , Stoyanov D . Gesture recognition in robotic surgery: a review. IEEE Trans Biomed Eng. 2021; 68: 2021- 35.

[21] Chen J , Oh PJ , Cheng N , Shah A , Montez J , Jarc A , et al. Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial. J Urol. 2018; 200: 895- 902.

[22] Gao Y , Vedula SS , Reiley CE , et al. JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: Modeling and Monitoring of Computer Assisted Interventions (M2CAI)-MICCAI Workshop, 2014.

[23] Ahmidi N , Tao L , Sefati S , Gao Y , Lea C , Haro BB , et al. A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng. 2017; 64: 2025- 41.

[24] van Amsterdam B , Funke I , Edwards E , Speidel S , Collins J , Sridhar A , et al. Gesture recognition in robotic surgery with multimodal attention. IEEE Trans Med Imaging. 2022; 41: 1677- 87.

[25] Shafiei SB , Durrani M , Jing Z , et al. Surgical hand gesture recognition utilizing electroencephalogram as input to the machine learning and network neuroscience algorithms. Sensors 2021. 2021; 21: 1733.

[26] Kiyasseh D , Ma R , Haque TF , Miles BJ , Wagner C , Donoho DA , et al. A vision transformer for decoding surgeon activity from surgical videos. Nat Biomed Eng. 2023; 7: 780- 96.

[27] Ma R , Wang J , Rich J , Huang SL , Otiato MX , Dalieh I , et al. MP07-13 contextualized surgical gestures in nerve-spare reveal heterogeneity in techniques and predicts erectile function recovery. J Urol. 2024; 211: e109.

[28] Nwoye CI , Gonzalez C , Yu T , et al. Recognition of instrument-tissue interactions in endoscopic videos via action triplets. In: Martel AL, Abolmaesumi P, Stoyanov D, et al., editors. Medical Image Computing and Computer Assisted Intervention-MICCAI 2020. Lecture Notes in Computer Science. Cham: Springer International Publishing; 2020. p. 364- 74.

[29] Nwoye CI , Yu T , Gonzalez C , Seeliger B , Mascagni P , Mutter D , et al. Rendezvous: attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Med Image Anal. 2022; 78: 102433.

[30] Nwoye CI , Alapatt D , Yu T , Vardazaryan A , Xia F , Zhao Z , et al. CholecTriplet2021: A benchmark challenge for surgical action triplet recognition. Med Image Anal. 2023; 86: 102803.

[31] Nwoye CI , Yu T , Sharma S , Murali A , Alapatt D , Vardazaryan A , et al. CholecTriplet2022: Show me a tool and tell me the triplet—an endoscopic vision challenge for surgical action triplet detection. Med Image Anal. 2023; 89: 102888.

[32] Sharma S , Nwoye CI , Mutter D , et al. Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In: Greenspan H, Madabhushi A, Mousavi P, et al., editors. Medical image computing and computer assisted intervention-MICCAI 2023. Cham: Springer Nature Switzerland; J Robot Surg. 2023. p. 505- 14.

[33] Inouye DA , Ma R , Nguyen JH , Laca J , Kocielnik R , Anandkumar A , et al. Assessing the efficacy of dissection gestures in robotic surgery. J Robot Surg. 2022; 17: 597- 603.

[34] Dergachyova O , Bouget D , Huaulmé A , Morandi X , Jannin P . Automatic data-driven real-time segmentation and recognition of surgical workflow. Int J Comput Assist Radiol Surg. 2016; 11: 1081- 9.

[35] Lalys F , Jannin P . Surgical process modelling: a review. Int J Comput Assist Radiol Surg. 2014; 9: 495- 511.

[36] Oleari E , Leporini A , Trojaniello D , et al. Enhancing Surgical Process Modeling for Artificial Intelligence development in robotics: the SARAS case study for Minimally Invasive Procedures. In: 2019 13th International Symposium on Medical Information and Communication Technology (ISMICT). Oslo, Norway, 2019. p. 1- 6.

RIGHTS & PERMISSIONS

The Author(s). UroPrecision published by John Wiley & Sons Australia, Ltd on behalf of Higher Education Press.