Mixed Reality spatial computing for preoperative surgical planning in orthopedic oncology: review of 72 cases and evolution of clinical implementation

Abstract

Aim: Planning orthopedic tumor surgery requires substantial cognitive effort to interpret 3D plans derived from 2D preoperative images and translate them into the patients’ actual anatomy. Mixed Reality (MR) 3D holograms overlaid on patients may help surgeons visualize surgical steps more intuitively before making skin incisions. This study evaluated the use of MR for preoperative assessment in 72 patients with primary or revision orthopedic oncology conditions, as well as the technical issues encountered during clinical implementation, between July 2021 and November 2025.

Methods: 3D Slicer or MIMICS software was used to generate tumor models and support surgical planning. A proprietary MR platform (versions 1 and 2) was developed to integrate patients’ medical images and 3D models into digital asset bundles, which were then downloaded to the MR headset in the operating room via the hospital’s Wi-Fi network. The surgeon examined each patient preoperatively using the conventional 2D method first, and then applied the MR 3D hologram method.

Results: A Likert-scale questionnaire showed that the MR 3D hologram group outperformed the 2D group across all aspects of spatial awareness of the patient’s pathoanatomy and was viewed as a more effective tool for preoperative planning. Regarding NASA-TLX scores, the overall cognitive workload during preoperative assessment was lower in the MR 3D hologram group. Since December 2024, generating cinematic-rendered 3D models with the upgraded MR software platform (version 2) has taken an average of 61 minutes (49–156). Engineer intervention was needed in 4 of 36 cases (11.1%). All cases were wirelessly accessible and completed an MR assessment. The average time to perform hologram-to-patient registration for the last 26 cases was 2.3 minutes (0.95–5.17).

Conclusions: Our results suggest that MR technology could enhance surgeons’ 3D spatial awareness in various orthopedic tumor surgeries and reduce cognitive load during the translation of surgical plans.

Keywords

Mixed Reality / orthopaedic oncology / sarcoma / holograms / cognitive load / navigation / 3D printing

References

[1] He F, Zhang W, Shen Y, Yu P, Bao Q, Wen J, et al. Effects of resection margins on local recurrence of osteosarcoma in extremity and pelvis: Systematic review and meta-analysis. Int J Surg. 2016; 36:283-92.

[2] Laitinen MK, Kurisunkal VJ, Parry MC, Morris GV, Stevenson JD, Jeys LM. Improving oncological outcomes for pelvic bone sarcomas: Is it possible? Eur J Surg Oncol. 2025; 51:110416.

[3] Wong KC, Kumta SM. Use of Computer Navigation in Orthopedic Oncology. Curr Surg Rep. 2014; 2:47.

[4] Wong KC, Sze KY, Wong IO, Wong CM, Kumta SM. Patient-specific instrument can achieve same accuracy with less resection time than navigation assistance in periacetabular pelvic tumor surgery: a cadaveric study. Int J Comput Assist Radiol Surg. 2015; 11:307-16.

[5] Bosma SE, Wong KC, Paul L, Gerbers JG, Jutte PC. A Cadaveric Comparative Study on the Surgical Accuracy of Freehand, Computer Navigation, and Patient-Specific Instruments in Joint-Preserving Bone Tumor Resections. Sarcoma. 2018; 2018:4065846.

[6] Benady A, Meyer JS, Freidin D, Ran Y, Golden E, Wong KC, et al. A Review of 3D Printing in Orthopedic Oncology. J 3D Print Med. 2022; 6:147-61.

[7] Nandra R, Matharu G, Stevenson J, Parry M, Grimer R, Jeys L. Long-term outcomes after an initial experience of computer-navigated resection of primary pelvic and sacral bone tumours. Bone Jt J. 2019; 101-B:484-90.

[8] Li Z, Li L, Deng Z, Yang Y, Duan F, Zhang Q, et al. Long-term outcomes of computer navigation-assisted resection for primary pelvic sarcomas. Bone Jt J. 2025; 107-B:561-70.

[9] Milgram P, Kishino F. A taxonomy of Mixed Reality visual displays. IEICE Trans Inform Syst. 1994; E77-D:1321-29.

[10] Wong KC, Sun YE, Kumta SM. Review and Future/Potential Application of Mixed Reality Technology in Orthopaedic Oncology. Orthop Res Rev. 2022; 14:169-86.

[11] Rito D, Barbosa RM, da Silva MV, Macedo CS, Alves V, Santos CP. Mixed Reality in orthopaedics: A systematic review and meta-analysis on clinical and technological aspects. Comput Methods Programs Biomed. 2025; 271:109011.

[12] Wong KC, Sun EY, Wong IOL, Kumta SM. Mixed Reality Improves 3D Visualization and Spatial Awareness of Bone Tumors for Surgical Planning in Orthopaedic Oncology: A Proof of Concept Study. Orthop Res Rev. 2023; 15:139-49.

[13] Bronowicki K, Antoniuk-Majchrzak J, Malesza I, Możarowski W, Szymborska A, Pachuta B, et al. An attempt to evaluate the use of Mixed Reality in surgically treated pediatric oncology patients. npj Digit Med. 2025; 8:262.

[14] Ghandour S, Turner L, Rikard B, Sawada M, Fazio Ferraciolli S, Lupp Mota A, et al. Establishing Medical Extended Reality Labs Within Healthcare Institutions: A Nationwide Perspective. J Med Ext Real. 2025; 2:223-9.

[15] Uttal DH, Meadow NG, Tipton E, Hand LL, Alden AR, Warren C, et al. The malleability of spatial skills: A meta-analysis of training studies. Psychol Bull. 2013; 139:352-402.

[16] Benton A, Tranel D. Visuoperceptual, visuospatial, and visuoconstructive disorders. In: Heilman KM, Valenstein E, editors. Clinical Neuropsychology. Washington, DC: American Psychological Association; 1993. pp. 165-213.

[17] Lu L, Wang H, Liu P, Liu R, Zhang J, Xie Y, et al. Applications of Mixed Reality Technology in Orthopedics Surgery: A Pilot Study. Front Bioeng Biotechnol. 2022; 10:740507.

[18] Colligan L, Potts HW, Finn CT, Sinkin RA. Cognitive workload changes for nurses transitioning from a legacy system with paper documentation to a commercial electronic health record. Int J Med Inform. 2015; 84:469-76.

[19] Hermanek P, Wittekind C. Residual tumor (R) classification and prognosis. Semin Surg Oncol. 2006; 10:12-20.

[20] Iwata S, Yonemoto T, Araki A, Ikebe D, Kamoda H, Hagiwara Y, et al. Impact of infiltrative growth on the outcome of patients with undifferentiated pleomorphic sarcoma and myxofibrosarcoma. J Surg Oncol. 2014; 110:707-11.

[21] Imanishi J, Slavin J, Pianta M, Jackett L, Ngan SY, Tanaka T, et al. Tail of Superficial Myxofibrosarcoma and Undifferentiated Pleomorphic Sarcoma After Preoperative Radiotherapy. Anticancer Res. 2016; 36:2339-44.

[22] Park JW, Kim HS, Lee C, Yoo HJ, Yun JY, Han I. Preoperative Factors Associated with Infiltrative Histologic Growth Patterns in Extremity Soft Tissue Sarcoma. Sarcoma. 2017; 2017:5419394.

[23] Lau HW, Wong KC, Chiu WK, Kumta SM. Local Recurrence After Minimally Invasive Curettage For Primary Giant Cell Tumor of Bone With Perioperative Bisphosphonate Is Comparable to Open Curettage: Retrospective Comparison With 9-Year Follow-Up. Arthrosc Sports Med Rehabil. 2021; 3:e1729-36.

[24] Woo AOF, Lau HW, Li MML, Chiu CWK, Wong KC. Is preoperative zoledronate or denosumab associated with lower recurrence for extremity giant cell tumor patients treated with curettage? J Orthop Trauma Rehabil. 2025; 33:65-74.

[25] Bonvalot S, Levy A, Terrier P, Tzanis D, Bellefqih S, Le Cesne A, et al. Primary Extremity Soft Tissue Sarcomas: Does Local Control Impact Survival? Ann Surg Oncol. 2016; 24:194-201.

[26] Chouliaras K, Patel N, Senehi R, Ethun CG, Poultsides G, Grignol V, et al. Impact of resection margin on outcomes in high-grade soft tissue sarcomas of the extremity-A USSC analysis. J Surg Oncol. 2020; 123:479-88.

[27] Pisters PW, Leung DH, Woodruff J, Shi W, Brennan MF. Analysis of prognostic factors in 1,041 patients with localized soft tissue sarcomas of the extremities. J Clin Oncol. 1996; 14:1679-89.

[28] Gronchi A, Lo Vullo S, Colombo C, Collini P, Stacchiotti S, Mariani L, et al. Extremity Soft Tissue Sarcoma in a Series of Patients Treated at a Single Institution: local control directly impacts survival. Ann Surg. 2010; 251:506-11.

[29] Duwelius PJ, Parvizi J, Matsen Ko L. New Technology: Safety, Efficacy, and Learning Curves. Clin Orthop Relat Res. 2014; 472:1080-5.

[30] Farfalli GL, Albergo JI, Ritacco LE, Ayerza MA, Milano FE, Aponte-Tinao LA. What Is the Expected Learning Curve in Computer-assisted Navigation for Bone Tumor Resection? Clin Orthop Relat Res. 2017; 475:668-75.

[31] Grunbeck IA, Kumar RP, Teatini A, Elle OJ, Wiig O. 4D Mixed Reality tool for orthopaedic surgery - a feasibility study. Comput Methods Biomech Biomed Eng: Imaging Vis. 2025; 13:e13.

[32] Javaheri H, Ghamarnejad O, Lukowicz P, Stavrou GA, Karolus J. LLMs Enable Context-Aware Augmented Reality in Surgical Navigation. In: Proceedings of the 2025 ACM Designing Interactive Systems Conference (DIS ‘25). New York, NY: Association for Computing Machinery; 2025. pp. 3205-20.

[33] Catchpole K, Cohen T, Alfred M, Lawton S, Kanji F, Shouhed D, et al. Human Factors Integration in Robotic Surgery. Hum Factors: J Hum Factors Ergon Soc. 2022; 66:683-700.

[34] Beberman R, Courtier J, Straker D. Protecting Patient Privacy: A Suggested Framework for Deploying Extended Reality Technologies. J Med Ext Real. 2025; 2:e2.