Robot-Assisted Augmented Reality (AR)-Guided Surgical Navigation for Periacetabular Osteotomy
Abstract
:1. Introduction
- We propose a robot-assisted, AR-guided surgical navigation system for PAO, built on a robot arm and a Microsoft HoloLens 2 headset (Microsoft, Redmond, USA), which is a state-of-the-art (SOTA) OST-HMD. Our system automatically aligns a bone saw with a preoperatively planned osteotomy plane and then provides surgeons with both virtual constraints and AR visual guidance, improving surgical accuracy and safety.
- We propose an optical marker-based AR registration method. Specifically, we control a robot arm to align an optical marker attached to the robot flange with predefined virtual models, collecting point sets in the optical tracker COS and the virtual space COS, respectively. The transformation is then estimated based on paired-point matching.
- Comprehensive experiments were conducted to evaluate both AR registration accuracy and osteotomy accuracy of the proposed system. Experimentally, the proposed AR registration method can accurately align virtual models with the corresponding physical counterparts while the navigation system achieved accurate osteotomy on sheep pelvises.
2. Related Works
2.1. Surgical Navigation in PAO
2.2. Robot Assistance in Osteotomy
2.3. AR Guidance in CAOS
3. Method
3.1. Overview of the Proposed Navigation System
3.2. Preoperative Planning
3.3. Image–Patient Registration
3.4. Bone Saw Calibration
3.5. AR Registration
3.6. Robot-Assisted AR-Guided Osteotomy
3.6.1. Robot Assistance
3.6.2. AR Guidance
4. Experiments and Results
4.1. Tasks and Evaluation Metrics
4.1.1. Evaluation of AR Registration Accuracy
4.1.2. Ablation Study
4.1.3. Evaluation of Time Cost of the Proposed AR Registration
4.1.4. Evaluation of Osteotomy Accuracy
4.2. Experimental Results
4.2.1. Accuracy of AR Registration
4.2.2. Ablation Study
4.2.3. Time Cost of the Proposed AR Registration
4.2.4. Osteotomy Accuracy
5. Discussion and Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
API | application programming interface |
AR | augmented reality |
CAOS | computer-assisted orthopedic surgeries |
COS | coordinate system |
CT | computed tomography |
DDH | developmental dysplasia of the hip |
DRB | dynamic reference base |
EM | electromagnetic |
FoV | field of view |
PAO | periacetabular osteotomy |
MR | magnetic resonance |
mADE | mean absolute distance error |
OST-HMD | optical see-through head-mounted display |
SOTA | state of the art |
3D | three-dimension |
References
- Ahmad, S.S.; Giebel, G.M.; Perka, C.; Meller, S.; Pumberger, M.; Hardt, S.; Stöckle, U.; Konrads, C. Survival of the dysplastic hip after periacetabular osteotomy: A meta-analysis. Hip Int. 2023, 33, 306–312. [Google Scholar] [CrossRef]
- Troelsen, A. Assessment of adult hip dysplasia and the outcome of surgical treatment. Dan. Med. J. 2012, 59, B4450. [Google Scholar] [PubMed]
- Liu, L.; Ecker, T.M.; Siebenrock, K.A.; Zheng, G. Computer assisted planning, simulation and navigation of periacetabular osteotomy. In Proceedings of the International Conference on Medical Imaging and Augmented Reality, Bern, Switzerland, 24–26 August 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 15–26. [Google Scholar]
- Pflugi, S.; Liu, L.; Ecker, T.M.; Schumann, S.; Larissa Cullmann, J.; Siebenrock, K.; Zheng, G. A cost-effective surgical navigation solution for periacetabular osteotomy (PAO) surgery. Int. J. Comput. Assist. Radiol. Surg. 2016, 11, 271–280. [Google Scholar] [CrossRef] [PubMed]
- Grupp, R.B.; Hegeman, R.A.; Murphy, R.J.; Alexander, C.P.; Otake, Y.; McArthur, B.A.; Armand, M.; Taylor, R.H. Pose estimation of periacetabular osteotomy fragments with intraoperative X-ray navigation. IEEE Trans. Biomed. Eng. 2019, 67, 441–452. [Google Scholar] [CrossRef] [PubMed]
- Liu, L.; Siebenrock, K.; Nolte, L.P.; Zheng, G. Computer-assisted planning, simulation, and navigation system for periacetabular osteotomy. In Intelligent Orthopaedics: Artificial Intelligence and Smart Image-Guided Technology for Orthopaedics; Springer: Singapore, 2018; pp. 143–155. [Google Scholar]
- Pflugi, S.; Vasireddy, R.; Lerch, T.; Ecker, T.M.; Tannast, M.; Boemke, N.; Siebenrock, K.; Zheng, G. Augmented marker tracking for peri-acetabular osteotomy surgery. Int. J. Comput. Assist. Radiol. Surg. 2018, 13, 291–304. [Google Scholar] [CrossRef] [PubMed]
- Inaba, Y.; Kobayashi, N.; Ike, H.; Kubota, S.; Saito, T. Computer-assisted rotational acetabular osteotomy for patients with acetabular dysplasia. Clin. Orthop. Surg. 2016, 8, 99. [Google Scholar] [CrossRef] [PubMed]
- Han, Z.; Tian, H.; Han, X.; Wu, J.; Zhang, W.; Li, C.; Qiu, L.; Duan, X.; Tian, W. A respiratory motion prediction method based on LSTM-AE with attention mechanism for spine surgery. Cyborg Bionic Syst. 2024, 5, 0063. [Google Scholar] [CrossRef]
- Fan, Y.; Xu, L.; Liu, S.; Li, J.; Xia, J.; Qin, X.; Li, Y.; Gao, T.; Tang, X. The state-of-the-art and perspectives of laser ablation for tumor treatment. Cyborg Bionic Syst. 2024, 5, 0062. [Google Scholar] [CrossRef]
- Mihalič, R.; Brumat, P.; Trebše, R. Bernese peri-acetabular osteotomy performed with navigation and patient-specific templates is a reproducible and safe procedure. Int. Orthop. 2021, 45, 883–889. [Google Scholar] [CrossRef]
- Mendicino, A.R.; Condino, S.; Carbone, M.; Cutolo, F.; Cattari, N.; Andreani, L.; Parchi, P.D.; Capanna, R.; Ferrari, V. Augmented Reality as a Tool to Guide Patient-Specific Templates Placement in Pelvic Resections. In Proceedings of the 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Glasgow, UK, 11–15 July 2022; pp. 3481–3484. [Google Scholar]
- Sun, M.; Lin, L.; Chen, X.; Xu, C.; Zin, M.A.; Han, W.; Chai, G. Robot-assisted mandibular angle osteotomy using electromagnetic navigation. Ann. Transl. Med. 2021, 9, 567. [Google Scholar] [CrossRef]
- Tian, H.; Duan, X.; Han, Z.; Cui, T.; He, R.; Wen, H.; Li, C. Virtual-fixtures based shared control method for curve-cutting with a reciprocating saw in robot-assisted osteotomy. IEEE Trans. Autom. Sci. Eng. 2023, 21, 1899–1910. [Google Scholar] [CrossRef]
- Shao, L.; Li, X.; Fu, T.; Meng, F.; Zhu, Z.; Zhao, R.; Huo, M.; Xiao, D.; Fan, J.; Lin, Y.; et al. Robot-assisted augmented reality surgical navigation based on optical tracking for mandibular reconstruction surgery. Med. Phys. 2024, 51, 363–377. [Google Scholar] [CrossRef] [PubMed]
- Liebmann, F.; Roner, S.; von Atzigen, M.; Scaramuzza, D.; Sutter, R.; Snedeker, J.; Farshad, M.; Fürnstahl, P. Pedicle screw navigation using surface digitization on the Microsoft HoloLens. Int. J. Comput. Assist. Radiol. Surg. 2019, 14, 1157–1165. [Google Scholar] [CrossRef]
- Hoch, A.; Liebmann, F.; Carrillo, F.; Farshad, M.; Rahm, S.; Zingg, P.O.; Fürnstahl, P. Augmented reality based surgical navigation of the periacetabular osteotomy of Ganz—A pilot cadaveric study. In Proceedings of the International Workshop on Medical and Service Robots, Basel, Switzerland, 8–10 July 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 192–201. [Google Scholar]
- Sun, Q.; Mai, Y.; Yang, R.; Ji, T.; Jiang, X.; Chen, X. Fast and accurate online calibration of optical see-through head-mounted display for AR-based surgical navigation using Microsoft HoloLens. Int. J. Comput. Assist. Radiol. Surg. 2020, 15, 1907–1919. [Google Scholar] [CrossRef] [PubMed]
- Tu, P.; Gao, Y.; Lungu, A.J.; Li, D.; Wang, H.; Chen, X. Augmented reality based navigation for distal interlocking of intramedullary nails utilizing Microsoft HoloLens 2. Comput. Biol. Med. 2021, 133, 104402. [Google Scholar] [CrossRef] [PubMed]
- Tu, P.; Qin, C.; Guo, Y.; Li, D.; Lungu, A.J.; Wang, H.; Chen, X. Ultrasound image guided and mixed reality-based surgical system with real-time soft tissue deformation computing for robotic cervical pedicle screw placement. IEEE Trans. Biomed. Eng. 2022, 69, 2593–2603. [Google Scholar] [CrossRef]
- Tu, P.; Wang, H.; Joskowicz, L.; Chen, X. A multi-view interactive virtual-physical registration method for mixed reality based surgical navigation in pelvic and acetabular fracture fixation. Int. J. Comput. Assist. Radiol. Surg. 2023, 18, 1715–1724. [Google Scholar] [CrossRef]
- Qian, L.; Wu, J.Y.; DiMaio, S.P.; Navab, N.; Kazanzides, P. A review of augmented reality in robotic-assisted surgery. IEEE Trans. Med. Robot. Bionics 2019, 2, 1–16. [Google Scholar] [CrossRef]
- Wang, X.; Guo, S.; Xu, Z.; Zhang, Z.; Sun, Z.; Xu, Y. A Robotic Teleoperation System Enhanced by Augmented Reality for Natural Human–Robot Interaction. Cyborg Bionic Syst. 2024, 5, 0098. [Google Scholar] [CrossRef]
- Condino, S.; Turini, G.; Parchi, P.D.; Viglialoro, R.M.; Piolanti, N.; Gesi, M.; Ferrari, M.; Ferrari, V. How to build a patient-specific hybrid simulator for orthopaedic open surgery: Benefits and limits of mixed-reality using the Microsoft HoloLens. J. Healthc. Eng. 2018, 2018, 5435097. [Google Scholar] [CrossRef]
- Safdari, A.; Ling, X.; Tradewell, M.B.; Kowalewski, T.M.; Sweet, R.M. Practical, non-invasive measurement of urinary catheter insertion forces and motions. Front. Biomed. Devices 2019, 41037, V001T06A016. [Google Scholar]
- Bhagvath, P.V.; Mercier, P.; Hall, A.F. Design and Accuracy Assessment of an Automated Image-Guided Robotic Osteotomy System. IEEE Trans. Med. Robot. Bionics 2023, 6, 96–109. [Google Scholar] [CrossRef]
- Sun, W.; Liu, J.; Zhao, Y.; Zheng, G. A Novel Point Set Registration-Based Hand–Eye Calibration Method for Robot-Assisted Surgery. Sensors 2022, 22, 8446. [Google Scholar] [CrossRef] [PubMed]
- We, L. Marching cubes: A high resolution 3D surface construction algorithm. Comput. Graph. 1987, 21, 7–12. [Google Scholar]
- Sorkine-Hornung, O.; Rabinovich, M. Least-squares rigid motion using svd. Computing 2017, 1, 1–5. [Google Scholar]
- Besl, P.J.; McKay, N.D. Method for registration of 3-D shapes. In Proceedings of the Sensor Fusion IV: Control Paradigms and Data Structures, Boston, MA, USA, 12–15 November 1991; SPIE: Bellingham, DC, USA, 1992; Volume 1611, pp. 586–606. [Google Scholar]
- Yan, B.; Zhang, W.; Cai, L.; Zheng, L.; Bao, K.; Rao, Y.; Yang, L.; Ye, W.; Guan, P.; Yang, W.; et al. Optics-guided robotic system for dental implant surgery. Chin. J. Mech. Eng. 2022, 35, 55. [Google Scholar] [CrossRef]
- Bharatha, A.; Hirose, M.; Hata, N.; Warfield, S.K.; Ferrant, M.; Zou, K.H.; Suarez-Santana, E.; Ruiz-Alzola, J.; D’amico, A.; Cormack, R.A.; et al. Evaluation of three-dimensional finite element-based deformable registration of pre-and intraoperative prostate imaging. Med. Phys. 2001, 28, 2551–2560. [Google Scholar] [CrossRef]
- DeLorenzo, C.; Papademetris, X.; Staib, L.H.; Vives, K.P.; Spencer, D.D.; Duncan, J.S. Volumetric intraoperative brain deformation compensation: Model development and phantom validation. IEEE Trans. Med. Imaging 2012, 31, 1607–1619. [Google Scholar] [CrossRef]
- Chen, X.; Li, Y.; Xu, L.; Sun, Y.; Politis, C.; Jiang, X. A real time image-guided reposition system for the loosed bone graft in orthognathic surgery. Comput. Assist. Surg. 2021, 26, 1–8. [Google Scholar] [CrossRef]
- Broers, H.; Jansing, N. How precise is navigation for minimally invasive surgery? Int. Orthop. 2007, 31, 39–42. [Google Scholar] [CrossRef]
- Ferrari, V.; Carbone, M.; Condino, S.; Cutolo, F. Are augmented reality headsets in surgery a dead end? Expert Rev. Med. Devices 2019, 16, 999–1001. [Google Scholar] [CrossRef] [PubMed]
Method | mADE (mm) | ||
---|---|---|---|
Average | Max | Min | |
Sun et al. [18] | 20.53 | 1.83 | |
Tu et al. [19] | 10.91 | 3.41 | |
Tu et al. [21] | 9.53 | 3.94 | |
Ours | 2.71 | 1.38 |
Strategies | mADE (mm) | ||||
---|---|---|---|---|---|
Controlling
Robot Arm |
Multi-View
Observation | Average | Max | Min | |
FR | 9.55 | 3.66 | |||
SRR | √ | 11.65 | 2.23 | ||
Ours | √ | √ | 2.71 | 1.38 |
Index | 1 | 2 | 3 | 4 | 5 | Average |
---|---|---|---|---|---|---|
Time(s) | 201 | 206 | 190 | 198 | 223 | 203.6 ± 11.0 |
Case | (mm) | (mm) | (°) |
---|---|---|---|
1 | 1.14 | 1.42 | 4.77 |
2 | 0.92 | 1.27 | 3.14 |
3 | 1.30 | 1.63 | 4.79 |
4 | 0.71 | 1.08 | 2.75 |
5 | 0.75 | 1.16 | 3.41 |
Average | 0.96 ± 0.23 | 1.31 ± 0.20 | 3.77 ± 0.85 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ding, H.; Sun, W.; Zheng, G. Robot-Assisted Augmented Reality (AR)-Guided Surgical Navigation for Periacetabular Osteotomy. Sensors 2024, 24, 4754. https://doi.org/10.3390/s24144754
Ding H, Sun W, Zheng G. Robot-Assisted Augmented Reality (AR)-Guided Surgical Navigation for Periacetabular Osteotomy. Sensors. 2024; 24(14):4754. https://doi.org/10.3390/s24144754
Chicago/Turabian StyleDing, Haoyan, Wenyuan Sun, and Guoyan Zheng. 2024. "Robot-Assisted Augmented Reality (AR)-Guided Surgical Navigation for Periacetabular Osteotomy" Sensors 24, no. 14: 4754. https://doi.org/10.3390/s24144754
APA StyleDing, H., Sun, W., & Zheng, G. (2024). Robot-Assisted Augmented Reality (AR)-Guided Surgical Navigation for Periacetabular Osteotomy. Sensors, 24(14), 4754. https://doi.org/10.3390/s24144754