Virtual Reality Simulation and Augmented Reality-Guided Surgery for Total Maxillectomy: A Case Report

Abstract

With the improvement in computer graphics and sensors, technologies like virtual reality (VR) and augmented reality (AR) have created new possibilities for developing diagnostic and surgical techniques in the field of surgery. VR and AR are the latest technological modalities that have been integrated into clinical practice and medical education, and are rapidly emerging as powerful tools in the field of maxillofacial surgery. In this report, we describe a case of total maxillectomy and orbital floor reconstruction in a patient with malignant fibrous histiocytoma of the maxilla, with preoperative planning via VR simulation and AR-guided surgery. Future developments in VR and AR technologies will increase their utility and effectiveness in the field of surgery.

1. Introduction

Virtual reality (VR) is a contemporary technology that is being used in medical education and simulations during surgery. It is important for the surgeons to understand the patient’s anatomy and practice accurate surgical skills. This is particularly important in maxillofacial surgery, which greatly influences patients’ appearance and masticatory ability [1]. The first use of VR in the medical field was reported in the 1990s, which involved visualization of complex medical data during surgery and establishing surgical plans preoperatively with the assistance of VR [2]. A VR medical simulation is usually composed of a virtual patient in a medical three-dimensional (3D) model, with a reconstructed pathological lesion, surgical instruments, and tools for a VR interface [3].

With the improvement in computer graphics and sensors, augmented reality (AR) has created new possibilities for developing diagnostics and surgical techniques in maxillofacial surgery [4]. Aschke et al. [5] reported a conceptual description of AR application in a three-dimensional stereoscopic overlay of the operating field. AR can provide the operator with visualization of a preplanned 3D model by overlaying the surgical field in real time during the surgical procedure [6].

Computer-assisted surgery includes surgical planning with simulation software and fabrication of customized surgical guides. However, the process of manufacturing surgical guides is time-consuming and involves additional costs. Furthermore, surgical guides cannot respond to various unexpected situations during surgery. These disadvantages can be overcome by replacing this process with VR simulation and AR-guided surgery [7].

This study describes the application of VR planning and AR-guided surgery for total maxillectomy in a patient with malignant fibrous histiocytoma in the maxilla.

2. Case Presentation

We followed the guidelines of the Helsinki Declaration throughout this study. We obtained approval from the Institutional Review Board of the Chosun University (2-1041055-AB-N-01-2019-08).

A 39-year-old Korean man was referred to the Department of Oral and Maxillofacial Surgery, with a complaint of swelling and pain in the left maxillary molar region and palatal area. There was no significant past medical or dental history. Clinical examination revealed diffuse swelling in the left cheek area with facial asymmetry. Extensive swelling was observed from the left maxillary canine to the first molar extending to the mid-palatal area. The size of the swelling was approximately 50 mm × 50 mm. Clinical examination of the neck showed no cervical lymphadenopathy at the time of diagnosis. Panoramic radiography (ProMax^® 2D S3, Planmeca Inc., Helsinki, Finland) revealed a radiolucent lesion with moderate to defined margins extending from the left maxilla to the maxillary sinus (Figure 1). There was no evidence of significant root resorption.

The Computed-Tomography (CT) equipment used was an Aquilion One CT system (Aquilion ONE, Canon Medical Systems, Otawara, Japan). The scanning conditions were as follows: tube current 250 mA, tube potential 120 kV, scanning time 0.5 s/scan, and slice thickness 0.5 mm. CT images demonstrated an expansile lesion obstructing the entire left maxillary sinus just below the orbital floor. The soft tissue mass extended to the superficial fascia of the face and protruded into the nasal cavity. The lateral and medial walls of the maxillary sinus were damaged (Figure 2).

Incisional biopsy was performed, and the patient was diagnosed with malignant fibrous histiocytoma. Based on the results of the biopsy, total maxillectomy of the left maxilla was planned.

2.1. VR for Surgical Planning

CT images and facial scan data were obtained for the computer-assisted surgery. Facial markers were attached to the patient’s skin before CT and facial scanning. CT and facial scans (Artec^® Space Spider; Artec^® Group; Luxembourg, Luxembourg) were sequentially performed. Using Mimics software (17.0, Materialise, Leuven, Belgium), lesions and normal bone tissue were segmented separately from the CT images. 3D images of normal bone tissue and lesions extracted from CT, and 3D images of skin through the facial scan, were converted into STL files. 3D models were registered according to the facial markers. We developed a VR simulation software using Unity 3D, which could simulate preoperative planning using a head mounted display (VIVE pro, HTC, Taoyuan, Taiwan) with intuition and immersiveness. Using this VR simulation, an osteotomy line was designed considering the anatomical structures, safety margins, and accessibility of the instruments used in the actual surgery. The osteotomy line was displayed in the form of a plane that indicated the cutting lines and directions in consideration of the visibility in the AR viewer developed by us (Figure 3). Preplanned 3D models were exported as STL files. The exported models were imported by the AR viewer. The AR viewer could track the markers automatically and overlay the 3D models with adjustable transparency on the actual patient. In addition, a customized polycaprolactone mesh (T&R Biofab Co., Ltd., Si-heung, Korea) was prepared for the reconstruction of the orbital floor. The polycaprolctone (PCL) orbital mesh is a patient specific implant, which was produced with biocompatible PCL in T&R Biofab’s 3D printing system. For the design of the customized mesh, standard data were established based on the data obtained from the computed tomography (CT) images. Through computer-aided designing, the 3D model of PCL mesh was extracted from the curvature to the orbital wall and floor. PCL orbital mesh size was 36 mm width, 39 mm length, 14 mm height, and 1.15 mm thickness. PCL mesh was fixated in the orbital floor by being sutured to the remained bone.

2.2. AR-guided Surgery and Postoperative Analysis

A modified Weber Ferguson incision was made, and the operating field was exposed. Thereafter, AR-guided surgery was performed using an AR viewer. The AR viewer used a 4k camcorder as an RGB camera to obtain the image of the surgical site and transmit it to a laptop using a USB 3.0 type frame grabber (Elgato 4 k cam link, Elgato, Munich, Germany). The 3D model was augmented using the AR viewer installed on the laptop in real time. The augmented screen was transferred to the operator’s head-mounted display and monitor. A registration marker was attached to the patient’s forehead to track the registered position in real time. In addition, the error between the actual image and the augmented 3d model was minimized by allowing the operator to directly see the actual image and the augmented 3d model and fine-tune the positional relationship. Four osteotomies were performed using the AR viewer in the exact place as planned with the RGB camera and marker system (Figure 4).

The orbital floor was reconstructed with PCL mesh. The surgical defect was packed with Vaseline gauze. Postoperative CT data were obtained, and the differences between the osteotomy line planned for the 3D model and that on the postoperative 3D model were evaluated.

The differences in the frontomaxillary, pterygomaxillary, premaxillary, and zygomaticomaxillary osteotomy lines were 4.99 mm, 2.25 mm, 1.95 mm, and 1.88 mm, respectively. The average difference in the AR-guided system for all four osteotomy lines was 2.77 ± 1.29 mm (Figure 5).

At the time of operation, an alginate impression was taken before the operation was completed, and an arylic resin obturator was made for sealing the surgical defect and delivered on the 3rd day after the operation. The patient was discharged 3 weeks after the operation. The patient was referred to the Department of Hematology after 2 months, and concurrent chemoradiation therapy (CCRT) was initiated with cisplatin, etoposide, and ifosfamide.

3. Discussion

Malignant fibrous histiocytoma is an aggressive high-grade sarcoma [8]. It has been recognized to occur most commonly in soft tissues, especially in torso and one of the extremities [9]. The most frequently observed area is the lower extremity, followed by the upper extremity, and the abdominal area, and the least frequently observed area is the head and neck region. It has been reported that approximately 1–3% of these tumors are present in the head and neck region [10]. The nasal and paranasal cavity is most commonly influenced, and involvement of the maxillary alveolar bone is often observed [11,12]. The present report describes a case of a malignant fibrous histiocytoma affecting the maxillary alveolar bone and maxillary sinus [13].

Depending on the primary interest of base surgery, Kim et al. [14] classified clinical AR studies into three groups: surgical training, planning, and navigation. The use of AR in this report consisted of planning and navigation. Moreover, if digital 3D models of the patient are converted into physical models, they can also be used for surgical training.

Computer-assisted surgery enables simulation surgery and helps the operator to make accurate surgery plans before surgery. In facial contouring surgery, Tsai et al. [15] used a haptic device for surgical simulation to reduce zygomatic bone projection and insert implant into the jaw. Simulation for reducing mandibular angle was also performed using a VR-based system [16]. Woo et al. [17] used computer simulation to create a 3D virtual plan for mandibular reconstruction. Olsson et al. [18] used facilities, including an immersive workstation and 3D glasses for cranio-maxillofacial surgery.

In order to allow the actual operation to be performed according to the preoperative plan, the operator may record a specific length of the surgical site, take an image or model of the surgical plan to the operating room, or prepare a surgical guide before surgery. Recently, surgery using AR has also been conducted experimentally. Fushima and Kobayashi [19] proposed a mixed reality-based system using a dental model and 3D maxillofacial model for maxillofacial surgery. This system synchronizes the movement of the real dental model and virtual 3D patient model. They adjusted the 3D model according to the modification of the dental model in order to plan orthognathic surgery. In this study, AR-based navigation surgery was performed on a real patient, and the results were acceptable.

Three major factors have made AR application possible in the medical field. First, with the breakthrough of computer performance, surgeons have been able to use volume rendering to process imaging data, eventually enabling digital image visualization [6]. Second, user accessibility has significantly improved due to the development of new 3D image processing software [20]. Lastly, advancements in visualization tools (digital monitors), digital recording (4 K camera), and 3D printers have made it possible to produce instruments for surgery intraoperatively [21]. Due to these new AR applications, it has been possible to consider AR-assisted surgeries in the cranio-maxillo-facial region. Currently, surgeons have been able to use hands-free 3D imaging and various AR-guided operation techniques in maxillofacial surgery [6].

We have made several attempts at AR-assisted surgery before this study. We used commercially available AR glasses and attempted to perform AR-guided surgery using a webcam and tablet. However, we were unable to obtain clear, high-resolution surgical images because of the high illumination of the operating room and restricted location of markers. In addition, ensuring that instruments for AR-assisted surgery do not interfere with the functioning of the operator and assistant was also a huge obstacle. For this, a 4k camcorder was installed right under the astral lamp using a self-made cradle, and satisfactory images were obtained. This operation cannot be considered as a perfect AR-guided surgery, because it did not perfectly match the visual field of the operator. However, AR guidance was sufficient for reference at important points in the surgery, such as formulating the osteotomy lines. In particular, the osteotomy lines, which were shown in the form of a plane, were easily recognizable in terms of position and direction during the actual operation. It was also possible to refer to the locations where it was difficult to use surgical guides, such as the pterygomaxillary junction osteotomy line. Accuracy is extremely important in surgical operations using VR/AR devices. In several studies, optical systems have featured precision within 5 mm, which is considered an acceptable difference range for clinical application [20,22,23,24]. Yoshino et al. [25] reported an accuracy of 2.9 ± 1.9 mm using a high-resolution magnetic resonance image for reconstruction and optimal tracking system. Gavaghan et al. [26] demonstrated a portable projection device with an accuracy of 1.3 mm. A few studies proposed that deviations of less than 2.7 mm could be accomplished using a head-mounted display [27,28]. RGB camera and marker system is reported to have an accuracy of 1.9–4.9 mm [29]. In the present study, the operation was performed with an accuracy of 2.77 ± 1.29 mm using an RGB camera and tracking marker system, which was within a reasonable range, similar to other established studies. To date, no clinical research has been conducted for maxillectomy using VR/AR aided surgical planning. Previously, maxillectomy and orbital reconstruction have been performed by computer-assisted surgery, including 3D printing guide or navigation system (

Table 1. Previous studies of surgical interventions for maxillectomy.

Clinical Application	Software	PICO (Populations, Intervention, Comparison, Outcome)
Maxillectomy and orbital reconstruction [30]	CMF 1.4 (Materialize, Belgium) iPlan CMF 3.0 (BrainLab, Germany)	P: Patients with maxillary and orbital defects: 19; benign tumor: 11, malignant tumor: 8, median age: 44 years. I: Surgery under guidance of a navigation system. C: CT scan analysis, globe projection, and orbital volume. O: Average maximum deviation was 2.4 ± 0.2 mm.
Maxillectomy and deep circumflex iliac artery (DCIA) flap reconstruction [31]	3D printing guide system (Aview, Coreline soft, Korea)	P: A 59-year-old male patient with squamous cell carcinoma. I: Total maxillectomy with surgical guide. C: Postoperative CT image was compared. O: Accurate positioning of DCIA flap as planned.
Maxillectomy and fibular reconstruction [32]	Advantage Windows 3D Anaysis Package Version 1.2 with GE scanner	P: A 21-year-old woman with squamous cell carcinoma. I: Surgery under rapid prototyping 3D resin models of total maxillectomy and fibula model. C: 3D CT reconstruction after 1 year. O: Adequate functional and esthetic results.
Maxillectomy and orbital reconstruction [33]	iPlan CMF 3.0 (BrainLab, Germany) ProPlan CMF (Materialise, Belgium)	P: Patients who underwent maxillectomy, including the orbital floor: 10, benign tumor: 4, malignant tumor: 6, mean age: 42.1 years. I: Surgery under computerized navigation system, prefabricated titanium mesh was fitted to the orbital floor defect. C: Postoperative globe projection and orbital volume on the reconstructed and unaffected sides were measured using iPlan CMF. O: Globe projection was 15.91 ± 1.80 mm on the reconstructed side and 16.24 ± 2.24 mm on the unaffected side (P = 0.505). The orbital volume was 26.01 ± 1.28 mL on the reconstructed side and 25.27 ± 1.89 mL on the unaffected side (P = 0.312).

). These surgeries involve additional costs for the fabrication of surgical guides and time-consuming procedures, such as registration and application of surgical guides. However, AR-guided surgery does not make the surgical guide and registration more convenient.

The error in the present study was similar to that of previous studies. However, beginning with this report, AR/VR application in oral and maxillofacial surgeries will increase. Standard guides need a definite reference point, like the bone and teeth of patients, and more dissection is needed to locate a particular point. However, the AR guide used in this case had the advantage of requiring less incision and dissection. For AR-guided surgeries, all images must be restructured by a complex algorithm, with a high-performance computer [34]. Although AR accelerates the surgical procedure, preparing the entire system, including registration and calibration, might take up more time depending on the intricacy of the system [7]. Additionally, the AR guide and surrounding interfaces can interrupt the surgeon’s field of vision and the surgeon can be distracted due to the increased information on the AR platform during operation [35]. Proper recognition of the 3D model and depth can also be a problem [7]. Excessive latency can reduce operation accuracy and lower the clinician’s comfort; therefore, overall system latency should be considered. However, these drawbacks can be overcome with the enhancement of AR technology. Combination of VR and AR will allow for better results in surgical procedures. Compared to conventional computer-assisted surgery with a surgical guide, VR/AR assisted procedure is more economical and convenient for the surgeon. In this report, the accuracy of the AR guide was verified. However, the absolute accuracy of the AR guide could not be assured because only one case was evaluated in this study. Assessment of the accuracy and practicality of the AR guide is necessary through additional studies.

4. Conclusions

Total maxillectomy was performed in a patient with malignant fibrous histiocytoma of the maxilla. The surgery was planned using VR simulation software developed using Unity 3D. AR-guided surgery was performed using a self-developed AR viewer. AR appears to be a powerful tool that may be applied in the field of surgery. In this report, we conducted preoperative planning via VR simulation and AR-guided surgery for total maxillectomy and orbital floor reconstruction. With the advancement in the field of VR and AR technologies, their utility in the field of surgery will further increase.