Augmented Reality in Surgical Navigation: A Review of Evaluation and Validation Metrics

Abstract

Pre-operative imaging has been used earlier to guide traditional surgical navigation systems. There has been a lot of effort in the last decade to integrate augmented reality into the operating room to help surgeons intra-operatively. An augmented reality (AR) based navigation system provides a clear three-dimensional picture of the interested areas over the patient to aid surgical navigation and operations, which is a promising approach. The goal of this study is to review the application of AR technology in various fields of surgery and how the technology is used for its performance in each field. Assessment of the available AR assisted navigation systems being used for surgery is reviewed in this paper. Furthermore, a discussion about the required evaluation and validation metric for these systems is also presented. The paper comprehensively reviews the literature since the year 2008 for providing relevant information on applying the AR technology for training, planning and surgical navigation. It also describes the limitations which need to be addressed before one can completely rely on this technology for surgery. Thus, additional research is desirable in this emerging field, particularly to evaluate and validate the use of AR technology for surgical navigation.

1. Introduction

Surgical techniques utilized in different parts of the world have advanced technologically in recent decades. To give surgeons reference information, traditional surgical visualization systems used ultrasound, magnetic resonance imaging (MRI), or patient pre-operative computed tomography (CT) and other medical imaging [1]. However, the surgeon should integrate the two-dimensional (2D) image with three-dimensional (3D) space throughout the surgery, resulting in a mismatch between pre-operative and intra-operative information. During the surgery, with a standard surgical visualization system, the practitioner needs to frequently switch his or her line of sight between the surgical scene and the auxiliary display, extending the operation duration. At the same time, surgical navigation is not achieved, which allows surgeons to precisely track as well as project the instrument position during surgery. Surgical targets and instruments are frequently concealed in other anatomical structures. As a result of these factors, surgical navigation accuracy falls short of expectations.

There has been significant progress in the surgeon’s search for safer, less intrusive, and more affordable techniques and the surgical community is becoming more interested in alternative, yet intuitive intra-operative visual guidance systems [2]. The disadvantages of the above mentioned surgical systems can be efficiently addressed by using an augmented reality visualization and navigation system [3,4,5]. Augmented reality (AR) based navigation systems were developed in response to a growing demand for visual feedback in the operating room [6]. Augmented reality is a new technology that works in a similar way as virtual reality (VR) [7]. AR technology is utilized to build 3D images from CT and/or MRI data and to overlay virtual images over a view of the surgical field. AR technology, in general, consists of 3D reconstruction, display, registration, and tracking techniques, and it has been recently used in surgical interventions. Consequently, AR technology can combine the pre-operative model with the intra-operative scenario, giving clinicians real-time information and surgical guidance [8].

Until recently, AR navigation technology has been widely employed in various fields of surgery, including neurosurgery [9,10,11,12,13], orthopedic and spine surgery [14,15,16], laparoscopic surgery [17,18], hepatobiliary surgery [19,20,21,22,23,24], oral surgery [25,26], etc. [27,28,29]. The goal to do brain surgery as minimally invasive as feasible may be seen throughout the entire history of neurosurgery. The reason for this is that neurosurgery is the practice of performing surgery on or within an organ that contains many sensitive regions that have a direct impact on a patient’s mental and physical well-being. As a result, neurosurgeons frequently accept new technology before others do, with the hope that it may reduce surgical risks and improve patient outcomes. Similarly, researchers have explored using AR technology for image guided surgery in various other fields to overcome surgical complexities by enhancing the operating room (OR) with virtual content.

The application of AR systems in different surgical fields and their performance is described in the next section. In Section 3, the evaluation pertaining to surgical techniques used in AR are described and some validation techniques have been suggested. The last section discusses the limitations of this technology in the surgical field and how the AR navigation systems can be improved so that it can prove to be beneficial for the society in the near future. For this study, many recent papers have been reviewed especially concentrating on articles published after 2008, since the adoption of AR technology began to rise in many surgical fields around that time. Our search criteria involved mainly the following keywords (but not limited to): augmented reality, evaluation, assessment, validation, surgery, etc. at the e-resources platform available within our university’s research collection. Figure 1 shows the distribution of the chosen papers taken into consideration for this study by year. The last search was conducted in May 2022 which explains the drop from 2021, although the dotted curve showing the trend-line over the years shows an exponential growth.

2. State of the Art AR Based Surgical Systems

Many areas of medical imaging, from simulation to training and therapy, are significantly benefiting from the use of mixed, augmented, and virtual reality technologies [30,31,32]. While VR is primarily used for training, AR may be used for everything from training to planning and navigation in the surgery room [33]. Most research compared user comprehension, which was assessed using Likert scale surveys, to the usability of the AR and VR system [34,35,36,37,38]. Overall, the findings imply that participants found AR/VR systems simple to comprehend. Some studies provided a thorough analysis of usability by evaluating particular AR/VR system tools and contrasting the selected interface with more conventional interfaces such as the keyboard and mouse [39]. Most users preferred the interface of the AR/VR system. Numerous studies also mentioned generic characteristics that improve usefulness in the learning environment, such as system mobility and adaptability. Finally, the ability of AR/VR technologies to provide a seamless transition between reality and the augmented or virtual world was a critical component of their usability in anatomy learning [40,41,42].

The beauty of augmented reality is its potential to show an easily recognized and simplified anatomy during surgery [43]. This section describes various fields in surgery where AR technology has been applied more often. Figure 2 shows the breakdown of the papers’ percentage contributions from the various surgical specialties under discussion. The level of flexibility, functionality, and work-flow that each surgical field, each institution, and each surgeon requires from their navigation systems varies. Every field utilizes different methods for evaluating their setup and the overall technology as described below.

2.1. Neurosurgery

The first surgical field to employ navigation and successfully incorporate it into regular practice was neurosurgery. In the last decade, a growing number of scientific and commercial systems for neurosurgical applications have emerged [44,45]. The visual perception and human–computer interface features of computer graphics for supplementing the display in neurosurgery are the main focus here [10]. AR has already been evaluated for the training of tumor resection planning in terms of perceptual and spatial reasoning ability [46]. Their technology was specifically created and tested with human aspects in mind, with the purpose of reducing cognitive stress. In comparison to traditional planning scenarios, their proposed method significantly increased novice performance, regardless of the sensorimotor tasks performed (as shown in Figure 3 and Figure 4). Furthermore, when practitioners used the technology, they were able to complete clinically relevant activities in significantly less time. This showed that augmented reality systems can help residents gain the spatial reasoning abilities they require to plan neurosurgical treatments and enhance patient outcomes.

Augmented reality systems represent a significant advancement over current neuro-navigation systems. In particular, the wide range of technical implementations gives the neurosurgeon viable options for various operations (generally neuro-oncological and neuro-vascular), treatment methods (endo-vascular, endo-nasal, open), and various stages of the same operation (microscopic and macroscopic part). Although no prospective randomized research has been published, available literature [9] confirms that this technology in neurosurgery is a reliable, versatile, and a promising tool. In the work presented by [47], a target registration error of 2.5 mm was achieved using augmented reality neuro-navigation (data from 5 publications), with no significant difference (2.6 mm) when compared to conventional infrared neuro-navigation systems (data from 23 publications).

According to the review of 12 publications on intra-operative medical applications in [48], the injection of 3D images with AR enables successful image integration in vascular, oncological, and other lesions without the need to look away from the surgical field which improves safety, surgical experience, and clinical outcome. According to the results of clinical trials presented in [44], the augmented reality system proved to be accurate and reliable for intra-operative image projection to the head, skull, and brain surface, with a mean registration time of 3.8 min and a mean projection error of 0.8 ± 0.25 mm. Based on the evaluation completed in [45], better visualization techniques which differentiate between arteries and veins and ascertain the absolute depth of a vessel of interest are required for more complicated anomalies such as arteriovenous malformations (AVM) and arteriovenous fistulae (AVF); however, AR is favourable technology in neuro-vascular surgery. To conclude, there is no universal method for reporting navigation accuracy in augmented reality neuro-navigation; comparative studies, however, still need to be assessed to identify its effectiveness [49].

2.2. Orthopedic and Spine Surgery

The increased interest in augmented reality in orthopedics and trauma is unsurprising considering that surgical techniques in orthopedics frequently involve visual data such as medical pictures gathered both pre- and intra-operatively. Additionally, mechanical stages such as osteotomies, screw or implant insertions, and the correction of deformities are routinely used during orthopedic surgery and can be witnessed in AR environments [50]. Spinal disorders such as degenerative ailments, deformities, trauma/spinal injuries, and spinal tumors are managed, assessed, and treated by the spine surgery. In the past ten years, minimally invasive spine surgery has made significant advancements and is now being utilized more frequently to address challenging situations [51].

Pedicle screw implantation (a distinctly difficult procedure) in spine surgery can offer a firm fixation to the spine, preventing harm when the back extends, bends, or rotates excessively [52]. Usually, a screw that is incorrectly placed can cause neurologic and vascular problems; therefore, a navigation system is employed to assist with correct positioning [53]. During surgery, however, the surgical targets and surgical devices such as pedicle screws may be concealed in other tissues. To address these issues, research on a 3D integral videography (IV) AR system [15] for pedicle screw placement (with ultrasound-assisted registration) has been proposed [54]. Figure 5 shows a patient-specific, 3D spine model with a screw inserted at the entry point, a 2D axial image of the screw(s), and a 3D translucent image of the screw(s). To eliminate the effect of extracting anatomical markers on the surface and limit radiation exposure, ultrasound is employed to accomplish registration between the pre-operative medical imaging and the patient. Experiments show that the suggested system has adequate targeting accuracy, and that both the patient and the surgeon are exposed to less radiation.

According to a 12-question survey conducted globally in [16], spine physicians recognise the usefulness of computer assisted surgery, but current technologies fall short of their expectations in terms of usability and integration into the surgical work flow. Speed, accuracy, and urgency are the key in spine or orthopedic treatments. For instance, the majority of orthopedic or spinal surgeries require anesthesia before the surgery because of the severity of pain and other procedural requirements. Thus, before the effect of anesthesia is over, the procedure has to be finished. The post Deep Venous Thrombosis (DVT) stage is so acute that the patient needs urgent attention. Thus, the computer assisted models should be affordable in terms of user friendliness, computationally inexpensive, and fast so that they can be used frequently. Even after reaching the state, where supercomputer is available (although not available abundantly), appropriate visualization still remains a problem, which the clinicians rank the most among others. Thus, in orthopedic and spine surgery, augmented reality has the potential and capability to save time, reduce radiation and risk, and improve precision.

2.3. Laparoscopic Surgery

Laparoscopy, in which physicians try surgical procedures through small incisions, is one of the most popular minimally invasive surgical treatments because of the development of video technology. However, despite the fact that a laparoscopic method is frequently favored over an open surgical approach due to the much lower patient morbidity and shortened recovery time [55], even for expert surgeons, it is highly difficult physically and cognitively. This is partly because of constraints in the typical arrangement, such as restricted flexibility in manipulation and poor depth perception in 2D displays, and partly because the surgeons’ perception is dislocated from the action site. However, for many laparoscopic procedures, whether it be reconstruction of a bile duct while avoiding critical blood vessels or resection of a tumor located deep within an organ parenchyma, surgeons need to see hidden surgical targets. Keyhole techniques restrict the surgeon’s 3D perception within the human body, despite the fact that minimally invasive surgery has several benefits [56,57]. For instance, the surgeon’s visual and tactile cues may be reduced by the restricted visual field of endoscopes and laparoscopes as well as the absence of haptic input.

Augmented reality is a viable contender in the field of laparoscopic imaging [58,59,60,61]. Given the standard endoscope or laparoscope’s small field of vision, it makes perfect sense to supplement the video image with a model of the organ being inspected [62,63]. This can include a depiction for internal organ features that are hidden in the typical laparoscopic picture or an extended surface view of the organ [64,65,66]. Many technical elements of correct registration and visualization of pre-operative imagery with the laparoscopic view, including the image registration, segmentation, calibration, tracking, and intra-operative imaging with MRI, CT, and ultrasound, have been addressed in [17,67]. Recent advancements have revealed promising findings, suggesting that the proposed method requires substantially less cognitive and physical effort than the current 2D ultrasound imaging method utilized in the operating room [68].

Figure 6 provides an example of visualization during laparoscopy through navigation system monitors. A thorough overview of the state of the art as well as the fundamentals of AR in laparoscopy is provided in [18], which makes it very useful. Several types of data, rendering, and display are accessible depending on the augmentation desired. A dependable technical foundation for laparoscopic liver resection is provided by the cutting-edge image navigation technology, as presented in [61]. Although the fundamental challenge facing laparoscopic AR is now its accuracy, the lack of accuracy necessitates the use of validation methods in surgical AR [1].

2.4. Oral Surgery

Oral surgery normally involves surgeries on the jaws and the teeth to alter the dentition. Drilling, cutting, fixing, resection, and implantation are the most common oral surgical procedures [69,70]. The tight space and the fact that perhaps the surgical targets may be covered, limit operations in most circumstances. To ensure surgical safety, surgeons must practice avoiding injury to the surrounding important structures. Dental surgery still experiences imperfect hand-eye coordination and a loss of 3D information in surgical navigation, despite the introduction of computer assisted oral surgery.

Using an IV based 3D image overlay system and stereo tracking, Tran and Wang created a 3D augmented reality based navigation system with automatic marker-free registration of image for oral surgery [71,72]. A stereo camera for tracking patients and equipment, a real-time patient’s 3D image registration method, an IV camera registration method, and an optical see-through device are all part of the proposed system [8]. The surgeon can use the AR information to intuitively examine hidden structures as well as the surgical tools. Figure 7 shows the AR model overlay with surgical scene and the surgical instrument in 3D, with an overlay showing the tool’s location in pink and nerve channel in blue.

Experiments were conducted to confirm the practicality, and the suggested system’s overlay error was less than 1 mm. Surgeons can obtain intuitive depth information and then treat while using the IV based 3D AR system. Another knowledge-based strategy for a contextual intra-operative support system is presented in [73], where two phantom experiments were performed examining the medical usability, accuracy, and recognition rate. In conclusion, the system demonstrated its ability to accurately identify circumstances and give beneficial visualizations in real-time; however, issues with low accuracy continue to exist. Future research will concentrate on increasing the accuracy and recognition rate as well as assessing the concepts on complex procedures.

To summarize,

Table 1. Table summarizing the evaluation and validation performed in various fields.

Surgical Field	Reference	Evaluation	Objective	Validation
Neuro Surgery	Meola et al., 2017 [9]	✓	Clinical applications	×
	Si et al., 2018 [11]	✓	Training & guidance	×
	Lee et al., 2018 [12]	✓	Medical imaging	×
	Shamir et al., 2011 [33]	✓	Pre-operative planning	×
	Cabrilo et al., 2014 [43]	✓	Surgery on patients	×
	Tabrizi et al., 2015 [44]	✓	Surgery on patients	×
	Kersten et al., 2015 [45]	✓	Surgery on patients	×
	Abhari et al., 2014 [46]	✓	Pre-operative planning	×
	Fick et al., 2021 [47]	✓	Clinical applications	×
	López et al., 2019 [48]	✓	Clinical applications	×
	Morineau et al., 2013 [74]	✓	Medical imaging	×
	Mikhail et al., 2019 [75]	✓	Clinical applications	×
Orthopedic & Spine Surgery	Zhang et al., 2016 [15]	✓	Medical imaging	×
	Härtl et al., 2013 [16]	✓	Surgical navigation	×
	Jud et al. 2020 [50]	✓	Clinical applications	×
	Ma et al., 2017 [54]	✓	Pre-clinical trials	×
	Andress et al. 2020 [76]	✓	Training & guidance	×
Laparoscopic Surgery	Nicolau et al., 2011 [17]	✓	Clinical testing	×
	Bernhardt et al., 2017 [18]	✓	Clinical testing	×
	Tsutsumi et al., 2013 [19]	✓	Surgery on patients	×
	Modrzejewski et al., 2019 [59]	✓	Registration testing	×
	Pelanis et al., 2021 [60]	✓	Surgical navigation	×
	Zhang et al., 2021 [61]	✓	Surgery on patients	×
	Teatini et al., 2019 [62]	✓	Pre-clinical trials	×
	Liu et al., 2020 [63]	✓	Pre-clinical trials	×
	Shekhar et al., 2010 [64]	✓	Pre-clinical trials	×
	Luo et al., 2020 [65]	✓	Pre-clinical trials	✓
	Schneider et al., 2021 [66]	✓	Clinical applications	×
	Bernhardt et al., 2016 [67]	✓	Pre-clinical trials	×
	Jayarathne et al., 2017 [68]	✓	Pre-clinical trials	×
Oral Surgery	Wang et al., 2017 [25]	✓	Clinical applications	✓
	Jiang et al. 2019 [26]	✓	Medical imaging	×
	Casap et al., 2011 [69]	✓	Training & guidance	×
	Yamaguchi et al., 2009 [70]	✓	Medical imaging	×
	Tran et al., 2011 [71]	✓	Medical imaging	×
	Wang et al., 2014 [72]	✓	Medical imaging	×
	Katić et al., 2010 [73]	✓	Medical imaging	×
	Wang et al., 2013 [77]	✓	Medical imaging	×

shows the publication list where evaluation and validation studies have been performed using AR technology. The kind of evaluation being conducted varied from testing the AR system for medical imaging, training and guidance, pre-operative planning and surgical navigation. The use of AR technology for pre-clinical trials, various clinical testing and applications, and surgery on patients is also evaluated. This table makes it very evident that the majority of the applications lack a proper validation on using AR surgical technology.

3. Evaluation and Validation of AR Surgical Technology

The clinical practice of simulation surgery utilizing 3D images has recently been accomplished, and AR is an effective tool in conducting surgical procedures, due to advancement of diagnostic imaging and creative computer programs [78]. In recent decades, surgery procedures aiming at minimizing invasiveness have grown in popularity [79]. AR is a technique that adds computer generated information to the perception of reality. When surgeons make decisions about anatomical structures and associated environment, they use their perceptual and spatial thinking skills. It is critical to develop methods and approaches to operationalize objective metrics of performance in tasks that involve the perception of and spatial reasoning regarding anatomical structures in order to study the facilitation offered by AR interactive displays of 2D and 3D, volumetric, biomedical visualization [80]. The driving force behind AR’s development was the need for an optimal head tracking system that continuously and perfectly tracks all subtle changes of the surgical area and transparently modifies the baseline registration all throughout the surgical intervention [81,82,83]. The display device must fit seamlessly into the physician’s workflow because AR visualization is supposed to make the physician’s task easier. As a result, the optimal presenting technique is one that blends in with the surroundings while delivering the necessary details.

The crucial point is that the human visual system responds to the resultant surface’s high frequency components, forming a sparse 3D picture of the surface. The buried virtual object no longer seems to be layered on top of a solid-looking surface, and the buried object’s relative depth perception is restored. The visual clarity achieved is suitable for the high AR depth perception, according to the experimental results using both phantom and in vivo data [84,85]. The goal of AR visualization is to be able to provide a selective and simplified portrayal of anatomy that may be easily understood during surgery in a variety of surgical specialties [86]. It is now also possible to boost surgeons’ sensory and motor skills during diverse surgical activities, from macro to micro levels, using computerized, robotics-assisted surgical systems. In comprehensive clinical investigations over the last two decades, the performance, advantages, and efficacy of established surgical robotic systems and many scientific platforms have been evaluated and validated [87,88,89].

The main challenge that AR technology is currently facing is the accuracy. Aside from a qualitative visual evaluation, there is currently no way to measure the correctness of an augmentation or predict the error. This is owing to the utilization of several complicated systems, each of which contributes to the uncertainty. For instance, optical tracking is widely utilized as a source of information in augmented reality for camera movement and fiducial pointing in AR [76]. However, as mentioned in [90], camera calibration, tracking, and hand-eye calibration errors are all accumulated by such systems. A clear requirement for validation methods in surgical AR is created by this lack of precision [1].

Validation requires demonstrating that perhaps the system performs as expected. Validation will mostly consist of establishing that implemented AR improves the knowledge and understanding of the operational domain if AR is designed to deliver an ideal understanding of the work domain. Validation has primarily been studied in terms of its impact on perception so far. As a result, validation is carried out as a research including a representative group of users, covering a wide range of abilities, age, technological skills, gender, and other applicable characteristics, as well as under unique situations such as fatigue and stress.

AR designers want to know how their system affects decision-making and action, and there are numerous metrics that may be used to evaluate and validate this system, which are mentioned in [91]. Figure 8 presents one of the evaluation metrics which can be classified into four groups. The first is about the user who is testing the AR system (e.g., head-mounted system). The system’s human elements, such as the system’s ergonomics, have a direct impact on the user (e.g., comfort of use). The system may also have an effect on the user’s technical (dexterity, tremor) and non-technical skills (such as situation awareness, stress, or risk anticipation). The technology may have an impact on the entire medical team by raising situation awareness and influencing human factors. Although it is well understood that the success of a process is primarily determined by its goal, the procedure used to achieve the goal, such as minimal duration, minimal energy, minimal workspace, cost, repeatability, or standardization, may be of interest and serve as a metric for evaluation of the AR system. Finally, the system may have an impact on the patient by affecting clinical scores, operating room time, or recovery time.

When conducting AR validation studies, it is critical to follow the standard procedures to ensure accurate reporting and interpretation of the findings. It is relevant to describe the clear design and reporting in the context of assessing image-guided navigation systems [92]. It is also critical to be able to clearly specify the conditions under which validation can be carried out in order to acquire a thorough grasp of the results and any potential bias. The suggested validation workflow is shown in

Table 2. Validation workflow based on the operator level (from low to high clinical realism).

Operator	Location	Medical Process	Patient
Engineer	Laboratory	Technical Scenario	Simulations
Medical Student	Simulated OR	Simulated Procedure	Physical Phantoms
Surgeon	Operation Room	Real Procedure	Clinical Dataset

based on the operator level, which ranges from low to high clinical realism (i.e., from engineer to medical student to the surgeon). It describes the parameters of the place/setup where the AR system is being used by the various operators, which is laboratory for an engineer, simulated operation room for a medical student and operation room for the surgeon. An engineer’s work is mainly focused on technical scenarios and testing with simulations, whereas a medical student performs a simulated procedure using phantoms, and the real procedure is being performed by the surgeon using a clinical dataset.

Following the above described evaluation and validation method step by step can aid in the development of a better AR surgical technique. We advise assessing the degree of expertise in the work domain for each operator. This can be calculated using surveys such as the one given in [74], which has questions that span the entire abstraction hierarchy of the relevant work domain. The framework used here can serve as a user-oriented approach in evaluating the amount of perceived information using image-guided clinical systems by surgeons and medical students and supporting their design from the viewpoint of cognitive engineer.

Figure 9 summarizes the suggested parameters which are relevant in evaluating and validating the current AR systems. To have the required precision of any of the AR system under consideration, visualization is the key factor which is further dependent upon many factors, mainly the human–computer interface. The source of real data and the virtual image are important factors affecting the visualization. With an efficient hand-eye coordination and display type, proper spatial reasoning and depth perception are obtained. Various other factors such as the registration technique, tracking modality, rendering time, etc. also contribute towards a reliable visualization. The system being user friendly will ensure the easy acceptance from the surgical community. The surgeon’s ability to use the equipment in the operating room may be constrained by a lack of intuitive design. For a human-centered AR system, the user experience is important additional to the user interface and should be in line with the requirements of the surgeon, the major stakeholder.

The overall functionality of the system and patient’s safety during the whole procedure are the useful factors to evaluate and validate the usage of the AR based navigation in surgery. As already discussed in this section, the surgeons’ skills (such as sensory, motor, and cognitive abilities) are also crucial in achieving a precise surgical result as well as managing the stress and exhaustion that arise during the procedure. Another crucial element in the validation of the AR systems should be the patient’s post-operative recovery period, success, and relapse rate. The effective approach to validate a surgical procedure is to look at the clinical outcome in terms of operating time and efficiency, which is the result of all these elements. Currently, it is challenging to compare accuracy measurements between various systems due to differences in evaluation techniques. However, feedback from surgeons can help determine whether the most advanced and current AR assisted navigation systems are effective or not [66,93,94].

4. Discussion

4.1. Limitations

In the last 20 years, augmented reality has been used in medical operations to aid physicians in understanding and performing procedures, as well as for training and planning. However, as other review papers have pointed out, AR has yet to attain its full potential in terms of utility and deployment in the medical workflow. Many reasons have been given for this; one is that the AR system design approach is primarily technology driven, necessitating the development of clinically useful AR applications. Another reason is the lack of meaningful assessments of planned AR systems to prove their benefits across the range of medical care [95]. The key challenges in evaluating AR technology include comparing results with and without the AR system, assuming that there will be a statistically significant difference in results attributable to the use of the navigation system, which is also not easy to achieve or demonstrate.

It should also be emphasized that, in the AR environment, a common concern for any surgeon is that the target organ may not behave as intended. All the organs in the human body are not rigid; they deform in response to heartbeat and respiratory rhythms, laparoscopic insufflation pressure, and when physically probed [96]. These physical conditions are more noticeable during surgery on the liver and intestines (pliable organs) than the surgery on the bones and brain (semi-rigid organs) [97]. These deformations result in registration problems during surgery, which is one of the factors limiting the usage of AR technology. This is due to the fact that actual surgical confirmation of AR would include medically exposing the concealed structures revealed by the augmentation without modifying the scene or shifting the structures, which is highly unlikely in these conditions.

In AR surgical applications, the tracking of calibrated tools is a basic necessity. In tracking and calibration, the use of error analysis techniques and numerical methods established for registration purposes has numerous purposes [98,99]. Registration methods have been used in tracking to improve target design, predict tracking error, and improve tracking accuracy [100]. Given the widespread usage of calibrated tools in augmented reality surgical applications, there is a lot of opportunity for expanded use of registration techniques in calibration [101,102]. Based on the review completed in [103], there is also a need to build adequate segmentation algorithms to overcome the limits of existing methodologies and imaging modalities.

It can be challenging to select the best metric for a certain image processing task. The work described in [104] highlights some of the most common faults in the most often used reference-based validation criteria in image processing. It is crucial to validate biomedical image analysis algorithms in order to advance understanding and put methodological research into implementation. Validation metrics, or the metrics by which system’s performance is measured, are a crucial part of the validation design phase. In addition, to improve the accuracy of current AR systems, more research is required in the fusion of different imaging modalities, increasing bio-mechanical modeling, and improving image processing and tracking technologies [21]. Efforts should be made to improve the setup of AR systems, creating them to be more user friendly throughout all stages of surgery (microscopic and macroscopic) and also across different surgical procedures. The virtual models must be refined so that they perfectly blend with the real environment. Finally, new imaging modalities such as intra-operative ultrasound or MRI have the potential to add new details to virtual models as well as improve registration.

Both pre-operative planning and intra-operative guidance for surgery are aided by improved visualization, especially when combined with intra-operative imaging for real-time visualization. The incorporation of supplementary technologies to assist the incorporation of AR systems into clinical use may address technical and clinical limitations [75]. AR technology provides a novel visualization paradigm with exciting clinical navigation and guidance potential. However, user acceptance of novel interfaces is always slower, and it is unclear if this is due to a history of slow adoption of new technology in the medical field, regulatory roadblocks, or inherent usability challenges rooted in problems with human perceptual system restrictions [105]. To summarize, when creating the user interface of AR systems in medicine, significant caution must be exercised. An otherwise reliable system may become useless if the representation loses its credibility or becomes unclear.

4.2. Future Outlook

An essential prerequisite for enabling AR assisted navigation is medical imaging. The progress of AR surgical navigation is still being influenced by ingenious surgeons. To address their surgical concerns, these surgeons pushed for the development of this new technology. Thus, to better comprehend how to correlate intuitive and natural multi-sensory manipulation with the patient, some more research must be conducted. The potential for shared control in robotics will be further expanded by the integration of AR with active guidance mechanisms, leading to advancements in navigation, spatial orientation, and intra-operative confidence. AR in medical must overcome its obstacles in addition to technological ones in order to justify its cost, portability, and utility.

In the field of AR based navigation surgery, the creation of new interaction strategies has received less attention than that of hardware development, precise and reliable calibration techniques, and AR visual analytics. However, properly designed interface strategies have the ability to make tasks simpler, enable more natural ways to handle image guided surgery systems and optimize surgical workflow, as well as improve how the guided visuals and augmented reality representations are perceived. Tangible user interfaces are another strategy that might offer a cutting-edge response to interactions for surgical domain management, even though they are not completely investigated in the literature.

There is a lot of room for future research, especially when it comes to letting the surgeon interact within the surgical field of vision to adjust for system errors (such as brain shift, registration error, etc.). As a result, AR based image guided surgery devices could be employed more frequently and with greater accuracy. For appropriate guidance, it is necessary to have a better understanding of the spatial relationships between virtual elements in the surgical field of view and virtual anatomy, as well as their depth perception. AR will become more prevalent in clinical practice if interactivity and visualization are combined in a way that improves perception and enables correct localization of anatomy. It is also crucial to keep in mind that representation and perception are user-dependent; therefore, depending on their area of expertise, level of skill, and psychological makeup, different users will respond differently to various representations.

In conclusion, this paper focused on the need for effective evaluation and validation of the AR assisted navigation systems being employed in surgery. This strategy will enable the creation of clinically applicable systems with an emphasis on evaluating adequate comprehension of key functional concepts (such as data, information, and knowledge) to support decision-making during surgery. To make sure that the reliance on the operational work domain is maintained, evaluation of the AR assisted navigation systems is necessary both at the clinical and technological levels. It will be challenging to guarantee the patient’s surgical safety without a thorough validation of the system by the surgeons. Thus, the AR approach in surgery cannot be translated into a genuine product until it does not meet the stringent certification standards with a credible validation methodology.

To successfully integrate AR technology into current practice, academic researchers, technical industries, and clinical users must work together to pool their knowledge, resources, and expertise in order to migrate AR based navigation technology from the research laboratory to the medical environment. With advances in computer technology and a move towards sophisticated information processing, AR navigation will soon become a more common part of surgical procedures, although further study is required to assess its long-term clinical effect on patients, surgeons, and administrators in hospitals. The advancement of augmented reality to make it more user-friendly has revived interest and will maximize the number of skilled and safe surgical hands in the twenty-first century.