Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Abstract

Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement.

1. Introduction

Robotic systems have experienced significant development in recent decades and have become increasingly prevalent in various applications, including medical, industrial, collaborative, bionics [1,2,3,4,5], etc. Among all the robotic applications, medical and industrial robots have gained extensive attention and have brought about a revolution in their respective fields. For instance, the integration of medical robotic systems has greatly empowered surgeons’ capabilities and performance for surgical task completion compared to traditional manual surgical procedures [6,7]. Specifically, robotic systems have high positioning capabilities, improved dexterity, fatigue avoidance, and reduced trauma, which relates to faster recovery, especially for delicate procedures [8]. Additionally, the advent of the industry 4.0 paradigm strives to achieve more intelligent manufacturing, perception, and monitoring processes [9], in which advanced robotic systems are indispensable. In particular, robotic systems outperform human operators in repetitive tasks, heavy load lifting, quality consistency, and more.

The interface for human–machine or robot interaction and collaboration plays an important role, with which the human operator can send commands and supervise the task execution, such as input from a keyboard, voice, gestures [10,11], etc. However, using traditional interfaces, such as a mouse and keyboard, the visual feedback is provided to the user on a screen and external monitor. Most of these visualizations are not intuitive enough and require high hand–eye coordination capability and could cause distractions [12]. Augmented reality (AR) has emerged as a transformative technology that provides the human operator with more transparent and immersive visual feedback, hence, further assisting the operator in decision-making procedures and improving task completion performance [13,14]. AR could enhance human–robot interaction (HRI) and human–robot collaboration (HRC) by improving communication, situational awareness, and remote collaboration, leading to more efficient and effective interactions.

The integration of AR can enhance the user experience, simplify robot programming, and enable remote collaboration, making robotic systems more accessible and versatile in various applications. In medical robotics, the AR technique has been employed in various applications, such as rehabilitation, surgery, and medical assistance. Especially in surgical fields, the AR technique has been introduced into multiple phases during surgical operations, including preoperative surgical task planning [15], intraoperative surgical guidance [16], and telesurgery [17]. The AR interface can be used to guide the surgeon to locate the incision position and provide real-time intraoperative guidance during robot-assisted surgery (RAS) procedures. Similarly, AR has been demonstrated to be highly beneficial in industrial areas, such as complicated assembly task guidance [18], collaborative task [19], and manufacturing process monitoring [20]. By leveraging the capabilities of AR, industrial robots can become more intuitive, efficient, and accessible tools, empowering operators and experts to collaborate effectively and precisely, enhancing productivity.

Several reviews have been implemented on the topic of AR for robotic systems in recent years, and these reviews cover topics from industrial robots to medical robots, including rehabilitation, assistive, and surgical applications. For example, Qian et al. [21] reviewed AR in RAS scenarios and the hardware components, application paradigms, and clinical relevance. Furthermore, future perspectives were summarized in this work. Bertolo et al. [22] performed a systematic review of AR in urological interventions and pointed out that the critical limitation of AR-assisted surgery was the inaccuracy in registration, which causes poor navigation precision. In addition, Makhataeva et al. [23] reviewed AR in medical, motion planning, human–robot interaction, and multi-agent system applications. Suzuki et al. [24] classified the AR-improved human–robot interaction interfaces for robotics. Recent years witnessed both hardware systems and human–machine interfaces for AR being improved significantly, as well as extensive research results. For example, the HoloLens2 (Microsoft, Redmond, WA, USA) was released in 2019 with improved immersive perception capability, ergonomics, positioning accuracy, gaze tracking function, human–machine interface, etc., compared with the first generation of HoloLens [25]. Furthermore, extensive applications have been investigated in robotic applications recently, such as surgical guidance, surgical training, standard manufacturing, and intelligent manufacturing. Hence, this paper aims to provide a brief review of recent works and advancements in AR techniques for both medical and industrial robot applications. Afterward, the current challenges and future perspectives in AR for robotics are summarized as well.

In this survey paper, the adopted definition of augmented reality is the widely accepted one given by Milgram et al. [26]: “any case in which an otherwise real environment is augmented by means of virtual (computer graphic) objects”. In contrast, a virtual reality (VR) environment is one in which the participant/observer is totally immersed and in which they can interact with a completely synthetic world. To this day, AR and VR still share similarities in terms of hardware, scope, and usability. The importance of VR in technology applications and its increasingly relevant role in research is undeniable; however, due to the differences in areas of application and the sizes of both research fields, this survey paper only focuses on AR applications.

The remainder of this manuscript is organized as follows: Section 2 gives an introduction to the development of AR and robotics. Afterward, Section 3 summarizes the AR for medical robotic applications and consists of Section 3.1, preoperative and surgical task planning; Section 3.2, image-guided robotic surgery; Section 3.3, surgical training and simulation; and Section 3.4, Telesurgery. Following that, Section 4 describes AR for industrial robotic applications and includes Section 4.1, human–robot interaction and collaboration; Section 4.2, path planning and task allocation; Section 4.3, training and simulation; and Section 4.4, teleoperation control/assistance. Then, a discussion is given in Section 5 and includes Section 5.1, limitations and challenges; and Section 5.2, future perspectives. Finally, Section 6 concludes the work.

2. Augmented Reality and Robotics

AR possesses the potential to transcend the physical limitations of conventional interaction by integrating holographic information onto real scenarios. This unique attribute provides the users with a more comprehensive perception capability of their surroundings and results in improved interactive experiences [27]. Over the past decades, AR has gained substantial popularity and is increasingly influential in diverse fields such as industry, medicine, entertainment, and education [28]. Benefiting from the inherent intuitive and efficient information presentation ability in interaction, AR-based solutions have been integrated into many robotic applications [24]. Figure 1 illustrates the concept of integrating AR techniques in medical and industrial robotic applications.

2.1. Augmented Reality

Although AR has achieved widespread application, a clear and consistent definition of this technology remains elusive both in academic and industrial fields. Milgram and Kishino et al. [26] introduced the concept of a “reality–virtuality continuum” to define augmented reality and augmented virtual environments based on surrounding environmental characteristics. Azuma et al. [29] subsequently proposed a widely adopted definition of AR as the seamless integration of 3D virtual objects with the real environment in real-time, characterized by three key features: the incorporation of virtual and real space, real-time interaction, and spatial registration. Considering the blurred boundaries between AR and mixed reality (MR), a more general description is nowadays widely adopted, positing that any system that enhances physical objects and environments can be deemed as AR, regardless of the technological form [23,24]. Despite these various definitions, the overarching objective of AR is to augment human perceptual capabilities of the physical environment while simultaneously improving information interaction efficiency [30].

AR devices encompass various fixed/mobile displays, projectors, and head-mounted displays (HMD) [31]. Display-based AR implementation necessitates a camera to capture real-world information, which is rendered with virtual information in the computer and finally exhibited on the monitor. In contrast, projection-based augmented reality solutions directly project digital content onto physical environments, leading to an enhanced perception of reality and enabling users to interact with virtual objects in a more direct way. In recent years, lightweight wearable displays have become the primary medium of AR, typically implemented through two primary modes: optical perspective and video perspective [32]. The optical perspective utilizes holographic optics to project virtual information onto the eye’s imaging system, superimposing it onto real-world environmental imaging. In contrast, the video perspective fuses virtual and environmental information, displaying it on an opaque HMD that is then optically delivered to the user’s eyes for perception.

In recent years, a number of commercially available AR devices have appeared on the market and are being used in various applications [33]. Display-based AR is the simplest and easiest to implement and can run on any display-equipped device, including computers, mobile phones, or tablets. Projection-based AR (also known as projection mapping and spatial AR) uses a projector instead of a display to project augmented information onto an irregular surface to enhance perception and provide the ability to interact spatially. One of the best-known HMD devices based on optical perception is Google Glass (Google Inc., Mountain View, CA, USA), which projects information onto a small screen located directly above and to the right of the user’s right eye, with little obstruction to the user’s vision. The prismatic projection structure is easy to implement, but the small screen and the monocular structure provide only limited immersion. Another commonly used device is Microsoft’s HoloLens, which is now in its second generation. HoloLens 2 is a complete AR system and contains a central processing unit, a custom-designed holographic processing unit, various types of sensors, and a holographic projector with see-through optics. Due to its rich sensing system and large field of view (FoV), it has become a mainstream device for AR applications in industry and medicine. In addition, Magic Leap One (Magic Leap Inc., Plantation, FL, USA) is an optical see-through head-mounted display (OST-HMD) AR glass that is capable of overlaying digital content in the real world, creating an immersive AR experience, and has been integrated into many robotic applications. Common video-based HMD devices currently available include the Samsung Gear VR2 (Samsung Corp., Seoul, Republic of Korea), which uses a smartphone screen and embedded camera to display the real world, and the ZED Mini (Stereolabs, San Francisco, CA, USA), which is specifically designed to provide a high-resolution stereoscopic view of the real world while the user wears the HMD. Notably, Apple’s recently announced Vision Pro (Apple Inc., Cupertino, CA, USA) is also a video see-through HMD and is expected to be the new most powerful AR HMD.

Table 1. Hardware parameters of the AR devices used in the applications collected in this survey. The devices are ordered chronologically by their release date.

Platform	Field of View (degrees)	Per-Eye Resolution (pixels)	Tracking Type (DoF)	Eye Tracking	Latency (ms)
Google Glass	-	640 × 360	3 Non-positional	No	-
HoloLens 1	34	1268 × 720	6 Inside-out	No	16
Magic Leap 1	50	1280 × 960	6 Inside-out	Yes	8
HoloLens 2	52	2048 × 1080	6 Inside-out	Yes	16
Magic Leap 2	70	1440 × 1760	6 Inside-out	Yes	8
Meta Quest Pro	96	1800 × 1920	6 Inside-out	Yes	10

collects and compares the hardware parameters of the most relevant AR devices on the market [34]. As technology advances, augmented reality devices are achieving unprecedented levels of performance, seamlessly blending virtual elements with the real world. Their enhanced processing power and refined optics are opening up exciting possibilities across various industries and research fields, revolutionizing how we interact with information and our environment. The improvement of display and tracking technologies will broaden the range of practical applications, especially when more precise alignments are needed [24].

2.2. Robotics Applications

Robotics is currently experiencing a paradigm shift from single-purpose applications and fixed workspace constraints towards the development of general-purpose collaborative robots: positional accuracy, repeatability, spatial awareness, and back-drivability are now key concepts shared by very diverse fields of applications.

Specifically, robotics has had a profound impact on both industrial and medical applications, revolutionizing these sectors in numerous ways. In the industrial domain, robots are extensively used in manufacturing processes, automating tasks such as welding, assembly, and material handling [35]. This integration enhances productivity, lowers production costs, and ensures consistent quality. Furthermore, collaborative robots, or cobots, work alongside human workers, augmenting their capabilities and creating safer work environments. In the medical field, robotics has brought about significant advancements, particularly in surgical procedures. Robotic surgical systems enable surgeons to perform complex operations with enhanced precision and dexterity, leading to reduced trauma, faster recovery times, and improved patient outcomes [36]. Additionally, telemedicine applications are leveraging robotics to facilitate remote surgeries and consultations, expanding access to healthcare in remote areas. Moreover, rehabilitation robots aid in physical therapy, helping patients to recover from injuries or neurological conditions more effectively. The continuous development of robotics technology promises even more innovative applications in both the industrial and medical realms, making processes more efficient and elevating the standard of care in healthcare.

Medical robotics and industrial robotics are two distinct yet interconnected branches of the robotics field, each with its specialized applications and unique set of challenges. Medical robotics focuses on the development of robotic systems and devices to assist healthcare professionals in surgeries, diagnostics, and patient care [37]. These robots are designed to be precise, compact, and capable of delicate movements to ensure safe and accurate medical interventions. In contrast, industrial robotics is geared towards automating manufacturing processes in various industries, ranging from automotive to electronics, where the emphasis is on high-speed production, efficiency, and cost effectiveness [38]. While both medical and industrial robotics employ cutting-edge technologies, they have divergent objectives and constraints. Medical robotics demands stringent safety and regulatory compliance due to the potential risks associated with human–robot interactions in sensitive medical procedures [39]. On the other hand, industrial robots primarily operate within controlled factory environments, and safety measures are mainly focused on protecting human workers from accidental collisions. In terms of complexity, medical robotics often involves intricate kinematics and sensor systems to navigate within the confined spaces of the human body and deliver precise surgical maneuvers. Industrial robots, however, are engineered to perform repetitive tasks with higher payloads and longer reach, requiring robust mechanical structures and control algorithms. The cost factor also differentiates these two domains significantly. Medical robots tend to be more expensive due to their specialized nature, advanced materials, and extensive research and development. Industrial robots, by contrast, are often designed for mass production and can benefit from economies of scale, resulting in comparatively lower costs.

Despite their distinct purposes, there are areas of convergence between medical and industrial robotics. Advancements in perception systems [40], machine learning, and artificial intelligence [41] have influenced both domains, leading to improved capabilities such as object recognition, adaptive control, and autonomous decision making. Moreover, developments in materials and miniaturization [42] have enabled the creation of more compact and versatile robots in both sectors.

In conclusion, while medical robotics and industrial robotics serve divergent purposes, they share common technological foundations and have shaped each other’s progress through interdisciplinary collaboration. As these fields continue to evolve, their combined contributions hold the potential to drive further innovations and improve the quality of life for individuals worldwide.

2.3. Robotic Applications with Augmented Reality

The collaboration between human operators and robots within overlapping workspaces requires a higher level of efficiency and flexibility in human–robot interaction. Traditional human–robot interaction relies on the robots’ internal physical and perceptual capabilities, such as gestures, audio, visual displays, natural language (text and spoken language), physical interaction, and haptic feedback [43]. However, these interaction methods are limited by their expression capacity, presenting major challenges in efficiency and convenience. The emergence of AR technology has the potential to address these challenges in HRI, particularly in medical and industrial applications.

For medical robots, the AR technique is usually integrated into preoperative planning and intraoperative navigation, which can effectively alleviate the cognitive burden on surgeons during surgeries and improve the efficiency and accuracy of surgical operations. For example, Porpiglia et al. [44] introduced elastic 3D virtual models in the da Vinci surgical system (Intuitive Surgical Inc., Sunnyvale, CA, USA) and superimposed them on the three-dimensional visual interface provided by the laparoscope to dynamically simulate the deformed organ and guide the prostate surgery execution. In addition to the lead surgeon, AR technology can also be used to assist paramedics with their tasks. For example, Qian et al. [45] proposed the concept of ARsist, which uses a head-mounted optical display to superimpose the laparotomy scene with a real operating scene to assist the first assistant in the auxiliary task of robotic surgery.

In industrial robotic applications, AR has been introduced for robot motion planning, control, task allocation, manufacturing system monitoring, etc. The use of AR technology allows for the direct mapping of virtual objects into real-world environments, facilitating rapid path planning and programming. Ong et al. [46] proposed an AR-assisted welding path programming system that simplifies the welding path and torch direction definition process through an AR interface, enabling accurate positioning based on user inputs. Ostanin et al. [47] developed an AR-based interactive programming method using the Unity engine and HoloLens glasses, which involved point cloud analysis of real robot positions to generate viable robot trajectories using virtual markers and menus. In another study, Wesley et al. [48] established an AR-based application for a carbon-fiber-reinforced polymer materials process, showing that AR can reduce physical exertion and task completion times compared to a manual positioning joystick while also improving robot utilization.

3. Augmented Reality for Medical Robotic Applications

AR has gained significant traction in the medical robot field and has been integrated into various applications to enhance preoperative and intraoperative surgical planning and surgical procedure guidance [31]. AR in medical robotics has a fascinating history that began with its early applications in the late 20th century. One of the pioneering works in this field was the development of the “virtual fixtures” system by Louis Rosenberg at the U.S. Air Force Research Laboratory in 1993, which introduced the concept of overlaying virtual information onto the real world for robotic-assisted tasks [49]. As technology advanced, AR found its way into surgical procedures, where it was used to enhance visualization, navigation, and precision. A notable example is the first reported use of AR in neurosurgery in 1998, when Kwoh et al. utilized AR visualization to assist in skull base tumor resections, achieving improved accuracy and safety [50]. Since then, AR in medical robotics has continued to evolve, revolutionizing various aspects of healthcare, from training and simulation to intraoperative assistance and post-operative monitoring. One of the most relevant examples proposed in the early 2000s was developed by Anderson et al. [51], who proposed a computer-based system for the simulation of image-guided cardiovascular procedures for physician and technician training.

These pioneering works still resonate in the more modern applications of the research community of medical robotics, medical imaging, and computer-aided surgery. One of the largest research groups is the Medical Image Computing and Computer-assisted Intervention (MICCAI) community, which embeds laboratories and researchers from all around the world, organizing annual conferences, workshops, and challenges to push the boundaries of AR [13] and medical robotics forward.

In this section, related works using AR for medical robots are reviewed and summarized, mainly categorized into four topics, namely, “preoperative and surgical task planning”, “image-guided robotic surgery”, “surgical training and simulation”, and “telesurgery”.

3.1. Preoperative and Surgical Task Planning

With medical imaging data, such as computerized tomography (CT) or magnetic resonance image (MRI) scans, surgeons can accurately locate targets and assess the depth and orientation of structures, improving surgical precision and reducing risks. In particular, AR can further assist surgeons in preoperative and surgical task planning by providing them with real-time and precise guidance and visualization. Moreover, optimizing the positioning of the surgical robot, the robot configuration, and the incision position can improve the surgeon’s ergonomics and reduce the probability of intraoperative failure during RAS. This subsection summarizes related works on using AR interfaces for surgical task planning, robot configuration optimization and port placement, and surgical tool insertion tasks. Related works using AR for preoperative and surgical task planning are summarized in Table 2.

AR techniques have been successfully integrated into the surgical task planning process to improve users’ precision capability and task completion efficiency by leveraging the information overlapped onto their view. For instance, Samei et al. [52] implemented a partial AR system with live ultrasound and registered preoperative MRI for guiding robot-assisted radical prostatectomy using the da Vinci (Intuitive Surgical, Sunnyvale, CA, USA) robotic system. Zhang et al. [53] proposed a multiple-view registration interface for planning robot-assisted spinal puncture procedures; comparison experiments with a single-view registration interface were conducted and showed the proposed approach achieved a registration accuracy of $1.70\pm 0.25$ mm, which met the requirements in clinical procedures. In ref. [15], an AR application was employed for trajectory planning on the adjacent surface of the full crown during robot-assisted tooth preparation procedures; the robot could also be controlled using the designed AR interface. Fu et al. [54] proposed an AR interface for interactive needle insertion path planning for robot-assisted percutaneous nephrolithotomy (PCNL) tasks. Moreover, Ho et al. [55] designed an AR-assisted task planning and navigation system for robot-assisted surgical drilling tasks to enhance safety. The implemented AR interfaces enabled the human operator to modify the robot’s trajectory, both the position and orientation of the robotic manipulator, and supervise the execution process of the surgical task.

In addition, AR visualization provides the surgeon with an intuitive and immersive interface for the robotic configuration verification, surgical system setup, and incision port selection on a patient’s body. Fotouhi et al. [56] proposed an AR-assisted interactive interface to address the challenge of perspective ambiguities during the virtual-to-real alignment procedure of the robotic arm in robot-assisted minimally invasive surgery (RA-MIS) tasks. Żelechowski et al. [57,58] investigated the application of using the AR OST-HMD to choose the position of the patient appropriately considering the limited workspace of the serial LBR iiwa (KUKA, AG, Augsburg, Germany) surgical robotic system in RA-MIS tasks. Similarly, Wörn et al. [59] explored the utilization of an AR interface to find the optimal placement of the trocars to minimize the probability of collision with the robot arm in manual laparoscopic surgery and the robot configuration initialization in teleoperated robot-assisted surgery using the da Vinci robotic system. Following this work, Weede et al. [60] developed a projection-based AR visualization system to overlap the optimized port positions onto the patient’s abdomen, taking into account the ergonomic working directions, collision avoidance, and reachability of the target areas during minimally invasive surgery (MIS) procedures. Moreover, an AR-assisted framework for repositioning the C-arm was implemented, and the X-ray technician was equipped with an HMD AR visualization interface to operate the C-arm interventionally in 3D via an integrated infrared sensor [61]. Fu et al. [62] designed an AR-assisted robot learning framework for MIS tasks, and an AR interface was adopted to verify the configuration of a serial redundant robot from HoloLens2 OST-HMD. To generate a robust desired trajectory to transfer to the robotic system for reproduction in the minimally invasive surgery scenario, the Gaussian mixture model (GMM) and Gaussian mixture regression (GMR) were employed to encode the human-demonstrated multiple trajectories in this work.

Surgical tool or needle insertion is a common and crucial step in surgical procedures, particularly for MIS tasks [31]. AR has been employed to assist surgeons in selecting the needle poses intuitively instead of observing the imaging feedback from external 2D monitors, which potentially has reduced surgeons’ mental and physical workloads [63]. Vörös et al. [64] created an AR interface, with HoloLens2 OST-HMD employed, to help the operator to position and adjust the pose of the drilling apparatus in robot-assisted pedicle screw surgical tasks. Qian et al. [21] designed the “ARssist” framework to guide the “first assistant” with preoperative instrument insertion and tool manipulation in robotic-assisted laparoscopic surgery. The experimental results showed that the proposed interface significantly enhanced task completion performance and efficiency while requiring lower hand–eye coordination than a typical 2D monitor visualization interface. Similarly, an AR interface was implemented for novice surgeons training using OST-HMD in PCNL tasks, particularly for the needle alignment and insertion procedures [65]. Except for wearable devices, a projection-based AR system was used to plan needle placement during a percutaneous radio-frequency (RF) liver ablation for liver resections. The average error in the experiment was 1.86 mm for virtual needle insertion, which was less than the clinical requirement of 2 mm [66]. In ref. [67], an AR-assisted robotic navigation system was designed for spinal surgery, and the pre-planned path was visualized using the wearable OST-HMD to guide the pedicle screw insertion process.

Table 2. Summary of AR use in medical robots for preoperative and surgical task planning.

Application	References	Robot Platform	AR Medium	Detailed Contents
Surgical task planning	Samei et al. [52]	da Vinci	Console	Prostatectomy
	Zhang et al. [53]	KUKA KR 6 R900	Projector	Spinal
	Jiang et al. [15]	Custom	Monitor	Dental
	Fu et al. [54]	KUKA LWR IV+	OST-HMD	PCNL
	Ho et al. [55]	Custom	OST-HMD	Spinal
Robot system setup	Fotouhi et al. [56]	KUKA iiwa 7 R800	OST-HMD	MIS
	Żelechowski et al. [57]	KUKA LBR iiwa Med	OST-HMD	MIS
	Wörn et al. [59]	da Vinci	Projector	Laparoscopy
	Weede et al. [60]	KUKA LWR IV+	Projector	MIS
	Fu et al. [62]	KUKA LWR IV+	OST-HMD	MIS
Needle insertion planning	Vörös et al. [64]	KUKA LWR IV+	OST-HMD	Spinal
	Qian et al. [21]	da Vinci	OST-HMD	MIS
	Ferraguti et al. [65]	KUKA LWR IV+	OST-HMD	PCNL
	Wen et al. [66]	Custom	Projector	RF Ablation
	Boles et al. [67]	Custom	OST-HMD	Spinal

Table 2. Summary of AR use in medical robots for preoperative and surgical task planning.

Application	References	Robot Platform	AR Medium	Detailed Contents
Surgical task planning	Samei et al. [52]	da Vinci	Console	Prostatectomy
	Zhang et al. [53]	KUKA KR 6 R900	Projector	Spinal
	Jiang et al. [15]	Custom	Monitor	Dental
	Fu et al. [54]	KUKA LWR IV+	OST-HMD	PCNL
	Ho et al. [55]	Custom	OST-HMD	Spinal
Robot system setup	Fotouhi et al. [56]	KUKA iiwa 7 R800	OST-HMD	MIS
	Żelechowski et al. [57]	KUKA LBR iiwa Med	OST-HMD	MIS
	Wörn et al. [59]	da Vinci	Projector	Laparoscopy
	Weede et al. [60]	KUKA LWR IV+	Projector	MIS
	Fu et al. [62]	KUKA LWR IV+	OST-HMD	MIS
Needle insertion planning	Vörös et al. [64]	KUKA LWR IV+	OST-HMD	Spinal
	Qian et al. [21]	da Vinci	OST-HMD	MIS
	Ferraguti et al. [65]	KUKA LWR IV+	OST-HMD	PCNL
	Wen et al. [66]	Custom	Projector	RF Ablation
	Boles et al. [67]	Custom	OST-HMD	Spinal

3.2. Image-Guided Robotic Surgery

Surgeons rely on the information from medical imaging not only for preoperative planning purposes but in the intraoperative phase as well. In most cases, the operating room (OR) is equipped with displays where the surgeon and the OR staff visualize the medical image in its 3D or sliced-2D form, using it as a reference for locating the anatomical structures of interest. In this scenario, the surgeon is constantly switching their focus of attention from the medical images to the surgical environment, requiring extraordinary hand–eye coordination and the ability to map the surgical scene and the motion of the instruments to the image space. The advent of AR and its introduction into the surgical field empowered surgeons with techniques for visualizing the information gathered from medical images directly superimposed onto their view of the surgical environment and seamlessly aligned with anatomical structures. All the applications of AR for image-guided robotic surgery presented in this subsection are summarized in Table 3.

When the surgical scenario is not subject to deformations, the registration is rigid, the accuracy is highest, and the benefits of AR technologies are the most obvious. Fotouhi et al. [68] guided the placement of K-wires in orthopedic surgery for total hip replacement (THP) by visualizing the optimal insertion path of the OST-HMD of the surgeon. Andress et al. [69] exploited co-registration between the OST-HMD and a C-arm for visualizing annotations on X-ray images and their real position in the real-world space, allowing for easier localization of orthopedical lesions. For spinal microscopic surgery, Carl et al. [70] integrated the visualization of vertebrae, discs, and tumors into the video feed of heads-up displays to allow an intraoperative see-through experience.

Teleoperated leader–follower (in the past known as master–slave) surgical robotic platforms are equipped with screens for real-time visualization of the endoscopic camera feed: AR applications in this context usually take advantage of such a display to visualize the anatomical structures directly superimposed. Chen et al. [71] performed an ultrasound-based reconstruction and registration of oropharyngeal anatomical structures, which were then displayed onto the high-resolution stereo viewers (HRSV) of a da Vinci robot during trans-oral robotic surgery (TORS). In the same context, Chan et al. [48] successfully conducted a feasibility cadaveric experiment, visualizing in real-time the carotid artery system during robotized head-and-neck surgery. Integrating the augmented visualization onto the display of the robotic platform may be advantageous compared to employing external hardware such as OST-HMDs, which come with their limitations in terms of battery life, connectivity issues, and visual line-of-sight impairments [72].

AR has also been integrated into surgical contexts where the anatomy is non-rigid and, therefore, registration must be performed with more advanced techniques that account for deformations. This challenge was pursued by Pessaux et al. [73], who built an advanced visualization system for the abdominal vasculature and anatomy to assist surgeons during liver resection procedures, accounting for pneumoperitoneal deformations into the registration phase. With a different approach, Marques et al. [74] proposed a framework for surgical assistance during minimally invasive robotic hepatic surgery, registering preoperative CT images to point clouds of the liver surface acquired intraoperatively from a stereo endoscope. Lee et al. [75] built a similar framework for visualization in robotic thyroid surgery. Shen et al. [76] developed a fully customized actuated system for trans-rectal ultrasound (TRUS) 3D reconstruction of the rectum, to display the augmented visualization enriched with the reconstructed anatomy and the target tumoral tissues on an OST-HMD. Kalia et al. [77] conducted a similar approach, where a preclinical study on a real-time AR-based guidance system for radical prostatectomy was embedded in a surgical robot aimed at assisting the surgeon in the intraoperative visualization of the prostate anatomy reconstructed from TRUS, the projected US scans, and the tumors to be targeted. Porpiglia et al. [78] employed a computer vision algorithm for the automated anchoring of virtual 3D models on intraoperative images during a robotized prostatectomy, effectively utilizing a learning-based approach for AR registration.

The work of Piana et al. [79] showcased how AR was employed to address some notable limitations of teleoperated surgical robotics. Specifically, their work highlighted how the lack of haptic feedback in robotic surgery was problematic in the identification of atheromatic plaques during robot-assisted kidney transplant, and, therefore, built an intraoperative visualization tool based on the patient’s 3D images, intending to support the localization process during minimally invasive surgery.

Bianchi et al. [16] notably conducted a 20-patient quantitative study demonstrating that embedding AR-enhanced visualizations of anatomical structures in radical prostatectomies allows for significantly reduced positive surgical margins. AR was validated as a technique for increasing the safety of surgical interventions in addition to the benefits to the workflow and cognitive load of the surgeon.

Edgcumbe et al. [80] introduced a custom-made sterilizable “dart-shaped” tracker to be surgically inserted into the patient’s body to allow its registration into the coordinate system of the surgical robot, effectively allowing the visualization of the relative pose of the instruments and the tracker itself. Combining the robot kinematics and the detected relative pose of the tracker, the surgeon visualizes the surgical structures and their 3D relative pose with respect to the instruments intraoperatively. Similarly, Qian et al. [45] exploited ArUco markers mounted onto the actuated arms of the da Vinci robot to project a hologram of the robot on an OST-HMD worn by the OR first assistant. They could, therefore, visualize the whole robot hologram projected in the OR and onto the patient, where the instruments inside the surgical scene were also visible.

Table 3. Summary of AR in image-guided surgery application.

Application	References	Platform	AR Medium	Detailed Contents
Head-and-Neck	Chen et al. [71]	da Vinci	HRSV	Anatomy
Head-and-Neck	Chan et al. [48]	da Vinci	HRSV	Anatomy
Hepatic	Pessaux et al. [73]	da Vinci	HRSV	Anatomy
Hepatic	Marques et al. [74]	da Vinci	OST-HMD	Anatomy
Thyroid	Lee et al. [75]	da Vinci	HRSV	Anatomy
Colorectal	Shen et al. [76]	Custom	OST-HMD	Anatomy
Colorectal	Kalia et al. [77]	da Vinci	HRSV	US Scans
Colorectal	Porpiglia et al. [78]	da Vinci	HRSV	Anatomy
Urology	Piana et al. [79]	da Vinci	HRSV	Anatomy
Urology	Edgcumbe et al. [80]	da Vinci	HRSV	Instruments
Abdominal Cavity	Qian et al. [45]	da Vinci	OST-HMD	Instruments

Table 3. Summary of AR in image-guided surgery application.

Application	References	Platform	AR Medium	Detailed Contents
Head-and-Neck	Chen et al. [71]	da Vinci	HRSV	Anatomy
Head-and-Neck	Chan et al. [48]	da Vinci	HRSV	Anatomy
Hepatic	Pessaux et al. [73]	da Vinci	HRSV	Anatomy
Hepatic	Marques et al. [74]	da Vinci	OST-HMD	Anatomy
Thyroid	Lee et al. [75]	da Vinci	HRSV	Anatomy
Colorectal	Shen et al. [76]	Custom	OST-HMD	Anatomy
Colorectal	Kalia et al. [77]	da Vinci	HRSV	US Scans
Colorectal	Porpiglia et al. [78]	da Vinci	HRSV	Anatomy
Urology	Piana et al. [79]	da Vinci	HRSV	Anatomy
Urology	Edgcumbe et al. [80]	da Vinci	HRSV	Instruments
Abdominal Cavity	Qian et al. [45]	da Vinci	OST-HMD	Instruments

3.3. Surgical Training and Simulation

The increase in minimally invasive surgical robotics procedures in the last decade has demanded an increasingly higher number of trained surgeons capable of teleoperating such advanced and complex systems while taking advantage of the benefits of RA-MIS safely and effectively. The role of surgical training and surgical robotics training is, hence, of key importance in the achievement of optimized learning time, skill retention, and skill transfer. In this context, AR acts as a guidance system and as an automated supervisor for personalized learning: Peden et al. [81] reported a significant improvement in the perceived quality and utility of the teaching curriculum of surgical skills for students that learned with AR-enhanced modalities. Long et al. [82] leveraged reinforcement learning to learn from expert demonstrations on a peg-transfer task and then generated a 3D guidance trajectory, providing prior context information of the surgical procedure.

The integration of advanced AR visualization systems into surgical robotics training curricula allows trainees to effectively visualize the correct motion paths that they should follow, to more quickly learn the anatomy and the structures involved in the task, and to smooth the learning curve. Rewkowski et al. [83], for example, proposed projecting visual cues on an OST-HMD worn by a trainee teleoperating a surgical robot during the execution of a peg-transfer task, with accurate calibration accuracy and real-time capabilities. Barresi et al. [84] developed an AR-enhanced simulator for learning robot-assisted laser microsurgery, where the Electroencephalography (EEG) of the operator was recorded and processed in real-time to estimate the level of focus and retract the virtual scalpel in low concentration phases. A similar framework was proposed by the work of Wang et al. [85] and Zeng et al. [86], where brain–computer interfaces (BCIs) were embedded with augmented reality feedback to control a robotic arm used as an assistance strategy for learning object grasping. Both studies were aimed at paralyzed subjects. Gras et al. [87] proposed an adaptive AR-enhanced surgical robotic simulator for neurosurgery embedded with a Gaussian process voter that automatically selects the level of AR assistance deployed to the trainee. The authors also conducted a user study that showed clear improvements in user perception of the surgical scene and task times during a tumor marking task. Condino et al. [88] proposed a “tactile” AR approach by building an actuated wearable fabric yielding device that mimics the haptic sensations on the fingertips for improving artery palpation training.

AR technology in surgical robotics training is also exploited to enhance the supervision phase as an advanced communication and mentoring tool. Specifically, Jørgensen et al. [89] addressed the crucial issue related to limited visual communication between the supervisor and the trainee and proposed a compact system to overlay the video streams on the HRSV daVinci with annotation and 3D computer graphics generated by the supervisor. All the aforementioned applications of AR for surgical training and simulations are summarized in

Table 4. Summary of AR applications in surgical training.

Application	References	Platform	AR Medium	Assessment
General Skills	Rewkowski et al. [83]	Custom	OST-HMD	None
General Skills	Long et al. [82]	daVinci	HRSV	Quantitative
Supervision	Jørgensen et al. [89]	daVinci	HRSV	Qualitative
Safety	Barresi et al. [84]	Custom	OST-HMD	Qualitative and Quantitative
Rehab	Wang et al. [85]	Custom	Screen	Quantitative
Rehab	Zeng et al. [86]	Custom	Screen	Quantitative
Injury Detection	Gras et al. [87]	Simulator	Display	Quantitative User Study
Preoperative Training	Si et al. [90]	Simulator	OST-HMD	Questionnaire
Haptic	Condino et al. [88]	Custom	Custom	Questionnaire

.

3.4. Telesurgery

Benefiting from the advances in robotic systems and telecommunication, telesurgery enables surgeons to perform complex surgical operations in a remote manner regardless of their physical location [91]. AR in telesurgery and telementoring further enables remote visualization, robot control, and proximity alerts. By leveraging these capabilities, AR facilitates seamless real-time remote and local collaboration, enhances surgical accuracy, expands access to specialized medical expertise, and ultimately improves patient care and outcomes [92]. Related works using AR for preoperative and surgical task planning are summarized in

Table 5. Summary of AR use in telesurgery in medical robot applications.

Application	Reference	Robot Platform	AR Medium	Detailed Contents
Remote visualization	Lin et al. [93]	KUKA LBR Med	OST-HMD	Interventions
	Gasques et al. [94]	Custom	OST-HMD	Trauma
	Black et al. [95]	Custom	OST-HMD	Ultrasound
	Qian et al. [96]	da Vinci	OST-HMD	MIS
	Huang et al. [97]	KUKA LBR iiwa	Monitor	Percutaneous
Teleoperation control	Lin et al. [98]	Custom	OST-HMD	Endoluminal
	Fu et al. [99]	Universal Robot 5	Monitor	Ultrasound
	Ho et al. [55]	da Vinci	Projector	Laparoscopy
	Ma et al. [100]	da Vinci and Custom	OST-HMD	Laparoscopy
Latency and motion prediction	Richter et al. [17]	da Vinci	Console	MIS
	Bonne et al. [101]	da Vinci	OST-HMD	MIS
	Gonzalez et al. [102]	da Vinci	Monitor	MIS
	Fu et al. [99]	Universal Robot 5	Monitor	Ultrasound

.

Several works have investigated the topic of how to improve users’ remote visualization capability and situational awareness in telesurgery with AR. In these works, AR was employed to provide the remote operator with intuitive and immersive visualization feedback of important anatomies, surgical instruments inside the human abdominal cavity, depth information, etc. Lin et al. [93] explored how to provide clinicians working remotely with synchronous visual feedback using an OST-HMD, which was mounted at the end effector of a serial redundant robotic manipulator. Gasques et al. [94] designed an immersive collaboration framework for surgical telementoring tasks. Experiments were conducted on cadavers with both expert and novice surgeons and exhibited the promising potential of using AR for telesurgery. Black et al. [95] explored the possibility of using the MR interface in a tele-ultrasound task. In detail, the follower was instructed to track the desired position and contact force to perform ultrasound scanning tasks demonstrated by a remote expert. In addition, Qian et al. [96] developed an AR-assisted framework “ARAMIS” to provide the surgeon with visual feedback on the patient’s internal structure during MIS in real-time and wirelessly. The end-to-end latency was reported as 178.3 ms and improved intuitiveness and reduced task completion time cost, as well having a higher success rate. In ref. [97], Huang et al. designed an auto-stereoscopic visualization system for telesurgery leveraging the AR techniques on a local site display. The planned path and model during the preoperative phase were fused with the point clouds acquired using an RGB-D camera from a remote environment.

Furthermore, telesurgery robotic systems using the AR medium have been extensively explored considering the advantages of enhanced hand–eye coordination reducing the cognitive load, remote collaboration possibilities, radiation exposure avoidance, etc. In ref. [98], Lin et al. designed an AR-assisted touchless teleoperation control interface designed to provide the operator with immersive visualization of the patient’s anatomy structure and to guide the surgical robot for endoluminal intervention procedures with human hand gesture recognition. Fu et al. [99] explored the usability of an AR visualization interface to ensure the synchronization between the local and remote sides in teleoperated robot-assisted ultrasound scanning tasks. After acquiring each frame of the image from the patient site, a Pose-ResNet artificial intelligence model was utilized to calculate the positions of 16 key points of the human body on the master site. Ho et al. [55] studied the AR-assisted supervised control modality of a robot-assisted drilling system. The surgical trajectory was projected onto a 3D vertebrae bone model using BT-300 AR glasses (Seiko Epson, Suwa, Japan), and the operator could modify the robot trajectory through a graphical user interface (GUI) remotely. Ma et al. [100] implemented a view adjustment framework using an OST-HMD to track the surgeon’s head movement for autonomous navigation of a robotic stereo flexible endoscope, which was equipped at the distal end of the da Vinci surgical robotic manipulator.

Latency is another critical issue in telesurgery, which could affect task completion accuracy and safety, especially in delicate tasks (e.g., delicate and fine manipulation tasks). Furthermore, the time delay could impose a high cognitive workload on the operator, cause impaired telepresence, and reduce the efficiency of task completion. Although 5G technology had been investigated for minimizing the latency in remote laser microsurgery [103], the economy and accessibility were not always available in all telesurgery applications. Instead, AR has been utilized in many robot teleoperation applications to overcome this challenge by merging digital information into physical scenarios and robot motion prediction.

For example, Richter et al. [17] developed a stereoscopic AR predictive display (SARPD) interface to deal with the time delay issue in telesurgery by displaying the predicted surgical instrument motion cues alongside the in situ scenarios immediately. Bonne et al. [101] implemented a digital twin system to deal with the network instability and delays challenge for peg-transfer surgical training tasks in telesurgery. The teleoperator performed the surgical operation by observing the digital twin robotic system and did not suffer from unstable or low-bandwidth communication anymore, and the remote robot executed the human command semi-autonomously. Similar work was performed by Gonzalez et al. in [102], an AR interface for predicting robotic arm motion visualization was implemented to provide the surgeon with real-time feedback to avoid fatal surgical errors caused by communication delays during telesurgery. Fu et al. [99] integrated AR visualization into a teleoperated robot-assisted ultrasound system incorporated with dynamic contact force prediction between the probe and the patient’s body to ensure safety. Although latency existed, the real robot motion was immediately displayed on the local side by overlapping the robot hologram model on the physical.

4. Augmented Reality for Industrial Robotic Applications

AR techniques facilitate widespread application of industrial robots across various domains, including collaborative systems, natural interaction, intuitive task planning, robot training, and remote control. In this section, related works utilizing AR for industrial robots are reviewed and summarized, primarily categorized into four topics, namely, “human–robot interaction and collaboration”, “path planning and task allocation”, “training and simulation”, and “teleoperation control and assistance”.

4.1. Human–Robot Interaction and Collaboration

HRI and HRC are vital nowadays for smart manufacturing transformation, especially toward human-centric, resilient, and sustainable principles [104]. HRI means one teammate communicates, guides, or controls the other remotely or physically in touch for the completion of a shared task [105]. HRC pays more attention to the parallel, coordinated, and synchronous activity of humans and robots in an overlapped workspace for a common task goal [106]. For both phases, AR technologies play a critical role in both efficient and effective collaborative work.

The deployment of an AR environment in HRI and HRC tasks allows human operators to intuitively teleoperate or remote control robots without expert knowledge of robot programming. For example, Wang et al. [19] developed a feasible AR system for closed-loop robot control, as presented in

Table 6. AR technique for HRI and HRC in industrial robotic applications.

Category	Reference	Robot	Medium	AR Content
Accurate robot control	Wang et al. [19]	KUKA KR6 R700	HoloLens	Move cube, rotate cube, and create waypoint
Interactive path planning	Ji et al. [107]	ABB IRB1200	HoloLens	Robot trajectory point, user interface
Pick-and-place selection	Sanna et al. [108]	COMAU e.DO manipulator	HoloLens 2	Explore and select NeuroTags
Safety-aware HRC	Choi et al. [109]	Universal Robot 3	HoloLens 2	Synchronized digital twin, safety information, motion preview
User-aware control	Umbrico et al. [110]	Universal Robot 10e	HoloLens 2 headset and Samsung Galaxy S4 tablet	3D model, arrow control guidance, task instruction, operator feedback
Mobile HRI	Aivaliotis et al. [111]	Mobile robot platform	HoloLens 2	Programming interface, production status, safety zones, and recovering instruction
Multi-user robot teleoperation	Szczurek et al. [112]	CERNBot	HoloLens 2	Video, 3D point cloud, and audio feedback
Engine assembly task	Hietanen et al. [113]	Universal Robot 5	HoloLens, LCD projector	danger zone, changed region, robot status, and control button
Carbon-fiber-reinforced polymer fabrication	Chan et al. [114]	KUKA IIWA LBR14	HoloLens	Virtual robot, workpiece model, and robot trajectory
Gearbox assembly task	Moya et al. [115]	Universal Robot 10	Tablet	3D model animation, audio, PDF file, image, and video
Maintenance manipulation	Liu et al. [116]	AE AIR4–560	HoloLens 2	Text, 3D model, execute task, remote expert

. The user was able to manipulate an industrial robot by planning the posture, trajectory points, and tasks of a virtual robot in the AR environment with gesture commands. Then, Ji et al. [107] integrated human eye blink input and AR feedback in HRI tasks. The human could interactively create and modify robotic path planning according to the different inputs and achieve an AR-based modify-and-preview process. Furthermore, Sanna et al. [108] combined BCI and AR for HRC in assembly tasks. This approach allowed users to visualize different parts to be assembled via AR and guide a robot picking-and-placing selection task via the NextMind. Especially, users in the system could free both hands and assemble objects that required manual work.

Beyond robot control, AR systems can improve human-centric user experience and respond to personalized requirements in industrial scenarios. For instance, Choi et al. [109] focused on safety-aware HRC by providing real-time safe distance and preview of a robot digital twin in an AR environment. Among the systems, the approach of deep-learning-based instance segmentation is used to estimate a 3D object point cloud between a real robot and virtual robot, i.e., the robot’s digital twin. Then, Umbrico et al. [110] presented a user-aware control method for HRC via the headset and tablet AR environment. This method integrated process decomposition, communication, and motion planning modules for indicative task execution, which matched users’ skills to the requirements of production tasks. In addition, Aivaliotis et al. [111] developed an AR software suite for HRC with mobile robot platforms. Humans could define robot motion and navigation goals in a virtual interface. Moreover, the AR suite allowed users to visualize task execution instructions and safety working zones and recover robots from unexpected events. Furthermore, Szczurek et al. [112] proposed a multi-user AR interface for the remote operations of robots. Multiple human operators could teleoperate with the robot through multimodal interaction, including hands, eyes and motion tracking, and voice recognition. The AR interface provided video, 3D point cloud, and audio as feedback for humans. Lastly, Liu et al. [116] introduced an AR system into tasks of human–robot collaborative maintenance. In the robot maintenance task, the decision was generated by a deep reinforcement learning module.

With these explorations, AR-based HRI and HRC systems can be applied in various manufacturing activities, such as welding [117], assembly [118], maintenance, etc. In this context, Hietanen et al. [113] explored a projector–mirror setup and wearable AR interface in a realistic industrial assembly task. The AR user interface could intuitively present a danger zone, changed regions, various control buttons, and robot status to humans. The AR-based HRC reduced the task completion time and robot idle time. Then, Chan et al. [114] evaluated an AR interface for HRC in large-scale, labor-intensive manufacturing tasks. The system allowed a user to specify the robot’s path, visualize the motion, and execute robot trajectories with speech, arm gestures, and gaze. Compared with joystick-based robot control, the AR interface was easy, fast, and more convenient to use. Furthermore, Moya et al. [115] proposed an AR content tool that supported non-expert users with a web interface in HRC. Humans could create, visualize, and maintain AR manuals based on different assets, such as 3D models, audio, PDF files, images, and video. The tool reduced the task load and assisted humans in training robots in assembly tasks. Lastly, Liu et al. [116] introduced an AR-based HRC system into the maintenance process. In the system, a robot could recognize human maintenance requests and execute maintenance tasks from human gestures, while the human was able to interact with the robot and operate auxiliary maintenance tasks.

4.2. Path Planning and Task Allocation

Path planning and task allocation are preconditions before a robot conducts various manipulations. AR techniques can be introduced into this process and optimize robot task execution, as shown in Table 7.

Ong et al. [119] utilized AR to plan collision-free robot paths in an unknown environment. A piecewise linear parameterization algorithm was introduced to interactively generate a 3D curve demonstrated by the user for robot task allocation and execution. Then, dynamic constraints of robots were added in the proposed AR system to resolve discrepancies between the planned and simulated paths of robots [120]. In addition, Young et al. [121] explored occlusion removal and path planning of robots using tablet-based AR systems. The occlusion effect in AR is eliminated by depth correction, whereas the robot task path is planned by a rapidly exploring random tree algorithm which avoids obstacles in the working environment. Furthermore, Solyman et al. [122] proposed a semi-automatic offline task programming method for robots in AR and stereo vision systems. Among the systems, stereo matching algorithms were used to match and overlay virtual graphics on real scene settings, while robot forward kinematics was leveraged to calculate the 3D position of robot arm joints for task programming.

The use of AR systems in robotics allows a robot to adapt, re-plan, and optimize its motion in a timely fashion. For instance, Tavares et al. [123] introduced an AR projector to allow humans to identify the metal part placement in welding tasks. After the calibration of an industrial robot in the projector, the robotic system adapted the part location information in a timely fashion and optimized the welding motions and poses. Furthermore, Mullen et al. [124] investigated using AR to communicate robot inference to humans. With human feedback, robots could re-plan various task planning procedures.

Table 7. AR application in path planning and task allocation.

Category	Reference	Method	Robot	Medium	AR Content
Intuitive robot programming	Ong et al. [119]	Piecewise linear parameterization algorithm for generation of 3D path curve from data points	Scorbot ER VII	HMD	Virtual robot, workspace’s physical entity, and probe marker
Occlusion removal and path planning	Young et al. [121]	Coordinate mapping between robot and tablet and rapidly exploring random tree for path planning	Industrial robot	Tablet PC	Virtual robot and planned path
Semi-automatic offline task programming	Solyman et al. [122]	Stereo matching algorithms for overlaying of virtual graphics on real scenes and interactive robot path planning	6-DoF robot arm	Tablet PC	2D workspace boundary, rendering robot path, and exception notification
Welding task optimization	Tavares et al. [123]	Laser scanner TCP calibration and genetic algorithm for robot trajectory optimization	Welding robot	MediaLas ILP 622 projector	Location where the operator should place and tack weld the metal parts
Inferred goals communication	Mullen et al. [124]	AR-based passive visualization and haptic wristband-based active prompts	7-DoF robot arm	HoloLens	Robot motion goal and text alert
Grasping task planning	Weisz et al. [125]	OpenRAVE for motion planning and RANSAC method for target object localization	BarrettHand gripper	Tablet PC	Scene point cloud, selection button, and object model
Navigation trajectory decision	Chadalavada et al. [126]	Eye-gaze-based human-to-robot intention transference	AGV system	Optoma X320UST projector	Line (robot path) and arrow (robot direction)
Multi-robot task planning	Li et al. [127]	AR for robot teleoperation, reinforcement learning for robot motion planning	Universal Robot 5	HoloLens 2	Task video, control button, and virtual robot model
Adaptive HRC task allocation	Zheng et al. [128]	Human action recognition, object 6-DoF estimation, 3D point cloud segmentation and knowledge graph for task strategy generation	Universal Robot 5	HoloLens 2	Robot states and task instruction
Task allocation under uncertainties	Zheng et al. [129]	Knowledge-graph-based task planning, human digital twin modeling, robot learning from demonstration	Universal Robot 5	HoloLens 2	Virtual robot model and overlapped distance

Table 7. AR application in path planning and task allocation.

Category	Reference	Method	Robot	Medium	AR Content
Intuitive robot programming	Ong et al. [119]	Piecewise linear parameterization algorithm for generation of 3D path curve from data points	Scorbot ER VII	HMD	Virtual robot, workspace’s physical entity, and probe marker
Occlusion removal and path planning	Young et al. [121]	Coordinate mapping between robot and tablet and rapidly exploring random tree for path planning	Industrial robot	Tablet PC	Virtual robot and planned path
Semi-automatic offline task programming	Solyman et al. [122]	Stereo matching algorithms for overlaying of virtual graphics on real scenes and interactive robot path planning	6-DoF robot arm	Tablet PC	2D workspace boundary, rendering robot path, and exception notification
Welding task optimization	Tavares et al. [123]	Laser scanner TCP calibration and genetic algorithm for robot trajectory optimization	Welding robot	MediaLas ILP 622 projector	Location where the operator should place and tack weld the metal parts
Inferred goals communication	Mullen et al. [124]	AR-based passive visualization and haptic wristband-based active prompts	7-DoF robot arm	HoloLens	Robot motion goal and text alert
Grasping task planning	Weisz et al. [125]	OpenRAVE for motion planning and RANSAC method for target object localization	BarrettHand gripper	Tablet PC	Scene point cloud, selection button, and object model
Navigation trajectory decision	Chadalavada et al. [126]	Eye-gaze-based human-to-robot intention transference	AGV system	Optoma X320UST projector	Line (robot path) and arrow (robot direction)
Multi-robot task planning	Li et al. [127]	AR for robot teleoperation, reinforcement learning for robot motion planning	Universal Robot 5	HoloLens 2	Task video, control button, and virtual robot model
Adaptive HRC task allocation	Zheng et al. [128]	Human action recognition, object 6-DoF estimation, 3D point cloud segmentation and knowledge graph for task strategy generation	Universal Robot 5	HoloLens 2	Robot states and task instruction
Task allocation under uncertainties	Zheng et al. [129]	Knowledge-graph-based task planning, human digital twin modeling, robot learning from demonstration	Universal Robot 5	HoloLens 2	Virtual robot model and overlapped distance

The exploration of AR systems provides a feasible solution to intuitively deliver robot task intentions to humans, based on which a user can identify and correct robot task allocation. Zheng et al. [128] leveraged an AR system to show task allocation strategies to human operators. The task planning strategy was generated by knowledge-graph-based artificial intelligence algorithms with a holistic perception of the surrounding environment, including human action recognition, object 6 degrees-of-freedoms (DoFs) estimation, and 3D point cloud segmentation. The knowledge-graph-based task allocation provided an explainable, graphical structure of robot tasks [130], which were easily learned by humans in the AR environment. Further, to tackle task allocation problems facing industrial uncertainties, Zheng et al. [129] investigated human digital twin modeling and robot learning from demonstration algorithms. A deep reinforcement learning algorithm was leveraged to let a human re-plan the robot motion. In this context, humans could assist robots in adjusting task planning for new situations, whereas robots could ensure human safety in unexpected events.

Beyond one fixed robot arm, AR-based path planning and task allocation show advantages in dexterous grippers, mobile manipulators, and multiple robot systems. For example, Weisz et al. [125] developed an assistive robotic grasping system with an AR interface. Users could make grasp tasks online for known and unknown objects in cluttered spaces. To improve robot navigation decisions, Chadalavada et al. [126] used eye-tracking glasses to record human trajectories and let humans choose a safer path or the shortest encounter manner in an AR environment. The projection-pattern-based AR manner was preferred among users and enhanced bi-directional communication in HRI. Then, Li et al. [127] integrated AR and digital twin techniques for multi-robot collaborative tasks. The approach utilized reinforcement learning algorithms to generate path trajectories of multiple robots and preview task planning in the AR environment.

4.3. Training and Simulation

By overlaying virtual instructions onto the real-world robot and work cell, AR enables hands-on learning experiences and simulations. Meanwhile, by visualizing the robot’s movements, operators can understand its interactions with the surroundings and train complex tasks in a safe and adjustable virtual environment. Therefore, AR can be utilized to train operators in the programming and operation of industrial robots and simulate the sequences of movements of robots.

In this context, Sievers et al. [131] designed a mixed-reality learning environment for employees’ skills training with collaborative robots. Specifically, the experimental modular assembly plant, a decentralized learning factory, which consists of reconfigurable autonomous sub-modules, was proposed. After that, Leutert and Schilling [132] proposed a projector-based AR system that can support shop-floor-level intuitive programming and modification of milling robot operations in real industrial scenarios. A large-scale metal workpiece with high production tolerances processing task was conducted, evaluating the practicability of the proposed approach.

Not only the robot programming but the verification of the planned trajectories is also essential. Wassermann et al. [133] developed a system for industrial robots’ workspace simulation and program verification based on augmented reality. Specifically, the environment was constructed by 2D images and 3D point clouds in the AR system, and the user performed task-oriented level programming. Then, the robot executed the safe trajectory after plausibility and collision checking. Similarly, Ong et al. [134] proposed an augmented reality system (including a head-mounted display and a handheld pointer) to simulate the work cell of serial industrial robots, enabling motion planning, collision detection, and plan validation. The users can wear the HMD to check the real-time situation of the work cell and define the tasks by 3D points and paths with the pointer. While Hernandez et al. [135] proposed a high-level augmented reality specifications (HARS) method, which can allow the users to just specify high-level requests to the robots. Instead of defining 3D points and paths, the users just place the virtual objects in target locations in the AR system, and the planner will compute the feasible configuration trajectory for the robot later. Expected visual input and audio information can also be applied. In [136], vision and speech were used for the interaction between the users and the AR to achieve intuitive robot programming.

In addition, to ensure a clear understanding of the robot’s motion intent for human collaborators, Rosen et al. [137] implemented a mixed reality head-mounted display (MR-HMD) system. The proposed HoloLens visualization interface for users showed sequences of virtual arm movement graphics overlaid in the real world, which were compared with the 2D display and no visualization interface. A conducted user study demonstrated the advantages of the MR interface in terms of task accuracy and time cost. Later, in [14], the MR-HMD interface was tested in a more complex pick-and-place task, where obstacle avoidance and conditional reasoning were included, compared with the 2D interface. The results reported the proposed interface could improve the users’ time efficiency, usability, and naturalness and reduce the cognitive workload.

Moreover, AR simulations can also be used to verify robot learning. For example, an AR interface was combined with a semantic-based learning system in [138]. The semantic learning system is driven by knowledge graph algorithms and can generate a human-readable description of the demonstrated task [139]. The user can decide if the new demonstration is necessary by evaluating the learned trajectories in the AR system. Later, Luebbers et al. [140] proposed constraint-based learning from a demonstrations method that allows users to maintain and adapt previously learned skills without providing new demos, based on an AR interface. Specifically, the users can directly modify the existing skills with defined constraints to fit new task requirements in the AR in situ visualizations. The aforementioned applications of AR for training and simulation in industrial robotic applications are summarized in

Table 8. Applications of AR techniques for training and simulation.

Category	Reference	Robot	Medium	AR Content
Human–robot collaboration learning environment	Sievers et al. [131]	Collaborative robots	HMD/Tablet	Experimental modular assembly plant
Industrial robot milling	Leutert and Schilling [132]	Industrial robot	Projector	Virtual processing paths and menu
Workspace simulation and program verification	Wassermann et al. [133]	KUKA KR6	Tablet	2D image, 3D point cloud
Robot work cell simulation	Ong et al. [134]	Industrial robot	Oculus Rift DK2 HMD	Work cell, 3D points and paths
High-level augmented reality specifications	Hernandez et al. [135]	Fetch	HMD	Virtual objects
Augmented trajectories simulation	Quintero et al. [136]	Barrett WAM	HoloLens	Robot-augmented trajectories
Robot motion intent visualization	Rosen et al. [137]	Baxter	HoloLens	Sequences of robot movement
Robot trajectory simulation	Gadre et al. [14]	Baxter	HoloLens	Motion preview
Robot learning verification	Diehl et al. [138]	UR5e	HoloLens/Tablet	Virtual robot, target objects
Robot learned skills modification	Luebbers et al. [140]	Sawyer	HoloLens	Virtual robot and tasks

.

4.4. Teleoperation Control/Assistance

Teleoperation is a technique that involves controlling a robot (actions and movements) remotely with a human operator. It is commonly used in various fields, including industrial settings, healthcare, space exploration, and hazardous environments where direct human presence may be unsafe or impractical. Since AR/MR systems can provide a reasonable visualization of the work environment, they are widely applied in robot teleoperation control and assistance. The applications of AR for teleoperation control/assistance in industrial robotic applications are summarized in Table 9.

For teleoperation, Solanes et al. [141] proposed an AR-based interface to industrial robots to replace the conventional teaching pendant. Especially, the computer-generated graphics in the real environment were shown in the HMD, and users could interact with the graphics with a gamepad to command the robot. The usability tests reported that the proposed interface was more intuitive, ergonomic, and easy to use, improving the speed of the teleoperation task. Later, the interface was implemented for bimanual industrial robot teleoperation in [142]. To meet the orientation and velocity requirements during teleoperation, Pan et al. [143] proposed an AR interface based on an RGB-D camera and handheld orientation teaching pendant for industrial robots. The path of the end effector (EE) was defined by the user by selecting several points with a mouse in the virtual work cell. Then, with the portable teaching device, the orientation and motion speed were also given by the user. In addition to fixed-base robots, AR can also be applied to the teleoperation of mobile manipulators. In this direction, Su et al. [144] proposed a 3D/2D vision-based MR interface for mobile manipulators. Three tasks were conducted to evaluate the proposed system compared with a typical 2D visual display method. The results reported that the MR method can reduce overall task completion time and minimize the training effort and cognitive workload. AR can also be used in a shared control manner in teleoperation tasks. In [145], an AR system was implemented to visualize the spraying process in real-time (i.e., not yet complete, complete, and overdosed) based on a proposed logical approach. According to the visual information from the HMD, the operator decided to move the handheld spray robot to the next target region. Similarly, Wonsick et al. [146] developed a virtual reality interface for robot telemanipulation. For the environment reconstruction in virtual reality, a deep learning approach was utilized to segment objects in the workspace and estimate the object’s 6D pose.

Meanwhile, haptic feedback is also important for teleoperation tasks. Lin et al. [147] compared the effect of haptic feedback and AR for assisting teleoperation. Four kinds of telemanipulation tasks were conducted for eight participants, namely, target location, constraint alert, grasping affordance, and grasp confirmation. The results showed that both the haptic feedback and AR assistance can significantly improve the performance of the teleoperation tasks. While the haptic feedback is suitable for tasks that need a prompt response, AR cues were preferred in system status monitoring. Moreover, the participants preferred reducing their cognitive workload regardless of increasing other efforts. In [148], an MR system with haptic feedback was designed for an industrial robot welding task teleoperation. With this system, the user’s hand movement was directly mapped to the robot’s EE in a velocity controlled manner, and the haptic feedback could guide the operator’s hand in following the conical instructions to align the torch for welding and constrain the movement within a collision-free space.

To make the AR system more accessible, Frank et al. [149] developed a mobile MR interface based on tablets to achieve object manipulation in a human–robot collaborative manner. Specifically, virtual objects were attached to physical objects in the robot workspace in augmented live video, and the operator could command the robot to move the specific object by moving the virtual one to the desired location. Later, Su et al. [150] also proposed a tablet-based AR system for industrial robots. Differently, this system could also ensure collision-free operation based on developed tools, which is an important feature for teleoperation. In terms of collision-free operation, Piyavichayanon et al. [151] proposed a collision-aware AR teleoperation approach based on depth mesh. With an RGB-D camera, the proposed system can reconstruct the robot’s work environment in AR. Then, the user can command the robots to generate collision-free movements based on the integrated collision checking function.

Table 9. AR implementation for teleoperation control/assistance in industrial robotic applications.

Category	Reference	Robot	Medium	AR Content
Industrial robot teleoperation	Solanes et al. [141]	6R industrial robot	HoloLens	Virtual work environment
Bimanual robot teleoperation	Garcia et al. [142]	Two industrial robots	HoloLens	Working environment simulation
Industrial robot teleoperation with orientation and speed requirements	Pan et al. [143]	ABB IRB120	PC screen	Robot work cell
Mobile manipulator teleoperation	Su et al. [144]	Mobile manipulator	HTC Vive	Virtual work scenarios
Spraying task assistance	Elsdon and Demiris [145]	Handheld spraying robot	HoloLens	Virtual processing paths and menu
Comparison of haptic and AR cues for assisting teleoperation	Lin et al. [147]	KINOVA Gen 3	HTC Vive	Robot workspace
Robotic welding with AR and haptic assistance	Su et al. [148]	UR5	HTC Vive HMD	Virtual scene
Object manipulation	Frank et al. [149]	Collaborative robots	Tablets	Virtural objects
Industrial robot teleoperation	Su et al. [150]	Industrial robot	HTablet	Virtual robot workspace
Collision-aware telemanipulation	Piyavichayanon et al. [151]	7-DOF manipulator	Mobilephone	Virtual robot workspace

Table 9. AR implementation for teleoperation control/assistance in industrial robotic applications.

Category	Reference	Robot	Medium	AR Content
Industrial robot teleoperation	Solanes et al. [141]	6R industrial robot	HoloLens	Virtual work environment
Bimanual robot teleoperation	Garcia et al. [142]	Two industrial robots	HoloLens	Working environment simulation
Industrial robot teleoperation with orientation and speed requirements	Pan et al. [143]	ABB IRB120	PC screen	Robot work cell
Mobile manipulator teleoperation	Su et al. [144]	Mobile manipulator	HTC Vive	Virtual work scenarios
Spraying task assistance	Elsdon and Demiris [145]	Handheld spraying robot	HoloLens	Virtual processing paths and menu
Comparison of haptic and AR cues for assisting teleoperation	Lin et al. [147]	KINOVA Gen 3	HTC Vive	Robot workspace
Robotic welding with AR and haptic assistance	Su et al. [148]	UR5	HTC Vive HMD	Virtual scene
Object manipulation	Frank et al. [149]	Collaborative robots	Tablets	Virtural objects
Industrial robot teleoperation	Su et al. [150]	Industrial robot	HTablet	Virtual robot workspace
Collision-aware telemanipulation	Piyavichayanon et al. [151]	7-DOF manipulator	Mobilephone	Virtual robot workspace

5. Discussion

Although AR techniques have been considered a promising paradigm for improving human operators’ situation awareness and visual feedback before and during task execution, several limitations and challenges should be tackled and emphasized. In this section, we discuss the general limitations and challenges of the use of AR in robotic applications from the view of transparency, hardware limitations, safety, and accuracy issues. In addition, we also summarize several potential perspectives that can be improved and addressed in future works on AR-related robotic applications.

5.1. Limitations and Challenges

Immersion and Transparency: AR can provide the user with improved visual feedback in robotic applications. However, immersion and transparency remain critical issues. Firstly, the FoV from the AR display (monitor, HMD, projector) could restrict the amount of information that is accessible to the operator. Moreover, the non-ergonomic interaction during HRI and HRC tasks will affect the task completion performance and impose high cognitive workloads, both mentally and physically. Hence, more transparent and intuitive interfaces should be implemented considering the possible customized or personalized requirements. Additionally, the absence of force and haptic feedback sensing is another limitation [152,153]. Haptic feedback integration into AR-related robotic systems could further provide the operator with informative feedback, such as texture, force, friction, etc., which would assist the user in decision-making and operations.

Hardware and Communication Efficiency: The computation capability in several applications could be limited by the computation capability of the hardware configuration. For instance, most commercial general-purpose AR devices possess only fundamental visualization and basic computing and graphics rendering capabilities, making it challenging to achieve the real-time rendering of the deformation of non-rigid objects and complicated applications. Furthermore, the low communication quality can lead to significant latency in robotics systems, particularly notable in teleoperated systems, thereby affecting the synchronization between the human operator and robotic systems during remote and teleoperation control tasks.

Accuracy and Safety: The accuracy of registration/alignment between the hologram model or the computed-generated image on the physical environment could cause misoperations, thus causing failures in task completion, for example, in fine manipulation, and delicate surgical operation tasks. For the OST-HMD, the hologram model can experience a displacement of 3–5 cm when the operator moves significantly. In addition, considering the latency and inaccurate registration, the safety issue between the robotic system and the environment in dynamic and unstructured environments is challenging and should be addressed. Therefore, safety enhancement strategies should be developed, such as collision avoidance, contact force control, manufacturing process monitoring, etc.

Simulation to Real Adaptation: AR-based simulation and digital twin systems have found their implementation in both medical and industrial domains. For example, a representative application of simulation to real (Sim2Real) transfer proves beneficial in tasks such as robot programming and trajectory planning for robot-assisted operations. However, considering the inherent limitations and challenges, such as the errors in modeling and the dynamics of the physical environment, and the noise/disturbance, the direct transfer of results from simulations to the real world still has a lot of obstacles. Appropriate strategies should be considered and implemented to adapt the results from simulations to physical scenarios.

5.2. Future Perspectives

Hardware System: The comfort level of utilizing AR systems remains a primary issue faced by current AR applications, which encompasses two aspects: visual comfort and wearing comfort. To ensure optimum visual experience, the optical system must offer various features, including a suitable field of view, well-matched depth imaging with high resolution, and ample contrast, and brightness. The attainment of wearing comfort relies on a more compact optical structure design to alleviate the burden on the head, low-power processors to reduce discomfort caused by heat, and advanced battery systems to balance capacity and portability. At the optical design level, the introduction of adjustable lenses is expected to lower visual accommodation conflict and relieve dizziness. The polarization film is also capable of providing a solution to the compact size problem, while free-form optics can provide sufficient FoV. Regarding the signal processing system, customized, dedicated chip solutions are expected to solve the balance between AR system performance, power consumption, and heat dissipation. Furthermore, with the participation of more consumer electronics companies, the costs and prices of AR systems are expected to decrease, which would accelerate the development of AR technology.

Interaction Modalities: A handheld controller, voice, gaze, and gestures are the most commonly used interaction methods in current AR-based systems. However, a handheld controller cannot be used in all-weather wearable applications, and the use of both hands makes it impossible to execute other operations in AR applications. Gesture operation requires real-time capture of hand movements by the camera, which has accuracy issues and can affect social acceptance of public gesture operations with privacy risks. Voice operation is susceptible to environmental noise and can disturb others in shared spaces. Eye-tracking technology has been proven to be faster in efficiency than pointing with fingers, and gaze-driven interfaces based on eye tracking have been well applied in tasks such as target selection and information placement. However, due to the current limitations of the interaction interface style, the gaze-driven experience is poor in complex interaction tasks. Optimizing the interface settings, combining user operating habits, and introducing dynamic UI and sticky capture features can improve and enhance the interactive experience based on eye-tracking technology, leading to significant improvements in AR interaction.

Multimodal Information to Improve the Perception Capability: As optical resolution and FoV improve, virtual objects in AR systems will become increasingly realistic. However, relying solely on visual feedback still leads to a disconnect between virtual objects and the real environment. Enhancing AR with additional audio, haptic, temperature, or olfactory feedback can provide users with a more immersive and natural interaction in AR environments. For example, haptic feedback can be used to create a sense of touch and improve the user’s spatial awareness, while auditory feedback can be used to indicate distance or direction. It is foreseeable that in the future, AR systems will introduce multimodal information feedback to provide alternative feedback modes that are better suited to individual needs, increase usability for users with sensory or cognitive impairments, and provide users with greater perception beyond the limits of the visual field.

6. Conclusions

This paper provides a review and summary of the recent advancements in robotic applications using AR technology. The primary research focuses of this paper are on the implementation of AR in medical and industrial scenarios. Additionally, this work offers a summary of the popular robotic platforms, medium, and AR system contents. Specifically, we classified AR in the medical robot context into four subsections according to their application: preoperative and surgical task planning, image-guided robotic surgery, surgical training and simulation, and telesurgery. For industrial robotic applications, this paper investigated and discussed the recent advancement in human–robot interaction and collaboration, path planning and task allocation, training and simulation, and teleoperation control/assistance. In the meantime, the limitations and challenges that exist in robotic applications are emphasized, such as transparency, interfaces, hardware, and safety issues. Following that, future perspectives are summarized to improve the application of AR in the field of robotic systems. This review paper aims to provide a reference for the future application and development of AR in the field of robotic systems.