Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

Abstract

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

1. Introduction

Robotic teleoperation is a technological approach allowing a human operator to perform a task by controlling a device or machine at a remote location. The distance can vary from the micron level (micro-manipulation) to the kilometer level (space applications). Human operators manifestly have the ability to adapt to the irregular environment and employ a variety of strategies to place the telerobots in favorable conditions, despite the complexity involved in remote sites.

Teleoperation provides an enormous advantage over other robotic controlling methods. A teleoperation system benefits from taking advantage of both human and robotic capabilities. A typical teleoperation system consists of a master device and a slave device. A human operator manipulates the master device and issues commands to the slave device, which is manipulated to achieve a certain goal. A robot manipulator that follows the motion of the master arm in real time rather than being pre-programmed is an example. The kinematics of the slave device can be identical to the kinematics of the master device, or they can be different. It can also be specifically scaled or even completely customized according to the task and requirements.

Despite the emergence and growing application of artificial intelligence, autonomous robots cannot reach the same level of intuition and reasoning as humans. While practical control algorithms can be used to process complex interactions, and kinematic aspects are better suited to humans, who perform such tasks effortlessly in their daily lives [1]. One of the advantages of teleoperation is that it empowers the robot to fulfill tasks in unstructured or hostile environments where situational perceptions, cognitive abilities, and professional experience have a predominant impact on task execution [2]. Therefore, for tasks requiring a variety of human judgments and professional skills, it is practical and feasible to utilize teleoperation to accomplish the goals.

In general, application scenarios of teleoperation could include deep-sea robotics applications [3], space exploration [4,5], de-mining operations, search and rescue in disasters, inspection in restricted spaces [6], medical robot-assisted surgery [7], hazardous material treatment, and micro-manipulation or minimally invasive surgery, among many possibilities. These applications fall into areas of either high risk to humans or high need, where human capability is not available or cannot be present for other reasons. Space or underwater applications are examples of high-risk scenarios, while telerobotic surgery in remote locales is an example of a high-need application, where a skilled surgeon may not be locally available [8].

More specifically, teleoperation is suitable for tasks meeting the following set of criteria:

• Tasks to be performed in unstructured and dynamic environments, such as deep-sea exploration and space applications.
• Tasks involving operations in hazardous situations where human health is severely harmed. For example, in the Fukushima nuclear disaster, there was a high demand for emergency treatment and rescue. Mining scenarios are also increasingly typical.
• Tasks requiring dexterity, especially the coordination between hands and eyes. Medical surgery performed with remote robots is a typical example.
• Tasks requiring object recognition, obstacle detection, or situational awareness, for example, inspection in confined spaces.

This literature review, unlike previous ones, specifically focuses on XR applications applied to human-in-the-loop telemanipulation and telemanufacturing scenarios. The objective of this review is to categorize the recent literature on XR-enhanced telerobotic frameworks for telemanipulation and telemanufacturing tasks, spanning the period from early 2016 to late 2023. Our aim is to comprehensively examine the evolution of this research field, pinpoint the primary areas and sectors where it is currently applied, elucidate the adopted technological solutions, and emphasize the key benefits attainable through this technology. In this context, we address substantial challenges and opportunities, with a specific focus on hardware and software integration to ensure stable and efficient communication and system adaptation for various scenarios. Our review delves into telemanipulation and tele-welding scenarios within industrial applications, offering a unique perspective on the integration of XR technology. Notably, our work innovatively summarizes how XR technology can enhance remote welding tasks, providing fresh and valuable insights in this domain. Additionally, our review recognizes the specific challenges encountered in integrating teleoperation systems with diverse hardware platforms and emphasizes the necessity for detailed information on the design of software architectures for interoperability. Through our meticulous analysis, we aspire to contribute significant knowledge to the evolving landscape of XR-enhanced telerobotic systems.

The scope of our work is focused on telemanipulation and telemanufacturing tasks in hazardous conditions. To achieve this, we conducted a search for papers published during a seven-year period between 2016 and 2023 in relevant resources for extended reality and robotics, including IEEE Xplore, MDPI, ACM Digital Library, and Science Direct. Additionally, we present and classify the application context of the reviewed articles into two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. We conducted a thorough literature review focusing on scholarly databases, academic journals, conference proceedings, and reputable online repositories related to the field of extended reality and robotics. Our searches were comprehensive and included keywords such as human–robot interaction, teleoperation, extended reality, virtual reality, mixed reality, and augmented reality and used Boolean operators to refine the search results. We ensured that the selected resources were recent and peer-reviewed, emphasizing studies and articles published within the last 7 years to maintain relevancy and accuracy.

This section provides a comprehensive overview of the article’s focus on immersive and intuitive telemanipulation frameworks using XR technologies. The rest of the article is structured as follows. Section 2 presents the prerequisites for effective robotic teleoperation, focusing on natural motion retargeting and multimodal feedback. Section 3 outlines the research methodology utilized in the literature and provides initial insights into intuitive and natural teleoperation. Section 4 delves into XR-enhanced telerobotic frameworks for industrial applications and classifies the application context into two groups: MR-enhanced robotic telemanipulation and MR-enhanced robotic tele-welding. Section 5 outlines the challenges and discusses future opportunities in the field of research. Finally, Section 6 summarizes the key findings from the reviewed works and emphasizes the integration of XR, particularly MR, in telemanipulation systems and remote robotic welding.

2. Categories of Robotic Teleoperation

2.1. Collocated and Separated Teleoperation

Based on the relative locations of the human and robot, robot teleoperation can be categorized into collocated and separated teleoperation. In collocated teleoperation [9,10], the operator can directly observe the robot and its working environment in the same space, and a visual feedback system is not necessary; augmented reality technology can be used in collocated teleoperation to provide users with informative virtual content superimposed on the user–robot shared space [11]. Collocated robot teleoperation presents users with a robotic platform that shares a physical environment with the users, who have a natural and clear 3D view of the robot’s workspace [12]. However, although humans and robots physically share the same space, users can often only observe from a third viewpoint alongside the robot and cannot inspect the robot’s environment from the robot’s main viewpoint.

Spatially separated teleoperation is used in a wider range of scenarios where the user and the robotic system are far apart or where the user must be separated from the robot for safety reasons and cannot directly view the robot’s movements. This type of robot teleoperation system typically requires visual and haptic feedback systems to show the user how the robot is working at a distance, allowing the operator to make timely adjustments. However, in the separated telemanipulation, users can typically only perceive the robot space through 2D camera streams and have difficulty synthesizing the 2D information collected by the robot with contextual knowledge of robot movements in its working space [13].

2.2. Ego-Centric and Eco-Centric Teleoperation

Depending on the operator’s perspective in observing the manipulation process, robot teleoperation can be divided into ego-centric and eco-centric teleoperation. In ego-centric robotic teleoperation, the user observes the remote manipulation process from the same perspective as the robot and perceives himself as one with the robot [14]. The operator controls the remote robot with human-level dynamics through immersive and multimodal control–feedback schemes. Ego-centric robotic teleoperation transfers human-level manipulation and proficiency in telemanipulation applications and equips robotic systems with human-level motor skills. The user does not need to consider the mapping relationship between the robot’s actions and the user’s input commands [15,16].

Eco-centric teleoperation usually involves inspecting the robot and its interactive environment from a third-person perspective while manipulating the robotic system with a broader view of observation (Figure 1) [17]. With the help of mixed reality technology, users can still experience coexistence with the robot in the same virtual space but without the immersion and intuition of being one with the robot.

2.3. Robotic Mechanism-Based and Motion Sensor-Based Teleoperation

Robot teleoperation can be divided into robotic mechanism–based and motion sensor–based teleoperation, according to the command input device used. Teleoperation systems that use robots as master mechanisms usually apply robotic devices of the same or different structure as the remotely controlled robot as the user command input system [18]. Using the same robots to form the master–slave architecture for teleoperation, motion synchronization can be accomplished by simple joint space mapping, and the user can directly visualize the pose of the slave robot by observing the master robot being controlled [19,20,21]. When using robots of different configurations as master and slave components for teleoperation, motion synchronization can be achieved using joint space motion remapping or TCP motion in the Cartesian space, and a scaling factor can be set to match the working space ranges of the master and slave robots.

The motion sensor–based robotic teleoperation platforms eliminate the need to introduce expensive and complex robot platforms as human–robot interaction command input devices. Commercial optical and IMU-based sensors, depth cameras, motion controllers, and gloves are often used to capture user motion information [22]. Compared to the robotic master devices, motion sensor–based robotic teleoperation platforms usually have a larger space for motion tracking, the user’s movement is not constrained by the mechanical structure, and the relatively lower price reduces the overall cost of the system [23,24].

3. Intuitive and Natural Teleoperation

In conventional telemanipulation systems, the user controls the remote robotic system with a joystick, gamepad, keyboard and mouse, or 3D mouse and simultaneously receives visual feedback from 2D displays [25,26]. Robot control is not intuitive and natural to the user [27,28]. The mismatch between the range of user control space and the limits of input device workspace can increase the difficulty of telemanipulation and lead to poor operation [29,30]. Another disadvantage of typical telerobotic systems is the lack of depth perception due to monocular, 2D visualization of the remote site [31,32,33], limiting operator performance [34,35] and any feelings of immersion and telepresence in the remote workspace [36,37].

Intuitive human–robot interaction schemes are needed so the operator can easily guide the robotic system using natural motion. A conventional intuitive teleoperation platform was developed in [38] (Figure 2). The user manipulates the seven degree-of-freedom (DOF) master robot to intuitively drive the slave robot at a distance as if they were directly operating the slave robot. This platform requires a symmetrical relationship between the input (master) and output mechanisms (slave). However, for 2D visualization of the telemanipulation process, the user lacks a sense of presence. Monitoring the interaction process with a 2D monitor on the user side can also distract the user from effective robot control activities. In addition, using the isomorphic master robot as the motion input device on the user side in telemanipulation systems often leads to a significant hardware investment.

Several noteworthy studies and frameworks have effectively implemented natural motion retargeting and multimodal feedback. One notable example is the work of Akshit et al. [39], in which the authors developed a telerobotic system integrating natural motion retargeting algorithms, significantly enhancing the operator’s control precision. Another novel interface that features a motion retargeting method for effective mimicry-based teleoperation of robot manipulators was introduced in [40]. This framework allows novice users to interact with the remote robotic manipulators effectively and intuitively. Additionally, the framework proposed by [41] successfully incorporates multimodal feedback, combining visual, auditory, and haptic cues, leading to improved user immersion and task performance. These studies demonstrate the practical implementation of natural motion retargeting and multimodal feedback in XR-enhanced telerobotics, emphasizing their impact on enhancing user experience and task efficiency.

With the inevitable limitations in the field of view of remote on-site 2D cameras and the poor quality of the visual feedback sent back from the robot’s workspace, users are often unable to achieve a level of situational awareness sufficient for effective and safe remote manipulation tasks [42]. The operator is unable to perceive the physical environment of the worksite in 3D due to the lack of stereoscopic depth information. Typical monocular visual feedback strategies greatly reduce the efficiency of remote-controlled robotic operations [43,44].

More specifically, the lack of depth information from 2D vision causes ambiguity about the spatial positions of objects in static images. Operators cannot directly perceive depth in monoscopic vision (MV) streams and have difficulty in accurately determining the distance between the robot end-effector and workpieces. Thus, the user perception from 2D visual feedback makes it insufficient to accomplish remote tasks effectively. Hence, the success and efficiency of teleoperated tasks mainly depend on operator skill and proficiency [45,46].

Multi-camera arrangements can provide the relative position of objects and allow users to view and judge the distance from various perspectives, compensating for the limits of a single monocular 2D camera. However, as telerobotic systems become more complex, avoiding robot collisions with surrounding objects with only multiple 2D feedbacks inevitably leads to low robot control efficiency and increased operator workload.

Different levels of depth perception in telemanipulation can improve task performance for various manipulation and grasping tasks [47,48,49]. Using binocular cameras can provide additional depth information to assist operators. Stereoscopic vision and point cloud are two methods of providing stereo-depth information and have been applied in various telerobotic systems for remote vision [50,51,52]. Binocular visual feedback for robotic teleoperation has been widely studied on unmanned aerial vehicles and mobile robots [53,54]. Leveraging depth perception in remote robot teleoperation tasks for general object manipulation is not well studied on mobile manipulators. In addition, neither multi-camera nor binocular camera setups allow the user the flexibility to continuously change the viewing angle to observe the remote work without introducing additional mechanisms on the robot side [55].

Incorporating haptic feedback in telerobotics offers numerous advantages in creating immersive and natural teleoperation schemes, notably enhancing the user’s tactile experience and augmenting immersion [56]. This technological integration enables users to perceive and manipulate objects remotely with a heightened sense of realism. Such realistic feedback proves invaluable in tasks necessitating a keen sense of touch, such as delicate surgical procedures or the handling of fragile items [57]. Additionally, haptic feedback serves to enhance the operator’s spatial awareness, offering vital cues about the robot’s environment and the forces it encounters. This heightened perception significantly contributes to the precise control and manipulation of the robotic system. Moreover, haptic interfaces alleviate the cognitive burden on operators by delivering intuitive, natural feedback. This intuitive interaction allows operators to concentrate more on the task at hand, reducing their dependence on interpreting visual information [44].

Nevertheless, the integration of haptic feedback into telerobotics is not without challenges. One of the primary hurdles involves ensuring real-time responsiveness, as any delay in haptic feedback can disrupt the user’s control and sense of immersion [58]. Achieving high-fidelity haptic feedback demands advanced hardware components and sophisticated algorithms, thereby augmenting the overall complexity and cost of the system. Additionally, the design of haptic interfaces catering to a diverse array of tasks and user preferences poses a formidable challenge [59]. Different applications may require varying levels of force, precision, and feedback sensations, necessitating a delicate balance to accommodate these diverse requirements effectively. Addressing these challenges is imperative for harnessing the full potential of haptic feedback in telerobotics, ensuring seamless integration and optimal user experience in various telerobotic applications.

In the process of integrating robotic teleoperation systems with diverse hardware platforms, several specific challenges were encountered. Ensuring compatibility between various hardware components, such as sensors, actuators, and communication interfaces, was a primary challenge due to differences in data formats, protocols, and communication speeds, which posed hurdles in achieving seamless integration [60]. Additionally, interfacing multiple sensors, including cameras, depth sensors, and force/torque sensors, required careful consideration and calibration to integrate data from different manufacturers in real time, ensuring accurate and coherent information for operators [61]. Communication delays between different hardware components could disrupt the real-time feedback loop, impacting teleoperation tasks that demand quick response times. Minimizing latency and optimizing communication protocols were crucial in maintaining stable and efficient communication [62]. Furthermore, coordinating control signals sent to different hardware modules, such as manipulator arms and grippers, necessitated precise synchronization to prevent erratic movements or inaccuracies, highlighting the importance of synchronization mechanisms for ensuring precise and coordinated actions.

To overcome these challenges and ensure stable and efficient communication in the integration of teleoperation systems with diverse hardware platforms, several strategies were employed. Implementing standardized communication protocols, such as ROS (Robot Operating System) messages, facilitated interoperability between different hardware components, ensuring consistent data exchange and compatibility across various devices that use different programming languages. Designing a unified software architecture with ROS and interface engines, such as Unity 3D, which abstracted underlying hardware complexities, was essential. This design allowed for the implementation of high-level control algorithms and user interfaces without the need to consider hardware-specific details, thereby promoting interoperability and ease of development [63,64]. In summary, the integration of ROS and Unity-based software architectures enable easy expansion and adaptation to new devices and sensors used in telemanufacturing aspects, such as remote robotic welding.

Teleoperation systems cope with dynamic environments by integrating advanced technologies such as sensor fusion, artificial intelligence, machine learning, adaptive algorithms, and human supervision [65,66]. These systems use various sensors to gather real-time data and employ AI algorithms to process and adapt to dynamic changes. Environmental mapping techniques, like simultaneous localization and mapping (SLAM), enable the system to create and update detailed maps, aiding in adaptive navigation. Dynamic path planning algorithms calculate optimal routes in real time, considering obstacles and changing environmental conditions [67]. Additionally, collaborative teleoperation strategies leverage human expertise for high-level decision making while allowing robots to execute tasks based on human guidance, ensuring adaptability and safe operation in complex and unpredictable environments. Human supervisors also play a vital role in remotely monitoring operations, providing contextual understanding and intervention capabilities when necessary [68].

In adapting teleoperation technology for telehealth scenarios, particularly in remote regions, several key strategies can be employed. Firstly, enhancing the robustness of internet connectivity through satellite-based solutions and low-bandwidth protocols can ensure real-time communication between remote robots and healthcare professionals [69]. Developing affordable and portable robotic platforms, customized for telehealth applications, can address cost and accessibility issues. Moreover, comprehensive training programs for local healthcare providers are crucial to ensure effective operation and maintenance of teleoperation equipment [70]. However, these implementations are not without challenges. Limited infrastructure, such as unstable internet connectivity and power supply, poses a significant hurdle. Finding cost-effective solutions, considering the financial constraints of remote regions, is crucial for the widespread adoption of teleoperation technology [71]. Addressing training gaps and ensuring cultural acceptance are equally vital. Ethical considerations, particularly related to patient consent and data security, require careful attention. Moreover, establishing efficient maintenance and technical support systems in remote regions, where specialized technicians are scarce, is essential to guarantee the continuous and effective operation of telehealth robots. By proactively addressing these challenges, teleoperation technology can be successfully adapted to transform healthcare delivery in remote areas [72,73].

This section on intuitive and natural teleoperation highlighted two important requirements for an effective and intuitive teleoperation platform. First, natural motion retargeting, or the use of natural human behavior to teleoperate robotic systems, effectively reduces training costs and grants human-level dynamics to telerobotic systems; it is an important component of natural and intuitive human–robot interaction systems. Second, sufficient multimodal feedback, particularly visual feedback, is pivotal in interactive telemanipulation systems, providing depth information and enhancing situational awareness to enable the operator to know the dynamic situation in the robot’s workspace; it is also critical in interactive telemanipulation systems. The human operator in the control loop team must be aware of the relative positions of the robot end-effector and the workpieces in the workspace to make decisions and avoid collisions or damage to the robotic system, the working area, and/or the environment.

4. Mixed Reality for Human-Robot Interaction

4.1. Human-Robot Interaction

Human–Computer Interaction (HCI) delves into the study of how individuals interact with computers and the extent to which computers are designed for successful human interaction [74]. This discipline encompasses the design, evaluation, and implementation of interactive computing systems tailored for human use, along with the examination of significant phenomena associated with them. HCI fundamentally revolves around comprehending the intricate relationship between humans and computers and crafting computer systems that facilitate meaningful and efficient interactions [75]. It places a strong emphasis on enhancing the usability and user experience of computer interfaces, ensuring technology is accessible, efficient, and enjoyable for users [76]. Human–Robot Interaction (HRI) constitutes a multidisciplinary field focused on comprehending, designing, and evaluating interactions between humans and robots [77]. This domain involves the comprehensive study, design, and analysis of behaviors, interfaces, and systems geared towards robots interacting with humans across diverse contexts [78]. The overarching objective of HRI is to develop robots capable of collaborating effectively and intuitively with humans, offering assistance or services in a variety of settings, including homes, workplaces, healthcare facilities, and public spaces [79].

4.2. Reality–Virtuality Continuum

Extended reality (XR) encompasses the mutual embedding and fusion of physical and virtual scenes, along with the human–computer interaction occurring in the generated environment [80], where X represents the degree of interpenetration and integration of real scenes and virtual content in spatial or immersive computing technologies, involving the user’s sense of presence and acquisition of perception [81]. XR spans the entire spectrum from entirely realistic to fully virtual, encompassing all representative forms and possible variations of AR, VR, AV, MR, and other interpolations between reality and virtuality [82,83]. The reality–virtuality continuum presented by Milgram et al. [84] outlines the XR ecosystem and the relationship between AR/VR/AV/MR and fully real and virtual environments, which are schematically illustrated in Figure 3.

Augmented reality (AR) overlays informative virtual 3D graphics onto the physical world and allows real-time interaction with the 3D graphics, enabling the user to reach into the augmented world and manipulate 3D objects directly as if they were real-world objects [85]. The advantage of AR is it superimposes important cues on physical objects or hovers the display in real space to indicate the user’s potential or actual future actions, or the robot’s motion planning, while eliminating distractions caused by typical display methods [86]. AR–HRI interfaces provide users with an exo-centric view of the robot and its surroundings and allow operators to maintain situational awareness of the robot and ensure intuitive interaction and communication between the human and robot using multimodal interfaces [87].

Virtual reality (VR) uses fully immersive computer-generated graphics with scenes and objects that appear to be real to completely replace the physical surroundings in which the user is located [88]. In the virtual environment, the users perceive immersion in an identical or completely different scene from the physical surroundings and perceive an imaginary environment with muti-modalities of visual, acoustic, and haptic information [89]. VR allows users to move around in the scene and manipulate the virtual objects by using wireless controllers or haptic robotic structures as input sources.

Mixed reality (MR) technology involves the merging of physical and imaginary spaces and does not occur exclusively in the physical or virtual world [90]. MR enables real-time visualization and interaction between physical and digital content [91] and takes full advantage of the visual information of real scenes and the three-dimensional immersion and interaction provided with virtual cues [92]. Head-mounted display (HMD)-based mixed reality can be achieved in two ways: either by displaying the camera captured video feedback of the real world in the HMD or by allowing the user some direct visibility of the real world in the HMD [93,94].

Augmented virtuality (AV) is a fully immersive computer-generated virtuality-dominated environment enhanced by sensory data from physical environments [95]. AV keeps the virtual environment central but superimposes the real-world elements on the virtual content [96]. AV is also classified as a subset of MR. The term AV is less commonly used in the literature for immersive telerobotic interfaces and is often generally referred to as MR.

In MR, the hierarchy of reality spans from partial sensory input to a proximity infinitely close to immersive reality [97]. The reviewed MR-enhanced intuitive and immersive teleoperation schemes in this article fall into the category of partially superimposing real-world visual inputs onto computer-generated virtual content.

4.3. Mixed Reality–Enhanced Telemanipulation

4.3.1. Mixed Reality–Enhanced Intuitive Telemanipulation

In recent years, mixed reality technologies have been widely applied in the field of robotics, and these research directions can be divided into three categories according to application aspects: (1) human–robot interaction (HRI): immersive teleoperation, intuitive telemanipulation, collaborative interaction, wearable robots, haptic effects, and virtual devices; (2) medical robotics: robot-assisted surgery (RAS), prosthetics, and robot-assisted rehabilitation and training devices; (3) robot motion planning and control: trajectory generation, robot programming, simulation, and manipulation.

Table 1. Differences between this article and comparable studies recently introduced in the literature on MR-enhanced robotics.

Reference	Year	Topics Focused on Recent Papers	Differences with This Article
[98]	2021	Review of the current state of VR/AR solutions in HRI and HRC Solutions summarized in four main categories: operator support, instruction, simulation, and manipulation Referenced citations were published between 2010 and 2019	Review of novel XR-enhanced robotic approaches for intuitive remote teleoperation applications This article focuses on XR technologies in robotic telemanipulation and telemanufacturing scenarios Articles presented were published from 2016 to 2023
[99]	2022	Review of existing AR-enhanced HRI and robotic interfaces Approaches presented in two main categories: augmenting robots and augmenting surroundings	This article reviews XR-enhanced telerobotic solutions, including the use of VR, MR, and AR The scope of this work is focused on telemanipulation and telemanufacturing tasks in hazardous conditions
[100]	2023	Review of existing VR/AR/MR tools and frameworks in HRC using game engines Recent literature on software tools categorized into communication tools and interaction tools Applications presented in four main categories: social robotics, programming of industrial robots, teleoperation, and human–robot collaboration	This article presents XR systems that extend the user’s reality and provide a more immersive and intuitive interface This article focuses on both software and hardware aspects This work classifies the industrial application context of the reviewed articles into two groups: XR-enhanced telemanipulation and XR-enhanced tele-welding

outlines the main differences between this article and previous research in the realm of MR-enhanced robotic manipulation.

MR interfaces enhance operator immersion through realistic 3D visualizations and interactive elements. To minimize fatigue, haptic feedback and intuitive gestures are integrated, enhancing the user experience. Additionally, optimizing rendering algorithms is essential. Techniques like foveated rendering, where high-quality graphics are focused on the user’s gaze, ensure a smooth experience while conserving computational resources [101]. MR has been applied to robotic teleoperation systems to optimize operator immersion and enhance user perception of the remote side to enable immersive robotic teleoperation (IRT) [102,103,104]. A three-dimensional virtual world similar to the slave side can be simulated through MR and displayed to the user on the master side. By implementing MR as a human–robot interaction interface (HRI), the user experiences a physical presence in the remote environment and co-existence with the robotic platform via MR subspace, while guiding and monitoring the robotic platform at the local user space [105,106]. MR interfaces blend virtual elements with the real world, providing operators with an immersive 3D visualization of the remote environment, such as a welding site [107]. This immersion allows operators to perceive the task space in depth, enhancing their understanding of the work area and minimizing cognitive load. By feeling present in the remote environment, operators can focus more effectively on their tasks, reducing distractions and mental fatigue [108]. MR-based user interfaces with natural control commands minimize fatigue by utilizing human instincts, such as motion and gestures, to interact with complex robotic systems [109]. MR-enhanced teleoperation allows direct mapping of control commands and actions between the user and the robot, presenting the potential for performance enhancements. It serves as an intermediary for integrating imitation-based motion mapping and 3D visualization techniques [110].

Recent research on MR-enhanced teleoperation systems has primarily concentrated on gathering demonstrations for robotic learning, incorporating haptic feedback and other sensory inputs, developing immersive manipulation, integrating AI and machine learning, and addressing issues related to poor virtual transparency [111,112,113]. However, these telerobotic systems do not fully exploit the potential performance enhancements provided by using an MR subspace as an intermediary for the integration of imitation-based motion mapping and 3D visualization mapping. Figure 4 shows the typical diagram and communication structure of the XR-enhanced intuitive and immersive teleoperation systems. The XR-enhanced robotic teleoperation system consists of four parts: (1) the XR-enhanced human–robot interaction unit; (2) the XR scene for vision and motion mapping; (3) the remotely controlled robot working unit; and (4) the bi-directional communication links. Dennis Krupke et al. [114] developed an MR-enhanced heterogeneous robot telemanipulation system that presents the real robot working space and the corresponding virtual scene presented to the operator in the robot telemanipulation architecture for immersive and intuitive remote grasping. The pose of the robot replica in MR is synchronized with the current pose of the physical robot via messages. The communication between the robot and MR scene is maintained via the ros-bridge. A virtual screen on the left wall augments the virtual scene by displaying an image stream from the camera mounted above the physical robot. However, intuitive teleoperation is only applied to the robotic hand instead of the arm-hand system in its entirety, limiting the workspace and overall application significantly.

In [111], researchers from the University of California, Berkeley, built an MR-based teleoperation system on a PR2 robot through imitation learning. The system allows the operator to teleoperate robots to perform complicated tasks naturally and intuitively. In the proposed teleoperation system, a robot is controlled at a distance by the human operator through an MR-based telerobotic interface with overlaid information, which is an effective approach to collect high-quality demonstrations for training the robot. Imitation learning techniques allow the robot to imitate human behavior and acquire skills through perceiving human demonstrations aiming at performing specific tasks.

An MR control room for dynamic vision and movement mapping between the operator and dual-arm robotic agent can be developed for tele-manufacturing (Figure 5) [115]. The multiple monocular sensor displays and motion mapping approach via MR outperforms telerobotic platforms with direct camera feeds. The control room has objects and controls floating in space, which allows the user to perform movements relative to 3D markers to command the remote-controlled robot. However, the researchers did not determine if the immersive experience led to performance improvements compared to conventional 2D HRI platforms, so the impact remains unknown and unquantified.

ROS Reality [103,104] is an open-source MR-based telerobotic manipulation framework that was developed at Brown University (Figure 6). This work enables communication and interaction between ROS-based robots and Unity-compatible MR systems. A total of 24 dexterous telerobotic manipulation tasks using ROS Reality were conducted by expert users compared to direct kinematic manipulation of the Baxter robot. the remote-controlled robotic platform targeted at expert teleoperators. However, user efficiency and system functionality were not verified with novice users for manufacturing-related tasks. This system can be used as a data acquisition and validation platform for learning from demonstration (LfD) and other machine learning approaches to transfer human expertise and skills to robots.

In [68], the authors developed an MR-based telemanipulation system to control a robotic arm-hand system. The MR scene is augmented by real-time data from the robot task space, to enhance the operator’s visual perception. The system incorporates a new interactive agent to control the robot and reduce the operator’s workload. Two control algorithms are introduced into the MR-based teleoperation system to improve the long-distance and fine motions of the robot. Telemanipulation experiments using a UR 10 robot arm and Robotiq-85 gripper demonstrate the feasibility of the proposed telerobotic paradigm.

The results of our review indicate that using motion input devices to build intuitive robot teleoperation with XR technology has become a focus of research in natural and immersive HRI applications. In this context, a common application is the use of VR controllers to capture user motion to drive physical robots, as described in [116]. However, few works compare the effectiveness and efficiency between different control–feedback methods for the teleoperation system. An exception is [117], which presents an integrated mapping of motion and visualization scheme based on the MR subspace approach for intuitive and immersive telemanipulation of robotic arm-hand systems. The effectiveness of different control–feedback methods for the teleoperation system is validated and compared.

Evaluating the effectiveness of virtual reality (VR) training technology for preparing operators in dynamic industrial environments, particularly for unforeseen situations during teleoperation, involves several key aspects and methodologies. Researchers use techniques such as assessing performance metrics like task completion time, accuracy, error rates, and response time [118]. They introduce scenario complexity through variations and stress tests, examining adaptability. Cognitive load and decision making are analyzed using eye tracking, think-aloud protocols, and surveys [119]. Biometric data like heart rate, galvanic skin response, and electroencephalography offer insights into emotional and cognitive states [120]. Feedback and iterative design involve post-training debriefing and continuous scenario refinement. Long-term skill retention is evaluated through follow-up assessments, and comparative studies compare VR-trained operators with those from traditional methods [40]. Real-world performance correlation assesses VR training’s translation into job performance. This comprehensive approach aids in understanding VR training effectiveness, informing improvements in preparing operators for unforeseen teleoperation challenges in dynamic industrial settings.

4.3.2. Mixed Reality–Based Vision Mapping and Merging

Mixed reality (MR) scenes feature immersive integration of multiple 3D/2D visual display modes to the users. At present, various teleoperation platforms have focused on MR-enhanced robot control [93]. It is not yet known whether using stereoscopic vision and point cloud within an MR environment can enhance users’ stereoscopic perception and task performance in separated telerobotic operations. In [115], an MR control room for dynamic vision and movement mapping between the operator and dual-arm robot is introduced. The multiple monocular sensor displays and motion mapping approach via MR outperforms telerobotic platforms with direct camera feeds. The control room has objects and controls floating in space, which allows the user to perform movements relative to 3D markers to command the remote-controlled robot. However, the authors did not determine if the immersive experience led to performance improvements.

In [31], the authors presented an MR teleoperation interface for mobile manipulation tasks with visual inputs from a monoscopic and stereoscopic camera setup for remote mobile manipulation tasks in hazardous production environments. This system is equipped with two monocular cameras and a stereoscopic camera at the robot’s working site. Users acquire multi-view 2D images and stereo vision with depth cues in a Unity-generated MR control room. However, the intuitive control of the robotic platform as a whole was not assessed or presented. In addition, no comparative tests with typical telemanipulation systems were performed to verify the performance and efficiency of the proposed system. To reduce the training time for teleoperation, a multi-view merging method via MR was designed (Figure 7) in [121], and the platform provides users with intuitive control of the robot’s motion by using commercial VR controllers.

Yiming Luo et al. [122] explored the use of stereoscopic view in an immersive manner for mobile robot teleoperation and navigation, and the results showed the stereoscopic view and immersive perception via virtual reality head-mounted displays (VR-HMDs) provided the user with depth cues and improved user performance and system usability. However, the effect of stereoscopic perception on telerobotic manipulation and the depth cues provided by other 3D visualization resources such as point cloud was not studied.

The only exception is [118]; this work evaluates the impact of depth perception and immersion provided by integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in an MR environment. In particular, the MR-enhanced systems maintain spatial awareness and perceptual salience of the remote scene in 3D, facilitating intuitive mixed-reality human–robot interaction (MR-HRI). This study compared two MR-integrated 3D/2D vision and motion mapping schemes against a typical 2D baseline visual display method through pick-and-place, assembly, and dexterous manufacturing tasks.

Incorporating 2D/3D vision mapping into mixed reality is just one facet of the growing adoption of XR-enhanced intuitive HRI; digital twins also play a pivotal role in facilitating intuitive teleoperation within the MR scene [116]. These digital twins, mirroring the physical robots, provide users with real-time information about the remote robot and seamlessly integrate into the human–robot interface [123,124]. Within the mixed reality environment, users can interact with these digital twins naturally, enabling precise and intuitive control over robotic systems while obtaining real-time status information about the remote robot from the digital twins [125,126]. Digital twins enhance the user’s sense of presence and control in teleoperation by providing visual representations, spatial awareness, real-time monitoring, predictive capabilities, and interactivity [127,128]. These capabilities empower operators, enabling them to teleoperate robots effectively and with a heightened sense of immersion and control. Several challenges are associated with the real-time integration of digital twins with robotic systems. Achieving real-time synchronization between the digital twin and the physical robot is crucial for seamless robotic telemanipulation [129]. Ensuring data accuracy and calibration poses another significant challenge, as digital twins heavily depend on precise sensor data to accurately represent the state of the physical robot [130]. Additionally, creating detailed and accurate digital twin models for complex robotic systems presents a formidable obstacle [131,132]. The complexity of the physical system must be faithfully mirrored in the digital twin to provide valuable and meaningful insights, adding to the complexity of the integration process [133].

Multimodal feedback and haptic interfaces play a crucial role in XR-enhanced telerobotics by providing users with enhanced immersion, increased spatial awareness, and improved perception [134,135]. Haptic interfaces provide users with a sense of touch and force feedback, allowing them to perceive and interact with virtual objects and environments more accurately and safely [57]. Several specific studies and experiments have highlighted their significance. In [136], the operator controls the remote robot in AR-based immersive and multimodal control–feedback schemes in telepresence robot applications. In [32], the authors studied XR-based multimodal interfaces and the improvements in teleoperation through comparison of how all combinations of audio, visual, and haptic interfaces affect performance during manipulation. The authors of [56,63,137] conducted experiments involving practical manipulating and manufacturing tasks, including item manipulation and item delivery of tools and components. The results demonstrate the feasibility and significance of implementing multimodal feedback, especially haptic feedback, in telerobotic systems. In summary, multimodal feedback and haptic interfaces are essential components of XR-enhanced telerobotics, significantly improving user experience and safety and the overall effectiveness of teleoperation tasks [44,58].

4.4. Mixed Reality-Enhanced Robotic Tele-Welding

4.4.1. Robotic Tele-Welding

Welding has been used extensively in the maintenance of nuclear plants, the construction of underwater structures, and the repair of spacecraft in outer space [138]. In these hazardous situations, in which human welders have no effective access, the judgment and intervention of the human operators is still required [139]. Customized production is also an application scenario for tele-welding, where welders often work in environments with dust, strong light, radiation, and explosion hazards [140]. Human-in-the-loop (HITL) robotic tele-welding strategies have become a feasible approach for removing humans from these dangerous, harmful, and unpleasant environments while performing welding operations [141]. Robotic tele-welding systems (RTWSs) combine the advantages of humans and robotics and coordinate the functions of all system components efficiently and safely [142]. For example, RTWSs can address geographical limitations for scarce welding professionals and bring a remote workforce into manufacturing.

Welding training is a time-consuming and costly process. Intensive instruction and training are usually required to bring unskilled welders to an intermediate skill level [143]. It is important to analyze the differences between the operating skills of professional and novice welders to facilitate the professional welding level of unskilled welders and to further improve the feasibility, efficiency, and welding quality of RTWSs used by novice welders during remote welding operations.

Thus, the expertise and skill extraction of professional welders as well as the application of robot assistance in on-site welding operations have become popular research topics [144]. The implementation of interactive robots can stabilize the hand movements of novice welders for improved welding quality. However, robot-assisted welding has not been studied in teleoperated welding scenarios. Welding motion capture systems were used in [145] and [146] to differentiate between professional and unskilled welders in terms of operational behavior in the gas tungsten arc welding (GTAW) process, providing an experimental basis for the development of robot-assisted tele-welding schemes. The experiments in [147] revealed the differences between professional and unskilled welders in the trajectory of the GTAW hand movements and indicated the main cause of unsatisfactory welding results is that novice welders make abrupt movements in the direction perpendicular to the weld surface. However, the operational difference of gas metal arc welding (GMAW) between professional and novice users was not well examined [148].

In industrial teleoperation contexts, control algorithms play a critical role in managing communication latency during human–robot interactions [149]. These algorithms employ predictive modeling and adaptive strategies to compensate for the delay between the operator’s input and the robot’s response [66,150]. By anticipating the robot’s movements based on the operator’s commands and real-time feedback, these algorithms help synchronize actions despite the latency [151]. The impact of communication latency on welding precision is noticeable. In tele-welding tasks, precision is paramount, requiring accurate control of the welding tool. Latency can lead to misalignments and imprecise movements, affecting the quality and integrity of the weld [152]. Control algorithms work to minimize this impact, ensuring that the robot’s actions closely align with the operator’s intentions, thus maintaining welding precision even in the presence of communication delays [153]. However, achieving real-time synchronization remains a significant challenge, necessitating continuous advancements in control algorithms to enhance both precision and efficiency in teleoperated robotic welding scenarios.

Welding automation has advanced significantly with the integration of teleoperation and artificial intelligence (AI). Teleoperation allows remote control and monitoring of welding robots, while AI algorithms enhance precision and efficiency [140]. Machine learning adapts welding parameters based on historical data, ensuring tailored and optimal welding conditions. AI-powered computer vision systems enable real-time quality control by detecting defects, leading to immediate adjustments. Predictive maintenance algorithms anticipate equipment issues, reducing downtime [154]. Additionally, AI optimizes path planning, minimizing unnecessary movements, and collaborative robots, guided by AI, work alongside human welders, ensuring precise and efficient welding outcomes [155]. These integrations enhance adaptability, productivity, quality, and cost-effectiveness across various industries.

4.4.2. Mixed Reality–Enhanced Robot-Assisted Welding

Recent research on human-centered robotic welding has focused on the development of MR-based robot-assisted welding training platforms, intuitive programming for telerobotic welding, interactive telerobotic welding design, and MR-enhanced tele-welding paradigms. A VR-based haptic-guided welder training system was introduced in [156]. This system provides guidance to welders, simulating a human welding trainer. Both novice and skilled welders can use this platform to improve their welding skills in a virtual environment. However, this system does not integrate real welding scenarios into the virtual environment to allow welders to adjust their movements in real time according to the welding pool status, nor does it transfer human movements to the robot for actual tele-welding operations.

Olaf Ciszak et al. [157] proposed a vision-guided approach for programming automated welding robot paths in 2D, where the programmer draws the target weld pattern in the user presentation space, a low-cost camera in the system captures the image, and an algorithm detects and processes the geometry (contour lines) drawn by the human. This intuitive remote programming system for welding is limited by programming of the contour lines in two-dimensional planes only and does not have the real-time capability of a telerobotic welding system. In [158], the authors analyzed the integration of advanced technologies, such as MR, robot vision, intuitive and immersive teleoperation, and artificial intelligence (AI), to build an interactive telerobotic welding system. This paradigm enables efficient human-centered collaboration between remote welding platforms and operators through multi-channel communication.

A teleoperated wall-climbing robotic welding system was developed to demonstrate the application of various technologies in an innovative robotic interaction system to best achieve natural human–robot interaction. However, the mobile wall-climbing welding robot presented in this system has a simple structure and does not have a flexible robot manipulator to mimic welders’ human-level manipulation and make dexterous welding adjustments. Natural human movement signals were not used to improve the system intuitiveness and control the robot for tele-welding tasks.

More recently, research attention has been focused on MR-enhanced tele-welding paradigms [159]. It was verified in [160] that there were no statistically significant differences in the total welding scores between participants in the physical welding group and the mixed reality–based welding groups. The mixed reality welding user interface gives operators the ability to perform welding at a distance while maintaining a level of manipulation [161]. An optical tracking-based telerobotic welding system was introduced in [162]. The Leap Motion sensor captures the trajectory of a virtual welding gun held by a human welder in the user space to control the remote welding robot for the welding task. However, this welding system requires the use of a physical replica of the welding workpiece in the user space to superimpose a real-time weld pool state and guide the welders to adjust their hand movements to the shape of the workpiece [163]. Thus, this MR welding system is not suitable for a wide range of workpieces.

Yukang Liu et al. [163] developed a projection-based MR telerobotic welding system for transferring welder skills and human-level dynamics to the welding robot (Figure 8). Human welders interact with the welding robot by moving a tracked virtual welding torch in 3D space. A UR5 industrial manipulator equipped with vision sensors is operated remotely to perform the welding task. The weld pool stream from the welding site is transmitted back to the user and projected on a mock-up of the workpiece. The human welder can monitor the work process and adjust the movements according to the projected weld pool status from the worksite. However, the operator does not have sufficient visual feedback to check the status of the robot during operation.

Wang et al. [164] developed an MR-based human-robot collaborative welding system (Figure 9). The collaborative tele-welding platform combines the strengths of humans and robots to perform weaving gas tungsten arc welding (GTAW) tasks. The welder can monitor the welding process through an MR display without the need to be physically present. Welding experiments indicated collaborative tele-welding provides better welding results compared to welding performed by humans or robots independently.

MR-based robot-assisted remote welding platforms were developed in [165] to provide the welders with more natural and immersive human–robot interaction (HRI). However, in these systems, the users rely on visual feedback for movement control and have no haptic effects to completely prevent accidental collisions between the robot and the workpiece when the operator controls the robot for welding from a distance. Hence, they are still limited.

A visual and haptic robot programming system based on mixed reality and force feedback was developed in [166]. However, the system was not suitable for real-time remote welding operations and was inefficient in unstructured and dynamic welding situations. Haptic feedback provides the welders with additional scene modality and increases the sense of presence in the remote environment, thereby improving the ability to perform complex tasks [167]. In contrast, it can be difficult to implement effectively.

The primary advantage of integrating haptic effects is to enhance both the performance of tele-welding tasks and the operator’s perception. Current remote robotic welding systems fail to fully utilize the potential improvements in performance offered by various forms of haptic feedback. The rapid advancement of MR-enhanced teleoperation has facilitated the merging of MR and virtual fixtures, aiming to enhance task performance and user perception. This integration effectively addresses the deficiencies in existing telerobotic welding systems. By creating an immersive and interactive MR environment, virtual workpieces can be generated in the user’s space. When combined with Virtual Fixture (VF) technology, this environment provides users with force feedback and guidance, improving the precision of robot movements and preventing accidental collisions [168,169].

The main limitation of existing studies on tele-welding lies in the insufficient incorporation of MR technology and virtual fixtures into remote-controlled robotic welding systems. This lack of integration hampers the elimination of potentially harmful collisions in the tele-welding process and prevents welding robots from achieving human-level dynamics for intricate Gas Metal Arc Welding (GMAW) tasks. Additionally, there has been no effort to simplify operations to assist inexperienced welders in performing welding tasks quickly, addressing the issues of time-consuming training and the shortage of qualified personnel.

An exception is [107], which introduces an integrated approach utilizing MR and haptic feedback for intuitive and immersive teleoperation of robotic welding systems. This approach involves incorporating MR technology, allowing the user to be fully immersed in a virtual operating space enhanced by real-time visual feedback from the robot’s working area. In this proposed robotic tele-welding system, the user’s hand movements are mapped to imitate the robot’s motions, enabling spatial velocity-based control of the robot’s tool center point (TCP).

Several design elements, including mixed reality, digital twin, and virtual fixtures, are integrated and implemented in user interfaces for intuitive comprehension of the robot’s state, especially in critical welding situations. Digital twin technology was utilized to capture the real-time pose of the physical UR5 robot during its operation, enabling welders to observe the rotational status of each joint. By combining virtual twin data with onsite video streams in the MR space, a comprehensive monitoring system was created, offering real-time insights into the robot’s operations. This setup facilitated accurate and efficient adjustments of welding motions based on the robot model data. To ensure optimal visualization within the MR welding workspace, the virtual UR5 robot’s data and motions were scaled at a ratio of 1:5 to match the user’s perspective. Within this framework, virtual fixtures (VFs) were employed and categorized into guidance and prevention fixtures. The proposed MRVF integrated both types, guiding users to the initial welding point effectively and preventing collisions between the torch tip and the workpiece.

In the welding process, it is essential for the electrode to contact the molten weld pool to transfer filler metal to the workpiece. Simultaneously, contact between the torch tip and the workpiece must be avoided to prevent damage [107]. In the MRVF tele-welding workspace (as illustrated in Figure 10), a transparent prevention VF panel was overlaid on the virtual workpiece. This panel displayed a 2D representation of the actual welding process, minimizing collisions between the user-manipulated torch tip and the workpiece.

5. Challenges and Future Opportunities

Based on the reviewed and analyzed research, several challenges associated with XR solutions for robotic telemanipulation and telemanufacturing have been identified. One major challenge lies in the technological integration of extended reality technologies such as AR, VR, and XR with robotic systems, giving rise to issues such as synchronization problems, latency, and the need for real-time data processing. Additionally, designing interfaces that offer an intuitive and immersive user experience proves to be a daunting task, demanding seamless interaction with augmented or virtual environments while simultaneously controlling robotic systems in real time. Providing accurate and meaningful haptic and sensory feedback to users presents another obstacle, as replicating real-world touch and feel in virtual environments adds complexity to system design. Furthermore, ensuring the safety of both users and the environment remains paramount, necessitating solutions for challenges like collision detection, emergency shutdown mechanisms, and real-time hazard recognition. Moreover, the cost associated with developing and implementing robust XR-enhanced robotic systems poses a significant barrier, limiting accessibility for smaller organizations or research initiatives aiming to delve into this innovative technology.

Given the dynamic nature of XR technology, we anticipate numerous exciting opportunities in XR-enhanced telerobotics. Advancements in immersive XR interfaces, incorporating enhanced gesture recognition, voice commands, and brain–computer interfaces, are expected, making teleoperation more seamless and intuitive. Customized interfaces tailored to individual preferences, coupled with advancements in MR and haptic feedback integration, promise a realistic and detailed sense of touch and force feedback, enhancing the operator’s ability to perform delicate tasks remotely. Additionally, developments in AI and machine learning could enhance robots’ autonomy and adaptability by learning from human operators, fostering more efficient and versatile telerobotic systems. In summary, innovations in immersive interfaces, haptic feedback, AI integration, and interdisciplinary research are poised to revolutionize remote teleoperation and human–robot collaboration, unlocking the full potential of XR technology in telemanipulation and telemanufacturing domains.

6. Conclusions

This article identifies and summarizes the intuitive telerobotic frameworks proposed in the human–robot interaction (HRI) literature, which aim to remove human workers from harmful working environments by equipping complex robotic systems with human intelligence and command/control via intuitive and natural human–robot interaction, including the implementation of XR techniques to improve the user’s situational awareness, depth perception, and spatial cognition, as fundamental to effective and efficient teleoperation. The article commences by delving into the prerequisites for achieving effective robotic teleoperation, highlighting two important requirements for an effective and intuitive teleoperation platform. First, natural motion retargeting, or the use of natural human behavior to teleoperate robotic systems, effectively reduces training costs and grants human-level dynamics to telerobotic systems; it is an important component of natural and intuitive human–robot interaction systems. Second, adequate multimodal feedback, especially visual feedback that provides depth information and situational awareness to enable the operator to know the dynamic situation in the robot’s workspace, is also critical in interactive telemanipulation systems.

This review has elucidated the state-of-the-art frameworks developed by the robotics community to integrate physical robotic platforms with XR technology. Specifically, it underscores MR-based 3D/2D vision mapping and merging as a method for providing immersive integration of diverse visual display modes for users. In the MR-enhanced telerobotic schemes, digital twins also play a pivotal role in facilitating intuitive teleoperation within the MR scene, in which users can interact with these digital twins naturally, enabling precise and intuitive control over robotic systems while obtaining real-time status information about the remote robot from the digital twins. We have presented the main contributions of the reviewed articles on XR-enhanced telerobotic schemes in industrial systems and classified their application contexts into three main groups: effective and intuitive teleoperation, MR-enhanced intuitive telemanipulation, and MR-based robotic tele-welding. Additionally, this review highlights the pivotal role of XR, especially MR, in the domain of teleoperation systems and robotic welding. Identified as a promising development, the incorporation of haptic interfaces in tele-welding systems holds the potential to enhance task performance and operator perception.