4.1. AR in Medical Robotics
In this section, we review AR applications developed during the five year period from 2015 to 2019 in medical robotics. We will cover AR-based Robot Assisted Surgery (RAS), surgical training platforms, and rehabilitation systems. Within this section of the paper we collected 26 papers from the Scopus database by the union of two search strategies: (1) keywords in the title (“augmented-reality-based" OR "augmented-reality” OR “augmented reality”) AND (“robot-assisted” OR “robotic rehabilitation” OR “robotic surgery” OR “robotic assisted” OR “robot-assisted” OR “robotics prosthesis” OR “robotic prosthesis”), (2) keywords in the title and in the keywords section: (“augmented-reality-based” OR “augmented-reality” OR “augmented reality”) AND (“robotic surgery” OR “robotic imaging” OR “medical robotics”). Then we excluded review papers, non-English papers, and papers that do not suit the scope of our work. Furthermore, we found additional papers via cross referencing from the previously considered works. Among the selected papers, ten works were included for the analysis presented in Table 1
An overview of AR-based robotic surgery platforms by Diana and Marescaux [43
] discusses issues related to safety and efficiency during the computer-assisted surgery. In [44
], AR is utilized for training medical residents for robot-assisted urethrovesical anastomosis surgery. In [45
], SAR technology is embedded to a mobile robot for robot-assisted rehabilitation. This rehabilitation framework consists of a projecting module that renders a virtual image on top of the real-world object. Later, Ocampo and Tavakoli [46
] developed an AR framework integrated with the haptic interface, where SAR technology and 2D projection display were utilized. Such system was designed to make the overall rehabilitation process more effective, fast, and reduce the cognitive load experienced by patients during rehabilitation.
Pessaux et al. [47
] developed a visualization system enhanced with AR technology for robot-assisted liver segmentectomy. This system assists the surgeon to precisely and safely recognize almost all of the important vascular structures during the operation. Liu et al. [48
] presented an AR-based navigation system to assist a surgeon to perform base-of-tongue tumor resection during the transoral robotic surgery. With the AR vision module, the da Vinci surgical robot can identify the location of the tumor in real-time and adjust the area for resection by following the instructions augmented into its stereoscopic view. As noted by Navab et al. [49
], AR-based vision tools together with intra-operative robotic imaging can improve the efficiency of computer-assisted surgery.
An AR compatible training system to assist urology residents to place an inflatable penile prosthesis is described in [50
]. An AR-based vision system for RAS was developed by Lin et al. [51
], where a three-dimensional AR display was used in an automatic navigation platform for craniomaxillofacial surgery. An AR navigation system used in oral and maxillofacial surgery was described by Wang et al. [52
]. This framework utilized a novel markerless video see-through method for registration that was capable of aligning virtual and real objects in real-time. Thus, the user did not require to align manually. Later, Zhou et al. [53
] developed an AR platform for assisting a surgeon during robotic mandibular plastic surgery. The framework generated navigation guidelines and prepared visual instructions for the preoperative surgical plan. With such system, inexperienced users have more chances to make proper decisions during operation. For instance, force feedback provided by the framework during the experiments facilitated the process of controlling the high-speed drill by new users preventing them from damaging bone or nerve tissues inadvertently.
Bostick et al. [54
] created an AR-based control framework to send manipulation signals to a prosthetic device. The system utilized an algorithm that identified an object and sent corresponding grasping commands to the upper-limb prosthesis. Qian et al. [55
] refer to the AR-based framework “ARssist” that consists of a teleoperated robot assistance module and a haptic feedback interface. It can generate visual instructions during minimally invasive surgery. The system renders the 3D models of the endoscope, utilized instruments, and handheld tools inside the patient’s body onto the surgeon’s view in real-time (Figure 4
a). Furthermore, AR-assisted surgery with the help of projector-camera technology is also gaining momentum [55
]. In [56
], an AR system was used in rendering the operational area during the urological robot-assisted surgery for "radical prostatectomy". In [57
], an overview of the literature addressing the issues of surgical AR in intra-abdominal minimally invasive surgery is presented. In [58
], different robotic systems and AR/VR technologies utilized in neurosurgery are summarized. For example, in spinal neurosurgery, authors refer to the robotic assisting platforms utilized for screw placement, e.g., Spine Assist, Renaissance, and da Vinci surgical systems. According to the authors, there is a high potential of AR/VR enhanced training systems and simulation platforms to improve the training process of surgeons, even though limitations of AR/VR for neurosurgery are still present. The pilot studies with experienced and inexperienced surgeons using this AR assistance system in an operational scenario was described later in [59
] (Figure 4
b,c). Analyzing the recorded results of 25 inexperienced surgeons, the authors state that AR assistance helped to increase the efficiency of the operation, improve patient safety, enhance hand-eye coordination, and reduce the time required for tool manipulation by the surgeon. According to Hanna et al. [60
], AR-integrated platforms can be used to detect and diagnose anatomic pathologies.
A recent work discussing AR and VR navigation frameworks for achieving higher accuracy and precision in RAS, specifically in oncologic liver surgery, was presented by Quero et al. [61
]. This work highlighted how imaging and 3D visualization can improve the perception of the operation area by the surgeon in RAS. These authors also developed an AR-based platform designed to provide the "X-ray see-through vision" of the operating area in real-time to the surgeon wearing the HMD [62
]. The performance of this system was tested during minimally invasive laparoscopic surgery on the deformable phantom. It was found that the system is prone to registration errors since the operational area was comprised of soft tissues.
Among robotic surgery platforms, da Vinci surgical robot is the most widely used robotic system utilized in RAS. In addition, console display of da Vinci system can be adapted to support AR techniques during the surgery obviating the need for additional displays for supporting AR during surgery. The state of the art in AR-integrated the Robot Assisted Surgery (RAS) was reviewed by Qian et al. (2019b) [63
]. A brief overview of AR applications developed within 5 year period in the area of medical robotics along with the corresponding list of technology utilized for the creation of AR setups, tested robotic systems, and observed overall limitations are summarized in Table 1
In Figure 5
, we show the distribution of the papers indexed by the Scopus database published between 1995 and 2019 within the area of medical robotics. Selected papers are classified into three groups, which are number of papers in MR, AR, VR. From this figure, we infer that popularity of AR and VR in medical robotics rose significantly from 2015 until 2019. The papers were collected by using following keywords throughout all sections of papers in Scopus database: (1) ALL(“mixed reality” AND NOT “augmented reality” AND “medical robotics”), (2) ALL(“augmented reality reality” AND NOT “mixed reality” AND “medical robotics”) and (3) ALL(“virtual reality” AND “medical robotics”).
Based on our literature survey, we foresee that AR has an immense potential to trigger a paradigm shift in surgery to improve outcomes during and post operation. On the other hand, this will presumably introduce new challenges in the training of new medical residents and surgeons who would be able to benefit from AR-enhanced visualization and navigation tools. Before AR could be further utilized in medicine and rehabilitation, technical limitations of the current systems should also be overcome. These include latency, calibration and registration errors.
4.2. AR in Robot Control and Planning
AR applications within this category focus on the control and planning domains of robotics. In order to find relevant papers for this section, we collected 49 papers from the Scopus database by the union of two search strategies: (1) keywords in the title (“augmented-reality-based” OR “AR-assisted” OR “augmented reality”) AND (“robot navigation” OR “robot control” OR “robot localization” OR “robot visualization” OR “automated planning” OR “robotic control” OR “robot programming”), (2) keywords in the title and in the “keywords section”: (“augmented-reality-based” OR “AR-assisted” OR “augmented reality”) AND (“robot programming”)). We did not consider non-English papers, review papers, papers duplicating earlier works and irrelevant ones. Other papers were added to the list via cross-referencing strategy. Among selected papers, 16 were included for the analysis presented in Table 2
Gacem et al. [64
] introduced an AR-integrated system, “Projection-Augmented Arm (PAA)”, that incorporates projection technology with a motorized robotic arm to assist human user during the processes of locating and finding objects in a dense environment. In [65
], an infrared camera was integrated to a mobile robot to enhance autonomous robot navigation via utilizing projection SAR technology. With the help of this framework, a mobile robot was able to follow a complicated path by receiving graphical feedback in the form of projected infrared signals. A similar AR-integrated mobile robot navigation system was described in the work of Dias et al. [66
]. This system used a visualization framework consisting of non-central catadioptric cameras, and an algorithm capable to determine the position of the robot from the camera data. The catadioptric optical system acquired a wide field of view using camera lenses and mirrors through central projection and a single viewpoint.
], industrial robot programmers assisted by AR were asked to complete the tool center point teaching, trajectory teaching, and overlap teaching tasks. Evaluation metrics used in the work were the efficiency of the performed work, overall time spent to complete the task, and the amount of the mental load experienced by the robot programmer. The following trends were observed: (1) mental load experienced by AR-assisted users was significantly lower than the amount of load experienced by users performing the tasks without AR assistance and (2) time spent for completing the task with AR support was significantly higher for non-experienced AR users. Thus, one can see that AR technology has the potential to reduce the mental load of industrial workers; however, time and training are needed for workers to get used to the AR system initially. In [38
], AR was embedded to the remote control module of a teleoperated maintenance robot. In this system, a handheld manipulator wirelessly transmitted a control signal and manipulation instructions to the real maintenance robot (Figure 6
a). An optical tracking system was utilized to identify the position of the handheld manipulator and to place properly the generated augmented scenes for the maintenance operator in the AR environment. In [68
], AR was emplyed for on-the-fly control of aerial robots utilized in structural inspection. In [69
], AR with haptic feedback was used during teleoperation of an unmanned aerial vehicle (UAV) equipped with a gamma-ray detector. With the help of this system comprised of a 3-DoF haptic device, fixed camera, computer screen, and a nuclear radiation detector, an operator was able to remotely control a drone during its search for the source of nuclear radiation (Figure 6
b). Later, Kundu et al. [70
] integrated AR interface with an omnidirectional robot for vision-based localization of a wheelchair during indoor navigation. As shown by Zhu and Veloso [71
], it is possible to interact with a smart flying robot, that generates the route on-the-fly and executes the planned motion via an AR capable interface. This interface can videotape the robot’s motion and overlay the recorded video with the synthetic data allowing human operators to study algorithms utilized by the robot during the flight.
Information from an AR simulation environment with the virtual robot was used to generate the motion instructions to the real robot [72
]. The presented framework was capable of simulating the pick-and-place task, in which the target object was placed into the computer numerical control (CNC) milling machine (Figure 6
c). Lee et al. [73
] in their work show that position of the AR camera with respect to the robot link can be estimated with the help of SAR technology in cases when the position of the camera cannot be calculated using methods purely based on kinematics. In another application, robot status and navigation instructions were generated by an AR platform and sent to the human operator via an AR display [74
]. Guhl et al. [75
] developed an AR interface that established communication between devices and the human within an industrial environment.An AR system that utilized the Fuzzy Cognitive Map optimization algorithm for the real-time control of a mobile robot is presented in [76
]. During the experiments, the robot was able to modify its intended motion following AR instructions comprised of glyphs and paths. An experimental setup containing an AR-supported vision system and haptic feedback technology which was used for the remote control of a welding robot is presented in [77
Liu et al. [78
] integrated an AR interface with Temporal And-Or Graph (T-AOG) algorithm to provide a robot programmer with information on the robot’s hidden states (e.g., latent forces during interaction). In [79
], an AR interface was used to program a seven degrees-of-freedom (DoF) manipulator via the method of augmented trajectories. The framework was able to perform trajectory generation, motion planning, parameter identification, and task execution in a simulation environment. In [80
], a robot programming assistant framework, developed on the basis of the Google Tango AR computing platform, was described. This AR system was tested during the remote programming of two KUKA Lightweight Robots. During the study, the human operator was asked to modify the robot’s joint configuration, change coordinates of the tool center point, control the gripper, and switch between different control modes. Later, Hoffmann and Daily [81
] introduced an AR framework to present information obtained from the 3D simulation environment in the form of 2D instructions displayed on an AR screen. The system records data from 2D camera and 3D sensor in order to place virtual data on top of the 3D objects within a real-world environment. AR was integrated with tactile feedback technology in [82
]. This system was designed to provide real-time assistance to a human operator during industrial robot programming and trajectory generation. The developed interface enabled a more natural and efficient communication during collaborative work with an industrial robot thanks to the hand gestures used to interact with the AR interface.
Recent work describing the state-of-the-art of simulation and design platforms for manufacturing was published by Mourtzis [83
]. Additionally, Mourtzis and Zogopoulos [84
] in their work consider AR-based interface to support an assembly process. Integration of AR to warehouse design utilized in the papermaking industry was described in [85
]. AR-based framework for industrial robot programming is described in [86
]. It was found that such an application can significantly ease the process of robot programming and motion planning, reducing the necessity for extensive training of the human workers. An AR interface designed to provide assistance during robotic welding was presented by Ong et al. [87
]. Another AR programming framework for welding robots was developed by Avalle et al. [88
]. This tool helped a new user to complete complex welding tasks without having expertise in robotics. Projection-based AR was used for providing assistance to human operators in robotic pick and place tasks in the work of Gong et al. [89
]. Such an interface reduces the cognitive load of an operator involved in robot grasping by providing visual feedback during robot-based object manipulation in real environment.
Integrating AR technology into the process of industrial robot programming, optimal path generation and the overall design of some production processes has become a common technique in industry. Among wide range of platforms developed for the creation of AR environments, Unity game development platform is the most highly utilized framework. AR environment allows to map the virtual and real robots and predict the motion of the real robot with the help of the virtual one. Furthermore, in some applications, AR provides instructions to the human programmer of an industrial robot such that the human operator can easily adapt to the programming of the new robotic system. In Table 2
, we provide a short summary of AR applications presented in this section together with the used technology for the creation of AR environments, robotic platforms and observed limitations of the developed AR systems.
Numbers of the papers indexed by the Scopus database published between 1995 and 2019 in robot control and planning are shown in Figure 7
. These papers were classified into three groups (MR, AR, and VR). Similar to the Figure 5
for medical robotics, the figure shows that the numbers of AR and VR papers on robot control and planning have been rising steadily, especially in the last five years. We also notice that the number of works in MR has not been changed significantly. This might be due to the trend in literature to combine fields of AR and MR into one area known as AR. In our search, we used the following keywords throughout all sections of papers in Scopus database: (1) ALL(“mixed reality” AND NOT “augmented reality” AND (“robot control” OR “robot planning”)), (2) ALL(“augmented reality reality” AND NOT “mixed reality” AND (“robot control” OR “robot planning”)) and (3) ALL(“virtual reality” AND (“robot control” OR “robot planning”)).
From the presented works, we infer that the integration of AR into the simulation process can lead to more efficient manufacturing with less cognitive load to the human operators. Furthermore, AR technology in the era of Industry 4.0 is going to serve as a new medium for human–robot collaboration in manufacturing. The developed applications and studies analyzed in this section reveal the premise of AR to facilitate human interaction with technology in the industry. However, the training of specialists in manufacturing, assembly and production needs to be transformed such that AR-assistance will increase the efficiency and can be adopted seamlessly.
4.3. AR for Human-Robot Interaction
Wide-ranging applications of AR in HRI have been developed to enhance the human experience during interaction with robotic systems or wearables within considered five year period. The papers analysed in this section were selected by the following method. First, we collected 44 papers from the Scopus database by the union of two search strategies: (1) keywords in the title (“augmented reality”) AND (“human–robot collaboration” OR “remote collaboration” OR “wearable robots” OR “remote control” OR “human-machine interaction” OR “human–robot interaction” OR “robotic teleoperation” OR “prosthesis” OR “brain-machine interface” ), (2) keywords in the title and in the “keywords section”: (“augmented reality”) and ((“augmented reality”) AND (“human–robot interaction”)). Then we removed survey papers, non English papers and papers not relevant for our topic. We also considered papers found by cross-referencing and ones suitable for the scope of this survey. In Table 3
, 19 papers were selected for the detailed analysis.
With the help of AR, it was possible to set a remote collaborative workflow between a supervisor and the local user as shown by Gurevich et al. [90
]. In this system, an AR-based projector was placed on top of a mobile robot and it enabled the transfer of information in a way that the user’s hands remain free to follow the remotely provided instructions. Another AR-based touchless interaction system allowed the users to interact with the AR environment via hand/feet gestures in [91
]. In the work of Clemente et al. [92
], an AR-based vision system was designed to deliver feedback to the sensory-motor control module of a robotic system during object manipulation. The system utilized two wearable devices: haptic data glove and MR goggles to facilitate user interaction during remote collaborative work with a robotic hand.
Gong et al. [93
] developed an AR-based remote control system for real-time communication with “IWI” human-size traffic cop robot. For real-time video monitoring, “Raspberry Pi” camera module was embedded into the police robot. This camera allowed human operators to send remote control commands to the robot based on the information transferred to the AR goggles or tablet displays. Piumatti et al. [94
] created a cloud-based robotic game integrated with Spatial Augmented Reality (SAR) technology. This game has different modules to set game logic, sound, graphics, artificial intelligence and player tracking in order to develop and integrate proper AR projections into the gaming experience. Dinh et al. [95
] leveraged AR for assisting a human operator working with a semi-automatic taping robot. The system used SAR and “Epson Moverio BT-200” AR goggles to generate visual instructions on the taping surface. Lin et al. [96
] integrated AR technology into the Robotic Teleoperation System comprised of Wheelchair Mounted Robotic Manipulator. AR was employed to remotely reconstruct the 3D scene of the working area and generate a specific scene, which was then augmented to the user’s field of view manipulating a virtual robotic arm. Finally, AR-based control module converted gestures applied to the virtual object into control commands for the robotic arm.
Shchekoldin et al. [97
] presented an AR framework to control a telepresence robot using an HMD integrated with an inertial measurement unit (IMU). The system utilized an adaptive algorithm that processed the IMU measurements and inferred the motion patterns (e.g., direction and magnitude of the angular velocity) due to the user’s head movement. An AR-based vision system for constructing the kinematic model of reconfigurable modular robots is developed in [98
]. Each robot module was marked with an AR tag with a unique identifier. This way, the state of each link was acquired and the distance between modules and joint angles were computed. Mourtzis et al. [99
] utilized cloud-based remote communication to establish an interaction between an expert and the remotely-assisted maintenance worker within the AR environment (Figure 8
Recently, Makhataeva et al. [100
] developed an AR-based visualization framework that aimed to increase an operator’s awareness of the danger by augmenting his/her view in AR goggles with an aura around the operating robot (Figure 8
c,d). The work introduced the concept of safety aura that was developed on the basis of a safety metric based on distance and velocity. AR can also be applied for relaying robot motion intent during human–robot communication. For example, Walker et al. [101
] developed an AR-based communication framework to interact with AscTec Hummingbird during its autonomous flight over a pre-defined trajectory. In the later work, Walker et al. [102
] introduced an AR interface that utilized the virtual model of the robots, such that before sending teleoperation commands to the real robot, a user could study the motion of the virtual robots. This interface was tested on the teleoperation scenario of an aerial robot designed to collect and process environmental data. In [103
], AR was embedded to the control interface of a teleoperated aerial robot (Parrot Bebop quadcopter) programmed for data collection during environmental inspection. In [104
], a novel teleoperator interface to remotely control the robot manipulators for nuclear waste cleanup is proposed. This interface could enhance the performance of a human operator using multi-model AR equipped with haptic feedback. ROS (Robot Operating System) was employed as the main platform for data sharing and system integration. Another example of robot teleoperation with AR-based visual feedback was invented by Brizzi et al. [39
]. The authors discussed how AR can improve the sense of embodiment in HRI based on experiments with Baxter robot which was teleoperated in an industrial assembly scenario.
In the area of wearable robotics, He et al. [105
] developed an AR simulation system which provided a patient with a realistic experience of donning a myoelectric virtual prosthetic hand. The framework consisted of an AR display, pose estimation module for aligning the virtual image of the prosthetic device with the real hand of a person and a feedback system that allowed a user to control virtual hand prosthesis with electromyography signals recorded from the hand. Meli et al. [106
] presented an overview of robotic frameworks embedded with haptic feedback and AR. This work also highlighted the effectiveness of wearable finger haptics in AR to perform robot manipulation, guidance, and gaming. Wang et al. [107
] integrated a BCI system with AR visual feedback in order to assist paralyzed patients in controlling a robot arm for grasping. The system was tested on five subjects. Obtained results indicated that the AR-based visual feedback system has an advantage over standard camera-based one (control time and error of the gripper aperture reduced to 5 s and 20%, respectively). Zeng et al. [108
] embedded AR into the Gaze-BCI system utilized in closed-loop control of a robotic arm. During the experiments, eight subjects were asked to grasp and lift objects with a robot. The obtained results were the following: amount of trigger commands necessary to complete the task reduced and amount of errors during the lifting process decreased more than 50% in comparison to the trials when a vision system without AR feedback was utilized. Si-Mohammed et al. [109
] introduced a BCI system with AR feedback to simplify the control of a mobile robot. Microsoft HoloLens MR goggles were used for visualization of the augmented environment.
In the area of HRI, Peng et al. [40
] developed an AR-based fabrication framework named as “Robotic Modeling Assistant (RoMA)”. With this framework (Figure 8
b), a user wearing AR goggles was able to control object modeling and printing performed by a ceiling-mounted robotic arm (Adept S850, Omron). Such a framework can accelerate design and 3D printing by allowing a user to interact with the printed object and make changes almost simultaneously. Urbani et al. [110
] introduced an AR-based inter-device communication framework to monitor and adjust operational parameters (e.g., status, shape, and position) of a multipurpose wrist wearable robot. According to Williams et al. [111
], the first workshop on “Virtual, Augmented, and Mixed Reality for Human-Robot Interactions (VAM-HRI)” brought together works where AR, VR, and MR were integrated with different robotic systems. According to Sheridan [112
], one of the major open problems in HRI is related to human safety during collaborative work with robots. The issue of safety during HRI was also addressed in [113
]. Specifically, the authors considered the concept of the virtual barrier in order to protect human users during collaboration with robots in a manufacturing environment. In addition, AR was applied for Concept-Constrained Learning from Demonstration as described in [114
]. Using this method, a user wearing AR goggles could see the start and end positions of robot motion and define the motion trajectories more intuitively.
AR technology for the remote control and teleportation has become popular in HRI research. In addition, visualization capabilities of AR-based applications increase their usability in human robot collaborative tasks. For example, AR increases the efficiency during the trajectory design and safety during testing of various robots. A key technological breakthrough in HRI application is the invention of the HMDs, e.g., Microsoft Hololens. ROS emerges the leading open source platform for interfacing AR devices with robots. Around twenty AR applications in HRI, along with the technology utilized in these setups, tested robotic systems, and observed limitations are summarized in Table 3
Within the area of HRI, the number of papers in MR, AR and VR indexed by the Scopus between 1995 and 2019 is shown in Figure 9
. The figure shows the significant rise in papers dedicated to the Human-Robot Interaction within AR and VR starting from the year of 2016. Even though number of papers both in AR and VR rise over the years, number of papers on VR is almost double that of AR. The search for papers was conducted by using the following keywords throughout all sections of papers in Scopus database: (1) ALL(“mixed reality” AND NOT “augmented reality” AND “human–robot interaction”), (2) ALL(“augmented reality” AND NOT “mixed reality” AND “human–robot interaction”) and (3) ALL(“virtual reality” AND “human–robot interaction”).
According to our literature analysis, we note AR can significantly shape human perception of collaborative work and interaction with robotic systems. For example, with the help of AR, interaction can become more intuitive and natural to humans. This is especially the case when the AR environment is experienced via AR goggles worn by a human during communication, control, and study of robotic systems. Furthermore, a multitude of AR applications developed for remote control and teleoperation indicate that AR will be extensively utilized in the future, primarily due to the rise of Industry 4.0. However, technological limitations of AR technology such as calibration issues, problems with optics and localization errors need to be further researched such that commercial AR applications for HRI can translate from research to the market.
4.4. AR-Based Swarm Robot Research
In this section, we consider AR applications developed for swarm robotics. Within this section, we found nine papers from the Scopus database by searching following keywords in the title and abstract ((“augmented reality”) AND (“robot swarm” OR “swarm robotics” OR “kilobots” OR “swarm e-pucks” OR "biohybrid systems”)). We removed irrelevant papers according to the scope of this work and included additional papers through cross-referencing. From the final list of chosen papers, we further selected seven for the analysis presented in Table 4
In swarm robotics, AR can be perceived through virtual sensing technology as demonstrated by Reina et al. [115
]. In their work, the authors presented an AR-based virtual sensing framework that consists of (1) a swarm of robots (15 e-pucks) each embedded with a virtual sensor, (2) a multi-camera tracking system that collects data of the real environment, and (3) a simulator that uses this data to generate augmented data and sends instructions to each robot based on the sensing range of the embedded virtual sensor (Figure 10
a). With the help of virtual sensing, the motion of robot swarms can be studied more effectively in AR, while in real environment mini-robots have limited sensing. In addition, this system can help researchers to control and design motion of robot swarms within complex scenarios in an AR-based synthetic environment.
AR can also be utilized to investigate the interaction between swarms of biohybrid (plant-robot) agents. AR was utilized to visualize the space covered by biohybrid swarms [41
]. Specifically, the system showed potential spatial changes as the plant-robot systems evolve or grow over time. As was demonstrated by Omidshafiei et al. [116
], AR was integrated with the robotic prototyping framework employed in the research of cyber-physical systems. The AR platform assisted the user during hardware prototyping and algorithm testing via providing real-time visual feedback from the system’s hidden states.
Experiments with the Kilobots in the ARGoS simulation environment were described by Pinciroli et al. [117
]. The authors designed a plugin to perform cross-compilation of the applications developed in the simulator according to a real robot scenario (Figure 10
d). Reina et al. [118
], in their work, integrated AR into the control software of Kilobots system. Within the developed platform, AR together with virtual sensing helped to extend the operational capabilities of the robots (Figure 10
b,c). This made them suitable for experiments where advanced sensors and actuators were required. Later, an open-source platform was created for the experiments with large numbers of Kilobots [119
]. The performance of this system was studied on an experimental setup where robot swarms were organized to execute obstacle avoidance, site selection, plant watering, and pheromone-based foraging tasks.
In the recent work of Reina et al. [120
], a group of Kilobots was programmed to perform actions during the filming process of a short movie where actors were all robots. The work described software utilized for the programming of robot swarms during filming. Specifically, the authors wrote an open-source code that transformed human-generated instructions to the C code, which was later loaded to Kilobots to make them perform corresponding actions in the real scene. In the work of Llenas et al. [121
], the behavior of large swarms were observed to study the concept of stigmergic foraging. During the study ARK platform also known as AR for Kilobots was used as the main simulation framework. The study on the collective foraging behavior continued [122
], where behaviors of around 200 Kilobots having access to the sensing data from the virtual sensors and actuators were studied. ARGoS framework was used as the main experimental platform.
Latest advance in the research of swarm robotics is the simulation platforms such as ARK or ARGoS. These simulation platforms enable to study motion of robot swarms integrated with the virtual sensors and virtual actuators. This way, perception of the environment by the mini robots with limited sensing abilities can be enhanced by virtual sensors, allowing researchers to study their motion in more complex scenarios. Considered AR applications developed within the research of robot swarms, their technology, the robotic systems, and limitations are listed in Table 4
Within the research area of swarm robotics, the distribution of the papers indexed by the Scopus database published between 1995 and 2019 in MR, AR and VR is shown in Figure 11
. From the figure, we observe that integration of MR, AR and VR technologies in the research of swarm robotics is not yet widely addressed in the literature. However, in the period of 2015–2019, the number of papers published within AR and VR rose from five to around 30–35 papers. The number of papers in MR is less than five throughout all years within the considered time period. The papers for this figure were collected by using the following keywords in Scopus database: (1) ALL(“mixed reality” AND NOT “augmented reality” AND (“swarm robotics” OR “robot swarms”)), (2) ALL(“augmented reality reality” AND NOT “mixed reality” AND (“swarm robotics” OR “robot swarms”)) and (3) ALL(“virtual reality” AND (“swarm robotics” OR “robot swarms”)).
The performed review outlined the lack of research on the integration of AR to robot swarms. However, the limited introduction of AR to the swarm robotics has shaped the human perception of the simulation environment and significantly eased the design and understanding of the collective behavior in robot groups. Currently, AR-based simulation platforms allow scientists to setup experiments involving thousands of mini-robots in simulation before actual real-world experiments. Even though few applications have been developed within this area of robotics until now, we foresee that further development of AR technology will significantly increase the number of works dedicated to the study of AR in swarm robotics.