The interaction between a human and a swarm can pose many problems and issues. Indeed, there are many obstacles that can prevent the swarm from achieving the human objective:
This must be attainable by the swarm according to its capabilities. If the target is too complex for the swarm functionalities, it will not be achieved.
To communicate their objectives, the operator must use an appropriate means of bidirectional communication that enable both the operator and swarm to be understood.
Depending on the environment, there are different problems involved in moving a swarm. In outdoor sites, weather conditions and fields of deployment are the main challenges to overpass. In indoor areas or building, communication between the swarm and operator can be very difficult due to loss of communication signals. The difficulty also increases if the operator does not have a line of sight on the swarm, and if he controls it through a graphical interface that gives him the essential information.
If a swarm is very dependent on the operator’s decision, the operator must constantly observe the evolution of the swarm and guides it swarm in its task. If the swarm has a high level of autonomy, this would not be the case. An optimal operational shared autonomy between a swarm and an operator depends on the complexity of the mission and environment. An operator should only submit commands at a strategic level. Of course, a complex mission could need submitting commands at a tactical level. The strategy chosen will influence the number of robots deployed.
With more robots composing the swarm, it becomes more difficult for the operator to control the swarm behavior considering all constraints such as battery voltage or state of charge, the current state of the mission and what has been accomplished in the mission.
4.1. Swarm Interaction Taxonomy
This section will present the studies that have been conducted for this purpose.
Figure 1 shows a possible taxonomy for these different means of interaction depending on the support used. In this figure, hybrid method is possible such as using Augmented Reality to see the swarm, Haptic to control the structure of the swarm and electrocardiogram to control, as an example, the velocity and orientation of the swarm.
In their article, Bowley et al. [
30] proposed to control a swarm of robots from a phone or tablet with their touch screen. It has several functions that can be used due to finger movements (touching or removing fingers, scanning the screen, enlarging or reducing with two fingers, etc.). With this interface, the operator uses an algorithm to influence the behavior of the swarm through several attractive or repulsive beacons:
It attracts the robots swarm towards its position.
It emits repulsive force so that the robots avoid going to its zone and thus avoid collision with the obstacle.
Similar to attractive beacon, it is used in an emergency or at the end of a test exercise.
It is supposed to lead the swarm towards this target.
It is a mix between the attractive, obstacle and the management beacon. It is used for zone control.
It is used to change the perception of the environment of robots in an area in order to change their behavior accordingly.
Each of the beacons located on the screen has a modifiable influence radius. Simulations were carried out to validate the operation of this concept, which allows the behavior of a swarm of robots to be intrinsically modified.
Crandall et al. [
31] developed an interface that allows an operator to interact directly with a swarm of modeled mobile robots following a bee colony. This is done in order to share decision making process and offer fault-tolerant capabilities. Thus, the goal of the swarm is to find quality sites to collect resources. Each robot behaves like a bee. It can enter different states: exploration, observation, pause, evaluation and dancing as a message. Each bee will initially explore an area at random. If she encounters a potential site, she will evaluate it and go back to the colony to dance more or less according to the quality of the site. Then she rests before starting the cycle again. Observers watch bees dance to visit potentially interesting sites. If many bees have detected a good site, the colony will exploit it. Initially, the project performed computer simulations of a bee colony. Subsequently, they wanted to improve the safety and speed of bee exploration. To do so, they allowed an operator to place beacons to guide bees in their tasks, and then they evaluated the impact of this interaction on the robots swarm. From this experience, they were able to define several categories of control on the swarm:
It can be achieved by exciting or inhibiting the behavior of bees in their exploration whether by specifying a direction of research or altering their speed.
The operator can directly control one robot of the swarm, which will then influence the overall swarm.
This is done by placing attractive or repulsive beacons in the bee environment.
It is to ensure that the swarm changes the allocation of its own goal in order to select the best strategy to adopt. In this case, it would be to reassess the quality of a site after a certain operating time.
In conclusion, Crandall et al. [
31] admit that these methods of influence work well if the operator knows exactly how to give the tasks to be carried out by the swarm and accepts sharing its control with others.
Kim et al. [
32] developed a swarm of mobile robots capable of tracking people’s movement. The system consists of three steps: (1) sequence of operation, (2) receiving/sending messages and (3) approximate location of robots. This interaction takes place through a connected watch and a connected belt. The swarm is composed of a leader who receives orders from the watch via a Bluetooth Low Energy (BLE) communication. The belt is used to assess the distance between the person and the swarm through infrared communication. The leader then sends instructions to the other robots by radio and infrared communication. The authors created the communication protocol for this swarm in order to keep it in formation. This system works for a small number of robots. Indeed, the system is tested with real mobile robots and realized that communication becomes noisy if the number of robots is high. The user can choose the formation of the swarm when moving according to several prefixed patterns.
To interact in various ways with a swarm of robots, Ferrer [
33] made an enumeration of various physical supports existing for this purpose. First of all, he takes a gesture taxonomy from the existing hand to be able to apply it to a swarm of mobile robots. This gesture recognition is done via a camera that associates the gesture with a command to be made for the swarm. Of course, hand gesture could also be executed with an electromyography (EMG) such as with an eight-channel armband [
34]. In their paper, Mendes et al. described how they can obtain better results by selecting the best feature reduction process of EMG signals data before the classification of gestures. Then another method of communication with the swarm is presented. Several studies have been carried out on the interaction between a swarm and a human via the haptic, especially with the aim of obtaining feedback from others instead of using visual information to help the operator in his control. The operator uses some haptic sensors which send some feedback to him. It does not make a human being an external operator of the swarm, but rather a special member of the swarm. Both methods are hard enough to put in work and cannot allow interacting with a large swarm. Subsequently, various means of interaction by augmented reality are presented. Finally, Ferrer concludes on portable tools on a human that can act as a support for interaction between a swarm and an operator. First, a gesture recognition can be done by an armband that can recognize the gestures of the fingers, hand and wrist thanks to the muscles of the forearm. The armband used was a Myo armband by Thalmic Labs. With each of these gestures, we can associate a command with the swarm. Then, usually for gesture recognition, it is possible to use the Leap Motion [
35] to detect the movement of the fingers via infrared light. It identifies the gestures of the fingers, their movements and their spatial coordinates if necessary. It is a precise tool that can provide a wide range of control for an operator. The last physical support presented is a vest for video game players acting as a connected garment. It is equipped with haptic devices that allow the user to feel immersed in a chosen environment. Ferrer concludes by comparing the advantages and disadvantages of different media of interaction.
In their work, Mc Donald et al. [
36] developed a method of interaction with a swarm of mobile robots based on haptic. The purpose of the robots swarm is to carry out patrols and encircle buildings at the request of an operator. When robots encircle a building, they are represented by virtual force fields which then allow the formation of the swarm to be represented by a flexible virtual ring. The operator can perform three types of handling when the robots are in encirclement mode:
The haptic tool allows the operator to feel the shape of the swarm without changing it. This is possible because of the virtual force field created by mobile robots.
This mode allows the operator to modify the formation of the swarm by means of the haptic remote control which changes the shape of the virtual ring.
In normal mode, the spacing between each robot is identical. This mode allows the operator to change these values. The operator also has actions to perform during the patrol of mobile robots.
This mode activates if the swarm has selected its target position to be reached and it is not in encirclement mode. Its purpose is to allow the operator to reach the target position faster.
During the work of the swarm, the operator may choose to feel the formation chosen by it without modifying it.
Mc Donald et al. were able to simulate their systems in order to validate them and test the effects of this physical medium on the performance of the operator’s controls on the swarm of mobile robots.
Kapellman et al. [
37] suggested using Goolge Glass as physical support. These allow an operator to guide a swarm of robots for the transportation of an object. One of the robots is appointed as the leader of the swarm, the one that the operator can influence. It acts as an intermediate target which the other robots are going to recognize and follow. The operator has the possibility of choosing the leader among the robots of the swarm. He can also check the state of each robot by selecting them and communicating orders via Bluetooth:
It is the basic behavior of the robot that is activated.
Movement of the robot can be directly controlled by the operator (go ahead, back, turn right/left, stop).
The robot must ignore all commands from a remote control other than glasses.
Via connection.
These instructions can be given by the voice command or by touching the glasses. This support could be tested with a real swarm of mobile robots. This medium allows the operator to have free hands to perform other actions. It was also demonstrated that interaction allows for dynamic selection of the target to reach.
In their work, Mondada et al. [
38] decided to process Control operator’s electroencephalography (EEG) signal so that it can select a swarm’s robot to control it. It is based on the stationary state of the potential evoked by vision (Steady-State visually evoked Potential: SSVEP). This detection is done by flashing light on each robot, allowing knowing whether the selected robot is the one the operator wants. For this, an EEG acquisition helmet is placed on the operator’s head. Three parameters are important to extract the SSVEP signal from the EEG: the flashing frequency of the lights, the color of the lights and the distance to the stimulus. Mondada et al. [
38] used existing literature to select the ranges of parameters to be tested. The blinking frequencies were chosen according to [
39] study. The distance between the target and the operator was chosen according to [
40] study. For the color of the LED, the scientific community is not able to give the best one (there is some debate between white, red, green and blue). Several tests were conducted with individuals. The results indicate that the success rate varies greatly from person to person (on average 75% success with a standard deviation of around 15% success depending on the frequencies used). More trained operators are in this process, the better the results will be. This method also delays for several seconds in the recognition of the signal, as does gesture recognition by image or voice. The main disadvantages are the uncontrollable factors for a real application such as the personal attitude of the different operators, the distance from the robots, the brightness, etc.
In their article, Setter et al. [
41] used the haptic to obtain feedback about the swarm of mobile robots. The swarm used is made up of a leading robot and other followers robots that maintain a given formation. The operator can control the speed of the leader, which can influence the behavior of the swarm. This is done through a haptic device. The feedback given by the force of the haptic device indicates to the operator whether his control is good or bad for the swarm, that is to say whether the speed of the following robots is more or less different from that of the leading robot. This information allows the operator to adjust the leader’s speed. The system was successfully experimented with a real swarm of mobile robots.
Podevijn et al. [
42] developed a gesture recognition interface capable of ordering a swarm of mobile robots. A Microsoft Kinect RGB-D sensor is used for body tracking and to identify the gestures of the user This interface allows the operator to dedicate himself fully to the management of his swarm. The contribution is to have a simple command interpreted by the swarm of decentralized robots and also to allow it to give some feedback. Since a swarm is too difficult to command directly, the swarm could be subdivided it into several sub-swarms. The following commands are used by the operator:
The operator can guide a sub-swarm to a target position.
The sub-swarm stops.
Creation of new sub-swarms.
Gathering of two sub-swarms.
The operator chooses the sub-swarm with which he wants to interact.
Each of these controls is associated with a gesture of the operator’s arms. Eighteen participants were able to test this interface with a real swarm of mobile robots.
Kolling et al. [
43] provided a 2D graphical interface, which is optimized to display only important information for the operator, to simulate interaction with a swarm of mobile robots. The robots move following Voronoï graphs based on [
44], in the environment to be explored. For each new information retrieved, they must return to a departure station that will update the swarm movement card. The operator can visualize these movements from his interface and interact with a mouse on the swarm via a few commands: stop, go to a zone, appointment point, deployment, random movement, update data, leave a zone. It can also use other means of control, such as a robot selection rectangle, which defines a sub-swarm that is obedient to different commands of the swarm in general, and also places a beacon that attracts robots to its area.
Diana et al. [
45] used a joystick made of modeling paste as a physical medium for interaction. This allows the operator to control the formation of the robots swarm with the geometry of the modeling paste. It uses modeling paste to define the desired formation for its swarm. A camera is used to take the form (scan the geometry) and compare it to a library for the identification and classification of the geometry. Once this is done, the information is sent to the swarm that performs the desired formation using a method that minimizes the energy of the system during its displacement. Simulations were carried out with a real swarm of mobile robots.
Alessandro et al. [
46] developed a human-swarm interaction based on the recognition of hand gestures. For this, 13 gestures was used and 70,000 images of these gestures was collected by cameras representing the position of all the fingers of the hand. These data were used to train a vector support machine that will perform the classification of the 13 gestures by affecting a probability of belonging to a category of the gesture to be recognized. Every swarm robot has a camera on it. The robot move around the operator to improve their point of view and facilitate gesture recognition. The robots then share the information obtained by their classification and the swarm makes a decision afterwards.