A Multi-Robot Collaborative Exploration Method Based on Deep Reinforcement Learning and Knowledge Distillation

Abstract

Multi-robot collaborative autonomous exploration in communication-constrained scenarios is essential in areas such as search and rescue. During the exploration process, the robot teams must minimize the occurrence of redundant scanning of the environment. To this end, we propose to view the robot team as an agent and obtain a policy network that can be centrally executed by training with an improved SAC deep reinforcement learning algorithm. In addition, we transform the obtained policy network into distributed networks that can be adapted to communication-constrained scenarios using knowledge distillation. Our proposed method offers an innovative solution to the decision-making problem for multiple robots. We conducted experiments on our proposed method within simulated environments. The experimental results show the adaptability of our proposed method to various sizes of environments and its superior performance compared to the current mainstream methods.

1. Introduction

The problem of multi-robot autonomous exploration involves the collaborative efforts of multiple robots to explore and create a map of an unknown environment autonomously [1]. This technology possesses a wide range of potential applications, including but not limited to floor-sweeping robots [2], search and rescue operations [3], and the construction of warehouse maps [4].

Multi-robot collaborative autonomous exploration methods can be classified as centralized [5] and decentralized [6]. Centralized execution approaches must prioritize maintaining reliable communication between individual robots throughout exploration. Consequently, this technique is unsuitable for scenarios with communication constraints.

Traditional distributed multi-robot collaborative autonomous exploration approaches often rely on heuristics to direct robots in choosing target points for exploration. These heuristics may consider factors such as the distance between the robot and the frontier (the boundary between free and unknown spaces) [7] or the amount of new information that can be obtained at a target point [8]. Conventional approaches are constrained by reliance on heuristics and exhibit poor performance in complex environments, particularly when communication is constrained.

The advancement of deep reinforcement learning (DRL) technology [9] has led to its progressive integration into robotics, demonstrating more excellent performance than primitive methods in various domains. Niroui et al. presented a novel DRL-based method to enable a single robot to explore unknown environments autonomously [10]. This methodology demonstrated superior performance compared to conventional methods. Additionally, specific multi-agent deep reinforcement learning (MADRL) techniques, such as VDN [11], MADDPG [12], and QMIX [13], are designed to address multi-agent decision-making problems. He et al. introduced a multi-robot autonomous exploration technique utilizing MADDPG [14]. The method involves selecting the closest frontier position as the exploration target point in a specific direction. However, due to the limited action space, not all frontier positions can be included, resulting in a significant constraint on the selection range of the target point. Meanwhile, utilizing multi-agent deep reinforcement learning methods directly with the centralized training with decentralized execution (CTDE) [15] structure requires complete retraining when modifying the network size, which consumes significant time interacting with the environment.

According to the limitations of the existing traditional and DRL-based methods, we are prompted to propose a new way of thinking to solve the problem of autonomous exploration of multiple robots in unknown environments. We have enhanced the SAC deep reinforcement learning technique [16] to accommodate multi-robot systems. We consider the multi-robot team as a whole and employ the improved SAC deep reinforcement learning method to acquire centrally executable autonomous exploration strategies for the team. Subsequently, we employ the knowledge distillation technique [17] to convert multi-robot autonomous exploration strategies that necessitate centralized execution into a network of various strategies that may be executed in a distributed manner on individual robots.

In general, the contributions of this paper include the following:

• We provide an enhanced deep reinforcement learning approach that builds upon the SAC deep reinforcement learning algorithm, enabling it to address decision-making challenges in multi-robot systems.
• We view the multi-agent system as a whole and obtain a policy network that allocates decisions to individual agents via centralized training. This network is converted into several distributed policy networks that can be implemented on individual agents with knowledge distillation techniques. Our proposed strategy offers an innovative approach to address the multi-agent collaboration task.

2. Related Work

2.1. Multi-Robot Exploration

Both traditional multi-robot autonomous exploration approaches and DRL-based multi-robot autonomous exploration approaches commonly employ the notion of frontier. A frontier is a boundary line where the free space of the environment meets the unknown space while a robot is exploring an unknown environment.

The Nearest Frontier method proposed by Yamauchi et al. is a straightforward method for multi-robot autonomous exploration [7]. The approach involves iteratively choosing the nearest frontiers to the robots for exploration. Stachniss et al. proposed utilizing AdaBoost [18] to train a classifier that can recognize the type of environment the robot is in to assist with exploration [19,20]. Haumann et al. introduced the Voronoi partitioning concept [21] and developed an exploration approach that optimizes an objective function. This function includes costs related to distance and direction and estimated information gain [22]. Colares et al. proposed using a utility function that considers information gain and distance cost to guide robots in exploring based on the frontier-based method [8]. Bautin et al. introduced a computationally efficient technique for assigning frontiers that considers the spatial distribution of the robots in the environment and reduces redundant scanning of the environment by the robot [23]. Similar utility-based approaches can be found in [24,25]. Aside from utility-based exploration approaches, some strategies introduce market-based multi-robot co-ordination approaches [26] to guide robots in distributed collaborative exploration [27,28,29].

In addition to conventional approaches, learning-based methodologies are utilized for multi-robot autonomous exploration. He et al. introduced a multi-robot autonomous exploration technique utilizing MADDPG [14]. The method involves selecting the closest frontier position as the exploration target point in a specific direction. He et al. suggested employing multi-agent deep reinforcement learning methods to develop a strategy for guiding a robot’s exploration [14]. This strategy involves analyzing the robot’s current state, selecting a direction, and identifying the nearest frontier position in that direction as the target. Zhang et al. introduced a coarse-to-fine exploration approach [30]. They combined graph neural networks (GNNs) with MADRL to help robots implicitly learn co-operative strategies. By applying the notion of macro actions [31], Tan et al. created a decentralized exploration planner based on macro actions that enabled the robot to consider the intentions of its teammates while conducting exploration [32].

2.2. Deep Reinforcement Learning

Developing artificial intelligence (AI) systems that can learn and respond effectively has been a persistent and challenging problem. Reinforcement learning (RL) is a mathematical framework that enables autonomous agents to acquire knowledge independently [33]. RL-based methods have achieved success in solving specific uncomplicated decision problems. Kohl et al. employed RL to direct the movement of a quadrupedal robot [34]. Ng et al. developed a controller for inverted helicopter flight using RL [35]. While RL-based methods have shown benefits in specific straightforward decision-making problems, they have become impractical for solving complex decision-making problems due to the issue of the curse of dimensionality.

With the development of deep neural network-based techniques, learning-based methods are gradually being used to solve decision-making tasks. A representative event was the building of the top Go program by Silver et al. using DRL methods [36]. Recently, DRL algorithms have been progressively used in robotic systems, including path planning [37], robotic manipulation [38], and autonomous driving [39]. SAC [16] is presently a widely utilized and highly effective DRL technique. This approach introduces the idea of maximizing the entropy of the action distribution. This enables the agent to explore actions more effectively, avoid falling into local optima, and maintain stable learning. Deep reinforcement learning algorithms for multi-agents commonly employ a CTDE scheme, such as MADDPG [12] and QMIX [13].

3. Methodology

This section defines the multi-robot autonomous exploration problem and presents our approach.

3.1. Problem Formulation

A multi-robot autonomous exploration mission refers to the systematic procedure wherein a group of robots, $I=\{I_1,I_2,\dots ,I_n\}$ comprising many units traverse an unknown environment and progressively create a global map, M, of that environment by collecting data through their onboard sensors. The global map, M, is stitched together from local maps $\{m_1,m_2,\dots ,m_n\}$ built by individual robots. In this study, each robot senses its surroundings using a LiDAR carried by itself with a detection range of 25 m. During exploration, the robots build a map of the unknown environment by scanning obstacles around themselves with LiDARs. Grid maps are commonly employed to represent environmental maps. These grids can be classified into three distinct types: unknown space, free space, and obstacle space. The term “unknown space” denotes an uncharted region yet to be explored, “free space” pertains to an unobstructed area that has been thoroughly investigated, while “obstacle space” designates a region characterized by the presence of obstacles. While a team of robots is exploring an unknown environment, they can interact with each other by exchanging local maps, robot positions, and goal positions. This communication is possible as long as the distance between the robots is smaller than $d_c$, which is 30 m in this article. The objective of multi-robot autonomous exploration is to minimize the time spent while completing the exploration of the unknown environment:

$$min\left[J=\sum _{i=1}^T{t}_i\right],$$

(1)

where T denotes the total number of time steps, and $t_i$ denotes the duration of the i-th time step.

3.2. Technical Framework

In this study, we present an innovative deep reinforcement learning framework for distributed multi-agent systems, which leverages the concept of knowledge distillation. This framework divides training into two phases: centralized reinforcement learning and knowledge distillation, as shown in Figure 1. During the phase of centralized reinforcement learning, the collective of robots is perceived as a unified agent, and the SAC approach is employed for reinforcement learning. Notably, the SAC approach employed in this paper is an enhanced version designed explicitly for multi-robot decision-making. During the knowledge distillation phase, the actor network acquired through the first phase of SAC is converted into the policy network for each robot using the knowledge distillation technique, which enables the multi-robot autonomous exploration policy to be distributed. In knowledge distillation, the observation data in the replay buffer is directly used for knowledge distillation, which avoids interaction with the environment and can save training time. There are two notable advantages to this approach. Firstly, it can leverage the efficiency of the SAC deep reinforcement learning method. Secondly, there is no necessity to re-interact with the environment while resizing the final network, resulting in significant time savings for training.

3.3. Centralized Reinforcement Learning

Centralized reinforcement learning is the first part of our proposed methodological framework. This part is based on the SAC deep reinforcement learning method to obtain a multi-robot autonomous exploration strategy that can be executed centrally. We improve the entropy calculation in the original SAC to be used for multi-robot decision-making problems. During the training process, the proposed method considers all robots as a whole, and the actor network directly outputs the actions of all robots.

3.3.1. Modeling

A Markov Decision Process (MDP) is defined by a tuple $<S,A,R,T,\gamma >$, where S is a set of states, A is a set of action vectors, R is a reward function $R(s,\overrightarrow{a})$, T is a transfer function $T\left(s^{\prime }\right|s,\overrightarrow{a})$, and $\gamma$ is a discount factor. In this paper, the value of $\gamma$ is empirically taken as 0.99. When the robots take actions, $\overrightarrow{a}=\{a_1,a_2,\dots ,a_n\}$, at state s, the environment reaches a new state, $s^{\prime }$, according to the probability distribution $T\left(s^{\prime }\right|s,\overrightarrow{a})$ and returns a reward $R(s,\overrightarrow{a})$. The goal is to find a centralized strategy, $\pi$, that allows a team of robots to capture the maximum cumulative reward when completing exploration tasks:

$${\pi }^{*}=\underset{\pi }{argmax}\mathbb{E}\left[\sum _{t=0}^{T-1}{\gamma }^{t}R(s_t,\overrightarrow{a_t})|s_0,\pi \right],$$

(2)

where T is the total number of time steps, and $s_0$ is the initial state.

3.3.2. Multi-Robot SAC

In this section, we present the Multi-Robot SAC (MR-SAC). The actor network in the primitive SAC method [16] outputs a single action probability distribution. In contrast, the actor network in this study’s modified SAC method optimized for the multi-robot decision-making problem outputs a set of action probability distributions, $\{p_1,p_2,\dots ,p_n\}$, to be assigned to individual robots in the robot team. In the method proposed in this paper, the entropy of the action probability distribution is calculated separately for each robot, while the corresponding temperature parameters are optimized individually. Thus, the soft Bellman residual used to train the parameters of the soft Q-function is given by the following equation:

$$J_Q\left(\theta \right)=\mathbb{E}\left[{\displaystyle \frac{1}{2}}{(Q_{\theta }(s_t,\overrightarrow{a_t})-(R(s_t,\overrightarrow{a_t})+\gamma \mathbb{E}\left[V_{\overline{\theta }}\left(s_{t+1}\right)\right]))}^2\right],$$

(3)

where

$$V_{\overline{\theta }}\left(s_t\right)=\mathbb{E}\left[Q_{\overline{\theta }}(s_t,\overrightarrow{a_t})-\sum _{i=1}^n{\alpha }_{i}log{p}_i\left(a_{ti}\right|s_t)\right],$$

(4)

$s_t$ and $\overrightarrow{a_t}=\{a_{t1},a_{t2},\dots ,a_{tn}\}$ (sampled from the replay buffer) denote the state at the t-th time step and the action taken by the actor network, respectively, and $p_i$ and ${\alpha }_i\in \overrightarrow{\alpha }$ are the action probability distribution and temperature parameter of the i-th robot respectively, and finally, $\theta$ and $\overline{\theta }$ are the parameters of the policy networks and target networks, respectively. The temperature parameter determines the relative importance of the entropy term to the reward, thus controlling the stochastic nature of the optimal policy [16]. Although we view the entire team of robots as a single agent, we perform independent entropy calculations and assign independent temperature parameters to each robot for the sake of the stochastic nature of each robot’s strategy in the team, which is very different from the traditional SAC approach.

Similarly, the loss function of the actor network is given by the following equation:

$$J_{\pi }\left(\varphi \right)=\mathbb{E}\left[\sum _{i=1}^n{\alpha }_{i}log\left(p_i\left(a_{ti}\right|s_t,\varphi )\right)-Q_{\theta }(s_t,\overrightarrow{a_t})\right],$$

(5)

where $\varphi$ represents the parameters of the actor network.

Optimizing the temperature parameter of each robot separately can make the entropy of each robot’s action probability distribution converge to the target entropy. The loss function for the temperature parameter is

$$J\left({\alpha }_i\right)=\mathbb{E}\left[-{\alpha }_{i}log{p}_i\left(a_{ti}\right|s_t)-{\alpha }_i{\mathcal{H}}_d\right],$$

(6)

where ${\mathcal{H}}_d$ is a hyper-parameter denoting the desired entropy of the probability distribution of each robot’s action.

The complete process of Soft Actor-Critic for Multi-Robot Decision Making is shown in Algorithm 1.

Algorithm 1 Soft Actor-Critic for Multi-Robot Decision Making
Input: ${\theta }_1,{\theta }_2,\varphi ,\overrightarrow{\alpha }$	▹ Initial parameters.
1: ${\overline{\theta }}_1\leftarrow {\theta }_1,{\overline{\theta }}_2\leftarrow {\theta }_2$	▹ Initialize target network.
2: $\mathcal{D}\leftarrow \varnothing$	▹ Initialize replay buffer.
3: for each iteration do
4:     for each environment step do
5:         $a_{ti}\sim p_i\left(a_{ti}\right|s_t,\varphi )$, $1⩽i⩽n$	▹ Sample actions from the policy.
6:         $s_{t+1}\sim T\left(s_{t+1}\right|s_t,\overrightarrow{a_t})$	▹ Sample a transition from the environment.
7:         $\mathcal{D}\leftarrow \mathcal{D}\cup \left\{(S_t,A_t,R(s_t,\overrightarrow{a_t}),s_{t+1})\right\}$      ▹ Store the transition in the replay buffer, where $S_t=\left\{s_i\right|1⩽i⩽t\}$, and $A_t=\left\{\overrightarrow{a_i}\right|1⩽i⩽t\}$. The complete sequence is saved to train the LSTM module.
8:     end for
9:     for each gradient step do
10:         ${\theta }_i\leftarrow {\theta }_i-{\lambda }_Q{\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right),i\in \{1,2\}$	▹ Update policy critic network weights.
11:         $\varphi \leftarrow \varphi -{\lambda }_{\pi }{\widehat{\nabla }}_{\varphi }{J}_{\pi }\left(\varphi \right)$	▹ Update actor network weights.
12:         ${\alpha }_i\leftarrow {\alpha }_i-\lambda {\widehat{\nabla }}_{{\alpha }_i}J\left({\alpha }_i\right),1⩽i⩽n$	▹ Adjust temperature.
13:         ${\overline{\theta }}_i\leftarrow \tau {\theta }_i+(1-\tau ){\overline{\theta }}_i,i\in \{1,2\}$  ▹ Update target critic network weights, where $\tau$ is the soft update coefficient.
14:     end for
15: end for
Output:${\theta }_1,{\theta }_2,\varphi ,\alpha$	▹ Optimized parameters

3.3.3. Learning

During exploration, the arrival of one of the robots at a goal point triggers a decision to assign a new goal point to each robot. We designed each robot’s action space as a set of all frontier center locations. To ensure that each robot is assigned a different goal point, it adds its position to the action space when the number of frontiers is less than the total number of robots, allowing it to stay stationary at this time step.

The reward function consists of five components: area explored, area of scanning overlap, distance traveled by the robot, cost of time, and number of decisions:

$$\begin{array}{cc}r=& r_{explore}+r_{overlap}+r_{travel}\\ & r_{time}+r_{decision}\end{array},$$

(7)

$$r_{explore}=s_{cell}·\left|U_t\cap \left(\bigcup _{i=1}^n{E}_i^t\right)\right|,$$

(8)

$$r_{overlap}=-0.3s_{cell}·\sum _{i=1}^{n-1}\sum _{j=i+1}^n\left|E_i^t\cap E_j^t\right|,$$

(9)

$$r_{travel}=-\sum _{i=1}^n{d}_i^t,$$

(10)

$$r_{time}=-\Delta t,$$

(11)

$$r_{decision}=-5,$$

(12)

where $s_{cell}$ denotes the area of each grid, $U_t$ denotes the unknown space, $E_i^t$ denotes the free space region scanned by the i-th robot in this step of exploration, n is the total number of robots, $d_i^t$ denotes the length of the distance traveled by the i-th robot in the step of exploration, and $\Delta t$ denotes the time spent in this step of exploration. The item area explored is the area of new free space discovered by the robot team at each exploration step. We designed the scanning overlap penalty term to drive the robots to spread out during exploration, defined as the area of the overlapping region scanned by two robots, where 0.3 is the weighting factor. After each exploration step, we drive the trained strategy to spend as little distance and time as possible by giving penalties based on the distance traveled and time spent by the robot, respectively. After each decision step, a fixed penalty prevents the robot from always choosing its own position as the goal point.

We use spatial information and a scale to represent a state. The spatial information is a two-dimensional tensor of $4+n$ layers corresponding to the unknown space, the free space, the obstacle space, the frontier centers, and the positions of the n robots. The scale used to represent the state is the edge length corresponding to each element in the tensor. Before generating the tensor and the corresponding scalar, the information needs to be preprocessed so that its data dimensions match the network input. The preprocessing process is shown in Figure 2 and can be divided into cropping, scaling, and padding:

• The smallest rectangular region containing valid information is boxed out, and irrelevant unknown regions are excluded;
• The boxed rectangular region is scaled so that the number of pixels on its long side matches the input dimension of the network (100 pixels in this study);
• The scaled rectangular region is padded into a square with unknown space, making it the same dimension as the input to the network (100 pixels × 100 pixels in this study).

Each layer in the tensor is a binary map with positive regions of 1 and negative regions of 0.

The structure of the actor network is shown in Figure 3 and consists of a map feature extractor and n policy heads, where n is the number of robots. The inputs to the actor network are categorized into spatial information and scale. The spatial information contains the map information, robots’ positions, and candidate target positions, and the scale is the side length of each pixel of the map after preprocessing (in meters). The map feature extractor obtains the feature vector corresponding to the current state from the input data. The strategy head obtains the corresponding action probability distribution of each robot through the spatial information and the feature vector corresponding to the current state. The filter in the policy head assigns the tensor of the non-frontier position to negative infinity so that the probability of the action corresponding to the non-frontier position is 0 after the softmax operation.

The structure of the critic network is shown in Figure 4. The input data of the critic network contains the input data of the actor network as well as the action probability distributions of the individual robots’ output by the actor network. The expected Q value of the action probability distribution is output by the critic network to evaluate the goodness of the action probability distribution.

3.4. Knowledge Distillation

Knowledge distillation is the second part of our proposed method, consisting of one teacher and n student networks, where n is the number of robots. We use the actor network obtained by training in MR-SAC to train the decentralized policy networks using offline knowledge distillation. In this case, the teacher network is the actor network trained in MR-SAC, and the student networks are the distributed policy networks that can be used on individual robots. In this section, we propose a novel approach to allow student networks to learn communication-invariant features by feeding global and local observations to the teacher network and student networks, respectively.

3.4.1. Training Dataset Generation

Primitive offline knowledge distillation is performed by feeding the same data to the teacher network and the student network and training the student network with the output of the teacher network as the labels. However, the direct use of the same input data does not allow the policy network obtained using knowledge distillation to be adapted to communication-constrained scenarios. We feed the teacher network the complete observations of all robots and feed the student networks the local observations available to individual robots in communication-constrained scenarios to force the student networks to be able to output big-picture strategies even when only local observations are available. During the centralized reinforcement learning process, we record the positions of individual robots at different time steps and the scanned environment information to generate the training dataset needed for the knowledge distillation session. On the one hand, we use these records to generate global observations as input to the teacher network. On the other hand, local observations of different robots are generated as inputs to the student network by introducing communication range and communication success rate in these records. When the distance between two robots is less than the communication range, they exchange information about their local maps. We also randomly simulate robots with communication failures when generating training data, and when a robot has a communication failure, it does not exchange information with other robots for 5 s.

3.4.2. Training Method

We use the actor network trained in MR-SAC as the teacher network and train the student network with the generated training dataset.

The training method is shown in Figure 5. The student networks are the policy networks used by each robot for independent decision-making. A student network is similar in structure to the teacher network, but smaller than the teacher network. The input to the teacher network is a global observation not constrained by communication; in contrast, the input to a student network is a local observation constrained by communication. The teacher network outputs action probability distributions for all robots, while a student network outputs an action probability distribution for only the corresponding robot. The action with the highest probability in the action probability distribution of each robot output by the teacher’s network is denoted as $a_{mi}$$(1\le i\le n)$. The center of each frontier is extracted through local observation as the action space of the corresponding robot, and the closest position in the action space to $a_{mi}$ is chosen to be noted as $a_{pi}$$(1\le i\le n)$ as the label to train the corresponding student network, where n is the number of robots. In most cases, $a_{pi}$ and $a_{mi}$ overlap in the environment.

This training method can force the robots to make decisions with a big-picture perspective even when only receiving information from local observations. At the same time, the knowledge distillation process does not require the robot to re-interact with the environment, and the size of the student network can be easily adjusted.

4. Training

The training process is divided into two parts: MR-SAC and knowledge distillation. The robot team is viewed as an agent that interacts with the environment. The MR-SAC approach is then utilized to train and develop a policy network. The knowledge distillation process converts the policy network obtained from MR-SAC, which requires centralized execution, into several policy networks that can be executed distributively. We use the first three environments shown in Figure 6 for training, with different obstacle distributions, sizes, and aspect ratios. The training involved scenarios of robot teams consisting of three, four, and five robots, utilizing two Intel Xeon Silver 4210 CPUs (Intel, California, USA) and one NVIDIA RTX 2080 Ti GPU (NVIDIA, California, USA).

In the MR-SAC part, it took an average of 300 h to complete training for one size of robot team. We cycled through the first three environments shown in Figure 6 for training and recorded the total reward received by the robot team each time it completed the exploration of Environment 1. The cumulative reward and elapsed exploration time during training are shown in Figure 7. The network can be trained to converge in around 80,000 episodes for every robot team size. Following the convergence of training, the team of five robots achieved the lowest cumulative reward but completed the exploration in the shortest time. This was due to a relatively high penalty term from overlapping scanning regions ($r_{overlap}$).

Within the knowledge distillation part, we designed student networks of varying sizes. The student networks share the same fundamental design, resembling the actor network in MR-SAC (depicted in Figure 3) but with only one policy head. The student networks sequentially pass through a map feature extractor, which is composed of multiple convolutional layers, two fully connected layers, one recurrent neural network (RNN), and a policy head that includes one fully connected layer, multiple transposed convolutional layers, and multiple convolutional layers. The batch size for the training procedure was set at 64, and an Adam optimizer was employed with an initial learning rate of 1 × ${10}^{-3}$. After completing a single epoch of training, the policy network that had undergone training was employed to explore Environment 1 (shown in Figure 6). Once the average value of the elapsed time of the last 100 completed explorations reaches a point of stagnation, the training process is considered to have reached convergence, leading to the termination of training.

Various policy networks of various sizes were established for training in the knowledge distillation part. Regardless of their respective sizes, all policy networks could achieve convergence within 1 h. According to

Table 1. Different student network depths.

Index	C1	RNN	TC	C2	Elapsed Time (s)
1	2	GRU	3	2	181.0, 170.6, 160.1
2	2	GRU	4	3	179.1, 162.6, 159.4
3	3	GRU	3	2	177.3, 164.3, 160.3
4	3	GRU	4	3	173.2, 160.1, 156.5
5	4	GRU	3	2	176.5, 161.5, 154.9
6	4	GRU	4	3	170.7, 156.3, 154.1
7	2	LSTM	3	2	182.0, 165.9, 160.8
8	2	LSTM	4	3	175.3, 164.5, 160.7
9	3	LSTM	3	2	176.9, 163.5, 159.4
10	3	LSTM	4	3	170.8, 159.8, 154.3
11	4	LSTM	3	2	175.0, 158.5, 154.4
12	4	LSTM	4	3	172.3, 159.1, 149.1

, there are 12 distinct sizes of student networks. In this table, C1 represents the number of layers in the convolutional layer of the map feature extractor. RNN represents the selected type of RNN, TC represents the number of layers in the transposed convolutional layer of the policy head, and C2 represents the number of layers in the convolutional layer of the policy head. The elapsed time in the table is the average elapsed time of a robot team consisting of three, four, and five robots completing exploration 100 times in Environment 1 (shown in Figure 6), respectively. At the start of each exploration, the robot team is assigned a random location on the map, and the timer starts. The timer ends when the robot team finishes exploring. All training was conducted with a communication success rate of 80%. According to the data presented in Table 1, the network’s performance improves as its size increases. However, the kind of RNN has little impact on performance.

5. Experiment

We conducted a comparative analysis of our proposed method (the networks are indexed as 6 in Table 1) and other multi-robot autonomous exploration methods in a simulated environment. The exploration of the 4th to 8th environments in Figure 6 was accomplished using these methods to compare the performance of the different methods. We built the simulation environment used for the experiments in CoppeliaSim. Within the simulation experiment, we set the communication success rate. The robot had a probability of communication failure with each communication attempt. When this failure occurs, the robot cannot communicate with other robots for 5 s. Furthermore, the generalization ability of our suggested approaches may be verified by considering the varying aspect ratios and sizes of these environments. The experiment compares the method suggested in this research, two classic methods, and a DRL-based method.

• Nearest Frontier [7]: The robots consistently choose the nearest positions on the frontiers as target points to explore;
• Information Gain [8]: The methodology involves the selection of goals for individual robots through the calculation of information gain resulting from the exploration of various frontier locations. This gain is then combined with the path length cost. Additionally, a co-ordination factor is employed to encourage the robot formation to disperse, thereby enhancing exploration efficiency;
• DME-DRL [14]: DME-DRL is based on the MADDPG deep reinforcement learning method with the introduction of structural information and time series. The robots employ this technique to choose the closest frontiers in specific directions as goal points for exploration.

We have tested the performance of different methods in different environments, and the results are shown in Figure 8. Our proposed method achieved the best results in most of the environments. This experimental result shows, on the one hand, that our proposed deep reinforcement learning-based exploration strategy outperforms the current mainstream exploration methods in terms of exploration efficiency. On the other hand, the knowledge distillation technique enables the robot team to explore unknown environments stably despite the communication constraints.

We used different methods to conduct 100 explorations in Environment 7, which has the largest free space, measuring 3093.1 m², using a team of four robots and a communication success rate of 80%. As shown in Figure 9, we collected data on the average area explored for each method over time. The figure shows that our proposed strategy demonstrated an early advantage in the exploration job. This is because our proposed method can effectively decentralize the robot’s formation.

6. Conclusions

This study presents a novel technique for treating a multi-agent system as a whole and learning the strategies using a single-agent deep reinforcement learning method. The learned strategies can be executed in a distributed manner using knowledge distillation. We apply this method to a multi-robot autonomous exploration task, and the training process demonstrates that our suggested approach can effortlessly modify the size of the policy network without the need to interface with the environment repeatedly. This capability significantly reduces the time required for environmental interaction. The experimental results indicate that our suggested method can adjust to environments of varying sizes and aspect ratios while outperforming traditional approaches. Nevertheless, our suggested approach still possesses certain constraints. Once the size of the robot team has been modified, our proposed technique needs to undergo retraining to accommodate the new team size. Further investigation is required to determine how the trained policy network may be directly extended to varying-sized robot teams.