Generating Function Reallocation to Handle Contingencies in Human–Robot Teaming Missions: The Cases in Lunar Surface Transportation

Abstract

On lunar missions, efficient and safe transportation of human–robot systems is essential for the success of human exploration and scientific endeavors. Given the fact that transportation constructs bottlenecks for numerous typical lunar missions, it is appealing to investigate what function allocation strategies can generate optimal task implementation paths for robots with low-human workloads when the situation changes. Thus, this paper presents a novel approach to dynamic human–robot function allocation explicitly designed for team transportation in lunar missions. The proposed dynamic allocation framework aims to optimize human–robot collaboration by responding to existing and potential contingencies. First, a fitness concept model is designed to quantify the factors that motivate the functional adaptation of each agent in dynamic lunar mission scenarios. A hierarchical reinforcement learning (HRL) algorithm with two layers is then employed for decision-making and optimization of human–robot function allocation. Finally, the validity of the framework and algorithm proposed is validated by a series of human–robot function allocation experiments on a simulated environment that mimics lunar transportation scenarios, and is compared with the performance of other algorithms. In the future, path-planning algorithms can be incorporated into the proposed framework to improve the adaptability and efficiency of the human–robot function allocation in lunar missions.

1. Introduction

As the exploration of space continues to evolve, the Moon has emerged as a promising destination for scientific research and human settlement. In lunar missions, the transportation of personnel, equipment and resources plays a crucial role in facilitating exploration and enabling the successful execution of mission objectives. Advances in autonomous technologies have the potential to increase the ability to work on large-scale operations and minimize human risk through the use of autonomous rovers [1]. However, there is also the risk that if an autonomous system fails, the astronauts would have to spend valuable time rescuing the rover or taking over the tasks, which is considered a difficult failure to solve in an early stage lunar mission. The challenges and complexities associated with lunar transportation necessitate innovative solutions to optimize efficiency, ensure safety, and enhance mission productivity. One promising avenue for addressing these challenges lies in the collaboration between humans and robots [2]. By leveraging the complementary capabilities of both humans and robots, a dynamic human–robot function allocation system can be developed to optimize the allocation of transportation tasks, leading to increased efficiency and improved adaptability to the variability of complex environments. This is especially the case in the resource-intensive, technologically risky and environmentally unstructured world of lunar surface operations [3].

In multi-human–robot lunar surface operations (LSO), astronauts are able to drive lunar rovers. When not driven by an astronaut, some vehicles might also function independently as robots, either teleoperated from the cabin or operated with supervised autonomy [4]. However, as the complexity of the environment and the task size expand, the function allocation efficiency is significantly suppressed. Due to the specificity of lunar mission scenarios, few studies have been found to construct human–robot task allocation methods for dynamic lunar missions. Fortunately, Thomas et al. proposed a task assignment method for lunar missions involving multiple agents that matches capabilities with specific functions, enabling a suitable assignment plan based on task decomposition and helping to address the unpredictable and changing nature of the lunar environment [5]. Okubo et al. leveraged an action graph to enhance the allocation of tasks and path planning for robots in dynamic environments, leading to faster and more efficient performance [6]. However, the aforementioned studies do not focus on optimizing lunar transportation, and the dynamic function allocation is still based on the predefined functions of each agent, whose capabilities will vary in different task scenarios and during the process of task implementation.

Machine learning is extremely promising for solving dynamic task allocation [7]. A series of human–robot function allocation algorithms have been proposed, such as the market-based auction algorithm [8,9], particle swarm optimization algorithm [10,11], greedy algorithm [12], heuristic algorithm [13], queen bee-assisted genetic algorithm [14], and simulation-based learning techniques [15,16]. Liu et al. introduced a task allocation approach that takes into account both capacity and time, with the objective of enhancing the efficiency of task allocation and accommodating current tasks as they arise [17]. Lyu et al. have suggested an approach that takes into account transportation time and collision avoidance, aiming to reduce the number of robots required [18]. Chen et al. suggested an algorithm for task allocation that optimizes the time required to complete the task [19]. Zitouni et al. optimized the task allocation algorithm by conducting the maximum number of tasks [20]. Al-Hussaini et al. have suggested methods that take into account the effect of possible mission uncertainties on both the robot and the task, intending to generate task reallocation suggestions automatically [21]. Considering the heterogeneity of robots, Tai et al. studied the different states of the robot, categorizing them as normal, delayed, or capable of recovering from a delay [22]. Huang et al. analyzed the capabilities of various types of robots to assess the range of tasks they could perform [23]. Wang et al. combined the terrain and task adaptabilities of humans and heterogeneous robots to obtain a more accurate method of quantifying the search and rescue ability [24]. The above studies shed light on addressing the dynamic challenge of assigning tasks and planning paths in the presence of contingencies. But they ignore how environmental changes will bring about corresponding variations in agent capabilities. In addition, the proposed method cannot be efficient in the LSO context when the lunar scenario specificity of the lunar environment and terrain is not carefully considered, and the various influences of lunar transport are not accurately quantified.

To address the above issues, we propose an approach for dynamic human–robot function allocation in unstructured lunar environments. First, we develop a set of three fitness functions to measure the adaptability of agents in lunar surface operations, including distance, task, and environment fitness. The idea of measuring the level of collaboration between humans and robots has been introduced to determine how well they work together. Then, a proposal has been constructed for the allocation of functions between humans and robots using the HRL algorithm with two layers. The optimization objective in the first layer is to maximize the fitness of the task and environment, which aims to obtain the astronaut type and robot type, as well as the level of human–robot collaboration that would best fit the task requirements. For layer two, The definitive optimization objective is to increase the distance fitness so that each task can be allocated to a designated astronaut and robot. Finally, the transportation scenario of a sequence of lunar surface operations, encompassing ground exploration, failed robot rescue, material transport, and return of lunar surface samples, is used as a case scenario to validate how well the human–robot function allocation approach works. Based on the experimental findings, it has been demonstrated that the proposed task allocation algorithm is capable of significantly enhancing the efficiency of task allocation. In the face of complex LSO tasks, only some of the most common scenarios and task execution subjects are considered in this paper. In order to expand the application area of the proposed approach, it would be beneficial to consider additional types of subjects and tasks in future work. In addition, we plan to incorporate path planning algorithms into the proposed framework to enhance the adaptability and effectiveness of assigning human–robot functions during lunar missions.

As shown in Figure 1, the following sections outline the paper’s contents. In Section 2, we explain the procedure for the human–robot function allocation strategy, which consists of defining and computing the fitness and using the HRL algorithm. We state the experimental design in Section 3 and analyze the results in Section 4. Finally, we conclude the work of this paper and suggest potential avenues for future improvement in Section 5.

2. Methods

This section presents the dynamic human–robot function allocation model, including its problem context, algorithmic framework, function definition, and some specific parameter settings. This algorithm is presented in the context of coping with existing or potential contingencies in the unstructured environment of multi-agent lunar surface operations (LSO) tasks. A two-layer HRL algorithm is used to measure the level of adaptation of agents to tasks, environments, and distances using fitness-based metrics.

2.1. Formulation of the Problem

Figure 2 shows the typical transportation scenarios and agents of the LSO mission. The environment contains unpredictable terrain and uncertain lighting, where agents are required to transport in a variety of LSO tasks, such as ground exploration, robot rescue, material transport, and lunar surface samples. Different agents have different tasks and environment adaptabilities, including astronauts, a small exploration robot, large rescue robot, material transport robot, and lunar rover. There are two types of LSO mission executions, the first being primarily teleoperation, where the astronaut monitors or directly operates the robot in the capsule, and the other being directly performed by the astronaut, where the astronaut exits and drives the lunar rover on the lunar surface. There are three typical terrains of the lunar surface, including powdery, rocky, undulating and unknown. When allocating tasks between humans and robots, it is crucial to take into account the nature of the task, its location, and the flexibility of different agents to cope with changing environmental conditions with the ultimate goal of real-time adaptation of task assignment plans on demand.

2.2. Overview of the Algorithm for Dynamic Human–Robot Function Allocation

Figure 3 shows the flow of the dynamic human–robot function allocation for LSO task transportation. First, the stylized description of the transport tasks and agents of the lunar surface operations is made. LSO agents include astronauts and humanoid robots. Then, based on the stylized description and the unpredictable characteristics of the environment, the values of three fitness are calculated. Finally, the dynamic human–robot function allocation is implemented using a two-layer HRL algorithm, which is optimized with the Q-learning algorithm. Time steps are taken into account, and the results of the task allocation are iterated continuously by real-time computation. In contrast, the state of agents and tasks is influenced by the task allocation results. Thus, the HRL algorithm can output in real-time a task allocation method that best matches the input of the dynamically changing environment.

The LSO agents include astronauts and robots with distinct perceptions, predictions, and manipulation capabilities. The LSO agents are defined as

$$\left\{\begin{array}{c}{H}_i^1=\left\{p_{i1}^1,e_{i1}^1,t_{i1}^1\right\}\\ ⋮\\ H_i^H=\left\{p_{i1}^H,e_{i1}^H,t_{i1}^H\right\}\\ R_j^1=\left\{p_{j1}^1,e_{j1}^1,t_{j1}^1\right\}\\ ⋮\\ R_j^R=\left\{p_{j1}^R,e_{j1}^R,t_{j1}^R\right\}\end{array}\right.\hspace{1em}\hspace{1em}i\in \left[1,I^{*}\right],j\in \left[1,J^{*}\right]$$

(1)

In Equation (1), $H_i$ and $R_j$ refer to the specific features of the astronaut and lunar robot, respectively. The variables $p_{*}^{*}$, $e_{*}^{*}$, moreover, $t_{*}^{*}$ indicate the position, environment types, and transportation related to LSO tasks for each astronaut and robot. Additionally, $I^{*}$ and $J$ represent the entire number of astronauts and robots, which belong to $*th$ kind.

The tasks are defined as

$$T_m=\left\{p_m^t,d_m^t,t_m^t\right\},m\in \left[1,M\right]$$

(2)

The total number of tasks is represented by $M$. The position, environment, and task specifications of the $m$th task are referred to as $p_m^t$, $d_m^t$ and $t_m^t$, respectively.

2.3. Definition and Calculation of the Fitness

Task fitness, environment fitness, and distance fitness are functions that quantify the adaptability of the agent to the LSO mission, the lunar surface environment, and the transport distance.

2.3.1. Task Fitness

Task fitness is calculated as follows:

$$TF\left(a_{i,j},t_m\right)=W_{i,j}\times L_m$$

(3)

The capabilities of the $i$th astronaut and $j$th lunar robot to perceive, make decisions, and operate are denoted by $W_{i,j}$, while the abilities required for the mth task are represented by $L_m$.

There is a sequence of LSO missions that encompasses ground exploration, robot rescue, material transport, and lunar surface samples. In the ground exploration task, the agent is supposed to collect environmental data and plan a moving path while avoiding obstacles. In robot rescue tasks, agents locate, inspect, repair, and move malfunctioning robots. In material transport tasks, agents are supposed to lift, move, and lower material. In the lunar surface sample task, the agent is supposed to plan the cut trajectory and perform the cut operation.

In

Table 1. The agents’ ability in LSO tasks.

		Astronaut in the Capsule	Astronaut Outside the Capsule	Small Exploration Robot	Large Recue Robot	Material Transport Robot	Lunar Rover
Ground Exploration	Terrain Detection	$\surd$	$\times$	$\surd$	$\times$	$\times$	$\times$
	Path Planning	$\surd$	$\times$	$\surd$	$\times$	$\times$	$\surd$
	Obstacle Avoidance	$\times$	$\surd$	$\surd$	$\surd$	$\surd$	$\surd$
Robot Rescue	Positioning Robot	$\surd$	$\times$	$\surd$	$\surd$	$\times$	$\surd$
	Trouble Shooting	$\surd$	$\surd$	$\times$	$\surd$	$\times$	$\times$
	Moving Robot	$\times$	$\times$	$\times$	$\surd$	$\surd$	$\surd$
Material Transportation	Lifting Material	$\surd$	$\surd$	$\times$	$\times$	$\surd$	$\times$
	Moving Material	$\times$	$\times$	$\times$	$\times$	$\surd$	$\surd$
	Lowering Material	$\surd$	$\surd$	$\times$	$\times$	$\surd$	$\times$
Lunar surface samples	Trajectory Planning	$\surd$	$\surd$	$\times$	$\times$	$\times$	$\surd$
	Cutting Operation	$\times$	$\surd$	$\times$	$\times$	$\times$	$\surd$

, the capabilities of agents for different LSO tasks can be found. The symbol “$\surd$” represents that the agent possesses the corresponding ability, while the symbol “$\times$” represents that the agent lacks the corresponding ability. The LSO tasks are rigorously categorized to provide a clear understanding of the fitness calculations for the tasks. The robots involved are small exploration robots, large rescue robots, material transport robots and lunar rovers. The crew consists of astronauts inside the capsule and outside.

The normalization of task fitness is calculated as

$$TF\left(a_{i,j},t_m\right)=\frac{TF\left(a_{i,j},t_m\right)}{\underset{k}{\max }TF\left(a_{i,j},t_m\right)}$$

(4)

To quantify the task fitness, a normalizing computation with specific rules is used. The task fitness has a range of 0.0 to 1.0. A score of 0 indicates that the agent is not sufficiently adaptive to the task, while a score of 1 indicates that the agent is sufficiently adaptive.

Figure 4 shows the results of the task fitness calculations for different agents. It can be found that the task fitness is low for all four task types, and collaboration between astronauts and robots could significantly improve task fitness. For ground exploration tasks, any combination of astronaut–robot collaborations has a higher task fitness than a single robot. For the robot rescue task, the highest fitness is calculated by combining the astronauts in the capsule and the large rescue robot. For material transport tasks, small exploration robots and large rescue robots have the lowest task fitness. In contrast, the task fitness of other robot–astronaut combinations is relatively high. Only the lunar rover or astronaut–rover collaboration has a relatively high fitness for lunar surface sample tasks.

2.3.2. Environment Fitness

The concept of environment fitness refers to the ability of the robot to adapt to the lunar environment.

$$EF\left(ro,te,mp,lt\right)=P\left(ro,te,mp,lt\right)\left[S\left(ro,te,mp,lt\right)+V\left(ro,te,mp,lt\right)\right]$$

(5)

$EF\left(ro,te,mp,lt\right)$ represents the terrain fitness of the robot $ro$ in the environmrnt with the type $te$ of terrain, with the map $mp$ and light $mp$. Where $te$ represents the type of topography, it may be one of four types, including powdery, rocky, undulating, and unknown. $mp$ represents the degree of unpredictability of the map of the task region, which may be of one of two types, including the case where only the coordinates are known but no detailed map is, and the case where the detailed map with the coordinates is known. $lt$ is the intensity of the light $lt$, divided into two types, dark and bright.

The passability, stationarity, and speed of the robot $ro$ in the environmrnt with the type $te$ of terrain are denoted by $P\left(ro,te,mp,lt\right)$, $S\left(ro,te,mp,lt\right)$, and $V\left(ro,te,mp,lt\right)$, respectively. It is worth noting that $EF\left(ro,te,mp,lt\right)$ is calculated using (5) only if $te$ is known. The parameters are defined and computed as follows.

The passability $P\left(ro,te,mp,lt\right)$ is defined as 1.0 when the robot $ro$ is able to pass through the terrain $te$ with the map $mp$ and light $mp$ successfully. $P\left(ro,te,mp,lt\right)$ is defined as 0.0 when the robot $ro$ is not able to pass through $te$ at all.

The stationarity $S\left(ro,te,mp,lt\right)$ is calculated through the Sperling equation, which represents the stability when the robot is moving.

$$S\left(ro,te,mp,lt\right)=2.7\sqrt[10]{a^3{f}^{5}Fc\left(f\right)}=0.896\sqrt[10]{\frac{j^3}{f}Fc\left(f\right)}$$

(6)

where $a$ is the amplitude of robot moving. $j$ is the acceleration of amplitude, $f$ is the frequency of vibration, and $Fc\left(f\right)$ is the correction factor of frequency.

$$Fc\left(f\right)=\left\{\begin{array}{ll}0.8f^2& f=0.5~5.4 \mathrm{Hz}\\ \frac{650}{f^2}& f=5.4~26 \mathrm{Hz}\\ 1& f>26 \mathrm{Hz}\end{array}\right.$$

(7)

The velocity $V\left(ro,te,mp,lt\right)$ is defined as the maximum speed that each robot can move on specific terrain.

If the type of terrain is unknown, then considering the types of terrain that might occur are known, which include powdery, rocky, and undulating. If all these types are adaptable (could be passed successfully) for the combination of astronaut and robot, $EF\left(ro,te,mp,lt\right)$ is defined as 1.0. Otherwise, $EF\left(ro,te,mp,lt\right)$ is defined as 0.0.

$$EF\left(ro,te,mp,lt\right)=\frac{EF\left(ro,te,mp,lt\right)}{\underset{k}{\max }EF\left(ro,te,mp,lt\right)}$$

(8)

The environment fitness is normalized in the same way as terrain fitness. The environment fitness range takes values from 0.0 to 1.0. $ro$ is considered not able to adapt to the lunar environment at all when $EF\left(ro,te,mp,lt\right)$ is 0.0, and $ro$ is considered able to adapt to the environment fully when $EF\left(ro,te,mp,lt\right)$ is 1.0.

To make the statistical graphics more concise, the 13 environment types are numbered using numbers 1–13, the correspondence of which is shown in

Table 2. Numbering of the 13 environmental types.

ID	1	2	3	4	5
Environment Type	unknown terrain	powdery, no-coordinates, dark	powdery, no-coordinates, bright	powdery, coordinates, dark	powdery, coordinates, bright
ID	6	7	8	9	10
Environment Type	rocky, no-coordinates, dark	rocky, no-coordinates, bright	rocky, coordinates, dark	rocky, coordinates, bright	undulating, no-coordinates, dark
ID	11	12	13
Environment Type	undulating, no-coordinates, bright	undulating, coordinates, dark	undulating, coordinates, bright

.

Figure 5 shows the environment fitness of different robots in various lunar environments. It can be found that only the small exploration robot has a higher environment fitness in unknown terrain. The environment fitness of the different robots is higher in powdery terrain than in rocky and undulating terrain, where the robots tend to be more adaptable in bright situations than in dark situations, and tend to pass faster and steadier in environments where coordinates are known than in environments where coordinates are unknown. Compared to the other three types of robots, the environment fitness of the small exploration robot is more consistently high across different types of environments.

2.3.3. Distance Fitness

The “distance fitness” means the level of effectiveness that changes depending on how far the person is from the task at hand. It is calculated using the following formula:

$${\mathit{DF}}_i\left(a_{i,j},t_m\right)=\min \left(DH\left(i,m\right),DR\left(j,m\right)\right)$$

(9)

The adaptabilities of distance from the $i$th astronaut and $j$th lunar robot to the $m$th mission are represented by $DH\left(i,m\right)$ and $DR\left(j,m\right)$. Agents that have higher fitness values are deemed more suitable for the task. When the astronauts in the capsule are selected, the distance fitness is calculated using the robot’s moving distance, and the degree of human–robot collaboration is designed to describe how closely astronauts and robots collaborate. When selecting an astronaut outside the capsule, the distance fitness is calculated using the larger moving distance between the astronaut and the robot. The detailed calculation of the mobility distance fitness is as follows:

$$\left\{\begin{array}{l} D{H}_{in}\left(i_{in},m\right)=\frac{HRCd\left(i_{in},m\right)}{\max HRCd\left(i_{in},m\right)}\hspace{1em}\hspace{1em}{i}_{in}ϵ\left[1,I_{in}\right]\\ D{H}_{out}\left(i_{out},m\right)=1-\frac{D{H}_{out}\left(i_{out},m\right)}{\max \left(D{H}_{out}\left(1,m\right),\cdots ,D{H}_{out}\left(I_{out},m\right)\right)}\\ DR\left(j,m\right)=1-\frac{DR\left(j,m\right)}{\max \left(DR\left(1,m\right),\cdots ,DR\left(J,m\right)\right)}\end{array}\right.$$

(10)

The range of HRCd for human–robot collaboration is taken from 0.0 to 1.0. The lower the HRCd, the higher the autonomy of the robot. When HRCd is 0.0, the robot is considered capable of performing the task entirely on its own, without the operation of the astronaut who is supposed to do nothing but confirm the results of the operation. When HRCd is 1.0, the robot is not considered capable of performing the task on its own at all, and the astronaut should teleoperate with the robot for the entire mission.

To calculate the distance traveled by an astronaut or a robot outside of the capsule during a mission, we use the straight line distance between two locations on the unstructured lunar surface, as detailed maps are not available. The straight line distance between the agent and the mission location is used to calculate the distance traveled by an astronaut or robot outside the capsule to the mission.

In order to account for the increase in distance around a barrier that is not accounted for by the calculation in a straight line, the dynamic HRL algorithm is used in this study. This algorithm is able to continuously compute the task distance in real-time, accounting for changes in time and the movement of the robot’s position, allowing for simultaneous running, computation, and planning. Additional details on the algorithm can be found in the following subsection.

2.4. Hierarchical Reinforcement Learning Algorithm

Reinforcement learning is considered as an efficient approach to solve the optimal allocation problem [25]. By learning to operate on different levels of temporal abstraction, hierarchical reinforcement learning (HRL) can solve the reinforcement learning expansion problem, and considering its advantages in data richness complexity, efficient data utilization and learning performance, this paper uses the HRL algorithm with two layers for solving the optimal human–robot functional assignment problem [26,27].

It is important to note that the position and state of the agent and task will change as the agent performs the task. To address this issue, the proposed method re-allocates tasks in real-time at each point in time with the goal of updating the task allocation scheme to adapt to shifts in the agent and task states. The initial value of time is set to 0 before the human–robot function allocation starts, and the initial state of the agent and the task is fed into the HRL algorithm. After the human–robot function allocation begins, the algorithm will collect the input information at fixed intervals, which includes the status of agents (whether agents are performing a task or not), the position of agents, and the status of tasks. Then, the algorithm could calculate the combination of the astronaut, robot, and the HRCd between them (if the astronaut is in the capsule, the HRCd should be considered), which has the highest fitness, based on the input information of the current point in time. At the same time, as the task is executed, the state of the agent and the task changes at the next point in time, and the updated input computes a fresh task allocation scheme to suit the current situation. As time changes, the input and output of the algorithm are continuously updated, forming a dynamic loop for the entire HRL algorithm, which may assist astronauts in supervising the transport of LSO tasks.

The first layer of HRL builds M Q-tables, one-to-one correspondence with tasks, and aims to compute the agent that best matches each mission. The HRL algorithm’s second layer takes the first layer’s output as its input and builds another Q-table using the Q-learning algorithm, with the goal of computing the agent that best matches all tasks. The HRL algorithm optimizes the first layer based on task and environment fitness. The optimization of the second layer is based on distance fitness.

2.4.1. HRL Algorithm-Layer One

To ensure efficient and effective mission execution, the first layer of HRL involves computing the most appropriate crew and robot based on mission and environment requirements. This is achieved by using a Q-learning algorithm with the goal of maximizing the task and environment fitness for each task. To calculate the sum of the task and environment fitness, the function of reward $R_m^1(s^1,a^1)$ is proposed.

$$R_m^1=\left(s^1,a^1\right)=\left[\begin{array}{cc}\alpha & \beta \end{array}\right]\left[\begin{array}{c}TF\left(a_{i,j},t_m\right)\\ EF\left(a_{i,j},t_m\right)\end{array}\right]$$

(11)

The task fitness of agent $a^1$ for the $m$th task is denoted by $TF\left(a_{i,j},t_m\right)$, while $EF\left(a_{i,j},t_m\right)$ represents the environment fitness of agent $a^1$ for the $m$th task. The total number of tasks is $M$, and $a^1$ refers to the combination of the ith type of $i$th type of astronaut and the $j$th type of robot for human–robot collaboration. Coefficients $\alpha$ and $\beta$ have also been defined.

$$a^1=h^i\cup r^j,iϵ\left[1,I\right],jϵ\left[1,J\right]$$

(12)

The variables $I$ and $J$ represent the entire number of astronauts and lunar robots, respectively.

$$s^1=\left[\begin{array}{cc}H& R\end{array}\right]$$

(13)

The variable $s^1$ denotes the state, comprising of both the astronaut ($H$) and lunar robot ($R$). The agent in this context is described as:

$$a^1=\left\{\begin{array}{ll}{h}^i& \hspace{1em}if{s}^1=H\\ r^j& \hspace{1em}if{s}^1=R\end{array}\right.$$

(14)

In Q-learning, the time difference method is used to update the formulation. The formula for updating is provided as follows:

$$Q^1\left(s^1,a^1\right)=Q^1\left(s^1,a^1\right)+\epsilon ∆Q^1\left(s^1,a^1\right)$$

(15)

The Q value of agent $a^1$ in state $s^1$ is represented by $Q^1\left(s^1,a^1\right)$. This value is updated by adding the increment to the total value and multiplying it by the learning rate ϵ.

$$∆Q^1\left(s^1,a^1\right)=R_n^1\left(s^1,a^1\right)+\delta \underset{A^1}{\max }{Q}^1\left({\left(s^1\right)}^{\prime },A^1\right)-Q^1\left(s^1,a^1\right)$$

(16)

To update the state ($s^1)$, we select the highest value of $Q^1$ for the next state ${\left(s^1\right)}^{\prime }$, and then multiply it by the decay factor 𝛿. This is then added to the present reward and subtracted from the present value of Q.

2.4.2. HRL Algorithm-Layer Two

In the first layer, the types of astronauts and robots are calculated, which are suitable for specific LSO tasks. This is used as input to the second layer, which aims to compute the best match between the astronaut and the robot for each task by maximizing the distance fitness. We designed a reward function that calculates the overall fitness score for all tasks by taking into account their distances fitness and its weights.

$$R^2\left(s^2,a^2\right)=\sum _{m\in M}DF\left(a^2,t_m\right)$$

(17)

When referring to the fitness distance of agent $a^2$ in task $t_m$, we define $DF\left(a^2,t_m\right)$ as distance fitness. Additionally, the state of the second layer is denoted as $s^2$.

$$s^2=\left[t_1,\cdots ,t_m\right]$$

(18)

The agents could be any possible combination of astronaut and robot.

$$a^2=\left\{\begin{array}{c}\left[h_1^1,\cdots ,h_{I1}^1\right]\cup \left[r_1^1,\cdots ,r_{J1}^1\right],\hspace{1em}{s}^2=t_1\\ \cdots \\ \left[h_1^m,\cdots ,h_{Im}^m\right]\cup \left[r_1^m,\cdots ,r_{Jm}^m\right],\hspace{1em}{s}^2=t_m\end{array}\right.$$

(19)

In the $m$th task, $h_i^m$ and $r_j^m$ represent the selected $i$th astronaut and $j$th lunar robot. The second layer is updated by the same formula as the first layer and is as follows:

$$Q^2\left(s^2,a^2\right)=Q^2\left(s^2,a^2\right)+\epsilon ∆Q^2\left(s^2,a^2\right)$$

(20)

$$∆Q^2\left(s^2,a^2\right)=R_n^2\left(s^2,a^2\right)+\delta \underset{A^2}{\mathit{max}}{Q}^2\left({\left(s^2\right)}^{\prime },A^2\right)-Q^2\left(s^2,a^2\right)$$

(21)

In this formula, $\epsilon$ and $\delta$ refer to the learning rate and value of attenuation, respectively, of the second layer.

3. Experiments

There are two comparison experiments designed as follows:

In the first experiment, HRL and RL, DRL, and HDRL algorithms are used for human–human function allocation with the same number of agents, respectively. In the second experiment, four types of algorithms are used for task allocation, each with a varying number of agents. The efficiency of task allocation is the metric used for comparison, and the experiments are designed as follows.

3.1. Experiment 1

The parameters of the HRL algorithm are designed as follows:

The coordinates of agents and tasks are designed to arrange in the task area randomly. The learning rate is 0.4, and the attenuation value is 0.6 in the HRL algorithm. It is worth noting that the coordinates of each agent change with their movement. In Figure 5, the numbers of optimization iterations for the HRL-based first and second layers are shown.

To demonstrate the proposed approach’s effectiveness, the RL algorithm is chosen as a representative of the traditional human–robot function allocation algorithm, and the same computation is performed with the same number of agents to compare the task assignment efficiency of the two algorithms.

The reward function of the RL algorithm is as follows:

$$R_m\left(s,a\right)=\left[\begin{array}{ccc}\alpha & \beta & \theta \end{array}\right]\left[\begin{array}{c}TF\left(a,t_m\right)\\ EF\left(a,t_m\right)\\ DF\left(a,t_m\right)\end{array}\right]$$

(22)

In the Q-learning algorithm, the state representing the astronaut and the robot is denoted by $s$, while the agent representing the human–robot collaborative combination is denoted by $a$. There are three coefficients defined as $\alpha$, $\beta$, and $\theta$.

$$\left\{\begin{array}{c}{S}^1=\left[H,R\right]\\ A=h_m^i\cup r_n^j\end{array},\hspace{1em}iϵ\left[0,I\right],jϵ\left[0,J\right]\right.$$

(23)

In the above equation, the variables $I$ and $J$ denote the total count of astronauts and robots, respectively. The variable $h_m^i$ represents the $m$th astronaut belonging to the $i$th type, whereas $r_n^j$ denotes the $n$th lunar robot belonging to the $j$th type. Here, is the formula for updating:

$$Q\left(s,a\right)=Q\left(s,a\right)+\delta ∆Q\left(s,a\right)$$

(24)

$$Q\left(s,a\right)=R\left(s,a\right)+\theta \underset{A}{\max }Q\left({\left(s\right)}^{\prime },a\right)-Q\left(s,a\right)$$

(25)

In this scenario, $s$ and ${\left(s\right)}^{\prime }$ denote the current state and the following state, respectively. $a$ represents the agent involved, while $Q\left(s,a\right)$ is referred to as the $Q$ value. Meanwhile, $\delta$ and $\theta$ stand for the learning rate and attenuation value of RL, respectively.

The DRL and HDRL algorithms are constructed by combining RL and HRL algorithms with a three-layer neural network with the number of neurons in the three layers being 10, 15 and 10, respectively.

Table 3. Experimental setting of multi-agent human–robot function allocation.

Item	Type	Number
Astronaut	Astronauts in the Capsule	4
	Astronauts outside the Capsule	4
Robot	Small Exploration Robot	4
	Large Rescue Robot	4
	Material transport Robot	4
	Lunar Rover	4
Task	Ground Exploration	1
	Robot Rescue	1
	Material transport	1
	Lunar surface samples	1

shows the setting of the agents and tasks in experiment 1. The experiment involves a total of 24 agents, comprising of four astronauts inside the capsule ($H_1^{in}$–$H_4^{in}$), four astronauts outside the capsule ($H_1^{out}–H_4^{out}$), four small exploration robots (${Re}_1–{Re}_4$), four large rescue robots (${Rr}_1–{Rr}_4$), four material transport robots (${Rt}_1$–${Rt}_4$), and four lunar rovers (${Rl}_1–{Rl}_4$). There are four types of tasks set up in the experiment, which include ground exploration (point E), robot rescue (point R), material transport (point T), and lunar surface samples (point S).

The proposed HRL and RL, DRL, and HDRL algorithms are used to compute the optimal human–robot function allocation scheme for the number of agents and tasks set in Table 3, respectively, and their task allocation efficiency is compared by the number of iterations of the algorithms.

3.2. Experiment 2

In Experiment 2, different numbers of agents are set. The detailed designs of the four types of algorithms is the same as in Experiment 1.

Table 4. Experimental setting of experiment 2.

Item	Type	Number
Astronaut	Astronauts in the Capsule	3	4	5	6	7
	Astronauts outside the Capsule	3	4	5	6	7
Robot	Small Exploration Robot	3	4	5	6	7
	Large Rescue Robot	3	4	5	6	7
	Material transport Robot	3	4	5	6	7
	Lunar Rover	3	4	5	6	7
Task	Ground Exploration	1	1	1	1	1
	Robot Rescue	1	1	1	1	1
	Material transport	1	1	1	1	1
	Lunar surface samples	1	1	1	1	1

shows the design of the number of agents in Experiment 2. The number of each agent (astronaut and robot) was set as 3, 4, 5, 6, and 7, respectively. The robustness of the proposed method is verified through comparing the magnitude of the shift in the number of iterations of HRL and the other algorithms when computing the optimal human–robot combination with different numbers of agents.

4. Results

This section shows the experimental results of the comparison experiments designed in the previous section, including the task assignment efficiency of each algorithm with different numbers of agents, and the computational efficiency of each algorithm with the same number of agents. The algorithm types include HRL (used in the model of this paper), RL, DRL, and HDRL. The effectiveness and superiority of the proposed human–robot allocation method is validated by two comparison experiments.

4.1. Comparison with Other Algorithms under Different Agent Numbers

Figure 6 shows the number of iterations based on the proposed HRL algorithm. Two lines are shown in the figure: the orange one shows the second layer’s optimization and the blue line shows the first layer’s optimization. It is evident that the first layer of the algorithm converges around 100 iterations and the second layer converges around 220 iterations. Eventually, the fitness function stabilizes around 1.0.

In Figure 7, one can observe the iteration curves of RL and DRL for the same computational case. Remarkably, the reward function of RL converges to approximately 1.0 after around 12,500 iterations, while the reward function of DRL converges to approximately 1.0 after around 18,000 iterations. Both algorithms have the same final stationary value of the fitness function as HRL. Nonetheless, their ergodic counts are significantly higher than those of HRL.

In Figure 8, there are two iteration curves representing the HDRL algorithm for the same computational situation. In particular, the blue one represents the optimisation curve for the first layer and the orange one for the second layer. Of interest is that the reward functions of the two layers of HDRL converge to approximately 1.0 after 580 and 780 iterations, respectively. Although the fitness function converges to the same value, the number of iterations is still higher for this algorithm than for HRL.

To sum up, the proposed HRL improves the efficiency of human–robot function allocation by no less than 98.24%, 98.78% and 71.79%, respectively, compared to the RL, DRL and HDRL algorithms for the same number of agents. It can be demonstrated that the proposed approach could effectively improve the efficiency of task allocation compared to traditional algorithms.

4.2. Comparison with Other Algorithms under the Same Agent Number

Figure 9 presents the comparison of the efficiency of four types of algorithms for human–robot function allocation as the number of agents varies. When comparing the iteration number between HRL and the alternative algorithms for different numbers of agents, we consistently find that the iteration number for HRL is always significantly less than the others. Furthermore, the iteration number of HRL is able to remain relatively stable to a higher degree as the number of agents increases, compared to alternative algorithms.

When the number of agents of each type is 3, 4, 5, 6, and 7, the number of iterations of HRL is 930, 1040, 1250, 1650, and 1960. Meanwhile, the number of iterations for RL is 6000, 10, 16, 22, and 31, respectively. The number of iterations for DRL is 12,000, 20,000, 28, 38, and 52,000, respectively. The number of iterations for HDRL is 5000, 9000, 12, 16, and 25, respectively.

The proposed HRL is at least 90.49% more computationally efficient than regular RL for different numbers of agents by at least 94.89%, compared to DRL; this is at least 88.26% better than the HDRL algorithm. It can be shown that the proposed approach is in general highly robust compared to other traditional algorithms.

In summary, a series of simulation experiments and calculations have been performed in randomly varying and complex environments with agents planning while marching, and with varying numbers of agents to verify that the proposed method is effective and robust. Combining the results of experiments 1 and 2, it appears that the proposed method not only efficiently increases the computational efficiency compared to regular RL, DRL and HDRL algorithms, but also remains primarily highly robust while the number of agents changes.

5. Conclusion

In this paper, an approach for allocating human–robot functions dynamically in LSO missions using the hierarchical reinforcement learning algorithm has been proposed. First, functions that quantify an agent’s ability in unstructured environments, including task, environment, and distance fitness, are designed. Next, an HRL algorithm is designed to accurately compute the most optimal human–robot function allocation scheme in real-time, which consists of two layers that take into account the mobility of the agents and the unpredictability of the lunar surface. Finally, we validate the effectiveness of the proposed approach by comparing different algorithms with the same or different number of agents. The main conclusions of the paper are summarized below.

• Compared to RL, DRL, and HDRL algorithms, the proposed method improves the task allocation efficiency by approximately 98.24%, 98.78%, and 71.79%, respectively, for the same number of agents;
• When the number of agents is varied, the proposed method improves the task allocation efficiency by about 90.49%, 94.89% and 88.26% compared to RL, DRL and HDRL algorithms, respectively. It can demonstrate the better robustness of the proposed approach when the number of agents varies.

In future work, the path planning algorithm will be integrated based on the parameter optimization of the task assignment method, with the aim of enhancing the adaptability and efficiency of the human–robot function allocation during the lunar mission.