Inner–Outer Loop Intelligent Morphology Optimization and Pursuit–Evasion Control for Space Modular Robot

Abstract

This paper proposes an inner–outer loop computational framework to address the morphology optimization and pursuit–evasion control problem for space modular robots. First, a morphological design space considering the functional characteristics of different modules is designed. Then, an elite genetic algorithm is applied to evolve the morphology within this space, and a proximal policy optimization algorithm is applied to control the space modular robot with evolved morphology. Considering symmetry, centrality, module cost, and average cumulative reward, a comprehensive morphological assessment is proposed to evaluate the morphology. And the assessment result serves as the fitness of evolution. In addition, by implementing the algorithm on the JAX framework for parallel computing, the computational efficiency was significantly enhanced, allowing the entire optimization process within 17.3 h. Comparative simulation results verify the effectiveness and superiority of the proposed computational framework.

1. Introduction

Due to the ability of adaptive reconfiguration according to task environments, modular robots serve as an ideal platform for performing complex tasks. Presently, modular robots have garnered increasing attention for their potential prospects in environment exploration, object transportation, emergency rescue, and other fields [1,2,3,4]. Space pursuit–evasion problems have been extensively studied as fundamental problems in fields such as search and rescue [5,6,7]. Compared with conventional and monolithic robots, modular robots promise to be versatile and resilient in the face of complex tasks [8]. These attributes may potentially enhance its efficiency in performing space pursuit–evasion tasks. However, research specifically addressing the relevant application of modular robots for space pursuit–evasion tasks is still lacking. The modular robot design for complex tasks is also quite a challenging issue.

With the advancement of computational design, a variety of optimization methods have been applied to solve morphology optimization and control problems of modular robots [9,10]. In terms of morphology optimization, some studies employ genetic algorithms (GA), particle swarm optimization (PSO), Bayesian optimization (BO), and other optimization algorithms to optimize kinematics, topology, and locomotion of robots [11,12,13,14]. With the increasing complexity of modular robots and tasks, recent works in morphology optimization primarily focus on developing different morphological encoding schemes for various modular robots and tasks. In [15], the hierarchy of articulated three-dimensional rigid parts is directly encoded as the directed graph of nodes and connections. In [16], the direct encoding scheme is used to achieve the morphological evolution of the reconfigurable organisms in simulation and further applied to the design of the practical completely biological machines. A powerful indirect encoding scheme called CPPN-NEAT encoding is tested on various morphological evolution in [17]. The results show that this encoding scheme can create complex regularities such as symmetry and repetition. A graph grammar is proposed in [14] to express possible arrangements of physical robot assemblies. Then, the robot with components including different joint types, links, and wheels is encoded as a sequence of grammar rules indirectly. Overall, direct encodings ensure complete morphological design space, while it struggles to consistently produce regular morphologies and coordinated behaviors. Indirect encodings facilitate the emergence of regularity and variation in morphology, but they restrict the morphological design space [17]. Therefore, it is crucial to design specific encoding schemes according to the requirements of the tasks and the characteristics of the modular robots.

Control for robots with complex locomotor systems is a difficult task, as successful and efficient movement requires precise coordination of numerous parameters [18]. Reinforcement learning (RL) is currently the mainstream approach used to address the control issues of robots with complex morphologies, and it has also been increasingly integrated with large language models (LLMs) and deep neural networks (DNNs) to enhance their decision-making and adaptability [19,20]. In [21], a computational framework named Deep Evolutionary Reinforcement Learning (DERL) is proposed to evolve and control the robot to learn locomotion and manipulation tasks in complex environments with different terrains. In [22], the REINFORCE, soft actor-critic, and proximal policy optimization (PPO) algorithm are adopted to simultaneously optimize the discrete and continuous parameters of the morphology and controller, therefore achieving the efficient search for various robot designs. In particular, the PPO algorithm is applied in [23] to train and guide the pursuer to approach the evader intelligently. In addition to the PPO algorithm, a highly efficient co-optimization scheme that combines the control policy pre-training based on meta-reinforcement learning with the gradient-based optimization of variable hardware parameters to optimize robotic hand morphology was proposed in [24]. In [25], a multi-objective collaborative deep reinforcement learning algorithm based on deep deterministic policy gradient (DDPG) is proposed to address poor robustness and height limitations in bipedal robot jumping. Moreover, external disturbances and actuator failures are critical challenges in robotic control [26], and reinforcement learning has been employed to address these issues. Two novel high-accuracy trajectory tracking control policies for a quadrotor unmanned aerial vehicle (UAV) subject to external disturbances are developed in [27] utilizing reinforcement learning methodology, which can balance the control cost and control performance effectively. In [28], an optimized intelligent control scheme based on adaptive dynamic programming is proposed to control a UAV exposed to disturbances and actuator failures. However, most of these studies primarily focus on generating basic behaviors, such as swimming, crawling, and manipulation [21,22,29]. Simplification of control objectives shifts the focus of research towards optimizing the fundamental capability of modular robots while neglecting comprehensive task-oriented control. Some studies have begun to focus on controlling modular robots to perform complex tasks [30,31], but research in this area is still relatively limited.

For modular robots with vast morphological design spaces, achieving convergence in evolution and learning is challenging. Common strategies adopted by prior work include constraining the morphological design spaces or focusing on finding optimal parameters given a fixed hand-designed morphology [32,33,34]. Several other approaches utilize GPUs and parallel computing to accelerate the computational speed of algorithms, thereby supporting large-scale evolution and learning [35,36,37]. Among them, JAX is a framework designed for high-performance computing and machine learning, which significantly enhances the execution speed of code through just-in-time (jit) compilation [38,39].

Inspired by the above findings, an intelligent morphology optimization and pursuit–evasion control approach is proposed for space modular robots. The proposed approach evolves the morphology and trains the space modular robot efficiently, achieving the globally approximate optimal combination of the morphology and the strategy ultimately. The main contributions are summarized as follows:

(1)

Aninner–outer loop computational framework alternating between the evolution of morphology and the training of modular robots is built for space pursuit–evasion task for the first time. This framework is implemented on JAX to achieve efficient parallel computation, ultimately resulting in the globally approximate optimal combination of the morphology and the strategy for the space pursuit–evasion task.

(2)

In the outer loop, a morphological evolutionary algorithm based on the elite GA (EGA) is developed to optimize the morphology of the modular robots. Unlike most studies on modular robots that focus on modules with straightforward functions [21,22], this study investigates a modular robot with various practical module functions. Therefore, a morphological design space that considers the various module functional characteristics and a corresponding morphological encoding scheme is designed. Additionally, a comprehensive morphological assessment is proposed to guide morphological evolution and ensure the evolved morphology has good structural and control performance.

(3)

In the inner loop, a pursuit–evasion control approach based on the PPO algorithm is proposed for the space modular robots to pursue the free-floating and maneuvering evaders. By introducing the fuel consumption punishment into the reward function, the pursuit strategy is near fuel-optimal. The proposed approach possesses the superiority in balancing the pursuit performance and control cost.

2. Model and Problem Statement

2.1. Model of a Space Modular Robot

To investigate the morphology optimization and pursuit–evasion control for the space modular robot with different functional modules, we introduce a morphological design space $\mathbb{M}$, which contains physically realistic morphologies that can learn pursuit strategy in complex task environments. Each morphology $\mathcal{M}\in \mathbb{M}$ is the structure composed of various functional modules interconnected via rigid face-to-face connections, as shown in Figure 1. The module types include a control module (blue), auxiliary module (green), energy supply module (yellow), propulsion module (red), and basic connection module (grey). Establish the body-fixed frame $F_B\left\{O_B{X}_B{Y}_B{Z}_B\right\}$ with the origin $O_B$ being the center of mass of the space modular robot, as illustrated in Figure 1a. Subsequently, the method of one module connects to another can be represented as $\pm X_B$, $\pm Y_B$, $\pm Z_B$, where $+X_B$ represents connecting the one module to another in the $+X$ direction.

Due to the uniqueness of the propulsion module as the actuator driving the the locomotion of the space modular robot in three-dimensional space, it must be connected to at least one energy supply module and cannot be connected by any other module. Meanwhile, each energy supply module can only be connected by at most one propulsion module. Additionally, each morphology must include at least one control module and one auxiliary module, with no volume collisions between the modules. Here, in addition to incorporating the necessary conditions to ensure engineering practicality, we have not imposed any additional constraints on the morphological design space, thus achieving an extremely expressive morphological design space with minimal priors and restrictions.

A robot morphology $\mathcal{M}$ can be represented as a multiway tree $\mathcal{G}$ via direct encoding, as shown in Figure 1b. The modules of the morphology are regarded as the nodes of the multiway tree, and the connections between the two modules are regarded as the edges between the two nodes. The edges possess six weights ($\pm 1$, $\pm 2$, $\pm 3$), corresponding to six connection methods; the nodes possess five attributes (0,1,2,3,4), corresponding to five module functions. Determined the root node as control module, so the attribute of the root node is always 0. For a multiway tree $\mathcal{G}$, denote the sets of nodes, node attributes, edges, edge weights are $\mathcal{V}$, $\mathcal{F}$, $\mathcal{E}$, $\mathcal{D}$, respectively, it can be represented as $\mathcal{G}=\{\mathcal{V},\mathcal{F},\mathcal{E},\mathcal{D}\}$. Thus, the robot morphology $\mathcal{M}$ in Figure 1a can be encoded as

$$\begin{array}{c}\mathcal{G}=\{\mathcal{V},\mathcal{F},\mathcal{E},\mathcal{D}\}\hfill \\ \mathcal{V}=\{0,1,2,3,4,5,6,7,8,9,10\}\hfill \\ \mathcal{F}=\{0,4,2,4,1,2,3,3,1,2,3\}\hfill \\ \mathcal{E}=\left\{\right(0,1),(1,0),(0,2),(2,0),(0,3),\hfill \\ (3,0),(0,4),(4,0),(1,5),(5,1),\hfill \\ (5,6),(6,5),(2,7),(7,2),(4,8),\hfill \\ (8,4),(4,9),(9,4),(9,10),(10,9)\}\hfill \\ \mathcal{D}=\{1,-2,-1,2,3,-2,-1,1,-1,-3\}\hfill \end{array}$$

(1)

To model the dynamics of the space modular robot, the Earth-Centered-Inertial coordinate frame $F_I\left\{O_I{X}_I{Y}_I{Z}_I\right\}$ is established, as illustrated in Figure 2. Let ${\mathit{x}}_i={[{\mathit{r}}_i,{\dot{\mathit{r}}}_i]}^{\mathrm{T}}$ be the position vector, where ${\mathit{r}}_i\in {\mathbb{R}}^3$ is the position relative to $F_I$ and expressed in $F_I$. The subscript $i=p,e$ represents the pursuer (space modular robot) and the evader, respectively. Based on Newton’s gravitation law and Newton’s second law of motion, and considering the Earth as a homogeneous sphere, the equations of motion of the space modular robot and the evader in the reference frame $F_I$ are given by

$${\dot{\mathit{x}}}_i={\mathit{f}}_i\left({\mathit{x}}_i\right)+{\mathit{g}}_i{\mathit{u}}_i$$

(2)

where ${\mathit{u}}_i\in {\mathbb{R}}^3$ is the control thrust relative to $F_I$ and expressed in $F_I$. ${\mathit{f}}_i\left({\mathit{x}}_i\right)$ and ${\mathit{g}}_i$ are denoted as

$${\mathit{f}}_i\left({\mathit{x}}_i\right)=\left[\begin{array}{c}{\dot{\mathit{r}}}_i\\ -{\displaystyle \frac{\mu }{{\parallel {\mathit{r}}_i\parallel }^3}}{\mathit{r}}_i\end{array}\right],{\mathit{g}}_i=\left[\begin{array}{c}{\mathbf{0}}_{3\times 3}\\ {\displaystyle \frac{1}{m_i}}{\mathit{I}}_{3\times 3}\end{array}\right]$$

(3)

where $m_i$ is the mass; $\mu$ is the gravitational constant of the Earth. Neglecting the impact of fuel consumption on mass, the mass $m_p$ depends solely on the number of modules that compose the space modular robot, and it is modeled as

$$m_p=kn$$

(4)

where k is the mass of one module; n is the total number of modules. Additionally, the control thrust ${\mathit{u}}_p={[u_{px},u_{py},u_{pz}]}^{\mathrm{T}}$ is constrained by

$$\left\{\begin{array}{c}{u}_{px}\le T_{max}\hfill \\ u_{py}\le T_{max}\hfill \\ u_{pz}\le T_{max}\hfill \\ T_{max}=T_s·max\left(∥{\mathit{q}}^{+}∥,∥{\mathit{q}}^{-}∥\right)\hfill \end{array}\right.$$

(5)

where $T_{max}$ is the maximum thrust of the space modular robot; $T_s$ is the maximum thrust that a propulsion module can generate; ${\mathit{q}}^{+}={[b_{x1},b_{y1},b_{z1}]}^{\mathrm{T}}$ and ${\mathit{q}}^{-}={[b_{x2},b_{y2},b_{z2}]}^{\mathrm{T}}$, with $b_{x1}$, $b_{x2}$, $b_{y1}$, $b_{y2}$, $b_{z1}$, $b_{z2}$ are the quantities of propulsion modules which operative face arranged along $\pm X_B$, $\pm Y_B$, $\pm Z_B$, respectively (the operative face of propulsion module is opposite to its connection face). Meanwhile, the total thrust the space modular robot can generate is limited by

$$\int ∥{\mathit{u}}_p∥\le I_{sp}{m}_{\mathrm{fuel}}{g}_0$$

(6)

where $I_{sp}$ is the specific impulse of the fuel; $g_0$ is the standard gravitational acceleration. $m_{\mathrm{fuel}}$ is the mass of fuel carried by the space modular robot, which depends on the number of energy supply modules $n_{\mathrm{energy}}$, i.e.,

$$m_{\mathrm{fuel}}=m_f{n}_{\mathrm{energy}}$$

(7)

where $m_f$ is the mass of fuel carried by one energy supply module.

2.2. Problem Statement

Denote the comprehensive morphological assessment (Section 3.2) of the morphology $\mathcal{M}$ with the policy parameters $\mathit{\theta }$ as $MA(\mathcal{M},\mathit{\theta })$, and treat it as the objective of the morphology optimization. The morphology optimization problem can be formulated as

$$\begin{array}{cc}\hfill \underset{\mathcal{M},\mathit{\theta }}{max}& J=MA\left(\mathcal{M},\mathit{\theta }\right)\hfill \\ \hfill s.t.& \left\{\begin{array}{c}\mathcal{M}\in \mathbb{M}\hfill \\ \mathit{\theta }\in {\mathbb{R}}^{N_{\mathit{\theta }}}\hfill \end{array}\right.\hfill \end{array}$$

(8)

where J is the objective that needs to be maximized; $N^{\mathit{\theta }}$ is the number of the policy parameters.

In practical space pursuit–evasion tasks, the evader has a certain perception range, and the evader will initiate evasive maneuvers only when the pursuer enters the perception range. Therefore, the space pursuit–evasion tasks are typically divided into two phases based on whether the pursuer enters the perception range of the evader. This paper focuses on the stage within the perception range illustrated in Figure 2. The pursuer’s objective in the task is to approach the evader through maneuvering. In view of the distance between them, the criteria for completing the space pursuit–evasion task can be defined as

$$d_{\mathrm{safe}}<d<d_{\mathrm{work}}$$

(9)

where $d_{\mathrm{work}}$ is the minimum distance that should satisfy for completing the task; $d_{\mathrm{safe}}$ is the safe distance the pursuer needs to maintain from the evader; $d=∥{\mathit{r}}_p-{\mathit{r}}_e∥$ is the distance between the pursuer and the evader. The pursuer achieves position maneuver by executing impulsive thrust with fixed impulse interval p. The pursuit–evasion control problem can be stated as follows: For the pursuer with the specific morphology, design a pursuit strategy to approach the evader where the completion condition is described in Equation (9) while minimizing fuel consumption.

3. Inner–Outer Loop Intelligent Morphology Optimization and Pursuit–Evasion Control

In order to improve the intelligence and autonomy of the space modular robot during the pursuit–evasion task, an inner–outer loop computational framework is built to achieve intelligent morphology optimization and pursuit–evasion control for the space modular robot based on EGA [40] and PPO [41], as shown in Figure 3.

At the initialization of alternation between morphological evolution and training, the outer evolutionary loop generates an initial population of P individuals, each with a random morphology $\mathcal{M}\in \mathbb{M}$ composed of $n_{\mathrm{init}}$ modules. Then, the initial population undergoes lifetime learning in the inner reinforcement learning loop, and the result of the comprehensive morphological assessment determines fitness. After initialization, the entire inner–outer loop computation begins with individual selection. A total of G tournaments [42] are conducted to generate the elite population, with each involving the random grouping of 4 individuals. The crossover and mutation operations are then acted on by the elite population to produce elite offspring, which undergo lifetime learning and comprehensive morphological assessment to evaluate their fitness. Finally, the population is updated to the new one by incorporating the elite offspring and eliminating the individuals with the lowest fitness to maintain the population size at P. Thus, the inner and outer loops create a closed-loop system, where evolution and learning proceed in an alternating fashion. The following provides a detailed introduction to the core genetic mechanisms and the learning algorithm.

3.1. Crossover and Mutation

(1) Crossover

In Section 2, the robot morphology is represented as a multiway tree using the direct encoding method. Consider the crossover operation as a transformation $M_c:({\mathcal{G}}_{p1},{\mathcal{G}}_{p2})\to {\mathcal{G}}_c$, which is acted on two different trees ${\mathcal{G}}_{p1}$ and ${\mathcal{G}}_{p2}$ (parents) to produce a new tree ${\mathcal{G}}_c$ (child). In this process, two nodes are selected as the crossover points by a random number generator (RNG), and the child will be produced by crossing subtrees of parents according to the crossover points. Figure 4a provides a concrete example illustrating the concept. The red nodes represent the crossover points. Denote the probability of crossover occurrence is $r_{\mathrm{c}}$, i.e., the crossover rate. Each multiway tree in the population does not occur. Crossover is preserved as a child directly.

(2) Mutation

Mutation operations are acted on a child produced by crossover, including function mutation and connection method mutation. They have the same probability of occurrence and will not be acted simultaneously on a child. Define the probability of mutation occurrence as $r_{\mathrm{m}}$, i.e., the mutation rate.

Function mutation: Randomly select a module (excluding the control module) and randomly change its module function to another one. In specific implementations, a RNG is used to select a leaf node and a function number (the root node is determined as the control module, so the control module is excluded from the leaf nodes). Consider the function mutation operation as a transformation $M_{\mathrm{f}}:{\mathcal{G}}_c(V,E,f_0)\to {{\mathcal{G}}_c}^{\prime }(V,E,f_{\mathrm{mut}})$, the computation process can be represented as

$$fnc\_mu{t}_{v_i}=RNG(\{1,2,3,4\}\setminus \left\{fn{c}_{v_i}\right\})$$

(10)

where $fn{c}_{v_i}$ and $fnc\_mu{t}_{v_i}$ are the node attributes of the leaf node $v_i$ before and after mutation, $v_i=RNG(V\setminus \left\{v_{\mathrm{root}}\right\})$ is the selected leaf node; $v_{\mathrm{root}}$ is the root node of ${\mathcal{G}}_c$; function $RNG\left(\mathcal{S}\right)$ represents the random selection of an element from set $\mathcal{S}$ using RNG. Figure 4b provides a concrete example illustrating the concept.

Connection method mutation: Randomly select a module other than the control module, then randomly change its module connection method to another one. Consider the function mutation operation as a transformation $M_{\mathrm{d}}:{\mathcal{G}}_c(V,E,f_0)\to {{\mathcal{G}}_c}^{\prime }(V,E,f_{\mathrm{mut}})$, the computation process can be represented as

$$con\_mu{t}_{v_i}=RNG(\{\pm 1,\pm 2,\pm 3\}\setminus \left\{co{n}_{v_i}\right\})$$

(11)

where $co{n}_{v_i}$, $con\_mu{t}_{v_i}$ are the edge weights of the edge with $v_i$ as its child node before and after mutation. Figure 4c provides a concrete example illustrating the concept.

During the crossover and mutation, it is inevitable that some morphologies may fail to meet the necessary conditions in Section 2.1. Therefore, in the actual implementation, reasonable judgment is performed on each offspring generated by the crossover and mutation of two parents. If an offspring fails to meet the necessary conditions, the two parents will undergo crossover and mutation again until a valid offspring is generated.

3.2. Comprehensive Morphological Assessment

Considering symmetry, centrality, module cost, and average cumulative reward in the inner reinforcement learning loop, a comprehensive morphological assessment is proposed. The result of the assessment serves as the fitness of morphology, determining the retention and elimination of the morphology in the outer evolutionary loop. To ensure good pursuit–evasion control performance of the space modular robot, the evolved morphology should possess high levels of symmetry and centrality. The symmetry and centrality are defined in the form of

$$f_s\left(\mathcal{M}\right)=(x_{\mathrm{s}}+y_{\mathrm{s}}+z_{\mathrm{s}})/3n$$

(12)

$$f_c\left(\mathcal{M}\right)=-\sum _{i=1}^n\parallel {\mathit{X}}_i-{\mathit{X}}_c\parallel /3n$$

(13)

where $x_s$, $y_s$, $z_s$ represent the number of modules symmetric about $X_B$, $Y_B$, $Z_B$, respectively. ${\mathit{X}}_i$ represents the coordinate of module i relative to the body-fixed frame $F_B$, and ${\mathit{X}}_c$ represents the coordinate of morphology’s center of mass relative to the body-fixed frame $F_B$. In practical engineering missions, different modules have different usage costs, and the number of modules assigned to a robot is limited. Therefore, it is necessary to consider the module cost as an optimization metric to indirectly optimize the module configuration of the robot. The module cost is defined as

$$f_m\left(\mathcal{M}\right)=\sum _{i=0}^4{n}_i{c}_i$$

(14)

where $n_i$ and $c_i$ are the number and cost of different functional modules; the subscript $i=0,1,2,3,4$ represents the control module, auxiliary module, energy supply module, propulsion module, and basic connection module, respectively. Then, the comprehensive morphological assessment rule is constructed in the form of

$$MA(\mathcal{M},{\mathit{\theta }}_{\mathcal{M}})=w_1{f}_s+w_2{f}_c+w_3{f}_m+w_{4}F(\mathcal{M},{\mathit{\theta }}_{\mathcal{M}})$$

(15)

where $w_1$, $w_2$, $w_3$, $w_4$ are positive constant parameters; $F(\mathcal{M},{\mathit{\theta }}_{\mathcal{M}})$ denotes the average of the cumulative reward attained by the space modular robot over the last 100 episodes at the end of its lifetime learning. This metric reflects the learning ability and performance of the space modular robot with specific morphology in achieving the task objective.

3.3. Proximal Policy Optimization Algorithm

The PPO algorithm is applied as a reinforcement learning algorithm to achieve pursuit–evasion control for the space modular robot. In this paper, we use an architecture (Figure 5) where the action probability distribution ${\pi }_{\mathit{\theta }}\left({\mathit{a}}_t\right|{\mathit{s}}_t)$ and the estimation of state value ${\widehat{V}}_{{\pi }_{\mathit{\theta }}}\left({\mathit{s}}_t\right)$ are generated by one neural network whose parameter is denoted as $\mathit{\theta }$. The action ${\mathit{a}}_{{\pi }_{\mathit{\theta }}}\left({\mathit{s}}_t\right)\in {\mathbb{R}}^3$ is derived by sampling from ${\pi }_{\mathit{\theta }}\left({\mathit{a}}_t\right|{\mathit{s}}_t)$. Then, the action ${\mathit{a}}_{{\pi }_{\mathit{\theta }}}\left({\mathit{s}}_t\right)$ serves as the control signal of the controller, which results in

$${\mathit{u}}_{\mathit{p}}\left(t\right)=T_{\max }{\mathit{a}}_{{\pi }_{\mathit{\theta }}}\left({\mathit{s}}_t\right)$$

(16)

The index to be maximized for PPO is defined as [41]

$$L_t^{PPO}\left(\mathit{\theta }\right)=E_t[L_t^{CLIP}\left(\mathit{\theta }\right)-\alpha L_t^{VF}\left(\mathit{\theta }\right)+\beta S\left[{\pi }_{\mathit{\theta }}\right]\left({\mathit{s}}_t\right)]$$

(17)

where $\alpha$, $\beta$ are constant parameters; $L_t^{CLIP}$ is the clipped advantage function; $L_t^{VF}$ represents the estimation error of the state value function; $S\left[{\pi }_{\mathit{\theta }}\right]\left({\mathit{s}}_t\right)$ is the cross-entropy that reflects the stochastic degree of the policy. To improve computational efficiency and training stability, multiple identical task environments are created for parallel training. The space modular robot interacts with the task environments by applying the action $a_{{\pi }_{\mathit{\theta }}}\left({\mathit{s}}_t\right)$. Subsequently, the task environments transition to a new state ${\mathit{s}}_{t+1}$. The space modular robot and evader are deemed to be parts of the environment. During the environmental interaction process, the state $\mathit{s}$, reward r, action ${\mathit{a}}_{{\pi }_{\mathit{\theta }}}$, state value ${\widehat{V}}_{{\pi }_{\mathit{\theta }}}$, and action probability ${\pi }_{\mathit{\theta }}\left(\mathit{a}\right|\mathit{s})$ at each time step are stored in the replay buffer as a batch of data. When the data reaches the batch size, the data from the replay buffer is used for subsequent network parameter updates. The parameters of the neural network are updated by adaptive moment estimation (Adam).

In the implementation of the algorithm for the space pursuit–evasion task, the input of the network is the position error vector ${\mathit{r}}_p-{\mathit{r}}_e$, velocity of the pursuer ${\dot{\mathit{r}}}_p$, and velocity of the evader ${\dot{\mathit{r}}}_e$. The neural network has two hidden layers, each with 64 nodes. The activation function for all hidden layers is tanh. In the initialization process, orthogonal initialization is used for the weight of the neural network to ensure stability throughout the training process.

3.4. Reward Function of the Space Pursuit–Evasion Task

The reward function plays a crucial role in guiding the space modular robot to complete the pursuit–evasion task, i.e., it directs the robot state to satisfy the completion criteria (9). On this basis, considering the environmental boundary and fuel consumption, the reward function at each time step is defined as

$$r_t=r_{\mathrm{success}}-r_b+{\omega }_1\Delta d-{\omega }_2∥{\mathit{u}}_p∥$$

(18)

where ${\omega }_1$ and ${\omega }_2$ are constant coefficients, the third term is continuous rewards, and $\Delta d$ represents the distance between the pursuer and the evader in the previous time step minus this same quantity in the current time step. The fourth term is continuous punishment. This term enables the algorithm to continuously optimize fuel consumption during task performance while ensuring that the robot completes the task. As a result, the trained robot can accomplish the task with reduced fuel consumption, which holds practical engineering significance. The first and second terms are terminal reward and punishment, respectively, which are defined as

$$r_{\mathrm{success}}=\left\{\begin{array}{cc}100,\hfill & \mathrm{Equation}\left(9\right)\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(19)

$$r_b=\left\{\begin{array}{cc}150,\hfill & ∥{\mathit{r}}_p∥\ge b_{\max }\hfill \\ 150,\hfill & ∥{\mathit{r}}_p∥\le b_{\min }\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(20)

where $b_{\max }$ and $b_{\min }$ are two constants. The position of the robot must be within these boundaries while performing the task. Once the terminal reward or punishment is triggered, the current episode is immediately terminated. For the task environment discussed in this paper, establishing environmental boundaries is necessary. During the initial training phase, the space modular robot is prone to maneuver to excessively high or low positions that are impractical for task completion within the limited time and fuel. Thus, setting environmental boundaries allows for the early termination of episodes when the robot cannot complete the task, thereby enhancing the overall training efficiency.

3.5. Outline of the Inner–Outer Loop Computational Framework Implementation

The pseudocode for the inner–outer loop computational framework is shown in Algorithm 1.

Remark 1. 

The proposed inner–outer loop computational framework is inspired by DERL [21]. It considers various practical module functions and connection methods within the morphological encoding scheme, thereby enabling its application to a wider range of modular robot design scenarios compared to DERL. In the outer evolutionary loop, it employs EGA to evolve the morphology of the space modular robot, which helps preserve advantageous genes and significantly improves global exploration capability and convergence speed of the algorithm. Moreover, the proposed framework uses the result of the comprehensive morphological assessment as the fitness for morphological evolution. By taking engineering factors into account, it possesses greater practical engineering significance. Additionally, the proposed framework adopts reinforcement learning to control the space modular robot without the need for a dynamics model, which makes it more applicable to scenarios where accurate robot modeling is challenging. By comparison, certain graph-based morphology optimization frameworks adopt model-based control algorithms such as model predictive control (MPC), which require an accurate dynamics model of the modular robot [14,30].

Algorithm 1: Inner–Outer Loop Computational Framework

1: Initialize the population ${\mathcal{P}}_0=\{{\mathcal{G}}_0^1,{\mathcal{G}}_0^2,...,{\mathcal{G}}_0^p\}$ 2: for $i=1$ to p do 3:    Run an inner reinforcement learning for ${\mathcal{G}}_0^i$ 4:    $F_0^i\leftarrow$ The average cumulative reward of the reinforcement learning 5:    $f_{s,0}^i,f_{c,0}^i,f_{m,0}^i\leftarrow$ Right-hand side of (12)–(14) 6:    $M{A}_0^i\leftarrow$ Right-hand side of (15) 7: end for 8: Fitness ${\mathcal{F}}_0\leftarrow \{M{A}_0^1,M{A}_0^2,...,M{A}_0^p\}$ 9: for $j=1$ to G do 10:    ${\mathcal{E}}_j\leftarrow$ The elites $\{{{\mathcal{G}}_j^1}^{\prime },{{\mathcal{G}}_j^2}^{\prime },...,{{\mathcal{G}}_j^{groups}}^{\prime }\}$ selected from ${\mathcal{P}}_{j-1}$ using tournament algorithm 11:    ${\mathcal{O}}_j\leftarrow$ The offspring $\{{{\mathcal{G}}_j^1}^{\prime \prime },{{\mathcal{G}}_j^2}^{\prime \prime },...,{{\mathcal{G}}_j^{groups}}^{\prime \prime }\}$ generated by acting crossover and mutation on ${\mathcal{E}}_j$ 12:    for $k=1$ to $groups$ do 13:      Run an inner reinforcement learning for ${{\mathcal{G}}_j^k}^{\prime \prime }$ 14:      $F_j^k\leftarrow$ The average cumulative reward of the reinforcement learning 15:      $f_{s,j}^k,f_{c,j}^k,f_{m,j}^k\leftarrow$ Right-hand side of (12)–(14) 16:      $M{A}_j^k\leftarrow$ Right-hand side of (15) 17:    end for 18:    ${\mathcal{F}}_j\leftarrow {\mathcal{F}}_{j-1}+\{M{A}_j^1,M{A}_j^2,...,M{A}_j^{groups}\}$ 19:    ${\mathcal{P}}_j\leftarrow {\mathcal{P}}_{j-1}+{\mathcal{O}}_j$ 20:    for $l=1$ to $groups$ do 21:      Delete $\underset{\mathcal{G}\in {\mathcal{P}}_j}{argmin}MA\in {\mathcal{F}}_j$ from ${\mathcal{P}}_j$ 22:    end for 23: end for

4. Numerical Simulation

To verify the performance of the proposed inner–outer loop computational framework, comparative simulation experiments are implemented using different space pursuit–evasion task environments and optimization algorithms on a server equipped with two Intel Xeon Silver 4210R Processors (Intel, Santa Clara, CA, USA). The experimental framework is implemented on JAX to enable parallel computation on CPUs, which significantly accelerates the computation speed. Additionally, the implementation of PPO is based on PureJaxRL [43].

4.1. Configuration of Task and Algorithm

In the initialization phase, the positions of the evader are randomly initialized on a spherical surface centered at $O_I$ with a radius of R, while the space modular robot is randomly initialized around the evader, ensuring a distance of $d_p$ (perception radius) between them. The core parameters of the space pursuit–evasion task are listed in

Table 1. Parameters of the space pursuit–evasion task.

Parameter	Value
$m_e$	150 kg
k	10 kg
$T_s$	50 N
$I_{sp}$	260 s
$m_f$	0.47 kg
$d_{\mathrm{safe}}$	100 m
$d_{\mathrm{work}}$	1000 m
p	100 s
$d_p$	50,000 m
R	6,971,393 m
$T_{\mathrm{maxe}}$	100 N

. The core hyper-parameters of EGA and PPO are listed in

Table 2. Hyper-parameters of the algorithm.

Parameter	Value
Population size	16
Generations	50
Tournament groups	16
Crossover rate	0.8
Mutation rate	0.5
Total steps	800,000
Learning rate	0.001
Epochs	10
Batch size	2048

. To ensure that the task performance of the space modular robot is solely influenced by its morphology, offspring do not inherit the neural network parameters of their parents during the evolutionary process. Additionally, the total number of steps for all inner-loop reinforcement learning is kept the same.

4.2. Simulation in Different Space Pursuit–Evasion Task Environments

Considering the state of the evader as part of the space pursuit–evasion task environments, the environments can be classified into free-floating evader environment (FEEnv) and maneuvering evader environment (MEEnv) based on whether the evader adopts a maneuvering strategy. During the pursuit–evasion process, the free-floating evader does not perform any maneuvers, i.e., ${\mathit{u}}_{\mathit{e}}=\mathbf{0}$, while the maneuvering evader adopts the following evasion strategy

$${\mathit{u}}_e={\displaystyle \frac{{\mathit{r}}_e-{\mathit{r}}_p}{∥{\mathit{r}}_e-{\mathit{r}}_p∥}}{\mathit{T}}_{\mathrm{maxe}}$$

(21)

where ${\mathit{T}}_{\mathrm{maxe}}$ is the maximum thrust of the maneuvering evader.

The entire optimization and learning process in each environment takes 17.3 h, resulting in 816 morphologies along with their pursuit strategies. Figure 6 shows the optimal (with the highest fitness) individuals from generation 0 (initial population), 25, and 50 in the two environments. Due to the consideration of symmetry and centrality in the fitness function, the optimal morphology in the final population is more regular compared to the initial population. In particular, the optimal morphology in the final population under MEEnv exhibits pronounced symmetry and centrality. Additionally, due to the incorporation of module cost in the fitness function, the morphologies in both environments evolved towards a lighter configuration. They gradually removed redundant propulsion and auxiliary modules to reduce module costs while still being able to complete their tasks. Figure 7 illustrates the fitness changes in populations in both environments throughout the evolution. The fitness of the populations is stably convergent during the evolutionary process. The average fitness values of the initial populations in both environments are below zero, indicating that a significant number of individuals in the initial populations have poor morphologies and are unable to complete the task through fixed-step reinforcement learning. However, after 50 generations of evolution, low-fitness individuals in both environments are eliminated, and the overall fitness of the populations reaches a higher level. Furthermore, the overall fitness of the population in MEEnv is lower than that in FEEnv, suggesting that the complexity of the environment has a certain impact on the evolution of morphology. Within the same number of evolutionary generations, individuals who evolved in a relatively simpler environment tend to have higher fitness values.

To better illustrate the pursuit–evasion process, a free-floating object that is initialized 50 km from the evader in the direction of the evader to $O_I$ is introduced as a reference, transforming the absolute positions of the pursuer and the evader into relative positions with respect to this reference. Define the position of the reference as ${\mathit{r}}_{\mathrm{ref}}$, then the relative positions of pursuer and evader are represented as ${\mathit{r}}_p-{\mathit{r}}_{\mathrm{ref}}$ and ${\mathit{r}}_e-{\mathit{r}}_{\mathrm{ref}}$. Figure 8 illustrates the results of inner-loop reinforcement learning for the optimal morphologies in the initial population (Figure 6a,d) and the final population (Figure 6c,f) in two environments. As seen in Figure 8a, the pursuer with the optimal morphology in the initial population can approach the evader and complete the space pursuit–evasion task in FEEnv. However, its pursuit trajectory is quite unstable. This maneuver leads to excessive fuel consumption and is not practical for real-world engineering. Therefore, this morphology and its corresponding control strategy have significant room for optimization. Furthermore, as shown in Figure 8c, the pursuer with the optimal morphology in the initial population fails to complete the space pursuit–evasion task in MEEnv. Throughout the entire task period, the pursuer is unable to close the gap with the evader to the required distance. In contrast, as clearly shown in Figure 8b,d, through the optimization of the inner–outer loop, the pursuers with the optimal morphologies in the final populations can effectively complete the space pursuit–evasion task. The pursuers can approach the evader within the constraints of limited fuel and time while maintaining the stability of the pursuit process. This demonstrates the effective optimization of the proposed computational framework. Figure 9 illustrates that the cumulative reward of the two individuals exhibits an increasing trend throughout the reinforcement learning process. Additionally, as shown in Figure 10, due to the consideration of fuel consumption in the reward function, the fuel required by the two individuals to complete the task shows a decreasing trend.

4.3. Simulation in Different Optimization Frameworks

Furthermore, to highlight the superiority of the proposed inner–outer loop computational framework, the following two optimization frameworks are employed for comparison. (1) Using traditional genetic algorithm (GA) as the outer loop evolution algorithm and the inner-loop reinforcement learning algorithm remains unchanged (denoted as GA+PPO). During the evolutionary process, each individual in the population generates offspring directly through crossover and mutation operators. The hyper-parameters of traditional GA and mutation operators remain the same as those of the EGA. (2) Random search morphologies in the outer loop and the inner-loop reinforcement learning algorithm remain unchanged (denoted as Random+PPO). This framework explores the entire design space in a fully stochastic way by assigning a random morphology for each space modular robot. It means that the changes in morphology are random and no longer depend on fitness.

MEEnv is set as the task environment for the comparative experiment. Figure 11 shows the optimal individuals from generation 0 (initial population), 25, and 50 generated by different optimization frameworks. Compared to the proposed optimization framework, the “GA+PPO” framework guides the morphology toward simpler structures. However, the comprehensive morphological assessment results of the resulting optimal morphologies are lower than those of the proposed framework. In contrast to the first two frameworks, the “Random+PPO” optimization framework generates highly random morphologies without a clear evolutionary trend. Furthermore, the comprehensive morphological assessment results of its optimal morphologies are significantly lower than those of the other two frameworks. Figure 12 illustrates the fitness changes in populations generated by different optimization frameworks throughout the evolution process. The fitness of the populations optimized by the proposed framework and the “GA+PPO” framework converges stably during the evolution, whereas that optimized by the “Random+PPO” framework does not show any convergence. Both the average and highest fitness of the population optimized by the proposed algorithm exceed those of the other two frameworks. Overall, the proposed framework demonstrates a strong ability to optimize morphology within the vast morphological design space.

5. Conclusions

The intelligent morphology optimization and pursuit–evasion control approach for space modular robots is investigated in this paper, and an inner–outer loop computational framework based on an elite genetic algorithm and proximal policy optimization is proposed. In the proposed framework, morphological evolution and reinforcement learning alternately execute as the inner and outer loops to calculate the globally approximate optimal combination of the morphology and the pursuit strategy for the space pursuit–evasion task. Considering the various module functional characteristics, a morphological design space and a corresponding morphological encoding scheme are designed in the outer evolutionary loop. Meanwhile, a comprehensive morphological assessment is proposed to guide morphological evolution. In the inner reinforcement learning loop, a reward function considering the fuel consumption is designed to guide the space modular robot with specific morphology to learn near fuel-optimal pursuit strategy. Finally, the comparative simulation demonstrates that the proposed computational framework is effective in handling the pursuit–evasion task with varying maneuverability of evaders and possesses superior optimization and control performance with lower fuel consumption.

In the future, an intelligent evasion strategy will be considered for the evader, or a self-play approach will be used where the control strategies of both the pursuer and the evader are trained by the same reinforcement learning algorithm. Additionally, space obstacles and environmental disturbances will be incorporated to establish more realistic task environments with environmental uncertainties. This will improve the intelligence of the resulting morphology and control strategy. Moreover, Pareto-based multi-objective optimization algorithms (e.g., NSGA-II) will be adopted in the outer evolutionary loop to obtain the Pareto-optimal morphologies for specific tasks.