Deep Reinforcement Learning for the Agile Earth Observation Satellite Scheduling Problem

Abstract

The agile earth observation satellite scheduling problem (AEOSSP) is a combinatorial optimization problem with time-dependent constraints. Recently, many construction heuristics and meta-heuristics have been proposed; however, existing methods cannot balance the requirements of efficiency and timeliness. In this paper, we propose a graph attention network-based decision neural network (GDNN) to solve the AEOSSP. Specifically, we first represent the task and time-dependent attitude transition constraints by a graph. We then describe the problem as a Markov decision process and perform feature engineering. On this basis, we design a GDNN to guide the construction of the solution sequence and train it with proximal policy optimization (PPO). Experimental results show that the proposed method outperforms construction heuristics at scheduling profit by at least 45%. The proposed method can also calculate the approximate profits of the state-of-the-art method with an error of less than 7% and reduce scheduling time markedly. Finally, we demonstrate the scalability of the proposed method.

1. Introduction

Agile earth observation satellites (AEOSs) are a new generation of earth observation satellites (EOSs) with three degrees of freedom: roll, pitch, and yaw. With an extensive observation range, long observation time, and no terrain limitations, AEOSs play an important role in weather forecasting, disaster warning, environment protection, ground mapping, and maritime search and rescue. Compared with traditional EOS that only has roll capability, AEOS has a longer visible time window (VTW) for ground target observation. The observation window (OW) represents the real observation time of the task, whose length is the observation duration requested by the user. The OW is variable and can be any period within the VTW that guarantees the integrity of the observation process, which makes the solution space of the AEOSSP large. When observing two targets in succession, the AEOS must transit attitude, and because AEOS attitude is related to the start and end times of OW, the attitude transition time between the two tasks is variable and time-dependent. The agile earth observation satellite scheduling problem (AEOSSP) requires the determination of the task observation sequence and the OW of each task to satisfy observation integrity, attitude transformation constraints, and some other hard constraints of satellites, such as memory and power consumption. Therefore, the AEOSSP is a typical combinatorial optimization problem with complex constraints that has been shown to be an NP-hard problem [1].

With the expansion of AEOS application fields, observation requests become frequent, and observation requirements become diverse. However, with better observation capabilities, AEOSs are still a scarce resource that cannot satisfy the high demand for observations. In addition, some emergencies, such as earthquakes and floods, require satellites to complete observations as soon as possible. Therefore, a fast and efficient scheduling algorithm is essential to improve the utilization rate of satellites.

In recent decades, many scholars have studied the AEOSSP. Lemaître et al. [2] were the first to research the AEOSSP, and described the AEOSSP as a combinatorial optimization problem considering the selection and scheduling of observation tasks. Due to the complexity of the problem, few exact methods have been proposed. Wang et al. [3] proposed a mixed-integer programming model for the AEOSSP and reduced problem complexity by discretizing the continuous observation angle into three angles. They obtained the approximate upper bound of the problem by CPLEX. Chu et al. [4] designed an implicit enumeration algorithm to construct a solution under the framework of depth-first search and designed three pruning strategies. For the AEOSSP, the exact methods all simplify the time-dependent constraint. In addition, ref [5] showed that solutions cannot be obtained in an acceptable time with the CPLEX solver when the number of VTWs exceeds 27. Currently, research on combinatorial optimization problems primarily focuses on heuristics and meta-heuristics. Heuristic and meta-heuristic algorithms can solve large-scale problems and are widely used in practice. A well-designed algorithm can significantly improve efficiency [6,7,8]. For AEOSSP, Lemaître et al. [2] proposed four heuristics: a greedy algorithm, dynamic programming, a constraint planning algorithm, and a local search algorithm. There are also several profit-based construction heuristics [3,9] and the iterative local search [10]. Many meta-heuristics include the tabu search algorithm [11,12], the hybrid differential evolution algorithm [13], the improved genetic algorithm [14,15,16,17], and the adaptive large neighbourhood search algorithm [5,18]. However, the search difficulty and solution time of these algorithms increase dramatically as the problem scale increases. Traditional heuristic and meta-heuristic algorithms cannot meet the requirements of high efficiency and fast response in practical applications. In addition, these rule-based algorithms rely heavily on the designer’s experience, and the solution’s quality is poor. Therefore, traditional methods are limited by their solution characteristics and cannot produce high-quality results timely.

In recent years, deep reinforcement learning (DRL) has been applied to many classical combinatorial optimization problems, such as the travelling salesman problem (TSP) and the vehicle routing problem (VRP). Vinyals et al. [19] first proposed using pointer networks (PN) to solve the TSP, and their model refers to the traditional sequence-to-sequence (Seq2Seq) structure. Bello et al. [20] then used the policy gradient and actor–critic algorithms to train the PN model, which can obtain approximate optimal solutions for TSP with 100 tasks. Nazari et al. [21] proposed an end-to-end framework based on DRL, which divides the problem input into static input and dynamic input to solve the VRP with dynamic characteristics. Joshi et al. [22] proposed a graph pointer network (GPN) to solve the TSP. Additionally, DRL methods have been used in several classical real-world problems [23,24].

The literature highlights the strong potential of DRL to solve combinatorial optimization problems, and some scholars have performed related studies. Chen et al. [25] proposed an end-to-end framework based on DRL for the AEOSSP as first attempt to apply DRL to AEOSSP. Zhao et al. [26] proposed a two-phase neural combination optimization method with reinforcement learning and used a neural combination optimization with the reinforcement learning method to determine the observation sequence and a reinforcement learning algorithm based on a deep deterministic policy gradient to determine the start time of the tasks. Wei et al. [27] proposed a deep reinforcement learning and parameter transfer-based approach (RLPT) to solve a multiobjective AEOSSP. All these methods simplify the time-dependent attitude transition time constraint, but this constraint is one of the important constraints of the AEOSSP. To better represent this constraint, we propose to model the AEOSSP as a graph, with edges representing the pose transition time. On this basis, we propose to use GNN to solve the problem.

In this paper, to solve the AEOSSP, we present a graph-based DRL method that is different from existing methods. The primary contributions of this study are as follows:

(1) We model the AEOSSP with the time-dependent attitude transition time as a graph, which can more accurately represent tasks and their relationships through nodes and edges.

(2) Based on the graph model of the AEOSSP, we design its MDP solution process and propose a graph attention network (GAT)-based decision neural network (GDNN) to represent the policy, which is trained by an RL method.

(3) We design extensive experiments to demonstrate the effectiveness and timeliness of the proposed method by comparing it with specific competitors. In addition, we perform a model study to verify the structure and generalization of GDNN.

The remainder of this paper is organized as follows. Section 2 models the graph formulation of the AEOSSP, the attitude transition time constraint, and the reformulation of the AEOSSP. Section 3 describes the proposed method, including the GDNN and the training method. Section 4 provides the computational experiments and analysis. Finally, Section 5 concludes the article and presents suggestions for future research.

2. Problem Description

2.1. Parameter

The parameters used in this paper are summarized in

Table 1. Parameter description.

Parameter	Description
$n_{tsk}$	The number of tasks
$i,j$	The index of tasks, $i,j=0,1,...,n_{tsk}$. 0 means a virtual task
$pr{i}_i$	The profit of $tas{k}_i$, $pr{i}_0=0$
${\theta }_{i,t}$	The roll angle of the satellite to $tas{k}_i$ at time t
${\phi }_{i,t}$	The pitch angle of the satellite to $tas{k}_i$ at time t
${\psi }_{i,t}$	The yaw angle of the satellite to $tas{k}_i$ at time t
$w{b}_i$	The VTW start time for $tas{k}_i$, $w{b}_0=0$
$w{e}_i$	The VTW end time for $tas{k}_i$, $w{e}_0=0$
$t{b}_i$	The OW start time of $tas{k}_i$, $t{b}_0=0$ is the initial state time
$t{e}_i$	The OW end time of $tas{k}_i$, $t{e}_0=0$ is the initial state time
$c{t}_i$	The observation duration of $tas{k}_i$, $c{t}_0=0$
$b{a}_i$	The satellite attitude at $t{b}_i$, which is determined by the angle of roll ${\theta }_{i,t{b}_i}$, pitch ${\phi }_{i,t{b}_i}$ and yaw ${\psi }_{i,t{b}_i}$. $b{a}_0$ is the initial attitude of the satellite
$e{a}_i$	The satellite attitude at $t{e}_i$, which is determined by the angle of roll ${\theta }_{i,t{e}_i}$, pitch ${\phi }_{i,t{e}_i}$ and yaw ${\psi }_{i,t{e}_i}$. $e{a}_0$ is the initial attitude of the satellite
${\rho }_{ij}$	The attitude transition angle between $tas{k}_i$ and $tas{k}_j$
$\mathrm{trans}\left(e{a}_i,b{a}_j\right)$	The attitude transition time between $tas{k}_i$ and $tas{k}_j$
$x_{ij}$	Binary decision variable, indicating whether $tas{k}_i$ is a former task for $tas{k}_j$

.

2.2. Mathematical Formulation

$$max\sum _{i=0}^{n_{tsk}}\sum _{j=0}^{n_{tsk}}{x}_{ij}{pri}_j$$

(1)

$$w{b}_i\le t{b}_i\le t{e}_i\le w{e}_i,\forall i\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(2)

$$t{b}_i+c{t}_i=t{e}_i,\forall i\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(3)

$$t{e}_i+trans\left(e{a}_i,b{a}_j\right)-t{b}_j\le 0,\forall x_{ij}=1$$

(4)

$$\sum _{i=0}^{n_{tsk}}\sum _{j=0}^{n_{tsk}}{x}_{ij}\left(c{t}_j·uie+ute·trans\left(e{a}_i,b{a}_j\right)\right)\le (1-\zeta )·Eg{y}_{max}$$

(5)

$$\sum _{i=0}^{n_{\mathrm{tsk}}}{x}_{ij}\le 1,\forall j\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(6)

$$\sum _{j=0}^{n_{tsk}}{x}_{ij}\le 1,\forall i\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(7)

$$x_{ii}=0,\forall i\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(8)

$$x_{ij}\in \left\{0,1\right\},\forall i,j\in \left\{0,1,2,\dots ,n_{tsk}\right\}$$

(9)

Equation (1) is the optimization objective function of the AEOSSP, which is to maximize the sum of completed task priorities; Equation (2) represents the time window constraint of the satellite, which means the task must be observed within the VTW of the satellite; Equation (3) represents the relationship between the start, end time, and duration of the task; Equation (4) represents the time-dependent transformation time constraint between the tasks; Equation (5) represents the the consumed power constraint of the satellite; Equation (6) indicates that there exists at most one former task for each task; Equation (7) indicates that there exists at most one latter task for each task; Equation (8) indicates that a task can neither be its former task nor its latter task; Equation (9) represents the value of the decision variable.

2.3. Time-Attitude Adjacency Graph of the AEOSSP

We model AEOSSP as an adjacency graph, which is a typical class of directed acyclic graphs. When the satellite passes directly over the target (i.e., the overhead moment), the pitch angle of the satellite is 0. We introduce time-attitude coordinates to represent the VTW of the task. As shown in Figure 1, the timeline of satellite operation is the x-axis, and the roll angle of the satellite is the y-axis. A task is a node in the graph whose coordinate $(t_i^{\mathrm{side}},{\theta }_i^{\mathrm{side}})$ consists of the satellite overhead moment and the roll angle. Each node has seven attributes: $pr{i}_i$, $w{b}_i$, $w{e}_i$, $t{b}_i$, $t{e}_i$, $b{a}_i$, and $e{a}_i$. The edge weight $w_{ij}=\mathrm{trans}\left(e{a}_i,b{a}_j\right)$ between $nod{e}_i$ and $nod{e}_j$ indicates the transition time between $tas{k}_i$ and $tas{k}_j$. The optimization objective is to find a path that begins at virtual node 0 and satisfies all constraints to maximize the sum of the profits of all nodes on the path.

2.4. Reformulation of AEOSSP

The solution construction can be seen as a sequential decision process. Each task decision can be considered to be a stage. As Figure 2 shows, in each stage, we can determine the next task node based on the current graph state according to the policy, which is represented using the decision neural network in the proposed method. Once the node of one stage is determined, the graph of the current state is updated. The next task node is selected according to the current state. Then, the process is repeated until a scheduling solution is constructed.

We model this construction process as a Markov decision process (MDP) defined by the 5-tuple $〈S,A,T,R,C〉$, where:

• S is the state set of the time-attitude adjacency graph model;
• A is the set of actions that a satellite can perform (i.e., the candidate task set);
• $T:S\times A\to S$ is the state transition function;
• $R:S\times A\to {\mathbf{R}}^{+}$ is the reward function, which represents the profit of the selected task;
• $C:S\times A\to \left\{0,1\right\}$ is the set of constraints, including constraints (2), (4) and (5). When $C\left(s,a\right)=0$, $T\left(s,a\right)=\perp$, which means that the constraints are not satisfied and the state transition is infeasible.

According to the Bellman equation, under the optimal policy ${\pi }^{\ast }$, the optimal value function satisfies:

$$V^{\ast }\left(s\right)=\underset{a\in A}{max}\left(R\left(s,a\right)+V\left(T\left(s,a\right)\right)\right)\forall s\in Ss.t.T\left(s,a\right)\ne \perp$$

(10)

The corresponding optimal strategy ${\pi }^{\ast }$ is:

$${\pi }^{\ast }\left(s\right)=\underset{a\in A}{\arg \max }\left(R\left(s,a\right)+V\left(T\left(s,a\right)\right)\right)\forall s\in Ss.t.T\left(s,a\right)\ne \perp$$

(11)

2.5. Attitude Transition Time Constraint

For each target, the satellite observation attitude is determined. When observing two consecutive targets, a specific attitude transition time is required and can be calculated using Equations (12) and (13), where $b_0=11.66$, $b_1=5$, $b_2=10$, $b_3=16$, $b_4=22$, $a_1=1.5$, $a_2=2$, $a_3=2.5$, $a_4=3$, $z_0=10$, $z_1=30$, $z_2=60$, and $z_3=90$. The length of the transition time depends on the ending attitude of the former task and the starting attitude of the latter task, both of which are time-dependent.

$$\mathrm{trans}\left(e{a}_i,b{a}_j\right)=\left\{\begin{array}{c}\begin{array}{cc}\hfill b_0,& {\rho }_{ij}\le z_0\hfill \\ \hfill b_1+{\rho }_{ij}/\phantom{{\rho }_{ij}{a}_1}{a}_1,& z_0<{\rho }_{ij}\le z_1\hfill \\ \hfill b_2+{\rho }_{ij}/\phantom{{\rho }_{ij}{a}_2}{a}_2,& z_1<{\rho }_{ij}\le z_2\hfill \\ \hfill b_3+{\rho }_{ij}/\phantom{{\rho }_{ij}{a}_3}{a}_3,& z_2<{\rho }_{ij}\le z_3\hfill \\ \hfill b_4+{\rho }_{ij}/\phantom{{\rho }_{ij}{a}_4}{a}_4,& {\rho }_{ij}>z_3\hfill \end{array}\hfill \end{array}\right.,\forall x_{ij}=1$$

(12)

$${\rho }_{ij}=\left|{\theta }_{i,t{e}_i}-{\theta }_{j,t{b}_j}\right|+\left|{\phi }_{i,t{e}_i}-{\phi }_{j,t{b}_j}\right|+\left|{\psi }_{i,t{e}_i}-{\psi }_{j,t{b}_j}\right|,\forall x_{ij}=1$$

(13)

To analyze the characteristics of this constraint, we propose a method to determine the earliest observation start time for the next task based on the current task. First, we define a time delay function using the same method as in [28].

Definition 1.

Time delay function: For consecutive $tas{k}_i$ and $tas{k}_j$, the time delay function under the time-dependent attitude transition time constraint $(t{e}_i,t{b}_j,$ $\mathit{trans}\left(e{a}_i,b{a}_j\right))$ can be defined as Equation (14).

$$\mathrm{tidy}\left(t{e}_i,t{b}_j\right)=t{e}_i+\mathrm{trans}\left(e{a}_i,b{a}_j\right)-t{b}_j$$

(14)

The satellite completes the observation of the former task at $t{e}_i$ and then begins the attitude transition. After $\mathrm{trans}\left(e{a}_i,b{a}_j\right)$, the satellite ends the transition and waits until $t{b}_j$ for the latter task observation. When $\mathrm{tidy}\left(t{e}_i,t{b}_j\right)<0$, the satellite has finished the attitude transition before observation, which satisfies the shortest attitude transition time. When $\mathrm{tidy}\left(t{e}_i,t{b}_j\right)>0$, the transition time is insufficient, and the constraint is violated. When $\mathrm{tidy}\left(t{e}_i,t{b}_j\right)=0$, the transition time is just sufficient.

Pralet et al. [28] proved that for agile satellites, the delay function $\mathrm{tidy}\left(t{e}_i,t{b}_j\right)$ monotonically increases with $t{e}_i$ and decreases with $t{b}_j$. This property shows that determining the earliest observation start time for the latter task can help compress the attitude transition time between two tasks and increase the availability of OWs. We designed the EarliestImageCal algorithm to obtain the earliest observation start time of the latter task. The core idea is to calculate it by the linear approximation iterative method. The pseudo-code of the algorithm is shown in Algorithm 1.

The EarliestImageCal algorithm divides the solution into three situations: (1) when the start time of the latter task VTW meets the attitude transition constraint, the task OW is satisfied with the constraint, as shown in Figure 3a; (2) when the end time of the latter task VTW does not meet the attitude transition constraint, the task OW is not satisfied with the constraint, as shown in Figure 3b; (3) when the start time of the latter task VTW does not meet but the end time meets, we use the linear approximation method to replace the delay function, as shown in Figure 3c.

Algorithm 1 EarliestImageCal

Require:  the end time of current task $t{e}_i$, the VTW of the latter task $\left[w{b}_j,w{e}_j\right]$, the maximum number of iterations $NumIter$, calculation time accuracy $prc$ Ensure:  the earliest start time of the latter task $t_m$   1: $h_1=\mathrm{tidy}\left(t{e}_i,w{b}_j\right)$   2: if $h_1\le 0$ then   3:  return  $w{b}_j$ //Observing at the earliest visible time of latter tasks   4: end if   5: $h_2=\mathrm{tidy}\left(t{e}_i,w{e}_j\right)$   6: if $h_1>0$ then   7:  return  $+\infty$ //The attitude transition cannot be completed in the entire window   8: end if   9: for $j=1$ to $NumIter$ do 10:    $t_m=\left(h_2·w{b}_j-h_1·w{e}_j\right)/\phantom{\left(h_2·w{b}_j-h_1·w{e}_j\right)\left(h_2-h_1\right)}\left(h_2-h_1\right)$ 11:    $h_m=\mathrm{tidy}\left(t{e}_i,t_m\right)$ 12:    if $\left|h_m\right|<prc$ then 13:   return $t_m$ 14:    end if 15:    if $h_m>0$ then 16:   $w{b}_j=t_m$ 17:   $h_1=h_m$ 18:    else 19:   $w{e}_j=t_m$ 20:   $h_2=h_m$ 21:    end if 22: end for 23: return  $w{e}_j$

3. Methodology

3.1. GDNN Decision-Making Process

As the solution construction process shows in Figure 4, we first update the features in the current state as input to the GDNN. The network interacts the input features and uses mask mechanism [29] to avoid infeasible tasks. The network outputs the probability of the tasks and selects the tasks. Then, the process will be repeated until the candidate task set is empty. Finally, we obtain the output solution.

3.2. Feature Engineering

Appropriate feature extraction is the foundation of network decisions. We describe the AEOSSP as the time-attitude adjacency graph in which node attributes and edge weights are equally important. Therefore, the features of the AEOSSP comprise ten node features and five edge features, as shown in Figure 5. The following parts describe the meaning of each feature. All features must be normalized to improve network generalization ability and avoid weakening or failure of the network decision-making effect caused by the difference in data distribution.

Node features can be divided into task, VTW, and status features. The profit $pr{i}_i$ and the observation duration $c{t}_i$ are task features proposed by the users. The former indicates the importance of $tas{k}_i$, and the latter indicates the shortest time required for completing the task observation. The VTW features represent the VTW of the task, including the overhead time $t_i^{\mathrm{side}}$, the overhead roll angle ${\theta }_i^{\mathrm{side}}$, the start time $w{b}_i$ and end time $w{e}_i$ of the VTW, the earliest start time $t_i^m$, and its corresponding roll angle ${\theta }_i^m$ of $tas{k}_i$. The status features $l_i^{\mathrm{wait}}$ and $l_i^{\mathrm{last}}$ are updated after each decision. $l_i^{\mathrm{wait}}$ indicates whether $tas{k}_i$ is the candidate task in the current state. When $tas{k}_i$ is among the candidate tasks, $l_i^{\mathrm{wait}}=1$. Otherwise, $l_i^{\mathrm{wait}}=0$. $l_i^{\mathrm{last}}$ indicates whether $tas{k}_i$ is the last task in the current solution sequence. If $tas{k}_i$ is the last task, $l_i^{\mathrm{last}}=1$. Otherwise, $l_i^{\mathrm{last}}=0$. After normalization, we obtain the node features ${\stackrel{\rightharpoonup }{v}}_i=\left({\widehat{pri}}_i,{\hat{t}}_i^{\mathrm{side}},{\hat{\theta }}_i^{\mathrm{side}},{\widehat{ct}}_i,{\widehat{wb}}_i,{\widehat{we}}_i,{\hat{t}}_i^m,{\hat{\theta }}_i^m,l_i^{\mathrm{wait}},l_i^{\mathrm{last}}\right)\in {\mathbf{R}}^{10}$ and the node feature vector $\mathit{v}$.

The matrix of edge features $\mathit{E}$ indicates the edge features between two nodes, where ${\stackrel{\rightharpoonup }{e}}_{ij}=\left({\hat{d}}_{ij},l_{ij}^{\mathrm{n}1},l_{ij}^{\mathrm{n}5},l_{ij}^{\mathrm{n}10},l_{ij}^{\mathrm{n}20}\right)\in {\mathbf{R}}^5$. $d_{ij}$ indicates the distance between two nodes. In time-attitude coordinates, the distance between two nodes indicates the satellite attitude transition angle between two tasks, which can be calculated as Equation (13). In practice, the satellite attitude transition angle is primarily determined by roll and pitch angles. The pitch angle of the satellite is time-dependent and related to the length of the VTW. Therefore, we use the overhead time $t_i^{\mathrm{side}}$ to represent the pitch angle, thus linking time to the attitude transition angle, as shown in Equation (15).

$$d_{ij}=\left|t_i^{\mathrm{side}}-t_j^{\mathrm{side}}\right|\ast 2{\phi }_{max}/\phantom{2{\phi }_{max}t{w}_{max}}t{w}_{max}+\left|{\theta }_i^{\mathrm{side}}-{\theta }_j^{\mathrm{side}}\right|$$

(15)

where $t{w}_{max}$ is the length of the longest VTW and ${\phi }_{max}$ is the maximum pitch angle of the satellite. In addition, $l_{ij}^{\mathrm{n}1}$, $l_{ij}^{\mathrm{n}5}$, $l_{ij}^{\mathrm{n}10}$ and $l_{ij}^{\mathrm{n}20}$ are features used to represent the relationship of two nodes. For $nod{e}_i$, we sort all $d_{ij}$ in ascending order. If $d_{ij}$ is ranked within the first Kth, then $l_i^{\mathrm{nK}}=1$. Otherwise, $l_i^{\mathrm{nK}}=0$.

3.3. GNDD Structure

The graph attention network (GAT) is a graph neural network structure proposed by Veličković [30]. The network introduces an attention mechanism into the graph neural network structure and can weigh the relationships between graph nodes. By extracting problem features, the GAT can calculate the probability of the following actions based on the features of the current state.

In the proposed method, we design the GAT-based decision neural network (GDNN) for problem sequence decision-making. As shown in Figure 6, the GDNN consists of nine layers.

The first four layers are embedding layers, and each embedding layer is a single-layer GAT network using the attention mechanism to weigh the node and edge features. The following five layers are all fully connected layers, which are only responsible for updating the attributes of features. The fifth layer is the middle layer, which converts the network dimensions. The sixth to eighth layers are hidden layers whose dimensions remain the same. The last layer is the output layer and outputs a one-dimensional action probability. The feature update of the entire network is independent of the graph structure.

In the proposed method, the node feature is ${\stackrel{\rightharpoonup }{v}}_i\in {\mathbf{R}}^{10}$ and the edge feature is ${\stackrel{\rightharpoonup }{e}}_{ij}\in {\mathbf{R}}^5$. To avoid complicating the network, the intermediate network structure has the same dimension, which is unified as $F_3$. The transfer process of the extracted features in the network layer is as follows, where l is the network layer identifier, $l\in \left[1,9\right]\wedge l\in {\mathrm{N}}^{+}$.

(1) Embedded layer and the transfer network ($l\in \left[1,4\right]$).

The node feature vector $\mathit{v}$ is transferred in the embedding layer network through Equations (16)–(18), where the LeakyReLU function is proposed in [31]. The condition shown in Equation (19) is satisfied, and the ReLU function [32] is used to activate between layers. The edge feature matrix $\mathit{E}$ is transferred in the embedding layer network though the Equation (20), and the conditions shown in Equations (21) and (22) are satisfied.

$${\stackrel{\rightharpoonup }{z}}_{ij}^{\left(l\right)}={\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}\left[{\stackrel{\rightharpoonup }{v}}_i^{\left(l\right)}∥{\stackrel{\rightharpoonup }{e}}_{ij}^{\left(l\right)}∥{\stackrel{\rightharpoonup }{v}}_j^{\left(l\right)}\right]$$

(16)

$${\alpha }_{ij}^{\left(l\right)}=\frac{exp\left(\mathrm{LeakyReLU}\left({{\stackrel{\rightharpoonup }{a}}^{\left(l\right)}}^T{\stackrel{\rightharpoonup }{z}}_{ij}^{\left(l\right)}\right)\right)}{{\sum }_{k\in {\mathrm{N}}_i}exp\left(\mathrm{LeakyReLU}\left({\stackrel{\rightharpoonup }{z}}_{ik}^{\left(l\right)}\right)\right)},{\stackrel{\rightharpoonup }{a}}^{\left(l\right)}\in {\mathbf{R}}^{F_3}$$

(17)

$${\stackrel{\rightharpoonup }{v}}_i^{(l+1)}=\mathrm{ReLU}\left(\sum _{j\in {\mathrm{N}}_i}{\alpha }_{ij}^{\left(l\right)}{\stackrel{\rightharpoonup }{z}}_{ij}^{\left(l\right)}\right)$$

(18)

$${\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}\in \left\{\begin{array}{c}\begin{array}{cc}\hfill {\mathbf{R}}^{F_3\times \left(2F_1+F_2\right)},& l=1\hfill \\ \hfill {\mathbf{R}}^{F_3\times \left(2F_3\right)},& l\in \left[2,4\right]\hfill \end{array}\hfill \end{array}\right.$$

(19)

$${\stackrel{\rightharpoonup }{e}}_{ij}^{\left(l+1\right)}=\mathrm{ReLU}\left({\mathbf{W}}_{\mathrm{E}}^{\left(l\right)}{\stackrel{\rightharpoonup }{v}}_i^{\left(l\right)}+{\mathbf{W}}_{\mathrm{E}}^{\left(l\right)}{\stackrel{\rightharpoonup }{v}}_j^{\left(l\right)}+{\mathbf{W}}_{\mathrm{EE}}^{\left(l\right)}{\stackrel{\rightharpoonup }{e}}_{ij}^{\left(l\right)}\right)$$

(20)

$${\mathbf{W}}_{\mathrm{E}}^{\left(l\right)}\in \left\{\begin{array}{c}{\mathbf{R}}^{F_3\times F_1},l=1\hfill \\ {\mathbf{R}}^{F_3\times F_3},l\in \left[2,4\right]\hfill \end{array}\right.$$

(21)

$${\mathbf{W}}_{\mathrm{EE}}^{\left(l\right)}\in \left\{\begin{array}{c}{\mathbf{R}}^{F_3\times F_2},l=1\hfill \\ {\mathbf{R}}^{F_3\times F_3},l\in \left[2,4\right]\hfill \end{array}\right.$$

(22)

(2) Middle layer and hidden layer network ($l\in \left[5,8\right]$).

The middle layer and hidden layer are all fully connected layers. The input and output dimensions are both $F_3$, and the feature transfer adopts the method shown in Equation (23).

$${\stackrel{\rightharpoonup }{v}}_i^{(l+1)}=\mathrm{ReLU}\left({\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}{\stackrel{\rightharpoonup }{v}}_i^{\left(l\right)}\right),{\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}\in {\mathbf{R}}^{F_3\times F_3}$$

(23)

(3) Output layer network transfer ($l=9$).

The output layer is also fully connected, and the output dimension is 1. The feature transmission uses the method shown in Equation (24).

$${\stackrel{\rightharpoonup }{v}}_i^{(l+1)}={\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}{\stackrel{\rightharpoonup }{v}}_i^{\left(l\right)},{\mathbf{W}}_{\mathrm{v}}^{\left(l\right)}\in {\mathbf{R}}^{1\times F_3}$$

(24)

(4) Mask mechanism

The mask mechanism is introduced to avoid infeasible action choices. For nodes that violate the constraints, the output probabilities are controlled to zero. The mask label of $nod{e}_i$ is $m_i$. When choosing the next node, if $nod{e}_i$ violates the constraint, $m_i=0$; otherwise, $m_i=1$. If the probability of the final output $nod{e}_i$ is $v_i^{\ast }$, Equations (25)–(28) are used to realize the mask mechanism.

$$v_i^{\ast }=v_i^{\ast }+\left|\underset{i}{min}{v}_i^{\ast }\right|$$

(25)

$$v_i^{\ast }=v_i^{\ast }-\underset{i}{max}\left(v_i^{\ast }{m}_i\right)$$

(26)

$$v_i^{\ast }=\frac{exp\left(v_i^{\ast }{m}_i\right)}{{\displaystyle \sum _k}exp\left(v_k^{\ast }{m}_k\right)}$$

(27)

$$v_i^{\ast }=\frac{v_i^{\ast }{m}_i}{{\displaystyle \sum _k}{v}_k^{\ast }{m}_k}$$

(28)

3.4. Training Method

The parameters of the GDNN must be obtained by learning from large batches of training data. In the proposed method, we apply the proximal policy optimization (PPO) proposed by [33] to train the GDNN.

The training framework of PPO follows actor–critic [34], which includes an actor network with the parameters ${\Theta }_Q$ and a critic network with the parameters ${\Theta }_{{}_V}$. The pseudo-code is shown in Algorithm 2.

Algorithm 2 GDNN-PPO algorithm

Require:  Initialize the training parameter cropping factor $ϵ$, the mean square error factor $c_1$, the entropy factor $c_2$, the batch size K, the parameter update step size $Tp$, the parameter update optimization times k, the number of training episodes N Ensure:  The GDNN optimal network parameters $\Theta$   1: repeat   2:    Generate the instance $Emp=\left\{\mathbf{E},\mathbf{v},S^{\mathrm{sat}}\right\}$.   3:    while $s_t$ is not done do   4:   Choose $a_t$ by sampling according to $p_{{\Theta }_Q}\left(a_t\left|s_t\right.\right)$.   5:   Execute $a_t$, gather $r_t$ and $p_{{\Theta }_Q}\left(a_t\left|s_t\right.\right)$, update $s_{t+1}=\left({\mathbf{E}}_{t+1},{\mathbf{v}}_{t+1}\right)$.   6:   Store $\left\{s_t,a_t,r_t,p_{{\Theta }_Q}\left(a_t\left|s_t\right.\right)\right\}$ in sampling pool.   7:   $tp=tp+1$   8:   if $tp=Tp$ then   9:      Update r 10:      repeat 11:     repeat 12:        $u_t\left(\Theta \right)=\frac{{p_{\Theta }}_{{}_V}\left(a_t\left|s_t\right.\right)}{{p_{\Theta }}_{{}_Q}\left(a_t\left|s_t\right.\right)}$ 13:        ${\hat{A}}_t={\hat{r}}_t-V_{\Theta }\left(a_t\left|s_t\right.\right)$ 14:        $L_t\left(\Theta \right)={\hat{\mathrm{E}}}_t\left[L_t^{CLIP}\left(\Theta \right)+c_1{L}_t^{VF}\left(\Theta \right)+c_{2}S\left[p_{\Theta }\right]\left(s_t\right)\right]$ 15:        Update $\Theta$ using $\mathbf{SGD}$. 16:     until All $⌊Tp/\phantom{TpK}K⌋$ batches trained 17:      until Update $\Theta$ with k epochs 18:      ${\Theta }_V={\Theta }_Q$ 19:      Clear Sampling pool. 20:      $tp=1$ 21:   end if 22:    end while 23: until All N instances end. 24: return $\Theta$

At each step $t_p$, we sample the action $a_t$ based on the probability of the actor output $p_{{\Theta }_Q}\left(a_t\left|s_t\right.\right)$ and save the sample $\left\{s_t,a_t,r_t,p_{{\Theta }_Q}\left(a_t\left|s_t\right.\right)\right\}$ in the sampling pool. The parameters are updated when the number of samples $t_p$ reaches the parameter update step size $Tp$. We first update the reward $r_t$ according to Equation (29) to represent an assessment of the expected reward of action $a_t$. The critic evaluates the value $V_{\Theta }\left(a_t\left|s_t\right.\right)$ of the actor and probability ${p_{\Theta }}_{{}_Q}\left(a_t\left|s_t\right.\right)$. Then, the loss is calculated in Lines 12–14, and the actor parameters ${\Theta }_Q$ are updated in Line 15 according to stochastic gradient descent (SGD) [35]. The parameters ${\Theta }_V$ of the critic are updated by copying the updated parameters of the actor. Finally, the optimal model parameters are generated through iteration.

$${\hat{r}}_i=\sum _{k=i}^n{r}_k=\sum _{k=i}^n\gamma \ast {pri}_{j_k}$$

(29)

4. Experiments

4.1. Experimental Setting for AEOSSP

4.1.1. Datasets

The instance generation of the AEOSSP follows the characteristics of the satellite resources and orbits. The parameters of the instances are randomly generated according to a normal distribution, which can better increase the conflict between tasks. The parameter distributions of the instances are shown in

Table 2. Parameter distribution of the instances.

Parameter	Distribution	Details
${\theta }^{\mathrm{side}}$	$\mathrm{U}\left(-{\theta }_{max},{\theta }_{max}\right)$	${\theta }_{max}=45{}^{\circ }$
$\chi$ 1	$\mathrm{U}(\lambda n_{tsk}+t{w}_{max}/\phantom{t{w}_{max}2}2,T_{plan}-\lambda n_{tsk}-t{w}_{max}/\phantom{t{w}_{max}2}2)$	$t{w}_{max}=300$ s, $T_{plan}=5400$ s
$t^{\mathrm{side}}$	$\mathrm{U}\left(\chi -\lambda n_{tsk},\chi +\lambda n_{tsk}\right)$	$\lambda =12$
$ct$	$\mathrm{U}\left(c{t}_{min},c{t}_{max}\right)$	$c{t}_{min}=5$ s, $c{t}_{max}=20$ s
$pri$	$\mathrm{U}\left(1,pr{i}_{max}\right)$	$pr{i}_{max}=10$
$l_{tw}$ 2	$\mathrm{U}\left(t{w}_{max}/\phantom{t{w}_{max}4,t{w}_{max}/\phantom{t{w}_{max}2}2}4,t{w}_{max}/\phantom{t{w}_{max}2}2\right)$	$t{w}_{max}=300$ s

¹ The intermediate time of the schedule period. ² The length of the VTW. The above parameters are generated as integers.

, and the satellite capability parameters are shown in

Table 3. Satellite capability parameters.

Parameter	Description	Value
${\theta }_{max}$	Maximum roll angle	${45}^{\circ }$
${\phi }_{max}$	Maximum pitch angle	${45}^{\circ }$
${\psi }_{max}$	Maximum yaw angle	${90}^{\circ }$
$Egy$	Initial power	5000 unit
$\zeta$	Minimum power	$0.05$ unit
${\theta }_0$	Initial roll angle	$0^{\circ }$
${\phi }_0$	Initial pitch angle	$0^{\circ }$
${\psi }_0$	Initial yaw angle	$0^{\circ }$
$t_0$	Initial state time	0 s
$ute$	Power consumption for attitude transition	2 unit/s
$uie$	Power consumption for observation	2 unit/s

.

Based on the above rules, the experiment generates instances with tasks scaled 40, 60, 80, and 100. The instance with a task scale of 40 is shown in Figure 7, where the label means the task ID and profit. The figure shows that the task VTW distribution is relatively dense, and the conflicts of tasks are sufficiently large to effectively reflect the performance of the algorithms.

We propose three indicators to measure the performance of the algorithms: the average scheduling profit (ASP), the average scheduling time (AST), and the percentage of excess scheduling profit (PSP). The ASP measures the solution quality of the algorithm at different scales, the AST is used to reflect the timeliness of the algorithm, and the PSP indicates the difference in the ASP between the proposed method and other algorithms. All network training and experiments use NVIDIA TITAN RTX, i9-11900K CPU, and 64.0 GB memory. The algorithms are coded in Python, and the deep learning framework uses PyTorch 1.9.0.

4.1.2. Competitors

To verify the validity of the proposed method, we use some construction heuristics as baselines of the AEOSSP. Specifically, we design the following four heuristics: the start time of observational time window ascending (STWA), the profit of task descending (PTD), the ratio of profit and image time descending (RPID), and the conflict degree of task descending (CDTD). The construction heuristics solving framework is shown in Algorithm 3. Each heuristic sorts the candidate tasks by the construction rule and inserts the tasks that satisfy the constraints into the solution sequence in turn until there is no task to insert. In CDTD, the conflict degree $C{d}_i$ means the number of VTW overlaps between $tas{k}_i$ and other tasks, defined as Equations (30) and (31). We constitute the GDNN-DQN by training the GDNN with the DQN.

$$C{d}_i=\sum _{k=1}^{n_{tsk}}{o}_{ik}$$

(30)

$$o_{ij}=\left\{\begin{array}{c}1,\mathrm{if}w{b}_i<w{e}_j\wedge w{b}_j<w{e}_i\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.$$

(31)

Algorithm 3 General framework of construction heuristics

Require:  Task sequence $Tsk$, sequence rule $pt$, ascending sign $l_{\mathrm{rank}}$ Ensure:  Scheduling solution $Sln$   1: $Sln=\varnothing$   2: $Sl{n}_{\mathrm{temp}}=\varnothing$   3: if  $1==l_{\mathrm{rank}}$  then   4:    $Tsk=\mathrm{RankAscendingBy}\left(Tsk,pt\right)$   5: else   6:    $Tsk=\mathrm{RankDescendingBy}\left(Tsk,pt\right)$   7: end if   8: for $i=0$ to $\left|Tsk\right|$ do   9:    $Sl{n}_{\mathrm{temp}}=Sln$ 10:    $Sl{n}_{\mathrm{temp}}.\mathrm{InsertTask}\left(Ts{k}_i\right)$ 11:    $isFeasible=\mathrm{ConstraintCheck}\left(Sl{n}_{\mathrm{temp}}\right)$ 12:    if $isFeasible==\mathrm{TRUE}$ then 13:   $Sln=Sl{n}_{\mathrm{temp}}$ 14:    end if 15: end for 16: return  $Sln$

In addition, to validate the efficiency of GDNN-PPO, some high-quality solution algorithms are required for comparison. As mentioned in Section 1, the existing exact methods simplify the time-dependent transition time, and the solution time of CPLEX is unacceptable when the number of VTWs exceeds 27; thus, we do not consider the exact algorithms. We compare the GDNN-PPO algorithm with three high-quality competitors: (1) GRILS [10], which is a state-of-the-art heuristic method for the AEOSSP; (2) self-adaptation differential evolution (SDE) [36], which is an algorithm that has been shown to be effective at solving the AEOSSP; and (3) the self-adaptation genetic algorithm (SGA) [14], which is an improved genetic algorithm designed to solve the AEOSSP. The decoding method of the SGA is the same as Algorithm 3. The crossover and mutation parameters are updated according to the proportion of entering the next generation. The parameters of the high-quality competitors are shown in

Table 4. Parameters setting of high-quality competitors.

Method	Parameter
GRILS	Maximum iteration number with no improvement: 300
	StartGreed = 1, GreedRange = 0.2
	GreedDecrease = 0.02
	Place of start removal $S_d=1$
	Number of observations removed each time $R_d=1$
SDE	Population size $Pop=50$
	Iteration number $N_{\mathrm{iter}}=200$
	Initial crossover probability $Cr=0.5$
	Initial mutation probability $Mp=0.2$
	Encoding method: real-valued encoding
	Offspring selection: championship
	Elite solution retention ratio: 0.1
SGA	Population size $Pop=50$
	Iteration number $N_{\mathrm{iter}}=200$
	Initial crossover probability $Cr=0.8$
	Initial mutation probability $Mp=0.2$
	Encoding method: integer encoding
	Offspring selection: roulette
	Elite solution retention ratio: 0.1

.

4.2. Training Process Analysis

4.2.1. Training Parameters

The GDNN parameter settings are shown in

Table 5. Main parameters of the GDNN training.

Parameter	Description	Value
$ϵ$	Trim function parameter	0.1
$c_1$	Mean square error factor	0.5
$c_2$	Entropy factor	0.001
K	Batch size	32
$T_p$	Parameter update step	1024
k	Parameter update optimization times	3
N	Training episode generation	50,000
$lr$	Learning rate	0.0005

.

4.2.2. GDNN Structure Optimization

A network with a structure that is too complex may affect the calculation speed and convergence, and a network with a structure that is too simple may be unable to characterize the problem well. To improve the efficiency of the GDNN, we analyze the GDNN structure.

Due to the limited availability of servers for large-scale training, we only considered three parameters: the embedding layer dimension $F_3$, the number of hidden layers $n_{hid}$, and the number of embedding layers $n_{em}$.

For the embedding layer dimension $F_3$ and the number of hidden layers $n_{hid}$, experiments are designed to test networks of $128\times 4$, $64\times 3$ and $32\times 2$ ($F_3\times n_{hid}$) corresponding to high, medium, and low dimensions. We train networks of different dimensions using instances with 100 tasks and training episodes of $10,000$ and test them with 50 instances with tasks scaled 40, 60, 80, and 100. Results are shown in

Table 6. Experimental results of different dimensional networks in instances with different scales.

Instance Scale	$32\times 2$		$64\times 3$		$128\times 4$
	ASP	AST (S)	ASP	AST (S)	ASP	AST(S)
40	176.4	0.240	183.7	0.263	183.5	0.309
60	263.9	0.443	266.3	0.488	269.8	0.637
80	349.7	0.757	356.6	0.868	354.0	1.180
100	433.2	1.244	439.4	1.466	437.6	1.843

Bold indicates the best ASP in all algorithms.

. In addition, the number of embedding layers is tested with 3, 4, 5 and 6. The networks are trained by instances with a task scale of 40 and training episodes of 10,000. We test them with 50 instances with a task scale of 40. The test result is shown in

Table 7. Experimental results of different numbers of embedding layers in instances with a task scale of 40.

Embedding Layers	3	4	5	6
ASP	186.3	188.6	188.3	186.2
AST (S)	0.192	0.228	0.269	0.301

Bold indicates the best ASP and AST in all algorithms.

.

From Table 6, at the current cost of training, the low-dimensional network is the fastest but least profitable. The solution time of the high-dimensional network is longer, but the improvement in solution quality is small compared to the medium-dimensional network. For the number of embedding layers, results in Table 7 show that the network with four embedding layers can obtain the best ASP. The networks with five and six embedding layers have longer scheduling times but no improvement in scheduling profits. Therefore, the network parameters are finally set to $F_3=64$, $n_{hid}=3$ and $n_{em}=4$ in this study, and thus, the GDNN-64x3 network with four embedding layers is considered the best network for the AEOSSP.

4.2.3. GDNN Training

In the proposed method, we train GDNN with tasks scaled of 40, 60, 80, and 100 by the PPO. The training processes are shown in Figure 8. Results show that the average profit increases rapidly in the first 5000 episodes, indicating that the network parameters are continuously being updated. When 10,000 episodes are reached, the profit improvement begins to level off and remains stable. In general, after 50,000 training episodes, the network converges stably.

4.3. Performance Analysis

We generate instances with scales of 40, 60, 80, and 100 to test the efficiency of the proposed method. Instances with different scales are solved by the GDNN trained on the corresponding scales. For comparison, we divide the competitors into construction heuristics and high-quality competitors.

4.3.1. Comparison with Construction Heuristics

For test data, we generate 1000 instances with 40, 60, 80 and 100 tasks, and experimental results are shown in

Table 8. Experimental results of construction heuristics in different instances.

Instance Scale		40.0	60.0	80.0	100.0
GDNN-PPO	ASP	190.3	277.4	364.0	451.7
	AST (S)	0.283	0.581	1.175	1.671
	PSP (%)	0.00	0.00	0.00	0.00
GDNN-DQN	ASP	180.1	257.3	328.1	370.2
	AST (S)	0.206	0.392	0.659	1.134
	PSP (%)	5.61	7.82	10.94	22.01
STWA	ASP	115.4	172.1	227.9	284.5
	AST (S)	0.004	0.006	0.008	0.010
	PSP (%)	64.82	61.24	59.71	58.75
PTD	ASP	125.6	187.8	249.1	311.1
	AST (S)	0.007	0.012	0.020	0.027
	PSP (%)	51.49	47.74	46.16	45.21
RPID	ASP	121.2	181.6	240.7	301.0
	AST (S)	0.007	0.013	0.020	0.028
	PSP (%)	57.02	52.75	51.22	50.07
CDTD	ASP	99.6	149.7	197.2	246.6
	AST (S)	0.007	0.013	0.020	0.028
	PSP (%)	90.93	85.36	84.63	83.17

Bold indicates the best ASP in all algorithms.

. GDNN-PPO outperforms all the competitors in all task-scale instances with the highest Wilcoxon p-value of 0. Specifically, the PSP of GDNN-DQN reaches 11.6%, and the PSPs of the other four algorithms are all over 45%. Therefore, GDNN-PPO can obtain high-quality solutions for solving the AEOSSP. Regarding AST, GDNN-PPO has the longest solution time in all instances. However, even for instances with 100 tasks, the AST of GDNN-PPO is within 2 s, which is acceptable for its high solution quality.

4.3.2. Comparison with High-Quality Competitors

Due to the randomness and high computational cost of meta-heuristics, we choose one instance with a task scaled of 40, 60, 80, and 100, and repeat it 30 times to obtain ASP, AST, and PSP. Experimental results are shown in

Table 9. Experimental results of meta-heuristics in different instances.

Instance Scale		40	60	80	100
GAT-PPO	ASP	197.0	272.0	380.0	454.0
	AST (S)	0.270	0.568	1.162	1.676
	PSP (%)	0.00	0.00	0.00	0.00
GRILS	ASP	201.5	281.4	397.8	487.4
	AST (S)	73.452	224.986	370.964	600.308
	PSP (%)	2.23	3.33	4.47	6.86
SDE	ASP	159.0	215.7	294.0	348.8
	AST (S)	58.760	162.508	342.463	618.514
	PSP (%)	23.90	26.10	29.27	30.17
SGA	ASP	154.5	194.6	294.1	336.1
	AST (S)	24.170	32.661	45.607	57.583
	PSP (%)	27.54	39.80	29.19	35.09

Bold indicates the best ASP in all algorithms.

.

GRILS is shown to obtain the best ASP in all scale instances, demonstrating its state-of-the-art capability to solve the AEOSSP. GDNN-PPO outperforms SDE and SGA in ASP by more than 23%, and its advantage becomes more apparent as the scale increases. Although GDNN-PPO does not perform as well as GRILS in ASP (4.2% lower on average), it decreases ASP by orders of magnitude. Also, unlike GDNN, which constructs the solution at one time, meta-heuristics must repeat the evaluation of the same solution, which is time-consuming. As shown in Figure 9, the ASTs of GRILS and SDE increase nonlinearly and dramatically with the scale of tasks. The AST of SGA increases marginally but is still much higher than that of GNDD-PPO. Therefore, GDNN-PPO has a time-sensitive advantage over high-quality competitors in solving large-scale AEOSSP.

4.3.3. Validation of Scalability

To validate the upwards and downwards scale-solving ability of the GDNN-PPO trained at a specific instance scale, we use GDNN-PPO-X, where X is the number of instance tasks used for training, to solve 50 test instances with different task scales and calculate the ASP. Results are shown in

Table 10. Experimental results of scalability in different instances.

Task Scale	40	60	80	100
GDNN-PPO-40	190.62	278.62	367.82	448.24
GDNN-PPO-60	188.90	277.58	366.74	449.14
GDNN-PPO-60	188.36	277.42	365.74	449.70
GDNN-PPO-100	186.96	277.42	366.00	447.02

Bold indicates the best ASP in all algorithms.

, and the gap between the ASP of GDNN-PPO-X and the optimal ASP is shown in Figure 10.

Because the conflict degree of VTWs increases with the task scale, the decision neural networks trained by instances with specific scales can learn different strategies. All GDNN-PPOs are shown to calculate a solution with a small difference from optimal solutions, highlighting the scalability of this method. Among these four models, GDNN-PPO-40 has better scalability and can achieve optimal performance in solving the AEOSSP with 40, 60 and 80 task scales.

5. Conclusions

This paper proposes a graph-based DRL method called GDNN-PPO to solve the AEOSSP with time-dependent attitude transition time. We model the AEOSSP with the time-attitude adjacency graph and reformulate the problem as the MDP. Then, we extract the features of the AEOSSP, including node and edge features, and design a GDNN to guide task choice. Finally, we train the GDNN by PPO and design experiments to verify the validity of the proposed method. Experimental results show that GDNN-PPO outperforms all construction heuristics in ASP by at least 45% and surpasses high-quality competitors except for the state-of-the-art algorithm GRILS with regard to ASP and AST. Regarding GRILS, the difference between the GDNN-PPO performance in all instances is less than 7%; However, the AST of GRILS is 345 times longer than that of GDNN-PPO on average. Thus, GDNN-PPO performs well at large scales and with rapid responses when solving the AEOSSP and has strong potential for future applications in large constellations and new management models.

Although GDNN-PPO demonstrates significant advantages in solution time, there is potential to enhance its scheduling priority. In future work, we plan to further improve the GDNN solving efficiency by optimizing the feature selection, structure design, etc. Additionally, we plan to combine the proposed method with other algorithms to solve more complex satellite scheduling problems, such as multi-satellite scheduling problems.