Dynamic Reporting Nodes Selection Method for Network Awareness Based on Active–Passive Integrated Network Telemetry in LEO Satellite Networks

Abstract

Satellite network awareness is a key technology for obtaining the status of onboard resources and supporting efficient network management. However, the dynamic characteristics of the network topology results in dynamic transmission delays of awareness data, while the instability of satellite-ground links poses challenges in selecting awareness reporting nodes. Therefore, a dynamic reporting nodes selection method for network awareness based on active-passive integrated network telemetry (DRNSM-APINT) is proposed. Specifically, an APINT model is built which can dynamically select the optimal reporting node and reduce awareness time through the bidirectional aggregation of telemetry packets along the service path. To address the requirements of various services, flexible division strategy through virtual boundaries and the integration of APINT models are combined to achieve customized selection of multiple reporting nodes on each path. Additionally, a lightweight Q-learning algorithm is proposed to dynamically solve the above issue with lower computational complexity. The simulation results demonstrate that the DRNSM-APINT method can decrease the awareness time by 34.5% and improve the overall awareness performance by 17.1%, thereby significantly enhancing the timeliness performance and dynamic characteristics of network awareness.

1. Introduction

With the continuous expansion of the low-Earth-orbit (LEO) satellite network, awareness of the status of multi-dimensional heterogeneous network resources can provide basic data support for delivering high-quality network services, and has become the key technology for the operation and maintenance of satellite network. However, the high dynamism of satellite network topology, coupled with the instability of inter-satellite links (ISLs) and satellite-ground links (SGLs), can result in substantial delays in the transmission of awareness data, reducing the real-time performance of network awareness [1]. Meanwhile, traditional terrestrial network awareness methods cannot meet the differentiated and dynamic service awareness requirements of satellite networks [2]. Therefore, efficient network awareness methods have emerged as a key area of focus in the ongoing development of LEO satellite networks.

Network awareness primarily consists of measurement and reporting process. Measurement mainly involves collecting the resource status information of nodes and transmitting awareness data in the network. Reporting refers to the process of aggregating awareness data and transmitting them to the ground control center [3]. Research on resource-aware optimization methods for terrestrial network has lasted for decades. However, to the best of our knowledge, there are no relevant studies on LEO satellite network scenarios. Additionally, most studies of terrestrial networks have typically concentrated on expanding the awareness scope [4] and optimization of awareness overhead [5], while there is a lack of research focusing on the flexibility and the timeliness of awareness method [6]. For example, Yang et al. [7] employed an active telemetry method for giant satellite constellations, meaning that both the measurement and reporting processes utilize separate links or frequency bands for transmission. By optimizing resource allocation during the awareness process, the volume of awareness data transmitted was enhanced. However, this method occupied substantial on-board resources, thereby increasing awareness overhead and the complexity of the constellation. In order to minimize the awareness overhead, in-band network telemetry (INT) technology has been proposed as a lightweight passive telemetry approach [8]. It conveys awareness data by utilizing the additional fields appended to the service data packet headers [9]. The transmission route of the service concurrently serves as the awareness path, enabling operations such as reading, writing, and deleting awareness data through programmable technology [10], which is suitable for multi-service concurrent scenarios in large-scale satellite network [11]. Liu et al. [12] proposed utilizing the Euler path planning algorithm to decrease the redundancy of INT paths, thereby optimizing the telemetry paths within LEO constellations. Meanwhile, regarding the reporting process, the awareness method based on INT often reports after the target node separates the awareness and service data. For long-distance paths with multi-hop, a multi node reporting method based on fixed hop intervals can also be adopted to mitigate cumulative costs and enhance the timeliness performance of reporting.

Unlike terrestrial networks, the unique, complex, and dynamic characteristics of satellite network pose three main challenges to awareness technology: (1) The real-time challenge brought by the large delay in the transmission path during the network measurement process [13]. Due to the high-speed movement of satellite nodes and the considerable distance between satellites, the propagation delay is relatively significant when compared to terrestrial networks [14]. Moreover, under different path length, the delay generated by each node increases further through multi-hop accumulation, greatly affecting the real-time awareness and reducing the accuracy of awareness data [15]. (2) The instability of the SGL results in a limited number of optional reporting nodes and poor dynamic selection of reporting nodes. The SGLs of nodes have short duration and poor stability leading to a limited number of reporting nodes that can be selected on the path. At this point, the awareness data must either wait on the satellite or switch to another reporting node, thereby increasing the reporting time. (3) The high complexity of dynamic awareness caused by the dual heterogeneity of network topology and services. The dynamism of the topology reduces the timeliness of awareness and increases the difficulty of overall network awareness. The dynamism of services enhances the complexity of adjusting awareness methods in different time-varying scenarios. The fixed measurement and reporting strategy exhibits limited flexibility when it comes to optimizing the measurement process or the selection of reporting nodes in dual dynamic scenarios, making it impossible to be adjusted adaptively based on actual awareness requirements.

In response to the challenges of network awareness in LEO satellite networks, a dynamic reporting nodes selection method for network awareness based on active-passive integrated network telemetry (DRNSM-APINT) is proposed, which can flexibly adjust the telemetry mechanism and dynamically select the optimal number and position of reporting nodes for different services. Specifically, the method proposes an active-passive integrated network telemetry (APINT) model. Unlike the existing active and passive telemetry systems, which are both initiated by a single node, in this method, passive telemetry and active telemetry are initiated by the source and target nodes, respectively, converging toward the center of the path to shorten the awareness path. The optimal location for the reporting node is selected based on constraints related to the transmission time of ISLs and the status of SGLs, thereby reducing the time of measurement and reporting, and enhancing the timeliness of awareness in single-node reporting scenarios. Furthermore, considering that the single-node reporting mechanism in LEO satellite network cannot meet the awareness timeliness requirements of multi-hop and long-distance paths, an optimized selection method for multiple reporting nodes is proposed. The concept of virtual boundaries is introduced to divide the awareness path. Additionally, within each segment, the APINT model is utilized to select the optimal reporting nodes. Then, by comparing the effective-ness of selecting multiple reporting nodes across various virtual boundary partitioning scenarios, the most suitable number of reporting nodes for the current awareness path is determined, thereby enabling optimized and flexible selection of multiple reporting nodes. However, due to variations in path hops, distance of ISLs and SGLs, and the characteristics of these links among different services, the optimal number of reporting nodes for each service path varies in multi-service scenarios. Consequently, it is necessary to dynamically devise an optimal reporting scheme for each service, which further complicates the solution process. Considering the varying selection requirements for reporting nodes across diverse service scenarios, based on the model-free characteristic and the strong adaptability to dynamic scenarios of Q-learning, a lightweight Q-learning algorithm based on a pre-planned virtual boundary is proposed. The boundary is divided using the equal division method, and the selection of virtual boundaries for each service path is pre planned through dynamic boundary floating. The method can exclude discontinuous boundary combinations, decrease the number of search iterations required in Q-learning, and simplify the complexity of the algorithm’s solution process. Simultaneously, based on the proposed lightweight Q-learning algorithm, the optimal number and position of reporting nodes for different paths in multi-service scenarios are dynamically solved, achieving efficient awareness timeliness and flexible dynamic selection of reporting nodes in LEO satellite networks.

The main contributions of this paper are as follows:

(1)

An APINT model is built for LEO satellite networks. Combining active and passive telemetry, the awareness path is shortened. Additionally, the optimal location for the reporting node is selected based on constraints related to the transmission time and status of SGLs, thereby reducing awareness time and enhancing awareness timeliness.

(2)

A dynamic reporting nodes selection method of network awareness based on active-passive integrated network telemetry (DRNSM-APINT) is proposed, introducing the concept of virtual boundaries to divide the awareness path into segments, and using the APINT model to find the optimal reporting node of each segment. By comparing the selection results of reporting nodes across various virtual boundary settings scenarios, the best reporting solution was chosen for the current awareness path, solving the problem of flexible selection of multiple reporting nodes.

(3)

A multi-reporting nodes selection algorithm based on lightweight Q-learning for various services is proposed. The dynamic pre-planning of virtual boundaries mitigates the computational complexity of Q-learning solutions. The proposed lightweight Q-learning algorithm dynamically determines the optimal number and location of reporting nodes for different service paths, achieving efficient timeliness of awareness and flexible selection of reporting nodes in dual time-varying scenarios of service and path.

The remainder of this paper is organized as follows: Section 2 introduces the composition of LEO satellite network and presents the APINT model. Section 3 provides the details of optimized selection method for multiple reporting nodes. Section 4 introduces a lightweight Q-learning algorithm based on pre-planned virtual boundary. Section 5 validates the proposed method through simulation experiments. Section 6 provides a brief summary of the paper.

2. Composition of LEO Satellite Network and APINT Model

2.1. Composition of LEO Satellite Network

The composition of LEO satellite network is shown in Figure 1. A group of LEO satellites in the same layer and orbit constructs the satellite network through ISLs, and wireless communication is used for the transmission of inter-satellite services between nodes. When flying over the service area, satellites establish links with ground gateway stations or user terminals through SGLs, allowing for the backhaul of network awareness data.

The LEO satellites in the network are represented as $S_i=\{S_1,S_2,\dots ,S_N\}$ with $N$ satellites. The set of ground gateway stations is represented as $\mathrm{Gat}=\{G_1,G_2,\dots ,G_J\}$. Considering the dynamic features of the network topology, a specific time range is divided into several time slots $T=\{t_1,t_2,\dots ,t_k\}$, each with a duration of $∆t$. Within each time slot, the network topology remains stable, but it experiences dynamic changes between different time slots. At the same time, the task tuple $\{Z,P_Z,T_Z\}$ is used to represent each service flow, where $Z$ is the set of service identifiers, $P_Z$ is the set of service transmission paths, and $T_Z$ represents the time from the initiation of service transmission to its reception at the gateway station.

Table 1. List of variables.

Symbol	Symbolic Meaning	Symbol	Symbolic Meaning
$S_i$	Satellite number	$T_{\mathrm{final}}^{S_{rp}}$	Awareness data reach ground time
$Z$	Service identifiers	$V_Y$	Set of segments
$P_Z$	Transmission path	$Y$	Number of segments
$T_Z$	Time slots	$N_{VY}$	Number of satellites in segment
$L_{\mathit{ij}}$	Inter-satellite distance	$\mathrm{Eva}$	Evaluation function value
$C$	Propagation rate	$\overline{T_V^Z}$	Average telemetry and reporting time
$t_{\mathrm{pro}}^i$	Propagation delay	$E_{Vi}$	Average awareness overhead
$t_{\mathrm{trans}}^i$	Transmission delay	$B_M^{S_{Vi}^x}$	Awareness overhead of metadata
$t_{\mathrm{meta}}^i$	Metadata writing delay	$\mathsf{∆}$	Floating range
$t_{\mathrm{op}}^i$	Passive telemetry delay	$W$	State of Q-learning
$t_{\mathrm{ap}}^i$	Active telemetry delay	$A$	Action of Q-learning
$t_{\mathrm{re}}^i$	SGL propagation time	$R\left(w,a\right)$	Reward function
$T_R^{S_{rp}}$	Awareness data arriving time	$Q\left(w,a\right)$	Q-values

lists the commonly used symbols in this article.

2.2. Propagation and Transmission Delay

In the LEO satellite network, each service is transmitted from the source node $S_a$ to the target node $S_b$ with the assumption that the topology remains unchanged during the transmission process. The service transmission path $P_z$ has $M$ satellites, and the propagation delay between the nodes is given by:

$$t_{\mathrm{pro}}^i={\displaystyle \frac{L_{\mathit{ij}}}{C}},$$

(1)

where $L_{\mathit{ij}}$ represents the inter-satellite distance from node $i$ to node $j$, and $\forall i,j\in P_z$. Additionally, $C$ is the propagation rate on the link. The time consumed during the process of node $i$ sending data packet to the next node $j$ is referred as the transmission delay:

$$t_{\mathrm{trans}}^i={\displaystyle \frac{E_z^i}{{\mathsf{\phi }}_i}},$$

(2)

where $E_z^i$ represents the size of the data for the $z$-th service sent by the node $i$, and ${\mathsf{\phi }}_i$ is the transmission rate of the switch on the satellite.

2.3. APINT Model

The awareness of target objects in the network mainly include various resources used by nodes to provide network services, such as bandwidth, computing, storage, and queue resources, as well as ISL status information [16]. Each node in the network has awareness capability [17]. Based on programmable technology, active telemetry and passive telemetry can be performed, and the awareness process can be designed, modified, and operated [18].

2.3.1. Passive Telemetry

The passive telemetry approach based on INT is employed within the network, where node resource status information is inscribed as metadata into telemetry packets as service flows through. The awareness data transmit with the service, and the service transmission path can be regarded as an awareness path [19]. The INT packet mainly consists of a telemetry packet header and telemetry data, where the telemetry packet header is a specified 12-byte content, and the telemetry data are primarily composed of the aggregated metadata from each node [20]. Due to the small length of INT packet, it can be directly placed in the header of the service packet without occupying payload space. After editing and integrating the resource status information of each node, the written metadata typically have a length of 4 to 8 bytes [11]. $H_{\max }$ represents the maximum cumulative number of hops that telemetry data can contain, which can be set and modified according to actual needs, with a common value of 8 hops. When the path length exceeds $H_{\max }$, subsequent nodes will no longer write metadata to the telemetry data or reset the telemetry data and cumulative number of hops after reporting the current telemetry data.

In addition, the CPU processing time consumed by the node to read the telemetry packet header and write metadata into the telemetry data is denoted as $t_{\mathrm{meta}}^i$. Since the average length of the metadata is relatively short, the time taken to write the metadata is much smaller than the node’s propagation delay and transmission delay, making $t_{\mathrm{meta}}^i$ a constant value. Therefore, the time consumed by each node for passive telemetry during the process of inserting awareness data, and transmitting them along with the service to the next node is as follows:

$$t_{\mathrm{op}}^i={ t}_{\mathrm{pro}}^i+{ t}_{\mathrm{trans} }^i+t_{\mathrm{meta}}^i, \forall i \in P_z,$$

(3)

2.3.2. Active Telemetry

Each node in the network also has the capability to actively send telemetry packets to other nodes along the path for active telemetry. Additionally, the nodes along the path can also perform metadata writing operations on the telemetry data. The propagation delay of active telemetry packet for each node is consistent with Equation (1). The active telemetry packet contains only a portion of the service packet header information, telemetry packet header, and telemetry data. The data length of active telemetry packet is relatively small, which means the transmission delay of the node can be set as a constant value $t_{\mathrm{at}}^i$. Therefore, the time consumed by the node to complete active awareness is as follows

$$t_{\mathrm{ap}}^i=t_{\mathrm{pro}}^i+t_{\mathrm{at}}^i+t_{\mathrm{meta}}^i,\forall i \in P_z,\forall z \in Z,$$

(4)

2.3.3. Reporting Process

The set of distance from each satellite on the current service awareness path $P_z$ to the nearest ground gateway station is $D_z=\left\{D_1,\dots ,D_M\right\}$. The propagation time for the SGL is as follows:

$$t_{\mathrm{re}}^i={\displaystyle \frac{D_i}{C}} , {\forall D}_i\in D_z,$$

(5)

where $C$ represents the wireless transmission rate. Unlike terrestrial network, few nodes in the network can establish stable SGLs at any given moment. Therefore, nodes can only be recognized as potential reporting nodes when stable SGLs exist. $T_z$ represents the time when the current service starts transmitting until the ground gateway station receives the service. The starting transmission time can be defined as $T_0$, and the time interval of the available link between node $i$ and the gateway station is $t_r^i=\left[t_s^i,t_e^i\right], t_e^i < T_{0 }+{ T}_z, i \in { P}_z$, where $t_s^i$ and $t_e^i$, respectively, represent the moments when the node starts establishing the link and when the link is disconnected, and data reporting can only occur within this interval.

2.3.4. Integrated Network Telemetry

Based on passive telemetry, active telemetry, and the reporting process, along with the time constraints of inter-satellite transmission time and available SGLs intervals, an APINT model is proposed. The telemetry and reporting process of APINT is illustrated in Figure 2.

When the service starts its transmission, the INT packet is placed in the header of the service packet for passive telemetry. At the same time, the target node sends active telemetry packets in the reverse direction along the service transmission path, towards the source node. During the transmission process, each node can perform metadata writing operations. Then, the INT packets and active telemetry packets converge at a certain node along the path, and the awareness information is integrated on the satellite and transmitted back to the ground along the SGL, while the service continues to transmit along the path to the target node. To avoid impacting the normal transmission of service, the arrival time of active telemetry packets sent in the reverse direction at the aggregation node should be earlier than the arrival time of the service, which means the selection of aggregation nodes on the awareness path is subject to the time constraints, as follows:

$${\sum }_{{i=S}_a}^{S_r-1}{t}_{\mathrm{op}}^i>{\sum }_{{j=S}_r}^{S_b-1}{t}_{\mathrm{ap}}^j,$$

(6)

where $S_r$ is the aggregation node, with $S_r \in { P}_z$. $S_a$ is the source node and $S_b$ is the target node. Multiple nodes may satisfy the above constraints, and these nodes are all potential selectable reporting nodes, denoted as $S_R=\{S_{r1},S_{r2},\dots ,S_{rP}\}$. The time for the awareness data arrives at the potential reporting nodes is:

$$T_R^{S_{rp} }={\sum }_{{i=S}_a}^{S_{rp}-1}{t}_{\mathrm{op}}^i, i\in P_z,S_{rp} \in S_R.$$

(7)

However, due to the dynamic and unstable characteristics of the SGLs within satellite networks, only the potential reporting nodes capable of establishing stable SGLs are able to report awareness data. Furthermore, by combining the available SGL intervals of potential reporting nodes, it is possible to determine whether the awareness data can be reported. In this case, three situations exist:

(1)

If the awareness data arrive at the potential reporting node before the establishment of the SGL, the awareness data must wait on the satellite until the SGL is successfully established, then the data can be transmitted back to the ground.

(2)

If the awareness data arrive at the potential reporting node within the visible link time range of the SGL, and there are no issues of link disconnection during the transmission period, the awareness data can be directly transmitted back to the ground.

(3)

If the awareness data arrives when the SGL is already disconnected, the awareness data must wait on the satellite until the next available SGL is established for transmission [21], which results in significant reporting delays.

In summary, the time taken for the awareness data to ultimately reach the gateway station is:

$$T_{\mathrm{final}}^{S_{rp}}=\left\{\begin{array}{c}{t}_s^{S_{rp}}+t_{\mathrm{re}}^{S_{rp}}, \mathrm{if} T_R^{S_{rp}}< t_s^{S_{rp}}\\ t_{\mathrm{re}}^{S_{rp}}+T_R^{S_{rp}}, \mathrm{if} t_s^{S_{rp}}<{ T}_R^{S_{rp}}<t_e^{S_{rp}}\\ t_{e+1}^{S_{rp}}+t_{\mathrm{re}}^i,{\mathrm{if} T}_{\mathrm{R}}^{{\mathrm{S}}_{\mathrm{r}p}}>t_e^{{\mathrm{S}}_{\mathrm{r}p}}\end{array}\right., \forall {\mathrm{S}}_{\mathrm{r}p}\in {\mathrm{S}}_{\mathrm{R}},$$

(8)

where $t_{\mathrm{re}}^{{\mathrm{S}}_{\mathrm{r}p}}$ is the potential satellite-ground transmission time for potential reporting nodes calculated based on Equation (5). $t_{\mathrm{e}+1}^{{\mathrm{S}}_{\mathrm{r}p}}$ represents the time for the next available SGL establishment for the potential reporting node. For each potential reporting node, the awareness time is calculated and the node with the minimum value is selected as the optimal reporting node. The target optimization function is as follows:

$$\min T_{\mathrm{final}}^{{\mathrm{S}}_{\mathrm{r}p}},{\forall \mathrm{S}}_{\mathrm{r}p}\in { \mathrm{S}}_{\mathrm{R}}$$

(9)

$$\begin{gathered} \mathrm{s}.\mathrm{t}. \mathrm{C}1 \mathrm{Equation} \left(6\right) \\ \mathrm{C}2 \mathrm{Equation} \left(7\right) \\ \mathrm{C}3 \mathrm{Equation} \left(8\right), \end{gathered}$$

where $C1$ is the transmission time constraints, and potential available reporting nodes on the awareness path can be discovered. $C2$ and $C3$ are the awareness time for each potential reporting node.

3. Dynamic Multi-Reporting Nodes Selection Method

In LEO satellite networks, the length of the end-to-end service path is relatively long, and the single-node reporting method can lead to significant awareness delay and overhead due to multi-hop accumulation. Therefore, multiple reporting nodes need to be selected for segmented reporting. When the INT method is applied to terrestrial networks, a fixed hop interval approach is commonly employed to select multiple reporting nodes, due to the relative stability of terrestrial network nodes and links. However, when applied to satellite networks, such methods may result in significant cumulative awareness delays due to unstable ISLs and may also prevent timely reporting of awareness data due to the intermittency of SGLs. In this section, a dynamic multi-reporting nodes selection method of network awareness based on the APINT model is proposed, which can select the optimal number and location of reporting nodes for the awareness path.

The procedure for selecting multi-reporting nodes is illustrated in Figure 3. The selection of the optimal reporting node requires consideration of both the number of reporting nodes and the length of their respective intervals, as both factors jointly influence the telemetry time of the awareness path and the specific location of the reporting node. The issue of selecting multiple reporting nodes can be converted into an interval partitioning optimization problem.

Therefore, the concept of virtual boundaries is introduced, dividing the multi-hop path ${\mathrm{P}}_z$ into $Y$ segments, with the set of virtual boundaries for all segments represented as $V=\{V_1,V_2,\dots ,V_Y\}$, where the number of satellites in each segment is $N_{\mathrm{V}}=\{N_{\mathrm{V}1},N_{\mathrm{V}2},\dots ,N_{\mathrm{V}Y}\}$. Using each segment as the optimization object, the best reporting node is searched for within each segment based on the APINT model. Specifically, upon reaching the virtual boundary starting point of the segment, the service initiates passive telemetry along the route. Conversely, upon reaching the virtual boundary endpoint, it sends active telemetry packets, which are aggregated and reported at the optimal reporting node within the segment. Then the service proceeds to transmit and enter the subsequent segment, repeating the process until it arrives at the target node.

However, the division of different virtual boundaries results in different reporting node selection results, necessitating the selection of the optimal solution from numerous results. Therefore, by considering the awareness time within segments, awareness overhead of segments, and the number of reporting nodes, an awareness evaluation function is constructed. The optimal telemetry and reporting solution for the path is determined by the minimum number of reporting nodes and their respective locations, as indicated by the awareness evaluation function across various scenarios involving different virtual boundary segmentations. The optimization objective function is:

$$\min \mathrm{Eva}=\beta ·\mathrm{F}(Y)+\delta ·\mathrm{F}(\overline{{\mathrm{T}}_{\mathrm{V}}^{\mathrm{Z}}})+\tau ·\mathrm{F}({\mathrm{E}}_{\mathrm{V}i}),$$

(10)

where $\beta$, $\delta$, and $\tau$ are the corresponding optimization weights, with $\beta +\delta +\tau =1$. The function of $\mathrm{F}$ is the normalization function. $\overline{{\mathrm{T}}_{\mathrm{V}}^{\mathrm{Z}}}$ is the average awareness time of segments. $\overline{{\mathrm{E}}_{\mathrm{Vi}}}$ is the average awareness overhead of segments. The specific calculation is as follows.

The telemetry and reporting time for each segment after selecting the optimal reporting node based on the APINT model are ${\mathrm{T}}_{\mathrm{V}}=\{{\mathrm{t}}_{\mathrm{V}1},{\mathrm{t}}_{\mathrm{V}2},\dots ,{\mathrm{t}}_{\mathrm{V}Y}\}$. Then, the average telemetry and reporting time for the segments along the entire path is:

$$\overline{{\mathrm{T}}_{\mathrm{V}}^{\mathrm{Z}}}={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^Y}{\mathrm{t}}_{\mathrm{V}i}}{Y}}.$$

(11)

Meanwhile, the identifiers of the nodes within each segment are denoted as ${\mathrm{S}}_{\mathrm{V}i}=\{{\mathrm{S}}_{\mathrm{V}i}^0,{\mathrm{S}}_{\mathrm{V}i}^1,\dots ,{\mathrm{S}}_{\mathrm{V}i}^{{\mathrm{N}}_{\mathrm{V}i}}\}$, and the awareness overhead generated by passive and active telemetry within the segment is as follows:

$${\mathrm{E}}_{\mathrm{V}i}={\sum }_{{\mathrm{S}}_{\mathrm{V}i}^0}^{{\mathrm{S}}_{\mathrm{V}i}^{\mathrm{r}p}-1}{\mathrm{B}}_{\mathrm{INT}}+{\sum }_{{j=\mathrm{S}}_{\mathrm{V}i}^1}^{{\mathrm{S}}_{\mathrm{V}i}^{\mathrm{r}p}-1}{\mathrm{B}}_{\mathrm{M}}^j+{\sum }_{{\mathrm{S}}_{\mathrm{V}i}^{\mathrm{r}p}}^{{\mathrm{S}}_{\mathrm{V}i}^{{\mathrm{N}}_{\mathrm{V}i}}-1}{\mathrm{B}}_{\mathrm{ac}}+{\sum }_{k={\mathrm{S}}_{\mathrm{V}i}^{\mathrm{r}p}}^{{\mathrm{S}}_{\mathrm{Vi}}^{{\mathrm{N}}_{\mathrm{V}i}}-1}{\mathrm{B}}_{\mathrm{M}}^k,$$

(12)

where ${\mathrm{B}}_{\mathrm{INT}}$ and ${\mathrm{B}}_{\mathrm{ac}}$, respectively, represent the lengths of the passive telemetry header and the primary telemetry header. ${\mathrm{B}}_{\mathrm{M}}^{{\mathrm{S}}_{\mathrm{V}i}^x}$ represents the awareness overhead generated during the metadata’s transmission process of node ${\mathrm{S}}_{\mathrm{V}i}^x$ within the segment, and its calculation equation is as follows:

$${\mathrm{B}}_{\mathrm{M}}^{{\mathrm{S}}_{\mathrm{V}i}^x}={\mathrm{Meta}}_{{\mathrm{S}}_{\mathrm{V}i}^x}·{\mathrm{H}}_{{\mathrm{S}}_{\mathrm{V}i}^x},$$

(13)

where ${\mathrm{Meta}}_{{\mathrm{S}}_{\mathrm{V}i}^x}$ is the length of the metadata generated by node ${\mathrm{S}}_{\mathrm{V}i}^x$, and ${\mathrm{H}}_{{\mathrm{S}}_{\mathrm{V}i}^x}$ represents the number of hops required for node ${\mathrm{S}}_{\mathrm{V}i}^x$ to reach the reporting node.

4. A Multi-Reporting Nodes Selection Algorithm Based on Lightweight Q-Learning for Various Services

The dynamic selection method for multiple reporting nodes can determine the optimal reporting solution for any service path. However, differences in path hops, distances of ISLs, and available statuses of SGLs among various services further enhance the dynamism and complexity of multi-reporting nodes selection in concurrent multi-service scenarios. Therefore, it is necessary to develop a dynamic solving algorithm to select the optimal reporting solution for various service scenarios. The difficulty of this problem lies in not only adaptively adjusting to the unique characteristics of different service paths, but also reducing the computational complexity of solving the reporting solution within each service path.

Traditional mathematical solving algorithms, such as dynamic programming and linear programming methods, rely on precise environmental models. For multi-service scenarios, such methods necessitate continuous re-modeling and are incapable of facilitating dynamic strategy updates. Machine learning and other artificial intelligence algorithms require substantial data support and lack proactive exploration mechanisms. Q-learning is a model-free reinforcement learning algorithm based on finite Markov decision processes that learns directly from interactions with the environment of uncertain or unknown settings [22]. It is particularly suitable for solving dynamic and complex problems and supports the timeliness of the decision-making process.

In response to the above problems, the Q-learning algorithm is introduced to dynamically solve the dynamic selection problem of multiple reporting nodes in multiple service scenarios. However, during the dynamic partitioning of virtual boundaries, the Q-learning algorithm may encounter the problem of non-adjacent virtual boundary combinations, meaning that the virtual boundaries are discontinuous and the segments fail to achieve comprehensive coverage of nodes along the path. The Q-learning-based dynamic optimization algorithm traverses the effects of all potential actions $\mathrm{A}$ on state $\mathrm{W}$ and generates the corresponding $\mathrm{Q}$-value table. As the number of virtual segments grows, the scope of search and calculation exhibits an exponential expansion, posing a significant challenge to the limited computational capabilities of satellite.

A Q-learning algorithm based on virtual boundary preplanning is proposed, which can optimize the Q-learning search process and reduce the number of iterations. The algorithm flow is shown in Figure 4. Specifically, considering that the flexible selection method for multiple reporting nodes requires awareness of all nodes, and the APINT model needs to start telemetry from the starting points of the virtual boundaries of the segments separately, the virtual boundaries of the sub segments must be adjacent, with no nodes present between them. Meanwhile, the APINT model can expand the awareness range, and to enhance the average telemetry duration of segments, the lengths of these segments are relatively similar. Thus, the equal division method for virtual boundary initialization division is used. The number of satellites within the path is $N_{\mathrm{p}z}$, and the satellite number of each segment is $N_{\mathrm{V}}=\{N_{\mathrm{V}1},N_{\mathrm{V}2},\dots ,N_{\mathrm{V}Y}\}$, with $N_{\mathrm{V}i}\simeq {\displaystyle \frac{N_{\mathrm{p}z}}{Y}}$. Then, starting from the end of the first segment and the starting point of the second segment, the boundary nodes float back and forth within a certain range $∆={\displaystyle \frac{N_{\mathrm{V}i}}{Y+x}}$, where $\mathrm{x}$ is the range adjustment factor, which can dynamically adjust the virtual boundary. The process is repeated, and each segment boundary can float based on $\mathsf{∆}$. Considering that the number of nodes within the segment after dynamic adjustment is:

$${\mathrm{N}}_{\mathrm{Vi}}^{\prime }={\displaystyle \frac{{\mathrm{N}}_{\mathrm{pz}}}{\mathrm{Y}}}+∆={\displaystyle \frac{{\mathrm{N}}_{\mathrm{pz}}}{\mathrm{Y}}}+{\displaystyle \frac{{\mathrm{N}}_{\mathrm{Vi}}}{\mathrm{Y}+\mathrm{x}}}={\displaystyle \frac{{\mathrm{N}}_{\mathrm{pz}}* (\mathrm{Y}+\mathrm{x}+1)}{\mathrm{Y}(\mathrm{Y}+\mathrm{x})}},$$

(14)

when $\mathrm{Y}$ is fixed, ${\displaystyle \frac{\mathrm{d}({\mathrm{N}}_{\mathrm{Vi}}^{\prime })}{\mathrm{dx}}}=-{\displaystyle \frac{1}{\mathrm{Y}{(\mathrm{Y}+\mathrm{x})}^2}}$, which means ${\mathrm{N}}_{\mathrm{Vi}}^{\prime }$ is monotonically decreasing with respect to $\mathrm{x}$. Therefore, to ensure the search scope and maximize both the floating range and the number of segment nodes, the floating range can be set as follows:

$$\mathit{∆}\le {\displaystyle \frac{{\mathrm{N}}_{\mathrm{V}y}}{Y+1}}.$$

(15)

At the same time, by dynamically adjusting the number of reporting nodes $\mathrm{Y}$, repeating the virtual boundary partitioning and dynamic floating process mentioned above, the optimization of the Q-learning search process can be achieved. Taking the division of the virtual boundary as state $W$. The increase or decrease in the number of segments and the changes to the virtual boundaries are regarded as the action space $A$. The reward function based on the awareness evaluation function can be constructed as:

$$\mathrm{R}\left(w,a\right)=-[\mathrm{Ev}\mathrm{a}\left(w^{\prime }\right)+\mu ·w^{\prime }],$$

(16)

where $\mathrm{Ev}\left(w^{\prime }\right)$ is the awareness evaluation function under the new state $w^{\prime }$, where $\mu$ is the regularization penalty coefficient, and $w^{\prime }$ represents the number of segments and the division of virtual boundaries in the new state. The $\mathrm{Q}$-values are updated using $\mathrm{R}$, and the update formula is as follows:

$$\mathrm{Q}\left(w,a\right)\leftarrow \mathrm{Q}\left(w,a\right)+\mathsf{\alpha }\left[r+\mathsf{\gamma }\underset{{\mathrm{a}}^{\prime }}{\max }\mathrm{Q}\left(w^{\prime },a^{\prime }\right)-\mathrm{Q}\left(w,a\right)\right],$$

(17)

where $\alpha$ is the learning rate; $\gamma$ is the update rate, and $0\le \alpha \le 1$, $0\le \gamma \le 1$. The pseudocode of the algorithm is presented in Algorithm 1:

Algorithm 1: Dynamic selection of multiple reporting node based on the improved Q-learning

Input: Number of services, $N_z$; transmission path of each service, ${\mathrm{P}}_z$; maximum number of segments for each service, $Y_z$; 1   for i $\leftarrow$ 1 to $N_z$ do 2     for j $\leftarrow$ 1 to $Y_z$ do (adjust the number of segments) 3        Divide ${\mathrm{P}}_i$ equally into segments with virtual boundary $\mathrm{V}=\{{\mathrm{V}}_1,{\mathrm{V}}_2,\dots ,{\mathrm{V}}_{Y_j}\}$;

4        Calculate floating range $\mathsf{∆}$ with Equation (15), and the number of potential combinations of virtual boundary is $N_{\mathsf{∆}}={(2\mathsf{∆}+1)}^{Y_j}$; 5        for k $\leftarrow$ 1 to $N_{\mathsf{∆}}$ do 6         Adjust the virtual boundary $\mathrm{V}=\{{\mathrm{V}}_1,{\mathrm{V}}_2,\dots ,{\mathrm{V}}_{{\mathrm{Y}}_j}\}$ with $∆$ to generate new ${\mathrm{V}}^{\prime }=\{{\mathrm{V}}_1+∆,{\mathrm{V}}_2-∆,\dots ,{\mathrm{V}}_{{\mathrm{Y}}_j}\}$, which is the state $W;$ 7           for h $\leftarrow$ 1 to $Y_j$ do (using APINT model for each segment) 8             Find potential reporting nodes within segment ${\mathrm{V}}_h$ using Equation (6); 9             Identify the optimal reporting nodes in subsegment ${\mathrm{V}}_h$ based on Equations (8) and (9), which is the action $A$; 10            Calculate telemetry and reporting time, as well as awareness overhead within subsegment; 11         end 12         Calculate the awareness evaluation function as the reward $R$ based on Equation (10); 13         Update the Q value table by using Equation (17); 14       end 15       Find the best virtual boundary division range and the best reporting node location; 16    end 17    Compare the maximum Q-values associated with the optimal virtual boundary divisions in scenarios with different number of segments; 18    Output the optimal reporting nodes’ number, location, awareness time, awareness overhead of current service; 19 end Output: Generate optimal reporting node selections for various services.

5. Simulation Results and Analysis

In this section, the proposed DRNSM-APINT method will be evaluated through simulation methods. A network of 1584 satellites has been simulated, with the structure similar to the first shell of Starlink, at an average orbital altitude of 550 km. There are 72 orbital planes, each containing 22 satellites. The network comprises 300 traffic services, with an average path length of 20 hops. The telemetry packet is placed at the head of the service packet and the telemetry header has a fixed size of 32 bytes. The maximum cumulative hop for telemetry data is 8 hops. Each node can insert 4 to 8 bytes of metadata, with an insertion time of 0.01 ms, and the propagation time for the active telemetry packet is uniformly set to 0.08 ms.

To the best of our knowledge, limited studies have been conducted on satellite network measurement and selection of reporting nodes. Therefore, this paper adopts the latest methods in terrestrial network awareness research for application in satellite network scenarios, and compares them with the methods presented in this paper.

(1)

Flexible and active network telemetry scheme (FAINT): telemetry packets can be actively gathered and measured from nodes along the path, as specified, through the utilization of programmable technology [23].

(2)

INT method for segmented routing (SRINT): hop-by-hop awareness of each node along the path utilizing the INT method [20].

(3)

Energy-efficient data aggregation method (EEDA): the nodes are selected where there is relatively high remaining of bandwidth on the SGLs for reporting [24].

Figure 5 shows a comparison of the awareness time among the FAINT, SRINT, EEDA, and APINT model at different length of segments. From the figure, it can be seen that due to the FAINT method initiating active telemetry from the reporting node and requiring round-trip collection, the awareness time exhibits a notable increase as the number of hops grows, and is significantly higher than that of the other methods. The SRINT method employs INT to reduce the awareness time, but the awareness time remains relatively higher under the mechanism of target node reporting. The EEDA method selects nodes with abundant resources for reporting, thereby sacrificing a portion of the awareness time. The awareness time of the APINT method is notably shorter than that of the other comparative methods, and the increase in awareness time remains steady as the hop increases. When the hop length is 9, the awareness time of the APINT model is reduced by 34.5% compared to FAINT, 13.7% compared to SRINT, and 8.4% compared to the EEDA method, indicating a significant improvement in awareness timeliness.

Figure 6 illustrates the comparison of awareness evaluation function values (EFV) and the awareness overhead under varying numbers of reporting nodes based on the DRNSM-APINT method. As can be seen from Figure 6a, when the path length is less than 10, the EFV corresponding to a single reporting node scenario is smaller. The DRNSM-APINT will prioritize selecting the optimal single reporting node for reporting. Figure 6b illustrates that within the path length interval of 11 to 16, the EFV is minimal in the scenario with two reporting nodes, followed by the scenario with three reporting nodes. And the EFV under single reporting node scenario is the highest. The DRNSM-APINT will prioritize selecting two reporting nodes. From Figure 6c, it can be observed that when the length of path further increases from 18 to 30, the DRNSM-APINT will give priority to deploying three reporting nodes. The corresponding EFV is smaller compared to scenarios where there are two or four reporting nodes. To sum up, the DRNSM-APINT can flexibly and dynamically adjust the number of reporting nodes based on varying service path lengths, and determine the optimal location of the reporting nodes in corresponding scenarios.

Figure 6d shows the awareness overhead associated with various reporting node scenarios across different path lengths. The graph reveals that an increase in the number of reporting nodes leads to a reduction in segment length, subsequently decreasing the awareness overhead caused by accumulation of awareness data within the segments. However, it is noticeable that when there are three or four reporting nodes, the difference in awareness overhead between the two scenarios is small, indicating that the same awareness effect can be achieved by reducing the number of reporting nodes.

Figure 7 shows the performance comparison of the optimal reporting scheme selected by the DRNSM-APINT and other different reporting node selection algorithms. The FHINT method, based on the official INT specification [11], conducts periodic reporting after a fixed number of hops. The SRINT and EEDA method are the same as before.

Figure 7a presents a comparison of EFV across four methods. Specifically, the DRNSM-APINT algorithm selects varying numbers of reporting nodes across different paths. Therefore, the EFV values presented in the figure represent the optimal solutions achieved by the DRNSM-APINT algorithm in various scenarios. From the figure, it can be seen that the FHINT method has the highest EFV, while the SRINT and EEDA methods have smaller EFV. The EFV of DRNSM-APINT algorithm is lower than that of the comparison algorithm under various path scenarios. Taking the 20 hops as an example, DRNSM-APINT can reduce EFV by 17.1% compared to FHINT, 12.3% compared to SRINT, and 8.2% compared to EEDA. This indicates that the DRNSM-APINT method can maintain better awareness performance across different path scenarios.

Figure 7b compares the average awareness time of segments obtained through different methods. It can be seen that the awareness time reported periodically by the FHINT method is generally large, but the fluctuation is small. The SRINT method is limited by the size of the onboard awareness data, resulting in differences in the number of service hops and significant fluctuations in awareness time. EEDA aims to maximize reporting capability, which may result in an increase in some segment links, thereby increasing awareness time. The awareness time of DRNSM-APINT is significantly lower than other compared methods. Taking the 20 hops as an example, DRNSM-APINT reduces awareness time by 17.4% compared to FHINT, 12.4% compared to SRINT, and 13% compared to EEDA, indicating that the proposed method can significantly reduce awareness time. Meanwhile, the DRNSM-APINT has relatively small time fluctuations, indicating that the algorithm has good robustness in complex dynamic scenes.

Figure 7c presents a comparison of the awareness overhead associated with various methods. From the figure, there is no significant difference among the four methods when the service path is short. However, as the path length increases, the overhead of the FHINT method gradually increases, while the overhead of the SRINT and DRNSM-APINT algorithms is relatively close, and the EEDA method is the smallest. This is because DRNSM-APINT adopts active-passive joint telemetry method, which introduces additional telemetry packet headers and increases the awareness overhead. EEDA sacrifices some awareness time to achieve low overhead measurement and reporting, while DRNSM-APINT still has lower overhead than the other two methods.

Figure 7d compares the total number of reporting nodes selected by the four methods across varying numbers of services. From the figure, it can be seen that the number of reporting nodes of EEDA is the largest. This is due to the fact that the method strives to select as many idle nodes with abundant resources as possible for reporting, leading to a substantial and fluctuating number of reporting nodes. The number of reporting nodes for FHINT and SRINT methods is relatively small. The DRNSM-APINT can select fewer reporting nodes than the other comparison methods. When the number of services is 300, the total number of reporting nodes in DRNSM-APINT decreases by 34% compared to FHINT, 20.1% compared to SRINT, and 42.8% compared to EEDA, which can significantly optimize the number of reporting nodes.

Figure 8 presents a comparison of the number of combinations of virtual segment divisions using various methods. The combinations of virtual segment divisions affect the calculation complexity of DRNSM-APINT. The traditional solving algorithm traverses all possible segment partitions along the path, encompassing scenarios where segments are discontinuous. Therefore, with the growth of the path, the number of partition combinations grows exponentially. In this case, the number of combinations can be calculated as the second type of Stirling number [25], and the calculation equation is $Y!·\mathrm{S} ({\mathrm{N}}_{\mathrm{pz}},Y)$, where $N_{\mathrm{p}z}$ is the number of satellites and $Y$ is the number of reporting nodes. The Q-learning algorithm also traverses all the possible outcomes, but it can optimize the problem into a combinatorial planning task of continuous segments division based on the reward function. The calculation formula of combination quantity is $({\mathrm{N}}_{\mathrm{pz}}-1)!/((Y-1)!·({\mathrm{N}}_{\mathrm{pz}}-Y)!)$. The dynamic partitioning of continuous segments division is conducted using a heuristic algorithm, resulting in a further reduction of the number of combinations to ${\mathrm{N}}_{\mathrm{pz}}·Y$, but it is easy to getting trapped in local optimal solutions. The lightweight Q-Learning algorithm based on virtual boundary preplanning that proposed in this paper, combined with the APINT model’s feature of being less influenced by the segment range in selecting reporting nodes, can markedly decrease the number of combinations. Compared with Q-Learning and heuristic algorithm, this method can reduce the number of combinations by one or two orders of magnitude. Meanwhile, considering that the figure only illustrates the calculation complexity involved in selecting reporting nodes within a single service path, the algorithm proposed in this paper demonstrates significant lightweight capabilities in massive service scenarios, alleviating the computational burden in large-scale satellite constellation.

6. Conclusions

In this paper, to address the problems of poor timeliness awareness and limited flexibility in selecting reporting nodes within LEO satellite networks, a dynamic reporting nodes selection method of network awareness based on active-passive integrated network telemetry is proposed. This method combines active and passive telemetry on the awareness path and constructs an APINT model to select the optimal reporting node based on the constraints of the inter satellite transmission time and status of SGLs. Furthermore, the awareness path is divided into multi segments, and the APINT model is used to select the optimal reporting node for each segment, achieving flexible selection of multiple reporting nodes. To enable dynamic selection of reporting nodes across various service paths in multiple service scenarios, a lightweight Q-learning algorithm is proposed. Through the pre-planning of virtual boundaries, the computational complexity of Q-learning is reduced, and the optimal number of reporting nodes on each service path is dynamically determined. Through simulation, it was found that the proposed method has good performance and can be flexibly applied to the multi-service scenarios in LEO satellite networks.