AoI Minimization Scheduling Using Integrated Collection-Relay in Multi-AUV Multi-Hop Underwater Networks

Abstract

Underwater surveillance systems require a multi-AUV-based multi-hop underwater data collection system to deliver data from sensor nodes distributed over a wide underwater area to a buoy. In this system, the node-visit of the data collection AUV and the relay transmission to the relay AUV jointly affect the end-to-end Age of Information (AoI) from each node to the buoy. This paper proposes a discrete soft actor–critic (SAC)-based integrated scheduling policy that jointly determines the node-visit and relay transmission of the data collection AUV within a single decision-making process. The proposed method represents the end-to-end information update process from data collection to buoy update in the state design and derives a relationship showing that the accumulated node-wise max AoI increment corresponds to the Mean Peak AoI and uses this increment as the decision step reward. Computer simulation results show that the proposed method achieves a lower Mean Peak AoI and higher Delivery than conventional methods that determine node visits and relay transmissions separately, including Visit-Only SAC, Age-Greedy threshold, and Age-Gain TX. The cumulative distribution function (CDF) analysis also shows more stable performance across episodes.

1. Introduction

1.1. Background and Motivation

Underwater surveillance systems observe various events occurring over wide underwater areas, such as coastal surveillance, underwater target detection, and marine environmental monitoring. Sensor nodes distributed across the underwater area detect information related to the surrounding environment or the approach of external objects, and the detected information is delivered to the final monitoring system through a buoy.

Data transmission in underwater environments mainly relies on acoustic communication. However, underwater acoustic communication has a lower data rate and more severe channel variation than terrestrial communication, and the transmission power of sensor nodes is also limited. Thus, it is difficult to directly deliver data generated by underwater sensor nodes to a buoy. To address these constraints, AUV-based data collection methods have been considered. An AUV can collect data from sensor nodes while moving through the surveillance area, making it useful in wide-area underwater surveillance environments. However, when the surveillance area is wide and sensor nodes are spatially distributed, if a single AUV performs both data collection and delivery to the buoy, information collection can be delayed due to travel time. Then, a multi-AUV-based multi-hop underwater data collection system is used, which includes a data collection AUV responsible for data collection and a relay AUV that supports data delivery to the buoy.

Each sensor node irregularly generates environmental sensing data; therefore, new data may be generated immediately after the data collection AUV visits a specific node. In this case, the data cannot be collected until the next visit of the data collection AUV, and the AoI can significantly increase [1]. For important environmental variations or target detection, it is important to deliver the data to the buoy with a low AoI. For this purpose, the order in which the data collection AUV visits nodes for data collection and the timing at which the collected data are transmitted to the relay AUV under suitable communication conditions affect the AoI. If the node-visit order and relay transmission are optimized independently, the AoI cannot be properly optimized from an end-to-end perspective. Therefore, to reduce the end-to-end AoI from the nodes to the buoy, a scheduling policy that jointly considers node visits and relay transmission of the AUV is required.

1.2. Related Works

AoI-related scheduling studies can be broadly divided into studies that optimize mobile-collector- or AUV-based node visits and studies that optimize the information delivery process mainly through relay communication. Yi et al. [2] proposed a Deep Reinforcement Learning (DRL)-based method to reduce AoI by optimizing the movement path of a UAV and sensor selection, and Al-Habob et al. [3] optimized the data collection location and residence time of an AUV in an underwater environment. Bu et al. [4] optimized the trajectory and communication mode of an AUV-based on DRL. Fan et al. [5] jointly optimized the UAV trajectory, IoT device scheduling, and reconfigurable intelligent surface (RIS) phase shifts based on SAC to minimize AoI in a three-dimensional urban environment. This research has primarily focused on optimizing data collection routes or mobility paths, while the process of delivering collected data to surface buoys via relay transmission is handled through separate procedures. Therefore, the AUV might visit a node when the relay-link is favorable, or attempt a transmission when it is poor, consequently delaying the AoI update. In addition, Kadota et al. [6] analyzed a scheduling policy that prioritizes sources with large AoI in wireless networks, and Choudhury et al. [7] proposed a relay scheduling policy for AoI minimization in a two-hop UAV relay network. Zakeri et al. [8] addressed a transmission scheduling policy in a multi-source relaying environment with unreliable links. De Jesus et al. [9] proposed an AoI-aware multiple-access and relay transmission scheme to prioritize the delivery of stale information in a multi-relay-based two-hop IoT network.

This research mainly focuses on either data collection or relay transmission scheduling. They do not jointly consider node visits and relay transmission within a single scheduling process. However, when the data collection AUV performs both node visits and relay transmissions, the two actions are mutually exclusive because they compete for the same time resource. In addition, the two actions have different durations, so simply combining them into a single RL action space does not sufficiently reflect the temporal coupling among data collection, buffer waiting, relay transmission, and buoy update. Therefore, an integrated scheduling policy is required in which the data collection AUV jointly considers the different durations and buffers of node visits and relay transmissions within a single decision-making process.

To solve this problem, a DRL structure is required that can efficiently reuse the transitions collected during training while maintaining exploration over various scheduling actions. DQN [10] uses Q-value-based ε-greedy exploration, so it does not directly learn a state-dependent stochastic scheduling policy. Its exploration level also depends on a predefined ε schedule, making it difficult to adaptively adjust exploration when the favorable action varies with the time-varying relay-link condition. PPO [11] directly learns a stochastic policy, but it uses on-policy updates, so the node-visit and relay transmission transitions with different durations are discarded after each update instead of being accumulated in a replay buffer for reuse. In contrast, SAC combines a maximum-entropy stochastic policy with off-policy experience replay, thereby maintaining exploration among scheduling actions while repeatedly reusing past transitions [12,13]. Based on these characteristics, this paper learns the integrated scheduling of the multi-AUV multi-hop environment through discrete SAC over a discrete action set composed of node-visit and relay transmission actions.

1.3. Contributions

This paper designs a decision step reward for Mean Peak AoI minimization and proposes an integrated collection-relay scheduling policy in a multi-AUV multi-hop data collection environment. The contributions of the proposed method are as follows.

• This paper proposes a discrete SAC-based integrated scheduling policy that jointly determines the node-visit and relay transmission of the data collection AUV within a single decision-making process to minimize the end-to-end Mean Peak AoI in a multi-AUV multi-hop data collection environment.
• This paper derives a relationship showing that the accumulated node-wise max AoI increment corresponds to the Mean Peak AoI and uses the node-wise max AoI increment as the decision step reward in the proposed discrete SAC framework.
• Computer simulation results over five heterogeneous data-generation scenarios show that the proposed method achieves the lowest Mean Peak AoI and the highest Delivery compared with conventional methods that determine node visits and relay transmissions separately, including Visit-Only SAC, Age-Greedy threshold, and Age-Gain TX.

2. Proposed Integrated Collection-Relay Scheduling for AoI Minimization

This section presents the proposed AoI minimization scheduling using integrated collection-relay in multi-AUV multi-hop underwater networks. To this end, the multi-AUV-based multi-hop underwater data collection environment is described, and Mean Peak AoI and decision time are defined. The state and reward of the proposed scheduling are designed to represent the conditions of AUV1 that affect AoI. The learning process is described in which AUV1 selects either a node-visit or relay transmission action at each decision step through a discrete SAC-based actor–critic structure with action masking and learns a policy that reduces the Mean Peak AoI of the overall system.

2.1. Multi-AUV Multi-Hop Underwater Data Collection Environment

The underwater data collection environment considered in this paper is a multi-AUV-based multi-hop underwater data collection system composed of distributed sensor nodes, AUV1 for data collection, AUV2 for supporting relay transmission, and a buoy. The sensor nodes are deployed at fixed positions within the surveillance area and randomly generate data at different frequencies. AUV1 moves through the surveillance area, collects data from the sensor nodes and stores the collected data in the AUV1 buffer. The AUV1 buffer is modeled as a latest-data buffer that stores only the most recent data for each sensor node. Thus, if AUV1 collects new data from a specific node before transmission, the existing data are replaced with new data. In addition, the new data are transmitted as a single frame. Then, AUV1 transmits the data to AUV2, which periodically moves around the buoy and delivers the data to the buoy. If a relay transmission fails, the corresponding data are retained in the buffer and can be retransmitted through a subsequent relay action. Since AUV2 moves along a circular trajectory while maintaining a constant distance from the buoy, it is assumed to maintain stable communication performance with the buoy. The system structure is shown in Figure 1.

Figure 1. Multi-AUV-based multi-hop underwater data collection system.

The relay transmission success probability, Mean Peak AoI and decision time considered in this paper are defined as follows.

• Relay transmission success probability

The relay-link between AUV1 and AUV2 is modeled as a time-varying relay transmission success probability, which is determined from the received SNR between the two AUVs. Let the received SNR of the AUV1–AUV2 link at time t be denoted by ${\gamma }_t$. Then, the relay transmission success probability is defined by the following sigmoid function [14]. The success or failure of each relay transmission is determined according to $p_{succ}\left(t\right)$.

$$p_{succ}\left(t\right)={\displaystyle \frac{1}{1+\mathit{exp}\left[-\frac{4}{{\mathsf{\Delta }}_{\gamma }}\left(\gamma \left(t\right)-{\gamma }_{\mathrm{t}\mathrm{h}}\right)\right]}}$$

(1)

${\gamma }_{\mathrm{t}\mathrm{h}}$ denotes the SNR threshold at which the relay success probability becomes 0.5, and $\mathsf{\Delta }\gamma$ denotes the margin width that controls the variation range of the sigmoid function with respect to the SNR variation.

• Mean Peak AoI

The AoI of node $n$, $A_n\left(t\right)$ is defined as the time interval from data generation at the node to its arrival at the buoy [1]. The Peak AoI is defined as the AoI value observed immediately before new data are reflected at the buoy [15].

To evaluate the maximum AoI of a specific node over an arbitrarily long observation time, this paper defines the observation-time Peak AoI $A_n^{peak}$ of node n as the maximum value of $A_n\left(t\right)$ over the observation time $T$.

$$A_n^{\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}=\underset{0\le t\le T}{\max }{A}_n\left(t\right)$$

(2)

The Mean Peak AoI ${\overline{A}}^{peak}$ is defined as the average of the observation-time Peak AoI of each node over the total number of nodes N.

$${\overline{A}}^{\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{\mathrm{n}=0}^{N-1}}{A}_n^{\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}$$

(3)

• Decision time

When the two actions are integrated into a single scheduling process, the node-visit action and the relay transmission action require different durations. Learning is performed by determining the next action whenever the selected action is completed, based on the AoI of each node, buffer state, and relay-link condition, which vary according to the elapsed duration. If the start time of the $k$-th decision step is denoted by ${\tau }_k$ and the duration of the selected action is denoted by ${\delta }_k$, the start time of the next decision step can be calculated as follows:

$${\tau }_{k+1}={\tau }_k+{\delta }_k$$

(4)

the decision step index is $k=1,\dots ,K$, and $K$ is the index of the last decision step within the observation time, with ${\tau }_{K+1}=T$.

2.2. AoI Minimization Scheduling Using Integrated Collection-Relay

In the underwater surveillance system in Figure 1, data collected from sensor nodes need to be delivered to the buoy with low delay to maintain information freshness. To reduce the AoI from the nodes to the buoy, AUV1 must jointly consider data collection and relay transmission. This requires AUV1 to visit nodes with high data generation rates and transmit the collected data to AUV2 when the inter-AUV communication condition is favorable. Thus, this paper proposes an SAC-based integrated scheduling policy to jointly optimize these two actions.

The proposed method consists of state and reward design for AUV1 and discrete SAC-based learning with action masking. First, AUV1 observes a state that includes the AoI of each node, data generation rate, relay waiting time, and the link condition with AUV2. After executing the selected action, AUV1 receives rewards based on the variation in the AoI penalty as well as data collection and relay transmission success. Through this process, AUV1 learns how node-visit and relay transmission actions affect the end-to-end information update process. Second, an SAC-based actor–critic structure with action masking is used to select a feasible action in the current state. The selected action is evaluated based on its contribution to the subsequent information update process and Mean Peak AoI minimization. This evaluation is reflected in subsequent action selection. By repeatedly performing this process, AUV1 learns a scheduling policy that reduces the Mean Peak AoI of the overall system.

2.2.1. Delay-Aware State and Reward Design

This subsection designs the state and reward so that AUV1 can jointly determine node-visit and relay transmission actions within the integrated collection-relay scheduling process for Mean Peak AoI minimization. At each decision step, AUV1 selects an action corresponding to either visiting a sensor node for data collection or relaying the data currently stored in the buffer to AUV2. The action space of AUV1 is defined as

$$\mathcal{A}=\left\{a_1^{\mathrm{col}},a_2^{\mathrm{col}},\dots ,a_n^{\mathrm{col}},\dots ,a_N^{\mathrm{col}},a^{\mathrm{tx}}\right\}$$

(5)

where $a_n^{\mathrm{c}\mathrm{o}\mathrm{l}}$ denotes the action of visiting the $n$-th sensor node for data collection, and $a^{\mathrm{t}\mathrm{x}}$ denotes the relay transmission action from AUV1 to AUV2. Since the scheduling action is defined as a finite set composed of node-visit actions and a relay transmission action, the proposed method adopts SAC over a discrete action space. Let $s_k$ denote the state observed by AUV1 at decision step $k$. To determine either a node-visit action or a relay transmission action, AUV1 observes the current decision time $t_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}$, travel time to each sensor node $T_k$, average data generation rate of each node $\lambda$, node-wise AoI observed at the buoy $A_k^{\mathrm{b}\mathrm{u}\mathrm{o}\mathrm{y}}$, and waiting time of data stored in the AUV1 buffer $A_k^{\mathrm{b}\mathrm{u}\mathrm{f}\mathrm{f}}$. In addition, AUV1 observes the periodic movement state of AUV2 $P_k$ and the elapsed time since the most recent relay success $\mathsf{\Delta }{t}_k$ to indirectly reflect the variation in the relay transmission success probability between AUV1 and AUV2. Accordingly, the state is composed as follows:

$$\begin{array}{c}{s}_k=\{t_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}},T_k,\lambda ,A_k^{\mathrm{b}\mathrm{u}\mathrm{o}\mathrm{y}},A_k^{\mathrm{b}\mathrm{u}\mathrm{f}\mathrm{f}},P_k,\mathsf{\Delta }{t}_k\}.\end{array}$$

(6)

For nodes without data in the buffer, $A_k^{\mathrm{b}\mathrm{u}\mathrm{f}\mathrm{f}}$ is set to 0.

AUV1 receives the state $s_k$ in (6) and selects one action from the discrete action set $\mathcal{A}$ in (5). The node-visit action does not immediately reduce the AoI at the buoy. Instead, it collects new data from a sensor node and enables future AoI updates through relay transmission. A relay transmission action cannot collect new data, but it directly reduces the AoI at the buoy by successfully delivering the data stored in the buffer to the buoy. Thus, the AoI at the buoy is updated only when data generated by a sensor node are collected through a node-visit and subsequently delivered through relay transmission.

If Mean Peak AoI is used directly as the reward, the agent cannot clearly learn how the selected action affects long-term AoI reduction. This is because node visits and relay transmissions affect the AoI after some delay. In addition, Mean Peak AoI is calculated over the entire episode, while actions are selected at each decision step. Therefore, using Mean Peak AoI as a stepwise reward can provide weak learning signals and make policy learning difficult.

Therefore, a reward that can minimize the Mean Peak AoI at each decision step within an episode is required. In this paper, the use of the node-wise max AoI increment as the reward to minimize the Mean Peak AoI is derived through Lemma 1.

Let the maximum AoI at node $n$ up to decision step $k$ be denoted by $A_{k,n}^{max}$. When the max AoI increment with respect to decision step $k-1$ is defined as the node-wise max AoI increment, the following Lemma 1 holds.

Lemma 1.

The accumulated node-wise max AoI increment, averaged over all nodes, corresponds to the Mean Peak AoI.

$${\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=0}^{N-1}}{\displaystyle \sum _{k=1}^K}{\left[A_{k,n}^{\mathrm{m}\mathrm{a}\mathrm{x}}-A_{k-1,n}^{\mathrm{m}\mathrm{a}\mathrm{x}}\right]}^{+}={\overline{A}}^{\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}$$

(7)

$[x{]}^{+}=\mathrm{m}\mathrm{a}\mathrm{x}(0,x)$. The proof is presented in Appendix A.

By Lemma 1, the accumulated node-wise max AoI increment corresponds to the Mean Peak AoI, so minimizing the node-wise max AoI increment at each decision step minimizes the Mean Peak AoI. Thus, this paper uses the node-wise max AoI increment computed at each decision step as the AoI penalty reward. The AoI penalty reward $r_k^{\mathrm{a}\mathrm{o}\mathrm{i}}$ and the final integrated reward $r_k$ at decision step $k$ are defined as follows:

$$r_k^{\mathrm{a}\mathrm{o}\mathrm{i}}=-{\omega }_A{\displaystyle \frac{1}{N}}\sum _{n=0}^{N-1}{\left[A_{k,n}^{\mathrm{m}\mathrm{a}\mathrm{x}}-A_{k-1,n}^{\mathrm{m}\mathrm{a}\mathrm{x}}\right]}^{+}$$

(8)

$$r_k=r_k^{\mathrm{aoi}}+\alpha ·1_{\left\{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a} \mathrm{c}\mathrm{o}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\right\}}+\beta ·1_{\left\{\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{a}\mathrm{y} \mathrm{s}\mathrm{u}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\right\}}$$

(9)

${\omega }_A$ denotes a weight that controls the magnitude of the AoI penalty, and $1_{\{\cdot \}}$ denotes an indicator function that has a value of 1 when the corresponding event succeeds. $\alpha$ and $\beta$ denote weighting parameters according to the success of each action. The two success rewards are auxiliary shaping terms that complement the credit assignment for the delayed effects of node visits and relay transmissions. If $\alpha$ is too large, the policy may excessively prefer data collection itself over reducing the AoI at the buoy, and if $\beta$ is too large, it may prioritize relay transmission rather than collecting sufficient new data. Conversely, if the two weights are too small, the immediate learning signal for the delayed effects of node visits and relay transmissions may be insufficient. Therefore, $\alpha$ and $\beta$ are set to assist the learning of the two actions while maintaining the AoI penalty reward as the main learning criterion.

2.2.2. SAC-Based Scheduling Policy Learning

This subsection explains the SAC-based policy learning process for the proposed integrated collection-relay scheduling, where AUV1 determines either a node-visit action or a relay transmission action at each decision step. The proposed method adopts the discrete SAC algorithm, which learns a stable stochastic policy over a discrete action set [12,16], and establishes an optimal scheduling policy for end-to-end Mean Peak AoI minimization.

The actor network receives the current state $s_k$ and outputs the policy distribution ${\pi }_{\theta }(a_k\mid s_k)$ over the action space $A$, which includes node-visit actions and the relay transmission action. Discrete SAC adds an entropy regularization term to the objective function. By maximizing both reward and policy entropy, the policy avoids early convergence to a local optimum or a specific action and maintains exploration in a changing environment.

In the discrete action space, the policy may select an action that does not create a new AoI update opportunity in the current state. Whether an action updates the AoI is determined solely by the current state, regardless of the degree of policy learning. Therefore, rather than gradually reducing such actions through policy learning, it is more efficient to exclude them directly at the execution stage [17,18]. To address this issue, the proposed method combines an action masking mechanism at the output of the actor network. Specifically, a masking function $m_k\left(a\right)\in \left\{\mathrm{0,1}\right\}$ is defined to determine the validity of an action at each decision step $k$. If action $a_k$ is executable and an efficient alternative in the current state $s_k$, then $m_k\left(a\right)=1$. If the action does not create a new AoI update opportunity in the current state, then $m_k\left(a\right)=0$ and the action is excluded from the selection process. By combining the original policy probability distribution ${\pi }_{\theta }\left(a_k|s_k\right)$ with the masking function, the final execution policy $\widetilde{{\pi }_{\theta }}\left(a_k|s_k\right)$ that interacts with the environment is defined as follows:

$$\widetilde{{\mathsf{\pi }}_{\mathsf{\theta }}}\left(a_k|s_k\right)={\displaystyle \frac{{\mathsf{\pi }}_{\mathsf{\theta }}\left(a_k|s_k\right)m_k\left(a_k\right)}{{\sum }_{a^{\prime }\in \mathcal{A}}{\mathsf{\pi }}_{\mathsf{\theta }}\left(a^{\prime }|s_k\right)m_k\left(a^{\prime }\right)}}$$

(10)

In (10), an unavailable action has a numerator of zero, resulting in a final selection probability of zero. Valid actions are renormalized using only the probabilities of selectable actions. This masking-based execution policy ensures that an action which does not create a new AoI update opportunity is not selected, regardless of the action of the learned policy.

Based on the masked policy ${\tilde{\pi }}_{\theta }$, the critic network (soft Q-network), evaluates the long-term value of the selected action. The critic estimates the value function using the immediate reward of the selected action, future states resulting from that action, and cumulative rewards of subsequent actions. This evaluation represents how much the current node-visit or relay transmission action contributes to minimizing the Mean Peak AoI of the overall system.

The actor network updates the policy parameter $\theta$ using the Q-value provided by the critic network. This update increases the selection probability of valid actions that are expected to produce larger long-term gains in the current state. It also helps maintain a balanced action distribution within the valid action space refined by masking.

The proposed actor–critic structure enables AUV1 to jointly optimize node visits and relay transmissions within a single integrated scheduling problem. By repeating this learning process, AUV1 learns to select valid actions in dynamic and uncertain communication environments. The optimal scheduling policy that minimizes the Mean Peak AoI of the overall system is defined as

$${\mathsf{\pi }}^{\ast }=\arg \underset{\mathsf{\pi }}{\min }{\mathbb{E}}_{\mathsf{\pi }}\left[\overline{A^{\mathrm{peak}}}\right]$$

(11)

The learned scheduling policy improves the information freshness of the overall system under irregular data generation patterns and time-varying relay-link conditions.

3. Simulation Results and Analysis

In this experiment, the end-to-end Mean Peak AoI performance of the proposed method was compared with those of Visit-Only SAC, Age-Greedy threshold, and Age-Gain TX. The proposed method jointly determined the node-visit order and relay transmission timing, whereas the comparison methods handled these actions separately. In addition, the proposed method performed an ablation analysis to examine the effect of action masking on the immediate reselection of the current node and on relay transmission under an empty-buffer state, and, based on the results, the method applying the Empty-Buffer Relay Mask was used in the experiments.

Existing learning-based AUV/UAV data collection studies and AoI-aware relay scheduling studies mostly focus on either node-visit or relay transmission decisions or assume a condition in which the data to be transmitted are already available. Therefore, it is difficult to apply these methods directly as comparison methods in the integrated collection-relay environment of this study. Accordingly, this study configures the representative decision-making principle used in each research direction to fit the same multi-AUV multi-hop data collection environment and uses them as comparison methods.

Visit-Only SAC uses SAC to learn the node-visit order in the same way as the proposed method and performs relay transmission according to a fixed threshold rule. That is, it corresponds to a comparison policy that keeps the node-visit learning condition identical to the proposed method but excludes relay transmission from the integrated learning, and it is used for the end-to-end performance comparison with the proposed integrated policy. Age-Greedy threshold visits the node with the largest current AoI and performs relay transmission when the relay success probability is greater than or equal to 0.5, representing a policy that combines AoI-priority-based greedy node selection with a channel-threshold relay rule. Age-Gain TX determines node visits in a round-robin manner and selects whether to transmit according to the expected AoI reduction upon successful relay transmission, representing a policy that combines fixed node-visit scheduling with AoI-gain-based relay scheduling.

Delivery was also used as an auxiliary metric to verify whether Mean Peak AoI reduction was accompanied by increased data delivery to the buoy. The cumulative distribution functions (CDFs) of Mean Peak AoI and Delivery were analyzed to examine episode-wise performance variation.

3.1. Simulation Environment

The buoy was located at coordinate point $\left[0,0\right]$ in the simulation environment. AUV2 periodically moved along a circular trajectory with a radius of 1000 m centered at the buoy. Nine sensor nodes were deployed at uniform intervals within a $5000\hspace{0.17em}\mathrm{m}\times 5000\hspace{0.17em}\mathrm{m}$ square node deployment area centered at coordinate point $0,-4500$. AUV1 started from the center of the node deployment area and moved within the communication range of each node to collect data. The communication range for data collection between AUV1 and each sensor node was set to 500 m [19]. The simulation environment is shown in Figure 2.

Figure 2. Simulation environment and sensor-node deployment.

The communication state between AUV1 and AUV2 was modeled by reflecting underwater acoustic communication conditions. Propagation loss was calculated using spreading loss and the Thorp absorption model [20], and underwater ambient noise was modeled using the Wenz noise model [21]. The received SNR computed in this way was used to obtain the packet-level relay success probability through the sigmoid model defined in Section 2.1 [14]. The success or failure of each relay transmission was determined based on this probability.

The data transmitted from a node were configured as one frame with a length of 10 s [22]. To reflect random event occurrence, the data generation time of each sensor node followed a Poisson process in each episode. As a result, an average of 180 packets was generated by all nine nodes [23,24]. Each node had a different average number of generated packets, and five scenarios were configured by changing the locations of nodes with high and low average generation numbers. The node locations used in the five scenarios and the average packet generation per node are presented in Table 1.

Table 1. Sensor node coordinates and scenario-specific average packet generation.

Node Index	X (m)	Y (m)	Scenario 1 (Packets/Episode)	Scenario 2 (Packets/Episode)	Scenario 3 (Packets/Episode)	Scenario 4 (Packets/Episode)	Scenario 5 (Packets/Episode)
0	−2125	−2875	30.0	10.0	10.0	10.0	30.0
1	0	−2875	27.5	12.5	12.5	12.5	10.0
2	2125	−2875	25.0	15.0	30.0	15.0	27.5
3	−2125	−5000	22.5	17.5	15.0	17.5	12.5
4	0	−5000	20.0	20.0	27.5	30.0	20.0
5	2125	−5000	17.5	22.5	17.5	20.0	15.0
6	−2125	−7125	15.0	25.0	25.0	22.5	25.0
7	0	−7125	12.5	27.5	20.0	27.5	17.5
8	2125	−7125	10.0	30.0	22.5	25.0	22.5

The episode time for training and evaluation was set to $T=\mathrm{15,000} \mathrm{s}$. The four methods, including the proposed method, were trained and evaluated in the same environment for five scenarios. The simulation and training parameters are listed in Table 2 and Table 3, respectively.

Table 2. Simulation parameters.

Parameter	Value
Episode length T	15,000 s
$\mathrm{AUV}1 \mathrm{speed} v_{auv1}$	4 knots
AUV2 Orbit period	3000 s
$\mathrm{Carrier} \mathrm{frequency} f_c$	15 kHz
$\mathrm{Bandwidth} B$	10 kHz
$\mathrm{Source} \mathrm{Level} SL$	120 dB re 1 μPa at 1 m
$\mathrm{Spreading} \mathrm{factor} k$	1.5
$\mathrm{SNR} \mathrm{threshold} {\gamma }_{th}$	14 dB
$\mathrm{SNR} \mathrm{margin} \mathrm{width} {\mathsf{\Delta }}_{th}$	4 dB
Average generated packets per node	10~30 packets/episode

Table 3. Training hyperparameter.

Parameter	Value
Hidden dimension	256
Learning rate	$3\times {10}^{-4}$
Discount factor	0.995
Batch size	256
Replay buffer size	${10}^6$
Soft update coefficient	0.005
Initial temperature	0.2

3.2. Mean Peak AoI and Delivery Performance Comparison

This subsection evaluates the average Mean Peak AoI and Delivery performance of the proposed method and the comparison methods. Training was performed 10,000 times for each of the five scenarios, and performance evaluation was conducted over 3000 episodes. In each evaluation episode, the Mean Peak AoI was computed by first obtaining the maximum AoI observed at each node during the episode and then averaging it over all nodes. Table 4 shows the average Mean Peak AoI, and Table 5 shows the average Delivery; the values in parentheses indicate the performance difference relative to the proposed method and the ratio delivered to the buoy among the total 180 generated packets, respectively.

Table 4. Mean Peak AoI comparison across five experimental environments.

Policy	Scenario 1 (sec)	Scenario 2 (sec)	Scenario 3 (sec)	Scenario 4 (sec)	Scenario 5 (sec)	Average (sec)
Age-Greedy (th = 0.5)	10,676 (98.6%)	10,520 (92.1%)	10,489 (95.5%)	10,484 (96.4%)	10,506 (95.3%)	10,535 (95.6%)
Visit-Only SAC (th = 0.5)	5818 (8.2%)	5927 (8.2%)	5873 (9.5%)	5965 (11.7%)	5824 (8.3%)	5881 (9.2%)
Age-Gain TX	5580 (3.8%)	5548 (1.3%)	5545 (3.4%)	5533 (3.6%)	5566 (3.5%)	5554 (3.1%)
Proposed method	5376 (0.0%)	5476 (0.0%)	5365 (0.0%)	5339 (0.0%)	5380 (0.0%)	5387 (0.0%)

Table 5. Delivery comparison across five experimental environments.

Policy	Scenario 1 (Packets)	Scenario 2 (Packets)	Scenario 3 (Packets)	Scenario 4 (Packets)	Scenario 5 (Packets)	Average (Packets)
Age-Greedy (th = 0.5)	12.9 (7.2%)	13.4 (7.4%)	13.3 (7.4%)	13.4 (7.4%)	13.3 (7.4%)	13.3 (7.4%)
Visit-Only SAC (th = 0.5)	43.7 (24.3%)	41.8 (23.2%)	42.0 (23.3%)	41.9 (23.3%)	42.4 (23.6%)	42.4 (23.5%)
Age-Gain TX	43.4 (24.1%)	42.6 (23.7%)	43.1 (23.9%)	42.7 (23.7%)	42.1 (23.4%)	42.8 (23.8%)
Proposed method	48.5 (26.9%)	46.1 (25.6%)	46.3 (25.7%)	47.6 (26.4%)	45.7 (25.4%)	46.8 (26.0%)

In Table 4, the proposed method achieved the lowest average Mean Peak AoI of 5387 s among the four methods. Age-Greedy (th = 0.5) showed the largest performance gap because it preferentially visited the node with the largest AoI without reflecting differences in data generation rates across scenarios and relied on a fixed threshold for relay transmission. Visit-Only SAC was 9.2% higher than the proposed method because it learned only the node-visit order and followed a fixed rule for relay transmission. Age-Gain TX was closest to the proposed method but was still 3.1% higher due to its fixed round-robin visit order. These results indicate that node-visit scheduling and relay transmission timing should be determined jointly.

In Table 5, the proposed method achieved the highest Delivery of 46.8 packets on average, corresponding to 26.0% of the total 180 generated packets. Age-Greedy threshold delayed transmission until the relay success probability exceeded the threshold, delivering 71.6% fewer packets than the proposed method. Visit-Only SAC failed to match the buffered data with favorable relay-link conditions due to its fixed relay rule, delivering 9.4% fewer packets. Age-Gain TX delivered 8.5% fewer packets because its fixed round-robin visits left insufficient fresh data in the buffer at the time relay transmission was possible.

These results demonstrate that integrated collection-relay scheduling is suitable for reducing Mean Peak AoI and improving data delivery in multi-AUV multi-hop underwater data collection environments.

3.3. CDF-Based Analysis of Performance Stability

This subsection analyzes the episode-wise performance distribution and stability that cannot be confirmed from average values alone. To this end, the CDFs of Mean Peak AoI and Delivery of the proposed method and the comparison methods are compared to verify whether the advantage in the central value is maintained over the entire distribution and in worst-case episodes.

Figure 3 and Figure 4 show the CDFs of Mean Peak AoI and Delivery, respectively. In both figures, the green, blue, orange, and red lines represent Age-Greedy threshold, Age-Gain TX, Visit-Only SAC, and the proposed method, respectively. (a)–(e) are the scenario-wise CDFs, and (f) is the average CDF obtained by aggregating all evaluation episodes of the five scenarios.

Figure 3. CDF comparison of Mean Peak AoI across five experimental environments.

Figure 4. CDF comparison of Delivery across five experimental environments.

In the Mean Peak AoI CDFs of Figure 3a–f, the proposed method was located to the left of the comparison methods in all scenarios and in the average result, which indicates that it achieved a lower Mean Peak AoI over the evaluation episodes. In addition, the shorter right tail of the proposed method in Figure 3f shows that episodes with significantly increased Mean Peak AoI were reduced, indicating that the proposed method reduced both the average AoI and the episode-wise performance variation.

Among the comparison methods, Age-Gain TX showed the most similar CDF shape to the proposed method but was located to the right of the proposed method in most scenarios and in the average result. Visit-Only SAC approached the low-AoI region in some scenarios, but its distribution extended to the high-AoI region depending on the scenario, indicating that performance stability is difficult to maintain when only the node-visit order is learned and relay transmission is handled by a separate rule.

In the Delivery CDFs of Figure 4a–f, the proposed method was located to the right of the comparison methods in all scenarios and in the average result, which indicates that it delivered more updates to the buoy over the evaluation episodes. In Figure 4f, the distribution of the proposed method extended less toward the low-Delivery region, indicating that episodes with significantly degraded Delivery were reduced. Age-Gain TX showed an improved Delivery distribution among the comparison methods, but compared with the proposed method, it was located further to the left or showed a wider distribution depending on the scenario.

As a result, the CDF results show that, by jointly considering node visits and relay transmission within a single decision-making process, the proposed method stably achieves a lower Mean Peak AoI and higher Delivery than the closest comparison method, Age-Gain TX, over the entire episode distribution.

3.4. End-to-End Delivery Bottleneck Analysis

This subsection analyzes the main factor that limits the absolute Delivery ratio. To this end, a reduced temporal-load environment, in which the data generation rate of each node in Scenario 1 was halved and the episode length was doubled, was additionally evaluated. Accordingly, the total average generation per episode was kept identical to the original at 180 packets, so that the change in the Delivery ratio originates from the difference in the temporal density at which data is generated rather than from a difference in the generated amount. Table 6 shows the Delivery results of the original nominal environment and the reduced temporal-load environment under the same scenario.

Table 6. Delivery analysis under different temporal-load conditions.

	Generated Data	Delivered Data	Delivery Ratio
Nominal environment	180	48.5	26.9%
Reduced temporal-load environment	180	67.1	37.3%

In Table 6, the Delivery ratio of the reduced temporal-load environment was 37.3%, which is 10.4% higher than the 26.9% of the nominal environment.

In the nominal environment, each sensor node generates an average of 10–30 data per episode, so the average data generation interval per node is 500–1500 s. Meanwhile, AUV1 moves at 4 knots, that is, about 2.06 m/s. Since the distance between adjacent node centers is 2125 m and the data collection range of each node is 500 m, AUV1 must travel at least 1125 m to move between the collection regions of adjacent nodes, which requires about 547 s. This is similar to the 500 s average data generation interval of a node with a high generation rate. Therefore, new data can be generated at some nodes even while AUV1 moves between adjacent nodes, and more data can be generated while AUV1 performs several node visits and relay transmissions to revisit the same node. Therefore, the main factor that limits the absolute Delivery ratio lies not in the loss at the relay transmission stage but in the data collection stage, which is determined by the mobility of AUV1 and the node revisit interval.

3.5. Action-Masking Ablation Analysis

This subsection analyzes the effect of action masking on two actions that hardly contribute to Mean Peak AoI minimization. First, for the action that immediately reselects the current node, no time has elapsed since the previous visit, so it is difficult to collect new data. Second, for the action that performs relay transmission under an empty-buffer state, there are no data to transmit, so the AoI cannot be reduced. Depending on whether masking was applied to these two actions, four configurations were evaluated: Full Masking, which masks both actions; w/o Current-Node Reselection Mask, which removes only the node-reselection mask; w/o Empty-Buffer Relay Mask, which removes only the Empty-Buffer Relay Mask; and w/o Both Masks, which removes both masks. Each configuration was trained independently for 10,000 episodes in Scenario 1 using the same parameters as in Section 3.1.

Table 7 presents the selection rate of the unmasked actions, the time-out ratio over all episodes when an infinite loop occurred, and the Mean Peak AoI, obtained by evaluating each configuration over 10,000 episodes. A time-out was identified as an infinite loop when the decision step performed the same action more than 100 consecutive times; in this case, the episode was forcibly terminated, and the remaining time until the end of the episode was added to the AoI of each node to compute the Mean Peak AoI.

Table 7. Results of the action-masking ablation analysis.

	Current-Node Reselection Rate (%)	Empty-Buffer Relay Selection Rate (%)	Time Out (%)	Mean Peak AoI (s)
Full Masking	0.00	0.00	0.00	$5385.7\pm 325.1$
w/o Current-Node Reselection Mask	0.00	0.00	0.00	$5398.1\pm 320.3$
w/o Empty-Buffer Relay Mask	0.00	12.44	22.86	$6149.7\pm 2047.0$
w/o Both Masks	0.00	8.40	17.06	$5944.5\pm 1557.0$

In Table 7, for w/o Current-Node Reselection Mask, the learned policy did not select the node-reselection action, so its Mean Peak AoI was similar to that of Full Masking. There was no performance change due to masking.

In contrast, for w/o Empty-Buffer Relay Mask, the selection rate of the corresponding action was 12.44%, the time-out ratio increased to 22.86%, and the Mean Peak AoI increased by 14.2% to 6149.7 s. In this case, the AoI became very large when an infinite loop occurred, so the standard deviation was about 5.3 times larger than that of Full Masking.

w/o Both Masks showed a time-out ratio of 17.06% and a Mean Peak AoI of 5944.5 s, which was slightly lower than that of w/o Empty-Buffer Relay Mask. The reason its Mean Peak AoI did not match that of w/o Empty-Buffer Relay Mask is that, during training, the mask was applied to the Current-Node Reselection action, which changed the statistical characteristics of selecting each individual action relative to all actions.

Therefore, this experiment showed that the Current-Node Reselection Mask did not affect the Mean Peak AoI performance, whereas the Empty-Buffer Relay Mask blocked, at the execution stage, invalid actions that are not eliminated by SAC policy learning alone, thereby preventing infinite loops and improving the Mean Peak AoI performance. Accordingly, the proposed method applied the Empty-Buffer Relay Mask to improve the Mean Peak AoI minimization performance.

3.6. Performance Evaluation Under Channel Variation

This subsection analyzes the effect on the performance of the proposed method between the channel model used in training and a varied channel environment.

To verify this, at each relay transmission attempt in the evaluation stage, a scaling factor independently generated from a uniform distribution between 0.5 and 2.0 times was multiplied by the sigmoid-based success probability used in training to introduce variability. As in the original evaluation stage, 3000 episodes were conducted and the performance was averaged; Table 8 and Table 9 show the Mean Peak AoI and Delivery performance in the channel variation environment.

Table 8. Mean Peak AoI comparison across channel variation environments.

Policy	Scenario 1 (sec)	Scenario 2 (sec)	Scenario 3 (sec)	Scenario 4 (sec)	Scenario 5 (sec)	Average (sec)
Age-Greedy (th = 0.5)	10,620 (97.5%)	10,460 (90.8%)	10,508 (96.0%)	10,433 (94.7%)	10,487 (94.6%)	10,502 (94.7%)
Visit-Only SAC (th = 0.5)	5809 (8.0%)	5937 (8.3%)	5922 (10.4%)	5953 (11.1%)	5882 (9.2%)	5901 (9.4%)
Age-Gain TX	5486 (2.0%)	5489 (0.1%)	5490 (2.4%)	5480 (2.2%)	5510 (2.3%)	5491 (1.8%)
Proposed method	5377 (0.0%)	5481 (0.0%)	5362 (0.0%)	5360 (0.0%)	5388 (0.0%)	5394 (0.0%)

Table 9. Delivery comparison across channel variation environment.

Policy	Scenario 1 (Packets)	Scenario 2 (Packets)	Scenario 3 (Packets)	Scenario 4 (Packets)	Scenario 5 (Packets)	Average (Packets)
Age-Greedy (th = 0.5)	13.0 (7.2%)	13.5 (7.5%)	13.3 (7.4%)	13.6 (7.6%)	13.4 (7.4%)	13.4 (7.4%)
Visit-Only SAC (th = 0.5)	43.8 (24.3%)	41.8 (23.2%)	41.7 (23.3%)	41.9 (23.3%)	42.2 (23.6%)	42.3 (23.5%)
Age-Gain TX	43.6 (24.2%)	42.9 (23.8%)	43.3 (24.1%)	43.0 (23.9%)	42.4 (23.6%)	43.0 (23.9%)
Proposed method	48.7 (27.1%)	46.1 (25.6%)	46.6 (25.9%)	47.6 (26.4%)	45.6 (25.3%)	46.9 (26.1%)

The proposed method achieved the lowest Mean Peak AoI and the highest Delivery in all scenarios, even in the channel variation environment. The average Mean Peak AoI increased by about 0.1% from 5387 s in the nominal environment to 5394 s, and the average Delivery was maintained at a similar level, from 46.8 packets to 46.9 packets. In addition, the relative performance advantage over the comparison methods was maintained. These results show that the proposed method does not rely only on the case where the sigmoid channel model used in training exactly matches the actual success probability and can maintain its performance advantage, even in a channel variation environment.

3.7. Performance Evaluation with an Increased Number of Sensor Nodes

This subsection verifies whether the performance advantage of the proposed method is maintained in an environment with an increased number of sensor nodes. Twelve nodes were deployed in a 3 × 4 grid with intervals of 1416 m in the x-direction and 2125 m in the y-direction within the same sensor deployment area as the previous experiment, and training and evaluation were conducted under the same episode length and simulation parameters as the previous experiment. The average generation rate per node is presented in Table 10, and the total number of packets is 240. Table 11 shows the Mean Peak AoI and Delivery comparison results in this environment.

Table 10. Sensor node scenario-specific average packet generation.

Node Index	0	1	2	3	4	5	6	7	8	9	10	11
Scenario (packets/episode)	30.0	15.5	17.3	28.2	13.6	19.1	20.9	11.8	26.4	10.0	22.7	24.6

Table 11. Mean Peak AoI and Delivery comparison in 12-node deployment environments.

Policy	Mean Peak AoI (s)	Delivery (Packets)
Age-Greedy (th = 0.5)	10,973.5 (97.5%)	15.4 (6.4%)
Visit-Only SAC (th = 0.5)	6195.5 (11.1%)	52.3 (21.8%)
Age-Gain TX	6066.8 (8.8%)	52.1 (21.7%)
Proposed method	5573.9 (0.0%)	57.2 (23.8%)

In Table 11, the proposed method achieved the lowest Mean Peak AoI of 5573.9 s and the highest Delivery of 57.2 packets, maintaining better performance than all comparison methods even in the 12-node environment. The relative performance ranking among the comparison methods was also maintained the same as in the previous nine-node experiment. In particular, the performance gap with Age-Gain TX was larger in the 12-node environment (8.8%) than in the 9-node environment (3.5%). The larger gap is attributed to the fact that, as the number of nodes increases, the inability of the fixed round-robin visit order to adapt to the heterogeneous data generation rate distribution becomes more severe, whereas the proposed method can continuously and jointly optimize node visits and relay transmissions through the learned scheduling policy. These results show that the proposed integrated collection-relay scheduling maintains better performance than the comparison methods, even in an environment with an increased number of nodes.

4. Conclusions

This paper proposed a discrete SAC-based integrated scheduling policy for reducing end-to-end Mean Peak AoI in a multi-AUV multi-hop underwater data collection environment. The proposed method jointly determined node-visit and relay transmission actions within a single decision-making process and reflected the information update process from data collection to buoy update in policy learning. This paper established Lemma 1, which shows that the accumulated node-wise max AoI increment corresponds to the Mean Peak AoI, and accordingly used the node-wise max AoI increment as the decision step reward of the discrete SAC framework. In addition, based on the action-masking ablation analysis, the empty-buffer relay transmission, which cannot contribute to Mean Peak AoI minimization and is not eliminated by policy learning alone, was masked. Using this configuration, simulation results showed that the proposed method achieved lower Mean Peak AoI and higher Delivery than Visit-Only SAC, Age-Greedy threshold, and Age-Gain TX, reduced high-AoI episodes in the CDF analysis, and maintained this advantage under channel variation and an increased number of sensor nodes. These results demonstrate that the proposed integrated collection-relay scheduling is advantageous for reducing end-to-end information update delay under irregular data generation and time-varying relay-link conditions.