Abstract
In complex environments, communication networks within an unmanned ground vehicle (UGV) swarm are vulnerable to disruption due to terrain-induced blockage, high node mobility, and limited communication ranges. Employing an unmanned aerial vehicle (UAV) as a mobile relay provides an effective means of maintaining reliable communication. However, under partially observable and high-noise conditions, noisy observations may fail to accurately reflect the actual motion states of ground nodes, thereby degrading the reliability of UAV relay decisions. To address this issue, this paper proposes a Kalman filter–Soft Actor–Critic (KF-SAC) algorithm. This method employs a Kalman filter to fuse the motion models of the ground nodes with noisy measurements, thereby obtaining posterior estimates of motion states, such as position and velocity, as well as the corresponding motion trends. By feeding these estimates into the SAC agent as state inputs, the method improves the reliability of UAV relay trajectory decisions and enhances adaptability to dynamic and noisy environments. Simulation results demonstrate that, compared with standard SAC, the proposed method effectively reduces the overall communication cost of the UGV swarm and improves communication quality amidst complex terrain and noisy observation conditions.
1. Introduction
Unmanned systems performing tasks such as environmental monitoring, area inspection, disaster search and rescue, and emergency response typically rely on stable and reliable communication networks to facilitate information sharing and coordinated action [1,2,3,4]. However, unmanned systems operating in mountainous, hilly, urban, or other large-scale complex environments may experience degraded or disrupted communication links due to terrain-induced blockage, dispersed node distribution, and limited communication range [5,6,7]. Moreover, node mobility causes continuous changes in network topology [8]. Under these conditions, communication methods relying on direct node connections or fixed infrastructure struggle to consistently meet the real-time performance, reliability, and robustness requirements of coordinated unmanned system operations [9,10]. Consequently, relay nodes have been introduced to extend communication coverage. Among various relay platforms, unmanned aerial vehicles (UAVs) have emerged as a promising solution owing to their high mobility and favorable line-of-sight communication capabilities, enabling flexible communication relaying for unmanned ground vehicles and other ground nodes [11,12,13].
Despite the significant potential of UAV-based communication relaying, efficient relay trajectory planning and decision-making in complex and dynamic environments remain challenging [13,14,15]. First, due to limited sensing capability and terrain-induced obstructions, the UAV may not always obtain accurate and complete state information of mobile ground nodes in real time, resulting in incomplete information for relay decision-making. Second, significant fluctuations in communication conditions complicate control efforts. In complex terrain, wireless signal propagation is influenced not only by distance but also by factors such as multipath effects and shadowing, leading to high levels of noise in observational data, such as received signal strength [16].
Early research on communication relaying for unmanned systems was primarily grounded in graph theory and network optimization, focusing on issues such as maintaining network connectivity and minimizing communication costs. Typically, unmanned systems are modeled as communication nodes, and communication topology graphs are constructed to optimize the connections between them; examples include the Minimum Spanning Tree (MST) method [17,18] and classical graph-theoretic approaches such as min-cut [19] and max-flow [20]. As unmanned aerial vehicles (UAVs) have increasingly been employed for mobile communication relaying, research has expanded beyond topology optimization to encompass UAV deployment, trajectory planning, and communication resource allocation. Nasrollahi et al. [21] improved the overall performance of UAV-assisted ground user communication by jointly optimizing UAV flight trajectories and power allocation. Xia et al. [22] modeled air-to-ground communication channels and analyzed the relationships among UAV flight altitude, line-of-sight (LoS) propagation probability, and path loss, thereby providing a theoretical basis for optimizing relay positioning. Meanwhile, Su et al. [23] investigated optimal deployment strategies for UAV relays subject to total energy consumption constraints.
As the mobility of unmanned system nodes and the dynamic nature of communication environments increase, reinforcement learning (RL) is increasingly applied to UAV relay trajectory planning, communication resource allocation, and air–ground collaborative decision-making. Lee et al. [24] proposed an RL framework for UAV relay communication in emergency scenarios, utilizing positioning information to assist in communication and path planning. Addressing scenarios where a UAV acts as a mobile relay between multiple ground users and a macro base station, Abohashish et al. [25] employed RL to optimize UAV trajectories and communication load distribution, thereby enhancing wireless resource utilization. Zhang et al. [26] further utilized RL to solve non-convex optimization problems in mobile UAV relaying, aiming to maximize the total data transmission volume between users and base stations. For communication environments subject to interference, Lu et al. [27] proposed a relay strategy combining deep reinforcement learning and transfer learning; this enables UAVs to autonomously select relay actions—even without accurate interference or cellular network models—to help the system mitigate interference and reduce bit error rates. In scenarios where UAVs serve as relays for device-to-device (D2D) communication networks, Deng et al. [28] utilized RL algorithms to optimize communication resource allocation. Furthermore, RL has been extended to multi-node collaborative communication scenarios. Ding et al. [29] employed a QMIX-based multi-agent reinforcement learning method to optimize flight trajectories, frequency allocation, and data transmission timing in multi-hop UAV relay networks; meanwhile, Luong et al. [30] and Wang et al. [31] investigated the joint design of UAV positioning and resource allocation, as well as trajectory planning and energy consumption optimization for multi-UAV collaborative networks, respectively. Li et al. [32] proposed an SAC algorithm (SAC-CC) that integrates energy management and covert communication constraints. By dynamically allocating sensing and communication bandwidth and optimizing the UAV flight trajectory, this algorithm solves the problem of covert transmission optimization under energy constraints. Qi et al. [33] combined reinforcement learning with LSTM and proposed a collaborative learning framework to address the collaborative interruption caused by intermittent transmission of unmanned aerial vehicles. Tang et al. [34] combined the Hungarian algorithm, the Generative Diffusion Model, and the Multi-Agent Deep Deterministic Policy Gradient method to propose a collaborative framework for unmanned aerial vehicles, ground-based mobile robots, and airships, which enhances the computing capabilities of the unmanned aerial vehicles and optimizes task decision-making. Jiang et al. [35] proposed a 6G air–ground cooperative perception and computing framework that integrates reinforcement learning and intelligent optimization algorithms, in order to alleviate the field-of-view limitations and computational resource constraints in autonomous driving. Collectively, these studies indicate that RL can address coupled decision-making problems involving UAV trajectories, link states, resource allocation, and collaborative relationships through continuous interaction with dynamic communication environments, providing a promising approach to autonomous communication relaying in complex environments.
Traditional relay methods based on rules or geometric optimization typically assume ideal communication models or known locations, making them ill-suited to such dynamic changes. While reinforcement learning algorithms offer adaptability, directly feeding observation data containing significant noise into the policy network can easily lead the agent to misjudge the environmental state; this triggers oscillations in action output, resulting in unstable UAV flight trajectories and making it difficult to maintain high-quality communication links.
Addressing the aforementioned issues of inaccurate observation and signal instability, relying solely on end-to-end reinforcement learning strategies makes it difficult to converge to an optimal solution. To achieve stable communication relaying in environments characterized by partial observability and high noise, it is essential to equip the agent with more accurate state-sensing capabilities. Therefore, this paper proposes a Kalman Filter-augmented Soft Actor–Critic (KF-SAC) method for optimizing communication relaying. By utilizing a Kalman filter at the perception front-end to recursively derive high-confidence position and velocity trends from sensor data, and employing the SAC algorithm at the decision-making back-end for maximum-entropy exploration, the method achieves robust flight control and ensures communication quality for UAVs operating in dynamic, constrained environments. Designed to tackle the challenges of partial observability and volatile communication states during UAV relay missions over complex terrain, this approach not only enhances sample efficiency in dynamic environments but also improves the stability of flight decisions in areas with weak connectivity, thereby ensuring continuous and reliable communication support.
The main work and contributions of this paper are as follows:
- A communication relay model is established for a single UAV assisting a mobile UGV swarm in complex terrain, taking into account terrain-induced blockage, node mobility, communication constraints, and observation noise.
- A KF-SAC algorithm is proposed by integrating ground-node state estimation with SAC-based trajectory decision-making, enabling the UAV to adaptively adjust its relay trajectory according to node dynamics and thereby reduce the overall communication cost of the UGV swarm.
- A communication relay simulation environment is developed using the Qt framework, and the effectiveness of the proposed method in reducing communication costs and enhancing adaptability to noisy environments is validated through comparative experiments with standard SAC.
The remainder of this paper is organized as follows: Section 2 addresses the communication relay mission scenario and system modeling. Section 3 introduces the design of the proposed method. Section 4 covers the experimental design, the setup of the simulation platform, the experiments, and the analysis of results. Section 5 provides the conclusion.
2. Communication Relay Mission Scenarios and System Modeling
This paper considers a typical air–ground cooperative communication scenario for an unmanned system operating in a complex environment, comprising a swarm of unmanned UGVs and a relay UAV. As illustrated in Figure 1, the UGV swarm, represented by yellow dots, performs a cooperative task in complex terrain represented by rectangular blocks of varying heights. The figure depicts the communication topology of the air–ground cooperative system. The solid red and blue lines represent the air–ground communication links between the UGVs and the UAV and the ground-to-ground communication links within the UGV swarm, respectively, while the solid cyan line indicates the UAV flight trajectory during dynamic relay operation.
Figure 1.
Schematic diagram of an air–ground relay communication scenario in a complex environment.
When unmanned vehicles execute missions in the field, their trajectories are typically determined by specific tactical objectives and constrained by terrain, exhibiting continuous and smooth maneuvering characteristics. Unlike simple random walk models, this paper employs the smooth turn mobility model [36] to characterize the movement of ground nodes. Let the state vector of the i-th ground unmanned vehicle at time t be denoted as
where denotes the position coordinate, is the linear velocity, is the heading angle, and is the angular velocity.
The kinematic equations of the UGV are modeled as follows:
where is the tangential acceleration for adjusting the velocity magnitude, and is the normal acceleration for adjusting the turn rate.
Given the endurance and coverage requirements of large-scale relay missions, this study employs a fixed-wing UAV as the relay platform due to its relatively long endurance and wide coverage. However, unlike rotary-wing UAVs, fixed-wing UAVs are subject to specific kinematic constraints, including a minimum flight speed required to generate sufficient lift and a minimum turning radius. Assuming the UAV flies at a constant altitude H, its state vector is defined as , and its motion is described by a first-order response model:
where is the velocity control input, is the roll angle control input, is the time constant of velocity control, and g is the gravitational acceleration.
To guarantee flight safety and physical feasibility, the control inputs must satisfy the following constraints:
The above equations impose limits on the flight velocity and maximum yaw rate of the UAV respectively. Furthermore, the minimum turning radius is defined as
Under the discrete-time framework with time step k, the position update equation can be expressed as:
The calculation of communication link quality serves as the core foundation for relay decision-making. This section establishes mathematical models for air-to-ground (A2G) and ground-to-ground (G2G) communication links. The model comprehensively incorporates line-of-sight (LoS) propagation probability, path loss, and random shadow fading in complex terrain, and is applied to calculate the end-to-end link capacity of each node, thereby providing state observations and reward calculation basis for the reinforcement learning agent.
2.1. Air-to-Ground Probabilistic Channel Model
The channel quality between the UAV and UGVs is significantly influenced by the UAV altitude and terrain features. It should be noted that although the instantaneous air–ground channel state at the physical level is mutually exclusive (i.e., the channel is either in LoS or non-line-of-sight (NLoS) state), it is difficult to obtain accurate global 3D terrain and building information in real time. Therefore, this section refers to the research of [37] and adopts a statistical approach for channel modeling. Specifically, a probabilistic LoS propagation model is constructed, which characterizes the channel state as a probability distribution between LoS and NLoS according to the environmental characteristics of ground nodes (e.g., suburban, urban, high-rise districts). In this model, the probability of establishing a LoS link between the UAV and a ground node mainly depends on their relative geometric relationship, namely the elevation angle of the UAV relative to the ground node. Intuitively, a larger elevation angle corresponds to a higher probability that the signal avoids occlusion by ground obstacles such as buildings. Let the elevation angle between the UAV and the i-th ground node be , then the LoS probability can be expressed as a Sigmoid function of the elevation angle:
where a and b are environmental fitting parameters, which reflect the building distribution density and statistical characteristics of building height under different environment types respectively.
Correspondingly, the probability that the channel is in the NLoS occluded state is
Corresponding to the two mutually exclusive propagation states above, wireless signals suffer from distinctly different path loss. In the LoS state, the signal is mainly affected by free-space propagation loss with minor additional loss; while in the NLoS state, the signal experiences severe shadow fading and additional loss due to obstacle occlusion and reflection. Accordingly, this section introduces the path attenuation factor and carrier frequency , and the path loss under the two states is modeled as:
where is the spatial Euclidean distance, c is the speed of light, and and denote the average additional attenuation factors under LoS and NLoS conditions respectively. In general, is significantly larger than to reflect the degraded communication quality under NLoS conditions.
To provide the reinforcement learning agent with stable observations that reflect the statistical characteristics of the environment, this section adopts mathematical expectation to quantify the current average channel quality. By integrating the occurrence probabilities of LoS and NLoS and their corresponding loss values, the average path loss of the A2G link is calculated as the probability-weighted sum of the two states:
This average loss model not only reflects the impact of distance on signal strength, but more critically captures the nonlinear effect of UAV altitude variation on LoS probability, providing a physical basis for subsequent trajectory optimization.
2.2. Ground-to-Ground Communication Link Mathematical Model
Compared with the A2G channel dominated by LoS probability, the G2G communication environment among UGVs is more complex and harsh. Affected by terrain undulation, vegetation coverage, and direct occlusion from ground buildings, G2G links suffer from more severe large-scale path loss as well as shadow fading caused by multipath effects. Given the unstructured distribution of ground obstacles, it is difficult to accurately calculate the loss of each link solely through geometric relationships. Therefore, this section adopts the log-normal shadow fading model to characterize the channel properties among UGVs. The model consists of a deterministic path loss component and a random shadow fading component, which can effectively reflect the statistical propagation law of wireless signals in complex ground environments. Specifically, for any two UGVs i and j, the received signal strength is modeled as the result of transmit power amplified by channel gain, attenuated by propagation distance, and perturbed by random noise. Its mathematical expression is:
where is the transmit power of the node, K is the constant channel gain determined by antenna characteristics and carrier frequency, is the Euclidean distance between the two communication nodes, and is the reference distance. Different from the A2G model, a path attenuation factor exclusive to G2G links is introduced here, whose value typically ranges from 2 to 5. A larger value indicates a harsher propagation environment and faster signal attenuation with distance.
The most critical term in the model is , which is a random variable following a zero-mean Gaussian distribution:
where is the standard deviation of shadow fading. This random term simulates the amplitude fluctuation of wireless signals caused by random obstacles during propagation. In the training environment of reinforcement learning, this modeling approach with introduced random noise is of great importance: it forces the agent not to make decisions merely based on deterministic geometric distance, but to take environmental uncertainty into account in the strategy, thereby learning a communication relay strategy robust to channel fluctuations.
With the expansion of the UGV formation scale, the communication network presents high complexity and heterogeneity. Two types of communication links with completely different physical characteristics, namely A2G and G2G links, coexist in the network. To uniformly evaluate the overall communication state of the network within the reinforcement learning decision framework, it is necessary to establish a standardized link quality evaluation metric that maps different types of channel models to a unified numerical dimension. Based on the received signal strength model constructed above, this section defines link communication cost as the core metric for measuring the connection quality between nodes. For any two nodes u and v in the network (either UAV or UGV), the received signal strength is calculated by selecting the corresponding channel model according to node types:
where is calculated based on the probabilistic LoS propagation model, and is calculated based on the log-normal shadow fading model.
To distinguish strong links from weak links and convert signal strength into a cost form compatible with reinforcement learning algorithms, this section adopts a nonlinear mapping function based on the reference signal strength. The communication cost weight of link is defined as:
where denotes the communication cost weight of the link; a smaller value indicates better signal quality and more reliable communication. is the received signal strength calculated above. is the reference received signal strength threshold, which can be determined according to the actual parameters of communication hardware and task requirements. is the scaling factor (), which adjusts the sensitivity of the cost function to signal strength variations. The channel model considered in this paper focuses on large-scale propagation effects related to propagation distance, LoS/NLoS conditions, and environmental blocking. Doppler shifts and Doppler shifts due to relative motion are not explicitly modeled.
This calculation model eliminates the dimensional difference between heterogeneous channel models through a unified cost function. In subsequent cooperative planning and decision-making, the agent will utilize this metric to evaluate the effectiveness of each potential link in the network in real time, identify node pairs with communication obstruction, and guide the UAV to maneuver to regions that can minimize the global communication cost (i.e., maximize the global signal strength).
3. The Proposed KF-SAC Algorithm
3.1. Reinforcement Learning and the SAC Algorithm
Reinforcement learning problems can generally be described by a Markov decision process (MDP). At each discrete time step t, the agent selects an action from state according to policy , the environment transitions to the next state and generates an immediate reward . With a discount factor , the discounted return starting from time step t is defined as
To evaluate the long-term return under a given policy , the state value function and action value function are defined as
and
The introduction of value functions provides a clear quantitative criterion for policy evaluation and lays the foundation for subsequent learning processes. However, in continuous high-dimensional state spaces or under unknown transition laws, directly calculating the above expectations by definition is often infeasible. Therefore, it is necessary to construct learnable update targets using their recursive structure. To characterize the recursive structure of value functions and provide a basis for subsequent algorithm derivation, value functions satisfy the Bellman expectation equation. Its meaning can be understood as: the value of the current state equals the sum of the immediate reward obtained in the current step and the discounted expectation of the next state value. Based on this recursive relationship, an iterative estimation method for value functions can be constructed. Under policy , and satisfy
and
In policy updates, to reduce the estimation variance based on , the advantage function is often defined to measure the return gain of an action relative to the average behavior of the policy
where indicates that action a outperforms the average level of the current policy at state s; conversely, it means the action is not conducive to improving long-term returns.
Since the environment dynamics are unknown or difficult to model accurately, value functions are usually learned approximately by sampling interaction data. Temporal-Difference (TD) learning adopts the idea of “bootstrapping”, taking the current reward and the value estimation of the next state as the learning target [38]. Taking the state value as an example, the one-step TD error is
Correspondingly, the action value function can also construct the update signal with the one-step target
In deep reinforcement learning, neural networks are usually used for function approximation of or , and network parameters are updated by minimizing the mean square form of the TD error, thereby realizing iterative estimation of the value function. For continuous action space problems, parameterized policy is commonly used and policy parameters are directly optimized through the policy gradient method. Using the advantage function can obtain a more stable form of policy gradient:
The Actor–Critic framework combines policy learning and value evaluation [39]: the Actor outputs the policy and generates actions, while the Critic estimates the value function and provides learning signals for the Actor. The Critic can be updated by minimizing the square of the TD error:
The Actor can achieve policy improvement by maximizing the expectation of the advantage:
where is the advantage estimator.
In complex continuous control tasks, to enhance exploration ability and improve policy robustness, maximum entropy reinforcement learning adds a policy entropy regularization term to the optimization objective, so that the policy maintains a certain degree of randomness while obtaining high returns [40]. Its objective can be expressed as
where the policy entropy is defined as
and is the temperature coefficient, used to balance return maximization and explorability. The SAC algorithm constructs soft value functions based on the maximum entropy idea and adopts a stochastic policy update mechanism, which has favorable sample efficiency and training stability in continuous action space tasks.
3.2. State Space and Action Space Modeling
To realize fine-grained control of the relay UAV in the continuous action space and transform the state estimation results of Kalman filtering into decision-making basis, this section models the communication relay path planning problem as a Markov decision process. Different from traditional methods that directly take raw sensor data with Gaussian noise as state input, the state space constructed in this section explicitly includes the optimal posterior estimation of ground nodes and their motion trend information calculated by the filtering algorithm. This design effectively transforms the passive tracking problem in partially observable environments into an active prediction problem based on high-confidence estimated states.
3.2.1. State Space Modeling
In the design of state space, to reduce the influence of observation noise and better characterize the motion states of the UGVs, the posterior kinematic estimates provided by the Kalman filter are incorporated into the state representation of the SAC agent.
where the UAV’s own state is , including its normalized two-dimensional coordinates in the mission area, current flight speed and heading angle. The core innovation of this state model lies in the construction of the ground cluster estimation state . The agent adopts the optimal posterior estimation values of the Kalman filter after time update and measurement update. For a UGV swarm with N nodes, its estimated state set is:
This set contains two key dimensions of information: First, the filtered and smoothed position estimation . Compared with raw observations, this estimation eliminates the influence of random measurement noise and can more accurately reflect the true distribution of nodes in complex terrain, thereby improving the reliability of air-to-ground Line-of-Sight (LoS) probability calculation. Second, the velocity vector estimation resolved by filtering. To accelerate the feature extraction and convergence of the neural network, the position coordinates in all the above state components are normalized based on the size of the mission area.
3.2.2. Action Space Modeling
For the design of action space , considering the non-holonomic constraint characteristics of fixed-wing UAVs with minimum flight speed and minimum turning radius limits, this section designs a continuous action space including speed control and heading control. The action vector output by the SAC policy network is
where the vector consists of two normalized components, both with a value range of . First, since the fixed-wing UAV must maintain a certain airspeed to generate lift and cannot hover like rotary-wing aircraft, the normalized speed action is linearly mapped into the physical speed range of the UAV, and the actual speed control command is calculated as
This mapping ensures that the UAV always operates within safe flight limits, and allows it to adjust its speed to adapt to the change rate of the ground cluster topology.
Second, the heading control component determines the turning maneuverability of the UAV. Limited by the maximum roll angle of the airframe, the heading angular velocity of the fixed-wing UAV has a physical upper limit
and the actual angular velocity command is mapped as
By outputting continuous angular velocity commands, the agent can control the UAV to realize the optimal relay trajectory while satisfying the physical constraint of minimum turning radius . In each control cycle, the Kalman filter module first processes the UGV swarm data to construct a high-confidence state, then the SAC network generates action commands based on this state, and finally drives the UAV to perform the optimal relay task in a dynamic environment.
3.3. Reward Function Design
This section constructs the reward function based on the link communication cost defined in the second section of this section. Since the goal of reinforcement learning is to maximize cumulative rewards, while the goal of communication tasks is to minimize network transmission costs, this paper defines the communication reward as a negative correlation function of communication cost, guiding the agent to minimize the global communication cost (i.e., maximize signal quality).
First, for the optimization of communication performance, the primary task of the UAV is to provide high-quality relay links for obstructed ground nodes. According to the previous definition, the communication cost weight of link has a nonlinear inverse relationship with the received signal strength: the lower the value, the better the channel quality. To guide the UAV to actively search for and maintain high-quality links, the communication reward term is designed as the negative value of the sum of communication costs of all relay-required links at the current moment. This dense reward signal can effectively overcome the problem of difficult policy convergence under traditional sparse rewards, and continuously motivate the agent to approach the optimal relay position with low cost and high signal strength.
Second, in addition to communication performance, path planning should consider the maneuvering energy consumption and flight smoothness of the fixed-wing UAV. Frequent rapid acceleration/deceleration or high-g turns will not only sharply increase flight energy consumption and shorten mission endurance, but also may cause unstable attitude of the communication antenna. Therefore, this section constructs an energy consumption penalty term based on the normalized control variables output from the action space. This term consists of the sum of squares of the normalized speed action and heading angular velocity action, weighted by respective coefficients. By deducting this maneuver cost from the reward function, the agent is prompted to prioritize planning flight trajectories with gentle changes and low energy consumption on the premise of meeting communication coverage requirements, so as to avoid paying excessive maneuver costs for minor communication gains. In addition, to ensure that the UAV always performs tasks within the preset tactical area and prevent the agent from flying out of the effective communication area during exploration, an out-of-bounds penalty mechanism is designed. Once the position coordinates of the UAV exceed the mission boundary , the system will impose a large negative reward penalty and force the current episode to terminate, thus limiting the agent’s exploration range to safe airspace through soft constraints at the algorithm level.
In summary, the single-step total reward function is defined as a weighted combination of the above communication gain, action penalty and boundary constraint:
where and are weight factors, used to dynamically adjust the priority between communication quality and maneuver cost.
Guided by this reward function, the KF-SAC algorithm can automatically balance signal strength and flight cost in complex dynamic environments, and search for an efficient relay strategy that can significantly reduce global communication cost while maintaining a smooth and safe flight trajectory.
3.4. KF-SAC Algorithm
Based on the filter-enhanced state space model and multi-objective communication reward function constructed above, this section formally proposes a cooperative communication relay algorithm, KF-SAC. The algorithm aims to construct a closed-loop control framework from estimation to decision-making. By deeply coupling the model-based Kalman filter state estimator with the model-free reinforcement learning policy network, the algorithm can extract high-confidence motion trends from noisy sensor data in complex dynamic terrain, and accordingly generate a robust flight control strategy that balances exploration and exploitation. The algorithm structure is shown in Figure 2.
Figure 2.
KF-SAC algorithm architecture diagram.
At the perception front-end of the algorithm, the state estimation subsystem is responsible for solving the problem of accurate positioning under the interference of observation noise. Considering that UGVs follow the smooth turning motion law, the system models them as a linear Gaussian dynamic system, and uses a Kalman filter to recursively calculate their posterior probability density. This process first performs the time update step: the filter performs prior prediction of the current state based on the posterior estimation and covariance matrix of the previous moment, using the state transition matrix derived from the discretization of the UGV kinematic equations:
where represents the process noise of uncertainty during the maneuver of the UGV. This prediction process deduces the motion trend of nodes at the physical level and provides a prior benchmark for subsequent correction.
Then, the system introduces the noisy sensor observation affected by multipath effect and occlusion at the current moment, and enters the measurement update phase. The filter determines the trust weight for observations and predictions by calculating the Kalman gain , and updates the state estimation and error covariance:
where is the observation matrix and is the observation noise covariance.
Through this correction process, the filter not only effectively smooths the random jitter of position coordinates, but more importantly, resolves the implicit velocity vector and angular velocity information from the noisy displacement difference. The finally constructed enhanced state vector has an extremely high signal-to-noise ratio, providing a reliable physical basis for subsequent reinforcement learning decision-making.
After obtaining high-precision state estimation, the control architecture shifts to the decision-making link based on maximum entropy policy optimization. Different from deterministic policy gradient algorithms, the core objective of the SAC algorithm is to maximize the long-term cumulative reward while maximizing the entropy of the policy, thereby encouraging the agent to conduct extensive random exploration in the early training stage and preventing the policy from converging to a local optimum prematurely. This maximum entropy objective function is defined as:
where is the discount factor, is the entropy of the policy, and is the temperature coefficient, which is used to dynamically adjust the weight between exploration entropy and exploitation reward.
To achieve this objective, the algorithm adopts the Actor–Critic architecture to train the policy network and value network in parallel. In terms of value evaluation, to solve the Q-value overestimation problem, the system introduces a clipped double Q-learning mechanism, maintains two independent Critic networks , and takes their minimum value as the target value. The parameters of the Critic network are updated by minimizing the soft Bellman residual, and its loss function is:
where the target value , combined with the entropy gain of the next moment, is calculated as:
Here, the target network parameters are iterated by a moving average with a soft update strategy to ensure training stability.
Meanwhile, the parameter optimization of the Actor policy network aims to output an action distribution that maximizes the sum of the expected Q value and entropy under this distribution. To realize gradient backpropagation in the continuous action space, the algorithm adopts the reparameterization trick. Specifically, the policy network outputs the mean and standard deviation of the Gaussian distribution, and the actual action is obtained by introducing standard Gaussian noise for sampling and mapping through the Tanh activation function
This transformation makes the sampled action a differentiable function with respect to the network parameter . The loss function of the Actor is defined as:
By minimizing this loss, the policy network can automatically adjust the action distribution to shift it toward high-value regions. By alternately performing the above state estimation update, Critic value evaluation and Actor policy improvement, the KF-SAC algorithm enhances the perception of dynamic environments using the velocity trend information provided by Kalman filtering in the data flow, and ensures policy robustness through the maximum entropy mechanism in the gradient flow, so that the UAV can automatically learn an efficient relay strategy that balances communication quality and flight stability. The algorithm is summarized in Algorithm 1.
| Algorithm 1 KF-SAC-based UAV-UGV swarm communication relay algorithm in complex terrain. |
|
4. Experiments and Analysis
This section aims to verify the effectiveness of the proposed cooperative communication relay method for UAV-to-UGV swarms in complex terrain based on the KF-SAC algorithm. The experimental verification is carried out from two perspectives: first, neural network training is performed in a Python simulation environment, focusing on verifying the convergence performance of the algorithm; second, engineering visual simulation is conducted using UAV relay simulation software developed with C++ 17/Qt 6.7.2, to intuitively demonstrate the real-time decision-making performance and communication link maintenance capability of the algorithm in dynamic relay tasks.
4.1. Simulation Environment Setup and Parameter Configuration
The training platform in this experiment is built based on Python 3.9 and the PyTorch 1.12.1 deep learning framework. In addition, this section develops an application for the UAV-to-UGV swarm cooperative communication relay method in complex terrain based on the cross-platform C++ application development framework Qt, to further implement engineering simulation. The specific parameters are listed in Table 1.
Table 1.
Parameters of the Qt application.
4.2. Algorithm Performance Comparison Experiment
In order to evaluate the performance of the KF-SAC algorithm in complex terrains, the SAC algorithm was selected as the benchmark for comparison. The experimental analysis focused on the training performance and communication overhead of the relay system. The neural network structure and the main hyperparameters of the KF-SAC algorithm are summarized in Table 2. In terms of network architecture, both the Actor policy network and the Critic value network adopt a two-layer Multi-Layer Perceptron (MLP) with 256 neurons per hidden layer, to ensure sufficient nonlinear feature extraction and fitting capability for the heterogeneous state space.
Table 2.
Network structure and training hyperparameter settings.
For the designed complex terrain scenario, four types of regional terrain are considered, and the corresponding channel parameters for different regions are shown in Table 3. Where represents the ratio of built-up area to total land area, represents the number of buildings per square kilometer, and is a scaling parameter describing the Rayleigh distribution of building heights. The parameters a and b are the context-dependent fitting parameters of the LoS probability model.
Table 3.
Channel parameters of different terrain regions.
The experimental results are shown in Figure 3. The reward curve illustrates the performance comparison between the KF-SAC algorithm and the traditional SAC algorithm in the UAV relay communication scenario over 1000 training episodes. Although the initial reward of KF-SAC (red line) is slightly lower than that of SAC, it starts to rise sharply at around 50 episodes with a significantly steeper slope than SAC (blue line). This is because the introduction of Kalman filtering for state reconstruction can rapidly identify the motion vectors of UGVs from noisy observations.
Figure 3.
Comparison of average reward curves between KF-SAC and baseline SAC algorithm.
After 650 episodes, the red line shows almost no significant performance degradation, while the blue line still exhibits small-range sawtooth-like fluctuations. In the reported training run, the KF-SAC reward curve exhibits smaller fluctuations during the later stage of training. At around 800 episodes, KF-SAC basically enters the convergence stage, while SAC still maintains high volatility, and its peak reward is still lower than the convergence value of KF-SAC, showing a more stable late-stage training trend in the reported experiment. In practical tasks, this means KF-SAC can sustain efficient communication for a longer duration, effectively solving the problem of delayed maneuver response in highly dynamic scenarios.
4.3. Communication Relay Simulation Platform and Experimental Analysis
The simulation platform is developed based on the Qt framework, integrating functional modules including UAV communication relay, UGV swarm maneuvering, and real-time communication link quality calculation. The trained KF-SAC model is loaded in the background via PyTorch, realizing a software-level closed-loop simulation from relay decision-making to trajectory and communication-link visualization.
The interface structure is shown in Figure 4. The left side of the interface serves as the real-time simulation display area, dynamically presenting the motion trajectories, relative positions, and communication links between the relay UAV and the UGV swarm in complex terrain. The right side contains the parameter configuration panel, which consists of three main functional modules arranged from top to bottom. The environment parameter configuration module allows the center coordinates and coverage radii of different terrain regions to be customized, thereby enabling the simulation of various complex terrain environments. The UAV parameter configuration module is used to initialize the kinematic state of the relay UAV, including its initial coordinates, cruise speed, and yaw angle. The UGV parameter configuration module allows the swarm size and the motion states of individual UGVs to be configured.
Figure 4.
Interface of the simulation platform.
After configuring the above initial conditions, clicking the “Run” button at the bottom activates the background KF-SAC algorithm engine and starts the deduction and verification of the communication relay task.
The simulation experiment parameters are listed in Table 4, and the simulation results are shown in Figure 5 and Figure 6.
Table 4.
Simulation parameter settings.
Figure 5.
Trajectory snapshots of the first 100 steps in KF-SAC simulation.
Figure 6.
Yaw rate control command of the relay UAV.
In Figure 5, the blue circle denotes the UAV, the yellow circles with numbers denote UGVs of different IDs, and orange circles with different shades denote different terrain regions. The magenta solid lines represent the established communication links between the UAV and UGVs, while the light purple solid lines represent the communication links between UGVs. The blue dashed lines and red dashed lines represent disconnected links among UGVs and between UGVs and the UAV, respectively. The blue and yellow solid lines represent the motion trajectories of the UAV and UGVs, respectively.
As can be observed from the figure, at around the 10th step (Figure 5b), the UAV responds quickly and adjusts its heading according to the cluster position, maintaining the communication cost at a low level. During the 30th to 60th steps (Figure 5c,d), facing the steering and acceleration of UGVs, the UAV predicts their future positions via Kalman filtering and maneuvers in advance. Without such prediction, the communication cost would deteriorate significantly in this stage. During the 80th to 100th steps (Figure 5e,f), the system enters a steady state. Despite potential LoS occlusion caused by terrain, the UAV always stays at positions with low overall communication cost, which verifies the active guarantee capability of the proposed algorithm.
Figure 6 presents the variation of the UAV yaw rate control command during the communication relay process. Since the UGV swarm keeps maneuvering, the communication topology of the system changes dynamically, leading to continuous real-time offset of the global optimal relay position in space. To maintain optimal link quality, the UAV has to generate continuous yaw correction commands to follow the optimal position dynamically. Nevertheless, the reasonable fluctuation is always constrained within a very narrow range, presenting a smooth and continuous transition characteristic.
This fully proves that the closed-loop relay algorithm proposed in this section not only has strong real-time robust tracking ability for highly dynamic maneuvering targets, but also outputs control commands that fully comply with the physical dynamic constraints of the UAV flight control servo, possessing high feasibility for engineering deployment. It should be noted that due to the high maneuverability of the UGV swarm, the yaw angle control command does not converge to a fixed value.
The performance difference between KF-SAC and SAC can also be observed from Figure 7. Starting from around the 10th step, the communication cost of the UAV increases abruptly due to the motion variation of the UGV swarm. Benefiting from the posterior kinematic information provided by the Kalman filter, the KF-SAC algorithm gradually adjusts the UAV position during the rising phase of the communication cost. As a result, the communication cost stops increasing at around the 25th step and remains relatively stable afterwards in the reported simulation. In contrast, SAC directly relies on noisy UGV observations and shows a slower response to the motion variation of the UGV swarm in this evaluation. Consequently, its communication cost rises above 9 and remains at a higher level during the subsequent simulation steps.
Figure 7.
Comparison of communication cost between KF-SAC and SAC.
5. Conclusions
In this paper, the UAV-assisted communication relaying problem for a mobile UGV swarm is modeled as a Markov decision process with multiple objectives and constraints. Additionally, the current results are based on simulation, so they do not constitute field verification. Future work will investigate multi-drone cooperative relaying, more diverse motion and channel conditions, different noise and covariance settings, long-term energy-aware operations, multiple sub-statistical evaluations, and channel calibration based on hardware-in-the-loop and field verification, tailored for specific environments. In order to alleviate the fluctuation of relay decision caused by UGV mobility and noisy state observation, this paper proposes a KF-SAC algorithm by combining the Kalman filter with the SAC framework. The Kalman filter provides a more reliable a posteriori kinematics estimate for the relay decision, thus improving the state information available to the SAC agent in a dynamic noise environment. In this paper, numerical simulations and QT-based simulation platforms are used to evaluate the proposed method. Under the considered simulation conditions, the KF-SAC algorithm has better training performance and lower communication cost compared to the standard SAC algorithm.
However, the method still has some limitations. The current framework only considers a single relay UAV and employs a simplified UGV motion model and a large-scale A2G/G2G channel model without explicitly modeling the physical layer effects such as Doppler effects and MIMO. This evaluation uses a fixed Kalman filter covariance setting and limited simulation time limit, does not consider the limitations of UAV battery or flight duration, and does not provide statistical features based on multiple independent random seeds. Moreover, the current results are based on simulation and therefore do not constitute field validation. Future work will investigate multi-UAV cooperative relaying, more diverse motion and channel conditions, different noise and covariance settings, long-term energy-aware operation, multi-seed statistical evaluation, and channel calibration for specific environments based on hardware in the loop and in the field validation.
Author Contributions
Conceptualization, Z.M., Y.P., L.D., Q.S. and H.S.; Methodology, Z.M., Y.P. and L.D.; Software, Z.M., Y.P. and L.D.; Validation, Z.M., Y.P. and L.D.; Formal analysis, Y.P.; Investigation, Z.M., Y.P. and L.D.; Writing—original draft, Z.M.; Visualization, Z.M., Y.P. and L.D.; Supervision, Q.S. and H.S.; Project administration, H.S.; Funding acquisition, H.S. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by the National Natural Science Foundation of China under Grant No. 62473210 and the Natural Science Foundation of Tianjin under Grant No. 2025JCQNJC01350.
Data Availability Statement
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Munasinghe, I.; Perera, A.; Deo, R.C. A Comprehensive Review of UAV-UGV Collaboration: Advancements and Challenges. J. Sens. Actuator Netw. 2024, 13, 81. [Google Scholar] [CrossRef] [Scilit]
- Liu, C.; Zhao, J.; Sun, N. A review of collaborative air-ground robots research. J. Intell. Robot. Syst. 2022, 106, 60. [Google Scholar] [CrossRef] [Scilit]
- Chen, J.; Zhang, X.; Xin, B.; Fang, H. Coordination Between Unmanned Aerial and Ground Vehicles: A Taxonomy and Optimization Perspective. IEEE Trans. Cybern. 2016, 46, 959–972. [Google Scholar] [CrossRef] [Scilit]
- Chai, R.; Guo, Y.; Zuo, Z.; Chen, K.; Shin, H.S.; Tsourdos, A. Cooperative motion planning and control for aerial-ground autonomous systems: Methods and applications. Prog. Aerosp. Sci. 2024, 146, 101005. [Google Scholar] [CrossRef] [Scilit]
- Javaid, S.; Saeed, N.; Qadir, Z.; Fahim, H.; He, B.; Song, H.; Bilal, M. Communication and Control in Collaborative UAVs: Recent Advances and Future Trends. IEEE Trans. Intell. Transp. Syst. 2023, 24, 5719–5739. [Google Scholar] [CrossRef] [Scilit]
- Lin, X.; Huang, M. An Autonomous Cooperative Navigation Approach for Multiple Unmanned Ground Vehicles in a Variable Communication Environment. Electronics 2024, 13, 3028. [Google Scholar] [CrossRef] [Scilit]
- Bekmezci, İ.; Sahingoz, O.K.; Temel, Ş. Flying Ad-Hoc Networks (FANETs): A survey. Ad Hoc Netw. 2013, 11, 1254–1270. [Google Scholar] [CrossRef] [Scilit]
- Huang, Z.; Wu, W.; Fu, C.; Liu, X.; Shan, F.; Wang, J.; Xu, X. Communication-Topology-preserving Motion Planning: Enabling Static Routing in UAV Networks. ACM Trans. Sen. Netw. 2023, 20, 1–39. [Google Scholar] [CrossRef] [Scilit]
- Gao, Y.; Chen, H.; Li, Y.; Lyu, C.; Liu, Y. Autonomous Wi-Fi Relay Placement With Mobile Robots. IEEE/ASME Trans. Mechatron. 2017, 22, 2532–2542. [Google Scholar] [CrossRef] [Scilit]
- Kim, J.; Ladosz, P.; Oh, H. Optimal communication relay positioning in mobile multi-node networks. Robot. Auton. Syst. 2020, 129, 103517. [Google Scholar] [CrossRef] [Scilit]
- Zeng, Y.; Zhang, R.; Lim, T.J. Wireless communications with unmanned aerial vehicles: Opportunities and challenges. IEEE Commun. Mag. 2016, 54, 36–42. [Google Scholar] [CrossRef] [Scilit]
- Tran, D.H.; Nguyen, V.D.; Chatzinotas, S.; Vu, T.X.; Ottersten, B. UAV Relay-Assisted Emergency Communications in IoT Networks: Resource Allocation and Trajectory Optimization. IEEE Trans. Wirel. Commun. 2022, 21, 1621–1637. [Google Scholar] [CrossRef] [Scilit]
- Tao, C.; Liu, B. Online Autonomous Motion Control of Communication-Relay UAV with Channel Prediction in Dynamic Urban Environments. Drones 2024, 8, 771. [Google Scholar] [CrossRef] [Scilit]
- Ladosz, P.; Oh, H.; Chen, W.H. Trajectory planning for communication relay unmanned aerial vehicles in urban dynamic environments. J. Intell. Robot. Syst. 2018, 89, 7–25. [Google Scholar] [CrossRef] [Scilit]
- Ho, L.; Jangsher, S. UAV Trajectory Optimization based on Predicted User Locations. In Proceedings of the 2024 IEEE Wireless Communications and Networking Conference (WCNC), Dubai, United Arab Emirates, 21–24 April 2024; pp. 1–6. [Google Scholar] [CrossRef] [Scilit]
- Sun, R.; Matolak, D.W. Air–Ground Channel Characterization for Unmanned Aircraft Systems Part II: Hilly and Mountainous Settings. IEEE Trans. Veh. Technol. 2017, 66, 1913–1925. [Google Scholar] [CrossRef] [Scilit]
- Barrow, J.D.; Bhavsar, S.P.; Sonoda, D.H. Minimal spanning trees, filaments and galaxy clustering. Mon. Not. R. Astron. Soc. 1985, 216, 17–35. [Google Scholar] [CrossRef] [Scilit]
- Tao, C.; Zhou, R. A method of UAV motion control to optimize air-ground relay network. Syst. Eng. Electron. 2024, 46, 1712–1723. [Google Scholar]
- Stoer, M.; Wagner, F. A simple min-cut algorithm. J. ACM 1997, 44, 585–591. [Google Scholar] [CrossRef] [Scilit]
- Ford, L.R.; Fulkerson, D.R. Maximal Flow Through a Network. Can. J. Math. 1956, 8, 399–404. [Google Scholar] [CrossRef] [Scilit]
- Nasrollahi, S.; Mirrezaei, S.M. Toward UAV-based communication: Improving throughput by optimum trajectory and power allocation. EURASIP J. Wirel. Commun. Netw. 2022, 2022, 9. [Google Scholar] [CrossRef] [Scilit]
- Xia, N.; Liu, Y.; Yu, Y. An Elevation-Aware Large-Scale Channel Model for UAV Air-to-Ground Links. Mathematics 2025, 13, 3377. [Google Scholar] [CrossRef] [Scilit]
- Su, Y.; Liwang, M.; Chen, Z.; Du, X. Toward Optimal Deployment of UAV Relays in UAV-Assisted Internet of Vehicles. IEEE Trans. Veh. Technol. 2023, 72, 13392–13405. [Google Scholar] [CrossRef] [Scilit]
- Lee, I.; Babu, V.; Caesar, M.; Nicol, D. Deep reinforcement learning for UAV-assisted emergency response. In Proceedings of the MobiQuitous 2020-17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, Virtual, 7–9 December 2020; pp. 327–336. [Google Scholar]
- Abohashish, S.M.; Rizk, R.Y.; Elsedimy, E. Trajectory optimization for UAV-assisted relay over 5G networks based on reinforcement learning framework. EURASIP J. Wirel. Commun. Netw. 2023, 2023, 55. [Google Scholar] [CrossRef] [Scilit]
- Zhang, C.; Li, X.; He, C.; Li, X.; Lin, D. Trajectory optimization for UAV-enabled relaying with reinforcement learning. Digit. Commun. Netw. 2025, 11, 200–209. [Google Scholar] [CrossRef] [Scilit]
- Lu, X.; Xiao, L.; Dai, C.; Dai, H. UAV-Aided Cellular Communications with Deep Reinforcement Learning Against Jamming. IEEE Wirel. Commun. 2020, 27, 48–53. [Google Scholar] [CrossRef] [Scilit]
- Deng, X.; Deng, Z.; Zheng, L.; Yang, C.; Chen, C. Optimization of D2D emergency communication resources for UAV relay based on DA-STD3. EURASIP J. Wirel. Commun. Netw. 2024, 2024, 88. [Google Scholar] [CrossRef] [Scilit]
- Ding, R.; Chen, J.; Wu, W.; Liu, J.; Gao, F.; Shen, X. Packet Routing in Dynamic Multi-Hop UAV Relay Network: A Multi-Agent Learning Approach. IEEE Trans. Veh. Technol. 2022, 71, 10059–10072. [Google Scholar] [CrossRef] [Scilit]
- Luong, P.; Gagnon, F.; Tran, L.N.; Labeau, F. Deep Reinforcement Learning-Based Resource Allocation in Cooperative UAV-Assisted Wireless Networks. IEEE Trans. Wirel. Commun. 2021, 20, 7610–7625. [Google Scholar] [CrossRef] [Scilit]
- Wang, L.; Wang, K.; Pan, C.; Xu, W.; Aslam, N.; Hanzo, L. Multi-Agent Deep Reinforcement Learning-Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing. IEEE Trans. Cogn. Commun. Netw. 2021, 7, 73–84. [Google Scholar] [CrossRef] [Scilit]
- Li, D.; Du, B.; Bai, Z. Deep Reinforcement Learning-Enabled Trajectory and Bandwidth Allocation Optimization for UAV-Assisted Integrated Sensing and Covert Communication. Drones 2025, 9, 160. [Google Scholar] [CrossRef] [Scilit]
- Qi, N.; Jiang, H.; Xiao, S.; Wu, D.; Zhou, F.; Li, C.; Jin, S. Robust and Energy-Efficient Multi-UAV Trajectory Planning for Data Collection: A Game-Theoretic and Deep Reinforcement Learning Approach. IEEE Trans. Green Commun. Netw. 2026, 10, 3086–3101. [Google Scholar] [CrossRef] [Scilit]
- Tang, X.; Chen, Q.; Weng, W.; Jin, C.; Liu, Z.; Wang, J.; Sun, G.; Li, X.; Niyato, D. Task Assignment and Exploration Optimization for Low Altitude UAV Rescue via Generative AI Enhanced Multi-Agent Reinforcement Learning. IEEE Trans. Mob. Comput. 2026, 25, 627–643. [Google Scholar] [CrossRef] [Scilit]
- Jiang, H.; Guo, J.; Xiao, Z.; Yang, K.; Li, T.; Li, B. Collaborative Perception and Computing Offloading in 6G Air-Ground Integrated Networks. IEEE J. Sel. Areas Commun. 2026, 44, 2821–2838. [Google Scholar] [CrossRef] [Scilit]
- Wan, Y.; Namuduri, K.; Zhou, Y.; Fu, S. A Smooth-Turn Mobility Model for Airborne Networks. IEEE Trans. Veh. Technol. 2013, 62, 3359–3370. [Google Scholar] [CrossRef] [Scilit]
- Al-Hourani, A.; Kandeepan, S.; Lardner, S. Optimal LAP Altitude for Maximum Coverage. IEEE Wirel. Commun. Lett. 2014, 3, 569–572. [Google Scholar] [CrossRef] [Scilit]
- Tesauro, G. Temporal difference learning and TD-Gammon. Commun. ACM 1995, 38, 58–68. [Google Scholar] [CrossRef] [Scilit]
- Konda, V.; Tsitsiklis, J. Actor-critic algorithms. Adv. Neural Inf. Process. Syst. 1999, 12. [Google Scholar]
- Ahmed, Z.; Le Roux, N.; Norouzi, M.; Schuurmans, D. Understanding the impact of entropy on policy optimization. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019; pp. 151–160. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.






