Hierarchical Cooperative Trajectory Planning for Air–Ground Robotic Systems in Communication-Constrained Urban Canyons

Abstract

Heterogeneous airground robotic systems, which integrate unmanned ground vehicles and unmanned aerial vehicles, have shown significant potential in complex autonomous missions. However, when deployed in urban canyons, dense high-rise buildings impose severe communication constraints on ground vehicles, necessitating the introduction of aerial vehicles as relays to maintain reliable connectivity. The resulting cooperative trajectory planning problem is challenging for three reasons. First, the kinematic and communication constraints are tightly coupled. Second, the optimization landscape is highly non-convex and non-differentiable. Third, the planner must balance topological exploration with real-time efficiency. To address these challenges, we propose a hierarchical cooperative trajectory planning framework for an air–ground robotic system. Specifically, in the upper layer, a heuristic-search-guided reinforcement learning mechanism is employed to narrow the search space and circumvent the sparse reward problem, rapidly generating an initial solution. Subsequently, the lower-layer planner utilizes an optimization-based solver, together with a corridor-based constraint formulation method, to refine the initial solution into a kinematically feasible cooperative trajectory. Ultimately, this strategy improves real-time efficiency while improving the quality of feasible cooperative trajectories. Extensive ablation studies and comparative experiments with representative baselines demonstrate that the proposed framework improves collision avoidance, communication reliability, trajectory smoothness, and computational efficiency in the tested urban canyon scenarios.

1. Introduction

Heterogeneous multi-robot collaboration, driven by recent advancements in autonomous robotics, has emerged as a promising paradigm for multi-agent trajectory planning [1]. Among various configurations, the air–ground cooperative system—integrating UAVs and UGVs—has attracted considerable attention due to its complementary mobility and payload capabilities. Accordingly, cooperative trajectory planning for such systems has become a critical research focus, with broad applications in last-mile delivery [2], disaster search and rescue [3], and large-scale environmental monitoring [4].

However, deploying UGV-UAV systems in dense urban areas introduces unique spatial challenges, especially in urban canyons. An urban canyon typically refers to a narrow street flanked by continuous high-rise buildings [5]. Such environments can severely degrade UGV communication due to non-line-of-sight (NLoS) propagation [6] and multipath fading [7], which consequently hinder reliable data exchange and real-time localization. To address this issue, a UAV can be deployed as an aerial relay. Benefiting from its high mobility and elevated position, the UAV can help re-establish a stable communication link between the UGV and external infrastructure, as shown in Figure 1. Therefore, a tailored cooperative planning framework is important for supporting both operational safety and planning efficiency in such constrained environments.

In the proposed system, the UAV is assumed to be docked on the UGV and is deployed as an aerial relay only when the UGV encounters a communication failure. Once deployed, the UAV is assumed to maintain connectivity with external infrastructure, thereby providing a reliable communication link for the integrated system. In our study, the LoS/NLoS state is used as a binary geometric connectivity indicator at the trajectory planning level, mainly to capture building-induced blockage between the UGV and UAV. This assumption is suitable when blockage is the dominant factor affecting communication and the relay distance remains within the operational range. However, it may become invalid when distance-dependent path loss, severe multipath fading, strong interference, dynamic blockage, adverse weather, or antenna-related constraints dominate the link quality. In such cases, a more detailed physical-layer channel model should be incorporated for link-quality-aware planning.

Despite the advantages of the UGV-UAV system, cooperative trajectory planning for the UGV-UAV system in an urban canyon still faces the following three core challenges.

First, cooperative trajectory planning for the UGV-UAV under multiple constraints in urban canyons presents significant challenges. In such complex 3D environments, both the UGV and the UAV must avoid obstacles. They also need to satisfy their own kinematic constraints and maintain a stable communication link. Under such multiple constraints, we need to generate a high-quality cooperative trajectory, which remains a challenging task for this research.

Second, the optimization problem formulated from the aforementioned complex scenarios exhibits pronounced non-convexity and non-differentiability. Urban canyons contain complex obstacle topologies. In addition, the UGV and UAV are coupled through kinematic and communication constraints. These factors create many local minima in the high-dimensional search space and make the problem highly non-convex. Furthermore, the inclusion of hard physical limits within the kinematic and communication constraints inevitably introduces non-smooth functions into the optimization framework, leading to a high degree of non-differentiability.

Third, striking a balance between topological exploration and real-time efficiency poses a significant challenge when solving the aforementioned complex optimization problem. To improve topological search capability, the computational complexity of conventional trajectory-searching algorithms increases substantially within the intricate topology of urban canyons, thereby potentially compromising real-time performance. Conversely, traditional optimization-based methods are often used to maintain real-time responsiveness. However, in complex environments, they are prone to local optima. This may degrade the quality and feasibility of the resulting trajectories.

To address the challenges outlined above, we propose a hierarchical framework. In the first stage, a guided RL algorithm is developed to navigate the strong non-convexity and non-differentiability of the optimization problem by leveraging the heuristic nature of the reward function, thereby yielding an initial solution. In the second stage, this solution serves as a warm start for an optimization-based solver, which refines the trajectory for smoothness and obstacle avoidance, enhancing real-time computational efficiency. The main contributions of this paper are summarized in the following three aspects.

• The UGV-UAV cooperative trajectory planning problem is decoupled into a hierarchical computational framework. A guided RL algorithm first derives a high-quality initial solution that mitigates the adverse effects of non-convexity and non-differentiability. This initial solution then serves as a warm start for an optimization-based solver, thereby enhancing real-time performance while improving the quality of feasible solutions.
• We introduce a heuristic search algorithm as prior guidance for the RL agent. The heuristic search provides feasible reference paths in complex topological spaces. Based on this guidance, the RL agent explores a reduced search space. This design helps alleviate the sparse-reward problem caused by complex obstacle configurations and improves the efficiency of initial solution generation.
• To further enhance solver efficiency, a corridor-based constraint formulation method is integrated into the optimization-based solver. This method transforms the original non-convex obstacle avoidance constraints into a set of convex safe corridors, enabling the solver to operate within convex domains and reducing computational overhead for real-time trajectory generation.

The rest of this article is structured as follows. Section 2 reviews the primary research leading up to cooperative trajectory planning of the UGV-UAV system. Section 3 provides the problem statement of our planning task. Section 4 introduces our framework from a macroscopic perspective, while Section 5 elaborates on our framework, including the core motivations and algorithms. Section 6 presents the simulation results and experimental results. Section 7 discusses the necessity of the proposed hierarchical framework through ablation studies and analyzes the limitations of the current work. Finally, Section 8 concludes the paper and outlines future research.

2. Related Research

This Section reviews studies related to cooperative trajectory planning for UGV-UAV systems. We first discuss trajectory planning methods for individual UGVs and UAVs. These methods provide the methodological basis for UGV-UAV cooperative planning. We review the research on cooperative trajectory planning of the UGV-UAV system. Additionally, the primary research within the context of urban canyon environments is introduced.

2.1. Trajectory Planning for UGVs or UAVs

This section primarily categorizes existing research on trajectory planning techniques for UGVs and UAVs into three major paradigms: search-based methods, optimization-based methods, and learning-based methods.

First, search-based methods operate by discretizing the environment into a graph or grid, utilizing graph-search algorithms to systematically find a collision-free geometric path. Specifically, Meng et al. [8] proposed a parameter-optimized Artificial Potential Field A* algorithm for unmanned aerial vehicles in workshop environments, which enhances the standard A* approach by integrating a variable step-size strategy and potential field values into the cost function to dynamically improve planning accuracy and flight stability.

Second, optimization-based methods formulate trajectory generation as a mathematical programming problem, optimizing objectives such as smoothness while strictly satisfying the vehicle’s kinematic constraints. Specifically, for UGVs, Qiao et al. [9] and Bao et al. [10] utilized mathematical programming to generate smooth, kinematically feasible trajectories by balancing energy efficiency and integrating landmark-based localization, respectively. Extending these principles to network scale, Chen et al. [11] formulated a coordinated routing policy for connected vehicles to optimize system-wide travel efficiency alongside individual trajectory smoothness. Furthermore, for UAVs, Wu et al. [12] developed a trajectory-based flight scheduling framework for UAVs in AirMetro systems, mathematically guaranteeing collision-free and efficient urban airspace operations.

Third, learning-based methods leverage data-driven models to map environmental observations directly to planning decisions, enabling highly adaptive and real-time responses in complex scenarios. Specifically, in the context of UGVs, Ayalew et al. [13] applied deep RL techniques to mobile robots, demonstrating how data-driven policies can dynamically adjust paths to handle environmental uncertainties and dynamic obstacles. Pushing the boundaries of these learning models to UAVs, Du et al. [14] proposed a cooperative pursuit framework for intercepting unauthorized UAVs in urban airspace, showcasing the real-time decision-making and adaptive capabilities of learning algorithms in highly dynamic, multi-agent adversarial environments.

2.2. Cooperative Trajectory Planning of UGV-UAV System

This section primarily categorizes existing research on cooperative trajectory planning of UGV-UAV systems into three major scenarios: UGV-aided UAV trajectory planning, UAV-aided UGV trajectory planning, and reciprocal UGV-UAV cooperative trajectory planning.

First, in UGV-aided UAV scenarios, ground vehicles typically act as mobile depots or edge servers to compensate for the limited battery capacity and computational payload of UAVs. Specifically, UGVs are utilized as mobile docking stations or nested depots to optimize vehicle routing, facilitate autonomous landing, and coordinate large-scale exploration [15,16,17]. Furthermore, UGVs are increasingly deployed as mobile edge computing nodes to provide computational support, providing various frameworks for efficient and secure task offloading in urban surveillance and disaster response missions [18,19].

Second, as for studies of UAV-aided UGV scenarios, aerial platforms serve as overhead scouts or communication relays to assist ground vehicles in obstacle avoidance and pathfinding in GPS-denied or occluded environments. Particularly in the context of overhead scouting, Xie et al. [20] designed collaborative path-planning frameworks where UAVs provide extended environmental visibility to optimize UGV trajectories and hybrid navigation strategies. Expanding beyond physical navigation to communication and operational coordination, Oubbati et al. [21] and Sun et al. [22] utilized aerial platforms to facilitate energy-efficient urban patrolling and joint resource scheduling for mobile edge computing within vehicular networks.

Third, UGV-UAV cooperative planning focuses on joint spatiotemporal optimization. It has been widely applied to last-mile delivery and heterogeneous formation control. In these tasks, the UGV and UAV operate simultaneously to improve task-level performance. In the domain of cooperative logistics and last-mile delivery, Jang et al. [23] and Li et al. [24] proposed dynamic truck-drone routing frameworks that jointly optimize path planning and vehicle scheduling to minimize overall travel costs. Expanding upon this routing paradigm, Yoon et al. [25] introduced an optimal load-balancing strategy for collaborative parcel pickup, while Li et al. [26] extended these joint routing principles to maritime environments through vessel–UAV collaboration. To handle the high-dimensional complexity of such joint missions, Ma et al. [27] applied adaptive graph neural networks for dynamic task allocation, and Zhu et al. [28] developed deep learning-enhanced trajectory predictors for cross-domain vehicle interactions. Conversely, addressing the strict kinematic and stability requirements of heterogeneous formation control, Tian et al. [29], Yan et al. [30], and Liang et al. [31] designed distributed adaptive control methods to ensure robust formation tracking, accommodate asymmetric physical constraints, and maintain prescribed performance even with human-in-the-loop participation.

Recent studies have also emphasized the importance of reliable state estimation and hierarchical planning in robotic systems. For example, Cosenza et al. [32] developed a virtual-sensor-based wheel angular-speed estimation method for differential-drive robots, showing that accurate motion-state estimation is important for reliable control and trajectory tracking. However, these studies mainly focus on single-robot planning, autonomous driving, or general hierarchical decision-motion coupling, and rarely consider UGV-UAV cooperative trajectory planning under simultaneous communication, obstacle avoidance, and heterogeneous kinematic constraints in urban canyon environments. Therefore, a dedicated hierarchical framework is still needed for communication-constrained air–ground cooperative planning.

2.3. Research on Urban Canyons

Research on urban canyon environments includes methods to improve GNSS positioning accuracy [33], techniques for signal correction [34] and optimization of planned trajectories [35]. In contrast, this paper primarily focuses on trajectory planning research under communication-constrained conditions within urban canyons.

3. Problem Statement

Essentially, our cooperative planning task can be formulated as an OCP subject to multiple constraints. Its core objective is to determine an optimal state sequence $X^{*}=\left\{x_g,x_a\right\}$ and a corresponding optimal control sequence $U^{*}=\left\{u_g,u_a\right\}$ to minimize the cost function, while satisfying the requirements of system kinematics, communication quality, and collision avoidance. The state sequence is defined as $X=\left\{x_g,x_a\right\}$, where the UGV state vector is $x_g={[p_{gx},p_{gy},{\mathsf{\psi }}_g,v_g]}^T$ and the UAV state vector is defined as $x_a={[p_{ax},p_{ay},p_{az},{\mathsf{\varphi }}_a,{\mathsf{\theta }}_a,{\mathsf{\psi }}_a]}^T$. The control sequence is defined as $U=\left\{u_g,u_a\right\}$, consisting of the UGV control vector $u_g={[a_g,{\mathsf{\delta }}_g]}^T$ and the UAV control vector $u_a={[v_{ax},v_{ay},v_{az},{\mathsf{\omega }}_{ax},{\mathsf{\omega }}_{ay},{\mathsf{\omega }}_{az}]}^T$. In the following sections, the formulation of this OCP will be elaborated. For clarity and consistency, we provide a notation table for all state and control variables, as shown in

Table 1. A notation table that illustrates all the state and control variables that we use in Section 3.

Symbol	Description
$p_g={[p_{gx},p_{gy}]}^T$	Position of the UGV
${\mathsf{\psi }}_g$	Heading angle of the UGV
$v_g$	Velocity of the UGV
$x_g={[p_{gx},p_{gy},{\mathsf{\psi }}_g,v_g]}^T$	UGV state vector
$u_g={[a_g,{\mathsf{\delta }}_g]}^T$	UGV control vector
$p_a={[p_{ax},p_{ay},p_{az}]}^T$	Position of the UAV
${\mathsf{\varphi }}_a,{\mathsf{\theta }}_a,{\mathsf{\psi }}_a$	UAV roll, pitch, and yaw angle
$x_a={[p_{ax},p_{ay},p_{az},{\mathsf{\varphi }}_a,{\mathsf{\theta }}_a,{\mathsf{\psi }}_a]}^T$	Complete UAV state vector
$v_a={[v_{ax},v_{ay},v_{az}]}^T$	Velocity of the UAV
${\mathsf{\omega }}_a={[{\mathsf{\omega }}_{ax},{\mathsf{\omega }}_{ay},{\mathsf{\omega }}_{az}]}^T$	Body-frame angular velocity of the UAV
$u_a={[v_{ax},v_{ay},v_{az},{\mathsf{\omega }}_{ax},{\mathsf{\omega }}_{ay},{\mathsf{\omega }}_{az}]}^T$	Complete UAV control vector
$X=\left\{x_g,x_a\right\}$	Joint state vector of the UGV-UAV system
$U=\left\{u_g,u_a\right\}$	Joint control vector of the UGV-UAV system
$T_a^k$	Transformation matrix from the body-frame angular velocities to the Euler angle rates

.

3.1. Cost Function

To ensure rapid planning task completion, guarantee communication quality in urban canyons and enhance the smoothness of the trajectory, the cost function $J$ is formulated as follows:

$$J={\mathrm{w}}_t(k_N-k_0)\mathsf{\Delta }t+{\displaystyle {\sum }_{k=0}^{N-1}({\mathrm{w}}_c{J}_{comm}^k+{\mathrm{w}}_s{J}_{smooth}^k)}$$

(1)

Specifically, the planning task time cost is denoted by $(k_N-k_0)\mathsf{\Delta }t$, where $k_0$ and $k_N$ represent the initial and terminal step index, respectively; $\mathsf{\Delta }t$ represents the time interval of each step; by tuning the weight coefficients ${\mathrm{w}}_t$, ${\mathrm{w}}_c$ and ${\mathrm{w}}_s$, the cost function can be adapted to accommodate different planning task priorities; the communication quality cost $J_{comm}^k$ is defined as

$$J_{comm}^k=\left\{\begin{array}{ll}\mathsf{\gamma },\hfill & \mathrm{if} \mathrm{Line}(p_g^k,p_a^k)\cap \mathcal{O}\ne \varnothing \hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

where $p_g^k$ and $p_a^k$ denote the positions of the UGV and UAV at step $k$, respectively; $\mathrm{Line}()\cap \mathcal{O}$ indicates whether the line segment between the UGV and UAV intersects the set of building obstacles $\mathcal{O}$; and $\mathsf{\gamma }$ represents the penalty constant associated with communication occlusion. The trajectory smoothness cost, $J_{smooth}^k$, is defined as

$$J_{smooth}^k={‖u_g^k‖}^2+{‖u_a^k‖}^2$$

(3)

where $u_g^k$ and $u_a^k$ denote the control vectors of the UGV and the UAV at step $k$, respectively.

3.2. Constraints

To guarantee the physical feasibility of the cooperative trajectories and ensure operational safety within complex urban environments, the formulated OCP must satisfy the following five constraints.

3.2.1. Two-Point Boundary Value Constraints

The two-point boundary value constraints are defined as

$$p_g^0=p_{start},v_g^0=0; p_a^0=p_{start},v_a^0=0$$

(4)

$$‖p_g^N-p_{goal}‖\le \mathsf{\epsilon },v_g^N=0; v_a^N=0$$

(5)

where $p_g^0$ and $p_a^0$ denote the starting points of the cooperative trajectories for the UGV and UAV, respectively; $p_{start}$ represents the starting coordinates in the map; $v_g^0$ and $v_a^0$ represent the initial velocities of the UGV and UAV; $p_g^N$ and $p_a^N$ denote the terminal points of the cooperative trajectories; $v_g^N$ and $v_a^N$ represent the terminal velocities of the UGV and UAV; and $\mathsf{\epsilon }$ denotes the terminal tolerance.

3.2.2. Kinematic Constraints

The state transitions of both the UGV and UAV at each time step must be governed by the discrete-time kinematic equations $x_{g,a}^{k+1}=f(x_{g,a}^k,u_{g,a}^k)$. For the UGV, the kinematic bicycle model is employed as presented in Figure 2a, and the specific equations are formulated as follows:

$$\left\{\begin{array}{l}{x}_g^{k+1}=x_g^k+v_g^k\cos ({\mathsf{\psi }}_g^k)\mathsf{\Delta }t\hfill \\ y_g^{k+1}=y_g^k+v_g^k\sin ({\mathsf{\psi }}_g^k)\mathsf{\Delta }t\hfill \\ {\mathsf{\psi }}_g^{k+1}={\mathsf{\psi }}_g^k+{\displaystyle \frac{v_g^k}{L}}\tan ({\mathsf{\delta }}_k)\mathsf{\Delta }t\hfill \\ v_g^{k+1}=v_g^k+a_g^k\mathsf{\Delta }t\hfill \end{array}\right.$$

(6)

where ${\mathsf{\psi }}_g^k$ and ${\mathsf{\delta }}_k$ denote the heading angle and the front wheel steering angle of the UGV at the $k$-th step, respectively; $a_g^k$ represents the acceleration of the UGV at the $k$-th step; $L$ is the wheelbase of the UGV; and $\mathsf{\Delta }t$ denotes the time interval between discrete steps.

For the UAV, the quadrotor model is adopted as presented in Figure 2b, and the specific equations are formulated as follows:

$$\left\{\begin{array}{l}{p}_a^{k+1}=p_a^k+v_a^k\mathsf{\Delta }t\hfill \\ {\mathsf{\chi }}_a^{k+1}={\mathsf{\chi }}_a^k+T_a^k{\mathsf{\omega }}_a^k\mathsf{\Delta }t\hfill \end{array}\right.$$

(7)

where $p_a^k={[p_{ax}^k,p_{ay}^k,p_{az}^k]}^T$ denotes the position vector of the UAV; ${\mathsf{\chi }}_a^k={[{\mathsf{\varphi }}_a^k,{\mathsf{\theta }}_a^k,{\mathsf{\psi }}_a^k]}^T$ represents the attitude vector, where ${\mathsf{\varphi }}_a^k$, ${\mathsf{\theta }}_a^k$, and ${\mathsf{\psi }}_a^k$ correspond to the roll (around the $x$-axis), pitch (around the $y$-axis), and yaw (around the $z$-axis) angles, respectively; $v_a^k$ represents the linear velocity vector of the UAV, and ${\mathsf{\omega }}_a^k$ denotes the angular velocity vector in the body-fixed frame; $\mathsf{\Delta }k$ is the discretized time step; the transformation matrix $T_a^k$ from the body-frame angular velocities to the Euler angle rates is defined as

$$T_a^k=\left[\begin{array}{ccc}1& \tan {\mathsf{\theta }}_k\sin {\mathsf{\varphi }}_a^k& \tan {\mathsf{\theta }}_k\cos {\mathsf{\varphi }}_a^k\\ 0& \cos {\mathsf{\varphi }}_a^k& -\sin {\mathsf{\varphi }}_a^k\\ 0& \sin {\mathsf{\varphi }}_a^k/\cos {\mathsf{\theta }}_a^k& \cos {\mathsf{\varphi }}_a^k/\cos {\mathsf{\theta }}_a^k\end{array}\right]$$

(8)

To prevent the cooperative trajectories from violating physical laws of motion, the velocities and control vectors of both the UGV and UAV must satisfy the following physical constraints

$$v_g\in \left[v_{\min },v_{\max }\right], a_g\in \left[a_{\min },a_{\max }\right], \left|{\mathsf{\delta }}_g\right|\le {\mathsf{\delta }}_{g,\max }$$

(9)

$$\Vert v_{a,xy}\Vert \le v_{xy,max}, \left|v_{a,z}\right|\le v_{z,max}, \left|{\mathsf{\omega }}_a\right|\le {\mathsf{\omega }}_{a,max}$$

(10)

where ${\mathsf{\omega }}_a={[{\mathsf{\omega }}_{ax},{\mathsf{\omega }}_{ay},{\mathsf{\omega }}_{az}]}^T$ denotes that the magnitude of the angular velocity vector of the UAV is bounded by its maximum allowable value ${\mathsf{\omega }}_{max}$ to ensure flight stability and satisfy actuator limits.

3.2.3. Collision Avoidance Constraints

The collision avoidance constraint is defined below

$$p_g^k\in C_g^k, p_a^k\in C_a^k$$

(11)

where $C_g^k,C_a^k$ represents the UGV/UAV safety corridor generated by the algorithm at the $k$-th step, the specific procedure of which will be elaborated in detail in Section 5.

3.2.4. Communication Range Constraints

The communication range constraint is defined below

$$‖p_a^k-p_g^k‖\le D_{\max }$$

(12)

where $D_{max}$ represents the maximum allowable communication distance between the UGV and the UAV.

3.3. Remarks

Integrating the above-mentioned components, the cooperative trajectory planning problem in this paper can be summarized in the following form:

$$\begin{array}{l}\underset{x(t),u(t),T}{\min } \left(1\right),\\ \mathrm{s}.\mathrm{t}. \mathrm{Kinematic} \mathrm{constraints} \left(6\right)–\left(10\right);\\ \mathrm{Boundary} \mathrm{constraints} \left(4\right) \mathrm{and} \left(5\right);\\ \mathrm{Collision}\text{−}\mathrm{avoidance} \mathrm{constraints} \left(11\right) \mathrm{and} \left(12\right).\end{array}$$

(13)

Directly solving the UGV and UAV trajectories independently using traditional path planning algorithms and corridor constraints [36] may not be suitable for the formulated problem. First, the communication constraint depends on the relative positions of the UGV and UAV and, therefore, introduces a nonlinear coupling between their trajectories. Second, the UGV and UAV follow different kinematic models, so independent planning cannot easily guarantee joint feasibility of the complete state-control sequence. Third, in a non-convex environment, a heuristic initial solution may constrain the optimizer to a limited feasible region, which can reduce the quality of the final cooperative trajectory.

4. Overall Framework

This Section presents the design motivation and detailed architecture of the proposed two-layered planning framework, as presented in Figure 3. Furthermore, it elaborates on how this framework is applied to solve the aforementioned UGV-UAV cooperative trajectory optimization problem in urban canyon environments.

4.1. Motivation

Inspired by the closed-loop mechanism of forward decision-making guidance and backward motion planning [37], a hierarchical cooperative planning framework is proposed to address the optimization problem formulated in Section 3.

In the upper layer, the strong non-convexity and non-differentiability of the problem render conventional optimization-based solvers highly susceptible to local optima. RL is therefore adopted, as it circumvents direct mathematical resolution through trial-and-error interactions with the environment. However, pure RL in complex urban canyon scenarios suffers from the sparse reward dilemma. Drawing on the from-coarse-to-fine paradigm [38], the A* algorithm is introduced to pre-generate a reference path as prior guidance, narrowing the search space and contributing to more stable RL training.

In the lower layer, since the upper-layer output typically violates certain complex constraints and lacks spatial smoothness, an optimization-based solver is employed for trajectory refinement. A safe corridor generation algorithm is further incorporated to transform non-convex obstacle-avoidance constraints into convex formulations, supporting collision-free guarantees while optimizing kinematic smoothness.

4.2. Two-Layered Planner

This study proposes a two-layered trajectory planning framework, which decouples the UGV-UAV cooperative trajectory planning problem into a discrete spatial exploration phase and a continuous numerical optimization phase. This approach enables the system to efficiently narrow down the search space prior to control execution, thereby achieving a balance between search efficiency and trajectory quality.

In the upper-layer framework, this study designs an initial trajectory generation strategy based on RL guided by heuristic search. The input to this layer consists of the environmental grid map, the encoded communication-connectivity map, and the start and goal configurations of the UGV. The corresponding output is a sequence of discrete waypoints for both the UGV and UAV that preliminarily satisfy the cooperative constraints.

In the lower-layer framework, this study formulates a trajectory optimizer based on nonlinear programming (NLP). The input to this layer consists of the initial cooperative waypoint sequence from the upper layer, along with the OCP cost function and constraints. The corresponding output is the refined feasible UGV-UAV cooperative trajectory.

The upper and lower layers achieve cooperative trajectory planning through the coordination of topological guidance and a warm-start mechanism. In highly non-convex environments, the traversable space is fragmented into multiple homotopy classes. While lower-layer NLP solvers excel at local refinement, they are not designed to perform topological transitions across obstacles. However, by directly utilizing the upper layer’s topologically informative initial trajectory as a warm-start input, the lower-layer solver reduces reliance on unguided local search. Since this initial solution is expected to lie closer to the optimum, the NLP solver only needs to handle the complex kinematic constraints of the UGV and UAV within the established safe corridor. This enables the algorithm to generate an optimal cooperative trajectory within minimal iterations.

5. Methodology

This Section elaborates on the specific architecture, operational logic, and algorithmic principles of the key modules within the proposed two-layer planner.

5.1. Upper-Layer Planner

The upper-layer planner consists of four modules designed to generate initial waypoint sequences $P_{init}={\{(p_{g,init}^k,p_{a,init}^k)\}}_{k=0}^N$ with topological guidance in urban canyon environments. In the following sections, we will sequentially introduce the constituent components of the upper-layer planner, its operational pipeline, the key RL algorithm employed, and the associated training network architecture.

5.1.1. Components and Operational Pipeline

The upper-layer planner is designed to generate an initial cooperative waypoint sequence with topological guidance in urban canyon environments, as shown in Figure 4. It consists of four tightly coupled modules: map construction, heuristic path guidance, reinforcement learning, and the DRL network.

First, the map construction module establishes a two-layered environmental representation. The first layer is an inflated 3D grid map $M_{obs}\in {ℝ}^{H\times W\times L}$ constructed from a manually curated polygonal dataset based on Google Earth real-world imagery, where grid values of 1 and 0 represent obstacles and free space, respectively. The second layer is an NLoS communication grid map $M_{nlos}\in {ℝ}^{H\times W\times L}$. Unlike $M_{obs}$, which is assumed to be a given urban canyon scenario, $M_{nlos}$ is dynamically updated during RL inference according to the current UGV position. Specifically, the LoS status between the UGV and UAV is evaluated based on $M_{obs}$, where 0 indicates visible and 1 indicates occluded. Therefore, $M_{nlos}$ is updated as the UGV moves.

Second, the heuristic path guidance module employs the A* algorithm to generate a 2D reference path $P_{ref,g}=\{p_{g,t\arg et}^0,p_{g,t\arg et}^1\dots p_{g,t\arg et}^{N-1}\}$ for the UGV and a 3D reference path $P_{ref,a}=\{p_{a,t\arg et}^0,p_{a,t\arg et}^1\dots p_{a,t\arg et}^{N-1}\}$ for the UAV according to the start and goal positions. To prevent colliding with obstacles, a distance transform field is incorporated into the 3D grid map $M_{obs}$ used by A*.

Third, the RL module treats the UGV and UAV as two cooperative agents in the RL environment $S=\{S_{img},p_g^k,p_a^k\}$. The image-based state $S_{img}$ is constructed by concatenating $M_{obs}$ and $M_{nlos}$. The action space $A\in {ℝ}^5$ consists of displacement vectors for the UGV in the XY plane and for the UAV in the XYZ space. The reward function $R$ is mapped from the objective function and constraints of the OCP, which will be elaborated in Section 5.1.2.

Finally, the DRL network adopts an Actor-Critic architecture. The CNN branch extracts features from the $S_{img}$ and maps them into a 256-dimensional feature vector, while the MLP branch maps the UGV-UAV position vector into a 64-dimensional representation. These two feature vectors are concatenated into a 320-dimensional mixed feature vector and further transformed into a 128-dimensional latent representation. Based on this representation, the Actor network outputs the UGV and UAV displacement actions, while the Critic network estimates the expected cumulative return to guide policy optimization during training.

During execution, the map construction and heuristic path guidance modules first perform preprocessing for the given scenario. $M_{obs}$ and $M_{nlos}$ are generated, and A* is used to obtain $P_{ref,g}$ and $P_{ref,a}$. The planner then enters the iterative RL inference loop. At each discrete time step, the updated $S_{img}$ and position vectors are fed into the pre-trained Actor-Critic network. The Actor outputs the cooperative displacement actions, after which the upper-layer planner performs a single-step forward operation in the urban canyon environment and updates the UGV-UAV positions. This process is repeated until the UGV reach the goal. The coordinates recorded at all discrete time steps form $P_{init}$, which is then passed to the lower-layer planner as the warm-start initial guess.

5.1.2. Forward Planning in RL

Forward planning in RL consists of a sequence of identical computational steps, with the detailed algorithmic logic for each step presented in Algorithm 1, which is described as follows:

At the onset of each discrete time step $k$, it dynamically retrieves the reference paths $P_{ref,g}$ and $P_{ref,a}$ pre-planned by the A* algorithm based on the current spatial coordinates of the UGV and UAV. From these paths, the local target guidance points $p_{g,target}$ and $p_{a,target}$ for the subsequent stage are extracted. These are completed by function $\mathrm{ExtractReferenceTarget}()$. Subsequently, the spatial positions of the UGV and UAV are updated according to the displacement vectors received from the Actor network, subject to a hard truncation function $\mathrm{ClipToMapBoundary}()$. Finally, a reward value based on the reward function is fed back based on the agent’s updated spatial configurations. The details of the reward function are illustrated as follows:

The objective functions and constraints of the OCP defined in Section 3 are mapped into the RL reward function $R_{total}$, as follows:

$$R_{total}=R_{progress}+R_{safety}+R_{comm}+R_{cons}$$

(14)

Specifically, the first term $R_{progress}$ evaluates navigation efficiency by measuring the reduction in the Euclidean distance between each agent and its corresponding A*-guided target point

$$R_{progress}={\mathrm{w}}_{p,g}(d_g^k-d_g^{k+1})+{\mathrm{w}}_{p,a}(d_a^k-d_a^{k+1})$$

(15)

where $d_g$ and $d_a$ denote the distances from the UGV and UAV to their corresponding A* reference points, respectively, and ${\mathrm{w}}_{p,g}$ and ${\mathrm{w}}_{p,a}$ are the progress reward weights. $R_{safety}$ corresponds to the collision constraints prior to corridor generation (which will be illustrated in Section 5.2.3)

$$R_{safety}=\left\{\begin{array}{ll}-{\mathrm{w}}_{collision},\hfill & \mathrm{if} p_{g,a}^{k+1}\in \mathcal{O}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(16)

where ${\mathrm{w}}_{collision}$ represents the collision penalty coefficient, $\mathcal{O}$ represents obstacles. $R_{comm}$ corresponds to the communication quality cost $J_{comm}^k$ defined in Section 3. The system queries $M_{nlos}$ in real time, and a fixed penalty is triggered once the NLoS state appears

$$R_{comm}=\left\{\begin{array}{ll}-{\mathrm{w}}_{comm},\hfill & \mathrm{if} \mathrm{Line}(p_g,p_a)\cap \mathcal{O}\ne \varnothing \hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(17)

where ${\mathrm{w}}_{comm}$ represents the communication penalty coefficient. $R_{cons}$ corresponds to the maximum communication distance constraint defined in Section 3

$$R_{cons}=\left\{\begin{array}{ll}-0.1\cdot {\mathrm{w}}_{comm}\cdot \left(\Vert p_a^k-p_g^k\Vert -D_{max}\right),\hfill & \mathrm{if} \Vert p_a^k-p_g^k\Vert >D_{max}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(18)

Upon computing the reward function, the environment immediately invokes the collision detection procedure. If the trajectory of either the UGV or the UAV is detected by function $\mathrm{CheckCollision}()$ for intruding into the obstacle regions within $M_{obs}$, the system triggers a ‘soft reset’ strategy. Unlike conventional logic, where collisions lead to immediate episode termination, we forcibly roll back the agents’ coordinates to their nearest A* reference waypoints, which is completed by function $\mathrm{RollbackToNearestAStarPoint}()$. This rollback mechanism effectively concludes the single-step update of the RL process.

Algorithm 1. RL Single-Step Execution Logic

Input: Current coordinates $p_g^k,p_a^k$, reference paths $P_{\mathrm{ref},g},P_{\mathrm{ref},a}$, action vectors Output: Updated coordinates $p_g^{k+1},p_a^{k+1}$, step total reward $R_{\mathrm{total}}$

1: $p_{g,t\arg et}^k,p_{a,t\arg et}^k\leftarrow \mathrm{ExtractReferenceTarget}(p_g^k,p_a^k,P_{\mathrm{ref},g},P_{\mathrm{ref},a})$ 2: // Update positions with boundary truncation 3: $p_g^{k+1}\leftarrow \mathrm{ClipToMapBoundary}(p_g^k+p_g^{\mathsf{\Delta }t})$ 4: $p_a^{k+1}\leftarrow \mathrm{ClipToMapBoundary}(p_a^k+p_a^{\mathsf{\Delta }t})$ 5: // Calculate reward components 6: $R_{\mathrm{progress}}\leftarrow 0$,$R_{safety}\leftarrow 0$,$R_{\mathrm{comm}}\leftarrow 0$, $R_{\mathrm{cons}}\leftarrow 0$ 7: $R_{\mathrm{progress}}\leftarrow {\mathrm{w}}_{p,g}(d_g^{k+1}-d_g^k)+{\mathrm{w}}_{p,a}(d_a^{k+1}-d_a^k)$ 8: if $p_g^{k+1}\in \mathcal{O}$ or $p_a^{k+1}\in \mathcal{O}$ then 9:   $R_{\mathrm{safety}}\leftarrow -{\mathrm{w}}_{\mathrm{collision}}$ 10: end if 11: if $\mathrm{Line}(p_g^{k+1},p_a^{k+1})\cap \mathcal{O}\ne \varnothing$ based on $M_{\mathrm{nlos}}$ then 12:   $R_{\mathrm{comm}}\leftarrow -{\mathrm{w}}_{\mathrm{comm}}$ 13: end if 14: if $\Vert p_a^{k+1}-p_g^{k+1}\Vert >D_{\max }$ then 15:   $R_{\mathrm{cons}}\leftarrow -0.1\cdot {\mathrm{w}}_{\mathrm{comm}}\cdot (\Vert p_a^{k+1}-p_g^{k+1}\Vert -D_{\max })$ 16: end if 17: $R_{\mathrm{total}}\leftarrow R_{\mathrm{progress}}+R_{\mathrm{safety}}+R_{\mathrm{comm}}+R_{\mathrm{cons}}$ 18: // ‘Soft reset’ strategy for collision avoidance 19: if $\mathrm{CheckCollision}(p_g^{k+1},p_a^{k+1},M_{\mathrm{obs}})$ then 20:   $p_g^{k+1}\leftarrow \mathrm{RollbackToNearestAStarPoint}(p_g^{k+1},P_{\mathrm{ref},g})$ 21:   $p_a^{k+1}\leftarrow \mathrm{RollbackToNearestAStarPoint}(p_a^{k+1},P_{\mathrm{ref},a})$ 22: end if

return $p_g^{k+1},p_a^{k+1},R_{\mathrm{total}}$

5.1.3. DRL Training Network

In the upper-layer cooperative planning framework, we train the Actor-Critic network using the Proximal Policy Optimization (PPO) algorithm. At each discrete time step $k$, the agent acquires the environmental state $S_k$. Based on this state, the old policy network ${\mathsf{\pi }}_{{\mathsf{\theta }}_{old}}$ outputs the distribution parameters ${\mathsf{\mu }}_k$ and ${\mathsf{\sigma }}_k$ to construct a multivariate normal distribution $\mathcal{N}({\mathsf{\mu }}_k,{\mathsf{\sigma }}_k^2)$, from which the joint action vector $a_k$ for the UGV and UAV is sampled. The mean action output of the Actor network is passed through a Tanh activation function, which bounds the normalized displacement command within $[-1,1]$ before it is scaled to the predefined RL step ranges of the UGV and UAV. After the environment executes $a_k$, it returns the reward evaluated by the previously defined reward Function (14). Subsequently, the system serializes this state transition data and pushes it into the experience replay buffer.

Entering the policy update phase, the algorithm first calculates and normalizes the discounted cumulative return ${\widehat{R}}_k$ backwards using the Monte Carlo method. The current policy network ${\mathsf{\pi }}_{\mathsf{\theta }}$ then re-evaluates the historical states, outputting the new action log-probabilities $\log {\mathsf{\pi }}_{\mathsf{\theta }}(a_k|S_k)$, the state value estimates $V_{\mathsf{\varphi }}(S_k)$, and the distribution entropy $H({\mathsf{\pi }}_{\mathsf{\theta }})$. Consequently, these outputs are utilized to compute the importance sampling ratio $r_k(\mathsf{\theta })=\exp (\log {\mathsf{\pi }}_{\mathsf{\theta }}-\log {\mathsf{\pi }}_{{\mathsf{\theta }}_{old}})$ and the advantage function ${\widehat{A}}_k={\widehat{R}}_k-V_{{\mathsf{\varphi }}_{old}}(S_k)$.

To improve iterative stability, the network employs the composite PPO loss function $L_{total}$ with a clipping mechanism to perform backpropagation over $K$ consecutive epochs

$$L_{total}=-\min \left(r_k(\mathsf{\theta }){\widehat{A}}_k,\mathrm{clip}(r_k(\mathsf{\theta }),1-ϵ,1+ϵ){\widehat{A}}_k\right)+c_1{\mathcal{L}}_{VF}-c_{2}H({\mathsf{\pi }}_{\mathsf{\theta }})$$

(19)

where $ϵ$ represents the clipping threshold, $L_{VF}=0.5\Vert V_{\mathsf{\varphi }}(S_k)-{\widehat{R}}_k{\Vert }^2$ denotes the mean squared error loss of the value network, and $c_1$ and $c_2$ are the weight coefficients for the value and entropy terms, respectively. During parameter optimization, we employ a grouped learning rate mechanism: the feature extraction modules and the Actor network are assigned a lower base learning rate to preserve the stability of state representations, whereas the Critic network is independently assigned a higher learning rate to accelerate the approximation of expected environmental returns. Ultimately, the system synchronizes the weights of the updated network ${\mathsf{\pi }}_{\mathsf{\theta }}$ to ${\mathsf{\pi }}_{{\mathsf{\theta }}_{old}}$ and flushes the replay buffer, thereby concluding a complete cycle of policy iteration. The detailed RL training hyperparameters used in the experiments, including reward penalty weights, PPO parameters, learning rates, batch size, and episode count, are reported in Section 6.

5.2. Lower-Layer Planner

Designed as an optimization-based refinement module, the lower-layer planner leverages the initial guess generated by the upper layer to map abstract topological guidance into a kinematically feasible and optimal physical trajectory, as presented in Figure 5. The fundamental workflow consists of three stages: initially, the OCP defined in Section 3 is discretized into a finite-dimensional NLP formulation. Subsequently, the solver-ready state and control decision vectors are initialized using the upper-layer output. Ultimately, by utilizing orthogonal collocation direct transcription (OCDT), these variables, subject to all system constraints, are evaluated by the NLP solver to compute the final trajectory. In the following sections, we provide a detailed breakdown of the lower-layer planner, elaborating on its internal architecture, operational pipeline, and the core algorithmic principles it employs.

5.2.1. Components and Operational Pipeline

The lower-layer planner is primarily composed of two modules: the upper-layer output processing module and the NLP solving module.

In the upper-layer output processing module, the waypoint sequence $P_{init}$ generated by the upper-layer planner is first densified through linear interpolation to obtain $P_{init}^{\prime }$. To avoid collisions caused by sparse waypoint sequences, a KD-tree-based fallback mechanism is introduced, as shown in Figure 6 and summarized in Algorithm 2. Specifically, the A*-guided reference path $P_{ref}$ is organized as a KD-tree for efficient nearest-neighbor search. For each adjacent waypoint pair in $P_{init}$, the algorithm checks whether the connecting segment intersects the inflated obstacle map. If the segment is collision-free, it is directly interpolated into $P_{init}^{\prime }$. Otherwise, the midpoint of the invalid segment is computed, and its nearest point in $P_{ref}$ is selected as the fallback endpoint. This process is repeated, subject to the maximum retry number $N_r$, until the interpolation is completed.

Algorithm 2. Linear interpolation with KD-tree-based Fallback Mechanism

Input: Initial waypoint sequence $P_{init}$, A*-guided reference path $P_{ref}$, Inflated obstacle map $M_{obs}$, Maximum retry number $N_r$ Output: Interpolated initial waypoint sequence $P_{init}^{\prime }$

1: Construct a KD-tree from the A*-guided path $P_{ref}$ 2: $P_{init}^{\prime }=P_{init}$ 3: for each candidate segment ($p_{start}$,$p_{end}$) generated from $P_{init}$ do 4:   $r=0$ 5:   while ($p_{start}$,$p_{end}$) collides with $M_{obs}$ and $r<N_r$ do 6:    $p_{mid}=(p_{start}+p_{end})/2$ 7:    Query KD-tree to find the nearest A*-guided point $p_d$ to $p_{mid}$ 8:    $p_{end}=p_d$ 9:    $r=r+1$ 10:   end while 11:   $p_{start}=p_{end}$ 12:   $P_{init}^{\prime }\leftarrow p_{end}$ 13: end for

return $P_{init}^{\prime }$

Based on the discrete spatial coordinates within $P_{init}^{\prime }$, numerical derivative approximation is then used to estimate the UGV heading angle ${\mathsf{\psi }}_g^k$ and velocity $v_g^k$, as well as the UAV three-axis velocities ${[v_{ax}^k,v_{ay}^k,v_{az}^k]}^T$ and attitude angles ${\mathsf{\varphi }}_k$. By concatenating $P_{init}^{\prime }$ with these supplementary state variables and zero-initialized control inputs, the initial state sequence $X_{init}$ and control sequence $U_{init}$ are constructed as a valid initial guess for the solver.

The NLP solving module then transforms the OCP into a constrained NLP problem using CasADi. The objective function is formulated as

$$J={\mathrm{w}}_{t}T+{\displaystyle \sum _{k=0}^{N-1}({\mathrm{w}}_r\Vert }{u}_g^k{\Vert }^2+{\mathrm{w}}_a\Vert u_a^k{\Vert }^2)$$

(20)

where $T=(k_N-k_0)\cdot \mathsf{\Delta }t$ is the total planning time, ${\mathrm{w}}_t$ is designed to guide the solver toward an optimal trajectory; $u_g^k$ and $u_a^k$ denote the UGV and UAV control vectors at step $k$; ${\mathrm{w}}_r$ and ${\mathrm{w}}_a$ regulate the smoothness of the cooperative trajectories. The NLP also includes system transition constraints based on UGV and UAV kinematics in Equations (6) and (7), physical bounds in Equations (9) and (10), safety corridors in Equation (11), two-point boundary value constraints in Equations (4) and (5), and the communication distance constraint in Equation (12).

5.2.2. NLP Resolution via OCDT

In establishing the numerical computation framework for the NLP problem, this paper employs the OCDT [39] (pp. 287–324), also referred to as a simultaneous full-discretization method. Within this framework, the system not only treats the control sequence $U=\{u_{g,a}^0,u_{g,a}^1,\dots ,u_{g,a}^{N-1}\}$ as decision variables but also incorporates the state sequence $X=\{x_{g,a}^0,x_{g,a}^1,\dots ,x_{g,a}^{N-1}\}$ into the optimization variable space. The dimension of the resulting NLP problem can be explicitly determined from the discretized state and control variables. At each discretization node, the UGV state contains 4 variables and the UAV state contains 6 variables, while the UGV and UAV control vectors contain 2 and 6 variables, respectively. Therefore, for an optimization horizon with N control intervals, the full-discretization NLP contains 10(N + 1) state variables and 8N control variables, resulting in a total of 18N + 10 decision variables. In the representative benchmark case used in the experiments, the interpolated trajectory was resampled into N = 120 control intervals, leading to 1210 state variables, 960 control variables, and 2170 decision variables in total.

Compared to partial discretization methods, the OCDT offers an advantage in terms of numerical stability and convergence. Because both state and control variables serve as independent optimization variables at each discrete time step, the continuous system kinematic equations are transcribed into equality constraints between adjacent nodes, denoted as $x_{g,a}^{k+1}=f(x_{g,a}^k,x_{g,a}^k,\mathsf{\Delta }t)$. This specific structure helps mitigate the trajectory ‘drift’ issue caused by the temporal accumulation of minute perturbations in the initial state, thereby improving the solver with enhanced numerical robustness when tackling highly nonlinear systems. Furthermore, this approach enables the explicit formulation of constraints as algebraic equations with respect to the optimization variables. This structure allows the IPOPT solver to effectively exploit the sparse Jacobian and Hessian matrices of these equations, thereby facilitating efficient computations.

5.2.3. Obstacle-Inflation-Based Safe Corridor Generation

In the OCDT framework, obstacle-inflation-based safety corridors are introduced to handle obstacle avoidance and improve numerical stability in non-convex environments. The UGV and UAV are modeled using a bounding disk and a bounding sphere, respectively. Directly imposing collision-avoidance constraints between these geometric envelopes and obstacles would introduce non-convex Euclidean distance constraints into the NLP problem, increasing computational complexity and potentially affecting convergence. Therefore, the 3D grid map is first inflated according to the maximum robot radius, allowing both agents to be treated as point-mass models. Based on the inflated map, safety corridors are constructed at each discrete node as convex feasible regions represented by linear inequalities. This formulation converts obstacle avoidance into linear boundary constraints and enables IPOPT to exploit the sparse structure of the resulting NLP problem more efficiently. Algorithm 3 summarizes the construction of the obstacle-inflation-based safety corridors.

The procedure begins with spatial preprocessing for obstacle inflation. A spatial operator is defined according to the maximum radii of the UGV and UAV. This operator slides over the boundary cells of the original obstacles and marks the covered voxels as occupied, thereby generating $M_{obs}^{\prime }$. The inflation process is illustrated in Figure 7.

Based on the inflated map $M_{obs}^{\prime }$, the system employs a local bounding box generation algorithm to construct the specific safety corridor constraints. Specifically, centered at the UGV-UAV positional coordinates for each discrete time step $k$ in $P_{init}^{\prime }$, the algorithm executes an adaptive expansion along the coordinate axes. The expansion logic is implemented as follows: for the UGV, the algorithm probes along four directions ${\mathsf{\Omega }}_{ugv}=\{+x,-x,+y,-y\}$, while for the UAV, it performs a six-direction detection ${\mathsf{\Omega }}_{uav}=\{+x,-x,+y,-y,+z,-z\}$. During the expansion process, the system monitors the box boundaries in real time to detect any contact with the inflated map. If there is no collision being detected, the expansion in a specific direction will start until a collision is detected or the preset maximum expansion distance $L_{limit}$ is reached. Then, this termination excludes one direction. Adaptive box expansion will stop when all the directions are excluded.

This procedure ultimately generates local box for each waypoint of UGV and UAV, denoted as ${\mathcal{C}}_k=\{{\mathrm{Box}}_{ugv,k},{\mathrm{Box}}_{uav,k}\}$. Ultimately, the final safety corridor is formed by the union of all local boxes generated for each waypoint, denoted as $\mathcal{C}=\{{\mathcal{C}}_0,\dots ,{\mathcal{C}}_N\}$.

Algorithm 3. Safety-Corridor Generation Based on Map Inflation

Input: 3D map $M_{obs}$, Unified robot radius $r_{max}$, Initial path ${\mathcal{P}}_{init}^{\prime }$, Expansion limit $L_{limit}$ Output: Safety corridors $\mathcal{C}$

1: Step 1: Map Inflation 2: Define Map Element $S=\{(i,j,k)\in {\mathbb{Z}}^3\mid \mathsf{\delta }\cdot \sqrt{i^2+j^2+k^2}\le r_{max}\}$ where $(i,j,k)$ are voxel index offsets relative to any point $p\in {\mathcal{P}}_{init}^{\prime }$ in $M_{obs}$ 3: $M_{obs}^{\prime }=\max \_\mathrm{filter}(M_{obs},\mathrm{S})$ 4: Step 2: Adaptive Box Expansion and Corridor Generation 5: for each $k\in \{0,\dots ,N\}$ do 6:   $(x_k,y_k,z_k)\in {\mathcal{P}}_{init}^{\prime }$ 7:   Initialize ${\mathrm{Box}}_{ugv,k},{\mathrm{Box}}_{uav,k}\leftarrow (x_k,y_k,z_k)$ 8:    For UGV expansion (4 directions): 9:     ${\mathsf{\Omega }}_{ugv}=\{+x,-x,+y,-y\}$ 10:     while ${\mathsf{\Omega }}_{ugv}\ne \varnothing$ do 11:      Expand ${\mathrm{Box}}_{ugv,k}$ along $d\in {\mathsf{\Omega }}_{ugv}$ by $\mathsf{\Delta }s$ 12:      if ${\mathrm{Box}}_{ugv,k}\cap M_{obs}^{\prime }\ne \varnothing$ or $\mathrm{dist}\ge L_{limit}$ then 13:       ${\mathsf{\Omega }}_{ugv}={\mathsf{\Omega }}_{ugv}\backslash \left\{d\right\}$ 14:      end if 15:     end while 16:   For UAV expansion (6 directions): 17:     ${\mathsf{\Omega }}_{uav}=\{+x,-x,+y,-y,+z,-z\}$ 18:     while ${\mathsf{\Omega }}_{uav}\ne \varnothing$ do 19:       Expand ${\mathrm{Box}}_{uav,k}$ along $d\in {\mathsf{\Omega }}_{uav}$ by $\mathsf{\Delta }s$ 20:       if ${\mathrm{Box}}_{uav,k}\cap M_{obs}^{\prime }\ne \varnothing$ or $\mathrm{dist}\ge L_{limit}$ then 21:       ${\mathsf{\Omega }}_{uav}={\mathsf{\Omega }}_{uav}\backslash \left\{d\right\}$ 22:       end if 23:     end while 24:    Construct corridor ${\mathcal{C}}_k=\{{\mathrm{Box}}_{ugv,k},{\mathrm{Box}}_{uav,k}\}$ 25: end for

return $\mathcal{C}=\{{\mathcal{C}}_0,\dots ,{\mathcal{C}}_N\}$

6. Experiments and Results

The proposed hierarchical framework is evaluated in a simulated 3D urban canyon environment constructed from a custom grid map with physical dimensions of 50 m × 50 m and a flight altitude limit of 40 m. The continuous workspace is discretized at a resolution of 0.5 m, resulting in a 100 × 100 × 40 3D grid. Static buildings are represented by a predefined polygonal obstacle dataset. To simplify collision checking and corridor generation, the obstacle map is inflated by an equivalent physical radius of 1.5 m, allowing the UGV and UAV to be treated as point-mass models in the subsequent NLP optimization. The algorithms are implemented in Python 3.10, with the DRL module developed using PyTorch 2.10.0 and the NLP problem solved by CasADi with the IPOPT solver. All simulations are conducted on a laptop equipped with an Intel(R) Core (TM) Ultra 9 275HX @ 2.70 GHz processor, 32 GB RAM, and an NVIDIA GeForce RTX 5070 Laptop GPU. The main simulation and planning parameters are summarized in

Table 2. Main parameters in our simulation scenario.

Parameter	Description and Unit	Value
Map Dimension	Length/Height of simulated urban canyon environment (m)	$50\times 50\times 40$
Resolution	Resolution of the map	$0.5$
Goal Tolerance	Threshold to determine a successful arrival at the goal	$0.5$
Obstacle Inflation Radius	Inflation radius to raw map (m)	$1.5$
Maximum Communication Range	Maximum stable communication link distance between UGV and UAV (m)	$15.0$
Wheelbase	Distance between the front and rear axles of UGV (m)	$2.5$
RL Step Range of UGV/UAV	UGV/UAV displacement per step under the RL policy (m)	$[1.0, 3.0]$/$[1.5, 4.0]$
UGV/UAV Velocity Range	Velocity range for UGV/UAV in the lower-layer planner (m/s)	$[0.0, 3.0]$/$[2.0, 8.0]$
UGV Acceleration Range	Acceleration range of the UGV in the lower-layer planner (m/s2)	$[0.0, 3.0]$
Maximum Steering Angle/Maximum Inclination Angle	Maximum of the UGV’s front-wheel steering angle (rad)/Maximum of UAV’s roll and pitch angles (rad)	$0.6$/$0.5$
Maximum Angular Velocity	Roll/Pitch/Yaw velocity of the UAV (rad/s)	$3.0$
Communication penalty constant	Unit penalty for each NLoS communication occurrence	1.0

.

6.1. Visualization of Planning Results

This Section details the operational pipeline of our proposed framework and presents both the final cooperative trajectories and the intermediate results at each stage. Given that 3D visualizations often fail to provide a clear perspective of the results, all figures—with the exception of the final cooperative trajectory plots, which include partial 3D views—are presented as 2D projections of the 3D scene onto the XY plane. Regarding the planning task setup, it is assumed that the UAV has already detached from the UGV at the starting position and is hovering at its minimum safe flight altitude directly above the UGV. Furthermore, the UGV is designated as the primary planning-task executor, while the UAV serves solely for communication support. Consequently, once the UGV reaches the goal, the UAV will hover at a specific distance from the UGV, with the assumption that it can autonomously navigate back to the destination of the UGV upon task completion. To examine planner performance, two distinct sets of start and goal positions are evaluated, as illustrated in Figure 8.

Initially, the start and goal positions illustrated in Figure 8a are fed into the upper-layer planner. Guided by a sampling-based A* path, the RL algorithm derives the initial set of waypoints ${\mathcal{P}}_{init}$ for the UGV-UAV system, as depicted in Figure 9. The DRL agent was trained for 4000 episodes. In our evaluation, the collision rate is defined as the ratio of the total collision occurrences to the aggregate number of steps within a single episode. The training results, as displayed in Figure 10, show a stable convergence trend of the reward function. Specifically, the initial paths generated by the RL agent exhibit an overall collision rate of 11.89%—comprising approximately 5–10% for the UGV and 20% for the UAV—along with an average NLoS communication rate of nearly 15%. To improve reproducibility, the detailed hyperparameters used for PPO training are summarized in

Table 3. RL training hyperparameters for the upper-layer planner.

Reward Function Parameters
Parameter	Symbol	Value	Description
UGV progress reward weight	${\mathrm{w}}_{p,g}$	2	Weight for UGV progress reward
UAV progress reward weight	${\mathrm{w}}_{p,a}$	4	Weight for UAV progress reward
Collision penalty weight	${\mathrm{w}}_{collision}$	20	Penalty for obstacle collision
Communication blockage penalty weight	${\mathrm{w}}_{comm}$	2	Penalty for NLoS communication
Neural Actor Network Parameters
Layers	Number	Size	Activation Function
Input	1	80 × 100 × 100 + 5	-
Hidden	6	32, 64, 256, 64, 64, 128	ReLU
Output	1	5	Tanh
Neural Critic Network Parameters
Layers	Number	Size	Activation Function
Input	1	80 × 100 × 100 + 5	-
Hidden	6	32, 64, 256, 64, 64, 128	ReLU
Output	1	1	-
PPO Training Parameters
Parameter	Symbol	Value	Description
Actor learning rate	${\mathsf{\alpha }}_{\mathsf{\pi }}$	2 × 10−4	Learning rate of the Actor network
Critic learning rate	${\mathsf{\alpha }}_V$	1 × 10−3	Learning rate of the Critic network
Discount factor	$\mathsf{\gamma }$	0.99	Discount factor for cumulative reward
PPO clipping threshold	$\mathsf{\epsilon }$	0.2	Clipping threshold in PPO objective
Entropy coefficient	$c_{ent}$	0.01	Entropy regularization coefficient
Replay buffer size	$N_{buffer}$	2048	Size of experience replay buffer
Batch size	$N_{batch}$	32	Mini-batch size for PPO update
Number of PPO epochs	$K$	10	Update epochs per iteration
Maximum steps per episode	$T_{\max }$	200	Maximum time steps in each episode
Total number of episodes	$N_{eps}$	4000	Total number of training episodes

.

Subsequently, we apply a linear interpolation process integrated with a back-off strategy to the initial waypoints ${\mathcal{P}}_{init}$ produced by the upper-layer planner. By interpolating 10 intermediate points between each pair of adjacent waypoints, we construct the refined initial solution $P_{init}^{\prime }$, which serves as the input for the lower-layer numerical solver. Then, an obstacle-inflation-based corridor construction algorithm is applied to each initial solution to generate the respective safety corridors for the UGV and the UAV, as illustrated in Figure 11.

Finally, based on $P_{init}^{\prime }$, we initialize the remaining state initial sequence $X$ and the control initial sequence $U$. These sequences, along with the safety corridor constraints, system-level constraints, and the objective function, are fed into the IPOPT solver. This numerical optimization process yields the final cooperative trajectory output, as presented in Figure 12.

To further examine the planning performance of the proposed hierarchical framework under a different start-goal configuration within the same simulated urban canyon environment, we conducted an additional evaluation based on the scenario shown in Figure 8b. In this test, the initial RL-generated paths exhibit a collision rate of 16.7% and an average NLoS communication rate of 17%. After being processed by the lower-layer optimization module with corridor-based constraints, the final cooperative UGV-UAV trajectories become smooth, collision-free, and communication-aware, as shown in Figure 13. These results indicate that the proposed hierarchical framework can improve the quality and feasibility of initial trajectories in the tested simulated urban canyon scenario.

The above experiments were conducted at the trajectory planning level in simulated urban canyon scenarios. The proposed framework is designed as a scenario-specific deployment approach rather than a map-agnostic reinforcement learning method. In practical deployment, the urban map is assumed to be known in advance, and the obstacle map, communication visibility structure, A*-based guidance paths, and RL state representation are constructed according to the given scenario. If the deployment environment changes, the map construction module needs to rebuild the obstacle and communication maps, and the A*-guided RL module should be retrained offline using the updated environmental information before online trajectory refinement is performed by the lower-layer optimizer. Therefore, the current experiments should be interpreted as planning-level validation in the tested urban canyon scenarios, rather than as evidence of zero-shot generalization across arbitrary unseen maps or full real-world system robustness.

6.2. Ablation and Comparison Experiments

This Section provides a systematic evaluation of the role of the proposed algorithm through multi-dimensional testing. First, ablation studies are conducted to assess the contribution of each module; subsequently, comparative experiments are presented to compare its performance with representative baseline methods. All these experiments are based on the set of start and end points in Figure 8a.

6.2.1. Ablation Experiments

To evaluate the role of the upper-layer RL module, ablation experiment 1 removes RL and directly uses the discrete path generated by the A* algorithm as the initial guess for the NLP solver. To define the cooperative terminal state, the UAV target node is aligned with the UGV destination. The CPU execution time of this A*-guided NLP-only planner is compared with that of the full hierarchical framework over 50 independent trials. As shown in Figure 14, the NLP-only planner can generate feasible cooperative trajectories, but it requires a longer and less stable solving process. The proposed framework achieves an average planning time of 5.180 s, whereas the NLP-only planner requires 11.266 s on average, with larger temporal fluctuations.

To examine the role of the lower-layer NLP solver, ablation experiment 2 removes the optimization phase and directly uses the DRL output as the cooperative trajectory. As shown in Figure 15, the pure RL planner can guide the UGV-UAV system toward the goal at the topological level, but the resulting trajectories are discrete and less smooth. The UGV and UAV paths exhibit step-like jumps and oscillations due to the sampled action outputs of the neural policy. Quantitatively, the pure RL strategy produces three collisions, including one UGV collision, two UAV collisions, and three NLoS events within a 26-step horizon.

To assess the intermediate waypoint-processing mechanism, ablation experiment 3 removes the KD-tree-based back-off logic and directly feeds the linearly interpolated RL waypoints into the lower-layer NLP solver. The resulting trajectory is shown in Figure 16.

Without the back-off correction, part of the interpolated UGV initial guess penetrates the inflated obstacles. As a result, the local corridor generation process produces infeasible or near-zero-volume safety regions around these invalid waypoints. These infeasible corridor constraints cause conflicts between equality and inequality constraints in the NLP formulation, leading to solver failure and distorted trajectories that intersect the inflated obstacles.

Overall, the ablation results indicate that the three components play complementary roles in the proposed framework. The upper-layer RL module provides a structured and communication-aware initial guess, reducing the computational burden of the NLP solver. The lower-layer NLP solver improves trajectory smoothness and enforces hard kinematic, collision-avoidance, and communication constraints. The intermediate waypoint-processing mechanism further improves the feasibility of the initial guess and supports reliable safety-corridor generation. Together, these modules contribute to the efficiency, feasibility, and robustness of the proposed hierarchical planner.

6.2.2. Comparison Experiment

To further evaluate the proposed hierarchical cooperative planner, we compare the Layered Planner with three representative baselines. The first baseline is sdSCA, which represents population-based meta-heuristic optimization [40]. The second baseline is RB-PPO, adapted from the multi-UAV path-planning method in [41]. The original RB-PPO improves PPO by using a replay buffer and a rollback-based objective for more stable policy updates. In this paper, it is adapted to the UGV-UAV relay task by replacing the original homogeneous multi-UAV state, action, and reward definitions with the proposed air–ground state representation, UGV/UAV displacement actions, and rewards related to goal progress, obstacle avoidance, NLoS penalty, and communication range. The trained policy directly outputs synchronized UGV-UAV trajectories without lower-layer NLP refinement. The third baseline is RRT*-NLP, adapted from the random-sampling and nonlinear-optimization framework in [42]. The original method first uses RRT* to generate a single-vehicle reference path and then applies nonlinear optimization to improve trajectory smoothness and constraint satisfaction. In this paper, RRT* is used to generate the UGV topological route, while the UAV relay waypoints are initialized according to the UGV route and communication constraints. The resulting cooperative initial trajectory is then refined by NLP under the same obstacle, NLoS, communication range, and heterogeneous kinematic constraints.

For stochastic baselines, including sdSCA and RRT*-NLP, 50 independent trials with different random seeds are conducted, and the results are reported as mean ± standard deviation in

Table 4. Quantitative comparison among the proposed Layered Planner, sdSCA, RB-PPO and RRT*-NLP.

Algorithm	UGV Collision Rate	UAV Collision Rate	NLoS Rate	UGV Smoothness	UAV Smoothness	Computational Times (s)	Solver Success Rate
Layered Planner	0.0%	0.0%	0.0%	0.0157	0.0165	6.172	100% (only once)
sdSCA	24.8 ± 15.5%	0.0 ± 0.0%	48.0 ± 10.1%	0.1842 ± 0.1500	0.7151 ± 0.0977	220.471 ± 75.693	60.9%
RB-PPO	3.8%	7.7%	3.7%	0.7785	0.443	0.594	100% (only once)
RRT*-NLP	0.1 ± 0.6%	0.0 ± 0.3%	0.5 ± 1.2%	0.2202 ± 0.0813	0.3097 ± 0.1331	16.679 ± 4.981	82%

Note: sdSCA and RRT*-NLP are evaluated over 50 trials, while the Layered Planner and RB-PPO are run once because their outputs are deterministic under the fixed policy, map, and start-goal pair.

. The proposed Layered Planner and RB-PPO are evaluated under a fixed trained policy, map, and start-goal pair, so single-run values are reported. Solver success rate is included to assess numerical robustness; for methods without an explicit NLP solver, it indicates whether a valid trajectory is generated.

The proposed Layered Planner achieves 0.0% UGV collision rate, 0.0% UAV collision rate, 0.0% NLoS rate, and the lowest UGV/UAV smoothness values of 0.0157 and 0.0165, respectively. Its total runtime is 6.172 s, including map and communication-visibility construction/update, 0.32 s; A*-based guidance path generation, 0.41 s; upper-layer RL inference, 0.48 s; waypoint interpolation and safe-corridor generation, 0.74 s; NLP transcription and CasADi model construction, 0.56 s; and IPOPT numerical optimization, 3.66 s.

This shows that NLP optimization is the main computational component, while the upper-layer guidance reduces the NLP search burden by providing a structured warm start. Compared with the proposed method, sdSCA has much higher UGV collision and NLoS rates, 24.8 ± 15.5% and 48.0 ± 10.1%, respectively, and a longer runtime of 220.471 ± 75.693 s. RB-PPO is the fastest method, requiring only 0.594 s, but it still produces UGV and UAV collision rates of 3.8% and 7.7%, indicating that learning-based planning alone may not strictly enforce hard feasibility constraints. RRT*-NLP achieves low collision and NLoS rates, but its smoothness, runtime, and solver success rate are still inferior to those of the proposed Layered Planner.

Statistical significance is further assessed by comparing each stochastic baseline with the proposed Layered Planner. A one-sample Wilcoxon signed-rank test with Holm–Bonferroni correction is used, and the significance levels are denoted as * for p < 0.05, ** for p < 0.01, and *** for p < 0.001. As shown in

Table 5. Statistical significance of comparative baselines against the proposed Layered Planner.

Comparison	UGV Collision Rate	UAV Collision Rate	NLoS Rate	UGV Smoothness	UAV Smoothness	Computational Times (s)
sdSCA vs. Layered Planner	1.78 × 10−7 ***	1.000 ns	5.27 × 10−9 ***	2.13 × 10−14 ***	2.13 × 10−14 ***	2.13 × 10−14 ***
Cooperative RRT*-NLP vs. Layered Planner	0.204 ns	0.635 ns	0.020 *	2.13 × 10−14 ***	2.13 × 10−14 ***	5.86 × 10−8 ***
RB-PPO vs. Layered Planner	n/a	n/a	n/a	n/a	n/a	n/a

Note: “ns” denotes no statistically significant difference, and “n/a” denotes that the statistical test is not applicable because the corresponding method is reported as a deterministic single-run result. * and *** indicate p < 0.05 and p < 0.001, respectively.

, the proposed method significantly outperforms sdSCA in UGV collision rate, NLoS rate, UGV/UAV smoothness, and computational time. Compared with RRT*-NLP, it shows significant improvements in NLoS rate, UGV/UAV smoothness, and computational time. These results indicate that the Layered Planner achieves a better balance among topological guidance, constraint satisfaction, trajectory smoothness, computational efficiency, and solver robustness.

7. Discussion

The proposed hierarchical framework is useful, as supported by our ablation studies. First, when the RL-guided upper layer is removed, the NLP solver can still generate feasible trajectories in some cases, but it becomes more computationally expensive and more sensitive to local minima, indicating that a structured initial solution is important for handling the highly non-convex cooperative planning problem. Second, when the lower-layer NLP refinement is removed, the raw RL output remains discrete, less smooth, and does not by itself strictly guarantee kinematic feasibility, collision avoidance, and communication satisfaction. Third, removing the intermediate waypoint-processing mechanism can lead to infeasible initial guesses and less reliable corridor generation, which may further affect solver convergence.

Nevertheless, the current study still has several limitations. Several open research questions and future directions arise from the current study. First, the communication model is based on a binary NLoS indicator. However, this simplified representation does not capture more realistic physical-layer effects, such as distance-dependent path loss, multipath fading, interference, or dynamic blockage. Future work will therefore investigate how to integrate more realistic physical-layer communication models into trajectory optimization. Second, experimental validation is currently restricted to simulated urban canyon scenarios. More specifically, the present evaluation still relies entirely on simplified grid-based simulations, without higher-fidelity simulation environments or real-world validation. Although these simulations provide useful insights into planning-level performance, high-fidelity simulations and real-world UGV-UAV experiments are still needed to further assess practical deployment performance and validate the robustness of the proposed framework.

8. Conclusions

This paper proposed a hierarchical cooperative trajectory planning framework for UGV-UAV systems in communication-constrained urban canyon environments. The framework uses A*-guided RL to generate structured cooperative waypoints and employs corridor-constrained NLP to refine them into feasible and smooth trajectories.

Simulation results show that the proposed framework can generate collision-free, communication-aware, and smooth cooperative trajectories in the tested urban canyon scenarios. Ablation and comparison experiments further indicate its advantages in trajectory feasibility, communication reliability, computational efficiency, and solver success rate over representative baseline methods.