Research on Imitation–Reinforcement Hybrid Machine Learning Algorithms: Application in Path Planning

Abstract

Path planning in complex, dynamic environments presents a significant challenge. Deep Reinforcement Learning (DRL) offers an end-to-end solution but suffers from critical sample inefficiency and a “cold-start” problem. Imitation Learning (IL) accelerates training but is constrained by a performance ceiling and poor generalization. To address these limitations, we propose a novel Imitation–Reinforcement Hybrid Machine Learning Algorithm (Hybrid IL-RL). This framework balances exploration and performance via a two-stage process: First, an offline pre-training phase uses Behavioral Cloning (BC) with “non-expert” A* data from static environments for a “warm start”. Second, an online fine-tuning phase uses a DRL algorithm (SAC) to adapt this policy in complex, dynamic environments, allowing the agent to surpass the teacher’s limitations. Simulation experiments validate the approach. The framework demonstrates significantly faster convergence than DRL algorithms trained from scratch. Most critically, in the dynamic environment, our Hybrid IL-RL algorithm achieved the highest success rate (82.4%), while pure IL methods (BC, GAIL) failed due to poor generalization (e.g., 82.1% collision rate) and pure DRL methods struggled (approx. 51–56% success rate). Our results confirm the hybrid framework effectively solves the cold-start problem while using DRL to break the IL performance ceiling.

1. Introduction

Autonomous path planning is a critical technology in modern robotics, autonomous driving, and autonomous aerial vehicles (AAVs) [1]. As these systems are deployed in increasingly complex and dynamic urban scenarios [2], agents must be able to plan safe and efficient paths in real-time amidst dense obstacles and unforeseen threats. Figure 1 illustrates the macroscopic application domain. Specifically, we target scenarios such as urban UAV navigation or autonomous ground delivery, where an agent must safely traverse a mapped environment containing permanent structures (buildings) and emergent dynamic threats.

Figure 1. Illustration of the target application domain (Urban UAV Navigation).

Traditional path planning algorithms, such as A* or RRT* [3,4], while effective in static environments, often rely on a complete global environment model. Furthermore, they face high computational complexity when dealing with high-dimensional state spaces and sudden obstacles [5], making it difficult to meet real-time decision-making requirements.

In recent years, machine learning methods, particularly Deep Reinforcement Learning (DRL), have provided an end-to-end solution for this problem [6]. DRL algorithms, through their inherent trial-and-error mechanism, enable agents to learn control policies directly from high-dimensional sensor inputs [7], demonstrating significant potential in understanding uncertain environments. However, the DRL paradigm suffers from a critical bottleneck: sample inefficiency issues [8,9]. In tasks with sparse rewards, such as path planning [10], the agent requires a massive number of interactions to accidentally discover a successful path during the initial training phase. This leads to extremely slow policy convergence, known as the “cold-start” problem [11].

To address the sample inefficiency of DRL, Imitation Learning (IL) has been widely studied as an alternative [12]. IL aims to quickly acquire a feasible policy by replicating demonstrations. Although this approach (e.g., Behavioral Cloning, BC [13]) can significantly accelerate training, it also introduces new challenges. First, it heavily relies on high-quality expert data [14], which can be difficult to obtain in complex or hazardous environments. Second, the performance of an IL policy is limited by the demonstrator’s proficiency; it cannot learn a policy that surpasses the performance ceiling of the data [15].

Addressing the respective limitations of DRL (slow start) and IL (low performance ceiling), this paper proposes a novel Imitation–Reinforcement Hybrid Machine Learning Algorithm framework. This framework aims to merge the advantages of both paradigms to achieve an optimal balance between exploration efficiency and final performance in path planning. The framework proposed herein decomposes the complex learning process into two stages. First, in the “pre-training” stage, the agent learns a foundational policy using IL from an easily obtainable “nonexpert” dataset (e.g., generated by a traditional A* algorithm in a static environment). This provides an efficient “warm start” for DRL training [16]. Subsequently, in the “fine-tuning” stage, the agent uses this foundational policy as a starting point to explore autonomously in a more complex environment that includes dynamic obstacles. Through DRL optimization, the agent progressively shifts toward surpassing the performance of the nonexpert teacher, learning to handle dynamic threats and more complex scenarios.

The main contributions of this paper are summarized as follows:

• A novel hybrid imitation–reinforcement learning framework is proposed that, through the combination of IL pre-training and DRL fine-tuning, effectively addresses the core “cold-start” and “sample inefficiency” problems of DRL in path planning tasks.
• The proposed framework reduces the dependency on high-quality “expert demonstrations,” instead leveraging easily generated “nonexpert” data from traditional algorithms for initialization. This ensures rapid convergence while using DRL to break through the performance ceiling imposed by IL.
• The superiority of the proposed algorithm is validated through extensive experiments in complex simulation environments containing both static and dynamic obstacles. The results demonstrate that the presented method achieves significant improvements in both convergence speed and final task success rate compared to pure DRL or pure IL approaches.

The remainder of this paper is organized as follows: Section 2 reviews related work on traditional, DRL-based, and IL-based path planning. Section 3 details the MDP formulation for path planning and the foundational algorithms of DRL and IL. Section 4 elaborates on the design of our proposed hybrid algorithm framework. Section 5 presents the simulation setup, comparative results, and analysis. Section 6 discusses the mechanism and limitations. Finally, Section 7 concludes the paper and discusses future work.

2. Related Work

This chapter aims to review and analyze relevant research in the field of path planning. The work presented in this paper lies at the intersection of Deep Reinforcement Learning (DRL) and Imitation Learning (IL). Therefore, this chapter will focus on two core themes: first, reviewing DRL-based path planning methods and their inherent sample efficiency bottlenecks; second, discussing reinforcement learning approaches that incorporate prior knowledge, particularly imitation learning and hybrid frameworks.

2.1. DRL-Based Path Planning

To overcome the limitations of traditional path planning algorithms, such as improved A* algorithms proposed by Wang et al. [17] or Neural RRT* by Wang et al. [18], which suffer from high computational costs and reliance on accurate models in high-dimensional and dynamic environments [19], researchers have turned to Deep Reinforcement Learning (DRL).

Sutton and Barto [20] laid the foundation for modern reinforcement learning, while the DQN algorithm introduced by Mnih et al. [21] demonstrated the significant potential of combining deep learning with reinforcement learning, enabling end-to-end decision learning. Numerous studies have successfully applied DRL algorithms to navigation tasks. For instance, the DDPG algorithm proposed by Wang and Hu [22] and the SAC algorithm by Haarnoja et al. [23] have achieved continuous action control and obstacle avoidance for robots in simulated environments. Zhao et al. [24] utilized DRL for dynamic target tracking control of UAVs. Zheng et al. [25] applied DRL to path planning for stratospheric airships in complex environments. These methods demonstrate the capability of DRL agents for autonomous exploration and generalization in unknown environments.

However, despite the great potential of DRL, its inherent trial-and-error mechanism leads to significant sample inefficiency issues [26]. In tasks with sparse rewards, such as path planning, the agent struggles to obtain positive reward signals through random exploration during the initial training phase (the “cold-start” problem). Several studies [27] point out that this lack of learning signals results in extremely slow policy convergence, requiring a massive amount of simulation interaction to learn an effective policy, which is often unacceptable in terms of computational resources and time costs.

2.2. Reinforcement Learning with Prior Knowledge and Imitation Learning

As a way to address the sample inefficiency problem of DRL, incorporating prior knowledge, especially through Imitation Learning (IL), has been widely studied [28]. Behavioral Cloning (BC) is the most direct form of IL [29]. The survey by Hussein et al. [30] provides a comprehensive overview of imitation learning methods. The advantage of BC lies in its fast training speed, enabling the agent to quickly acquire basic capabilities. However, BC itself suffers from serious drawbacks. Foster et al. [31] noted that BC performs poorly in long-horizon tasks. Codevilla et al. [32] also revealed the limitations of BC’s generalization ability in autonomous driving. Furthermore, it faces the challenge of “distributional shift,” a problem highlighted by Menda et al. [33], where even small errors during execution can lead the agent into states unseen in the demonstration data, causing error accumulation.

Given the distinct shortcomings of pure DRL (slow training) and pure IL (low performance, poor generalization), researchers have begun exploring hybrid learning algorithms that combine both. Hester et al. (whose work relates to DQfD, though the specific reference provided is Chen et al. [34] on Active DQfD) proposed accelerating DRL training by incorporating “expert” demonstration data into the experience replay buffer. The survey by Argall et al. [35] reviews various methods for robot learning from demonstration, often assuming expert data. Sun et al. [36] also combined RL with LfD (Learning from Demonstrations) for nonprehensile manipulation tasks. While these methods improve sample efficiency to some extent, they still heavily rely on high-quality and often difficult-to-obtain “expert demonstrations”.

Recent studies have started exploring the use of “nonexpert” demonstrations or offline data. For example, Zhang et al. [27] proposed using Artificial Potential Field (APF) as a nonexpert teacher to guide DRL training online. Ho and Ermon [37] introduced Generative Adversarial Imitation Learning (GAIL), which learns policies from demonstrations via adversarial training and has relatively lower requirements on data quality. Ball et al. [38] investigated how to efficiently combine offline data with online RL training. Shenfeld et al. [39] studied reinforcement learning under the guidance of an online teacher. These works recognize that even suboptimal prior knowledge can effectively guide the early exploration of DRL.

In summary, although hybrid learning is a prevailing trend, existing methods mostly focus on leveraging “expert” data or require complex “online teacher” guidance mechanisms. How to efficiently utilize the most readily available “nonexpert” offline data (such as static paths generated by the A* algorithm) to address the “cold-start” problem of DRL, and how to design a concise two-stage framework where the DRL stage can surpass the limitations of the nonexpert teacher (e.g., learning to handle dynamic obstacles), remains an issue requiring further investigation. A detailed comparison of these methods is provided in Table 1.

Table 1. Comparison of related path planning methodologies discussed in this section.

Method Category	Key Representatives	Advantages	Limitations
Traditional Planning	A*, RRT* [3,4]	Optimality in static maps; Theoretical completeness.	High computational cost in high-dim spaces; Fails in dynamic environments without fast replanning.
Deep Reinforcement Learning (DRL)	SAC, DDPG, PPO [22,23]	Handles high-dim sensory inputs; Continuous control; Learns dynamic avoidance.	Sample inefficiency; “Cold-start” problem; Unstable convergence.
Imitation Learning (IL)	BC, GAIL [13,37]	Fast training; No need for complex reward engineering.	Performance ceiling (limited by demonstrator); Poor generalization to unseen states (Distribution Shift).
Hybrid IL-RL (Ours)	BC Pre-training + SAC Fine-tuning	Fast convergence (Warm Start); Surpasses teacher performance; Robust dynamic avoidance.	Requires offline dataset generation; Two-stage training complexity.

The research work presented in this paper is developed against this background.

3. Problem Formulation and Foundational Algorithms

This chapter establishes the theoretical foundation for the hybrid imitation–reinforcement learning algorithm. First, the path planning application problem is mathematically formulated as a Markov Decision Process (MDP). Following this, the fundamental principles of Deep Reinforcement Learning (DRL) and Imitation Learning (IL) are introduced.

3.1. Path Planning Problem Formulation

The objective of the path planning task is to enable an agent to autonomously find an optimal path from a starting point $p_{start}$ to a target point $p_{goal}$ within a potentially complex environment possibly containing dynamic obstacles. This sequential decision-making problem is modeled as a Markov Decision Process (MDP), defined by a tuple $(\mathcal{S},\mathcal{A},P,R,\gamma )$ [40].

The state space $\mathcal{S}$ represents the complete observation of the environment available to the agent at time t. A well-designed state $s_t\in \mathcal{S}$ is crucial for algorithmic success. In this study, the state comprises proprioceptive information, such as the agent’s position $p_t$ and velocity $v_t$; environmental perception, like local obstacle information $o_t$ obtained through sensors; and task-relevant information, such as the relative vector to the goal $(p_{goal}-p_t)$ [41].

The action space $\mathcal{A}$ defines the control commands the agent can execute. To ensure smooth trajectories and reduce the simulation-to-reality gap, a continuous action space is employed. For instance, the action $a_t\in \mathcal{A}$ could be defined as $a_t=[v,\omega ]$, representing the target linear velocity v and angular velocity $\omega$ for the agent at time t.

The state transition probability $P\left(s^{\prime }\right|s,a)$ describes the probability of transitioning to state $s^{\prime }$ after executing action a in state s. In model-free DRL, this transition function is typically unknown, and the agent learns implicitly through interaction with a simulator or the real environment.

The reward function $R(s,a,s^{\prime })$ provides the core learning signal guiding the policy optimization, mapping a transition $(s,a,s^{\prime })$ to a scalar reward value $r_t$. To achieve multiple objectives such as efficiency and safety, the reward is often designed as a composite function, potentially including components like a goal reward $R_{goal}$, collision penalties $R_{collision}$ and $R_{dynamic}$, and a step penalty $R_{step}$.

The discount factor $\gamma \in [0,1]$ balances the importance of immediate versus future rewards. Figure 2 visualizes these components: the agent (blue circle) perceives the environment through sensor readings $o_t$ and the relative goal vector $(p_{goal}-p_t)$, while deciding its next move $a_t$ to navigate towards the goal (star) avoiding both static and dynamic obstacles.

Figure 2. MDP formulation of the path planning problem.

The path planning problem is formally defined as finding an optimal policy ${\pi }^{*}$ that maximizes the expected cumulative discounted reward $J\left(\pi \right)$ subject to kinematic and safety constraints:

$${\pi }^{*}=arg\underset{\pi }{max}J\left(\pi \right)=arg\underset{\pi }{max}{\mathbb{E}}_{\tau \sim \pi }\left[\sum _{t=0}^T{\gamma }^{t}r(s_t,a_t)\right]$$

(1)

subject to

$$\begin{array}{cc}\hfill s_{t+1}& =f_{trans}(s_t,a_t)(\mathrm{State}\mathrm{Transition})\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill \underset{x\in {\mathcal{O}}_j}{min}\left|\right|p_t-x{\left|\right|}_2& \ge R_{agent},\forall j(\mathrm{Collision}\mathrm{Avoidance})\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill a_t& \in [0,v_{max}]\times [-{\omega }_{max},{\omega }_{max}](\mathrm{Actuation}\mathrm{Limits})\hfill \end{array}$$

(4)

where $f_{trans}$ denotes the kinematic model governing the agent’s motion, ${\mathcal{O}}_j$ represents the set of points occupied by the j-th obstacle (including both static buildings and dynamic threats), and $R_{agent}$ denotes the safety radius of the agent. Equation (3) ensures that the minimum distance from the agent’s center to any point within an obstacle remains larger than the agent’s physical size.

To ensure the problem is tractable, we make the following standard assumptions:

(1)

The agent operates in a 2D plane with continuous state and action spaces.

(2)

The agent has access to its precise global position (e.g., via GPS/IMU).

(3)

Static obstacles are known a priori, while dynamic obstacles are detected only within the sensor range.

3.2. Foundational Algorithms

The hybrid framework presented in this paper builds upon two fundamental machine learning algorithms: Imitation Learning (IL) and Deep Reinforcement Learning (DRL).

3.2.1. Imitation Learning (IL)

Imitation learning aims to acquire a policy by learning from demonstrations provided by a teacher (Expert or Nonexpert Demonstrator). A demonstration dataset $D_{demo}={\left\{(s_i,a_i)\right\}}_{i=1}^N$ is assumed, containing N state–action pairs. Behavioral Cloning (BC) represents the most direct form of IL. BC frames the imitation task as a supervised learning problem. Its goal is to train a policy network ${\pi }_{\theta }\left(s\right)$, parameterized by $\theta$, to minimize the discrepancy between its output action and the teacher’s action. This is typically achieved by minimizing a loss function, such as the Mean Squared Error (MSE), denoted as $L_{BC}\left(\theta \right)$ [28]:

$$L_{BC}\left(\theta \right)={\mathbb{E}}_{(s,a)\sim D_{demo}}\left[\left|\right|{\pi }_{\theta }\left(s\right)-a{\left|\right|}^2\right]$$

(5)

The main advantage of BC is its simplicity and fast training, making it suitable for providing a reasonable initial policy and addressing the “cold-start” issue in DRL.

3.2.2. Deep Reinforcement Learning (DRL)

Unlike IL, Deep Reinforcement Learning (DRL) enables an agent to learn autonomously through trial-and-error interactions with its environment, guided solely by the reward signal. By leveraging the function approximation capabilities of deep neural networks to represent the policy ${\pi }_{\theta }\left(a\right|s)$ and/or value functions (like the Q-function $Q_{\varphi }(s,a)$ or V-function $V_{\varphi }\left(s\right)$), DRL can handle high-dimensional state and action spaces. Actor-Critic (AC) methods are a prominent class of DRL algorithms well-suited for continuous action spaces. Algorithms like Soft Actor-Critic (SAC) further enhance exploration and stability by incorporating techniques such as entropy maximization. The primary advantage of DRL is its ability to potentially discover optimal policies that surpass human performance or traditional algorithms.

4. Imitation–Reinforcement Hybrid Machine Learning Algorithm Design

This chapter details the design and implementation of the proposed Imitation–Reinforcement Hybrid Machine Learning Algorithm. The hybrid framework designed herein aims to integrate the strengths of both paradigms, leveraging the prior knowledge from IL to address DRL’s cold-start problem, and subsequently employing DRL’s autonomous exploration capability to overcome the performance limitations of IL.

4.1. Algorithm Framework Overview

To address the challenge of blind exploration and difficult convergence in the early stages of DRL training for path planning tasks, a two-stage hybrid machine learning algorithm framework is proposed. The execution flow of this framework decomposes the complex policy learning task into two core phases:

4.1.1. Phase 1: Policy Pre-Training (Offline)

The primary objective of this stage is to resolve the cold-start problem. Instead of relying on DRL’s random exploration, the Behavioral Cloning (BC) paradigm from imitation learning is utilized. A policy network is trained in a supervised manner using a “nonexpert demonstration” dataset $D_{\mathit{non-exp}}$, which is generated by a traditional algorithm in static environments. This phase yields a pre-trained policy ${\pi }_{pre}$, represented by parameters ${\theta }_{pre}$.

4.1.2. Phase 2: Policy Fine-Tuning (Online)

The main goal here is to surpass the performance ceiling inherent in IL derived from nonexpert data and adapt to dynamic environments. The pre-trained policy ${\pi }_{pre}$ obtained from the first phase is used as the initial policy for a DRL agent (Warm Start). Through interaction with this dynamic environment and guided by trial-and-error learning driven by a DRL algorithm (e.g., SAC), the agent autonomously explores and refines its policy, eventually converging to an optimal policy ${\pi }_{final}$. Figure 3 illustrates the detailed execution flow of this architecture. The left panel shows the offline pre-training phase where non-expert data from A* is cloned to initialize the policy. The right panel depicts the online fine-tuning phase, where the agent interacts with the dynamic environment using the DRL loop (Store-Sample-Update), progressively optimizing the policy parameters $\theta$ and $\varphi$ to handle dynamic threats.

Figure 3. Architecture of the proposed two-stage Imitation–Reinforcement Hybrid learning framework.

4.2. Phase 1: Policy Pre-Training via Behavioral Cloning

The objective of this phase is to quickly obtain a competent initial policy.

4.2.1. Nonexpert Dataset Construction

The prior knowledge utilized originates from the A* algorithm, serving as a nonexpert teacher. It is termed nonexpert as A* generates optimal paths only for static environments, lacking awareness of dynamic elements present in the target task. The dataset construction involves generating K distinct static simulation environments. In each, A* plans paths ${\tau }_{\mathit{non-exp}}$ between random start and goal points. State–action pairs $(s_i,a_i)$ are sampled along these paths. These pairs are aggregated across all environments to form the nonexpert dataset $D_{\mathit{non-exp}}$ [35]:

$$D_{\mathit{non-exp}}={\left\{(s_i,a_i)\right\}}_{i=1}^N$$

(6)

where N is the total number of samples collected.

4.2.2. Policy Network Architecture

A deep neural network ${\pi }_{\theta }\left(s\right)$, parameterized by $\theta$, represents the policy. Its input is the state $s_t$, and its output is the continuous action $a_t$. For example, the action could be defined as target linear and angular velocities [20]:

$$a_t=[v,\omega ]$$

(7)

4.2.3. Pre-Training Process

This phase trains the policy network ${\pi }_{\theta }$ to mimic the behavior in $D_{\mathit{non-exp}}$ using supervised learning. Batches b containing $(s_b,a_b)$ pairs are sampled from $D_{\mathit{non-exp}}$, and the network parameters $\theta$ are optimized by minimizing the Mean Squared Error (MSE) loss function $L_{BC}\left(\theta \right)$, defined as [28]

$$L_{BC}\left(\theta \right)={\mathbb{E}}_{(s,a)\sim D_{\mathit{non-exp}}}\left[\left|\right|{\pi }_{\theta }\left(s\right)-a{\left|\right|}^2\right]$$

(8)

4.3. Phase 2: Policy Fine-Tuning via DRL

This phase aims to adapt the policy to dynamic environments and improve upon the nonexpert’s capabilities.

4.3.1. DRL Algorithm Selection and Agent Initialization

Soft Actor-Critic (SAC) is selected as the DRL algorithm for fine-tuning due to its sample efficiency and stability in continuous action spaces. The SAC agent comprises an Actor network (${\pi }_{\theta }$) and two Critic networks ($Q_{{\varphi }_1},Q_{{\varphi }_2}$). The Actor network’s parameters $\theta$ are initialized with ${\theta }_{pre}$ from Phase 1. This warm start significantly accelerates the DRL learning process.

4.3.2. Reward Function for DRL Training

The reward function $r_t$ guiding the DRL phase is designed to encourage safe and efficient goal-reaching behavior. The reward at time step t is composed as follows [20]:

$$r_t=R_{goal}+R_{static\_collision}+R_{dynamic\_threat}+R_{step}$$

(9)

4.3.3. DRL Training Process and Update Rules

Fine-tuning follows the standard off-policy Actor-Critic loop. The agent interacts with the dynamic environment $E_{dynamic}$ using its current policy ${\pi }_{\theta }$, storing transitions in a replay buffer $D_{RL}$. The Critic networks ($Q_{{\varphi }_1},Q_{{\varphi }_2}$) are updated by minimizing the Bellman error. The target Q-value $y_t$ incorporates the next state action $a_{t+1}$, the minimum of the target Q-networks’ values, and the entropy term. One has [23]

$$\begin{array}{cc}\hfill y_t=& r_t+\gamma {\mathbb{E}}_{a_{t+1}\sim {\pi }_{\theta }(·|s_{t+1})}[\underset{i=1,2}{min}{Q}_{{\varphi }_i^{\prime }}(s_{t+1},a_{t+1})\hfill \\ & -\alpha log{\pi }_{\theta }\left(a_{t+1}\right|s_{t+1})]\hfill \end{array}$$

(10)

The Critic loss $L_{Critic}\left({\varphi }_i\right)$ is then computed as [23]

$$L_{Critic}\left({\varphi }_i\right)={\mathbb{E}}_{(s_t,a_t,r_t,s_{t+1})\sim D_{RL}}\left[{(Q_{{\varphi }_i}(s_t,a_t)-y_t)}^2\right],i=1,2$$

(11)

The Actor network ${\pi }_{\theta }$ is updated by maximizing the expected Q-value of its actions plus the policy entropy. The Actor loss $L_{Actor}\left(\theta \right)$ is defined as [23]

$$L_{Actor}\left(\theta \right)={\mathbb{E}}_{s_t\sim D_{RL},a_t\sim {\pi }_{\theta }(·|s_t)}\left[\alpha log{\pi }_{\theta }\left(a_t\right|s_t)-\underset{i=1,2}{min}{Q}_{{\varphi }_i}(s_t,a_t)\right]$$

(12)

4.4. Hybrid Algorithm Pseudocode

The complete procedure of the proposed Imitation–Reinforcement Hybrid Machine Learning Algorithm is summarized in Algorithm 1.

4.5. Numerical Illustration of the Learning Process

To clarify the internal mechanism of the proposed hybrid algorithm, we present a simplified numerical walkthrough based on a critical decision snapshot from the dynamic scenario (refer to the experimental results in Section 5). Consider a time step t where the agent is located at $p_t=(10, 10)$ and a dynamic obstacle is approaching from $p_{obs}=(12, 10)$.

4.5.1. Phase 1: BC Pre-Training Calculation

Suppose the non-expert teacher (A*) suggests an action $a_{expert}=[1.0, 0.0]$ (move straight at max speed). The initialized policy network ${\pi }_{\theta }\left(s_t\right)$ outputs a slightly deviant action $a_{pred}=[0.8, 0.2]$. According to Equation (8), the Behavioral Cloning loss for this sample is calculated as

$$L_{BC}=\left|\right|[0.8, 0.2]-{[1.0, 0.0]\left|\right|}^2={(0.8 - 1.0)}^2+{(0.2-0.0)}^2=0.04+0.04=0.08$$

(13)

Minimizing this loss aligns the agent’s initial tendency with the teacher’s path.

4.5.2. Phase 2: DRL Fine-Tuning Calculation

In the fine-tuning phase, assume the agent takes action $a_t$ and enters a state $s_{t+1}$ where the distance to the obstacle drops to $0.15m$ (collision, as $R_{agent}=0.2m$). The reward function (Equation (9)) triggers a penalty. Assuming $R_{collision}=-10$ and $R_{step}=-0.01$,

$$r_t=0+(-10)+0+(-0.01)=-10.01$$

(14)

For the network update, assume the current Q-value estimate is $Q(s_t,a_t)=-2.0$, the discount factor $\gamma =0.99$, and the target Q-value for the next state is estimated as $min{Q}_{target}(s_{t+1},a_{t+1})=-5.0$ (predicting future failure). The target $y_t$ (ignoring entropy for brevity) is

$$y_t\approx r_t+\gamma ·(-5.0)=-10.01+0.99(-5.0)=-14.96$$

(15)

The Critic loss (Equation (11)) seeks to close the gap between the current estimate ($-2.0$) and the new reality ($-14.96$):

$$L_{Critic}={(-2.0-(-14.96))}^2={\left(12.96\right)}^2\approx 167.96$$

(16)

This large gradient strongly discourages the action leading to collision, enabling the agent to learn dynamic avoidance where the A* teacher failed.

Algorithm 1 Imitation–Reinforcement Hybrid Algorithm

1: Input: Set of static environments $E_{static}$, Dynamic environment $E_{dynamic}$, DRL algorithm ${\mathcal{ALG}}_{DRL}$ (e.g., SAC)   2: Output: Final policy ${\pi }_{final}$   3: // Phase 1: Imitation Learning Pre-training (Offline)   4: Initialize nonexpert dataset $D_{\mathit{non-exp}}\leftarrow \varnothing$   5: for each static environment $e\in E_{static}$ do   6:     Plan path ${\tau }_{\mathit{non-exp}}$ in e using A* (nonexpert teacher)  7:      Sample state–action pairs $(s,a)$ from ${\tau }_{\mathit{non-exp}}$   8:     Add $(s,a)$ to $D_{\mathit{non-exp}}$   9: end for 10: Initialize policy network ${\pi }_{\theta }$ with parameters $\theta$ 11: repeat 12:     Sample a batch $(s_b,a_b)$ from $D_{\mathit{non-exp}}$ 13:     Compute BC loss $L_{BC}\left(\theta \right)$ using Equation (8) 14:     Update $\theta$ using gradient descent on $L_{BC}\left(\theta \right)$ 15: until $\theta$ converges 16: Set pre-trained parameters ${\theta }_{pre}\leftarrow \theta$ 17: // Phase 2: Reinforcement Learning Fine-tuning (Online) 18: Initialize DRL agent’s policy network parameters $\theta \leftarrow {\theta }_{pre}$ (Warm start) 19: Initialize Critic network parameters ${\varphi }_1,{\varphi }_2$ randomly 20: Initialize Target Critic networks ${\varphi }_1^{\prime }\leftarrow {\varphi }_1,{\varphi }_2^{\prime }\leftarrow {\varphi }_2$ 21: Initialize DRL replay buffer $D_{RL}\leftarrow \varnothing$ 22: for $episode=1$ to M do 23:     Reset $E_{dynamic}$, get initial state $s_0$ 24:     for $t=0$ to $T-1$ do 25:         Select action $a_t\leftarrow {\pi }_{\theta }\left(s_t\right)+\mathrm{noise}$ 26:         Execute $a_t$ in $E_{dynamic}$, observe $s_{t+1}$, $r_t$ 27:         Store transition $(s_t,a_t,r_t,s_{t+1})$ in $D_{RL}$ 28:         Sample a minibatch of transitions from $D_{RL}$ 29:         Compute target Q-value $y_t$ using Equation (10) 30:         Update Critic parameters ${\varphi }_1,{\varphi }_2$ by minimizing $L_{Critic}$ (Equation (11)) 31:         Update Actor parameters $\theta$ by minimizing $L_{Actor}$ (Equation (12)) 32:         Update temperature $\alpha$ (if using auto-tuning SAC) 33:         Update Target Critic networks 34:         $s_t\leftarrow s_{t+1}$ 35:         if terminal state reached then 36:             break 37:         end if 38:     end for 39: end for 40: return Final policy ${\pi }_{final}\leftarrow {\pi }_{\theta }$

5. Experiments and Results Analysis

This chapter quantitatively evaluates the performance of the Hybrid IL-RL algorithm. The experiments compare the hybrid algorithm against several baseline methods to validate its advantages in terms of training efficiency, task success rate, and path quality.

5.1. Experimental Setup

5.1.1. Simulation Environment

The experiments utilize a 2D simulation environment built based on Python Gym (v0.26.2). Specific parameters are detailed in Table 2. The environment contains randomly generated static circular obstacles and dynamic rectangular obstacles.

Table 2. Simulation Environment Parameters.

Parameter	Value	Unit	Description
Map Size	20 × 20	m	Dimensions of area
Num. Static Obstacles	10		Circular, radius 1.5 m
Num. Dynamic Obstacles	3		Rect., 3 × 3 m, speed 0.5 m/s
Agent Radius	0.2	m	For collision detection
Goal Region Radius	0.3	m	Threshold for reaching goal
Max Episode Steps	500	steps	Timeout threshold
Simulation Timestep $\Delta t$	0.1	s	Time per decision step

5.1.2. Agent State and Action Spaces

The agent’s state $s_t$ is designed to include its own kinematic information and environmental perception. The agent’s action $a_t$ is defined as a 2D continuous vector $[v,\omega ]$. Details are provided in Table 3.

Table 3. Agent State and Action Space Definitions.

Type	Component	Range	Description
State $s_t$	Rel. Goal Dist/Angle	2D	Norm. dist and angle
	Agent Linear Vel. v	[−1, 1]	Relative to $v_{max}$
	Agent Angular Vel. $\omega$	[−1, 1]	Relative to ${\omega }_{max}$
	LiDAR Readings $o_t$	10D	Sim. distances (max 5 m)
Action $a_t$	Target Linear Vel. v	[0, 1]	Range [0, $v_{max}$]
	Target Ang. Vel. $\omega$	[−1, 1]	Range [−${\omega }_{max}$, ${\omega }_{max}$]
	$v_{max}$ (Max Vel.)	1.0	m/s
	${\omega }_{max}$ (Max Ang.)	$\pi /2$	rad/s

5.1.3. Algorithm Implementation and Hyperparameters

All neural network-based algorithms are implemented using PyTorch (v1.13.1). Core hyperparameter settings are listed in Table 4. The nonexpert dataset $D_{\mathit{non-exp}}$ is generated by running the A* algorithm in 100 different static environments, totaling approximately 1 million state–action pair samples.

Table 4. Core Algorithm Hyperparameters.

Parameter	Value	Description
Learning Rate	3 × ${10}^{-4}$	Adam optimizer
Discount Factor $\gamma$	0.99
Replay Buffer Size ($D_{RL}$)	1 × ${10}^6$
Training Batch Size	256
Network Architecture (MLP)	[256, 256]	Hidden layer units
Activation Function	ReLU
SAC: Target Entropy	−dim($\mathcal{A}$)	Negative action dimension
SAC: Target Network Update $\tau$	0.005	Soft update coefficient
PPO: Clip Range $ϵ$	0.2	PPO clipping parameter
GAIL: Discriminator Network	[128, 128]
Total Training Steps	2 × ${10}^6$

5.2. Baseline Algorithms

The Hybrid IL-RL is compared against:

• SAC-from-Scratch (SAC-FS): SAC trained without pre-training.
• PPO-from-Scratch (PPO-FS): PPO trained without pre-training.
• Behavioral Cloning (BC): Pure imitation learning using ${\pi }_{pre}$.
• Generative Adversarial Imitation Learning (GAIL): Trained using $D_{\mathit{non-exp}}$.
• A* Algorithm: Traditional planning with periodic replanning.

5.3. Results Presentation and Analysis

5.3.1. Training Efficiency Analysis

Figure 4 displays the training curves. The horizontal axis represents training steps, and the vertical axis represents the average success rate in the dynamic environment. Benefiting from the warm start provided by BC pre-training, Hybrid IL-RL exhibits a much higher initial success rate compared to SAC-FS and PPO-FS. Furthermore, Hybrid IL-RL requires significantly fewer training steps to reach a relatively high stable success rate.

Figure 4. Training curves comparing the convergence speed and success rate of different algorithms.

5.3.2. Final Performance Comparison

Table 5 summarizes the final performance metrics. In the Static Environment, A* achieves the highest success rate (99.5%). Hybrid IL-RL and BC also exhibit high success rates (around 95–97%), demonstrating effective obstacle avoidance learned from A*. Pure DRL algorithms show a slightly larger drop in success rate. In the Dynamic Environment, Hybrid IL-RL maintains a clear advantage, achieving the highest success rate (82.4%). Its collision rate (14.3%) is the lowest among learning methods. Pure imitation learning algorithms (BC and GAIL) exhibit drastically reduced success rates, confirming their inability to generalize. Pure DRL algorithms also struggle, resulting in much lower success rates (around 51–56%).

Table 5. Final performance comparison of different algorithms in static and dynamic environments. Results are averaged over 5 random seeds (reported as Mean ± Standard Deviation).

Scenario	Algorithm	Success Rate (%)	Path Length (m)	Collision Rate (%)	Time (ms)
Static	Hybrid IL-RL	94.8 ± 1.1	25.8 ± 0.5	3.0 ± 1.0	2.1 ± 0.2
	SAC-FS	91.4 ± 2.0	26.3 ± 1.0	5.3 ± 1.4	2.1 ± 0.1
	PPO-FS	93.2 ± 1.8	27.5 ± 0.9	5.3 ± 1.0	1.8 ± 0.1
	BC	96.6 ± 0.9	25.6 ± 0.6	3.0 ± 0.6	1.6 ± 0.1
	GAIL	93.2 ± 1.4	26.1 ± 0.7	3.5 ± 0.9	1.9 ± 0.2
	A*	99.5 ± 0.1	24.9 ± 0.0	0.1 ± 0.0	40.1 ± 12.5
Dynamic	Hybrid IL-RL	82.4 ± 2.5	29.3 ± 1.8	14.3 ± 2.1	2.1 ± 0.2
	SAC-FS	56.1 ± 5.8	31.7 ± 3.3	40.5 ± 4.5	2.1 ± 0.1
	PPO-FS	51.1 ± 6.0	32.4 ± 2.8	43.9 ± 5.0	1.8 ± 0.2
	BC	14.4 ± 1.1	27.0 ± 2.4	82.1 ± 1.3	1.5 ± 0.1
	GAIL	20.7 ± 2.4	28.5 ± 2.1	76.5 ± 3.0	2.1 ± 0.2
	A* (replan)	48.8 ± 8.0	31.8 ± 4.0	48.0 ± 7.5	103.5 ± 45.1

Regarding computational efficiency, the Hybrid IL-RL agent averages 2.1 ms per decision step, which comfortably satisfies the real-time control requirement of 100 ms (10 Hz) for autonomous systems. In contrast, the A* algorithm (with replanning) averages 103.5 ms, which strictly exceeds this safety latency threshold, rendering it unsuitable for high-speed dynamic avoidance despite its theoretical optimality in static maps.

Furthermore, independent two-sample t-tests confirm that the success rate improvement of Hybrid IL-RL over SAC-FS is statistically significant ($p<0.05$), validating the robustness of the proposed method against random initialization seeds.

Figure 5 and Figure 6 visualize the trajectory differences generated within the 20 m × 20 m simulation environment.In the static case (Figure 5), the paths generated by BC and Hybrid IL-RL largely overlap with the optimal A* path, confirming the successful imitation of global navigation skills. In contrast, pure RL baselines (SAC-FS and PPO-FS) fail to converge to the optimal geometric path, exhibiting unnecessary detours and oscillations.However, in the dynamic scenario (Figure 6), the divergence is significant. While the A* planner (black line) attempts to follow a geometrically shortest path regardless of the moving threat, inevitably leading to a collision, the Hybrid IL-RL agent (blue line) proactively deviates from the reference trajectory to perform an evasive maneuver. This demonstrates that the hybrid agent has learned to utilize the available space within the 20 m × 20 m area to bypass dynamic obstacles safely before realigning with the goal direction.

Figure 5. Comparison of example paths generated by different algorithms in a typical static environment test case.

Figure 6. Comparison of example paths generated by different algorithms in a typical dynamic environment test case.

5.4. Ablation Study: Impact of Non-Expert Dataset Size

To evaluate the robustness of the proposed framework and the influence of the non-expert dataset size on the “warm start” quality, we conducted an ablation study. We trained the policy using subsets of the original dataset (containing 20%, 50%, and 100% of the generated A* trajectories) and observed the performance during the initial phase and the final convergence.

Table 6 summarizes the results. “Initial Success Rate” refers to the performance of the agent in the dynamic environment immediately after BC pre-training (before DRL fine-tuning).

Table 6. Ablation study on the size of the non-expert dataset ($D_{\mathit{non-exp}}$). The 100% dataset corresponds to the default setting used in previous experiments.

Dataset Size (Ratio)	Trajectories (Count)	Initial Success Rate (After BC)	Final Success Rate (After Hybrid)	Convergence Speed (Steps to 80%)
20%	200	12.1%	80.5%	∼1.65 M
50%	500	25.8%	81.7%	∼1.32 M
100%	1000	39.2%	82.4%	∼1.05 M

As shown in Table 6, reducing the dataset size significantly impacts the Initial Success Rate. The policy trained with only 20% of the data provides a limited starting point (12.1%), forcing the DRL stage to spend more interaction steps (∼1.65 M) to recover and learn dynamic avoidance. In contrast, the full dataset (100%) provides a decent warm start (39.2%, matching the initial performance in Figure 4), allowing the agent to reach the target performance much faster (∼1.05 M steps). Crucially, the Final Success Rate remains relatively stable across all settings (∼80–82%), demonstrating that the DRL fine-tuning stage is robust enough to compensate for a suboptimal pre-training, provided sufficient training time is allowed.

6. Discussion

The experimental results presented in Table 5 reveal a notable phenomenon: the Hybrid IL-RL agent significantly outperforms its “teacher” (the A* algorithm) in the dynamic environment (82.4% vs. 48.8% success rate). This section analyzes the underlying mechanisms driving this performance gap and discusses the limitations of the proposed framework.

6.1. Mechanism of Performance Improvement

The core question raised by these results is: How can a DRL agent learning from a “non-expert” teacher eventually surpass that teacher’s capabilities? The answer lies in the hierarchical decomposition of the navigation task into “Global Topology” and “Local Adaptation”.

The A* algorithm, despite its failure in dynamic settings, provides near-optimal guidance for static obstacle avoidance and global pathfinding. In the Pre-training Phase, the agent absorbs this topological knowledge, effectively learning “where to go” and “how to avoid walls.” This resolves the “cold-start” problem of pure DRL, which otherwise struggles to find the goal through random exploration (as seen in the low success rates of SAC-FS and PPO-FS).

In the subsequent Fine-tuning Phase, the DRL agent does not merely replicate the teacher. Instead, starting from a policy that already understands the static environment, the Reinforcement Learning process focuses almost exclusively on the residual task: handling dynamic threats. Since the agent no longer needs to expend sample complexity on learning basic navigation, it can efficiently adapt its policy to perform local evasive maneuvers that the A* teacher could not demonstrate. This “Warm Start + Fine-Tuning” strategy allows the student to retain the teacher’s global wisdom while acquiring new local reflexes.

6.2. Failure Analysis of Baseline Methods

The poor performance of the A* algorithm in the dynamic scenario (48.8% success rate) is attributed to its reliance on a static global map. Although we implemented a re-planning mechanism, the discrete update frequency and the lack of velocity prediction for dynamic obstacles inevitably lead to collisions when the threat speed exceeds the planner’s reaction horizon. Similarly, pure IL methods (BC and GAIL) failed (approx. 15–20% success rate) because they fundamentally lack the ability to generalize to states not seen in the static demonstrations. They blindly follow the “perfect” static path, colliding with any dynamic object that intersects it.

6.3. Limitations and Future Work

While the Hybrid IL-RL framework effectively balances efficiency and performance, it relies on the availability of a known static map to generate A* demonstrations. In scenarios where the static environment is completely unknown, this initialization method would not be applicable. Additionally, while this study focused on gradient-based DRL methods due to their sample efficiency, evolutionary approaches such as Genetic Algorithms (GA) or Artificial Bee Colony (ABC) combined with deep learning could offer alternative solutions for escaping local optima, which merits investigation in future work.

7. Conclusions

This paper proposed a novel Imitation–Reinforcement Hybrid Machine Learning algorithm framework (Hybrid IL-RL). The framework adopts a two-stage strategy: First, in the offline pre-training phase, “non-expert” data generated by A* is used via Behavioral Cloning (BC) to provide an efficient “warm start”. Subsequently, in the online fine-tuning phase, this policy is iteratively optimized using a DRL algorithm (SAC) to learn to handle dynamic threats. Simulation experiment results robustly demonstrate the framework’s effectiveness. Hybrid IL-RL converged significantly faster than DRL algorithms trained from scratch. In terms of final performance in the dynamic environment, Hybrid IL-RL showed clear superiority, achieving the highest task success rate (82.4%), whereas pure imitation learning methods failed almost completely, and pure DRL methods struggled. Future work will focus on applying this framework to more complex 3D environments and multi-agent systems.