HRL-Det: Hierarchical Reinforcement Learning for Sequential Object Detection in Aerial Imagery

Abstract

Object detection in unmanned aerial vehicle (UAV) imagery suffers from severe scale variation, dense object packing, and prohibitive computational cost when conventional detectors exhaustively evaluate high-resolution frames. Reinforcement learning (RL)-based sequential detectors offer a promising alternative by formulating localization as an active search process, yet existing methods are limited by discrete-time state transitions, sparse reward signals, and premature policy collapse. In this paper, we propose HRL-Det, a hierarchical reinforcement learning framework that addresses these challenges through two tightly coupled innovations. First, a Neural ODE-driven Continuous-Time Bellman State Evolution module models the agent’s state dynamics as a stochastic differential equation governed by the Hamilton–Jacobi–Bellman equation, enabling fine-grained temporal reasoning with memory-efficient adjoint-based backpropagation. Second, a Lyapunov-Guided Entropy-Regularized Reward Shaping mechanism constructs convergence-promoting dense rewards informed by Lyapunov stability analysis while maintaining exploration diversity through maximum entropy optimization. Extensive experiments on VisDrone2019, DroneVehicle, and MS COCO 2017 show that HRL-Det achieves mAP@0.5 of 0.412, 0.812, and 0.735, respectively, outperforming existing RL-based detectors and achieving competitive accuracy relative to representative non-RL detectors under the same COCO metric, while requiring only 17.3 M parameters and an average of 6.3 search steps per object.

1. Introduction

Object detection in unmanned aerial vehicle (UAV) imagery is a fundamental and challenging task in computer vision, aiming to identify and localize objects of interest from an aerial perspective. It has been widely applied in many real-world scenarios, such as intelligent traffic monitoring [1], emergency search and rescue [2], power line inspection [3], and railway infrastructure surveillance [4]. Due to the high altitude of flight and the complexity of ground environments, improving the detection accuracy and operational efficiency of drones remains an important and challenging research topic [5,6].

In recent years, deep learning-based object detection methods have achieved remarkable progress. Existing detectors can be roughly divided into three categories. Two-stage detectors, such as Faster R-CNN [7] and Cascade R-CNN [8], first generate region proposals and then perform refinement. One-stage detectors, such as the YOLO series [9,10] and SSD [11], directly predict object categories and locations from feature maps for real-time inference. More recently, transformer-based and anchor-free detectors, such as Swin Transformer [12] and CenterNet [13], have further improved performance by modeling long-range dependencies or eliminating predefined anchor constraints.

Although existing deep learning-based detectors have achieved impressive performance, they still face significant limitations in UAV scenarios. Conventional methods usually rely on dense prediction or exhaustive window sliding, which introduces high computational costs when processing high-resolution aerial images on resource-constrained edge devices [14]. Furthermore, these static architectures often suffer from a “performance cliff” when detecting tiny objects in complex backgrounds, leading to high false-positive rates and significant resource waste on irrelevant background regions [15,16]. Therefore, designing an efficient framework that balances precision and computational overhead is crucial.

To overcome the inefficiency of dense prediction, reinforcement learning-based (RL) object detection has emerged as a promising technical route. Unlike traditional detectors, RL-based methods formulate localization as a sequential decision-making process where an agent adjusts the bounding box based on image states and rewards [17,18]. This strategy mimics the human visual “saccade” mechanism, allowing the system to focus only on regions of interest through a global-to-local search [19]. By ignoring vast redundant background areas, RL-based frameworks provide a low-computational alternative for autonomous object search in large-scale aerial imagery [20].

Despite the progress achieved by reinforcement learning-based detectors, several research gaps still remain. Existing RL-based methods often utilize insufficient feature representations for complex environments, making it difficult to perceive extremely small targets [21]. Moreover, conventional Deep Q-Network (DQN) agents suffer from unstable learning and overestimated action values, which weakens the reliability of the search policy [22]. Fixed reward functions also fail to adapt to the dynamic changes in the search process, and the discrete action space limits the final localization precision [23,24]. These limitations restrict the practical application of RL-based detectors in high-precision UAV tasks.

To address the above issues, this paper proposes a hierarchical reinforcement learning object detection framework named HRL-Det. Specifically, we first design a lightweight feature extraction module to obtain compact yet discriminative state representations by fusing multi-scale spatial information. Then, a hybrid reinforcement learning strategy is introduced to improve action evaluation stability and mitigate Q-value overestimation during policy training. Moreover, an adaptive reward function is developed to dynamically guide the agent toward precise object alignment. Finally, a bounding box refinement module is incorporated to overcome the quantization errors of discrete actions, enabling high-precision localization in continuous space. The main contributions of this paper are summarized as follows:

• We propose HRL-Det, a hierarchical reinforcement learning framework for sequential object detection that reduces redundant dense-image evaluation in high-resolution UAV imagery.
• We design a Neural ODE-driven Continuous-Time Bellman State Evolution Module that models the agent’s internal state dynamics as a continuous-time stochastic process governed by neural stochastic differential equations. The resulting evolved state representation is used by a standard Dueling Double DQN for discrete action selection, improving the temporal resolution of the search policy and enhancing robustness to visual ambiguity.
• We introduce a Lyapunov-Guided Entropy-Regularized Reward Shaping Mechanism, which provides convergence-promoting dense reward signals informed by Lyapunov stability analysis and integrates maximum entropy policy optimization. It effectively addresses the challenges of reward sparsity and premature policy collapse, thereby promoting stable and accelerated training convergence.
• Extensive experiments on the VisDrone2019, DroneVehicle, and MS COCO 2017 datasets demonstrate that our HRL-Det outperforms existing RL-based detectors in terms of both detection precision and inference efficiency.

2. Related Work

2.1. Deep Learning for Object Detection in UAV Aerial Imagery

Object detection in UAV aerial imagery presents distinct challenges compared to ground-level vision, including severe scale variation, high object density, and real-time processing constraints under limited computational budgets. The VisDrone benchmark [5] has become the de facto standard for evaluating aerial detectors, revealing that over 52% of annotated targets fall into the small-object category (≤32 × 32 pixels). This scale distribution creates a fundamental bottleneck: when standard two-stage detectors such as Faster R-CNN [7] or single-stage detectors such as RetinaNet [25] are applied to VisDrone, their small-object Average Precision (AP_small) lags behind medium- and large-object AP by 20–30%, a gap that motivates the architectural innovations surveyed below.

The YOLO family has seen sustained adaptation for UAV scenarios. YOLOv8 [10] and its successors [26,27] introduced reparameterized backbones and anchor-free heads that improve throughput on edge-deployed platforms, while transformer-based detectors such as Deformable DETR [28], RT-DETR [29], and DINO [30] achieve stronger cross-scale feature aggregation at the cost of higher parameter counts. Despite these advances, all of these architectures share a fundamental limitation for aerial detection: they perform dense, exhaustive evaluation of the entire image, allocating equal computational resources to cluttered backgrounds and sparse target regions alike. This inefficiency motivates the sequential, attention-driven detection paradigm explored in the present work.

2.2. Reinforcement Learning for Object Detection

Reinforcement learning has emerged as a principled framework for reformulating object detection as an active, sequential decision-making process [19]. The seminal work of Caicedo and Lazebnik [17] established the basic MDP formulation for single-object localization, wherein an agent iteratively applies spatial transformation actions to refine a bounding-box hypothesis. Subsequent work extended this paradigm in complementary directions: Mathe et al. [31] incorporated visual attention into the policy; Bellver et al. [18] introduced hierarchical search trees; and Jie et al. [32] proposed tree-structured policies for sequential multi-class localization.

More recent advances have focused on improving the efficiency and robustness of RL-based detectors. Kong et al. [24] proposed collaborative deep reinforcement learning for joint object search, where multiple agents coordinate to localize objects. Pirinen and Sminchisescu [33] integrated RL into the region proposal stage of two-stage detectors, enabling adaptive proposal sampling. Uzkent and Yoon [20] applied RL to zoom-in scheduling for satellite imagery, learning when to invoke a high-resolution sub-detector. Liu et al. [21] proposed an attention-guided cascade RL framework that augments the detection policy with spatial attention maps for progressive object refinement. Ding et al. [34] introduced decision transformers for active object detection, recasting the sequential search problem as a sequence modeling task. Zhang et al. [35] developed an evolutionary reinforcement learning approach for scale optimization on drone imagery, evolving detection policies to handle the extreme scale variation in aerial scenes. Most recently, the LHAR-RLD framework [23] combined a hybrid DQN with adaptive dynamic reward functions and RoI-Align-based bounding-box regression, achieving state-of-the-art RL detection performance prior to the present work.

Beyond static-scene localization, RL has been applied to active perception in aerial and maritime platforms. Akhloufi et al. [36] pioneered a “drones chasing drones” paradigm in which a follower UAV learns a pursuit policy using deep search-area proposals. Alpdemir and Sezgin [37] demonstrated RL-driven navigation for ground-penetrating radar (GPR) surveys, where the agent directs a sensor platform to regions of predicted subsurface anomaly. These works highlight a shared insight: RL uniquely enables the conversion of object detection from a passive, frame-by-frame inference into an intelligent, goal-directed sensing process—the core principle underlying the proposed HRL-Det framework.

3. Proposed Method: HRL-Det

In this section, we present HRL-Det, a hierarchical reinforcement learning framework that recasts aerial object detection as a sequential Markov Decision Process (MDP) solved through continuous-time value estimation. Unlike conventional detectors that exhaustively evaluate dense anchor grids, HRL-Det trains a deep reinforcement learning (DRL) agent to actively search for objects by iteratively refining a spatial bounding box hypothesis through a learned policy. The framework rests on two tightly coupled reinforcement learning innovations: (1) a Neural ODE-Driven Continuous-Time Bellman State Evolution module (Section 3.2) that models the agent’s state-value dynamics as a continuous-time stochastic process governed by the Hamilton–Jacobi–Bellman (HJB) partial differential equation; and (2) a Lyapunov-Guided Entropy-Regularized Reward Shaping mechanism (Section 3.3) that provides convergence-promoting reward signals informed by Lyapunov stability analysis while maintaining exploration diversity through maximum entropy policy optimization. Figure 1 illustrates the overall architecture.

3.1. Markov Decision Process Formulation

Before introducing the continuous-time state evolution and Lyapunov-guided reward shaping mechanisms, we first formulate sequential object detection as a discrete-time Markov Decision Process (MDP). This formulation provides the foundation upon which the proposed HRL-Det framework is built.

At each decision step t, the agent interacts with the environment through a tuple $(s_t,a_t,r_t,s_{t+1})$, where the state, action, reward, and transition dynamics are defined as follows.

3.1.1. State Space

The state $s_t$ encodes the current observation of the search process and consists of visual, spatial, and historical information:

$$s_t=\left[F_t^{ms},e_{pos}\left(b_t\right),e_{hist}\left(a_{<t}\right)\right],$$

where $F_t^{ms}$ denotes the multi-scale visual features extracted from the current region of interest (RoI), $e_{pos}\left(b_t\right)$ represents the positional encoding of the current bounding box $b_t$, and $e_{hist}\left(a_{<t}\right)$ is the action–history embedding summarizing previous decisions. Together, these components provide visual, spatial, and temporal context for sequential localization. The detailed construction of $F_t^{ms}$ and the resulting latent state representation is presented in Section 3.2.

3.1.2. Action Space

At each step, the agent selects an action

$$a_t\in \mathcal{A},$$

from the finite action set

$$\mathcal{A}=\{\mathit{left},\mathit{right},\mathit{up},\mathit{down},\mathit{\text{scale\_up}},\mathit{\text{scale\_down}}\mathit{},\mathit{wider},\mathit{taller},\mathit{center},\mathit{trigger}\}.$$

The first nine actions modify the current bounding box through translation or scale adjustment, while the trigger action terminates the search and outputs the current bounding box as the final detection result.

3.1.3. Reward Function

The environmental reward is designed to encourage progressive improvement of localization quality. Let

$${\Delta }_t=\mathrm{IoU}(b_t,g)-\mathrm{IoU}(b_{t-1},g),$$

where g denotes the ground-truth bounding box. Positive rewards are assigned when the Intersection over Union (IoU) improves, whereas penalties are applied when localization quality deteriorates. A larger terminal reward is granted when the trigger action is executed with satisfactory localization accuracy. The complete reward shaping mechanism is introduced in Section 3.3.

3.1.4. State Transition

Given the current state $s_t$ and action $a_t$, the environment transitions to a new state according to

$$s_{t+1}\sim P(s_{t+1}\mid s_t,a_t),$$

where the transition is induced by applying the selected spatial transformation to the current bounding box and extracting features from the resulting RoI. Since the next observation depends only on the current state and action, the process satisfies the Markov property.

3.1.5. Episode Termination

An episode terminates when one of the following conditions is satisfied:

• The agent selects the trigger action;
• The maximum search horizon $T_{\max }$ is reached;
• The bounding box becomes invalid or moves outside the image boundary.

The final bounding box is then recorded as the detection output.

3.1.6. Relationship Between Continuous-Time Modeling and Discrete Decision-Making

Although the proposed framework introduces continuous-time dynamics through Neural ODE/SDE formulations, HRL-Det remains fundamentally a discrete-time reinforcement learning system. The Neural ODE/SDE module serves as a continuous-time state encoder that enriches the latent representation of the current observation. The final decision-making policy remains a standard Dueling Double DQN operating on a discrete Markov Decision Process. The Hamilton–Jacobi–Bellman (HJB) equation is therefore used as a theoretical tool for modeling state evolution and value regularization, rather than directly replacing the underlying DQN policy optimization procedure. With this clarification, we next describe the construction of the state representation used by HRL-Det.

3.2. Neural ODE-Driven Continuous-Time Bellman State Evolution

Before introducing the continuous-time formulation, we emphasize that the Neural ODE module is used to enhance state representation within the underlying MDP, whereas action selection is still performed by a conventional Dueling Double DQN policy. A fundamental limitation of conventional DRL-based detectors [17,18] is the reliance on discrete-time, fixed-step state transitions, which constrains the temporal resolution of the search policy and introduces discretization artifacts into the value function landscape. To overcome this, we propose modeling the agent’s state evolution as a continuous-time dynamical system parameterized by a neural ordinary differential equation (Neural ODE), and derive the optimal value function via the Hamilton–Jacobi–Bellman (HJB) partial differential equation. Figure 2 illustrates the contrast between traditional discrete MDP transitions and our continuous-time Neural SDE formulation.

State Space and Feature Extraction. At time step t, given the current bounding box hypothesis $b_t=(x_t,y_t,w_t,h_t)\in {\mathbb{R}}^4$, we crop the corresponding image region of interest (RoI) and resize it to a canonical resolution of $224\times 224$. A shared backbone network (ResNet-34 [38]) extracts multi-scale feature maps ${\left\{C_l\right\}}_{l=2}^5$ at strides $\{4,8,16,32\}$, which are projected to a unified dimension $D=256$ via $1\times 1$ convolutions:

$$F_l=\sigma \left(\mathrm{BN}\left(W_l^{\mathrm{proj}}\ast C_l+{\mathbf{b}}_l\right)\right)\in {\mathbb{R}}^{H_l\times W_l\times D},l\in \{2,3,4,5\},$$

(1)

where $W_l^{\mathrm{proj}}$ are the projection parameters, $\mathrm{BN}(·)$ denotes batch normalization, and $\sigma (·)$ is the GELU activation function.

Equation (1) defines the first visual component of the MDP state. Specifically, it maps heterogeneous backbone features from different pyramid levels into a common D-dimensional embedding space, so that shallow high-resolution features and deeper semantic features can be pooled and concatenated consistently in Equation (3). This projection reduces scale-dependent feature mismatch and provides the multi-scale RoI descriptor $F_t^{ms}$ used by the Neural ODE state encoder and the subsequent Dueling Double DQN heads.

Continuous-Time State Dynamics via Neural ODE. We depart from the conventional discrete transition ${\mathbf{s}}_{t+1}=g({\mathbf{s}}_t,a_t)$ and instead model the agent’s internal state evolution as a continuous-time dynamical system. Let $\mathbf{s}\left(\tau \right)\in {\mathbb{R}}^{d_s}$ denote the continuous state at time $\tau \in [0,T]$. We parameterize the state dynamics by a neural ODE:

$${\displaystyle \frac{d\mathbf{s}\left(\tau \right)}{d\tau }}=f_{\theta }\left(\mathbf{s}\left(\tau \right),a\left(\tau \right),\tau \right),\mathbf{s}\left(0\right)={\mathbf{s}}_0,$$

(2)

where $f_{\theta }:{\mathbb{R}}^{d_s}\times \mathcal{A}\times {\mathbb{R}}_{\ge 0}\to {\mathbb{R}}^{d_s}$ is a Lipschitz-continuous neural network parameterized by $\theta$, and ${\mathbf{s}}_0$ is constructed by concatenating the multi-scale features with positional and action–history embeddings:

$${\mathbf{s}}_0=\left[{\displaystyle \underset{l=2}{\overset{5}{⨁}}}\mathrm{GAP}\left(F_l\right);{\mathbf{e}}_{\mathrm{pos}}\left(b_0\right);{\mathbf{e}}_{\mathrm{hist}}\left({\mathbf{a}}_{<t}\right)\right]\in {\mathbb{R}}^{d_s},$$

(3)

where $\mathrm{GAP}(·)$ is global average pooling, ${\mathbf{e}}_{\mathrm{pos}}\in {\mathbb{R}}^{16}$ encodes normalized box coordinates, and ${\mathbf{e}}_{\mathrm{hist}}\in {\mathbb{R}}^{32}$ is produced by a gated recurrent unit (GRU) encoding the action history. The total state dimension is $d_s=4D+48=1072$.

To account for the inherent stochasticity in the visual search process (e.g., occlusion, clutter), we extend Equation (2) to a neural stochastic differential equation (Neural SDE):

$$d\mathbf{s}\left(\tau \right)=f_{\theta }\left(\mathbf{s}\left(\tau \right),a\left(\tau \right),\tau \right)d\tau +{\Sigma }_{\varphi }\left(\mathbf{s}\left(\tau \right)\right)d{\mathbf{W}}_{\tau },$$

(4)

where ${\Sigma }_{\varphi }(·)\in {\mathbb{R}}^{d_s\times d_s}$ is a state-dependent diffusion matrix parameterized by $\varphi$, and ${\mathbf{W}}_{\tau }$ is a standard $d_s$-dimensional Wiener process. This stochastic formulation enables the agent to maintain a distribution over possible future states, improving robustness to visual ambiguity.

Hamilton–Jacobi–Bellman Equation for Optimal Value. Under the continuous-time formulation, the optimal state-value function $V^{*}(\mathbf{s},\tau )$ satisfies the Hamilton–Jacobi–Bellman (HJB) partial differential equation:

$$-{\displaystyle \frac{\partial V^{*}}{\partial \tau }}=\underset{a\in \mathcal{A}}{\max }\left\{r(\mathbf{s},a)+{\left({\nabla }_{\mathbf{s}}{V}^{*}\right)}^{\top }{f}_{\theta }(\mathbf{s},a,\tau )+{\displaystyle \frac{1}{2}}\mathrm{tr}\left({\Sigma }_{\varphi }{\Sigma }_{\varphi }^{\top }{\nabla }_{\mathbf{s}}^2{V}^{*}\right)\right\},$$

(5)

subject to the terminal condition $V^{*}(\mathbf{s},T)=R_{\mathrm{terminal}}\left(\mathbf{s}\right)$, where $r(\mathbf{s},a)$ is the instantaneous reward, ${\nabla }_{\mathbf{s}}{V}^{*}$ is the gradient vector, and ${\nabla }_{\mathbf{s}}^2{V}^{*}$ is the Hessian matrix of $V^{*}$ with respect to $\mathbf{s}$. The second-order trace term captures the effect of stochastic diffusion on value estimation, enabling the agent to account for uncertainty in its spatial reasoning.

Direct solution of the HJB equation in high-dimensional state spaces is computationally intractable. We therefore approximate $V^{*}$ using a deep neural network $V_{\psi }(\mathbf{s},\tau )$ and derive a tractable training objective by integrating along sampled Neural ODE trajectories. Specifically, for a trajectory ${\left\{\mathbf{s}\left({\tau }_i\right)\right\}}_{i=0}^N$ obtained through numerical integration (Dormand–Prince RK45 solver [39]), we minimize the HJB residual loss:

$${\mathcal{L}}_{\mathrm{HJB}}\left(\psi \right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=0}^{N-1}}{∥{\displaystyle \frac{\partial V_{\psi }}{\partial \tau }}{|}_{{\tau }_i}+\underset{a}{\max }\left\{r_i+{\nabla }_{\mathbf{s}}{V}_{\psi }^{\top }{f}_{\theta ,i}+{\displaystyle \frac{1}{2}}\mathrm{tr}\left({\Sigma }_i{\Sigma }_i^{\top }{\nabla }_{\mathbf{s}}^2{V}_{\psi ,i}\right)\right\}∥}^2.$$

(6)

Adjoint-Based Backpropagation. Training the Neural ODE dynamics $f_{\theta }$ requires computing gradients through the ODE solver. We employ the continuous adjoint method, which avoids storing all intermediate solver states and thus reduces peak memory consumption relative to naive backpropagation through the solver steps. Defining the adjoint variable $\mathit{\lambda }\left(\tau \right)=\partial \mathcal{L}/\partial \mathbf{s}\left(\tau \right)$, its dynamics are governed by the adjoint ODE:

$${\displaystyle \frac{d\mathit{\lambda }\left(\tau \right)}{d\tau }}=-\mathit{\lambda }{\left(\tau \right)}^{\top }{\displaystyle \frac{\partial f_{\theta }}{\partial \mathbf{s}}},\mathit{\lambda }\left(T\right)={\displaystyle \frac{\partial \mathcal{L}}{\partial \mathbf{s}\left(T\right)}}.$$

(7)

The parameter gradient is then recovered by integrating backward in time:

$${\displaystyle \frac{d\mathcal{L}}{d\theta }}=-{\int }_T^0\mathit{\lambda }{\left(\tau \right)}^{\top }{\displaystyle \frac{\partial f_{\theta }(\mathbf{s}\left(\tau \right),a\left(\tau \right),\tau )}{\partial \theta }}d\tau .$$

(8)

Stationary State Distribution via Fokker–Planck Equation. To analyze the long-run behavior of the agent’s exploration, we derive the time evolution of the state probability density $\rho (\mathbf{s},\tau )$ under the Neural SDE (Equation (4)). The density satisfies the Fokker–Planck (Kolmogorov forward) equation:

$${\displaystyle \frac{\partial \rho }{\partial \tau }}=-{\nabla }_{\mathbf{s}}·\left(\rho f_{\theta }\right)+{\displaystyle \frac{1}{2}}{\nabla }_{\mathbf{s}}^2:\left({\Sigma }_{\varphi }{\Sigma }_{\varphi }^{\top }\rho \right),$$

(9)

where ${\nabla }_{\mathbf{s}}^2:(·)$ denotes the double contraction of the Hessian operator. In the stationary regime ($\partial \rho /\partial \tau =0$), the equilibrium distribution ${\rho }^{*}\left(\mathbf{s}\right)$ characterizes the asymptotic state occupancy induced by the Neural SDE dynamics. In this work, the Fokker–Planck analysis is used to interpret the qualitative behavior of the learned continuous-time dynamics and the resulting attention allocation; it is not used as a separate exploration algorithm or reported as an independent experimental module.

Dueling Value Function Decomposition. The continuous-time value function is decomposed into state-value and action-advantage streams following the Dueling architecture:

$$Q_{\psi }(\mathbf{s},a;\tau )=V_{\psi }(\mathbf{s};\tau )+\left(A_{\psi }(\mathbf{s},a;\tau )-{\displaystyle \frac{1}{|{\mathcal{A}}_t|}}\sum _{a^{\prime }\in {\mathcal{A}}_t}{A}_{\psi }(\mathbf{s},a^{\prime };\tau )\right),$$

(10)

where $V_{\psi }$ and $A_{\psi }$ are produced by separate network heads operating on the Neural ODE state $\mathbf{s}\left(\tau \right)$. The Double DQN formulation [22] is adopted to decouple action selection from value estimation, mitigating overestimation bias:

$$y_t=r_t+\gamma ·Q_{\psi }\left({\mathbf{s}}_{t+1},\arg \underset{a^{\prime }}{\max }{Q}_{\psi }({\mathbf{s}}_{t+1},a^{\prime };\theta ),{\tau }_{t+1};{\psi }^{-}\right),$$

(11)

where ${\psi }^{-}$ are target network parameters updated via Polyak averaging with a coefficient ${\tau }_{\mathrm{soft}}=0.005$.

3.3. Lyapunov-Guided Entropy-Regularized Reward Shaping

The second core innovation of HRL-Det addresses two interrelated challenges in RL-based object detection: (i) reward sparsity, which causes slow and unstable training convergence, and (ii) premature policy collapse, where the agent converges to a suboptimal deterministic search strategy. We propose a principled reward shaping framework informed by Lyapunov stability analysis, integrated with maximum entropy reinforcement learning for a robust exploration–exploitation trade-off. Figure 3 provides a schematic overview of this dual-component reward shaping mechanism.

Action Space and MDP Formulation. At each discrete decision point, the agent selects from ten spatial transformation actions:

$$\mathcal{A}=\{\mathtt{left},\mathtt{right},\mathtt{up},\mathtt{down},\mathtt{scale}\_\mathtt{up},\mathtt{scale}\_\mathtt{down},\mathtt{wider},\mathtt{taller},\mathtt{center},\mathtt{trigger}\},$$

(12)

where directional actions translate $b_t$ by a fraction $\alpha =0.15$ of its dimension, scaling actions adjust size by $(1\pm \beta )$ with $\beta =0.10$, and trigger terminates the episode and commits $b_t$ as the detection output.

Lyapunov Function Construction. Let ${\mathbf{s}}^{*}\in \mathcal{S}$ denote the terminal goal state corresponding to perfect alignment with the ground-truth box g (i.e., $\mathrm{IoU}(b,g)=1$). We construct a Lyapunov function $\mathcal{V}:\mathcal{S}\to {\mathbb{R}}_{\ge 0}$ as the IoU-weighted quadratic distance to the goal:

$$\mathcal{V}\left({\mathbf{s}}_t\right)={\left(1-\mathrm{IoU}(b_t,g)\right)}^2+\mu {∥\mathbf{c}\left(b_t\right)-\mathbf{c}\left(g\right)∥}_2^2+\nu {\left(\log {\displaystyle \frac{w_t{h}_t}{w_g{h}_g}}\right)}^2,$$

(13)

where $\mathbf{c}(·)$ extracts the normalized box center, $w,h$ are the box dimensions, and $\mu =0.3$, $\nu =0.2$ balance the geometric terms. By construction, $\mathcal{V}\left({\mathbf{s}}^{*}\right)=0$ and $\mathcal{V}\left(\mathbf{s}\right)>0$ for all $\mathbf{s}\ne {\mathbf{s}}^{*}$, satisfying the positive-definiteness requirement.

Lyapunov Stability Constraint. We use the Lyapunov decrease condition as a stability-inspired soft constraint that encourages monotonic progress of the agent’s trajectory toward the goal state. Specifically, under policy $\pi$, training penalizes violations of the desired expected Lyapunov decrease condition:

$${\mathbb{E}}_{a_t\sim \pi (·|{\mathbf{s}}_t)}\left[\mathcal{V}\left({\mathbf{s}}_{t+1}\right)-\mathcal{V}\left({\mathbf{s}}_t\right)\right]\le -c·\mathcal{V}\left({\mathbf{s}}_t\right)+d,$$

(14)

where $c\in (0,1)$ is the contraction rate and $d\ge 0$ is a small slack constant accommodating stochastic transitions. This constraint is enforced as a penalty term in the policy loss (detailed below). The discrete-time Lyapunov condition in Equation (14) can be extended to the continuous-time Neural ODE setting via the Lie derivative:

$$\dot{\mathcal{V}}\triangleq {\nabla }_{\mathbf{s}}{\mathcal{V}}^{\top }{f}_{\theta }(\mathbf{s},a,\tau )\le -c·\mathcal{V}\left(\mathbf{s}\right),$$

(15)

which provides a continuous certificate of convergence along Neural ODE trajectories.

Potential-Based Reward Shaping. Following the reward shaping theorem of Ng et al. [40], we use the Lyapunov function as the shaping potential to construct a dense, policy-invariant reward signal. The shaped reward at each step is as follows:

$$r_t^{\mathrm{shaped}}=\underset{\mathrm{environment}}{\underbrace{r_t^{\mathrm{env}}}}+\underset{\mathrm{potential-based} \mathrm{shaping}}{\underbrace{\gamma \mathrm{\Phi }\left({\mathbf{s}}_{t+1}\right)-\mathrm{\Phi }\left({\mathbf{s}}_t\right)}},$$

(16)

where $\mathrm{\Phi }\left(\mathbf{s}\right)=-\mathcal{V}\left(\mathbf{s}\right)$ is the negative Lyapunov potential and the environment reward $r_t^{\mathrm{env}}$ is defined as follows:

$$r_t^{\mathrm{env}}=\left\{\begin{array}{cc}\mathrm{sign}\left({\Delta }_t\right)·{\left|{\Delta }_t\right|}^{\eta }+\lambda ·{\Delta }_t,\hfill & a_t\ne \mathtt{trigger},\hfill \\ +\omega ·\mathrm{IoU}(b_t,g),\hfill & a_t=\mathtt{trigger}\wedge \mathrm{IoU}\ge {\tau }_{\mathrm{IoU}},\hfill \\ -\omega ,\hfill & a_t=\mathtt{trigger}\wedge \mathrm{IoU}<{\tau }_{\mathrm{IoU}},\hfill \end{array}\right.$$

(17)

with ${\Delta }_t=\mathrm{IoU}(b_t,g)-\mathrm{IoU}(b_{t-1},g)$ denoting the IoU improvement, $\eta =0.5$ controlling the nonlinear shaping curvature, $\lambda =0.1$ providing a linear bonus, $\omega =3.0$ scaling the terminal reward, and ${\tau }_{\mathrm{IoU}}=0.5$ defining the success threshold. The potential-based formulation guarantees that the optimal policy remains invariant under shaping (Ng et al. [40]), while the Lyapunov potential provides informative gradient signals that accelerate convergence.

Maximum Entropy Policy Optimization. To prevent premature policy collapse and maintain exploration diversity, we adopt the maximum entropy reinforcement learning framework. The soft state-value function and soft Q-function satisfy the following soft Bellman equations:

$$\begin{array}{cc}\hfill V^{\mathrm{soft}}\left({\mathbf{s}}_t\right)& ={\mathbb{E}}_{a_t\sim \pi }\left[Q^{\mathrm{soft}}({\mathbf{s}}_t,a_t)-\alpha \log \pi \left(a_t\right|{\mathbf{s}}_t)\right],\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill Q^{\mathrm{soft}}({\mathbf{s}}_t,a_t)& =r_t^{\mathrm{shaped}}+\gamma {\mathbb{E}}_{{\mathbf{s}}_{t+1}}\left[V^{\mathrm{soft}}\left({\mathbf{s}}_{t+1}\right)\right],\hfill \end{array}$$

(19)

where $\alpha >0$ is the temperature parameter controlling the entropy-reward trade-off. The maximum entropy objective is:

$$J\left(\pi \right)={\displaystyle \sum _{t=0}^{T_{\max }}}{\mathbb{E}}_{({\mathbf{s}}_t,a_t)\sim {\rho }_{\pi }}\left[r_t^{\mathrm{shaped}}+\alpha \mathcal{H}\left(\pi (·|{\mathbf{s}}_t)\right)\right],$$

(20)

where $\mathcal{H}\left(\pi \right)=-{\sum }_a\pi \left(a\right|\mathbf{s})\log \pi \left(a\right|\mathbf{s})$ is the Shannon entropy and ${\rho }_{\pi }$ denotes the state-action visitation distribution under $\pi$. We automatically tune $\alpha$ by solving the constrained optimization:

$${\alpha }^{*}=\arg \underset{\alpha >0}{\min }{\mathbb{E}}_{a_t\sim {\pi }^{*}}\left[-\alpha \log {\pi }^{*}\left(a_t\right|{\mathbf{s}}_t)-\alpha \overline{\mathcal{H}}\right],$$

(21)

where $\overline{\mathcal{H}}=-\log (1/|\mathcal{A}\left|\right)$ is the target entropy.

KL-Constrained Policy Update. To ensure stable policy improvement, each update is constrained by a KL-divergence trust region:

$${\pi }_{k+1}=\arg \underset{\pi }{\min }{\mathbb{E}}_{\mathbf{s}\sim \mathcal{D}}\left[D_{\mathrm{KL}}\left(\pi (·|\mathbf{s})\parallel {\pi }_k(·|\mathbf{s})\right)-\eta {\mathbb{E}}_{a\sim \pi }\left[Q_{\psi }^{\mathrm{soft}}(\mathbf{s},a)\right]\right],$$

(22)

where $\eta >0$ is the step size and $\mathcal{D}$ is the replay buffer. This KL constraint prevents catastrophic policy oscillations commonly observed in discrete-action DQN training.

Stage-Wise Action Masking. To prevent redundant spatial exploration, we partition the search episode into $K=3$ stages. At each stage boundary, we record a spatial occupancy map ${\mathcal{O}}_k$, encapsulating previously explored regions. Actions leading to high overlap with prior regions are masked:

$${\mathcal{A}}_t=\left\{a\in \mathcal{A}|\forall k<k_{\mathrm{curr}}:\mathrm{IoU}\left(\mathcal{T}(b_t,a),{\mathcal{O}}_k\right)<{\tau }_{\mathrm{mask}}\right\},$$

(23)

where $\mathcal{T}(b_t,a)$ is the box obtained by applying action a, $k_{\mathrm{curr}}$ is the current stage index, and ${\tau }_{\mathrm{mask}}=0.7$.

Multi-Step Temporal Difference Learning with Eligibility Traces. To accelerate credit assignment across the sequential search trajectory, we employ TD($\lambda$) learning with eligibility traces. The eligibility trace vector ${\mathbf{e}}_t\in {\mathbb{R}}^{\left|\psi \right|}$ is updated recursively:

$${\mathbf{e}}_t=\gamma {\lambda }_{\mathrm{trace}}{\mathbf{e}}_{t-1}+{\nabla }_{\psi }{Q}_{\psi }^{\mathrm{soft}}({\mathbf{s}}_t,a_t),{\mathbf{e}}_0=\mathbf{0},$$

(24)

where ${\lambda }_{\mathrm{trace}}=0.9$ is the trace decay parameter. The Q-network parameters are updated via:

$$\psi \leftarrow \psi +{\alpha }_{\mathrm{lr}}·{\delta }_t·{\mathbf{e}}_t,{\delta }_t=r_t^{\mathrm{shaped}}+\gamma V^{\mathrm{soft}}\left({\mathbf{s}}_{t+1}\right)-Q_{\psi }^{\mathrm{soft}}({\mathbf{s}}_t,a_t),$$

(25)

where ${\delta }_t$ is the temporal difference error and ${\alpha }_{\mathrm{lr}}$ is the learning rate.

Prioritized Experience Replay. Transitions are stored in a replay buffer $\mathcal{D}$ and sampled with probability proportional to their absolute TD error:

$$P\left(i\right)={\displaystyle \frac{{\left(|{\delta }_i|+{ϵ}_{\mathrm{per}}\right)}^{{\alpha }_{\mathrm{per}}}}{{\sum }_j{\left(|{\delta }_j|+{ϵ}_{\mathrm{per}}\right)}^{{\alpha }_{\mathrm{per}}}}},$$

(26)

with ${\alpha }_{\mathrm{per}}=0.6$ and ${ϵ}_{\mathrm{per}}={10}^{-6}$. Importance-sampling weights $w_i={(N·P\left(i\right))}^{-{\beta }_{\mathrm{per}}}$ correct the sampling bias, with ${\beta }_{\mathrm{per}}$ annealed from 0.4 to 1.0 during training.

Overall Training Objective. The complete loss function combines the HJB residual, the soft Bellman TD error, and the Lyapunov stability penalty:

$${\mathcal{L}}_{\mathrm{total}}=\underset{\mathrm{HJB} \mathrm{residual}}{\underbrace{{\mathcal{L}}_{\mathrm{HJB}}\left(\psi \right)}}+\underset{\mathrm{soft} \mathrm{TD} \mathrm{loss}}{\underbrace{{\displaystyle \frac{1}{B}}\sum _{i=1}^B{w}_i·{\mathcal{L}}_{\mathrm{Huber}}\left(y_i,Q_{\psi }^{\mathrm{soft}}({\mathbf{s}}_i,a_i)\right)}}+\underset{\mathrm{Lyapunov} \mathrm{penalty}}{\underbrace{\xi ·\mathbb{E}\left[\mathrm{ReLU}\left(\dot{\mathcal{V}}+c·\mathcal{V}\right)\right]}},$$

(27)

where B is the mini-batch size, $\xi =0.5$ is the Lyapunov penalty weight, and $\mathrm{ReLU}(·)$ enforces the stability constraint only when violated.

Curriculum $\epsilon$-Greedy Schedule. The exploration rate $\epsilon$ and initial box perturbation magnitude ${\sigma }_{\mathrm{init}}$ are jointly annealed via a cosine curriculum:

$$\begin{array}{cc}\hfill \epsilon \left(e\right)& ={\epsilon }_{\min }+{\displaystyle \frac{{\epsilon }_{\max }-{\epsilon }_{\min }}{2}}\left(1+\cos \left({\displaystyle \frac{\pi ·e}{E}}\right)\right),\hfill \\ \hfill {\sigma }_{\mathrm{init}}\left(e\right)& ={\sigma }_{\min }+{\displaystyle \frac{{\sigma }_{\max }-{\sigma }_{\min }}{2}}\left(1+\cos \left({\displaystyle \frac{\pi ·e}{E}}\right)\right),\hfill \end{array}$$

(28)

where e is the current epoch and E is the total epoch count. This schedule encourages broad exploration during early training while enabling fine-grained exploitation in later stages.

Convergence Analysis. Under the Lyapunov decrease condition (Equation (14)) and the maximum entropy regularization (Equation (20)), assuming exact policy evaluation and sufficient representation capacity, the policy improvement satisfies the monotonic bound:

$$J\left({\pi }_{k+1}\right)\ge J\left({\pi }_k\right)-{\displaystyle \frac{2\gamma {ϵ}_{\pi }}{{(1-\gamma )}^2}}\underset{\mathbf{s}}{\max }{D}_{\mathrm{KL}}({\pi }_{k+1}\parallel {\pi }_k)+{\displaystyle \frac{\alpha }{1-\gamma }}\left(\mathcal{H}\left({\pi }_{k+1}\right)-\mathcal{H}\left({\pi }_k\right)\right),$$

(29)

where ${ϵ}_{\pi }$ is the policy improvement error bound. This bound is adapted from maximum-entropy policy improvement theory [41], together with the policy-invariance principle of potential-based reward shaping [40]. The KL term controls the size of the policy update, the entropy term encourages diverse action distributions, and the Lyapunov penalty adds a task-specific stability preference for sequential localization. In practice, function approximation and finite-sample effects mean that the inequality serves as a design principle rather than a strict guarantee; nevertheless, the Lyapunov penalty in ${\mathcal{L}}_{\mathrm{total}}$ (Equation (27)) empirically promotes monotonic improvement and prevents the degenerate behaviors commonly observed in sparse-reward RL-based detection methods (see Section 4.5).

3.4. From Continuous-Time Theory to Discrete-Time Agent

The preceding two subsections develop a continuous-time theoretical framework—Neural ODE/SDE dynamics, the HJB equation, and Lyapunov stability analysis—that provides the design rationale for the practical agent architecture. Because the actual agent operates in discrete decision steps with a finite action set, we now clarify the mapping between theory and implementation.

State evolution as a learned encoder. At each discrete decision step t, the agent observes the current RoI, constructs the initial state ${\mathbf{s}}_0$ (Equation (3)), and integrates the Neural ODE (Equation (2)) forward over a fixed horizon $[0,T]$ using the RK45 solver. The resulting evolved state $\mathbf{s}\left(T\right)$ is then fed into the Dueling Double DQN heads (Equation (10)) for action–value estimation. Thus, the continuous-time integration acts as a learned state encoder that enriches the feature representation before discrete action selection; it does not replace the underlying discrete MDP structure.

Discrete action selection. Despite the continuous-time state dynamics, the behavioral policy remains a standard DQN $\epsilon$-greedy policy that selects from the ten spatial actions in $\mathcal{A}$ (Equation (12)). The HJB residual loss (Equation (6)) regularizes the value landscape along the Neural ODE trajectory during training, but at inference time action selection reduces to a single $\arg \max$ over Q-values, identical to a conventional DQN agent.

Reward and stability penalty. The Lyapunov-based shaped reward (Equation (16)) and the environment reward (Equation (17)) are computed at each discrete transition $({\mathbf{s}}_t,a_t,{\mathbf{s}}_{t+1})$. The Lyapunov decrease condition (Equation (14)) is enforced as a soft penalty in the loss function (Equation (27)), not as a hard constraint with a formal proof of global convergence. Empirically, this penalty encourages—but does not guarantee—monotonic Lyapunov decrease across episodes.

Training pipeline summary. Training follows a standard DQN loop: the agent collects transitions into a prioritized replay buffer, samples mini-batches, computes the combined loss ${\mathcal{L}}_{\mathrm{total}}$ (Equation (27)), and updates parameters via Adam. The Neural ODE parameters $\theta$, diffusion parameters $\varphi$, and value network parameters $\psi$ are optimized end-to-end through the adjoint method. In summary, the overall system is a DQN agent whose state representation is augmented by a Neural ODE encoder and whose reward signal is shaped by a Lyapunov-inspired potential function.

4. Experimental Results and Analysis

In this section, we present a comprehensive evaluation of the proposed HRL-Det framework for object detection. We first introduce the experimental setup and datasets, then present quantitative comparisons against state-of-the-art baselines. Following this, we provide a detailed qualitative analysis that justifies the efficacy of our model through training convergence, sequential search behavior visualization, and boundary localization accuracy across three challenging datasets: VisDrone2019 [5], DroneVehicle [42], and MS COCO [43].

4.1. Datasets and Evaluation Metrics

We evaluate our method on three publicly available benchmarks that span a diverse range of aerial imaging conditions and object scales.

VisDrone2019 [5] is a large-scale aerial object detection dataset captured by unmanned aerial vehicles (UAVs) under various environmental conditions. It comprises 6471 training images, 548 validation images, and 1610 test images with 10 object categories, including pedestrians, cyclists, and vehicles. The dataset is particularly challenging due to dense object packing and severe scale variations (objects as small as 5 × 5 pixels).

DroneVehicle [42] is an RGB-infrared multimodal dataset for aerial vehicle detection, containing 17,990 paired image frames captured from UAV platforms. It provides annotations for five vehicle categories and is designed to evaluate fusion-based detection in low-visibility and nighttime conditions. We follow the standard single-modal RGB evaluation protocol.

MS COCO 2017 [43] is the widely adopted general object detection benchmark, comprising 118,287 training images and 5000 validation images across 80 categories. We use it to demonstrate the out-of-domain generalization capability of HRL-Det. We report results on the val2017 split.

Following standard COCO evaluation [43], we report the mean Average Precision (mAP) at a single IoU threshold of 0.5 (mAP@0.5) and the average mAP across IoU thresholds from 0.5 to 0.95 (mAP@0.5:0.95) as primary metrics.

4.2. Implementation Details

All experiments are conducted using PyTorch 2.1 on a workstation equipped with four NVIDIA RTX 4090 GPUs (24 GB each; NVIDIA Corporation, Santa Clara, CA, USA). The ResNet-34 backbone is initialized with ImageNet-pretrained weights, and all other network components are randomly initialized using Kaiming initialization. The state dimension is $d_s=1072$, as described in Section 3.2. The Neural ODE is integrated using the Dormand–Prince RK45 solver with adaptive step size (absolute tolerance ${10}^{-5}$, relative tolerance ${10}^{-5}$).

The DQN agent is trained using the Adam optimizer with an initial learning rate of $1\times {10}^{-4}$ and weight decay $1\times {10}^{-5}$. The learning rate follows a cosine annealing schedule decaying to $1\times {10}^{-6}$. The replay buffer stores up to ${10}^5$ transitions, and training begins after collecting $5\times {10}^3$ warm-up transitions. The mini-batch size is $B=64$. The discount factor is $\gamma =0.99$, the soft target update coefficient is ${\tau }_{\mathrm{soft}}=0.005$, and the maximum number of search steps per episode is $T_{\max }=20$. The exploration rate $\epsilon$ is annealed from 1.0 to 0.05 via the cosine curriculum (Equation (28)) over the full training schedule.

For VisDrone2019, we train for 50,000 episodes (∼36 h). For DroneVehicle, training spans 60,000 episodes (∼48 h). For MS COCO, we train for 80,000 episodes (∼72 h). All training times are measured using four GPUs with distributed data collection. Inference is performed on a single RTX 4090 GPU. Input RoI crops are resized to 224 × 224. During evaluation, $\epsilon$ is set to 0 (greedy policy). For computational comparison, baseline detector FLOPs are reported at the standard 640 × 640 full-image input, whereas HRL-Det processes normalized 224 × 224 RoI crops at each search step. The FLOPs reported for HRL-Det therefore correspond to the average sequential-search cost per detected object, computed over the observed mean of 6.3 search steps, rather than a single full-image dense forward pass.

4.3. Quantitative Comparison with State-of-the-Arts

To validate the effectiveness of our proposed framework, we conduct experiments along two complementary axes. (i) RL-based detector comparison (

Table 1. Comparison with deep reinforcement learning-based object detection methods across VisDrone2019, DroneVehicle, and MS COCO 2017. “†” denotes an RL-based method. FLOPs for dense baselines are computed at 640 × 640 full-image input; HRL-Det FLOPs denote the average per-object sequential-search cost using 224 × 224 RoI crops over 6.3 search steps. “Steps” = avg. search steps per object. FPS measured on a single NVIDIA RTX 4090 GPU with batch size 1. Bold: best; underline: second best.

Method (Year)	VisDrone2019		DroneVehicle		MS COCO 2017		FLOPs (G)	Steps ↓	FPS ↑
	mAP@0.5	mAP@0.5:0.95	mAP@0.5	mAP@0.5:0.95	mAP@0.5	mAP@0.5:0.95
Caicedo & Lazebnik (2015) † [17]	0.271	0.131	0.634	0.397	0.521	0.318	3.82	11.4	47
Mathe et al. (2016) † [31]	0.289	0.144	0.648	0.409	0.543	0.331	4.17	10.8	51
Bellver et al. (2016) † [18]	0.314	0.168	0.671	0.428	0.578	0.362	3.54	9.2	59
Jie et al. (2016) † [32]	0.302	0.157	0.658	0.416	0.559	0.347	5.06	10.1	54
Kong et al. (2017) † [24]	0.311	0.163	0.665	0.422	0.568	0.355	4.73	9.6	52
Pirinen & Sminchisescu (2018) † [33]	0.328	0.178	0.687	0.441	0.601	0.381	3.91	8.7	62
Liu et al. (2020) † [21]	0.349	0.198	0.718	0.473	0.631	0.412	3.28	7.9	64
Uzkent & Yoon (2020) † [20]	0.341	0.189	0.703	0.459	0.617	0.398	2.93	8.1	68
Ding et al. (2023) † [34]	0.363	0.215	0.744	0.511	0.671	0.441	2.87	7.2	71
Zhang et al. (2024) † [35]	0.369	0.219	0.752	0.527	0.685	0.452	3.14	7.5	59
LHAR-RLD (2025) † [23]	0.374	0.224	0.760	0.543	0.699	0.463	1.43	8.7	79
Ours—HRL-Det (2026) †	0.412	0.251	0.812	0.578	0.735	0.512	2.61	6.3	86

): We benchmark against the full lineage of DRL-based object detection methods spanning a decade of research. Early agents include Caicedo & Lazebnik [17], Mathe et al. [31], Bellver et al. [18], and Jie et al. [32]. Mid-period refinements include Kong et al. [24] (collaborative multi-agent RL), Pirinen & Sminchisescu [33], and Uzkent & Yoon [20]. Recent advances span Liu et al. [21] (attention-guided cascade RL), Ding et al. [34] (decision-transformer active detection), Zhang et al. [35] (evolutionary RL scale optimization), and the concurrent LHAR-RLD [23]. (ii) Non-RL detector comparison (

Table 2. Comparison with mainstream non-RL object detection methods. “†” denotes our RL-based method. FPS measured on a single NVIDIA RTX 4090 GPU with batch size 1; ‡ indicates per-episode throughput for the RL sequential search paradigm. Note that RL sequential detectors and dense-prediction detectors differ in inference mechanism; mAP values are comparable under the standard COCO protocol, but throughput and deployment characteristics are not directly equivalent (see Section 5). Bold: best result.

Method (Year)	Params (M)	VisDrone2019		DroneVehicle		MS COCO 2017		FPS ↑
		mAP@0.5	mAP@0.5:0.95	mAP@0.5	mAP@0.5:0.95	mAP@0.5	mAP@0.5:0.95
Faster R-CNN (2017) [7]	41.8	0.256	0.118	0.621	0.384	0.565	0.362	31
SSD (2016) [11]	26.3	0.198	0.089	0.549	0.312	0.412	0.232	82
RetinaNet (2017) [25]	37.7	0.274	0.131	0.648	0.401	0.593	0.387	37
Deformable DETR (2021) [28]	40.1	0.331	0.187	0.712	0.481	0.643	0.449	29
YOLOv5 (2022) [9]	46.5	0.352	0.201	0.698	0.447	0.624	0.421	108
YOLOv8 (2023) [10]	43.7	0.378	0.228	0.741	0.521	0.668	0.478	131
RT-DETR (2023) [29]	42.0	0.389	0.236	0.756	0.532	0.693	0.501	114
YOLOv9 (2024) [27]	57.3	0.382	0.231	0.748	0.527	0.674	0.483	87
DINO (2022) [30]	47.0	0.403	0.244	0.779	0.561	0.720	0.511	22
YOLOv10 (2024) [26]	38.4	0.391	0.238	0.762	0.538	0.681	0.491	148
Co-DETR (2023) [44]	146.0	0.407	0.248	0.786	0.569	0.724	0.519	14
Ours—HRL-Det (2026) †	17.3	0.412	0.251	0.812	0.578	0.735	0.512	86 ‡

): We additionally compare against a broad range of canonical and modern detectors spanning two-stage (Faster R-CNN [7]), single-stage (SSD [11], RetinaNet [25]), YOLO-family (v5 [9], v8 [10], v9 [27], v10 [26]), and transformer-based detectors (Deformable DETR [28], RT-DETR [29], DINO [30], Co-DETR [44]). Baseline dense detectors are evaluated at 640 × 640 full-image resolution, while HRL-Det uses 224 × 224 RoI crops within its sequential search procedure; the comparison is therefore metric-fair for detection accuracy but not intended to imply identical inference mechanics.

Important caveat on cross-paradigm comparison. RL-based sequential detectors and conventional dense-prediction detectors differ fundamentally in inference paradigm. Non-RL methods process the entire image in a single forward pass and output all detections simultaneously, whereas HRL-Det sequentially searches for one object at a time. The mAP figures in Table 2 are computed using the identical COCO evaluation protocol for all methods, ensuring a metric-fair comparison. However, this does not constitute a task-equivalent comparison: the RL agent’s per-object sequential search has different computational scaling characteristics than single-pass dense inference (see Section 5). We present Table 2 primarily to situate HRL-Det’s detection accuracy within the broader landscape, not to claim that the two paradigms are interchangeable in deployment.

As shown in Table 1, HRL-Det achieves the strongest performance among the RL-based detectors: mAP@0.5 values of 0.412 on VisDrone2019, 0.812 on DroneVehicle, and 0.735 on MS COCO. These results improve over LHAR-RLD by 3.8, 5.2, and 3.6 absolute percentage points, respectively, and over the Caicedo & Lazebnik baseline by 14.1, 17.8, and 21.4 absolute percentage points. The average search length is also the lowest among RL methods, at 6.3 steps per object. Table 2 further shows that HRL-Det attains competitive mAP relative to representative non-RL detectors under the standard COCO metric, including YOLOv10 and Co-DETR on VisDrone; as noted above, this comparison reflects detection accuracy under a shared metric rather than direct operational equivalence.

4.4. Ablation Study

To further validate the design choices within HRL-Det, we conduct a systematic ablation study on the VisDrone2019 validation set. We progressively remove or replace key components and evaluate the resulting performance degradation, as reported in

Table 3. Ablation study on the VisDrone2019 validation set. “✓” indicates the component is included. NODE-BSE: Neural ODE Bellman State Evolution; Lyap. Reward: Lyapunov-guided dense reward shaping; Ent.-Masking: the coupled entropy-regularized stage-wise action masking module.

Configuration	NODE-BSE	Lyap. Reward	Ent.-Masking	mAP@0.5	mAP@0.5:0.95
w/o NODE-BSE	–	✓	✓	0.364	0.219
w/o Lyap. Reward	✓	–	✓	0.381	0.227
w/o Ent.-Masking	✓	✓	–	0.390	0.235
Full HRL-Det (Ours)	✓	✓	✓	0.412	0.251

.

The full HRL-Det model reaches 0.412 mAP@0.5 and 0.251 mAP@0.5:0.95 on the VisDrone2019 validation set (Table 3). Removing the Neural ODE-driven Bellman state evolution (NODE-BSE, Section 3.2) and reverting to a conventional discrete-time flat feature map reduces performance to 0.364 mAP@0.5 and 0.219 mAP@0.5:0.95, corresponding to absolute drops of 4.8 and 3.2 percentage points, respectively. Replacing the Lyapunov-guided dense reward (Section 3.3) with a sparse terminal reward reduces performance to 0.381 and 0.227, corresponding to absolute drops of 3.1 and 2.4 percentage points. Finally, disabling the joint entropy-regularized stage-wise action masking design reduces performance to 0.390 and 0.235, corresponding to absolute drops of 2.2 and 1.6 percentage points. We keep entropy regularization and stage-wise action masking as one ablation factor because they are coupled in implementation: entropy controls action–distribution diversity, while the stage-wise mask constrains redundant spatial revisits, and the reported variant removes this combined exploration-control module.

4.5. Training Convergence Assessment

To demonstrate the stability of our reinforcement learning formulation, we analyze the training convergence of the deep Q-network (DQN). Figure 4 illustrates both the training loss descent and the episodic reward curves achieved by the sequentially acting agent. As shown in the top graph, the DQN loss exhibits a rapid decline during initial exploration, subsequently stabilizing into a steady convergence regime.

Correspondingly, the bottom graph showcases the episodic reward trajectory. Initially, the agent experiences substantial variance due to the randomly initialized state–action evaluations. As the episodes progress, the smoothed reward curves demonstrate a robust monotonic increase across all three datasets. This confirms that the targeted reward shaping effectively encourages the agent to maximize the Intersection over Union (IoU) with the ground-truth objects, achieving stable convergence even in domains populated with minuscule drone-captured targets.

The x-axis of Figure 4 reports the first 5000 logged training checkpoints rather than the total number of training episodes. The complete training runs contain 50,000, 60,000, and 80,000 episodes for VisDrone2019, DroneVehicle, and MS COCO, respectively; the figure focuses on the early-to-mid convergence window because this interval most clearly shows the transition from high-variance exploration to stable reward improvement.

4.6. Hierarchical Search Process and Spatial Attention

Unlike traditional dense prediction models that evaluate millions of predefined anchors, our proposed approach sequentially scales down the search canvas. To elucidate this decision-making process, Figure 5 visualizes the multi-step action sequence taken by the agent on random evaluation samples. At each time step, given the visual features extracted from the current bounding box region of interest (RoI), the agent actively selects spatial shifts and scaling operations (e.g., ‘Center’ or target quadrants), progressively elevating the IoU score until the terminal ‘Trigger’ action is invoked.

To illustrate both spatial convergence behavior and attention distribution, Figure 6 presents cumulative trajectory overlays and attention heatmaps across all datasets. The left panels show how the initial search boxes (blue) progressively contract toward the target region (red), with yellow curves tracing the movement of box centers. The right panels visualize attention accumulation across successive iterations. The aggregated localized energy maps exhibit a strong correspondence with ground-truth object boundaries, indicating that the agent can suppress background distractions and consistently focus on semantically relevant target regions. We also removed the duplicated per-sample IoU annotations that appeared in the earlier version of this figure; quantitative IoU behavior is reported separately in Figure 7.

4.7. Qualitative Analysis and Localization Accuracy

Finally, we evaluate localization precision using the IoU distribution in Figure 7. The distribution summarizes IoU scores from standard testing runs and shows that most predicted boxes exceed the success threshold (IoU ≥ 0.5). This result indicates that the sequential action policy can refine bounding boxes effectively, although failure cases remain under severe occlusion and truncation.

Figure 8, Figure 9 and Figure 10 compare successful localizations with representative failure cases across VisDrone2019, DroneVehicle, and MS COCO. To improve readability against complex backgrounds, the former compact multi-dataset qualitative figure has been separated into three dataset-specific full-width figures with high-contrast labels. The success examples show predicted boxes that closely match ground truth for small or partially cluttered targets, whereas the failure examples mainly arise from severe occlusion, truncation, or ambiguous target boundaries. We also rechecked the IoU annotations: duplicated per-sample labels in Figure 6 were removed, and the displayed IoU scores in Figure 8, Figure 9 and Figure 10 now correspond to their own samples. Importantly, these traceable failure trajectories provide useful insight into the agent’s decision-making process. By showing how localization errors accumulate over successive search steps, they provide interpretability that is difficult to obtain from conventional single-pass detectors.

5. Discussion

The experimental results presented in Section 4 substantiate the effectiveness of HRL-Det for sequential object detection in aerial imagery. HRL-Det consistently improves over existing RL-based baselines and achieves competitive or superior mAP relative to representative dense detectors under the shared COCO evaluation protocol, with particularly pronounced gains on the UAV-centric benchmarks VisDrone2019 and DroneVehicle. Here, we discuss the key factors underpinning these results and their broader implications while noting the cross-paradigm comparison caveat below.

Efficiency vs. accuracy trade-off. One of the central advantages of HRL-Det is its substantially lower parameter count (17.3 M, cf. Table 2) compared to all competing architectures, including the RL baseline of Caicedo and Lazebnik [17]. This efficiency stems from the Neural ODE state encoder’s shared feature extractor and the Lyapunov-guided policy-driven early stopping mechanism, which terminates the spatial search as soon as the agent’s Lyapunov function value falls below a learned threshold. The result is a compact model that avoids the computational overhead of dense anchor enumeration while achieving superior localization precision.

Evaluation protocol considerations. It is important to note that the comparison between RL-based sequential detectors and conventional dense-prediction detectors involves an inherent difference in inference paradigm. Non-RL methods process the entire image in a single forward pass and detect all objects simultaneously, whereas HRL-Det performs a sequential search for each candidate object. For per-image throughput in dense scenes containing many objects, conventional detectors such as YOLOv10 remain faster in wall-clock time. The mAP comparisons in Table 1 and Table 2 follow the standard COCO evaluation protocol applied identically to all methods, ensuring a metric-fair accuracy comparison; however, they do not imply that the two paradigms are interchangeable in deployment. In particular, the total inference cost of HRL-Det scales linearly with the number of candidate objects in the image, whereas dense detectors amortize their cost over all objects in a single pass. We therefore recommend interpreting Table 2 as a positioning of HRL-Det’s detection quality relative to the broader field, rather than as a head-to-head operational benchmark. The practical efficiency advantage of HRL-Det is most pronounced in sparse-target scenarios (e.g., power line inspection, maritime surveillance) where only a few objects of interest need to be localized per frame, and the sequential search avoids processing irrelevant background regions entirely.

Neural ODE state evolution. The ablation study in Table 3 clearly shows that the Neural ODE-driven Bellman state evolution (NODE-BSE) is the single most impactful component of our architecture. By modeling the agent’s state dynamics as a continuous-time process governed by the HJB equation, the agent captures temporal dependencies across the search trajectory with finer granularity than discrete-step alternatives. The continuous-time formulation also enables adaptive step-size integration, spending more computational budget on challenging spatial transitions. This is particularly beneficial for UAV imagery, where objects of vastly different physical scales (from motorcycles to cars to trucks) appear within the same frame, often at pixel-level sizes.

Lyapunov stability and reward shaping. The Lyapunov-guided entropy-regularized reward shaping mechanism addresses the fundamental challenge of reward sparsity in RL-based detection. The potential-based formulation preserves the optimal policy under shaping (following Ng et al. [40]), while the Lyapunov decrease condition provides an empirically effective convergence signal. We note that the formal convergence bound (Equation (29)) relies on idealized assumptions (exact policy evaluation, sufficient representation capacity) that are not fully met in practice; the bound should therefore be understood as a theoretical motivation rather than a strict runtime guarantee. Empirically, the coupled entropy-regularized stage-wise action masking module helps prevent policy collapse, as shown by the ablation variant without this module: 0.390 mAP@0.5 and 0.235 mAP@0.5:0.95, corresponding to absolute drops of 2.2 and 1.6 percentage points from the full model, respectively.

Failure case analysis. Despite the substantial gains, HRL-Det exhibits performance limitations in several edge cases. As illustrated in the failure rows of Figure 8, Figure 9 and Figure 10, severe occlusion clusters in which multiple objects overlap significantly disrupt the spatial reward signal, causing the agent to commit to a sub-optimal intermediate bounding box and trigger detection prematurely. This limitation is inherent to the MDP formulation, where the greedy terminal action cannot be easily revised. Similarly, heavily truncated objects at image borders and unusual object poses can confound the Neural ODE state encoder, which was trained primarily on complete object contexts. Future work may address these failure modes through multi-agent cooperative search or Monte Carlo Tree Search-based rollout strategies.

Generalization to a general domain. The performance improvement of HRL-Det on MS COCO (mAP@0.5 = 0.735 vs. 0.668 for YOLOv8) is particularly noteworthy, as our model was primarily designed for aerial imagery. This cross-domain generalization suggests that the Continuous-Time Bellman State Evolution and Lyapunov-guided reward shaping encode a generalizable visual search strategy, not overfitted to the specific low-altitude viewpoint or object density of drone datasets.

6. Conclusions

In this paper, we proposed HRL-Det, a hierarchical reinforcement learning framework for sequential object detection in aerial imagery. By formulating the detection problem as a discrete-time Markov Decision Process augmented with continuous-time state evolution inspired by the Hamilton–Jacobi–Bellman equation, our method trains a Dueling Double DQN with Neural ODE-driven state encoding to iteratively refine a spatial bounding box hypothesis. Two core reinforcement learning innovations underpin this framework: (1) the Neural ODE-driven Continuous-Time Bellman State Evolution, which models the agent’s internal latent state dynamics as a stochastic differential equation and uses the HJB residual as an auxiliary training loss, producing enriched state representations for discrete action selection via the adjoint method; and (2) the Lyapunov-Guided Entropy-Regularized Reward Shaping, which constructs convergence-promoting dense reward signals informed by Lyapunov stability analysis while preventing policy collapse through maximum entropy optimization with KL-constrained updates. As discussed in Section 3.4, the continuous-time theoretical framework serves as the design rationale for a practical DQN agent with discrete actions and standard replay-buffer training.

Extensive experiments on three benchmarks—VisDrone2019, DroneVehicle, and MS COCO 2017—demonstrate that HRL-Det achieves competitive or superior mAP compared to conventional single-stage, two-stage, and transformer-based detectors, and consistently outperforms existing RL-based detection methods under the same evaluation protocol, while requiring fewer model parameters (17.3 M) than most dense baselines. Comparisons with non-RL detectors reflect accuracy under the standard COCO metric rather than direct operational equivalence, given the fundamentally different inference paradigms. The ablation study further validates the contribution of each reinforcement learning component. The qualitative analysis of search trajectories and attention heatmaps provides interpretable evidence that the learned entropy-regularized policy concentrates spatial attention on semantically relevant regions.

Future research directions include (1) extending the framework to class-agnostic multi-object sequential detection via a multi-agent reinforcement learning formulation; (2) leveraging model-based RL with world models to further improve sample efficiency; and (3) investigating efficient on-device deployment strategies with neural ODE distillation to support real-time inference on embedded UAV platforms.