Research on Vision-Based Servoing and Trajectory Prediction Strategy for Capturing Illegal Drones

Abstract

A proposed strategy for managing airspace and preventing illegal drones from compromising security involves the use of autonomous drones equipped with three key functionalities. Firstly, the implementation of YOLO-v5 technology allows for the identification of illegal drones and the establishment of a visual-servo system to determine their relative position to the autonomous drone. Secondly, an extended Kalman filter algorithm predicts the flight trajectory of illegal drones, enabling the autonomous drone to compensate in advance and significantly enhance the capture success rate. Lastly, to ensure system robustness and suppress interference from illegal drones, an adaptive fast nonsingular terminal sliding mode technique is employed. This technique achieves finite time convergence of the system state and utilizes delay estimation technology for the real-time compensation of unknown disturbances. The stability of the closed-loop system is confirmed through Lyapunov theory, and a model-based hardware-in-the-loop simulation strategy is adopted to streamline system development and improve efficiency. Experimental results demonstrate that the designed autonomous drone accurately predicts the trajectory of illegal drones, effectively captures them using a robotic arm, and maintains stable flight throughout the process.

1. Introduction

Rotor drones [1], known for their vertical takeoff and landing capabilities, exceptional stability during hovering, agile flight maneuvers, and ease of control, have found widespread applications in aerial photography, mapping, logistics, and search and rescue operations [2]. However, the affordability and accessibility of rotor drones have led to a proliferation of illegal usage driven by curiosity, invasion of privacy, commercial interests, or political motives [3]. Illegal drones pose a grave threat by infringing upon personal privacy and disrupting social order. Existing measures for managing airspace primarily rely on radio frequency jammers and similar devices to interfere with and neutralize illegal drones [4]. Nevertheless, the effectiveness of radio frequency jammers is limited due to their shorter range and inability to detect autonomous drones, resulting in decreased efficiency in congested radio frequency areas [5]. Moreover, using radio frequency jammers in densely populated regions is unsuitable as it may endanger ground personnel when the illegal drones plummet to the ground [6]. Therefore, the most viable approach to combatting illegal drones lies in autonomously identifying, tracking, and capturing them, while ensuring a stable flight state and transporting them to secure zones. Considering the high mobility characteristic of illegal drones [7] and maintaining stable flight conditions during the identification, tracking, and capture process poses a formidable challenge.

In recent years, there has been a widespread application of intelligent perception components such as cameras [8], millimeter-wave radar [9], and LIDAR [10] in mobile robots. By incorporating cameras with deep learning-based object detection algorithms like YOLO (You Only Look Once) [11], SSD (Single Shot MultiBox Detector) [12], and RetinaNet [13], these robots have demonstrated exceptional recognition capabilities for shapes and colors, enabling the accurate tracking of illegal unmanned aerial vehicles (drones). Cameras, with their lightweight and compact design, do not compromise the payload capacity of autonomous drones [14]. When compared to millimeter-wave radar and LIDAR, cameras are the most suitable choice for identifying illegal drones.

Visual servoing applies visual sensors to acquire real-time image information and utilizes these data to accomplish objectives such as target tracking, localization and navigation, posture control, and feedback control [15]. This technology enables the precise positioning, path planning, posture adjustment, and autonomous control of robots [16]. Visual servoing encompasses Position-based Visual Servoing (PBVS) and Image-based Visual Servoing (IBVS) [17]. PBVS involves establishing a mapping relationship between the autonomous drone and the target drone in relation to the inertial coordinate system [18], requiring GPS information. However, commercial GPS information has a positioning error of up to 1 m, which is larger than the size of the target drones. On the other hand, IBVS directly defines image plane coordinates using image features, rather than task space [19]. Its fundamental principle involves calculating control variables from the error signal and converting them into the motion space of the autonomous drone. By employing the obtained image error for closed-loop feedback control, the autonomous drone can move towards the target drone and accomplish the tracking process.

Considering the maneuverability of illegal drones, if the position of the illegal drone in the next moment can be predicted in advance during the tracking process, it would be beneficial for the trajectory planning [20] of the autonomous drone, allowing it to reach the capture position earlier and improve capture efficiency. Currently, trajectory prediction methods mainly include interpolation methods, linear regression, filtering methods, dynamic model methods, and machine learning methods [21]. The position signal of the illegal drone obtained using computer vision techniques contains noise [22]. Interpolation methods are sensitive to noisy data or outliers, resulting in prediction deviations. In addition, interpolation methods are usually based on linear or polynomial interpolation principles, making it difficult to reflect complex nonlinear relationships. Linear regression models have a trade-off between bias and variance. When the model complexity is insufficient, underfitting may occur, meaning the model fails to fit the data well. On the other hand, when the model complexity is too high, overfitting may occur, whereby the model performs well on the training data but poorly on new data. Dynamic model methods and machine learning methods both involve solving complex mathematical optimization problems, especially for nonlinear and non-convex problems, which can be time-consuming. Filtering methods mainly include Kalman filtering [23], particle filtering, etc. Among them, the Extended Kalman Filter (EKF) algorithm [24], which is an extension of the KF algorithm, can be used for nonlinear models, and is highly acclaimed for its low computational complexity and fewer computational resource requirements. It provides a feasible solution for real-time and efficient state estimation and trajectory prediction.

Configuring an automatically triggered robotic arm beneath the autonomous drone, which closes and grasps the unauthorized drone upon catching up with it, is crucial. However, during this process, the autonomous drone is susceptible to strong interference, which can potentially lead to a crash. Enhancing the robustness of the system is vital to mitigate the risk of a crash. The real-time estimation of external disturbances and compensation for unknown disturbances are key challenges in achieving stable flight for autonomous drones. Time Delay Estimation Control (TDC) is a nonlinear control strategy that effectively addresses unknown disturbances in dynamic control problems with complex nonlinear effects [25]. TDC utilizes the previous state of the system to estimate the current collective dynamics. It offers a concise and efficient model-free control strategy. TDC is suitable for handling continuous and slow-varying unknown disturbances [26]. However, when impact signals exist within the system, the control effectiveness of TDC can be weakened. Therefore, combining TDC with other robust control algorithms can compensate for estimation errors and enhance the robustness of the control system.

Sliding mode control is a robust, fast-response, and simple implementation nonlinear control strategy [27]. It suppresses system parameter disturbances and external disturbances by introducing sliding manifolds and quickly responds to changes in system states [28]. With advances in sliding mode control theory [29], Fast Non-Singular Terminal Sliding Mode Control (FNTSM) has emerged in recent years [30]. It achieves the finite-time convergence of system states and improves control robustness. Therefore, for a class of underactuated drone systems containing time-varying uncertain nonlinearities, unknown, and impulsive disturbances, the complex external disturbances are generalized as an aggregated unknown term. Based on Terminal Sliding Mode Control (TSMC), FNTSM is introduced to construct a composite controller. The traditional PD-type sliding manifold is upgraded to a PID-type sliding manifold, which obtains faster transient response and smaller steady-state error through the integral term [31]. Furthermore, to avoid the drawback of fixed high gain coefficients that may lead to system oscillations and trigger high-frequency unmodeled terms, a parameter adaptive updating mechanism is introduced into the PID-type sliding manifold to achieve the adaptive tuning of control parameters. The derivative of the control gain is proportional to the sliding manifold, which is beneficial for suppressing the adverse effects of noise. Without changing the structure of the sliding manifold, the increase in gain automatically adjusts with the change of the sliding manifold and accelerates convergence speed when the sliding variable deviates from the sliding manifold.

The contributions of this work can be further emphasized as follows: (1) In order to suppress the interference of illegal drones on autonomous drones and maintain stable flight, sliding mode control is introduced into Target Drone Control (TDC), forming a new robust control strategy integrating adaptive PID-type sliding manifold, TDC, and fast non-singular terminal sliding mode, which has strong robustness and characteristics of model-free control. (2) In order to reach the capture position in advance and improve capture efficiency, the EKF algorithm is integrated into IBVS to estimate the future position of the target drone, providing feasible solutions for real-time and efficient state estimation and trajectory prediction. The superiority of the proposed methods is verified through counter drone capture experiments.

The remaining parts of this paper are organized as follows: Section 2 presents the system design, including the modeling of the counter drone dynamics, IBVS method, EKF trajectory prediction, and controller design. Section 3 presents hardware-in-the-loop simulation and flight experiments. Section 4 concludes the paper.

2. System Design

The system diagram in Figure 1 depicts the remote communication between the ground station and the counter drone, which is equipped with an onboard computer featuring autonomous flight capabilities and intelligent perception using a camera to detect the illegal drone. The proposed airspace management strategy incorporates three critical technologies: IBVS, a trajectory prediction module based on the EKF, and a non-linear robust controller.

The core idea of the counter capture strategy is to integrate IBVS and EKF algorithms to estimate the future position of the illegal drone. The mechanical arm is then used to capture the target at a specified velocity, ultimately bringing the illegal drone to a designated area. Subsequently, key issues such as the dynamic modeling of the counter drone, IBVS, and trajectory prediction will be discussed.

2.1. Counter Drone Dynamics Modeling

The dynamics model of the drone is utilized for designing robust controllers. The counter drone is a quadcopter, which is a nonlinear, underactuated, and strongly coupled dynamic system with four input control signals and six output states. As shown in Figure 2, the counter drone is considered as a rigid body model, experiencing the lift generated by four motors and its own weight, with three position coordinates and three attitude angles. The figure depicts the world coordinate system $o_e-x_e{y}_e{z}_e$ denoted as {e} and the body coordinate system $o_b-x_b{y}_b{z}_b$ denoted as {b}.

The counter drone dynamics model consists of both position dynamics and attitude dynamics models, with the position dynamics model encompassing horizontal and vertical channel dynamics models [32].

The horizontal channel dynamics model is established based on the Newton–Euler dynamics theory [33]:

$${\ddot{\mathit{p}}}_{\mathit{h}}^{\mathit{e}}=g{\mathit{R}}_{\mathit{\psi }}{\mathit{\theta }}_{\mathit{h}}-{\mathit{\eta }}_{\mathit{h}}$$

(1)

where ${\ddot{\mathit{p}}}_{\mathit{h}}^{\mathit{e}}=\left[\begin{array}{cc}\ddot{X}& \ddot{Y}\end{array}\right]$ represents the horizontal acceleration in the world coordinate system; X and Y represent the positions in the x_e and y_e directions under the world coordinate system, respectively; $\dot{X}$ and $\dot{Y}$ represent the speeds in the x_e and y_e directions in the world coordinate system, respectively; ${\mathit{R}}_{\mathsf{\psi }}=\left[\begin{array}{c}\sin \psi \cos \psi \\ -\cos \psi \sin \psi \end{array}\right]$; ${\mathit{\theta }}_{\mathit{h}}={\left[\varphi \theta \right]}^{\mathrm{T}}$; $\psi$ represents the yaw angle; $\varphi$ represents the roll angle; $\theta$ represents the pitch angle; ${\mathit{\eta }}_{\mathit{h}}$ represents the horizontal disturbance.

The dynamic model of the vertical channel is established as follows:

$${\ddot{\mathit{p}}}_{\mathit{z}}^{\mathit{e}}=\frac{f}{m}\cos \varphi \cos \theta -g-\frac{l{\dot{\mathit{p}}}_{\mathit{z}}^{\mathit{e}}}{m}+\Delta P$$

(2)

where $f=f_1+f_2+f_3+f_4$ is the total thrust of the drone; ${\mathit{p}}_{\mathit{z}}^{\mathit{e}}=[\mathit{Z}]$, ${\dot{\mathit{p}}}_{\mathit{z}}^{\mathit{e}}=[\dot{\mathit{Z}}]$, and ${\ddot{\mathit{p}}}_{\mathit{z}}^{\mathit{e}}=[\ddot{\mathit{Z}}]$ are the displacements, velocities, and accelerations in the z direction in the world coordinate system, respectively; m is the total mass of the drone; g is the acceleration of gravity; l is the drag coefficient; and ΔP is the mechanical hand dynamics term.

We define ${\mathit{\eta }}_{\mathit{z}}$ as follows:

$${\mathit{\eta }}_{\mathit{Z}}=\mathit{f}+l{\dot{\mathit{p}}}_{\mathit{Z}}^e+\left[\begin{array}{c}0\\ 0\\ mg\end{array}\right]-\mathit{f}\cos \varphi \cos \theta -\Delta \mathit{P}\times m$$

(3)

where ${\mathit{\eta }}_{\mathit{z}}$ can be considered as a lumped uncertainty term that includes counter drone gravity, torque, manipulator dynamics, and external disturbances. Then the above formula can be changed to:

$$m{\ddot{\mathit{p}}}_z^e=\mathit{f}-{\mathit{\eta }}_z$$

(4)

The following content is the posture dynamics model

$${\mathit{G}}_a+\mathit{\tau }=\mathit{J}{\dot{\mathit{\omega }}}^{\mathit{b}}+{\omega }^{\mathit{b}}\times \mathit{J}{\mathit{\omega }}^{\mathit{b}}$$

(5)

where $\mathit{\tau }={\left[{\tau }_x {\tau }_y {\tau }_z\right]}^{\mathrm{T}}$ represents the torque of the propeller relative to the body axis; $\mathit{J}=\left[\begin{array}{l}{\mathit{J}}_{\mathrm{xx}} \\ {\mathit{J}}_{\mathrm{yy}} \\ {\mathit{J}}_{\mathrm{zz}}\end{array}\right]$ denotes the rotational inertia about the three axes of the counter drone; ${\mathit{\omega }}^{\mathrm{b}}={\left[{\omega }_{\mathrm{x}}^{\mathrm{b}} {\omega }_{\mathrm{y}}^{\mathrm{b}} {\omega }_{\mathit{z}}^{\mathrm{b}}\right]}^{\mathrm{T}}={\left[\mathit{p} q r\right]}^{\mathrm{T}}$ corresponds to the angular velocity in the body coordinate system; ${\dot{\mathit{\omega }}}^{\mathrm{b}}$ stands for the angular acceleration in the body coordinate system; and ${\mathit{G}}_{\mathrm{a}}={\left[G_{\mathrm{a}\varphi } G_{\mathrm{a}\theta } G_{\mathrm{a}\psi }\right]}^{\mathrm{T}}$ signifies the gyroscopic torque experienced by the drone.

$$\left[\begin{array}{c}{G}_{\mathrm{a}\varphi }\\ G_{\mathrm{a}\theta }\\ G_{\mathrm{a}\psi }\end{array}\right]=\left[\begin{array}{c}{\mathit{J}}_{\mathit{R}\mathit{P}}q\left({\omega }_1-{\omega }_2-{\omega }_3+{\omega }_4\right)\\ {\mathit{J}}_{\mathit{R}\mathit{P}}q\left(-{\omega }_1+{\omega }_2-{\omega }_3+{\omega }_4\right)\\ 0\end{array}\right].$$

(6)

Based on the small perturbation assumption, we obtain

$$\left\{\begin{array}{c}\dot{p}=\frac{1}{I_{\mathrm{x}\mathrm{x}}}\left[{\tau }_{\mathrm{x}}+qr(I_{\mathrm{y}\mathrm{y}}-I_{\mathrm{z}\mathrm{z}})-{\mathit{J}}_{\mathit{R}\mathit{P}}q\Omega \right]\\ \dot{q}=\frac{1}{I_{\mathrm{y}\mathrm{y}}}\left[{\tau }_{\mathrm{y}}+qr(I_{\mathrm{z}\mathrm{z}}-I_{\mathrm{x}\mathrm{x}})-{\mathit{J}}_{\mathit{R}\mathit{P}}q\Omega \right]\\ \dot{r}=\frac{1}{I_{\mathrm{z}\mathrm{z}}}\left[{\tau }_{\mathrm{z}}+qr(I_{\mathrm{z}\mathrm{z}}-I_{\mathrm{x}\mathrm{x}})\right]\end{array}\right..$$

(7)

where $\Omega ={\omega }_1-{\omega }_2-{\omega }_3+{\omega }_4$.

2.2. Image-Based Visual Servoing Method

During the theoretical analysis phase, two fundamental assumptions are provided.

Assumption 1:

The relative position between the counter drone’s body coordinate system and the camera coordinate system remains constant.

Rotation matrix

$${\mathit{R}}_{\mathrm{c}}^{\mathrm{b}}=\left[\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\\ 0& 1& 0\end{array}\right].$$

(8)

Assumption 2:

The counter drone can record in real time the center coordinates $\mathit{p}(x_{\mathrm{i}},y_{\mathrm{i}})$of the illegal drone, which are acquired via machine vision methodologies, such as YOLO-v5 [34], and can be used to predict its future trajectory center coordinates $\hat{\mathit{p}}({\hat{x}}_{\mathrm{i}},{\hat{y}}_{\mathrm{i}})$.

The core idea of IBVS is to achieve the coincidence of the future trajectory center coordinates $\hat{\mathit{p}}({\hat{x}}_i,{\hat{y}}_i)$ of the illegal drone and the center coordinates of the manipulator claw in the pixel coordinate system {i} [35]. The relative positional relationships between the camera, camera plane, and the illegal drone are illustrated in Figure 3. In the figure, $o_c-x_c{y}_c{z}_c$ represents the camera coordinate system, and $o_i-un$ represents the pixel coordinate system.

The counter drone is equipped with a forward pinhole camera fixed on its lower part, denoted as coordinate system $C=\left\{X_c,Y_c,{\mathit{Z}}_c\right\}$. ${\left[{}^{v}u, {}^{v}n\right]}^T$ and $\lambda$ represent the image plane axis and the camera focal length, respectively. The quadcopter drone is an underactuated system. To decouple the dynamics of the quadcopter, a virtual camera frame $\mathrm{\Xi }=\left\{X_v,Y_v,{\mathit{Z}}_v\right\}$ is introduced. The origin of $\mathrm{\Xi }\left({\mathit{O}}_c\right)$ coincides with C, and the $X_v$ axis is always parallel to the $X_e$ axis. Additionally, the definition of the virtual image plane is similar to the definition of the image plane in C.

Assuming the illegal drone is a stationary target point P, with the world coordinate and the virtual camera frame, $\mathrm{\Xi }\left({\mathit{O}}_c\right)$ is ${}^e\mathit{p}={\left\{{}^{e}x,{}^{e}y,{}^{e}z\right\}}^T$ and ${}^v\mathit{p}={\left\{{}^{v}x,{}^{v}y,{}^{v}z\right\}}^T$, respectively. Their transformation equations can be expressed as

$${}^v\mathit{p}={\mathit{R}}_{\varphi }^T\left({}^e\mathit{p}-{\mathit{O}}_v\right)$$

(9)

where

$${\mathit{R}}_{\varphi }=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \varphi & \sin \varphi \\ 0& -\sin \varphi & \cos \varphi \end{array}\right].$$

(10)

Differentiating ${}^v\mathit{p}$ with respect to time

$$\begin{array}{ll}{}^v\dot{\mathit{p}}\hfill & ={\dot{\mathit{R}}}_{\varphi }^T\left({}^e\mathit{p}-{\mathit{O}}_v\right)-{\mathit{R}}_{\varphi }^T{\dot{\mathit{O}}}_v\hfill \\ & =-\mathit{sk}\left(\dot{\varphi }{\mathit{e}}_x\right){\mathit{R}}_{\varphi }^T\left({}^e\mathit{p}-{\mathit{O}}_v\right)-{\mathit{R}}_{\varphi }^T{\dot{\mathit{O}}}_v\hfill \\ & =-\mathit{sk}\left(\dot{\varphi }{\mathit{e}}_x\right){}^v\mathit{p}-\mathit{v}\hfill \end{array}.$$

(11)

where the linear velocities of the illegal drone in the framesworld coordinate and virtual camera frame $\mathrm{\Xi }\left({\mathit{O}}_c\right)$ are denoted as ${\dot{\mathit{O}}}_v={\left[{}^{e}v_x, {}^{e}v_y, {}^{e}v_z\right]}^T$ and $\mathit{v}={\left[{}^{v}v_x, {}^{v}v_y, {}^{v}v_z\right]}^T$, respectively, with the unit vector ${\mathit{e}}_x={\left[1,0,0\right]}^T$.

Let the coordinates of the illegal drone P in the virtual image plane be labeled as P. The perspective projection equation in camera frame $\mathrm{\Xi }\left({\mathit{O}}_c\right)$ is then expressed as:

$$\mathit{p}=\left[\begin{array}{c}{}^{v}u\\ {}^{v}n\end{array}\right]=\frac{\lambda }{{}^{v}x}\left[\begin{array}{c}{}^{v}y\\ {}^{v}z\end{array}\right]$$

(12)

Substituting the above equation into Equation (11), the visual servoing equation is obtained, and the expression for the illegal drone P in the virtual image plane is:

$$\dot{\mathit{p}}={\mathit{J}}_v{\dot{\mathit{r}}}_v$$

(13)

where

$${\mathit{J}}_v=\left[\begin{array}{cccc}{}^{v}u/{}^{v}x& -\lambda /{}^{v}x& 0& -{}^{v}n\\ {}^{v}n/{}^{v}x& 0& -\lambda /{}^{v}x& {}^{v}u\end{array}\right]$$

(14)

Through the Jacobian matrix J_v in Equation (13), it is evident that ${\dot{\mathit{r}}}_{\mathit{v}}={\left({\mathit{v}}^T,\dot{\varphi }\right)}^T$ is not related to the pitch and yaw angles of the counter drone. To maintain the stability of the counter drone’s flight, the desired pitch and yaw angles are set to zero.

Equation (13) expresses how the illegal drone moves in the image plane when the camera is in motion. For visual servoing, the focus is on its inverse problem—given the motion of the illegal drone in the camera plane, we determine the camera’s motion.

We then transform Equation (13) into:

$${\dot{\mathit{r}}}_v={\mathit{J}}_v^{-1}\dot{\mathit{p}}$$

(15)

$$\dot{\mathit{p}}={\lambda }_{\mathrm{p}}\left({\mathit{p}}_d-\mathit{p}\right)$$

(16)

The first term of ${\dot{\mathit{r}}}_{\mathit{v}}={\left({\mathit{v}}^T,\dot{\varphi }\right)}^T$ corresponds to the velocity of the drone in the z-axis direction, denoted as B. Under the assumption of small perturbation linearization, the second term represents the angular rate of the desired roll angle. With this, the dynamic deconstruction of the counter drone in visual servoing is concluded.

2.3. Extended Kalman Filter Trajectory Prediction

To reach the interception position in advance, the EKF algorithm is employed to predict the position of the illegal drone in pixel coordinates. The EKF algorithm utilizes Taylor series expansion to linearize the treatment of nonlinear systems, followed by recursive computation and estimation. Its nonlinear expression is given by

$$\left\{\begin{array}{l}{\mathit{x}}_k=\mathrm{f}\left({\widehat{\mathit{x}}}_{k-1},{\mathit{u}}_{k-1},{\overline{\varpi }}_{k-1}\right)+\mathit{A}\left({\mathit{x}}_k-{\widehat{\mathit{x}}}_{k-1}\right)+{\overline{\varpi }}_k{\overline{\varpi }}_{k-1}\\ {\mathit{z}}_k=\mathit{h}\left({\tilde{\mathit{x}}}_k,{\mathit{v}}_k\right)+\mathit{H}\left({\mathit{x}}_k-{\tilde{\mathit{x}}}_k\right)+\mathit{V}{\mathit{v}}_k\end{array}\right..$$

(17)

In the equation, f(x) and h(x) represent the nonlinear mapping relationship; k is the time step; ${\mathit{x}}_k$ is the state vector; ${\mathit{u}}_k$ is the control vector; ${\overline{\varpi }}_k$ is the process noise; ${\mathit{z}}_k$ is the measurement vector; $v_k$ is the measurement noise; k−1 corresponds to the previous time step; $\mathit{p}({\overline{\varpi }}_{k-1})\sim N(0,\mathit{Q})$; $\mathit{p}(v_{k-1})\sim N(0,\mathit{R})$; $\mathit{p}({\overline{\varpi }}_k{\overline{\varpi }}_{k-1})\sim N(0,{\overline{\varpi }}_k\mathit{Q}{\overline{\varpi }}_k^{\mathrm{T}})$; $\mathit{p}(\mathit{V}{\mathit{v}}_{k-1})\sim N(0,\mathit{VQ}{\mathit{V}}^{\mathit{T}})$; $\mathit{Q}=E\left[{\overline{\varpi }}_{k-1} {{\overline{\varpi }}_{k-1}}^{\mathrm{T}}\right]$ is the observation error covariance matrix; $\mathit{R}=E\left[{\mathit{v}}_k {{\mathit{v}}_k}^{\mathrm{T}}\right]$ is the measurement error covariance matrix.

The linearization process is as follows:

$$\left\{\begin{array}{l}\mathit{A}={\left.\frac{\partial f}{\partial x}\right|}_{{\widehat{x}}_{k-1},u_{k-1}}\\ \overline{\varpi }={\left.\frac{\partial f}{\partial \overline{\varpi }}\right|}_{{\widehat{\lambda }}_{k-1},u_{k-1}}\\ \mathit{H}={\left.\frac{\partial h}{\partial x}\right|}_{{\tilde{x}}_k}\\ \mathit{V}={\left.\frac{\partial h}{\partial v}\right|}_{{\tilde{x}}_k}\end{array}\right.$$

(18)

The linearized equation for Formula (11) is as follows:

$$\left\{\begin{array}{l}{\mathit{x}}_k={\tilde{\mathit{x}}}_k+\mathit{A}\left({\mathit{x}}_k-{\hat{\mathit{x}}}_{k-1}\right)+{\overline{\varpi }}_k{\overline{\varpi }}_{k-1z}\\ {\mathit{z}}_k={\tilde{\mathit{z}}}_k+\mathit{H}\left({\mathit{x}}_k-{\tilde{\mathit{x}}}_k\right)+\mathit{V}{\mathit{v}}_k\end{array}\right.$$

(19)

The prediction process comprises a prediction equation and a correction equation.

Prediction Equation:

$$\left\{\begin{array}{l}{\widehat{x}}_K^{-}=\mathrm{f}\left({\tilde{\mathit{x}}}_{k-1},{\mathit{u}}_{k-1},0\right)\\ {\mathit{P}}_K^{-}=\mathit{A}{\mathit{P}}_{k-1}{\mathit{A}}^{\mathrm{T}}+{\overline{\varpi }}_k\mathit{Q}{\overline{\varpi }}_k^{\mathrm{T}}\end{array}\right.$$

(20)

Correction Equation:

$$\left\{\begin{array}{l}{\mathit{K}}_k=\frac{{\mathit{P}}_k^{-}{\mathit{H}}^{\mathrm{T}}}{\mathit{H}{\mathit{P}}_k^{-}{\mathit{H}}^T+{\mathit{VRV}}^{\mathrm{T}}}\\ {\widehat{\mathit{x}}}_k^{-}={\widehat{\mathit{x}}}_k^{-}+{\mathit{K}}_k\left({\mathit{Z}}_k-\mathrm{h}({\widehat{\mathit{x}}}_k^{-},0)\right)\\ {\mathit{P}}_k=\left(\mathit{I}-{\mathit{K}}_k\mathit{H}\right){\mathit{P}}_k^{-}\end{array}\right..$$

(21)

where ${\mathit{P}}_k^{-}=E\left[({\mathit{x}}_k-{\widehat{\mathit{x}}}_k^{-}){({\mathit{x}}_k-{\widehat{\mathit{x}}}_k^{-})}^{\mathrm{T}}\right]$; ${\mathit{Z}}_k$ represents the true value of the visual servoing feedback; ${\widehat{\mathit{x}}}_k^{-}$ is the a priori estimate; I is the identity matrix.

Directly predicting the coordinates in (x,y), we can define:

$$f\left(x,y\right)=\left[\begin{array}{c}{x}^2+xy+2\\ y^2+xy+2\end{array}\right].$$

(22)

obtaining

$$A=\left[\begin{array}{cc}2x+y& x\\ y& 2y+x\end{array}\right].$$

(23)

We then define

$$h\left(L,\theta \right)=\left[\begin{array}{c}\sqrt{x^2+y^2}\\ \arctan \frac{y}{x}\end{array}\right]$$

(24)

and obtain

$$H=\left[\begin{array}{cc}\frac{x}{\sqrt{x^2+y^2}}& \frac{y}{\sqrt{x^2+y^2}}\\ -\frac{y}{x^2\left(\frac{y^2}{x^2}+1\right)}& \frac{1}{x^2\left(\frac{y^2}{x^2}+1\right)}\end{array}\right].$$

(25)

2.4. Controller Design

The goal of controller design is to endow the counter drone with high maneuverability and precision during the interception of the illegal drone, suppress disturbances caused by the illegal drone to the system, and achieve strong robustness.

Following the core idea of IBVS, the tracking error is defined as:

$$\left\{\begin{array}{l}{e}_{\mathrm{x}} = x_r - {\widehat{x}}_{\mathrm{i}}\\ e_{\mathrm{y}} = y_r - {\widehat{y}}_{\mathrm{i}}\end{array}\right.$$

(26)

In the equation, $e_{\mathrm{x}}$ and $e_{\mathrm{y}}$ represent the tracking errors in the horizontal and vertical directions between the predicted center and the visual servoing center, respectively. $(x_r,y_r)$ denotes the coordinates of the visual servoing target point, chosen here as the center of the manipulator claw in the camera plane. $\widehat{\mathit{p}}({\widehat{x}}_{\mathrm{i}},{\widehat{y}}_{\mathrm{i}})$ represents the coordinates of the predicted center.

The drone’s control process is illustrated in Figure 4. To ensure the precise and rapid tracking of the target center in visual servoing, a controller is designed. We take the vertical channel as an example.

The following sliding mode surface [20] is chosen

$$s={\dot{s}}_{\mathrm{y}}+k_1{s}_{\mathrm{y}}+k_2\mathrm{sig}{(s_{\mathrm{y}})}^{\mathit{p}}.$$

(27)

where $k_1, k_2>0$; $0<\mathit{p}<1$.

To eliminate oscillations and achieve finite-time convergence, the FTSM reaching law is employed.

$$s_{\mathrm{y}}=e_{\mathrm{y}}+\beta \mathrm{sig}{({\dot{e}}_{\mathrm{y}})}^{\mathsf{\gamma }}$$

(28)

where $\mathrm{sig}{(y)}^{\mathsf{\gamma }}={\left|y\right|}^{\mathsf{\gamma }}\mathrm{sign}(y)$, $\mathrm{sign}()$ is the sign function; $\beta >0$; $1<\mathsf{\gamma }<2$.

The system’s stabilization time is

$$t_s=\frac{{\beta }^{-\frac{1}{\gamma }}}{1-\frac{1}{\gamma }}\left|e_{\mathrm{y}}\right.{\left.\left(0\right)\right|}^{1-\frac{1}{\gamma }}$$

(29)

Since ${\ddot{e}}_{\mathrm{y}}={\ddot{y}}_{\mathrm{r}}-{\ddot{\widehat{y}}}_{\mathrm{i}}$, considering Equation (4), the control law is designed as follows:

$$u_z=m{v}_z+{\widehat{\mathit{\eta }}}_z$$

(30)

where $v={\ddot{y}}_{\mathrm{r}}+{\beta }^{-1}{\mathsf{\gamma }}^{-1}\mathrm{sig}{({\dot{e}}_{\mathrm{y}})}^{2-\mathsf{\gamma }}+k_1{s}_{\mathrm{y}}+k_2\mathrm{sig}{({\dot{s}}_{\mathrm{y}})}^{\mathit{p}}$.

Due to the inability to establish an accurate mathematical model for the impact disturbance at the moment of interception, the TDE technique is employed for online estimation and real-time compensation of unknown disturbances. This approach enables the acquisition of the nominal model of the controlled object, thereby simplifying the controller structure.

Based on the vertical channel’s dynamic model (4), in conjunction with Equation (27), the TDE technique is utilized to obtain ${\widehat{\mathit{\eta }}}_{\mathit{z}}$:

$${\widehat{\eta }}_{\mathit{z}}={\eta }_{z(t-k)} = u_{z(t-k)} - m{\ddot{y}}_{r(t-k)}.$$

(31)

where t represents the current time, and (t − k) corresponds to the current time delayed by k time. The complete expression of the control law is:

$$\begin{array}{c}{u}_{\mathit{z}}=\underset{\mathrm{FNTSM} }{\underbrace{m({\ddot{y}}_{\mathrm{r}}+{\beta }^{-1}{\mathsf{\gamma }}^{-1}\mathrm{sig}{(e_{\mathrm{y}})}^{2-\mathsf{\gamma }}+k_1{s}_{\mathrm{y}}+k_2\mathrm{sig}{(s_{\mathrm{y}})}^{\mathit{p}})}}+\hfill \\ \underset{\mathrm{TDE}}{\underbrace{u_{(t-k)}-m{\ddot{y}}_{\mathrm{r}(t-k)}}}\hfill \end{array}.$$

(32)

Similarly, the control law’s expression for the horizontal channel is

$$\begin{array}{c}{u}_{\mathit{h}}=\underset{\mathrm{FMTSM}}{\underbrace{m\left({\ddot{x}}_{\mathrm{r}}+{\beta }^{-1}{\gamma }^{-1}\mathrm{sig}{(e_{\mathrm{x}})}^{2-\gamma }+k_1{s}_{\mathrm{x}}+k_2\mathrm{sig}{(s_{\mathrm{x}})}^p\right)+}}\hfill \\ \underset{\mathrm{TDE}}{\underbrace{u_{\left(t-k\right)}-m{\ddot{x}}_{\mathrm{r}}(t-k)}}\hfill \end{array}.$$

(33)

Taking the vertical channel as an example to demonstrate the stability of the proposed controller, the Lyapunov design is formulated as

$$V=\frac{1}{2}{s}_{\mathrm{y}}^2.$$

(34)

The derivative with respect to time is

$$\dot{V}=s_{\mathrm{y}}{\dot{s}}_y.$$

(35)

Taking the derivative of Equation (28) and substituting it into Equation (35), we get

$$V=s_{\mathrm{y}}\left({\dot{e}}_y+\left.\beta \gamma \right|{\left.{\dot{e}}_y\right|}^{\gamma -1}{\ddot{e}}_y\right).$$

(36)

By simultaneously solving Equations (30) and (31), we obtain

$${\ddot{e}}_{\mathrm{y}}+{\beta }^{-1}{\mathsf{\gamma }}^{-1}\mathrm{sig}{({\dot{e}}_{\mathrm{y}})}^{2-\mathsf{\gamma }}+k_1{s}_{\mathrm{y}}+k_2\mathrm{sig}{(s_{\mathrm{y}})}^{\mathit{p}}=\sigma .$$

(37)

For the estimation error $\sigma$ of TDE, there always exists a positive number $\phi$ such that $\sigma$ is bounded, i.e., $\left|\sigma \right|\le \phi$.

Substituting Equation (37) into Equation (36), we get:

$$\dot{V}=-s_{\mathrm{y}}\beta \mathsf{\gamma }{\left|e_{\mathrm{y}}\right|}^{\mathsf{\gamma }-1}\left[-\sigma +k_1{s}_{\mathrm{y}}+k_2{\mathrm{sig}(\mathrm{s}}_{\mathrm{y}}{)}^{\mathit{p}}\right].$$

(38)

The above equation is further discussed in the two cases

$$\dot{V}=-s_{\mathrm{y}}\beta \gamma {\left|{\dot{e}}_{\mathrm{y}}\right|}^{\gamma -1}\left[\left(k_1-\frac{\sigma }{s_{\mathrm{y}}}\right)s_{\mathrm{y}}+k_2\mathrm{sig}{\left(s_{\mathrm{y}}\right)}^p\right],$$

(39)

and

$$\dot{V}=-s_{\text{y}}\beta \gamma {\left|{\dot{e}}_{\text{y}}\right|}^{\gamma -1}\left[k_1{s}_{\text{y}}+\left(k_2\frac{\sigma }{\mathrm{sig}{\left(s_{\text{y}}\right)}^p}\right)\mathrm{sig}{\left(s_{\text{y}}\right)}^p\right]$$

(40)

For equation (39), when $\left|s_{\mathrm{y}}\right|>k_1^{-1}\left|\sigma \right|$, $\dot{V}<0$, the sliding surface can converge to the region in finite time

$$\left|s_{\mathrm{y}}\right|\le {\epsilon }_1.$$

(41)

where ${\epsilon }_1\triangleq k_1^{-1}\phi$.

For Equation (40), when ${\left|s_{\mathrm{y}}\right|}^{\mathrm{p}}>k_2^{-1}\left|\sigma \right|$, $\dot{V}<0$, the sliding surface can converge to the region in finite time

$$\left|s_{\mathrm{y}}\right|\le {\epsilon }_2.$$

(42)

where ${\epsilon }_2\triangleq \sqrt[\mathit{p}]{(k_2^{-1}\varphi )}$.

Combining Equations (41) and (42), the sliding surface converges in finite time to

$$\left|s_y\right|\le {\epsilon }_{\min }.$$

(43)

where ${\epsilon }_{\min }\triangleq \min ({\epsilon }_1,{\epsilon }_2)$.

3. Simulation and Experimentation

To validate the effectiveness of the proposed control strategy, Hardware in the Loop (HITL) [36] simulation experiments were conducted for algorithm verification. The HITL experiment involved the Robot Operating System (ROS) [37], Gazebo11 3D dynamic simulation software [38], and the Pixhawk 4 (PX 4) flight controller [39]. A repeatable, risk-free, and realistic simulation environment was established, incorporating physics characteristics such as collisions and aerodynamics to enhance development quality and efficiency. Subsequently, real flight experiments were conducted to optimize parameters during actual flights, further confirming the effectiveness of the proposed control strategy.

3.1. Hardware in the Loop Experiment

To simulate the states of actual flight, multiple states were set in the HITL simulation experiment, including environment initialization, take-off direction, target search, and tracking. Firstly, relevant scenes were created in the Gazebo 3D dynamic simulation environment, configured with capture objects, and simulated using an RGB camera to emulate a CCD industrial camera. Leveraging the distributed communication of the ROS system, target information was acquired, and the drone state was dynamically adjusted according to the proposed control strategy, reflecting real-time changes in the simulation scene. Finally, the PX 4 was used to control the drone to execute target capture tasks, as depicted in Figure 5 in the simulation scenario.

To reduce the requirements placed on the graphics card and enhance simulation efficiency, a red ball is used as a substitute for the illegal unmanned aerial vehicle. In the experiment, the red ball undergoes three-dimensional random movement with speeds set at 1 m/s, 5 m/s, and 10 m/s. The counter drone employs the same robust control strategy for tracking and capture. In the absence of EKF prediction, the tracking error in pixel coordinates is shown in Figure 6. When the tracking error is less than 20 pixels, it is considered as successfully tracking the target. In the HITL simulation experiments, the red ball moves at the aforementioned speeds, and the probabilities of the counter drone successfully tracking the target are 100%, 90%, and 12%, respectively. When EKF prediction is applied, the tracking error in pixel coordinates is shown in Figure 7, and the probabilities of the counter drone successfully tracking the target are 100%, 98%, and 80%, respectively. A comparison between Figure 6 and Figure 7 indicates that the use of EKF prediction improves the capture success rate and shortens the capture time. By predicting the future position of the target in advance, the drone can adjust its own state earlier for better target capture. When the red ball moves at a speed of 10 m/s, the counter drone approaches its thrust limit to catch up with the ball, and the pitch angle approaches 70° during acceleration. The HITL simulation results demonstrate that using EKF prediction allows for better target capture. As the target speed increases, both the drone’s flight speed and pitch angle show an increasing trend. Lower speeds result in a more stable capture process. The HITL simulation experiments also validate the robustness of the FNTSM-TDE control strategy.

Through HITL simulation experiments, the capture strategy is adjusted and optimized to more accurately capture the illegal unmanned aerial vehicle. In the visual servo system, controllers are designed by fully utilizing visual information, and the EKF algorithm is employed to predict the future trajectory of the illegal drone. Using the proposed controller as the underlying flight control strategy, the predicted trajectory is combined with the drone’s state, enabling the counter drone to obtain a feedforward control effect and enhance the capture success rate.

3.2. Flight Experiment

The counter drone comprises the drone mainframe and a manipulator suspended underneath. The top of the counter drone is equipped with the MID-360 laser radar, utilizing Simultaneous Localization and Mapping (SLAM) technology to create a high-resolution environmental map. By integrating SLAM technology and Inertial Measurement Unit (IMU) data, the counter drone obtains its own position, velocity, and acceleration. The system also includes an industrial camera and an on-board computer. The on-board computer is equipped with an Intel Core i7-1165G7 processor with 4 cores and 8 threads, used for running SLAM programs, recognizing the illegal drone, and running the EKF trajectory prediction program. The FNTSM-TDE control strategy operates within the PX 4, controlling the drone state and triggering manipulator actions.

The hardware composition schematic is shown in Figure 8. To meet the demands of rapid capture, the manipulator employs a passive triggering mechanism, utilizing elastic elements to store potential energy. When in contact with the illegal drone, the potential energy is converted into kinetic energy, causing the manipulator claws to close and capture the illegal drones. This design avoids the need for electronic devices such as sensors and drivers, and the weight of the manipulator is only 50 g, reducing the complexity and weight of the counter system.

In the actual capture experiment, YOLOv5 technology was employed for target detection. To enhance tracking performance, a dataset comprising 6000 relevant drone images was collected and created. After 2500 iterations, the average Intersection over Union (IOU) on the test set reached 80%. To ensure accuracy and real-time capability, the video stream captured by the camera had a resolution of 1920 × 1080 pixels and a frame rate of 60 FPS. The overall system workflow is illustrated in Figure 9.

The flight experiment code is identical to the HITL, with the main difference lying in the source of sensor data. In the flight experiment, sensor signals come from the real world, whereas HITL receives sensor signals from the Gazebo 3D dynamic simulation software. In the experiment, the drone initially operates in the position mode, moving to a specified location. The counter drone possesses autonomous search capabilities, rotating around the Z-axis of its body to locate the target and bring the illegal drone into the camera’s field of view. Three capture experiments are conducted with the illegal drone flying at speeds of 1 m/s, 2 m/s, and 4 m/s, respectively. At the beginning of each experiment, the two drones are 8 m apart. Once the counter drone identifies the illegal drone, it autonomously predicts the future trajectory, adjusts its own state in real time, and tracks and captures the target. The capture process from the camera’s perspective is illustrated in Figure 10, where the YOLOv5 algorithm interprets the position of the illegal drone in the camera plane, and the EKF estimates its future trajectory. The four sub images represent the images and recognition effects captured by the drone at different positions and attitude angles, respectively. The pixel error curve during the capture process is shown in Figure 11. Due to the EKF algorithm predicting the trajectory of the illegal drone in advance, the tracking error rapidly converges to zero. In experiments without the EKF algorithm, the convergence time of the tracking error increased by a factor of 2.

The eight subplots in Figure 12 depict capture scenes of the counter drone, demonstrating its autonomous tracking of the illegal drone and the automatic triggering and capture of the illegal drone by the manipulator. Once the manipulator makes contact with the illegal drone, the elastic elements are automatically triggered, and the claws of the manipulator capture the illegal drone.

4. Conclusions

This paper presents an airspace management strategy based on IBVS, utilizing a counter drone equipped with a robotic arm to capture illegal drones. The strategy achieves triple functionality: utilizing a neural network model based on the YOLO-v5 architecture for the visual recognition of an illegal drone, employing EKF for the advanced prediction of illegal drone flight trajectories to enhance capture success rates, utilizing a lightweight passive trigger mechanical hand weighing only 50 g for drone capture to reduce the weight and complexity of the capture system, and proposing the FNTSM-TDE control strategy to suppress the interference of rogue drones with counter UAVs during capture moments. This strategy features advantages of model-free control and finite-time convergence. The effectiveness of the proposed control strategy is validated through HITL simulation experiments and real flight tests. The EKF algorithm is employed to predict rogue drone trajectories in advance, resulting in the accelerated convergence of tracking errors to zero. Compared to capture strategies without EKF configuration, the convergence time is halved, thus enhancing capture success rates. Future efforts will focus on conducting capture experiments in more diverse scenarios to further enhance the robustness and practicality of the system.

5. Patents

The key technology disclosed in the paper has been applied for an invention patent in China, named an unpowered adaptive fast linkage manipulator and a drone capture device, patent No. 202311365289.2.