Real-Time Visual Tracking of Moving Targets Using a Low-Cost Unmanned Aerial Vehicle with a 3-Axis Stabilized Gimbal System

Featured Application

A complete solution for visual detection and autonomous tracking of a moving target is presented, which is applied to low-cost aerial vehicles in reconnaissance, surveillance, and target acquisition (RSTA) tasks.

Abstract

Unmanned Aerial Vehicles (UAVs) have recently shown great performance collecting visual data through autonomous exploration and mapping, which are widely used in reconnaissance, surveillance, and target acquisition (RSTA) applications. In this paper, we present an onboard vision-based system for low-cost UAVs to autonomously track a moving target. Real-time visual tracking is achieved by using an object detection algorithm based on the Kernelized Correlation Filter (KCF) tracker. A 3-axis gimbaled camera with separate Inertial Measurement Unit (IMU) is used to aim at the selected target during flights. The flight control algorithm for tracking tasks is implemented on a customized quadrotor equipped with an onboard computer and a microcontroller. The proposed system is experimentally validated by successfully chasing a ground and aerial target in an outdoor environment, which has proven its reliability and efficiency.

1. Introduction

The past decade has witnessed an explosive growth in the utilization of unmanned aerial vehicles (UAVs), attracting more and more attention from research institutions around the world [1,2]. With a series of significant advances in technology domains like micro-electro-mechanical system (MEMS), many UAV platforms and mission-oriented sensors limited to military affairs in the past are now widely applied to industrial and commercial sectors [3,4,5,6,7]. UAV-based target tracking is one of the most challenging tasks, which is closely related to applications such as traffic monitoring, reconnaissance, surveillance, and target acquisition (RSTA), search and rescue (SAR), inspection of power cables, etc. [8,9,10,11,12,13]. The tracking system is an important part of UAVs which detects a target of interest rapidly in a large area, and then performs continuous surveillance of the selected target in the tracking phase [14]. In order to achieve target acquisition and localization, military UAVs are usually equipped with airborne radars [15] or guided seekers [16]; however, they are too heavy and unaffordable for most civilian used UAVs. As one of the most popular UAV platforms, quadrotors are more stable and have a lower manufacturing cost than helicopters. For portability and flexibility, the takeoff weight of most quadrotors is less than 15 kg, so payload and battery endurance for onboard equipment are very limited. In this situation, precisely and robustly tracking a moving target by using a small-scale UAV platform is still a challenging task because the onboard computational capability is poor and sensors are low-cost [17].

Compared with many other detection sensors, cameras seem to have an inherent potential for UAV-based target tracking tasks, for they passively receive the environmental information and visual features, while still being low-cost and lightweight [18]. In recent years, various methods are studied in many research works to achieve target tracking by using cameras. Wenzel et al. [19] presented a visual tracking system for autonomous landing of a Hummingbird quadrotor by using an infrared (IR) camera that was extremely cheap. In [20], a vision-based landing control algorithm for an autonomous helicopter was designed and implemented by using a downward-pointing charge coupled device (CCD) camera. The onboard system was integrated with an algorithm for visual acquisition of the target (a moving helipad) and state estimation, which calculated the six degrees of freedom (DOF) pose with respect to the landing pad. The helipad which consists of a letter “H” surrounded by a circle is a typical structured pattern widely used in vision-based tracking and landing tasks because it can be easily detected and recognized in cluttered environments [21]. The research on square-based fiducial markers, such as AprilTag [22] and ArUco [23], has also aroused increasing interests toward marker-based visual tracking. Ref. [24] presented a fully autonomous flight control system for tasks of target recognition, geo-location, following and finally landing on a AprilTag marker which is attached to a high-speed moving car. Ref. [25] proposed a vision-based swarming approach for employing three or more Parrot Ar.Drone 2.0 quadrotors without any extra positioning sensors, while the ArUco markers fixed on the obstacles were used for localization and mapping. However, artificial markers or cooperative targets can only be applied to some specific scenarios such as autonomous landing and indoor navigation, in which the actual size and main features of the pattern are already known [26]. Since the vision system may be requested to track an arbitrary target with unknown size, shape, and motion, it is impossible to have already acquired information and know how to recognize it.

Object tracking is one of the most fundamental fields in computer vision and has a wide range of applications in many areas. For this task, many algorithms for visual tracking have been proposed, which can be generally categorized into generative and discriminative methods according to their appearance models. Generative methods typically search for the image region which is most similar to the target, such as incremental subspace tracker (IVT [27]), L1 tracker [28], real-time compressive sensing tracking (RTCST [29]), superpixel tracking [30], mean shift algorithm [31,32,33], and the continuously adaptive mean shift (CAMSHIFT) algorithm [34]. Discriminative methods or tracking-by-detection methods deal with the tracking problem as a binary classification task and separate the target object from the background, such as multiple instance learning (MIL) tracker [35], Struck [36], on-line AdaBoost (OAB) tracker [37], and Tracking-Learning-Detecting (TLD) [38]. The potential of correlation filters for visual tracking has aroused tremendous research interest because it reduces the overhead time through fast Fourier transformation (FFT) [39]. Bolme et al. [40] presented a minimum output sum of squared error (MOSSE) filter, which produced stable correlation filters when initialized using a single frame. It is able to quickly adapt to variations in scale, rotation, and lighting while operating at 669 frames per second. Henriques et al. [41] proposed the correlation filter of the circulant structure with kernel (CSK), and used the kernel trick to learn the appearance model. After that, the Kernelized Correlation Filter (KCF) tracker that Henriques proposed in [42] further improved the CSK method by using the histogram of oriented gradients (HOG) feature instead of gray feature to represent the object, which showed an amazing speed on the OTB2013 dataset [43]. However, the original KCF tracker uses a fixed size template and is not able to handle scale changes and occlusions, leading to bad performance in some scenarios. To alleviate theses drawbacks, Montero et al. [44] proposed fast scalable solution based on KCF framework which used an adjustable Gaussian window function and a keypoint-based model for scale estimation to deal with the fixed size limitation in the Kernelized Correlation Filter. Zhang et al. [45] presented an improved KCF tracker by adopting a cascade classifier composed of multi-scale correlation filter and NN (Neighbor Nearest) classifier, which showed favorable performance in accuracy and robustness.

To actually track a moving target in real time with a vision-based system onboard a multi-rotor UAV platform, three steps are required. The first step is to detect the target and localize it in each image frame as described above. The second step involves getting the line-of-sight (LOS) angle or the position of the target and maintaining the target in the center of the camera’s field of view (FOV). The third step is to use this information obtained by the vision system to define the control task of an autonomous UAV when flying around the target. Generally, a downward-facing strap-down camera has advantages with its small size, light weight, and simplicity [46]. However, the output of the strap-down camera cannot be directly used in the flight control system, for it couples with the UAV body angular motion. In addition, the restricted FOV of the camera makes it a challenging task to keep the fast-moving target in the image, which also means the FOV constraints must be fully considered in the guidance law [47]. To solve these problems, a gimbal system is widely used to provide inertial stability to the camera by isolating it from the UAV motion and vibration. A gimbaled camera now available on the market provides a decoupling along the roll, pitch, and yaw attitudes, which has become an important unit of the UAV system. In [48], a 3-axis gimbaled camera (DJI Zenmuse X3) is used to solve the problem, which enabled autonomous landing of a quadrotor on a high-speed ground vehicle. The system is experimentally implemented and validated by successfully landing a commercial quadrotor on the roof of a car moving at speeds of up to 50 km/h. Ref. [49] presented a field-tested mini gimbal mechanism to produce an estimation of the target position, which allowed a flying-wing UAV to fly around the target. Jakobsen et al. [50] presented the architecture of a pan/tilt/roll camera system implemented on the Georgia Tech’s UAV research helicopter. Each axis is driven by a servo and optical encoders are utilized to measure the gimbal orientation. Whitacre et al. [51] performed flight tests using a SeaScan UAV with a gimballing camera to track ground targets and studied the effect of altitude, camera FOV, and orbit center offsets within the geo-location tracking performance for both stationary and moving targets.

In this paper, motivated by the existing challenges, an onboard vision-based system for tracking arbitrary 3D objects moving at unknown velocities is proposed by utilizing a 3-axis gimbal system. A KCF tracker is used to detect and localize the target of interest from images acquired by the gimbaled camera. The 3-axis stabilized gimbal system is driven by brushless direct current motors (BLDCMs) and a magnetic rotary encoder is attached to each axis with a high-resolution output of the angular position. Then, the results of the KCF tracker are put into a proportional-derivative (PD) controller, which aligns the optical axis of the gimbaled camera with the LOS joining the camera and the target. A tracking strategy for multi-rotor UAVs is proposed based on the proportional navigation (PN) method, which makes it possible to keep following the target, although the target position is not known. A low-cost experimental quadrotor is customized for real-time flight tests, which is equipped with a microcontroller using consumer sensors and an onboard computer for image processing. The presented system is experimentally implemented and validated by successfully tracking a commercial quadrotor flying along an unknown path. By analyzing the flight data of both the target and the interceptor, the system is demonstrated to be reliable and cost-effective.

The rest of this paper is organized as follows. The vision system and control of the gimbaled camera are described in Section 2. The visual tracking algorithm used to detect and localize the target in image stream is presented in Section 3. Section 4 established the dynamic model of the quadrotor, and the target tracking strategy as well as the control law is presented. Section 5 describes preparation for flight experiments, and the results of the flight tests are given in this part. Finally, Section 6 concludes this paper and presents the future work.

2. Gimbal System

2.1. Problem Formulation and System Architecture

Although there are many algorithms capable of tracking targets from the video streams, techniques reported in the field of computer vision cannot be easily extended to airborne applications because of high dynamic UAV-target relative motion. In this section, a small commercial drone (41 cm in diameter) is considered as the target with unknown speed. If this unwanted aerial visitor flies into places such as airports, prisons, and military bases where consumer drones are not allowed, it can cause a big problem. Therefore, a lot of anti-UAV defense systems are being introduced to combat the growing threats of malicious UAVs. One way is to use a rifle-like device which sends a high-power electromagnetic wave to jam the UAV control systems and force them to land immediately. Another option is to capture the target in mid-air, using a UAV platform that carries a net gun.

Consider Figure 1, which depicts the UAV-Target relative kinematics and defines the coordinate systems. Let $B$ denote the body frame that moves with the UAV, $C$ the camera frame that is attached to the UAV but rotates with respect to frame $B$, $N$ a North-East-Down (NED) coordinate system taken as an inertial reference frame, $I$, the image frame. The origin of the camera frame $C$ is the optical center and $Z_c$ coincides with the optical axis of the camera. Following the notation introduced in [52], let $p_C={\left[x_c,y_c,z_c\right]}^T$ denote the position of the target in frame $C$. The rotation transformation ${}_c{}^{n}R$ from frame $C$ to frame $N$ is:

$${}_c{}^{n}R={}_b{}^{n}R{}_c{}^{b}R$$

(1)

where the transformation ${}_b{}^{n}R$ is calculated by the roll, pitch, and yaw angles of the UAV given by the flight controller. ${}_c{}^{b}R$ can be computed by the gimbal system using the relative angular position measured by the encoders. Let $p$ be the position of the target with respect to the optical center of camera resolved in $N$, which is given as:

$$p={}_b{}^{n}R{}_c{}^{b}R\cdot p_C$$

(2)

Thus, the target position $p_T$ in NED coordinate system can be estimated, which is given as:

$$p_T=p_B+{}_b{}^{n}R{p}_{BC}+p$$

(3)

where $p_{BC}$ is the position of gimbaled camera relative to the UAV body in frame $B$, and $p_B$ is the UAV position in $N$.

To deal with the task of moving target tracking, an autonomous quadrotor UAV system equipped with a 3-axis gimbaled camera is constructed to detect and follow the flying object in the pursuit-evasion scenario. The visual tracking system processes the images and drives the gimbal to search the target areas. Once an intruding drone is detected, the location of the target in each image frame is acquired, and this is utilized by the automated targeting module for aiming control of the gimbal. While the target is locked down, the camera pose can be used as input information to control the UAV flight. Figure 2 presents the proposed vision-based system.

For most civil UAV applications, the gimbal system and the UAV flight control system are independent of each other. However, to achieve autonomous tracking of a moving target, the two systems are coordinated by the proposed vision algorithm which is implemented in the onboard computer running a Linux based system.

2.2. Kinematics of the Gimbal System

While the FOV of a single camera is limited, the gimbal systems are able to rotate the camera to a desired direction, which are widely applied to many fields such as filming and monitoring. When these systems are mounted onboard an UAV, the torque motors are activated by the IMUs and other angular sensors to compensate for all the rotations resulting from the UAV flight, which returns the stable member to its original attitude. As shown in Figure 3 and Figure 4, the gimbal system which is used in this paper consists of direct current (DC) motors that balance the platform, magnetic rotary encoders that sense the relative rotation, embedded stabilization controller that process all the sensors information and output the control signals, the vibration damper that connects the outer gimbal to the UAV body, and the camera that captures the images.

The 3-axis gimbaled camera supporting structure consists of the case, outer frame, middle frame and inner frame as depicted in Figure 5. The kinematic relations are set as a yaw, roll, pitch sequence and four reference frames are introduced: the body-fixed frame $F$, the outer frame $O$, the middle frame $M$, the inner frame $G$ connected by three revolute joints. Considering the common structure of gimbals in [53,54,55], relative angles are defined as yaw (${\theta }_Y$), roll (${\theta }_R$) and pitch (${\theta }_P$). The frame $F$ is carried into frame $O$ by rotation ${\theta }_Y$ around the axis $z_F$. Frame $O$ is carried into frame $M$ by rotation ${\theta }_R$ around the axis $x_O$. Finally, frame $M$ is carried into frame $G$ by rotation ${\theta }_P$ around the axis $y_M$. The coordinate systems of the gimbal (Figure 5) are placed parallel to each other as the initial state in the configuration (${\theta }_Y$,${\theta }_R$,${\theta }_P$) = (0, 0, 0).

The coordinate of an arbitrary point $P$ in frame $G$ denoted as the vector ${}^{G}r_P$ can be described in a different coordinate frame $F$ using the rotation matrix ${}^{F}R_G$ and the translation vector ${}^{F}d_G$ between the frames according to the above relationship:

$${}^{B}r_P={}^{F}R_G{}^{G}r_P+{}^{F}d_G$$

(4)

where, in a 3D environment,

$${}^{G}r_P=\left[\begin{array}{c}{}^{G}x_P\\ {}^{G}y_P\\ {}^{G}z_P\end{array}\right]$$

(5)

A more convenient way to describe such transformation is to use homogeneous transformation matrices ${}^{F}T_G$ given as:

$${}^{F}T_G=\left[\begin{array}{cc}{}^{F}R_G& {}^{F}d_G\\ 0_{1\times 3}& 1\end{array}\right]$$

(6)

Several intermediate transformations are required to get the final transformation in Equation (4) given as

$$\left[\begin{array}{c}{}^{B}r_P\\ 1\end{array}\right]={}^{F}T_G\left[\begin{array}{c}{}^{G}r_P\\ 1\end{array}\right]$$

(7)

where

$${}^{F}T_G={}^{F}T_O{}^{O}T_M{}^{M}T_G$$

(8)

With the parameters $l_1$, $l_2$, $h_1$, $h_3$ and $b_2$ in Figure 5, the transformations between the frames are as follows.

The transformation between the frame $F$ and frame $O$:

$${}^{F}R_O=\left[\begin{array}{ccc}\cos {\theta }_Y& -\sin {\theta }_Y& 0\\ \sin {\theta }_Y& \cos {\theta }_Y& 0\\ 0& 0& 1\end{array}\right]$$

(9)

$${}^{F}d_O=\left[\begin{array}{c}-l_1\cos {\theta }_Y\\ -l_1\sin {\theta }_Y\\ h_1\end{array}\right]$$

(10)

or

$${}^{F}T_O=\left[\begin{array}{cccc}\cos {\theta }_Y& -\sin {\theta }_Y& 0& -l_1\cos {\theta }_Y\\ \sin {\theta }_Y& \cos {\theta }_Y& 0& -l_1\sin {\theta }_Y\\ 0& 0& 1& h_1\\ 0& 0& 0& 1\end{array}\right]$$

(11)

The transformation between the frame $O$ and frame $M$:

$${}^{O}R_M=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos {\theta }_R& -\sin {\theta }_R\\ 0& \sin {\theta }_R& \cos {\theta }_R\end{array}\right]$$

(12)

$${}^{O}d_M=\left[\begin{array}{c}{l}_2\\ -b_2\cos {\theta }_R\\ -b_2\sin {\theta }_R\end{array}\right]$$

(13)

or

$${}^{O}T_M=\left[\begin{array}{cccc}1& 0& 0& l_2\\ 0& \cos {\theta }_R& -\sin {\theta }_R& -b_2\cos {\theta }_R\\ 0& \sin {\theta }_R& \cos {\theta }_R& -b_2\sin {\theta }_R\\ 0& 0& 0& 1\end{array}\right]$$

(14)

The transformation between the frame $M$ and frame $G$:

$${}^{M}R_G=\left[\begin{array}{ccc}\cos {\theta }_P& 0& \sin {\theta }_P\\ 0& 1& 0\\ -\sin {\theta }_P& 0& \cos {\theta }_P\end{array}\right]$$

(15)

$${}^{M}d_G=\left[\begin{array}{c}{h}_3\sin {\theta }_P\\ b_2\\ h_3\cos {\theta }_P\end{array}\right]$$

(16)

or

$${}^{M}T_G=\left[\begin{array}{cccc}\cos {\theta }_P& 0& \sin {\theta }_P& h_3\sin {\theta }_P\\ 0& 1& 0& b_2\\ -\sin {\theta }_P& 0& \cos {\theta }_P& h_3\cos {\theta }_P\\ 0& 0& 0& 1\end{array}\right]$$

(17)

Thus, the total rotation matrix ${}^{F}R_G$ and translation vector ${}^{F}d_G$ between frame $F$ and frame $G$ is:

$${}^{F}R_G=\left[\begin{array}{ccc}\cos {\theta }_Y\cos {\theta }_P-\sin {\theta }_Y\sin {\theta }_R\sin {\theta }_P& -\cos {\theta }_R\sin {\theta }_Y& \cos {\theta }_Y\sin {\theta }_P+\cos {\theta }_P\sin {\theta }_Y\sin {\theta }_R\\ \cos {\theta }_P\sin {\theta }_Y+\cos {\theta }_Y\sin {\theta }_R\sin {\theta }_P& \cos {\theta }_Y\cos {\theta }_R& \sin {\theta }_Y\sin {\theta }_P-\cos {\theta }_Y\cos {\theta }_P\sin {\theta }_R\\ -\cos {\theta }_R\sin {\theta }_P& \sin {\theta }_R& \cos {\theta }_R\cos {\theta }_P\end{array}\right]$$

(18)

$$\begin{array}{ll}{}^{F}d_G& ={}^{F}R_O{}^{O}R_M{}^{M}d_G+{}^{F}R_O{}^{O}d_M+{}^{O}d_F\hfill \\ & =\left[\begin{array}{c}{l}_2\cos {\theta }_Y-l_1\cos {\theta }_Y+h_3\cos {\theta }_Y\sin {\theta }_P+h_3\cos {\theta }_P\sin {\theta }_Y\sin {\theta }_R\\ l_2\sin {\theta }_Y-l_1\sin {\theta }_Y+h_3\sin {\theta }_Y\sin {\theta }_P-h_3\cos {\theta }_Y\cos {\theta }_P\sin {\theta }_R\\ h_1+h_3\cos {\theta }_R\cos {\theta }_P\end{array}\right]\hfill \end{array}$$

(19)

The ${}^{F}R_G$ is called a pitch-roll-yaw rotation matrix according to the order in which the rotation matrices are successively multiplied. In a similar way, the rotation between frame and inertial reference frame can be described as:

$${}^{N}R_G=\left[\begin{array}{ccc}\cos {\alpha }_Y\cos {\alpha }_P-\sin {\alpha }_Y\sin {\alpha }_R\sin {\alpha }_P& -\cos {\alpha }_R\sin {\alpha }_Y& \cos {\alpha }_Y\sin {\alpha }_P+\cos {\alpha }_P\sin {\alpha }_Y\sin {\alpha }_R\\ \cos {\alpha }_P\sin {\alpha }_Y+\cos {\alpha }_Y\sin {\alpha }_R\sin {\alpha }_P& \cos {\alpha }_Y\cos {\alpha }_R& \sin {\alpha }_Y\sin {\alpha }_P-\cos {\alpha }_Y\cos {\alpha }_P\sin {\alpha }_R\\ -\cos {\alpha }_R\sin {\alpha }_P& \sin {\alpha }_R& \cos {\alpha }_R\cos {\alpha }_P\end{array}\right]$$

(20)

where ${\alpha }_Y$, ${\alpha }_R$, and ${\alpha }_P$ are derived using the information from the gyros and accelerometers on the IMU attached to the camera. The ${}^{N}R_G$ can also be derived using the rotation matrix ${}^{F}R_G$ and ${}^{N}R_F$, as below:

$${}^{N}R_G={}^{N}R_F{}^{F}R_G$$

(21)

where ${}^F\omega _{NF}$, and the angular velocity of frame $F$ respect to frame $N$ introduced in frame $F$ is measured and available given as:

$${}^F\omega _{NF}={\left[\begin{array}{ccc}p& q& r\end{array}\right]}^T$$

(22)

Angular velocities of frame $O$, $M$, and $G$ respect to frame $N$ introduced in its own frames are as follows:

$${}^O\omega _{NO}={}^O\omega _{NF}+{}^O\omega _{FO}={\left({}^{F}R_O\right)}^T{}^F\omega _{NF}+\left[\begin{array}{c}0\\ 0\\ {\dot{\theta }}_Y\end{array}\right]=\left[\begin{array}{c}p\cos {\theta }_Y+q\sin {\theta }_Y\\ -p\sin {\theta }_Y+q\cos {\theta }_Y\\ r+{\dot{\theta }}_Y\end{array}\right]$$

(23)

$${}^M\omega _{NM}={}^M\omega _{NO}+{}^M\omega _{OM}={\left({}^{O}R_M\right)}^T{}^O\omega _{NO}+\left[\begin{array}{c}{\dot{\theta }}_R\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}p\cos {\theta }_Y+q\sin {\theta }_Y+{\dot{\theta }}_R\\ \cos {\theta }_R(-p\sin {\theta }_Y+q\cos {\theta }_Y)+\sin {\theta }_R(r+{\dot{\theta }}_Y)\\ \sin {\theta }_R(p\sin {\theta }_Y-q\cos {\theta }_Y)+\cos {\theta }_R(r+{\dot{\theta }}_Y)\end{array}\right]$$

(24)

$$\begin{array}{ll}{}^G\omega _{NG}& ={}^G\omega _{NM}+{}^G\omega _{MG}={\left({}^{M}R_G\right)}^T{}^M\omega _{NM}+\left[\begin{array}{c}0\\ {\dot{\theta }}_P\\ 0\end{array}\right]\hfill \\ & =\left[\begin{array}{c}\cos {\theta }_P(p\cos {\theta }_Y+q\sin {\theta }_Y+{\dot{\theta }}_R)-\sin {\theta }_P(\sin {\theta }_R(p\sin {\theta }_Y-q\cos {\theta }_Y)+\cos {\theta }_R(r+{\dot{\theta }}_Y))\\ \cos {\theta }_R(-p\sin {\theta }_Y+q\cos {\theta }_Y)+\sin {\theta }_R(r+{\dot{\theta }}_Y)+{\dot{\theta }}_P\\ \sin {\theta }_P(p\cos {\theta }_Y+q\sin {\theta }_Y+{\dot{\theta }}_R)+\cos {\theta }_P(\sin {\theta }_R(p\sin {\theta }_Y-q\cos {\theta }_Y)+\cos {\theta }_R(r+{\dot{\theta }}_Y))\end{array}\right]\hfill \end{array}$$

(25)

The inertia matrices of the outer gimbal, middle gimbal, and inner gimbal are:

$$J_O=diag\left\{\begin{array}{ccc}{J}_{Ox}& J_{Oy}& J_{Oz}\end{array}\right\}$$

(26)

$$J_M=diag\left\{\begin{array}{ccc}{J}_{Mx}& J_{My}& J_{Mz}\end{array}\right\}$$

(27)

$$J_G=diag\left\{\begin{array}{ccc}{J}_{Gx}& J_{Gy}& J_{Gz}\end{array}\right\}$$

(28)

where $J_{kx}$, $J_{ky}$, $J_{kz}\begin{array}{c}\end{array}(k=O,M,G)$ refer to the diagonal elements. For simplicity, it is assumed that the off-diagonal elements of inertia matrices can be neglected and only the moments of inertia are considered. The angular momentum of the pitch gimbal is:

$$H_G=J_G{}^G\omega _{NG}$$

(29)

the roll gimbal is:

$$H_M=J_M{}^M\omega _{NM}+{}^{M}R_G{J}_G{}^G\omega _{NG}$$

(30)

the yaw gimbal is:

$$H_O=J_O{}^O\omega _{NO}+{}^{O}R_M(J_N{}^M\omega _{NM}+{}^{M}R_G{J}_G{}^G\omega _{NG})$$

(31)

and each member of the gimbal system is treated as a rigid body and the moment equation can be written as:

$${\tau }_k=\frac{d{H}_k}{dt}+{}^k\omega _{Nk}\times H_k\begin{array}{c}\end{array}(k=O,M,G)$$

(32)

where external torques ${\tau }_O$, ${\tau }_M$ and ${\tau }_G$, about $z_F$, $x_O$ and $y_M$, respectively, are applied to gimbals from motor and other external disturbance torques.

2.3. Stabilization and Aiming

The camera fitted on the innermost frame is inertially stabilized and controlled by the gimbal system. Furthermore, the system is required to align its optical axis in elevation and azimuth with a LOS joining the camera and target. Figure 6 describes the angular geometry of how the gimbaled camera aims at the target, where $\theta$ is the pitch angle of the UAV body, $\delta$ is the boresight angle, $\lambda$ is the LOS angle, ${\theta }_P$ is the pitch angle of the gimbal frame, and $\epsilon$ is the boresight error angle.

There are four operation modes, namely, the preset angle mode, search mode, stabilize mode, and tracking mode as shown in Figure 7. When the gimbal system is powered on and initialized, its direction is kept at $\delta =0$ in inertial space. In preset angle mode, the optical axis of the camera will be set to a given angle and the control system will maintain the desired direction despite disturbances. Then, the system may switch to search mode, in which the gimbal will rotate circularly between its minimum and maximum angle to search a larger range of area.

When a target is confirmed, the control system will switch to the tracking mode and keep the target in the center of the camera view. In addition, if the target is lost and cannot be recaptured in a few seconds, the system will return to the search mode and try to find it again. Figure 8 shows how the control system works which contains two loops: tracking loop and stabilizing loop.

Based on the image information and measurement data received from angular sensors, the tracking loop generates a rate command to direct the boresight towards the target LOS so that the pointing error can be kept near zero. On the other hand, the stabilizing loop isolates the camera from UAV motion and external disturbances, which would perturb the aim-point. The control loops in roll, elevation, and azimuth channels are related by the cross coupling unit based on the gimbal system dynamics, which may be defined as the impact on one axis with the rotation of another [56].

3. Target Tracking

3.1. KCF Tracker

The KCF tracker [42,57] that is used in this paper considers the process of sample training as a Ridge Regression problem, which is also a regular minimization problem with a closed solution. Consider a $n\times 1$ vector $x={\left[\begin{array}{cccc}{x}_1& x_2& \cdots & x_n\end{array}\right]}^T$ as the base sample, which represents a patch with the target of interest. A small translation of this vector is given as:

$$Px={\left[\begin{array}{cccc}{x}_n& x_1& \cdots & x_{n-1}\end{array}\right]}^T$$

(33)

where $P$ is the permutation matrix:

$$P=\left[\begin{array}{ccccc}0& 0& 0& \cdots & 1\\ 1& 0& 0& \cdots & 0\\ 0& 1& 0& \cdots & 0\\ ⋮& ⋮& \ddots & \ddots & ⋮\\ 0& 0& \cdots & 1& 0\end{array}\right]$$

(34)

and $u$ shifts can be made to achieve a larger translation by using the matrix power $P^{u}x$. By cyclic shifting operations, we can use these vectors to constitute a circulant matrix as:

$$X=C(x)=\left[\begin{array}{c}{(P^{0}x)}^T\\ {(P^{1}x)}^T\\ {(P^{2}x)}^T\\ ⋮\\ {(P^{n-1}x)}^T\end{array}\right]=\left[\begin{array}{ccccc}{x}_1& x_2& x_3& \cdots & x_n\\ x_n& x_1& x_2& \cdots & x_{n-1}\\ x_{n-1}& x_n& x_1& \cdots & x_{n-2}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ x_2& x_3& x_4& \cdots & x_1\end{array}\right]$$

(35)

and it is useful that all circulant matrices are diagonalized by the Discrete Fourier Transform (DFT), regardless of the base sample $x$, which can be expressed as:

$$X=Fdiag(\widehat{x})F^H$$

(36)

where $\widehat{x}$ is the DFT of base sample, $\widehat{x}=F(x)$, and $F$ is a constant matrix known as the DFT matrix that does not depend on $x$.

Based on Ridge Regression, the goal of training is to find a function $f(z)$ that minimizes the squared error over samples $x_i$ and their regression targets $y_i$, as shown below:

$$E=\underset{w}{\min }{\displaystyle \sum _i{(f(x_i)-y_i)}^2+\lambda {\Vert w\Vert }^2}$$

(37)

where the regularization parameter $\lambda$ is used to control over fitting, as in the Support Vector Machines (SVM) [57] and $w$ represents the filter coefficients.

Consider a linear regression function $f(z)=w^{T}z$, the minimizer has a closed-form solution given as:

$$w={(X^{T}X+\lambda I)}^{-1}{X}^{T}y$$

(38)

where $X$ is the circulant matrix with one sample per row $x_i$, each element of $y$ is a regression target $y_i$, and $I$ is an identity matrix. By utilizing the diagonalization of the circulant matrices, Equation (38) can be expressed in Fourier domain as:

$$\widehat{w}=\frac{{\widehat{x}}^{*}\odot \widehat{y}}{{\widehat{x}}^{*}\odot \widehat{x}+\lambda }$$

(39)

where $\widehat{w}$, $\widehat{x}$, $\widehat{y}$ are the DFT of $w$, $x$, $y$, respectively, and ${\widehat{x}}^{*}$ is the complex-conjugate. In addition, $w$ can be easily recovered in the spatial domain with Inverse Discrete Fourier Transform (IDFT).

When regression function $f(z)$ is nonlinear, the kernel trick is used to map the inputs of a linear problem to a nonlinear and high-dimensional feature space $\phi (x)$:

$$w={\displaystyle \sum _i{\alpha }_i\phi (x_i)}$$

(40)

Then, the variables under optimization are $\alpha$ instead of $w$. The kernel function $k$ is used to compute the algorithm in terms of dot-products, as shown below:

$${\phi }^T(x)\phi (x^{\prime })=k(x,x^{\prime })$$

(41)

where all the dot-products between samples are stored in a $n\times n$ kernel matrix $K$, with elements:

$$K_{ij}=k(x_i,x_j)$$

(42)

and the regression function $f(z)$ can be expressed as:

$$f(z)=w^{T}z={\displaystyle \sum _{i=1}^n{\alpha }_{i}k(z,x_i)}$$

(43)

The solution of this regression function can be given as:

$$\alpha ={(K+\lambda I)}^{-1}y$$

(44)

where $K$ is the kernel matrix and $\alpha$ is the vector of coefficients ${\alpha }_i$, which express the solution in the dual space. By making $K$ circulant, Equation (44) can be diagonalized as in the linear case, obtaining:

$$\widehat{\alpha }=\frac{\widehat{y}}{{\widehat{k}}^{xx}+\lambda }$$

(45)

where ${\widehat{k}}^{xx}$ is the correlation kernel of $x$ with itself in Fourier domain and $\widehat{\alpha }$, $\widehat{y}$ are the DFT of vector $\alpha$, $y$.

For the kernel matrix $K^z$ between all training samples (cyclic shifts of $x$) and candidate image patches (cyclic shifts of base patch $z$), each element of $K^z$ is given by $k(P^{i-1}z,P^{j-1}x)$. From Equation (43), the regression function can be computed for all candidate patches with:

$$f(z)={(K^z)}^T\alpha$$

(46)

where $f(z)$ is a vector, containing the output of all cyclic shift of $z$, which is the full detection response. The position where the output response takes the maximum value is the position of the target in a new frame. To compute Equation (46) efficiently, it is diagonalized as shown below:

$$\widehat{f}(z)={\widehat{k}}^{xz}\odot \widehat{\alpha }$$

(47)

where a hat $\wedge$ denotes the DFT of a vector. In this paper, given the nonlinear Gaussian kernel $k(x,x^{\prime })=\exp (-\frac{1}{{\sigma }^2}{\Vert x-x^{,}\Vert }^2)$, we get:

$$k^{x{x}^{,}}=\exp \left(-\frac{1}{{\sigma }^2}\left({\Vert x\Vert }^2+{\Vert x^{,}\Vert }^2-2F^{-1}\left({\widehat{x}}^{*}\odot {\widehat{x}}^{\prime }\right)\right)\right)$$

(48)

where the kernel correlation can be computed by using a few DFT/IDFT and element-wise operations in $O(n\log n)$ time. Henriques et al. [42] proved that the conversion from inverse operation of matrix in spatial domain to matrix multiplication in Fourier domain would greatly reduce the computational complexity and shorten the computation time.

In the tracking process, considering the target variations, such as illumination, scale, occlusion, and deformation factors, the target apparent model and coefficient vector are updated after each frame [58], as shown below:

$$x_t=(1-{\eta }_t)x_{t-1}+{\eta }_{t}x$$

(49)

$${\alpha }_t=(1-{\eta }_t){\alpha }_{t-1}+{\eta }_t\alpha$$

(50)

where $x_{t-1}$, $x_t$ are the target model updated after the $t-1$ and the $t$ frame. ${\alpha }_{t-1}$, ${\alpha }_t$ are the coefficient vector updated after the $t-1$ and the $t$ frame. ${\eta }_t$ is the learning rate.

3.2. Target Localization

Consider a monocular camera model as shown in Figure 9.

When an arbitrary point $p(x_w,y_w,z_w)$ in world frame $W$ is detected by the camera, its 2D position $p_i(x_i,y_i)$ on the image plane can be given as:

$$x_i=\frac{f{x}_c}{z_c}=(u-u_0)dx$$

(51)

$$y_i=\frac{f{y}_c}{z_c}=(v-v_0)dy$$

(52)

where $(x_c,y_c,z_c)$ is the position of p in the camera frame, $f$ is the focal length, $(u,v)$ is the target position in pixel values, $(u_0,v_0)$ is the intersection of the optical axis, and the image plane, $dx$ and $dy$ are the physical length per pixel in the $x_i$ and $y_i$ axis directions. Equations (51) and (52) can be integrated as:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=M_{in}\left[\begin{array}{c}{x}_c/z_c\\ y_c/z_c\\ 1\end{array}\right]=M_{in}\left[\begin{array}{c}{x}_{1c}\\ y_{1c}\\ 1\end{array}\right]$$

(53)

where ${[x_{1c},y_{1c},1]}^T$ is the point on the normalized plane and $M_{in}$ is the intrinsic parameter matrix of the camera, which is given as:

$$M_{in}=\left[\begin{array}{ccc}f/dx& 0& u_0\\ 0& f/dy& v_0\\ 0& 0& 1\end{array}\right]$$

(54)

With a calibrated camera, the pointing error angle $\epsilon$ in elevation and azimuth can be computed by using the above relations, as shown below:

$${\epsilon }_{\chi }=\arctan (x_c/z_c)$$

(55)

$${\epsilon }_{\gamma }=\arctan (y_c/z_c)$$

(56)

where ${\epsilon }_{\chi }$ and ${\epsilon }_{\gamma }$ are the pointing error or boresight error in azimuth and elevation, respectively, which can be put into the target aiming system as input information to control the gimbal.

A scalable KCF tracker is used in this paper, with the scale changes of the target taken into consideration. The tracker not only updates the centroid position of the target in the image frame, but also outputs the target size in pixel values. This can be used to control the distance between the interceptor and the target while tracking, though the physical size of the target is unknown. The running results of tracking a pedestrian with the proposed tracker are shown in Figure 10. The bounding box changes with adaption to the target variations in the video streams.

4. Flight Control Algorithm

4.1. UAV Dynamic Model

The 6 DOF motion of a rigid quadrotor is described in Figure 11.

Let $m$ denote the mass of the quadrotor and $J$ the moment of inertia. The external forces and torques which act on the quadrotor platform are primarily caused by propellers and gravity. A local NED frame $N$ and body-fixed frame B are introduced to describe the motion of the quadrotor. ${}^{n}p={\left[\begin{array}{ccc}{}^{n}p_x& {}^{n}p_y& {}^{n}p_z\end{array}\right]}^T$ and ${}^{n}v={\left[\begin{array}{ccc}{}^{n}v_x& {}^{n}v_y& {}^{n}v_z\end{array}\right]}^T$ are the position and linear velocity of the quadrotor’s mass center relative to N. Θ = [ϕ θ Ψ]^T is the roll/pitch/yaw angles, which represents the orientation of the quadrotor in N. The rotation matrix $R_b^n$ from B to N is expressed as:

$${}^{N}R_B=\left[\begin{array}{ccc}\cos \theta \cos \psi & \sin \varphi \sin \theta \cos \psi -\cos \varphi \sin \psi & \cos \varphi \sin \theta \cos \psi +\sin \varphi \sin \psi \\ \cos \theta \sin \psi & \sin \varphi \sin \theta \sin \psi +\cos \varphi \cos \psi & \cos \varphi \sin \theta \sin \psi -\sin \varphi \cos \psi \\ -\sin \theta & \sin \varphi \cos \theta & \cos \varphi \cos \theta \end{array}\right]$$

(57)

The equations of motion can be described as:

$${}^n\dot{p}={}^{n}v$$

(58)

$${}^n\dot{v}=g{n}_3-\frac{f_b}{m}{R}_b^n{n}_3$$

(59)

$$J\cdot {}^b\dot{\omega }=-{}^b\omega \times \left(J\cdot {}^b\omega \right)+G_a+\tau$$

(60)

where $f_b$ is the force applied to the quadrotor given in $B$, $\tau$ is the torque, $g$ is the gravitational acceleration and ${}^b\omega$ is the angular velocity of the quadrotor in $B$. The gyroscopic moment $G_a$ is mainly produced by propellers, which can be neglected. In addition, the translational dynamics shown in Equations (58) and (59) can be simplified as:

$${}^n\ddot{p}_x=-\frac{f_b}{m}\left(\sin \varphi \sin \psi +\cos \varphi \sin \theta \cos \psi \right)$$

(61)

$${}^n\ddot{p}_y=-\frac{f_b}{m}\left(-\sin \varphi \cos \psi +\cos \varphi \sin \theta \sin \psi \right)$$

(62)

$${}^n\ddot{p}_z=g-\frac{f_b}{m}\cos \varphi \cos \theta$$

(63)

Furthermore, it can be assumed that $\sin \varphi \approx \varphi$, $\cos \varphi \approx 1$, $\sin \theta \approx \theta$, and $\cos \theta \approx 1$ for small angle approximation, which leads to a simplified dynamic model as described in [59].

4.2. Tracking Strategy

The tracking strategy used in this paper is based on proportional navigation (PN), which is a well-known guidance law and has been widely used to enable a missile to catch its target in optimal time. The constant bearing approach considers that the missile will finally collide with the detected target if the LOS angle is kept constant. The PN method improves the constant bearing approach to accommodate for target maneuver by accelerating the missile in a direction lateral to the LOS with magnitude proportional to the rate change of the LOS angle. There are different types of PN methods according to their different mathematical formulations and their performances have been analyzed in [60], when applied to guidance of a quadrotor.

The desired acceleration ${\ddot{u}}^{des}$ obtained by the PN method can be expressed as:

$${\ddot{u}}^{des}=NL\times \dot{\lambda }$$

(64)

where $\dot{\lambda }$ is rate change of the LOS angle, $N$ is the navigation gain, and $L$ is the normal direction of the acceleration command that is calculated for different ways as follows:

$$L_{RTPN}=u_T-u$$

(65)

$$L_{IPN}={\dot{u}}_T-\dot{u}$$

(66)

$$L_{PPN}=-\dot{u}$$

(67)

$$L_{NGL}=\frac{\left|\dot{u}\right|u}{\left|u_T-u\right|}\cdot \frac{\sin \beta }{\left|\dot{\lambda }\right|}$$

(68)

where $L_{RTPN}$, $L_{IPN}$, $L_{PPN}$, and $L_{NGL}$ represent for Realistic True Proportional Navigation (RTPN) [61], Ideal Proportional Navigation (IPN) [62], Pure Proportional Navigation (PPN) [63], and Nonlinear Guidance Law (NGL) [64], respectively. $u_T,{\dot{u}}_T,{\ddot{u}}_T\in R^2$ are the position, velocity, and acceleration of the target, respectively, $u,\dot{u},\ddot{u}\in R^2$ are the position, velocity, and acceleration of the interceptor, respectively. $|\cdot |$ represents the magnitude of the vector and $\beta$ is the angle between interception velocity and LOS of the target.

In addition, $\dot{\lambda }$ can be described as:

$$\dot{\lambda }=\frac{(u_T-u)\times ({\dot{u}}_T-\dot{u})}{{\left|u_T-u\right|}^2}$$

(69)

The aim of the research in this paper is to track a moving target with a quadrotor platform, and coordinated control of the UAV and gimbaled camera is considered. As shown in Figure 12a,b, it can be done in two directions: the longitudinal direction and the lateral direction [65].

The ${\lambda }_{\chi }$, ${\lambda }_{\gamma }$ are the lateral and longitudinal LOS angle of the target, respectively. The ${\epsilon }_{\chi }$, ${\epsilon }_{\gamma }$ are the boresight error angle in lateral and longitudinal direction, respectively, which are controlled to be zero (${\epsilon }_{\chi },{\epsilon }_{\gamma }\to 0$). The ${\dot{u}}_T^{\chi }$, ${\dot{u}}_T^{\gamma }$ are the velocity of the target in lateral and longitudinal directions, respectively. The ${\dot{u}}^{\chi }$ is the velocity of the UAV in lateral direction that is aligned with the axis $x_b$ of the body frame $B$ and the desired yaw angle of the UAV is ${\psi }^{des}={\lambda }_{\chi }$. A pure pursuit guidance law is used for tracking in the lateral direction, which works on the principle that, if the interceptor persistently points towards the target ($\psi \to {\lambda }_{\chi },\dot{\psi }\to {\dot{\lambda }}_{\chi }$), then it will ultimately intercept it.

When the tracking is initiated, the UAV follows the target by tracking its lateral LOS angle with a forward speed ${\dot{u}}^{\chi }$, which is the horizontal component of the approaching velocity ${\dot{u}}_{App}$. To keep following the target moving at unknown speed, the approaching velocity ${\dot{u}}_{App}$ is decided by the scale changes of the target in the image frame. Then, a PPN guidance law will activate and the acceleration command ${\ddot{u}}_{PPN}^{des}$ will be applied to the UAV according to the rate changes of ${\lambda }_{\gamma }$, as shown in Figure 12b.

4.3. Flight Control System

The quadrotor is a typical underactuated system with only four independent inputs less than the degrees of freedom, so only the desired position and desired yaw angle can be directly tracked. The desired roll angle and the desired pitch angle are determined by the known ones. The flight control system of a quadrotor is described in Figure 13.

The pixel values $u_T,v_T$ are the centroid of the target position in each image frame, and $S_T$ is the scale change of the target, which is given by the proposed KCF tracker. The state of the quadrotor $(\Theta ,\omega )$ expressed in global NED frame is given by the autopilot using an Extended Kalman Filter (EKF). The desired position ${\left[\begin{array}{ccc}{x}_d& y_d& z_d\end{array}\right]}^T$ and the desired yaw angle ${\psi }_d$ are given by the gimbal system in the global NED using the above-mentioned tracking strategy. A cascade proportional-integral-derivative (PID) controller is designed to individually control the 6 DOF motion of the quadrotor. The attitude control loop is implemented on the microcontroller, while the outer loop for position control is implemented on the onboard computer. All PID gains have been preliminarily tuned in hovering flight tests. The outputs of the cascade PID controller are the desired force $f_d$ and the desired torque ${\tau }_d$, which are applied to the UAV body. The mixer gives the desired angular velocity of each motor to the electronic speed controller (ESC), which is expressed as a Pulse Width Modulation (PWM) signal.

It is worth mentioning that the thrust value is not only determined by the desired position, but also by the takeoff weight of the quadrotor platform. Thus, the height control can be considered as two parts: a slightly changed base value for hovering control and a fast controller for position control. When the target is selected, the quadrotor will keep the distance relative to the target based on the estimation of its scale changes in the image frame, which is shown in Supplementary Materials.

5. Experiments and Results

5.1. Experimental Setup

Most experimental UAVs are equipped with expensive sensors and devices, such as high-precision IMUs, 3D light detection, and ranging (Lidar) sensors and differential global positioning system (DGPS), which will definitely improve the control accuracy but are unaffordable in many practical applications. To test the proposed tracking system in this paper, a customized quadrotor platform (65 cm in diameter) is used to perform all the flight experiments, which weighs 4.2 kg including all the payloads, as shown in Figure 14. The cost is much lower than the other platforms (e.g., DJI Matrice 200). The 3-axis gimbal at a dimension of 108 × 86.2 × 137.3 mm³, weighs only 409 g, as shown in Figure 15a. The IMUs and encoders are consumer sensors at very low prices. The camera with focal lengths ranging from 4.9–49 mm has a maximum resolution of 1920 × 1080 at 60 frames per second (fps). The cost of the gimbaled camera is less than $400, which makes it very attractive considering its great performance. The AS5048A magnetic rotary encoder used in the gimbal system measures the angular position of each axis, which has a 14-bit high resolution output (0.0219 deg/LSB).

As shown in Figure 15b, the quadrotor is equipped with an embedded microcontroller developed by the Pixhawk team at ETU Zürich [66]. The selected firmware version is 1.9.2.

To achieve real-time image processing, a NVIDIA Jetson TX2 module is used as an onboard computer to implement the tracking algorithm, which is almost the fastest and most power-efficient embedded AI computing device. The output data of the vision system can be transferred from the onboard computer to the Pixhawk flight controller using serial communication, which is based on a MAVLINK [67] extendable communication node for the Robot Operation System (ROS) [68]. The control rate is at 30 Hz limited by the onboard processing speed. By using a 2.4 GHz remote controller (RC), the tracking process can be initiated by switching to the offboard mode when a target is selected.

5.2. Experimental Results and Analysis

To evaluate the performance of the proposed tracking and targeting system, we test it in different situations. After selecting a target in the video streams, the gimbal system is activated and rotates the camera to point at the selected target, which can be regarded as a step response. The boresight error pixels are plotted in azimuth and elevation, respectively, which are also printed on the top left of the screen.

As shown in Figure 16, the system responds rapidly and the steady-state error is about ±3 pixels, while the initial errors are hundreds of pixels. This test is usually used to tune all the control parameters of the gimbal system, which can be completed on the ground.

Then, the gimbal system and the onboard computer are fixed on the experimental quadrotor platform for further tests. To aim at a target from the UAV, the motion and vibration of the platform should be isolated. Once the target is locked, the UAV is in a fully autonomous mode controlled by the integrated system. Figure 17 shows the results of the boresight error while the UAV is following a pedestrian moving at 0.9–2 m/s. In about 250 seconds’ flight, an accuracy of ±9.34 pixels in the azimuth and ±5.07 pixels in the elevation was achieved.

While tracking a ground moving pedestrian, the altitude change of the target can be ignored in most situations, which makes the task less difficult. However, tracking a flying drone is much more complicated. The drones are able to change its position and velocity in a very short time, which may cause tracking errors or failures. During the flight experiment, autonomous tracking of an intruded drone has been achieved. Figure 18 shows the results of the boresight error while the UAV is tracking a flying drone.

Some oscillations still remain in the current configuration, which occurred when the flight path of the target suddenly changed. It is a great challenge for the system to catch up with the target in such a short time. The deviations caused by the image transmission delay cannot be ignored, which is about 220 milliseconds for the current system. Other factors such as image noises and illumination changes may also have an impact on the tracking accuracy to some extent. The root mean square errors (RMSEs) of boresight errors in drone tracking experiment are listed in

Table 1. Root mean square errors (RMSEs) of boresight errors in different cases.

RMSE (Pixel)	Azimuth	Elevation	2D
Step test	31.57	4.83	31.94
Pedestrian following	9.34	5.07	10.63
Drone tracking	21.50	19.28	28.88

, compared with the other two tests.

Figure 19 shows the process of a successful drone tracking experiment. During the flight, the roll/pitch/yaw angle of the UAV, camera, and gimbal frames are plotted, respectively, in Figure 20a–c. The actual approaching speed of the UAV has tracked the setpoint changes accurately as shown in Figure 20d. The trajectories of the intruded drone and the interceptor are plotted in a local NED frame as shown in Figure 21.

Higher control rate and less image transmission delay would significantly improve the response speed and the accuracy, if better hardware configuration were used. However, considering that all the sensors and onboard devices are low-cost, the performance of the proposed tracking system is very attractive in practical applications. A video of the experiments is available in the Supplementary Materials section (Video S1).

6. Conclusions

In the presented work, we proposed an onboard visual tracking system which consists of a gimbaled camera, an onboard computer for image processing and a microcontroller to control the UAV to approach the moving target. Our system used a KCF-based algorithm to detect and track an arbitrary object in real time, which has proved its efficiency and reliability in experiments. With the visual information, the 3-axis gimbal system autonomously aims at the selected target, which has achieved good performance during real flights. The proposed system has been demonstrated through real-time target tracking experiments, which enabled a low-cost quadrotor to chase a flying drone as shown in the video.

Future work may include using a laser ranging module attached to the camera, which is able to provide an accurate distance of the UAV with respect to the target. Even though this will increase the cost of the system, we look forward to its potential applications such as target geo-location and autonomous landing. Performance improvements could also be achieved by using deep learning-based detection algorithms combined with a large number of sample images. The CMOS sensors used in this paper are low-cost, which could lead to the effect of rolling shutter [69]. If the error is dramatic, compensation should be made to handle this issue.