Vision-Based Target Detection and Tracking for a Miniature Pan-Tilt Inertially Stabilized Platform

Abstract

This paper presents a novel visual servoing sheme for a miniature pan-tilt intertially stabilized platform (ISP). A fully customized ISP can be mounted on a miniature quadcopter to achieve stationary or moving target detection and tracking. The airborne pan-tilt ISP can effectively isolate a disturbing rotational motion of the carrier, ensuring the stabilization of the optical axis of the camera in order to obtain a clear video image. Meanwhile, the ISP guarantees that the target is always on the optical axis of the camera, so as to achieve the target detection and tracking. The vision-based tracking control design adopts a cascaded control structure based on the mathematical model, which can accurately reflect the dynamic characteristics of the ISP. The inner loop of the proposed controller employs a proportional lag compensator to improve the stability of the optical axis, and the outer loop adopts the feedback linearization-based sliding mode control method to achieve the target tracking. Numerical simulations and laboratory experiments demonstrate that the proposed controller can achieve satisfactory tracking performance.

1. Introduction

Commonly used small-scale unmanned aerial vehicles (UAVs) can be classified into fixed-wing aircraft, helicopters and multicopters. Before 2010, fixed-wing aircraft and helicopters took overwhelming dominance both in military and civil fields. However, due to the simple structure, ease of use, high reliability and low cost, the quadcopter as the most popular multicopter has consolidated its dominance in the market of small-scale UAVs in recent years [1]. Quadcopters have the characteristics of hovering, vertical take-off and landing, low-speed at low altitude and multi-attitude fight. Besides the small size and light weight, they have low requirements for take-off and landing sites, suitable for flying in confined spaces and in close proximity to people, such as within urban canyons and even inside buildings. Unfortunately, flying within these environments is very challenging from a navigation perspective, as the Global Positioning System (GPS) signal will be largely degraded or even unavailable due to dropouts and multipath effects [2]. Driven by this actual requirement, vision-based navigation and active ranging sensor-based navigation in GPS-denied environments are formulated to replace traditional GPS and inertial sensor navigation systems [3].

The quadcopter without payloads has no practical value in airborne flight. Only when the quadcopter carries mission payloads, such as a camera or a laser detection and ranging (LADAR) system, can it effectively improve the adaptability to the environment to perform various actual tasks, as mentioned in [4]. In order to achieve superior performance, these payloads need to be installed on a inertially stabilized platform (ISP). Airborne ISP, which takes the advantage of the gyro characteristics to maintain its own stability, is the core device of the navigation system. In order to accomplish the tasks of detecting and tracking ground or low-altitude targets, it is necessary to ensure that the vision sensor can always obtain stable and high-quality target images. Therefore, the vision sensor must be mounted on a highly stabilized platform. The ISP generally has two functions. On the one hand, it ensures the stabilization of the optical axis direction with respect to inertial space by measuring the attitude change of the carrier and reacting to isolate the rotational movement of the carrier. On the other hand, it can quickly respond to the control command, rotating with the desired angular velocity to ensure that the optical axis of the camera points to the desired orientation, so as to achieve the target detection and tracking. The ISP can reduce the effects of external factors on payloads to achieve a stable low-altitude shooting based on the inertial line-of-sight (LOS) stabilization. Recently, due to the continuous miniaturization of the quadcopter, most available commercial ISPs, such as the DJI Zemmuse Z15 camera platform, are no longer suitable for the miniature quadcopter, whose weight is less than 1 Kg. Thus, it is crucial to customize an appropriate ISP for the miniature UAV.

The ISP greatly increases the possibilities of target detection and tracking. In general, it will have a successful behavior regardless of the rotation movement and vibration of the quadcopter or the unexpected movement of the object. Related to this topic, visually stabilized platform technology was studied extensively in the past few decades. Adaptive and fuzzy controllers that decouple the axes for the passive LOS stabilization system were reported in earlier studies [5,6,7]. A rigorous analysis of control problems related to a standard double gimbal system is presented in [8]; however, it is not directly applicable to inertial stabilization. In order to obtain good adaptability for the servo system with a nonlinear property and uncertain factors, various kinds of fuzzy proportional–integral–derivative (PID) control methods have been proposed for the LOS stabilization of two-axis ISP in [9,10,11]. Considering the model uncertainties and carrier vibration or external disturbance, robust and disturbance rejection controllers are presented in [12,13,14,15,16,17]. Survey paper [18] confirms that this topic is still relevant within the engineering community and for defence technological needs. The above-cited works mostly focus on the task of inertial stabilization only. Another survey paper [19] discusses in detail how to extend the inertial rate stabilizing feedback loop to a visual tracking system by suggesting the common cascaded control structure for every rotational degree-of-freedom (DOF). According to this design method, decoupled cascaded controllers for two-axis gimbal are reported in [20,21]. Considering the nonlinear characteristics of camera motion, a feedback linearizaiton-based visual pointing and tracking control scheme for an inertially stabilized double-gimbal airborne camera platform is designed in [22,23]. The proposed scheme is thoroughly simulated and verified by laboratory experiments and compared against the more intuitive decoupled control scheme. In addition, pan-tilt camera platform postion controllers with a single feedback loop for each axis are designed in [24,25].

This paper presents a new vision-based target tracking controller for a customized miniature pan-tilt ISP in the presence of carrier disturbance and target movement. The target can be detected and recognized based on a computer vision algorithm. In order to facilitate the control design, a mathematical model that can accurately reflect the dynamic characteristics of the ISP is established, combining theoretical analysis and system identification. The target tracking controller adopts a two-layer cascaded control method based on the established mathematical model. The parameters of the controller are easier to determine than the model-free controllers that are designed in [9,11]. The proposed controller elaborately combines a lag compensator in the frequency domain with the feedback linearization-based sliding mode control method in the time domain to achieve satisfactory tracking performance. Based on the identified model, the inner loop of the proposed controller employs a proportional lag compensator to ensure the stability of LOS, while achieving satisfactory dynamic and steady-state performance. According to the nonlinear characteristics of camera motion [26], the outer loop adopts the feedback linearization-based sliding mode control method to achieve precise target tracking. The stabilization loop accepts commands from the output of the corresponding position loop. As is well known, sliding mode control can effectively adapt to model uncertainties and suppress external disturbances. When a disturbing rotational movement of the carrier or an unexpected movement of the object exists, the proposed controller will offer a significant improvement over the single feedback loop controllers as employed in [24,25]. A host-based control system (HCS) [27] is built as the avionics architecture of the ISP. The control algorithm is real-time-implemented based on the MATLAB software platform in the ground control station (GCS), so we can easily change the code and debug the program through wireless communication. In order to support the theoretical analysis, numerical simulations and laboratory experiments are also given in this paper. This is the most important extension of [19], which only presents the theoretical analysis of control design. This paper improves the control method and makes some important theoretical extensions of the previous conference paper [28].

This paper is organized as follows. In the next section, the modeling methodology of the ISP is introduced in detail. Section 3 is the main body of this paper, which presents the vision-based target tracking control design for the ISP. Numerical simulations and laboratory experiments are given in Section 4 and Section 5, respectively. The conclusions and future works are presented in Section 6.

2. Mathematical Model

2.1. Coordinate Systems

Because the ISP with a standard double-gimbal configuration can detect any stationary or moving target within a certain range on the ground, the ISP is designed in a simplified 2-DOF form. As shown in Figure 1, the mathematical model of the double-gimbal camera platform involves several coordinate systems, such as the inertial coordinate $O_{\mathrm{n}}{X}_{\mathrm{n}}{Y}_{\mathrm{n}}{Z}_{\mathrm{n}}$, the body coordinate $O{X}_{\mathrm{b}}{Y}_{\mathrm{b}}{Z}_{\mathrm{b}}$, the azimuth frame $O{X}_{\mathrm{a}}{Y}_{\mathrm{a}}{Z}_{\mathrm{a}}$ and the elevation frame $O{X}_{\mathrm{e}}{Y}_{\mathrm{e}}{Z}_{\mathrm{e}}$. The mechanism of the ISP is composed of two gimbals. The outer gimbal, which is fixed on the bottom of the quadrotor UAV by the support shaft $O{Z}_{\mathrm{a}}$, realizes the rotation of the platform around the azimuth axis. The azimuth angle, denoted by ${\theta }_{\mathrm{a}}$, is the angle from the azimuth frame $O{X}_{\mathrm{a}}{Y}_{\mathrm{a}}{Z}_{\mathrm{a}}$ rotating around the $O{Z}_{\mathrm{a}}$ axis to the body frame $O{X}_{\mathrm{b}}{Y}_{\mathrm{b}}{Z}_{\mathrm{b}}$. The inner gimbal, which is fixed on the azimuth gimbal by the support shift $O{Y}_{\mathrm{e}}$, achieves the rotation of the platform around the elevation axis. The elevation angle, denoted by ${\theta }_{\mathrm{e}}$, is the angle from the elevation frame $O{X}_{\mathrm{e}}{Y}_{\mathrm{e}}{Z}_{\mathrm{e}}$ rotating around the $O{Y}_{\mathrm{e}}$ axis to the body frame $O{X}_{\mathrm{b}}{Y}_{\mathrm{b}}{Z}_{\mathrm{b}}$. The support shift $O{Z}_{\mathrm{a}}$ and $O{Y}_{\mathrm{e}}$ are perpendicular to each other. The airborne camera whose optical axis direction is parallel to the $O{X}_{\mathrm{e}}$ axis is mounted on the elevation gimbal.

2.2. Stabilization Principle

When the miniature quadrotor UAV has a disturbing angular velocity ${\mathbf{\omega }}_{\mathrm{b}}={\left[\begin{array}{ccc}{\omega }_{bx}& {\omega }_{by}& {\omega }_{bz}\end{array}\right]}^{\mathrm{T}}$ with respect to the body coordinate, it will be transmitted to the angular velocity ${\mathbf{\omega }}_{\mathrm{e}}^{\mathrm{dis}}={\left[\begin{array}{ccc}{\omega }_{ex}^{\mathrm{dis}}& {\omega }_{ey}^{\mathrm{dis}}& {\omega }_{ez}^{\mathrm{dis}}\end{array}\right]}^{\mathrm{T}}$ with respect to the elevation frame through the mechanism of the platform, resulting in instability of the optical axis. The relationship can be obtained through two Euler rotations as follows:

$$\left[\begin{array}{c}{\omega }_{ex}^{\mathrm{dis}}\\ {\omega }_{ey}^{\mathrm{dis}}\\ {\omega }_{ez}^{\mathrm{dis}}\end{array}\right]=\left[\begin{array}{ccc}{C}_{{\theta }_{\mathrm{a}}}{C}_{{\theta }_{\mathrm{e}}}& S_{{\theta }_{\mathrm{a}}}{C}_{{\theta }_{\mathrm{e}}}& -S_{{\theta }_{\mathrm{e}}}\\ -S_{{\theta }_{\mathrm{a}}}& C_{{\theta }_{\mathrm{a}}}& 0\\ C_{{\theta }_{\mathrm{a}}}{S}_{{\theta }_{\mathrm{e}}}& S_{{\theta }_{\mathrm{a}}}{S}_{{\theta }_{\mathrm{e}}}& C_{{\theta }_{\mathrm{e}}}\end{array}\right]\left[\begin{array}{c}{\omega }_{bx}\\ {\omega }_{by}\\ {\omega }_{bz}\end{array}\right].$$

(1)

Hereafter, the abbreviations $S_{·}$, $C_{·}$ and $T_{·}$ represent the trigonometric functions $cos(·)$, $sin(·)$ and $tan(·)$, respectively.

In order to compensate for the disturbing rotational movement of the carrier, the servo controller needs to output the compensative angular velocity ${\dot{\theta }}_{\mathrm{e}}$ and ${\dot{\theta }}_{\mathrm{a}}$, which are projected to the angular velocity ${\mathbf{\omega }}_{\mathrm{e}}^{\mathrm{com}}={\left[\begin{array}{ccc}{\omega }_{ex}^{\mathrm{com}}& {\omega }_{ey}^{\mathrm{com}}& {\omega }_{ez}^{\mathrm{com}}\end{array}\right]}^{\mathrm{T}}$ with respect to the elevation frame. Thus, they can be expressed in components as follows:

$$\left[\begin{array}{c}{\omega }_{ex}^{\mathrm{com}}\\ {\omega }_{ey}^{\mathrm{com}}\\ {\omega }_{ez}^{\mathrm{com}}\end{array}\right]=\left[\begin{array}{c}{\dot{\theta }}_{\mathrm{a}}{S}_{{\theta }_{\mathrm{e}}}\\ {\dot{\theta }}_{\mathrm{e}}\\ {\dot{\theta }}_{\mathrm{a}}{C}_{{\theta }_{\mathrm{e}}}\end{array}\right].$$

(2)

Taking a superposition of (1) and (2), we can obtain

$$\left\{\begin{array}{c}{\omega }_{ex}={\dot{\theta }}_{\mathrm{a}}{S}_{{\theta }_{\mathrm{e}}}+{\omega }_{bx}{C}_{{\theta }_{\mathrm{a}}}{C}_{{\theta }_{\mathrm{e}}}+{\omega }_{by}{S}_{{\theta }_{\mathrm{a}}}{C}_{{\theta }_{\mathrm{e}}}-{\omega }_{bz}{S}_{{\theta }_{\mathrm{e}}}\hfill \\ {\omega }_{ey}={\dot{\theta }}_{\mathrm{e}}-{\omega }_{bx}{S}_{{\theta }_{\mathrm{a}}}+{\omega }_{by}{C}_{{\theta }_{\mathrm{a}}}\hfill \\ {\omega }_{ez}={\dot{\theta }}_{\mathrm{a}}{C}_{{\theta }_{\mathrm{e}}}+{\omega }_{bx}{C}_{{\theta }_{\mathrm{a}}}{S}_{{\theta }_{\mathrm{e}}}+{\omega }_{by}{S}_{{\theta }_{\mathrm{a}}}{S}_{{\theta }_{\mathrm{e}}}+{\omega }_{bz}{C}_{{\theta }_{\mathrm{e}}}\hfill \end{array}\right.,$$

(3)

where ${\omega }_{ex}$, ${\omega }_{ey}$, ${\omega }_{ez}$ are three angular velocity components with respect to the elevation frame. To maintain the LOS orientation of azimuth axis and elevation axis stabilization, the conditions are required as follows:

$$\left\{\begin{array}{c}{\omega }_{ey}=0\hfill \\ {\omega }_{ez}=0\hfill \end{array}\right..$$

(4)

Substituting (3) into (4), we can obtain the compensative angular velocity equation as

$$\left\{\begin{array}{c}{\dot{\theta }}_{\mathrm{e}}={\omega }_{bx}{S}_{{\theta }_{\mathrm{a}}}-{\omega }_{by}{C}_{{\theta }_{\mathrm{a}}}\hfill \\ {\dot{\theta }}_{\mathrm{a}}=-{\omega }_{bx}{C}_{{\theta }_{\mathrm{a}}}{T}_{{\theta }_{\mathrm{e}}}-{\omega }_{by}{S}_{{\theta }_{\mathrm{a}}}{T}_{{\theta }_{\mathrm{e}}}-{\omega }_{bz}\hfill \end{array}\right..$$

(5)

Substituting (1) into (5), the simplified compensative angular velocity equation can be expressed as follows

$$\left\{\begin{array}{c}{\dot{\theta }}_{\mathrm{e}}=-{\omega }_{ey}^{\mathrm{dis}}\hfill \\ {\dot{\theta }}_{\mathrm{a}}=-\frac{{\omega }_{ez}^{\mathrm{dis}}}{C_{{\theta }_{\mathrm{e}}}}\hfill \end{array}\right..$$

(6)

The integrated three-axis gyroscope mounted on the elevation gimbal measures the angular velocity ${\omega }_{ey}$ and ${\omega }_{ez}$, respectively. The stabilized loop governs servo actuators to rotate with the angular rate ${\dot{\theta }}_{\mathrm{e}}$ and ${\dot{\theta }}_{\mathrm{a}}$ around the elevation axis and azimuth axis, respectively. When (6) is satisfied, the ISP can isolate the disturbing rotational movement of the carrier to achieve the stabilization of the optical axis.

2.3. Dynamic Model of the Gimbal

In order to facilitate the performance analysis of the system, we need to establish a mathematical model that can describe the dynamic performance of the system accurately. Generally, theoretical modeling and experimental modeling are two methods to establish a mathematical model of an electromechanical system. Theoretical modeling is accomplished by analyzing the motion characteristics of the process with known laws. Experimental modeling is realized based on the measurements of the input and output signals containing the dynamic characteristics of the process. Due to the complexity of environmental conditions and the uncertainty of model parameters, there are some differences between the actual physical model and the mathematical model based on theoretical analysis. Furthermore, as an on-the-shelf product, the electromechanical parameters of a servo actuator are difficult to obtain. When the analytical method cannot accurately establish a mathematical model of the system, the experimental method will show its advantages. In this paper, the mathematical model of the system is established by combining theoretical analysis with system identification. Based on the theoretical analysis, the structure and order of the model are determined. Then, we use system identification to estimate the unknown parameters, so as to determine the best-fitting model for the measured system.

According to the operation principle, the electromechanical model of the servo actuator can be equivalent to a DC torque motor. The double gimbals and the camera are added to the shafts of servo actuators as inertial loads. The equivalent circuit diagram of the servo actuator is shown in Figure 2.

As shown in Figure 2, $U_a$ is the armature voltage of the motor, and $I_a$ is the armature current of the motor. $\Omega$ is the angular velocity of the output shaft of the servo motor. $L_a$ is the motor armature inductance, and $R_a$ is the motor armature resistance. $E_b$ is the counter electromotive force induced by $\Omega$. According to the analysis of mechanical and electrical principles [13,29], the block diagram of the gimbal electromechanical model is shown in Figure 3.

Generally, the transfer function of current regulator $W_i\left(s\right)$ in Figure 3 is designed to be a simple proportional–integral (PI) controller, which can be expressed as

$$W_i\left(s\right)=K_p(1+\frac{1}{T_{i}s}),$$

(7)

where $K_p$ presents proportional control gain, and $T_i$ represents the integral time constant. $U_c$ is the normalized control input of the servo motor, and $M_m$ is the electromagnetic torque of the motor. $\theta$ is the rotational angle of the output shaft of the servo motor. $k_{pwm}$ denotes the power amplification factor of the pulse-width modulation (PWM) driver. $k_i$ indicates the calibration coefficient of the current loop. $C_m$ is the motor torque constant, and $C_e$ is the counter electromotive force constant. J denotes the equivalent total moment of inertia on the motor shaft. According to the equivalent transformation rule of the block diagram, the transfer function between the normalized servo input $U_c$ and the gimbal angular velocity $\Omega$ will take the following form:

$$\frac{\Omega \left(s\right)}{U_c\left(s\right)}=\frac{K(T_{z}s+1)}{(T_{\omega }^2{s}^2+2\zeta T_{\omega }s+1)(T_{p}s+1)}.$$

(8)

From (8), we know that the gimbal dynamic model is a third-order under-damped and zero-point system; the expressions of the parameters are given as follows:

$$\left\{\begin{array}{c}K=\frac{R_a}{k_i{C}_e}\hfill \\ \zeta =\frac{T_i(R_a+K_p{k}_{pwm}{k}_i)}{2\sqrt{K_p{k}_{pwm}{k}_i{T}_i{L}_a}}\hfill \\ T_{\omega }=\sqrt{\frac{T_i{L}_a}{K_p{k}_{pwm}{k}_i}}\hfill \\ T_z=T_i\hfill \\ T_p=\frac{J{R}_a}{C_e{C}_m}\hfill \end{array}\right..$$

(9)

In order to estimate the unknown parameters of the linear system, frequency domain system identification is adopted using the MATLAB System Identification App. According to the model structure obtained from the theoretical analysis, the recursive least square method is used to estimate the unknown parameters. Using sinusoidal signals as the excitation signal, the dynamic model of the gimbal is obtained by changing the frequency and amplitude of the input excitation signal. In order to ensure identification accuracy, the input excitation signal with 0.4 to 10 Hz frequency range is applied to identify and validate the model. The number of samples is 210, and the number of iterations is 20. Neglecting the dynamic coupling between the azimuth axis and elevation axis, the transfer function between the normalized servo input of azimuth axis $u_{cx}$ and azimuth angular velocity ${\omega }_{ez}$ can be expressed as

$$G_{\mathrm{a}}\left(s\right)=\frac{0.3352\times (4.541s+1)}{(\frac{s^2}{{10.48}^2}+\frac{0.821s}{10.48}+1)(\frac{s}{4.328}+1)}.$$

(10)

The transfer function between the normalized servo input of the elevation axis $u_{cy}$ and elevation angular velocity ${\omega }_{ey}$ can be expressed as

$$G_{\mathrm{e}}\left(s\right)=\frac{0.2557\times (6.058s+1)}{(\frac{s^2}{{9.285}^2}+\frac{0.9075s}{9.285}+1)(\frac{s}{4.604}+1)}.$$

(11)

In order to validate the correctness of the identified model, the actual measured output signal is compared with the simulated output signal based on the identified model and input excitation signal. As shown in Figure 4 and Figure 5, the actual measured output signal, depicted by solid blue line, and the simulated output signal, depicted by dashed red line, fit well.

2.4. Camera Motion and Interaction Matrix

In order to facilitate the design and implement of a vision-based controller for the task of target pointing and tracking, a camera model must be developed. As shown in Figure 6, the frontal pinhole projection is a commonly used camera model. It is simple and convenient and involves placing the image plane in front of the optical center. The coordinates of the target expressed in the camera frame $O_{\mathrm{c}}{X}_{\mathrm{c}}{Y}_{\mathrm{c}}{Z}_{\mathrm{c}}$ are given by ${\mathit{p}}_{\mathrm{o}}={\left[\begin{array}{ccc}x& y& z\end{array}\right]}^{\mathrm{T}}$. The image coordinate frame $O_{\mathrm{i}}{X}_{\mathrm{i}}{Y}_{\mathrm{i}}$ is orthogonal to the optical axis, located at a focal distance $\lambda$ from the origin of the camera coordinate frame. The coordinates of the intersection point of the line connecting the object with the origin are ${\mathit{p}}_{\mathrm{i}}={\left[\begin{array}{ccc}u& w& \lambda \end{array}\right]}^{\mathrm{T}}$ expressed in the camera frame. Correspondingly, the vector $\mathit{s}={\left[\begin{array}{cc}u& w\end{array}\right]}^{\mathrm{T}}$ gives the image coordinates. According to the geometrical relationship of the pinhole projection model, we have that

$$\left\{\begin{array}{c}u=\lambda \frac{x}{z}\hfill \\ w=\lambda \frac{y}{z}\hfill \end{array}\right..$$

(12)

In general, the movement of the camera in the inertial space is characterized by its translational velocity ${\mathit{v}}_{\mathrm{c}}={\left[\begin{array}{ccc}{v}_{cx}& v_{cy}& v_{cz}\end{array}\right]}^{\mathrm{T}}$ and rotational velocity ${\mathbf{\omega }}_{\mathrm{c}}={\left[\begin{array}{ccc}{\omega }_{cz}& {\omega }_{cy}& {\omega }_{cz}\end{array}\right]}^{\mathrm{T}}$, both expressed in the camera frame. Stacking them together, a time-dependent vector $\mathbf{\xi }\left(t\right)={\left[\begin{array}{cc}{\mathit{v}}_{\mathrm{c}}\left(t\right)& {\mathbf{\omega }}_{\mathrm{c}}\left(t\right)\end{array}\right]}^{\mathrm{T}}\in {\mathbb{R}}^6$ is formed. We consider the simple case of a single-point feature and assume that the ground object does not move. Extension of the results stated here to the case of a moving ground target is feasible, but the resulting interaction matrix will be a function of the velocity of the ground object. The motion of the object as viewed by the so-called image feature velocity $\dot{\mathit{s}}\left(t\right)$ can be obtained as a time derivative of the image feature vector $\mathit{s}\left(t\right)$. It is possible to relate the velocity of the camera $\mathbf{\xi }$ to the velocity of the image of the point $\dot{\mathit{s}}$ by an interaction matrix transform. This matrix transform is derived in detail in [26,30] as

$$\left[\begin{array}{c}\dot{u}\\ \dot{w}\end{array}\right]=\left[\begin{array}{cccccc}-\frac{\lambda }{z}& 0& \frac{u}{z}& \frac{uw}{\lambda }& -\frac{{\lambda }^2+u^2}{\lambda }& w\\ 0& -\frac{\lambda }{z}& \frac{w}{z}& \frac{{\lambda }^2+w^2}{\lambda }& -\frac{uw}{\lambda }& -u\end{array}\right]\left[\begin{array}{c}{v}_{cx}\\ v_{cy}\\ v_{cz}\\ {\omega }_{cx}\\ {\omega }_{cy}\\ {\omega }_{cz}\end{array}\right].$$

(13)

Note that the interaction matrix is a function of the image coordinates of point ${\mathit{p}}_{\mathrm{o}}$ and the depth of the point with respect to the camera frame. The focal length $\lambda$ is regarded as a fixed parameter. Therefore, this equation is typically written as

$$\dot{\mathit{s}}\left(t\right)=\mathit{L}(u,w,z)\mathbf{\xi }\left(t\right).$$

(14)

It is useful to decouple and rewrite (14) as a composition of two parts:

$$\dot{\mathit{s}}\left(t\right)={\mathit{L}}_v(u,w,z){\mathit{v}}_{\mathrm{c}}\left(t\right)+{\mathit{L}}_{\omega }(u,w){\mathbf{\omega }}_{\mathrm{c}}\left(t\right).$$

(15)

In (15), the part corresponding to the translation of the camera frame ${\mathit{L}}_v(u,w,z)$ is a function of both the image coordinates of the point and its depth, while the part corresponding to the rotation of the camera frame ${\mathit{L}}_{\omega }(u,w)$ is a function of only the image coordinates of the point. This can be beneficial in real-world situations when the exact value of z may not be known. In such a case, errors in the value of z merely cause a scaling of the matrix ${\mathit{L}}_v(u,w,z)$, and this kind of scaling effect can be compensated by using fairly simple control methods.

The relationship between the inertial angular rate vector ${\omega }_c$ with respect to the camera frame and the angular rate vector ${\mathbf{\omega }}_e$ with respect to the elevation frame can be expressed using a constant rotation matrix ${\mathit{R}}_C^E$:

$${\mathbf{\omega }}_{\mathrm{e}}={\mathit{R}}_C^E{\mathbf{\omega }}_{\mathrm{c}}=\left[\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\\ 0& 1& 0\end{array}\right]{\mathbf{\omega }}_{\mathrm{c}}.$$

(16)

3. Control Design

In practical application, the ISP is inevitably subject to a variety of carrier disturbances and external interferences. These disturbances will lead to a serious decline in the stabilization and accuracy of the system, meaning that the target cannot be tracked. Therefore, it is necessary to design a reliable servo controller to overcome the interference and improve the system performance. The superior control design for the ISP can effectively isolate the attitude change of the quadrotor UAV during the flight in order to ensure the stabilization of the optical axis, so that the camera can obtain a stable video image. Meanwhile, the control system steers the gimbal to move the target into the center of the field of view, so as to achieve stationary or moving target detection and tracking. In the following, the visual servoing scheme will be introduced in detail.

3.1. Control System Structure

The visual servoing control design adopts the cascaded control method. The inner loop is the speed loop, which controls the angular velocity of the gimbal to ensure the stabilization of the platform. The outer loop is the position loop, which controls the orientation of the optical axis to achieve the target tracking. As shown in Figure 7, the proposed cascaded controller is composed of the inertial rate stabilization controller and the visual tracking controller. The gyroscope is an inertial rate sensor of the stabilization controller, which can effectively isolate the disturbance of the carrier and stabilize the platform, ensuring that the camera can obtain a stable video image. Meanwhile, the visual tracking controller enables the camera gimbal to rotate as instructed, ensuring that the optical axis points to the desired orientation for dynamic target tracking. Next, the control algorithm design will be introduced in detail.

3.2. Inertial Rate Stabilization Controller

When the quadrotor UAV has a disturbing angular velocity with respect to the inertial coordinate, it will be transmitted to the camera frame through the mechanism of the gimbal, resulting in instability of the optical axis. Therefore, a reliable servo control algorithm based on the gyroscopic angular velocity signal needs to be designed to effectively isolate the angular velocity interference of the carrier, so that the optical axis can be stabilized at the preset orientation with respect to the inertial coordinate system, regardless of the movement of the carrier. As shown in Figure 7, the output of the visual tracking controller is used as the desired angular velocity of the inertial rate stabilization controller. It is compared with the corresponding axial angular velocity measured by the gyroscope to obtain the angular velocity error, which serves as the input of the inertial rate stabilization controller. The control output, which is calculated through the control algorithm, is sent to drive the servo actuator of the corresponding gimbal.

In order to facilitate the control design, the bode diagrams of the azimuth axis and elevation axis can be depicted as shown in Figure 8 and Figure 9. The bode diagrams of uncalibrated system, calibrated system and closed-loop system are depicted by the dashed blue line, green dotted line and red solid line, respectively. According to the identified dynamics, the inertial rate stabilization controller is designed as

$$G_c\left(s\right)=\frac{K_p(T_{d}s+1)}{T_{z}s+1},$$

(17)

where $K_p$ is the proportional gain, and $T_d$ denotes the time constant.

Proposition 1.

The control law of (17) ensures the stability of the closed-loop system of the inertial rate tracking errors, provided that the control gain $K_p$ and time constant $T_d$ are selected to satisfy the following inequalities:

$$\left\{\begin{array}{cc}0<K_p<\frac{2\zeta (T_{\omega }^2+2\zeta T_{\omega }{T}_p+T_p^2)}{K(T_{\omega }{T}_p-T_{\omega }{T}_d-2\zeta T_p{T}_d)},\hfill & 0<T_d⩽T_{dth}\hfill \\ K_p>0,\hfill & T_d>T_{dth}\hfill \end{array}\right.,$$

(18)

$$\left\{\begin{array}{cc}{T}_d>0,\hfill & 0<K_p⩽K_{pth}\hfill \\ T_d>\frac{K_{p}K{T}_{\omega }{T}_p-2\zeta (T_{\omega }^2+2\zeta T_{\omega }{T}_p+T_p^2)}{K_{p}K{T}_{\omega }+2\zeta K_{p}K{T}_p},\hfill & K_p>K_{pth}\hfill \end{array}\right.,$$

(19)

where $T_{dth}=\frac{T_{\omega }{T}_p}{T_{\omega }+2\zeta T_p}$ and $K_{pth}=\frac{2\zeta (T_{\omega }^2+2\zeta T_{\omega }{T}_p+T_p^2)}{K{T}_{\omega }{T}_p}$.

Proof. 

According to the control law of (17), the characteristic equation of the closed-loop system is expressed as

$$D\left(s\right)=a_0{s}^3+a_1{s}^2+a_{2}s+a_3,$$

(20)

where $a_0=T_{\omega }^2{T}_p$, $a_1=T_{\omega }^2+2\zeta T_{\omega }{T}_p$, $a_2=2\zeta T_{\omega }+T_p+K_{p}K{T}_d$ and $a_3=K_{p}K+1$.

In order to ensure the stability of closed-loop system of $e_{\omega }$, the inequality (18) and (19) can be deduced based on the Routh-Herwitz stability criterion. □

As shown in Figure 8, the azimuth axis steady-state error of the unit step input signal of the closed-loop system is 0.2716 when $K_p=8$ is chosen. The cut-off frequency of the uncalibrated system is 76.5 rad/s, and the phase margin is 9.6 deg. In order to improve the phase margin, lag calibration is adopted. The phase margin will be increased to 27.1 deg at 25.3 rad/s when $T_d=0.4541$ is chosen, and the bandwidth of the closed-loop system is 41.67 rad/s. As shown in Figure 9, the elevation axis steady-state error of the unit step input signal of the closed-loop system is 0.3283 when $K_p=8$ is chosen. The cut-off frequency of the uncalibrated system is 70.5 rad/s, and the phase margin is 10.5 deg. In order to improve the phase margin, lag calibration is adopted. The phase margin will be increased to 30.8 deg at 23 rad/s when $T_d=0.6058$ is chosen, and the bandwidth of the closed-loop system is 39.17 rad/s.

3.3. Visual Tracking Controller

The visual tracking control design is based on the camera model in (13). A tracking error in the image plane is defined as

$$\mathit{e}\left(t\right)={\mathit{s}}^{\mathrm{ref}}\left(t\right)-\mathit{s}\left(t\right),$$

(21)

where ${\mathit{s}}^{\mathrm{ref}}={\left[\begin{array}{cc}{u}^{\mathrm{ref}}& w^{\mathrm{ref}}\end{array}\right]}^{\mathrm{T}}$ presents the reference trajectory. By taking the time derivative of $\mathit{e}$, the open-loop visual tracking error dynamics can be obtained as

$$\dot{\mathit{e}}={\dot{\mathit{s}}}^{\mathrm{ref}}-{\mathit{L}}_v(u,w,z){\mathit{v}}_{\mathrm{c}}-{\mathit{L}}_{\omega }(u,w){\mathbf{\omega }}_{\mathrm{c}}.$$

(22)

Let the angular velocity tracking error be defined as

$${\mathit{e}}_{\omega }\left(t\right)={\mathbf{\omega }}_{\mathrm{c}}^{\mathrm{des}}\left(t\right)-{\mathbf{\omega }}_{\mathrm{c}}\left(t\right),$$

(23)

where ${\mathbf{\omega }}_{\mathrm{c}}^{\mathrm{des}}={\left[\begin{array}{ccc}{\omega }_{cx}^{\mathrm{des}}& {\omega }_{cy}^{\mathrm{des}}& {\omega }_{cz}^{\mathrm{des}}\end{array}\right]}^{\mathrm{T}}$ denotes the desired angular velocity with respect to the camera frame. According to the constant rotation matrix in (16), the open-loop visual tracking error dynamics can be rewritten as

$$\begin{array}{ccc}\dot{\mathit{e}}& =& {\dot{\mathit{s}}}^{\mathrm{ref}}-{\mathit{L}}_v(u,w,z)v_{\mathrm{c}}+{\mathit{L}}_{\omega }^{\prime }(u,w)\left[\begin{array}{c}{e}_{\omega y}\\ e_{\omega z}\end{array}\right]\\ & & -\left[\begin{array}{c}w\\ -u\end{array}\right]{\omega }_{ex}-{\mathit{L}}_{\omega }^{\prime }(u,w)\left[\begin{array}{c}{\omega }_{ey}^{\mathrm{des}}\\ {\omega }_{ez}^{\mathrm{des}}\end{array}\right],\end{array}$$

(24)

where ${\mathit{L}}_{\omega }^{\prime }(u,w)=\left[\begin{array}{cc}\frac{uw}{\lambda }& -\frac{{\lambda }^2+u^2}{\lambda }\\ \frac{{\lambda }^2+w^2}{\lambda }& -\frac{uw}{\lambda }\end{array}\right]$. Based on the structure of (24) and subsequent stability analysis, the desired angular velocity vector is designed as

$$\left[\begin{array}{c}{\omega }_{ey}^{\mathrm{des}}\\ {\omega }_{ez}^{\mathrm{des}}\end{array}\right]={\mathit{L}}_{\omega }^{\prime -1}\left({\dot{\mathit{s}}}^{\mathrm{ref}}-\left[\begin{array}{c}w\\ -u\end{array}\right]{\omega }_{ex}+\rho \mathrm{sgn}\left(e\right)+ke\right),$$

(25)

where $\rho ,k\in {\mathbb{R}}^{+}$ are control gains. The inverse matrix can be expressed as

$${\mathit{L}}_{\omega }^{\prime -1}=\left[\begin{array}{cc}-\frac{uw}{\lambda ({\lambda }^2+u^2+w^2)}& \frac{{\lambda }^2+u^2}{\lambda ({\lambda }^2+u^2+w^2)}\\ -\frac{{\lambda }^2+w^2}{\lambda ({\lambda }^2+u^2+w^2)}& \frac{uw}{\lambda ({\lambda }^2+u^2+w^2)}\end{array}\right].$$

(26)

After substituting (25) into (24), the closed-loop dynamics of $\mathit{e}\left(t\right)$ can be obtained

$$\dot{e}=-{\mathit{L}}_v(u,w,z){\mathit{v}}_{\mathrm{c}}+{\mathit{L}}_{\omega }^{\prime }(u,w)\left[\begin{array}{c}{e}_{\omega y}\\ e_{\omega z}\end{array}\right]-\rho sgn\left(\mathit{e}\right)-k\mathit{e}.$$

(27)

Proposition 2.

The control law of (25) ensures the global exponential convergence of the visual tracking error as illustrated by

$$\underset{t\to \infty }{lim}e\left(t\right)=0,$$

(28)

provided that the control gain ρ is selected to satisfy the following inequality:

$$∥{\mathit{L}}_v(u,w,z){\mathit{v}}_{\mathrm{c}}-{\mathit{L}}_{\omega }^{\prime }(u,w)\left[\begin{array}{c}{e}_{\omega y}\\ e_{\omega z}\end{array}\right]∥⩽\rho .$$

(29)

Proof. 

To prove the above proposition, we define a Lyapunov function candidate $V\left(t\right)\in \mathbb{R}$ as follows:

$$V=\frac{1}{2}{e}^{\mathrm{T}}e.$$

(30)

After taking the time derivative of (30), and substituting (27) into the resulting equation, we can obtain

$$\begin{array}{ccc}\hfill \dot{V}& =& -e^{\mathrm{T}}{\mathit{L}}_v(u,w,z){\mathit{v}}_{\mathrm{c}}+e^{\mathrm{T}}{\mathit{L}}_{\omega }^{\prime }(u,w)\left[\begin{array}{c}{e}_{\omega y}\\ e_{\omega z}\end{array}\right]-\rho e^{\mathrm{T}}sgn\left(e\right)-k{e}^{\mathrm{T}}e.\hfill \end{array}$$

(31)

By using the condition in (29), the expression in (31) can be upper-bounded by

$$\dot{V}⩽-k{e}^{\mathrm{T}}e=-2kV.$$

(32)

From (32), it can be concluded that $V\left(t\right)$ is exponential convergence. Since $V\left(t\right)$ of (30) is a non-negative function, we can conclude that $V\left(t\right)\in L_{\infty }$. According to (32), we know that $e\left(t\right)$ is exponentially stable. Thus, the result in (28) is proven. □

4. Numerical Simulations

This section presents the simulation results of the proposed control algorithm. The resolution of the camera CCD chip is $320\times 240$ pixels. The geometric parameter of camera focal length $\lambda =7.5$ mm is estimated by the MATLAB Camera Calibrator App. Closed-loop responses with the proposed controller are simulated when the image of the observed object is initially located outside the center of the image frame. The control goal is to bring the observed object into the center of the field of view. The carrier disturbing angular velocity caused by a gust or turbulence and the target moving velocity are shown in Figure 10, and this will be transmitted to the camera frame through the mechanism of the gimbal.

In order to validate the superior performance of the vision-based tracking controller, comparative numerical simulations between the proposed controller and the cascaded PID controller are conducted. The sampling time of numerical simulations is 0.01 s. The simulation results of the proposed controller and the cascaded PID controller are depicted by solid blue line and dashed green line, respectively. For avoidance of the chatter generated by the signum function in (25), we replace $sgn\left(e\right)$ with a saturation function $sat(e/B)$, where B is the boundary layer thickness. The trajectories of horizontal and vertical tracking errors in the image pixel coordinate frame are illustrated in Figure 11. When the miniature quadrotor UAV has a disturbing angular velocity ${\omega }_{bz}$ with respect to the body coordinate from 14 s to 16 s, and the target moves with a sinusoidal translational velocity $v_{cz}$ from 20 s to 25 s in the simulations, the maximum horizontal tracking error u is within $\pm 10$ px for the proposed controller, and it is within $\pm 30$ px for the cascaded PID controller. Moreover, it can be seen that the horizontal tracking error u of the proposed controller is driven to 0 at last in the simulations, and the convergence speed is obviously faster than that of the cascaded PID controller. Therefore, we can conclude that the disturbance rejection ability and tracking performance of the proposed controller are better than those of the cascaded PID controller in the presence of carrier disturbance and target movement. The normalized control inputs of the azimuth axis and elevation axis are provided in Figure 12. It can be seen that the changes of the control inputs are milder than those of the cascaded PID controller, and they are all with reasonable values.

5. Experimental Results

In order to validate the target detection and tracking performance of the ISP, we have customized an experimental prototype for the miniature pan-tilt ISP. As shown in Figure 13, the hardware components of the self-developed miniature pan-tilt ISP consist of a servo controller, servo actuators, inertial measurement unit (IMU), vision sensor, etc. The volume of the miniature ISP is 85 mm × 38 mm × 75 mm, and the total weight is only 60 g; thus, it is much smaller and lighter than other commercial ISPs. It is suitable for a miniature UAV with a very limited load weight. The servo actuators installed in the azimuth gimbal and elevation gimbal steer the gimbal around the corresponding shaft, respectively. The vision sensor and IMU are mounted on the elevation gimbal. The servo controller and other electronic equipments are installed in the electronic equipment compartment of the quadrotor UAV. In order to facilitate the control design and program debugging, HCS is built as the avionics architecture of the miniature ISP [28].

5.1. Target Detection

In the target detection and tracking mission, the target selected is a small white ball. In order to accomplish this mission, video image acquisition and preprocessing are required. First of all, we create a video input object in the MATLAB environment, setting the video device to grayscale mode. Next, we create a timer object, setting the timer callback function to trigger the event. After starting the timer, the video image acquisition and preprocessing program is executed regularly. As shown in Figure 14, the image acquisition is performed by triggering image, and the image preprocessing is executed in the timer callback function subsequently. According to the threshold, the gray image is converted into a binary image, which can reduce the calculation load and enhance the real-time tracking performance. Afterwards, the small object generated by the noise signal is removed from the binary image by a filtering algorithm of mathematical morphology. The outer boundary of the region in the filtered binary image is searched using an edge detection algorithm, and then the geometric properties of the region, such as area and center coordinates, are calculated. Based on the thresholds set in advance, each region is traversed according to the geometric parameters to recognize the target. As shown in Figure 14, the target being tracked in natural background is marked with a small circle at the center of the small ball. After seeking out the target region in the image, the image-based tracking algorithm will be performed.

5.2. Target Tracking Experiments

The whole process of the target tracking mission is accomplished in an indoor environment. The avionics architecture of HCS introduced in [28] is used to validate the functionality and performance of the proposed visual servoing scheme. The experimental tests can be conducted after finishing the electrical connection of ISP and GCS, while the small ball to be tracked should be within the field of view of the airborne camera.

In order to further demonstrate the superior performance of the target tracking control scheme, comparative experimental results between the proposed controller and the cascaded PID controller of the previous conference paper [28] are given in this section. The control period of the target tracking experiments is 0.1 s. The experimental results of the proposed controller and the cascaded PID controller are depicted by the solid blue line and dashed green line, respectively. The trajectories of the horizontal and vertical tracking errors in the image pixel coordinate system are illustrated in Figure 15. The carrier disturbance occurs between 7 s and 12 s, and the target moves after 17 s during this experimental test. It can be seen that the maximum horizontal tracking error u is within $\pm 40$ px, the maximum vertical tracking error w is within $\pm 30$ px, and they are all driven near to 0 at last during this test, which means that the proposed controller steers the gimbal to move the target into the center of the field of view. Furthermore, the comparative tracking error trajectories of the cascaded PID controller are also given in Figure 15. Similarly, the carrier disturbance occurs between 2 s and 7 s, and the target moves after 11 s during the experimental test. It can be seen that the maximum horizontal tracking error u is within $\pm 60$ px, the maximum vertical tracking error w is beyond $\pm 40$ px, and they are not driven to 0 at last during the test. Thus, the disturbance rejection ability and tracking performance of the cascaded PID controller are weaker than those of the proposed controller. The trajectories of the azimuth and elevation angular velocities are shown in Figure 16, where it can be seen that the angular velocities are driven to 0 quickly in the presence of external disturbance and target movement. Obviously, the changes of angular velocities, especially ${\omega }_{ey}$ for the cascaded PID controller, are sharper than those of the proposed controller, which means that the camera image will become obscure when using the cascaded PID controller in the presence of carrier disturbance and target movement. The normalized servo inputs of the azimuth axis and elevation axis are provided in Figure 17, where it can be seen that the changes of control inputs of the proposed controller are milder than those of the cascaded PID controller and they have reasonable values. Therefore, we can conclude that satisfactory tracking performance of the proposed controller in the experimental tests is achieved in the presence of carrier disturbance and target movement.

6. Conclusions

This paper has presented a new visual servoing scheme for a miniature pan-tilt ISP. For the purpose of control design, the mathematical model of the ISP is established, combining theory analysis and system identification. The proposed controller adopts a cascaded control method based on the identified dynamics. The inner loop uses a proportional lag compensator to ensure the stabilization of the ISP, and the outer loop employs the feedback linearization-based sliding mode control method to achieve target tracking. The parameter domains for the global exponential stability of the closed-loop system are given as propositions, and they are proven by the frequency domain approach and Lyapunov-based method. The superior performance of the proposed controller compared with a traditional cascaded PID controller is verified by numerical simulations and experimental tests in the presence of carrier disturbance and target movement. The miniature pan-tilt ISP will be installed on a quadrotor UAV for actual flight experiments in the near future.