Online Predictive Visual Servo Control for Constrained Target Tracking of Fixed-Wing Unmanned Aerial Vehicles

Abstract

This paper proposes an online predictive control method for fixed-wing unmanned aerial vehicles (UAVs) with a pan-tilt camera in target tracking. It aims to achieve long-term tracking while concurrently maintaining the target near the image center. Particularly, this work takes the UAV and pan-tilt camera as an overall system and deals with the target tracking problem via joint optimization, so that the tracking ability of the UAV can be improved. The image captured by the pan-tilt camera is the unique input associated with the target, and model predictive control (MPC) is used to solve the optimization problem with constraints that cannot be performed by the classic image-based visual servoing (IBVS). In addition to the dynamic constraint of the UAV, the perception constraint of the camera is also taken into consideration, which is described by the maximum distance between the target and the camera. The accurate detection of the target depends on the amount of its feature information contained in the image, which is highly related to the relative distance between the target and the camera. Moreover, considering the real-time requirements of practical applications, an MPC strategy based on soft constraints and a warm start is presented. Furthermore, a switching-based approach is proposed to return the target back to the perception range quickly once it exceeds the range, and the exponential asymptotic stability of the switched controller is proven as well. Both numerical and hardware-in-the-loop (HITL) simulations are conducted to verify the effectiveness and superiority of the proposed method compared with the existing method.

1. Introduction

Target tracking is not only the essential basis for UAVs to perform Earth observation tasks, but also the foundation for subsequent intention analysis and situation generation. It can be applied to border surveillance, urban security, and aerial photography. Compared with a fixed surveillance camera, the field of view (FOV) of the camera is greatly expanded with the aid of a UAV, thereby enhancing the persistent monitoring capability of the target. Moreover, considering that fixed-wing UAVs usually exhibit longer endurance and faster flight speed than multi-rotor UAVs, it is meaningful to study target tracking for fixed-wing UAVs.

With the aid of vision sensors, the visual servoing method can be adopted to achieve target tracking. Based on the difference in feedback information, the method is mainly divided into two categories: position-based visual servoing (PBVS) and IBVS [1]. In terms of PBVS, it is required to calculate the relative position of the target to the UAV after obtaining the feature information of the target. Li et al. [2] achieved the passive vision-based tracking and motion estimation of a ground vehicle, which provided real-time estimation of the position, speed, and heading of the target. By designing a nonlinear adaptive observer to estimate the states, parameters, and the position of the target, Choi et al. [3] proposed a guidance law for target tracking and UAV maneuvers for a persistent excitation condition. Furthermore, Wang et al. [4] conducted actual flight tests to verify the proposed framework for tracking a mobile ground target. Nevertheless, due to the inevitable errors caused by camera calibration, there usually exist positioning deviations in PBVS which affect tracking accuracy.

In comparison, IBVS bypasses the positioning stage and directly designs the controller for the UAVs in the image plane. Therefore, it is robust to sensor models and calibration errors. Based on this method, Le Bras et al. [5] proposed a control algorithm to stabilize a UAV along a circular trajectory using a single visual landmark, which did not require independent position or velocity measurements. Similarly, Peliti et al. [6] proposed a feedback control approach aimed at guiding the UAV along a circular trajectory centered above the target, and a method that integrates IBVS using the least square method (LSM) is presented in our previous work [7] to enable continuous target tracking. Triputra et al. [8] presented a full dynamic visual servoing flight controller design using a command filtered backstepping control law for a fixed-wing UAV, which enables it to perform target tracking and the pursuing task effectively.

However, the IBVS method just focuses on the motion of the image feature information, but is unable to handle the constraints. For example, the dynamic constraint of the fixed-wing UAV and the visibility constraint of the vision sensors cannot be taken into consideration. As a result, the optimal solution may not be obtained; even worse, the target will be out of sight during tracking. To address this, MPC was introduced to solve the optimization problem with constraints, thereby improving the performance of the IBVS method. By combining standard MPC with IBVS, refs. [9,10,11] paid attention to the image kinematics and solved the constrained visual servoing problem for a 6-degrees-of-freedom (DOF) camera. Furthermore, refs. [12,13,14] took the dynamics of the 6-DOF system into consideration, which solved the visibility constraint and dynamic constraint at the same time. With regard to systems with uncertain or unknown parameters, refs. [15,16] integrated adaptive MPC and IBVS to improve the accuracy and stability of the control system. Furthermore, refs. [17,18] adopted a robust MPC to enhance the robustness of the IBVS system to uncertainty and interference. However, all of the above investigations are based on a full actuated system.

At present, there still exist very few investigations of MPC-based IBVS used in UAVs that are underactuated. This is mainly due to the challenge of concurrently considering the complex dynamic model of the UAV, the nonlinear IBVS system, and real-time requirements. Among related works, ref. [19] proposed an observer-based MPC scheme for the IBVS of a quadrotor, the objective of which was to regulate the relative pose of the quadrotor to a planar ground target using a monocular camera and an inertia measurement unit. In order to produce safe control inputs of the quadrotor in an obstacle-containing environment, ref. [20] designed a model predictive IBVS scheme by optimizing a cost function of the predicted image trajectories under the constraints of visibility and velocity. In addition, ref. [21] proposed a robust nonlinear MPC scheme for the visual servoing of quadrotors subject to external disturbances. Note that all of the above works are based on quadrotors. However, due to the highly dynamic nature of the moving target and fixed-wing UAV, it is challenging to achieve target tracking based on the real-time MPC-based IBVS method. It is imperative that the method demonstrates satisfactory real-time performance to be viable for online deployment. To the best of our knowledge, there is still a lack of relevant studies.

In this paper, we propose an online MPC-based IBVS method for a fixed-wing UAV in target tracking. With the aid of the method, the UAV is able to achieve persistent tracking of the target. Simultaneously, a pan-tilt camera is controlled to keep the target near the image center. Given the limited perception capability of the camera, when the target is far away from the camera, it may not be detected correctly due to the lack of image feature information. To this end, an MPC is adopted to consider both the perception constraint of the camera and the dynamic constraint of the fixed-wing UAV. Furthermore, considering that the target may exceed the clear perception range of the camera with increase in its speed, a switching-based control strategy is designed to quickly return the target to the perception range. Last but not least, both numerical and HITL simulations are conducted to verify the feasibility and superiority of the proposed method. The contributions of this paper are summarized as follows:

• A novel MPC-based IBVS method is proposed for the integrated system comprising a fixed-wing UAV and a pan-tilt camera in target tracking, with a focus on the camera’s perception capability. In contrast to prior research on target tracking [5,6,7,8], our method ensures that the target remains within the clear perception range, facilitating the generation of more image feature information for accurate detection.
• A switching-based MPC strategy incorporating soft constraints and a warm start is developed to ensure real-time performance in practical applications. By enabling smooth transitions between the different optimization controllers, this strategy empowers the UAV to promptly approach the target once it exceeds the camera’s clear perception range, thereby ensuring continuous tracking. Moreover, the stability of the closed-loop system is demonstrated.
• Extensive simulations, including numerical and HITL simulations, are conducted to verify the effectiveness and superiority of the proposed method compared with the existing method.

2. Problem Statement

The paper aims to achieve target tracking for a fixed-wing UAV using a pan-tilt camera based on a method combining MPC and IBVS. As Figure 1 shows, the UAV can circle around the target and concurrently keep the target near the image center with the aid of the pan-tilt camera. Note that the position and motion of the target considered in this work is unknown to the UAV, and the only source of the information about the target is the image captured by the pan-tilt camera. In terms of the pan-tilt camera, it has 2-DOF that can rotate in the horizontal and vertical planes, as shown in Figure 2. Three different frames are primarily involved. The body frame ${\mathcal{F}}_b=\{o_b,x_b,y_b,z_b\}$ is fixed on the UAV, where $o_b$ is located at the center of gravity of the fixed-wing UAV and the $x_b$-axis points to the head direction of the UAV. The origin $o_c$ of the camera frame ${\mathcal{F}}_c=\{o_c,x_c,y_c,z_c\}$ lies at the optical center, and the $z_c$-axis is along the optic axis. As for the image frame ${\mathcal{F}}_i=\{o_i,x_i,y_i\}$, its $x_i$-axis and $y_i$-axis are parallel to the $x_c$-axis and the $y_c$-axis, respectively.

2.1. System Modeling

The system used here is composed of a fixed-wing UAV and a pan-tilt camera. In contrast to rotor UAVs that are capable of executing various maneuvers [22,23], fixed-wing UAVs are constrained by the minimum stall speed and the turning radius. Therefore, they are assumed to fly at a constant speed and altitude in this study. The configuration used has been commonly employed in research on formation control [24] and path following [25]. Moreover, the unicycle model can be adopted to analyze the kinematics of the UAV. Denote the flight speed and altitude of the UAV by $V_t$ and H, respectively. The yaw, pitch, and roll angles of the UAV are denoted by $\psi$, $\theta$, and $\varphi$, respectively. Then, the UAV model can be represented as

$$\left\{\begin{array}{c}{V}_x=V_t·cos\psi ,\hfill \\ V_y=V_t·sin\psi ,\hfill \\ \dot{\psi }=u_{\psi },\hfill \end{array}\right.$$

(1)

where $u_{\psi }$ is the control input of the UAV. It is noted that the model is also used in [24,25].

Next, the pan-tilt camera with 2-DOF is modeled, which is shown in Figure 2. Denote the horizontal plane by $S_p$, and the $x_t$-axis lies in $S_p$, while the $x_p$-axis is perpendicular to $S_p$. Correspondingly, the pitch angle and the yaw angle of the pan-tilt are denoted by ${\theta }_t$ and ${\theta }_p$, respectively. Specifically, this paper adopts a pan-tilt with omnidirectional deflection around the $x_p$-axis. Therefore, the pan-tilt model can be represented as

$$\left\{\begin{array}{c}\dot{{\theta }_p}=u_p,{\theta }_p\in [-{\gamma }_p,{\gamma }_p]\hfill \\ \dot{{\theta }_t}=u_t,{\theta }_t\in [{\gamma }_1,{\gamma }_2]\hfill \end{array}\right.$$

(2)

where $u_p$ and $u_t$ are the control inputs of the pan-tilt, and ${\gamma }_1>-\frac{\pi }{2}$ while ${\gamma }_2>0$. Note that ${\theta }_t=0$ when the $z_c$-axis lies in $S_p$, as shown in Figure 2.

To sum up, we have the following overall system composed of a UAV and a pan-tilt camera:

$$\left\{\begin{array}{c}{V}_x=V_t·cos\psi \hfill \\ V_y=V_t·sin\psi \hfill \\ \dot{\psi }=u_{\psi }\hfill \\ \dot{{\theta }_p}=u_p,{\theta }_p\in [-\pi ,\pi ]\hfill \\ \dot{{\theta }_t}=u_t,{\theta }_t\in [{\gamma }_1,{\gamma }_2]\hfill \end{array}\right.$$

(3)

Problem 1.

How to design a controller for the overall system composed of a fixed-wing UAV and a pan-tilt camera based on the model of (3), such that the target will not be beyond the field of view or far away from the UAV to make it undetectable, and the UAV can achieve long-term tracking of the fast moving target while keeping it near the image center?

2.2. Image Kinematics

This part aims to construct the image kinematic model so that it can be used to predict the future behavior of the feature point during the process of the MPC. Denote the coordinates of the target in ${\mathcal{F}}_c$ and ${\mathcal{F}}_i$ by $(x_c,y_c,z_c)$ and $(u_i,v_i)$, respectively. According to the triangle similarity, there exists the following relationship:

$$\frac{u_i}{x_c}=\frac{v_i}{y_c}=\frac{f}{z_c},$$

(4)

where f represents the focal length.

Furthermore, the motion of the camera needs to be taken into consideration. Denote the linear velocity vector along the three axes of ${\mathcal{F}}_c$ by $V_c={[V_x,V_y,V_z]}^{\top }$. Simultaneously, the corresponding angular velocity vector is denoted by ${\mathrm{\Omega }}_c={[{\omega }_x,{\omega }_y,{\omega }_z]}^{\top }$. Let $s_i={[u_i,v_i]}^{\top }$; then, the following equation can be obtained based on [1]:

$${\dot{s}}_i=L_s·\left[\begin{array}{c}{V}_c\\ {\mathrm{\Omega }}_c\end{array}\right],$$

(5)

where $L_s\in {\mathbb{R}}^{2\times 6}$ is called an image Jacobian matrix, which is equal to

$$L_s=\left[\begin{array}{cccccc}-\frac{f}{z_c}& 0& \frac{u_i}{z_c}& \frac{u_i{v}_i}{f}& -\frac{f^2+u_i^2}{f}& v_i\\ 0& -\frac{f}{z_c}& \frac{v_i}{z_c}& \frac{f^2+v_i^2}{f}& -\frac{u_i{v}_i}{f}& -u_i\end{array}\right].$$

(6)

3. Controller Design

3.1. MPC Optimization Problem Formulation

This part aims to solve Problem 1.

Based on our previous work [7], a reference state named the Ideal State was proposed to construct the relationship between the image feature vector and $u_{\psi }$. As for the reference state, it has the following constant state:

$${\theta }_p=-\frac{\pi }{2},{\theta }_t=-\alpha ,\theta =0,\varphi =0.$$

(7)

Note that $\alpha \in (0,\frac{\pi }{2}]$ is a given constant. When given the desired circling radius $R_d$, $\alpha$ can be expressed as $\alpha =arctan\left(\frac{H}{R_d}\right)$. Denote the coordinates of the feature point in ${\mathcal{F}}_i$ by $(u_1,v_1)$, and in the reference state by $(u_2,v_2)$; then, the relationship can be written as:

$$\left[\begin{array}{c}{\dot{u}}_2\\ {\dot{v}}_2\end{array}\right]=\left[\begin{array}{cc}\frac{f^2+{u_2}^2}{f}·cos\alpha -v_2·sin\alpha & \frac{f}{z_c}\\ \frac{u_2·v_2}{f}·cos\alpha +u_2·sin\alpha & 0\end{array}\right]·\left[\begin{array}{c}{u}_{\psi }\\ V_t\end{array}\right].$$

(8)

Simultaneously, $u_p$ and $u_t$ can be obtained from (20) in [7], which is

$$\left\{\begin{array}{c}{u}_p=arctan\left(\frac{{\dot{u}}_1}{f}\right),\hfill \\ u_t=arctan\left(\frac{{\dot{v}}_1}{f}\right).\hfill \end{array}\right.$$

(9)

Define $s_2={[u_2,v_2]}^{\top }$ and based on (8), the discrete time state-space model can be expressed as

$$s_2(k+1)=s_2\left(k\right)+{\dot{s}}_2\left(k\right)·\Delta t,$$

(10)

where $\Delta t$ is the sampling time.

For convenience of subsequent analysis, the discrete form is rewritten as

$$s_2(k+1)={\mathrm{f}}_s(s_2,u_{\psi },k),$$

(11)

Since the yaw rate of the fixed-wing UAV is restricted, then the following constraint needs to be satisfied:

$$u_{\psi }\in U_{set},U_{set}=[-u_{max},u_{max}].$$

(12)

In order to guarantee that the change in $u_{\psi }$ can be smooth, the acceleration constraint is also given:

$$\Delta u_{\psi }\in a_{set},a_{set}=[-a_{max},a_{max}].$$

(13)

where

$$\Delta u_{\psi }(k+i)=u_{\psi }(k+i)-u_{\psi }(k+i-1).$$

(14)

Furthermore, the perception capability of the camera is taken into consideration as well. Under the condition of invariant camera parameters, whether the target can be detected correctly usually depends on the amount of feature information contained in the image. More specifically, it is related to the maximum perceived distance of the camera, which is denoted by $L_M$. Therefore, the clear perception range of the camera is equivalent to a cone, as shown in Figure 3.

In order to determine whether the target is within the cone, the feature point in the current image plane is first mapped into that in the horizontal image plane. Define the feature point in the latter plane as $(u_3,v_3)$, considering that ${\theta }_t=-\alpha$ of (7) in the Ideal State; then, $(u_3,v_3)$ can be obtained from $(u_2,v_2)$ based on the following transformation:

$$\left[\begin{array}{c}{u}_3\\ v_3\\ f\end{array}\right]=k_3\left[\begin{array}{ccc}1& 0& 0\\ 0& sin\alpha & -cos\alpha \\ 0& cos\alpha & sin\alpha \end{array}\right]\left[\begin{array}{c}{u}_2\\ v_2\\ f\end{array}\right],$$

(15)

where $k_3\in {\mathbb{R}}^{+}$. After solving the above three equations, the value of $u_3$ and $v_3$ can be obtained.

Since the flight altitude H of the UAV is constant in this paper, the maximum perceived distance can be converted into the maximum horizontal distance, which is denoted by $R_M$. Therefore, the target is recognized to be within the perception range if, and only if, the following condition is met:

$$\frac{\sqrt{{u_3}^2+{v_3}^2}}{f}\le \frac{R_M}{H}.$$

(16)

After that, in order to keep the target within the perception range when the UAV circles around it, $\alpha$ in (7) needs to satisfy the following condition:

$$tan(\frac{\pi }{2}-\alpha )\le \frac{R_M}{H},$$

(17)

Simultaneously, considering the constraint of $u_{\psi }$ in (12), there exists

$$tan(\frac{\pi }{2}-\alpha )\ge \frac{V_t}{u_{max}·H}.$$

(18)

Define ${\eta }_1=arctan\frac{R_M}{H}$, ${\eta }_2=arctan\frac{V_t}{u_{max}·H}$, then $\alpha \in [\frac{\pi }{2}-{\eta }_1,\frac{\pi }{2}-{\eta }_2]$. Based on (16), the perception constraint at time-step k can be defined as

$${\mathrm{g}}_1\left(k\right)\le 0,$$

(19)

where

$${\mathrm{g}}_1:=\frac{{u_3}^2+{v_3}^2}{f^2}-\frac{R_M^2}{H^2}$$

(20)

Denote the error of the feature point by

$$\Delta s_2\left(i\right|k)=s_d-s_2\left(i\right|k),$$

(21)

where $s_d={[0,0]}^{\top }$ is the desired value of $s_2$, and $s_2\left(0\right|k)=s_2\left(k\right)$ is the observed value at time-step k. Since $\Delta u_{\psi }\left(0\right|k)=u_{\psi }\left(0\right|k)-u_{\psi }(k-1)$, then the cost function can be defined as follows:

$$J_1\left(k\right)=\sum _{i=1}^{N_p}\left(\parallel \Delta s_2{\left(i\right|k)\parallel }_{Q_s}^2+{\parallel \Delta u_{\psi }(i-1|k)\parallel }_{Q_u}^2\right)=\sum _{i=1}^{N_p}{J}_r\left(i\right|k),$$

(22)

where $\parallel \Delta s_2{\parallel }_{Q_s}^2=\Delta s_2^{\top }·Q_s·\Delta s_2$ and similar to $\parallel \Delta u_{\psi }{\parallel }_{Q_u}^2$, $Q_s=diag\{q_1,q_2\}$ is positive definite and $Q_u\in {\mathbb{R}}^{+}$. The control sequence is

$$\mathbf{U}=\{u_{\psi }\left(0\right|k),u_{\psi }\left(1\right|k),\cdots ,u_{\psi }(N_p-1|k)\},$$

(23)

$N_p\in {\mathbb{R}}^{+}$ is the prediction horizon.

Note that $J_1=0$ if, and only if, $\Delta s_2\left(i\right|k)=0$ and $\Delta u_{\psi }(i-1|k)$ for $i\in \{1,\cdots ,N_p\}$. That is, the constant control input of the UAV makes it circle around the static target.

Due to the fact that only the first control input of $\mathbf{U}$ in (23) is used, the following contraction constraint is provided to ensure the stability of the system (which will be proven in Section 3.3):

$$C{C}_1\left(k\right)=\parallel \Delta s_2\left(1\right|k)\parallel -\delta ·\parallel \Delta s_2\left(k\right)\parallel \le 0,$$

(24)

where $\parallel \Delta s_2\left(k\right)\parallel$ represents the 2-norm of $s_2\left(k\right)$, $\delta \in (0,1)$.

Therefore, based on Equations (11)–(13), (19), (22), and (24), the MPC optimization problem can be formulated as

$$\begin{array}{cc}{\mathcal{P}}_1:\hfill & arg\underset{\mathbf{U}\left(k\right)}{min}{J}_1\left(k\right)\hfill \end{array}$$

(25a)

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.\hfill & s_2\left(i\right|k)={\mathrm{f}}_s\left(s_2,u_{\psi },i-1|k\right),\hfill \end{array}$$

(25b)

$$\begin{array}{l}{u}_{\psi }(i-1|k)\in {\mathcal{U}}_{set},\end{array}$$

(25c)

$$\begin{array}{l}\Delta u_{\psi }(i-1|k)\in a_{set},\end{array}$$

(25d)

$$\begin{array}{l}{\mathrm{g}}_1\left(i\right|k)\le 0, \end{array}$$

(25e)

$$\begin{array}{l}C{C}_1\left(k\right)\le 0, \end{array}$$

(25f)

where $i=1,2,\cdots ,N_p$.

3.2. Switching-Based Optimization Control

Problem 2.

With the increase in $N_p$ and limited by the computing capability of the processors, it is time-consuming to solve the optimization problem of (25) with nonlinear hard constraints (25e) and (25f) online. Even worse, they may also lead to the problem of an infeasible solution within the predefined iterations. Then, the question arises of how to design an online MPC strategy for practical application?

To solve the problem, this paper considers adding the hard constraints into the cost function, so that these constraints need not be strictly satisfied.

Firstly, add the constraint (25e) into $J_r$; then, we have

$$J_{r,1}\left(i\right|k)=J_r\left(i\right|k)+{\beta }_1·max\{{\mathrm{g}}_1\left(i\right|k),0\},$$

(26)

where ${\beta }_1\in {\mathbb{R}}^{+}$ is a constant.

Remark 1.

In the context of discrete systems as addressed in this work, the discontinuous nature of the sampling points renders the exact occurrence at ${\mathrm{g}}_1\left(i\right|k)=0$ highly improbable. Furthermore, it is essential to acknowledge that the cost function focuses on the overall optimization objective, thus rendering the non-differentiability of specific points inconsequential to the overall optimization.

Furthermore, considering the constraints of (12), (13), and the unknown motion of the target, it may be unable to make ${\mathrm{g}}_1(k+1)\le 0$ when ${\mathrm{g}}_1\left(k\right)>0$. Therefore, ${\mathrm{g}}_1$ in (20) is redefined as follows to make ${\beta }_1·max\{·\}$ in (26) function earlier:

$${\mathrm{g}}_2:=\frac{{u_3}^2+{v_3}^2}{f^2}-\frac{{(R_M-\Delta R)}^2}{H^2},$$

(27)

where $\Delta R>0$. Then, (26) is rewritten as

$$J_{r,1}\left(i\right|k)=J_r\left(i\right|k)+{\beta }_1·max\{{\mathrm{g}}_2\left(i\right|k),0\},$$

(28)

After that, turn (24) into a soft constraint as well; then, the cost function can be represented as

$$J_{1,soft}\left(k\right)=\sum _{i=1}^{N_p}{J}_{r,1}\left(i\right|k)+{\beta }_2·max\{C{C}_1\left(k\right),0\}.$$

(29)

Compared with the soft constraints-based method that introduces slack variables like [26], the method that uses the max function is more efficient. When ${\mathrm{g}}_2\left(i\right|k)\le 0$ is satisfied, the nonlinear function does not participate in solving the gradient in each iteration when using the interior point method. However, the gradients of all the ${\mathrm{g}}_2\left(i\right|k)$ still need to be calculated in each iteration for the former method, which is more time-consuming.

Therefore, the MPC optimization problem can be reformulated as

$$\begin{array}{cc}\hfill {\mathcal{P}}_{1,soft}:& arg\underset{\mathbf{U}\left(k\right)}{min}{J}_{1,soft}\left(k\right)\hfill \end{array}$$

(30a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& s_2\left(t\right|k)={\mathrm{f}}_s\left(s_2,u_{\psi },i-1|k\right),\hfill \end{array}$$

(30b)

$$\begin{array}{cc}\hfill & u_{\psi }(i-1|k)\in {\mathcal{U}}_{set}, \hfill \end{array}$$

(30c)

$$\begin{array}{cc}\hfill & \Delta u_{\psi }(i-1|k)\in a_{set}, \hfill \end{array}$$

(30d)

where $i=1,2,\cdots ,N_p$.

A warm start is also used in this work. Based on our previous work [7], the control input for the UAV can be obtained with the aid of LSM and (11), which is denoted by $u_{\psi }\left(i\right|0),i=0,\cdots ,N_p-1$. Note that $u_{{\psi }_i}\in U_{set}$ is required; then, the initial control sequence can be chosen as

$${\mathbf{U}}_0=\{u_{\psi }\left(0\right|0),u_{\psi }\left(1\right|0),\cdots ,u_{\psi }(N_p-1|0)\}.$$

(31)

Obviously, ${\mathbf{U}}_0$ satisfies the inequality constraints of (30c) and (30d). This indicates that ${\mathbf{U}}_0$ is the feasible solution of the optimization problem.

Furthermore, denote the optimal solution at time-step k by ${\mathbf{U}}^{\ast }\left(k\right)$; then, the feasible solution at time-step $k+1$, denoted by $\mathbf{U}(k+1)$, can be chosen as

$$u_{\psi }\left(i\right|k+1)=\left\{\begin{array}{c}{u}_{\psi }^{\ast }(i+1|k),i=0,\cdots ,N_p-2\hfill \\ u_{\psi }^{\ast }(N_p-1|k),i=N_p-1\hfill \end{array}\right.$$

(32)

With increase in the target speed, the relative distance between the fixed-wing UAV and the target may exceed the maximum perceived distance limited by the constraints of (12) and (13). Although the target may still be recognized with the aid of a detection algorithm, the accuracy of detection will decline and the details of the target cannot be clearly distinguished. It is noted that investigating the details of the target is a basic need for UAV target tracking applications.

The items of ${\beta }_1·max\{{\mathrm{g}}_2\left(i\right|k),0\}(i=1,\cdots ,N_p)$ in the cost function $J_{r,1}$ of (28) help to avoid the relative distance from exceeding $R_M$. However, the optimization problem of (30) is still more concerned with the convergence of $s_2$. Moreover, both the gradients about $(u_2,v_2)$ and $(u_3,v_3)$ need to be calculated online, which increases the workload of computation.

To solve the problem, a switching-based control method is proposed. When the relative distance keeps within the maximum perceived distance, the optimization problem (30) is considered to enable the UAV to fly around the target. However, if the condition is unsatisfied, the other optimization problem is designed to return the target back to the perception range quickly.

According to (16), the perception constraint is directly affected by $(u_3,v_3)$. However, the optimization problem ${\mathcal{P}}_{1,soft}$ focuses on the convergence of $(u_2,v_2)$ to $s_d$. Therefore, in order to accelerate the convergence of ${\mathrm{g}}_1\left(k\right)$ to 0 once the perception constraint is unsatisfied, a new cost function is defined as follows:

$$J_g\left(k\right)=\sum _{i=1}^{N_p}\left(\parallel \Delta s_3{\left(i\right|k)\parallel }_{Q_{s,g}}^2+{\parallel \Delta u_{\psi }(i-1|k)\parallel }_{Q_u}^2\right)=\sum _{i=1}^{N_p}{l}_g\left(i\right|k),$$

(33)

where $Q_{s,g}=diag\{q_{1,g},q_{2,g}\}$ is positive definite, $Q_u\in {\mathbb{R}}^{+}$ and

$$\left\{\begin{array}{c}{s}_3={[u_3,v_3]}^{\top },\hfill \\ \Delta s_3\left(i\right|k)=s_d-s_3\left(i\right|k).\hfill \end{array}\right.$$

(34)

In addition, to ensure the convergence of $\Delta s_3$, the following contraction constraint is also given:

$$C{C}_g\left(t\right|k)=\parallel \Delta s_3\left(1\right|k)\parallel -\delta ·\parallel \Delta s_3\left(k\right)\parallel \le 0,$$

(35)

where $\parallel \Delta s_3\left(k\right)\parallel$ represents the 2-norm of $\Delta s_3\left(k\right)$. Similar to (29), in order to avoid the nonlinear constraint of (35), the cost function of $J_g\left(k\right)$ in (33) is rewritten as

$$J_{g,soft}\left(k\right)=\sum _{i=1}^{N_p}{l}_g\left(i\right|k)+{\beta }_g·max\{C{C}_g\left(k\right),0\},$$

(36)

where ${\beta }_g\in {\mathbb{R}}^{+}$.

Remark 2.

The convergence of $C{C}_1$ in (24) does not theoretically guarantee the convergence of $C{C}_g$ in (35) around $g_1\left(k\right)=0$. Only when ${u_2}^2$ and ${v_2}^2$ converge simultaneously can the convergence of (35) be ensured (refer to Section 3.3 for the proof), thereby ensuring the stability of the switching process.

To this end, a new contractive constraint is defined as follows:

$$\left\{\begin{array}{c}C{C}_{2a}=|\Delta u_3\left(1\right|k)|-\delta ·|\Delta u_3\left(k\right)\parallel \le 0,\\ C{C}_{2b}=|\Delta v_3\left(1\right|k)|-\delta ·|\Delta v_3\left(k\right)\parallel \le 0.\end{array}\right.$$

(37)

Referring to (36), the corresponding cost function can be written as

$$J_{2,soft}\left(k\right)=\sum _{t=1}^{N_p}{l}_g\left(t\right|k)+a_2·\left(max\{C{C}_{2a}\left(k\right),0\}+max\{C{C}_{2b}\left(k\right),0\}\right).$$

(38)

Considering that (37) necessitates the simultaneous convergence of both $|u_2|$ and $|v_2|$, whereas (35) only mandates the convergence of $s_3$, it is evident that the solution requirements for the former are more stringent. By comprehensively considering the stability of the switching process and alleviating the solution constraints, we reformulate the control optimization problem based on Equations (30), (36), and (38) as

$$\begin{array}{cc}\hfill {\mathcal{P}}_{switch}:& arg\underset{\mathbf{U}\left(k\right)}{min}{J}_{switch}\left(k\right)\hfill \end{array}$$

(39a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathit{s}}_2\left(t\right|k)={\mathrm{f}}_s\left({\mathit{s}}_2,u_{\psi },t-1|k\right), \hfill \end{array}$$

(39b)

$$\begin{array}{cc}\hfill & u_{\psi }(t-1|k)\in {\mathcal{U}}_{set},\hfill \end{array}$$

(39c)

$$\begin{array}{cc}\hfill & \Delta u_{\psi }(t-1|k)\in a_{set}, \hfill \end{array}$$

(39d)

where $t=1,2,\cdots ,N_p$, and

$$J_{switch}=\left\{\begin{array}{l}{J}_{1,soft}\left(k\right),n_1=0,\\ J_{2,soft}\left(k\right),0<n_1<n_{max},\\ J_{g,soft}\left(k\right),n_1=n_{max}\end{array}\right.$$

(40)

Note that $n_1\in {\mathbb{Z}}^{+}$ functions as a counter that increases by one when ${\mathrm{g}}_1\left(k\right)\le 0$ and resets to 0 when ${\mathrm{g}}_1\left(k\right)>0$. Moreover, $n_{max}\in {\mathbb{Z}}^{+}$ represents the counting threshold. The constraint (35) is employed to relax the solution limit only after $n_1$ reaches $n_{max}$, thereby mitigating the potential issue of an unstable switching process.

The update of $n_1$ is formulated as

$$\left\{\begin{array}{cc}{n}_1=\mathrm{sat}(n_1+1),& {\mathrm{g}}_1\left(k\right)\le 0,\\ n_1=0,& {\mathrm{g}}_1\left(k\right)>0,\end{array}\right.$$

(41)

where the saturation function $sat(·)$ is defined as

$$\mathrm{sat}\left(x\right)=\left\{\begin{array}{cc}{n}_{max},& x>n_{max},\\ x,& else.\end{array}\right.$$

(42)

The initialization of $n_1$ is as follows:

$$\left\{\begin{array}{cc}{n}_1=n_{max},& {\mathrm{g}}_1\left(0\right)<0,\\ n_1=0,& else,\end{array}\right.$$

(43)

The implementation of the proposed method is shown as Algorithm 1.

Algorithm 1:Switching-based optimization control for target tracking

Require: The image captured by camera

Ensure:$(u_1,v_1)\to 0$, $(u_2,v_2)\to 0$

1: Let $k=0$, calculate ${\mathrm{g}}_1\left(0\right)$, and initialize $n_1$ based on (43).     2: while Discover the target do     3:    Detect the centroid coordinates $(u_1,v_1)$;     4:    Calculate $(u_2,v_2)$ of the reference state (7);     5:    if $k=0$ then     6:      Obtain initial control sequence $\mathbf{U}\left(k\right)$ based on (31).     7:    else     8:      Obtain initial control sequence $\mathbf{U}\left(k\right)$ based on (32).     9:    end if     10:    Solve the optimization problem (39) online to obtain ${\mathbf{U}}^{\ast }\left(k\right)$.     11:    Choose the first element of ${\mathbf{U}}^{\ast }\left(k\right)$ to be the control input.     12:    $k=k+1$.     13:    Obtain the control input of the pan-tilt based on (9).     14:    Calculate ${\mathrm{g}}_1\left(k\right)$ and update $n_1$ based on (41).     15: end while

3.3. Stability Analysis

Theorem 1.

Based on the model of (11), given the contractive constraints (24), (35), and (37), the control input obtained from the control optimization problem (39) ensures the stability of the closed-loop system.

Proof. 

Since (39) is a switching-based control optimization problem, then the stability analysis is divided into three parts, which correspond to the steps when ${\mathrm{g}}_1\left(k\right)>0$, ${\mathrm{g}}_1\left(k\right)\le 0$ for the switching process.

(1) ${\mathrm{g}}_1\left(k\right)>0$

Based on (35), there exists the following relationship:

$$\parallel \Delta s_3(k+1)\parallel \le \delta ·\parallel \Delta s_3\left(k\right)\parallel \le \cdots \le {\delta }^{k+1}·\parallel \Delta s_3\left(0\right)\parallel .$$

(44)

When $\delta \in (0,1)$, there exists $e^{(\delta -1)}\ge \delta$; then, we have

$$e^{(\delta -1)k}\ge {\delta }^k\ge 0,\forall k\in {\mathbb{Z}}^{+}.$$

(45)

Substitute (45) into (44) and we obtain

$$\parallel \Delta s_3(k+1)\parallel \le \parallel \Delta s_3\left(0\right)\parallel ·e^{-(1-\delta )(k+1)}.$$

(46)

Since $\underset{k\to \infty }{lim}{e}^{-(1-\delta )(k+1)}\to 0$ and $\parallel \Delta s_3(k+1)\parallel \ge 0$, then

$$\underset{k\to \infty }{lim}\parallel \Delta s_3(k+1)\parallel \to 0.$$

(47)

Due to $\Delta s_3\left(k\right)=s_3\left(k\right)$, then

$$\underset{k\to \infty }{lim}{s}_3\left(k\right)\to 0.$$

(48)

This implies that $s_3$ will exponentially converge until the condition ${\mathrm{g}}_1\left(k\right)\le 0$ is met.

(2) ${\mathrm{g}}_1\left(k\right)\le 0$

When $0<n_1<n_{max}$, the contractive constraint (24) takes effect, resulting in the following inequalities:

$$\parallel \Delta {\mathit{s}}_2\left(1\right|k)\parallel -\delta ·\parallel \Delta {\mathit{s}}_2\left(k\right)\parallel \le 0.$$

(49)

By referring to (44) and (45), we obtain

$$\parallel \Delta s_2(k+1)\parallel \le \parallel \Delta s_2\left(0\right)\parallel ·e^{-(1-\delta )(k+1)}.$$

(50)

Consequently, it follows that

$$\underset{k\to \infty }{lim}{s}_2\left(k\right)\to 0,$$

(51)

indicating an exponential convergence of $s_2$ to 0.

Moreover, the norm of $s_2$ is expressed as

$$\parallel {\mathit{s}}_2(k+1)\parallel =\sqrt{{u_2(k+1)}^2+{v_2(k+1)}^2},$$

(52)

When $n_1=n_{max}$, the contractive constraint (37) comes into play, leading to the following relationship based on (52):

$$\parallel {\mathit{s}}_2(k+1)\parallel \le \sqrt{{\delta }^2·\left({u_2\left(k\right)}^2+{v_2\left(k\right)}^2\right)}=\delta ·\parallel {\mathit{s}}_2\left(k\right)\parallel ,$$

(53)

thus demonstrating exponential convergence in accordance with (50) and (51).

(3) The switching process

According to (15), $u_3$ and $v_3$ can be expressed as

$$u_3=\frac{f·u_2}{v_2·cos\alpha +f·sin\alpha },v_3=\frac{f·(v_2·sin\alpha -f·cos\alpha )}{v_2·cos\alpha +f·sin\alpha }.$$

(54)

Then, we have

$$\begin{array}{cc}\hfill {u_3}^2+{v_3}^2& =\frac{f^2·\left({u_2}^2+{(v_2·sin\alpha -f·cos\alpha )}^2\right)}{{(v_2·cos\alpha +f·sin\alpha )}^2}\hfill \\ \hfill & =\frac{f^2·\left({u_2}^2+{v_2}^2+f^2-{(v_{2}cos\alpha +f·sin\alpha )}^2\right)}{{(v_2·cos\alpha +f·sin\alpha )}^2}\hfill \\ \hfill & =\frac{f^2·({u_2}^2+{v_2}^2+f^2)}{{(v_2·cos\alpha +f·sin\alpha )}^2}-f^2.\hfill \end{array}$$

(55)

Based on (55), the derivative of ${u_3}^2+{v_3}^2$ with respect to ${u_2}^2$ is

$$\frac{\partial ({u_3}^2+{v_3}^2)}{\partial {u_2}^2}=\frac{f^2}{{(v_2·cos\alpha +f·sin\alpha )}^2}>0.$$

(56)

This indicates that ${u_3}^2+{v_3}^2$ is positively correlated with ${u_2}^2$.

Since $v_2<0$ when $s_2$ is near the area defined by ${\mathrm{g}}_1=0$, then we can first solve the derivative of ${u_3}^2+{v_3}^2$ with respect to $v_2$, which is

$$\begin{array}{cc}\hfill \frac{\partial ({u_3}^2+{v_3}^2)}{\partial v_2}=& \frac{f^2·{(v_2·cos\alpha +f·sin\alpha )}^2·\left(2v_2\right)}{{(v_2·cos\alpha +f·sin\alpha )}^4}-\hfill \\ \hfill & \frac{f^2·({u_2}^2+{v_2}^2+f^2)·2(v_2·cos\alpha +f·sin\alpha )·cos\alpha }{{(v_2·cos\alpha +f·sin\alpha )}^4}\hfill \\ \hfill =& \frac{2f^2·\left((v_2·cos\alpha +f·sin\alpha )·v_2-({u_2}^2+{v_2}^2+f^2)·cos\alpha \right)}{{(v_2·cos\alpha +f·sin\alpha )}^3}\hfill \\ \hfill =& \frac{2f^2·\left(f·v_2·sin\alpha -({u_2}^2+f^2)·cos\alpha \right)}{{(v_2·cos\alpha +f·sin\alpha )}^3}.\hfill \end{array}$$

(57)

It is noted that $sin\alpha >0$ and $cos\alpha >0$, then

$$f·v_2·sin\alpha -({u_2}^2+f^2)·cos\alpha <0.$$

(58)

Simultaneously, there exists $arctan\left(\frac{-v_2}{f}\right)<\alpha$ according to the definition of $\alpha$, which can be rewritten as

$$v_2·cos\alpha +f·sin\alpha >0.$$

(59)

Substituting (58) and (59) into (57), we have

$$\frac{\partial ({u_3}^2+{v_3}^2)}{\partial v_2}<0.$$

(60)

Considering that $v_2<0$, then ${u_3}^2+{v_3}^2$ is also positively correlated with ${v_2}^2$. Therefore, if both ${u_2}^2$ and ${v_2}^2$ are convergent when $v_2<0$, then $\parallel \Delta s_3\parallel$ is convergent. □

4. Numerical Simulations and Results

In the numerical simulations, the parameters are set as

Table 1. Parameters and their values.

Parameter	Value	Parameter	Value	Parameter	Value
$V_t$	16 m/s	H	100 m	$R_M$	100 m
$\Delta R$	10 m	$R_d$	50 m	$u_{max}$	0.5 rad/s
$a_{max}$	0.02 rad/s	$\Delta t$	0.05 s	$Q_s$	diag{0.5,2}
$Q_u$	$1.0\times {10}^6$	$Q_{s,g}$	diag{1,1}	$\delta$	0.99
${\beta }_1$	$1.0\times {10}^4$	${\beta }_2$	$1.0\times {10}^3$	${\beta }_g$	$1.0\times {10}^3$
f	537	$N_p$	15	$n_{max}$	100

shows. Simultaneously, for convenience of the following analysis, the target speed and relative horizontal distance between the target and UAV are denoted by $V_c$ and R, respectively. The initial position and bearing of the UAV in the global frame are $(-85,0,50)$ and $\frac{\pi }{2}$, while the initial position of the target lies at the origin. Note that the fmincon function in MATLAB is adopted to solve the MPC problem.

4.1. Comparative Experiments of the LSM Method [7] and the Proposed Method

This part compares the tracking effects of the UAV for a moving target with $V_c=7$ m/s by the LSM method and the proposed method of (39). The simulation results for them are shown in Figure 4. Figure 4a intuitively shows the tracking trajectories of the UAVs. Figure 4b provides the details of control. It can be found that $u_{\psi }$ of the proposed method does not change as smoothly as that of the LSM method, which is to trade off the benefits between making $s_2$ tend to $s_d$ and keeping the target within the perception range. It is obvious that R corresponding to the LSM method exceeds $R_M$ in Figure 4c, where $R_M$ is represented by the black dashed line. However, R corresponding to the proposed method is always less than $R_M$. Furthermore, the changes in $s_2$ are analyzed as well. Here, we introduce the root mean square error (RMSE) to analyze the deviations, which is defined as $\mathbf{RMSE}=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N}{\parallel \Delta s_2\left(i\right)\parallel }_2^2}$, where N represents the total sampling number. After calculating, it can be obtained that the $\mathbf{RMSE}\left(LSM\right)\approx 226.55$ of the LSM method is significantly larger than the $\mathbf{RMSE}\left(pro\right)\approx 153.63$ of the proposed method (unit: px), which decreases by 32.19%. Therefore, the above comparisons confirm the superiority of the proposed method over the LSM method in target tracking.

4.2. Comparative Experiments of ${\mathcal{P}}_{1,soft}$ and the Proposed Method

This part compares the tracking effects of the UAV for a moving target with $V_c=9$ m/s by the proposed method of (39) and ${\mathcal{P}}_{1,soft}$ of (30) without switch. The corresponding simulation results are shown in Figure 5. When the relative horizontal distance exceeds $R_M$, the UAV is expected to reenter the perception range as soon as possible. It can be found that R corresponding to the method of (30) in Figure 5c exceeds $R_M$, and each time out of the perception range is larger than 9 s. However, once R exceeds $R_M$, it will soon drop below $R_M$ under the action of the proposed method. Furthermore, in order to analyze the deviations from $s_2$ to $s_d$, the RMSE is taken into consideration as well. It can be calculated that the $\mathbf{RMSE}\left({\mathcal{P}}_{1,soft}\right)\approx 228.56$ while the $\mathbf{RMSE}\left(pro\right)\approx 238.38$ (unit: px), which just increases by 4.3%. To sum up, compared with the method of (30) without the switch, the proposed method enables the target to quickly return to the perception range with little increase in the RMSE.

4.3. Comparative Experiments on Complex Movements of the Target

To further demonstrate the effectiveness of our method, a comparative simulation involving situations where the target executes complex movements is conducted. Specifically, we compare our proposed method with the PBVS method presented in [4] and the LSM method, and the corresponding results are depicted in Figure 6. The target’s speed and yaw rate are denoted by $v_c$ and ${\omega }_c$, respectively. Based on the trajectories illustrated in Figure 6a and the target velocity in Figure 6b, it is evident that the target’s motion is divided into four distinct phases as follows: (1) uniform linear motion at 0∼100 s; (2) uniform nonlinear motion at 100∼250 s; (3) linear motion at a changing speed at 250∼400 s; (4) nonlinear motion at a changing speed at 400∼500 s. Furthermore, Figure 6c displays the variation in the relative horizontal distance R between the UAV and the target. It is apparent that when employing the method in [4] and the LSM method, R frequently exceeds $R_m$ = 100 m, with the maximum of R reaching 124 m and 158 m, respectively. In comparison, our method effectively prevents R from exceeding $R_m$, with only one instance occurring at about 430 s due to the rapid movement of the target and significant changes in its yaw. Even so, the maximum of R reaches just 109 m, which is smaller than that achieved by the other two methods. Additionally, the changes in the control inputs are depicted in Figure 6d. It is noticeable that the amplitude change of $u_{\psi }$ in our method exceeds that in the other two methods. This is beneficial in preventing R from exceeding $R_m$, thereby enabling clear observation of the target.

5. HITL Simulations and Results

5.1. Simulation Setup

In order to achieve the practical application of the proposed method, a high-fidelity simulation system is built as shown Figure 7, which is based on the XTdrone [27] developed by our team. Note that we developed a full chain of verification from HITL simulation to flight test in the previous work [7,28], so that the program verified in the HITL simulation can be migrated to the physical platform directly, with just minor modifications in the interfaces. The simulation system is mainly composed of four parts: A jetson Xavier NX is adopted as the onboard processor with a robot operation system (ROS) installed, which is used for program deployment and provides interfaces to communicate with Gazebo; PX4 is a platform-agnostic autopilot software that communicates with ROS by MAVROS; QGroundControl (QGC) is a ground control station of PX4 that provides a visual interface to display the flight status and trajectory of the UAV; Gazebo is a 3D dynamic simulator and communicates with PX4 by MAVLink, where a black car represents the tracked target; the resolution of the pan-tilt camera is 1280 px × 720 px, and ${\theta }_t\in [-1.55,0.17]$ (unit: rad). The tiny version of YOLOv7 [29] is adopted in this work to achieve real-time and accurate detection with an average rate of 16 fps. The open-source nonlinear optimization solver NLopt is adopted to solve the MPC optimization problem; the computational time on the onboard processor is illustrated in Figure 8. Each group comprises 500 samples and the result demonstrates that the computational time is consistently below 0.025 s. This fulfills the real-time requirement. Figure 9 displays snapshots of the target tracking in different instants, which include the top views in Gazebo, the detected images with the target bounding box, and the trajectories in QGC. Note that the parameters used in the HITL simulations are the same as those in the numerical simulations.

5.2. Comparative Experiments of the LSM Method [7] and the Proposed Method

In the set of simulations, the tracking effects of the UAV for a moving target with $V_c=5$ m/s by the LSM method and the proposed method are compared; the simulation results are shown in Figure 10 and Figure 11, respectively. It can be observed that the maximum of R reaches 125 m in Figure 10b, which goes far beyond $R_M$. However, R is less than $R_M$ throughout the tracking in Figure 11b. Moreover, the RMSE inside the blue zone shown in Figure 10b is $\mathbf{RMSE}\left(LSM\right)\approx 203.33$, while that in Figure 11b is $\mathbf{RMSE}\left(pro\right)\approx 178.70$ (unit: px), which decreases by 12.11%. The above results show the advantages of the proposed method over the LSM method.

The changes in $(u_1,v_1)$ and the pose of the pan-tilt are presented as well. It can be observed that the curves in Figure 10e are smoother than those in Figure 11e. To account for the phenomenon, the change in $u_{\psi }$ needs to be taken into account. Since the change in $u_{\psi }$ in Figure 10d is smoother than that in Figure 11d, then the attitude change of the UAV in the former will be smoother as well, which leads to a small offset of the feature point in the image. Therefore, it can be found that the range of $u_1$ and $v_1$ are just about [−10, 10] and [−13, 5] (unit: px) in Figure 10e, while they are about [−32, 46] and [−36, 42] (unit: px) in Figure 11e, respectively. In this case, the feature point will return to the image center as long as the pan-tilt rotates smoothly in a small range; the comparison results are given in Figure 10f and Figure 11f, respectively. Even so, the feature point is still close to the image center with the proposed method.

5.3. Comparative Experiments of ${\mathcal{P}}_{1,soft}$ and the Proposed Method

In the set of simulations, the tracking effects of the UAV for a moving target with $V_c=7$ m/s by ${\mathcal{P}}_{1,soft}$ of (30) and the proposed method are compared; the corresponding results are shown in Figure 12 and Figure 13, respectively. It can be found in Figure 12b that when using the control input obtained from ${\mathcal{P}}_{1,soft}$, the maximum of R exceeds 120 m, and the average time that the target returns perception range ($R<R_M$) is about 8.7 s. However, according to Figure 13b, the maximum of R does not exceed 110 m with the aid of the proposed method, and is even less than 105 m after 10 s, and the target will return to the perception range faster. Furthermore, the RMSE inside the blue zone shown in Figure 12c is about $\mathbf{RMSE}\left({\mathcal{P}}_{1,soft}\right)\approx 271.53$, while in Figure 13c is about $\mathbf{RMSE}\left(pro\right)\approx 258.17$ (unit: px), which decreases by 4.9%. Compared with the results of numerical simulation in Section 4.2, it can be found that the changes in the RMSE are different. This is caused by two factors: one is the different target speed; the other is that there is a difference between the given speed and the real speed in the HITL simulation. Due to the second factor, the UAV is unable to achieve the expected motion to make the target remain in the perception range; then, R in both Figure 12b and Figure 13b exceeds $R_M$. In comparison, R will soon drop below $R_M$ in Figure 13b, and then $s_2$ will approach $s_d$ to decrease the RMSE with the aid of the proposed method. Therefore, the RMSE in Figure 13b is smaller than that in Figure 12b. In terms of the change for $u_1$ and $v_1$, it can be observed from Figure 12e and Figure 13e that their ranges are almost the same. Note that the sharp jitter of the curves is caused by the attitude change of the UAV and the time delay of the pan-tilt control. Figure 12f and Figure 13f present the changes in the pan-tilt.

5.4. Tracking Target with Nonlinear Motion

In the set of simulations, two typical nonlinear motions of the target are considered, including circular motion and sharp turns. The tracking results are shown as Figure 14, where the target moves at a speed of 5 m/s. According to Figure 14b, it is obvious that R is always less than $R_M$. When the target moves in a sharp turn, it can be observed that R exceeds $R_M$ at about 80 s in Figure 14d. This is because the sharp turn of the target changes the relative movement trend between them, which leads to increase in R. Even so, the proposed method can make the target return to the perception range immediately within about 4 s. Based on the above simulations, the effectiveness of the proposed method is further verified.

6. Conclusions

In this paper, we have proposed an online MPC-based IBVS method for a fixed-wing UAV with a 2-DOF pan-tilt camera for target tracking. The method enables the UAV to achieve persistent tracking of the target, while concurrently maintaining the target near the image center. With the aid of MPC, the dynamic constraint of the UAV and the perception constraint of the camera can be considered when solving the optimization problem. After that, a soft constraint method is designed for practical implementation combined with a warm start.

Furthermore, a switching-based control strategy is proposed to return the target to the perception range quickly once it is outside the range, and the asymptotic stability for the controller is proven. Extensive comparative simulation experiments including numerical simulations and HITL simulations were conducted, which demonstrated the effectiveness and superiority of the proposed method. In the future, estimation of the target speed will be considered, and real flight tests will be conducted as well.