Adaptive Visual Servoing for Obstacle Avoidance of Micro Unmanned Aerial Vehicle with Optical Flow and Switched System Model

Abstract

A vision-based adaptive switching controller that uses optical flow information to avoid obstacles for micro unmanned aerial vehicles (MUAV) is proposed in this paper. To use the optical flow to indicate the distance between the MUAV and the environment, we propose an algorithm with multi-thread processing such that the optical flow information is obtained reliably and continuously in the entire camera field of view. The flying behavior of considered MUAV is regarded as a switching system when considering different flying modes during the mission of obstacle avoidance. By the required flight direction for obstacle avoidance specified by the detected optical flow, an adaptive control scheme is designed to track the required trajectory in switching modes. The simulation result shows the tracking performances of the adaptive control with the switching system. The experiment of the whole system is completed to verify the obstacle avoidance capability of our system.

1. Introduction

Recently, 3D robots have presented an increasingly prosperous market and rapid development. Many industries, such as automobiles, automation, surveillance, medical, etc., require robots to replace humans for various purposes. As for 3D robots, a lot of research of this field is specifically contributing to the aerial platforms. A micro unmanned aerial vehicle (MUAV) is an aircraft that can fly autonomously in response to dangerous situations, such as fire sites, chemical or nuclear disaster areas, and battlefields. Due to its low cost, small size and light weight, MUAV is light in weight and flexible, making it particularly suitable for performing tasks in indoor environments. In this article, we consider the navigation control of MUAV flying through stairs between building floors. We focus on designing a novel vision-based adaptive controller for MUAV obstacle avoidance.

To avoid obstacles during the flight missions, we need to estimate the distance through sonar, laser ranging, and infrared sensors to observe the environment around the MUAV. Nonetheless, the ranging or depth sensors are usually too big or have high power consumption such that they are hard to be mounted on an MUAV. Therefore, low-cost, small-size pinhole cameras are commonly used on MUAVs. Stereo cameras that are used on MUAVs [1,2,3] can also help to obtain the information for depth around the camera in the obstacle avoidance task. Considering the single camera, ground vehicles with a monocular camera installed at a fixed height can be used to estimate the distance to the off-road area [4]. When a single camera is installed on the MUAV, the optical flow is usually estimated to denote the distance between the camera and objects [5].

Optical flow has been widely applied in computer vision and robotics. The object tracking task may involve the feature point tracking of optical flow in the motion-based tracking method [6,7]. The optical flow can also be used to construct the environmental maps and obtain the motion of the camera itself [2,8]. Recently, some research has developed parallel computing algorithms on the graphics processing unit (GPU) [9] to quickly obtain optical flow; however, the cost of additional hardware and hard programming make its implementation difficult. In this work, we use a single camera to quickly and reliably extract the optical flow information in front of the MUAV, and then, the optical flow will help to generate the input commands designed by the controller, followed by processing them in parallel on the CPU computing platform. With the feedback signals extracted from the vision sensor and then utilized to control the motion for desired purpose, visual servoing is widely applied to different applications and robotic systems. In the visual servoing systems [10,11], the camera is mounted on a moving platform, such as the robot arm or ground/aerial vehicle, which is usually controlled to follow the target through the feedback of the target information in the images. In [12,13], the visual servoing system can assist the robot arm and agriculture robot to accomplish the tasks and achieve the satisfactory performance.

On the other hand, there are many works discussing the control of MUAV with various models and methods. The model and design of the considered MUAV is discussed in detail in [14,15], where some adaptive controller design approaches for different models are provided. In [16,17], the nonlinear control methods are proposed for MUAV with uncertain parameters. The state feedback adaptive controller for quadrotors with varying centers of gravity is also presented in [18]. In [19,20,21], a robust adaptive control is designed for MUAV with output feedback adaptive control. In [22], a visual adaptive control is proposed, in which they consider the vehicle dynamics and the image information to construct the controller. In [23], the visual servoing control is considered for an underactuated rotorcraft UAV to regulate its translational motion and heading orientation. Besides, variable structure control is also robust to the disturbances and is applied to the UAV system, as discussed in [24,25,26]. Most research considers the state feedback design, and it is assumed that the states can be obtained from the sensor. However, such an approach is expensive and may be impractical. In [27], the output feedback visual servoing control is employed to navigate the line following of UAV. Here, we treat the MUAV system as a black box with a switching transfer function and only employ the sensing information from a single camera. Moreover, the output feedback control of the switching system is designed with the observed image sequence, which is more practical for the implementation of the MUAV system.

In this work, we implement the obstacle avoidance task of the MUAV system on which the front camera installed, and we use the adaptive control with optical flow information. The feature point tracking by the Kanade–Lucas–Tomasi (KLT) [28,29] algorithm is first applied to generate the information of optical flow. However, the original KLT tracker has some limitations; for example, the tracked feature points should be in the texture image block. When tracking a common image block, the corresponding relationship of this point is only correct between several consecutive frames. Therefore, if we want to detect the optical flow for obstacle avoidance in the entire image space, the points will be drawn uniformly and quickly redrawn. Here, we applied the multi-threaded processing scheme to supplement the optical flow when redrawing. Since objects closer to the camera will generate greater optical flow, these image areas with large optical flow can be regarded as an infeasible flight space for obstacle avoidance. In addition, because it is undesirable to change the original flight direction of the MUAV mission during obstacle avoidance, the flight strategy of the MUAV is chosen to perform the forward and lateral movements without moving in the yaw direction. After selecting the desired collision avoidance flight direction, the forward and lateral actions will be activated in sequence to reduce the airflow disturbance from the MAV itself. Therefore, the dynamics of the MUAV are modeled as a switching system. The main contributions of this work are that we use the optical flow information by the vision sensor and design an adaptive controller with the output in which the transfer function is unknown and composed of switching parameters. Many existing results related to MUAV control use information about the entire state to control the system. Instead of using the state-space model, the plant of MUAV is considered as a black box with a switching system description under different operating modes.

The rest of this article is organized as follows. In Section 2, the multi-thread KLT tracker with uniform point distribution is proposed such that the optical flow information can be obtained in a more sufficient and reliable way. Section 3 describes the obstacle avoidance flight strategy with optical flow and the considered problems. Here, we also propose visual servoing and adaptive control of the switching system for MUAV. Section 4 provides some simulation and experimental results. Section 5 gives the conclusion.

2. Optical Flow Estimation for Detecting Obstacle

Since only a camera is mounted on the MUAV to capture the front view, the feasible flight path must be decided from the observed image frames. An MAV can estimate the relative distance between itself and an object by moving in the scene, which is based on the concept of visual odometry (VO) [30] or structure of motion (SFM) [31]. The image scene consistency between consecutive frames employs the KLT feature point tracker [29]. The relative distance between the MUAV and the surrounding object can be denoted by the feature point displacement, which is also called optical flow. The traditional KLT tracker is easy to be misled by disturbers and may not estimate abundant optical flow in the whole image frame. Therefore, we improve the KLT tracker to be a multi-threaded process with uniform point distribution such that the optical flow in whole image frame can be quickly and consistently obtained.

2.1. The Construction of Uniform Point Distribution

The traditional KLT tracker only adopts the significant feature points, which could be reliably identified across image frames, such as Harris corner detection and scale invariant feature transform (SIFT). As illustrated in Figure 1a, the selected corner points can be recognized robustly for a long time by using the KLT tracker with pyramid computation architecture [32]. However, the obtained optical flow is limited to the textured area. The image region with uniform appearance cannot detect the optical flow, where the relative distance information will be lost. The available flight space of MUAV would not be clearly identified for the visual servoing of obstacle avoidance.

Therefore, the optical flow information that will be utilized for the obstacle avoidance must fill the entire front camera’s field of view. Instead of selecting the corner points, we uniformly draw the points, which are then tracked by the KLT method in the image, as shown in Figure 1b. The points are also drawn on the uniform appearance image region. However, as demonstrated in Figure 2, the points drawn on the uniform appearance region as Figure 2a are easily misled by the similar neighbors after KLT tracking over than ten frames. These wrong tracked points also result in the incorrect optical flow as shown in Figure 2b. Since the life cycle of insignificant point tracking is very short, in order to improve their reliability, the uniformly drawn points must be quickly redrawn after KLT tracking several frames. As demonstrated in Figure 3, most of the points uniformly drawn in Frame #1 are successfully tracked in the following two frames (#2 and #3), even in the uniform appearance image region. In addition, during the MUAV flight, the identified points may disappear due to moving out of the camera’s field of view or being occluded by other objects. As shown in in Frame #1 and #4, considering the fast redrawing frequency can also quickly compensate the optical flow information when the points are missing or tracking is lost.

2.2. Multi-Thread Processing

The KLT tracker with high frequency of point uniformly redrawing brings the disadvantage that identified optical flow is all erased when the tracked points are redrawn. At that time, the MUAV has no information to navigate the MUAV for obstacle avoidance. Here, a multi-thread processing for continuously obtaining the optical flow information is proposed to solve this problem. As shown in Figure 4, multiple threads are set to execute the pyramid Lucas–Kanade algorithm at the same time, and the point redrawing time of each thread is different. Assuming that there are $N$ KLT threads, each thread redraws the points at every $M$ frames, which cannot be too long, as described above. If the first thread redraws the point at frame $t$, the following $i\mathrm{th}$ thread will redraw the points at frame $t+(i-1)M/N$. The image displacement is more pronounced on threads with longer tracking points. Therefore, the considered optical flow uses the thread $i^{\ast }$ farthest from its point redrawing moment and is defined as

$$i^{\ast }=\arg \underset{i}{\max }\left(t-T_i\right),$$

(1)

where $t$ is current time, and $T_i$ is the recent point redrawing instant of the $i\mathrm{th}$ thread. The tracking process of multiple KLTs running one image frame in serial programming will be very slow, so point tracking is easy to fail. Fortunately, with the multi-core architecture of CPU, we can assign each tracking thread to a core of CPU. Then, as illustrated in Figure 5a, the optical flow in the entire camera field of view can be reliably and continuously detected with real-time performance.

Since the object closer to the camera will cause the larger image displacement across image frames, the optical flow estimated through the multi-thread process can be used to avoid obstacles in front of the MUAV. By calibrating the camera on the MUAV, the maximum allowable length of the optical flow, which denotes the minimum safe distance between a stationary object and the fastest-moving MUAV, can be measured in the beginning. As presented in Figure 5, the optical flow with length shorter than the maximum allowable length will result in the feasible flight space. The image centroid $(x_c,y_c)$ of the estimated feasible flight space can then be employed to navigate the flight trajectory of the MUAV.

3. Visual Servoing with Adaptive Control of Switched Systems

We discuss the visual servoing problem of MUAV that uses adaptive control to implement obstacle avoidance commands. In the first stage, the MUAV is modeled as a third-order system with different parameters, when operating in different modes, such as rotation or translation. Then, the controlled object is regarded as a switching system, and an adaptive control design is proposed for the switching linear system with unknown parameters in the MUAV model.

3.1. System Description and Problem Formulation

Consider a quad-rotor MUAV equipped with a camera to capture the front view of the MUAV. Assume that the optical center of the camera is the same as the centroid of the MUAV, and the camera coordinates are the same as the coordinates $(x_b,y_b,z_b)$ of the MUAV body, as shown in Figure 6. To maintain the original flight direction of the MUAV mission, its heading direction will not change during the obstacle avoidance. The control input adopts the up and down speed $W$ along the $z_b$ axis, the lateral speed $V$ along the $y_b$ axis, and the forward speed $U$ along the $x_b$ axis. The horizontal image position $x_c$ is used to determine the horizontal movement of the MUAV, and the vertical image position $y_c$ is used to draw the vertical movement. Here, the speed $W$ is only used to keep the height fixed, and no information of $y_c$ is needed. The forward and lateral speeds are designed to avoid obstacles. As shown in Figure 7, given image observation $x_c$ and camera parameter $f$, tangent function of the flight direction ${\alpha }_t$ required to avoid obstacles in the transverse plane is expressed as

$$\tan {\alpha }_t=\frac{x_c}{f}=\frac{Y_t}{X_t},$$

(2)

in which $X_t$ and $Y_t$ are the needed forward and lateral displacement at current time instant $t$, respectively. In practical implementation, values of $X_t$ and $Y_t$ during one sampling period will be bounded in a feasible range, and the forward displacement $X_t$ is first selected with the inverse proportional to the maximum length of detected optical flow $d_t$ or proportional to the minimum estimated distance $D_t$ of the surroundings [6] at every frame. Notice that the lateral displacement $Y_t$ is equal to zero when the desired flying direction is exactly in front of the MUAV, i.e., ${\alpha }_t=0$.

By the operating behavior and dynamics of MUAV system, we make the assumption that the dynamical model of MUAV is approximated by a third-order transfer function whose relative degree is 1, which is from the observation of the experimental setting with the step reference. Through system identification, the modes of the quadrotor can be modeled as the following different transfer functions. The forward mode TF1 is $G_{p1}(s)$, which is the forward translation in speed $U_t$. The transverse mode TF2 is $G_{p2}(s)$, which is the translation in speed $V_t$.

From the observation point of view, the MUAV is easy to be disturbed by the airflow in the indoor environment when executing the commands of the two modes at each instant. Therefore, it is desirable to activate the flight mode in turn at each moment. The entire structure of the closed-loop system is given in Figure 8. The considered system decouples the $x_b$ axis and the $y_b$ axis separately, and they correspond to the forward mode and the lateral mode, respectively. For the $x_b$ axis, suppose the system is in forward mode, then its dynamics will be TF1. If the system changes to the lateral mode, its model will be TF2. The same is true for the $y_b$ axis. It is worth mentioning that for the $x_b$ axis, the reference $X_{d,t}$ is zero in the lateral mode (TF2) and remains unchanged in the forward mode (TF1). In contrast, when considering the $y_b$ axis, the reference $Y_{d,t}$ is zero in the forward mode (TF1) and remains unchanged in the lateral mode (TF2). In Figure 8, blocks $C_x$ and $C_y$ are the controllers with respect to $x_b$ axis and the $y_b$ axis. Due to the inaccuracy of the approximate model and the uncertainty, the MUAV system parameters are regarded as unknown, and a robust adaptive control design is proposed to control the switching plants with unknown parameters in each mode.

3.2. Adaptive Control for Switched System

As shown in Figure 8, we design the adaptive control for the MUAV system with different modes and regard it as a switching system. For simplicity, we will discuss the general form in (3) below, which can be applied to the situation considered in Figure 8. For switching system with the transfer functions

$$y=W_{p\sigma (t)}(s)[u]=\frac{Z_{p\sigma (t)}(s)}{R_{p\sigma (t)}(s)}[u],$$

(3)

where $\sigma (t)\in P=\{1,2\}$ is the switching signal, and

$$Z_{pi}(s)=s^m+b_{m-1}^i{s}^{m-1}+\cdots +b_1^{i}s+b_0^i,$$

$$R_{pi}(s)=s^n+a_{n-1}^i{s}^{n-1}+\cdots +a_1^{i}s+a_0^i,\hspace{1em}i=1,2$$

The output y in (3) may be represented by $X_t$ or $Y_t$, and the input u is $U_t$ or $V_t$, respectively.

The $W_{pi}(s), i=1,2$ are assumed to be strictly proper with unknown parameters. Our control goal is to design the adaptive control so that the output signal will track the desired signal $y_m$. The signals $y_m=X_{d,t}$ and $y_m=Y_{d,t}$ are the references for the $x_b$ axis and the $y_b$ axis, respectively. In order to design an adaptive controller for the switching system (3), we pose the following conditions which are common in adaptive control system:

(A1)

The relative degree of the transfer function is $\rho =n-m=1$ and the order is n.

(A2)

The first $\rho$ time derivatives of the signal $y_m$ are all known and bounded.

(A3)

The transfer functions are completely controllable and observable.

(A4)

The signs of $b_m^i$ are known and are the same for all $i\in P$.

(A5)

The unknown parameters are bounded in a compact set which is known.

In order to design the adaptive control of the switching system, we adopted the tuning function method introduced in [29]. An output feedback backstepping control based on a variable structure is designed to make the tracking error under any switching signal converge to zero. In the first step, we use the adaptive observer with K filter [33] to estimate the states. System (3) is rewritten as

$$\begin{array}{l}\dot{x}=A_{0}x+F{(y,u)}^T{\theta }^{\sigma }\\ y=C^{T}x\end{array}$$

(4)

Here,

$$A_0={\left[\begin{array}{cc}\begin{array}{c}0\\ ⋮\end{array}& I_{n-1}\\ 0& \begin{array}{cc}\cdots & 0\end{array}\end{array}\right]}_{n\times n},$$

$${\theta }^{\sigma }={\left[{\begin{array}{cc}{(b^{\sigma })}^T& (a^{\sigma })\end{array}}^T\right]}^T={\left[\begin{array}{llllll}{b}_m^{\sigma }\hfill & \cdots \hfill & b_0^{\sigma }\hfill & a_1^{\sigma }\hfill & \cdots \hfill & a_n^{\sigma }\hfill \end{array}\right]}^T,$$

and

$$F{(y,u)}^T={\left[\begin{array}{ccc}\begin{array}{c}{0}_{(\rho -1)\times (m+1)}\\ I_{(m+1)\times (m+1)}\end{array}& |& -I_{n\times n}\cdot y\end{array}\right]}_{n\times (n+m+1)}$$

Then, we define $e_i$ as the ith coordinate vector, in which all components are zero, but the ith component is 1. Let $k={[k_1,\dots ,k_n]}^T$ such that $A_0=A-k{e}_1^T$ is Hurwitz, by which there will exist a positive definite matrix $P=P^T>0$ that satisfies

$$A_0^{T}P+P{A}_0=-I.$$

(5)

The K-filters design observer is:

$$\begin{array}{l}\dot{\xi }=A_0\xi +ky\\ {\dot{\Omega }}^T=A_0{\Omega }^T+F{(y,u)}^T\end{array}$$

(6)

Define the virtual state estimator to be

$$\widehat{x}=\xi +{\Omega }^T{\theta }^{\sigma },$$

(7)

and let the error of estimation as $ϵ=x-\widehat{x}$. The value of $ϵ$ will decrease to zero exponentially with $\dot{ϵ}=A_0ϵ$. According to the form of $F{(y,u)}^T$, we denote ${\Omega }^T=[v_m,\dots ,v_0,\Xi ]$, where $v_j\in R^n,\hspace{0.17em}j=0,1,\dots ,m,$ are column vectors that are found from a filter of input u, and $\Xi \in R^{n\times n}$ are obtained from the filter of output $y$. The K-filters are implemented as follows

$$\begin{array}{l}\dot{\eta }=A_0\eta +e_{n}y,\hspace{1em}\dot{\lambda }=A_0\lambda +e_{n}u\\ {\Omega }^T=[v_m,\dots ,v_0,\Xi ],\\ v_j={(A_0)}^j\lambda , j=0,1,\dots ,m\\ \Xi =-[{(A_0)}^{n-1}\eta ,\dots ,A_0\eta ,\eta ]\\ \xi =-{(A_0)}^n\eta \end{array}$$

Based on the above discussion, we design a backstepping adaptive controller with the K-filters and the tuning function approach. As the relative degree is one, according to (4),

$${\dot{x}}_1=x_2-a_1^{\sigma }y+b_m^{\sigma }u=x_2-y{e}_1^T{a}^{\sigma }+b_m^{\sigma }u,$$

(8)

From (5) and (8), we can represent $x_2$ as

$$x_2={\xi }_2+[v_{m,2},v_{m-1,2},\dots ,v_{0,2},{\Xi }_{(2)}]{\theta }^{\sigma }+{ϵ}_2,$$

(9)

The sub-index 2 stands for the second component in the column vector $v_j,j=0,1,\dots ,m,$ and ${\Xi }_{(2)}$ stands for the second row of $\Xi$. Thus,

$$\begin{array}{l}{\dot{x}}_1={\xi }_2+[v_{m,2},v_{m-1,2},\dots ,v_{0,2},{\Xi }_{(2)}-y{e}_1^T]{\theta }^{\sigma }+{ϵ}_2+b_m^{\sigma }u\\ \hspace{1em}={\xi }_2+v^T{\theta }^{\sigma }+{ϵ}_2+b_m^{\sigma }u\end{array}$$

(10)

Denote the output error as $z_1=y-y_m$, then

$${\dot{z}}_1={\xi }_2+v^T{\theta }^{\sigma }+{ϵ}_2+b_m^{\sigma }u-{\dot{y}}_m,$$

(11)

Thus, the control is chosen as

$$u=\widehat{\gamma }{\overline{\alpha }}_1,$$

(12)

where $\widehat{\gamma }$ is the estimate of ${\gamma }^{\sigma }=1/(b_m^{\sigma })$, and

$${\overline{\alpha }}_1=-c_1{z}_1-d_1{z}_1-{\xi }_2-v_a^T\widehat{\theta }+{\dot{y}}_m,$$

(13)

where $v_a={[|v_1|,\dots ,|v_{n+m+1}|]}^T$. To handle the parameters with switching, our control parameters are designed by the variable-structure-based adaptive laws

$$\begin{array}{l}{\widehat{\theta }}_i=\mathrm{sgn}(z_1){\overline{\theta }}_i,\hspace{1em}{\overline{\theta }}_i>\left|{\theta }_i\right|,\\ \widehat{\gamma }=-\overline{\gamma }\mathrm{sgn}(b_m^{\sigma })\mathrm{sgn}({\overline{\alpha }}_1{z}_1),\hspace{1em}\overline{\gamma }>1/\left|b_m^{\sigma }\right|\end{array}$$

(14)

$i=1,2,\dots ,n+m+1.$ The adaptive control is proposed by the Lyapunov theorem, where the Lyapunov function candidate is

$$V_1=\frac{1}{2}{(z_1)}^2+\frac{1}{2d_1}{ϵ}^{T}Pϵ$$

(15)

Thus,

$$\begin{gathered} \begin{array}{l}{\dot{V}}_1=-c_1{(z_1)}^2+(v^T{\theta }^{\sigma }-v_a^T\widehat{\theta })z_1-b_m^{\sigma }{\tilde{\gamma }}^{\sigma }{\overline{\alpha }}_1{z}_1-\frac{1}{4d_1}{ϵ}^Tϵ\\ \hspace{1em}\hspace{1em}-d_1{(z_1)}^2+z_1{ϵ}_2-\frac{1}{4d_1}{ϵ}^Tϵ\end{array} \\ \begin{array}{l}\hspace{1em}\le -c_1{(z_1)}^2+(v^T{\theta }^{\sigma }-v_a^T\widehat{\theta })z_1-b_m^{\sigma }{\tilde{\gamma }}^{\sigma }{\overline{\alpha }}_1{z}_1-\frac{1}{4d_1}{ϵ}^Tϵ\\ \hspace{1em}=-c_1{(z_1)}^2+{\displaystyle {\sum }_{i=1}^{2n}({\theta }_i^{\sigma }{v}_i{z}_1-{\overline{\theta }}_i|v_i{z}_1|)}\\ \hspace{1em}\hspace{1em}-(|b_m^{\sigma }\left|{\gamma }^{\sigma }\right|{\overline{\alpha }}_1{z}_1|)-\frac{1}{4d_1}{ϵ}^Tϵ\\ \hspace{1em}\le -c_1{(z_1)}^2-\frac{1}{4d_1}{ϵ}^Tϵ<0.\end{array} \end{gathered}$$

(16)

Note that in the first inequality in (16), the following relation is used

$$\begin{array}{l}-d_1{(z_1)}^2+z_1{ϵ}_2-\frac{1}{4d_1}{ϵ}^Tϵ\\ =-d_1{(z_1-\frac{1}{2}{ϵ}_2)}^2-\frac{1}{4d_1}{({ϵ}_1+{ϵ}_3+\dots +{ϵ}_n)}^2\le 0.\end{array}$$

(17)

Here, ${\tilde{\gamma }}^{\sigma }={\gamma }^{\sigma }-\widehat{\gamma },\hspace{0.17em}{\widehat{\theta }}^{\sigma }={\theta }^{\sigma }-\widehat{\theta }$, and we use (14) and the relation

$${\dot{z}}_1={\xi }_2+v^T{\theta }^{\sigma }+{ϵ}_2-b_m^{\sigma }{\tilde{\gamma }}^{\sigma }{\overline{\alpha }}_1+{\overline{\alpha }}_1-{\dot{y}}_m,$$

(18)

for our derivations. By the analysis, we obtain that $z_1=y-y_m$ and $ϵ$ will converge to zero exponentially and, thus, guarantee that $y=x_1$ is bounded. We then show that the remaining states $x_2,\dots ,x_n$ are bounded by the filters. By (13) and the boundedness of $z_1$, the control $u$ is bounded. Since $A_0$ is stable, and $\lambda$ is bounded, this implies the boundedness of $v_j,j=0,1,\dots ,m,$. Furthermore, $\eta$ is bounded due to the stability of $A_0$ and boundedness of y. Follow the above discussion and with (10), the boundedness of $x_2$ is guaranteed. Thus, the boundedness of $x_3,\dots ,x_n$ is obtained by the boundedness of ${\Omega }^T$. From the analysis, the whole system is stable, and the output signal will asymptotically track the reference signal.

Remark 1. In the general relative degree case of tuning function adaptive control design, an assumption on minimum phase is required [33]. Here, this assumption is unnecessary, since we only consider the case where the relative degree is 1.

4. Simulation Analysis and Experimental Results

4.1. Simulation Analysis

The following simulation is first performed to verify the results of Section 3. In order to demonstrate the robustness of proposed controller in switching modes, we assume that the reference $X_{d,t}$, the trajectory in $x_b$ axis, and mode of the plane are changing every second, which are given in Figure 9. Similar to results in the literature such as [34], the transfer function we consider is as follows, which is obtained through experiments.

$$G_{P1}(s)=\frac{-1.377S^2+3.103S-33.87}{S^3+6.537S^2+20.77S-0.6013}\hspace{1em}\hspace{1em}(TF1),$$

$$G_{P2}(s)=\frac{-2.685S^2+11.29S+102.7}{S^3+5.975S^2+8.297S-0.1247}\hspace{1em}\hspace{1em}(TF2),$$

We can see that no matter how the controlled object mode is switched, the output will gradually track the reference. Nevertheless, some transience caused by the switching effect of the reference trajectory exist. The transient behavior here is not caused by switching parameters, but by the reference. The switching effects are suppressed by the robust variable structure adaptive controller, and thus, in the simulations, there is no obvious switching effect on the state trajectory. It is similar to the response $y_b$. In addition, in (11), we utilize the condition ${\dot{y}}_m=0$ for the case under consideration because of the measure zero property of the nondifferentiable points.

4.2. Experimental Results with Visual Servoing

As discussed in Section 2, our visual detection of obstacles uses the mentioned optical flow estimation. The multi-threaded process for detecting the optical flow is realized on four threads of a PC with quad-core 3.3GHz CPU, and each thread redraws the points turn by turn in every four frames. We use Drone 2.0 of Parrot AR as the MUAV to test our visual servoing method. The real-time performance, i.e., 30 frames per second, can be achieved for the computation process of visual detection and visual servoing.

In Figure 10, the blue circle represents the desired flight direction in the image frame, and the MUAV flight snapshot received from the third view is shown in Figure 11. The original flight direction to the MUAV is chosen to pass through the corridor indoor. In the flight procedure, the MUAV first detects the brown box in the corridor. The desired flight direction is specified by the center of mass $(x_c,y_c)$ of feasible flight space for avoiding the obstacles. The feasible flight space is adjusted to the left, and the MUAV takes turns to activate the forward mode and the lateral mode through the box. However, this avoidance caused the MUAV to hit the wall at Frame #562. As previously mentioned, the wall caused a larger optical flow as the MUAV moves forward; the flight direction then changed to the right at Frame #585. The MUAV will also avoid the desk after Frame #602. With the vision-based obstacle avoidance method and the adaptive control of switching modes, the MUAV can avoid the obstacles and maintain the moving direction such that it can successfully traverse the corridor.

Moreover, as shown in Figure 12, we utilize the proposed visual servoing methodology to implement the moving obstacle avoidance task of MUAV. The blue circle denotes the desired flying direction in the image domain. We can see that from Frame #1 to Frame #170, the original desired flying direction is going toward the center of the corridor. The moving obstacle (human body) is observed by the multi-thread optical flow estimation in Frame #172, and the desired flying direction successfully lead the MUAV to pass the obstacle from Frame #177 to Frame #189. Afterwards, from Frame #199 to Frame #238, the MUAV can go back and continue the original task of heading toward the corridor.

5. Conclusions

In this research, we propose a vision-based obstacle avoidance system with respect to the MUAV flight control, including visual flight control and visual obstacle detection. In order to determine the infeasible flight space, obstacle detection considers the optical flow information. In addition, for having sufficient optical flow with KLT, we need the feature points to be spread evenly in entire image frames. We develop the multi-threaded process by feature point tracking with KLT and take turns to quickly redraw the points. Thus, reliable optical flow is extracted. According to the information from optical flow, we select the direction of desired flight trajectory, and the MUAV moves forward and laterally to avoid obstacles. This paper proposes a visual servo controller with adaptive law for MUAV. As for the obstacle avoidance tasks, MUAV is modeled by the switching system, and we design the adaptive control to fulfill our control goals. Many existing results consider state feedback control and use a lot of sensors; here, we only consider one single camera to be the sensor for the proposed control system.

In future works, a visual servoing with a nonlinear switched model will be considered for the MUAV system. More flying modes of the switched system will be modeled for the MUAV behaviors in the indoor environment, such as the wondering, corridor or staircase crossing, object tracking, etc. The visual information of various environments will also be extracted as the basis of visual servoing in the corresponding scenarios.