A Two-Step Controller for Vision-Based Autonomous Landing of a Multirotor with a Gimbal Camera

Abstract

This article presents a novel vision-based autonomous landing method utilizing a multirotor and a gimbal camera, which is designed to be applicable from any initial position within a broad space by addressing the problems of a field of view and singularity to ensure stable performance. The proposed method employs a two-step controller based on integrated dynamics for the multirotor and the gimbal camera, where the multirotor approaches the landing site horizontally in the first step and descends vertically in the second step. The multirotor and the camera converge simultaneously to the desired configuration because we design the stabilizing controller for the integrated dynamics of the multirotor and the gimbal camera. The controller requires only one feature point and decreases unnecessary camera rolling. The effectiveness of the proposed method is demonstrated through simulation and real environment experiments.

1. Introduction

Multirotors have gained popularity as a preferred platform for unmanned aerial vehicles (UAVs) due to their maneuverability, ability to take off vertically and land, low cost, and minimal maintenance requirements [1]. Recently, researchers have begun equipping multirotors with cameras to enable autonomous and precise control through visual input from the surrounding environment [2,3]. This trend has become widespread with the development of lightweight and small cameras. One area of research in this field is vision-based autonomous landing (VBAL), where a UAV uses a camera to autonomously detect, approach, and land on a designated site [4].

Various techniques are used in VBAL, most of which rely on visual servoing for UAV control. Visual servoing refers to the use of an image alone for UAV control, although additional sensor information such as an ultrasonic sensor [5], inertial measurement unit (IMU) [6], GPS [7], and LiDAR [7] may be additionally utilized in some cases to obtain the depth of a target, attitude, position, and altitude of the multirotor. Using this visual servoing technique, we can conduct a sequence of procedures that involves detecting a landing site, approaching it at close range, and landing safely.

Several studies on VBAL employ UAVs equipped with fixed cameras positioned mostly downward. For instance, Kumar [5] uses a fixed stereo camera and an ultrasonic sensor to determine the relative position of the quadrotor with respect to the landing site and applies four basic visual servoing controllers independently. Many recent studies have introduced the concept of a virtual camera. Lee et al. [8] devise a VBAL technique utilizing an adaptive sliding mode controller and a virtual camera that requires at least four point features in an image, while Zheng et al. [6] design a controller that employs a virtual camera and the backstepping approach, which requires at least three point features. Li et al. [9] propose a VBAL control scheme for an arbitrarily oriented planar visual target using the virtual camera and the image moment feature that still requires at least two coplanar feature points. One issue with the fixed camera is that the target features within its field of view can disappear during flight. Some solutions to this problem include using a control barrier function [10] or nonlinear model predictive control [11], which employ optimization with a visibility constraint to generate control inputs while keeping the landing site in the field of view. Recently, visual servoing control using reinforcement learning has also been conducted to keep the target within the field of view [12].

An alternative approach to address the issue is to mount a gimbal camera on the UAV [13]. As the UAV moves along any trajectory, the gimbal camera adjusts its attitude to keep the landing site close to the center of the image. Some VBAL systems that use gimbal cameras have separate controllers for the camera and UAV. These systems typically use an image-tracking algorithm for the gimbal camera controller, which is independent of the UAV dynamics. Examples of such systems include those using target tracking error dynamics [14], proportional navigation [15], or classical visual servoing with a tilting camera [7], model predictive control with a pan-tilt camera [16]. In these systems, the image tracking algorithm begins to control the gimbal camera after the target moves away from the center, which implies that the gimbal moves after the UAV moves. Thus, fast image tracking is necessary to prevent losing the landing site, but using a high gain may exacerbate the impact of noise and cause control input chattering. As Jiang et al. [14] have noted, there is a trade-off between speed and stability when tracking the image after the camera deviates from the target. Cabecinhas et al. [17] have proposed an attitude estimator and an integrated controller that considers both a pan-tilt camera and the quadrotor dynamics, but it still necessitates at least four feature points.

The focus of our study is to present a two-step VBAL controller for a multirotor with a gimbal camera to address the aforementioned issues. The integrated dynamics of the multirotor and the gimbal camera are derived using one feature point, enabling them to converge simultaneously to the desired configurations using feedback linearization. We show theoretically that our proposed method allows the multirotor to stably approach, track a target, and land from any starting position (even at a low altitude and over a long horizontal distance from the target). Importantly, this implementation relies solely on onboard sensor information. In the first step, the multirotor moves horizontally toward the landing site. Here, we define a five-dimensional state consisting of the image coordinates of one feature point, the depth of the feature point, the camera rolling variable relative to the world frame, and the altitude of the multirotor. As the multirotor gets close enough to the landing site horizontally, we initiate the second step, which we call vertical approaching. Here, we define a three-dimensional state consisting of the virtual camera coordinates of one feature point and the altitude of the multirotor. We conduct the method through real experiments, validating that the proposed states reach their desired values, leading to successful landings at the landing site.

This paper is organized as follows. Section 2 describes our two-step controller along with the derivation of the integrated dynamics. Section 3 shows the performance of the controller in simulation and real environments. The conclusion is presented in Section 4.

2. Materials and Methods

2.1. Notation

The set of all $3\times 3$ rotation matrices is defined as $SO\left(3\right)=\{A\in {\mathbb{R}}^{3\times 3}\mid A^{T}A=I,detA=1\}$. The set of all $3\times 3$ skew symmetric matrices is defined as $\mathfrak{so}\left(3\right)=\left\{A\in {\mathbb{R}}^{3\times 3}\mid A^T+A=0\right\}$. The hat map $\wedge :{\mathbb{R}}^3\to \mathfrak{so}\left(3\right)$ is defined as

$$\widehat{\Omega }=\left[\begin{array}{ccc}0& -{\Omega }_3& {\Omega }_2\\ {\Omega }_3& 0& -{\Omega }_1\\ -{\Omega }_2& {\Omega }_1& 0\end{array}\right]$$

for $\Omega ={[{\Omega }_1,{\Omega }_2,{\Omega }_3]}^T\in {\mathbb{R}}^3$. The standard basis vectors are denoted as $e_1={[1,0,0]}^T,e_2={[0,1,0]}^T$, and $e_3={[0,0,1]}^T$.

 Lemma 1. 

1. 

$\widehat{u}v=u\times v=-\widehat{v}u$ for all $u,v\in {\mathbb{R}}^3$.

2. 

$R\widehat{\Omega }{R}^T=\widehat{(R\Omega )}$ for all $R\in SO\left(3\right)$ and $\Omega \in {\mathbb{R}}^3$.

We define three coordinate frames for our multirotor environment as shown in Figure 1. The world frame $\left\{w\right\}$, which has $X_w{Y}_w{Z}_w$ coordinates, is an inertial frame. The body frame $\left\{b\right\}$, which has $X_b{Y}_b{Z}_b$ coordinates, is attached to the multirotor with its origin at the center of the multirotor. The gimbal camera frame $\left\{c\right\}$, which has $X_c{Y}_c{Z}_c$ coordinates, is attached to the gimbal camera with its z-axis coinciding with its optical axis. The rotation, position, and linear velocity of a frame $\left\{q\right\}$ with respect to a frame $\left\{p\right\}$ are denoted as $R_{pq}\in SO\left(3\right)$, $P_{pq}={[X_{pq},Y_{pq},Z_{pq}]}^T\in {\mathbb{R}}^3$, and $V_{pq}\in {\mathbb{R}}^3$, respectively. The body angular velocity of a frame $\left\{q\right\}$ in its own frame is denoted as ${\Omega }_q\in {\mathbb{R}}^3$. They satisfy

$$\begin{array}{cc}\hfill {\dot{P}}_{pq}& =V_{pq},\hfill \\ \hfill {\dot{R}}_{pq}& =R_{pq}{\widehat{\Omega }}_q.\hfill \end{array}$$

(1)

Define a function $L:{\mathbb{R}}^2\to {\mathbb{R}}^{2\times 3}$ as

$$L(u,n)=\left[\begin{array}{c}{e}_1^T\\ e_2^T\end{array}\right]\left(K-\left[\begin{array}{c}u\\ n\\ 1\end{array}\right]e_3^T\right)$$

(2)

for ${[u,n]}^T\in {\mathbb{R}}^2$, where the constant camera intrinsic matrix is given by

$$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right],$$

(3)

with $f_x$ and $f_y$ being the focal lengths in pixels, and $c_x$ and $c_y$ being the pixel coordinates of the principal point in the camera image. Define another function $M:{\mathbb{R}}^2\to \mathfrak{so}\left(3\right)$ as

$$M(u,n)=\widehat{\left(K^{-1}\left[\begin{array}{c}u\\ n\\ 1\end{array}\right]\right)}$$

(4)

for ${[u,n]}^T\in {\mathbb{R}}^2$. Finally, $\tau \in \mathbb{R}$ used in this paper represents time.

2.2. The First Step: Horizontal Approaching

In this section, we devise a controller that controls a multirotor to approach a target landing site in the horizontal direction. Let us assume that the landing site is fixed in the world frame $\left\{w\right\}$ and is already detected by the camera on the multirotor. The state is defined as

$$s_1={[u_c,n_c,Z_{ct},\eta ,h]}^T\in {\mathbb{R}}^5,$$

where ${[u_c,n_c]}^T$ is the pixel coordinates of the detected feature, which is the center of the landing site in the camera image, $Z_{ct}$ is the depth of the feature in the camera frame $\left\{c\right\}$, which is the third component of

$$P_{ct}={[X_{ct},Y_{ct},Z_{ct}]}^T,$$

(5)

representing the position of the target in the camera frame $\left\{c\right\}$, and

$$\eta =e_3^T{R}_{wc}{e}_1$$

(6)

is the quantitative value representing the amount of the camera rolling, which can be utilized for stable feature tracking. The final component h is the altitude of the multirotor, which is the same as the third component of

$$P_{wb}={[X_{wb},Y_{wb},Z_{wb}]}^T,$$

(7)

representing the position of the body frame in the world frame $\left\{w\right\}$. Figure 1 illustrates the state variables.

 Theorem 1. 

The dynamics of the state $s_1$ is given by

$${\dot{s}}_1=A_1\left(s_1\right)+B_1\left(s_1\right)\left[\begin{array}{c}\alpha \\ \beta \end{array}\right],$$

(8)

where

$$\begin{array}{cc}\hfill A_1\left(s_1\right)=& \left[\begin{array}{c}-{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)\\ -e_3^T\\ 0_{1\times 3}\\ 0_{1\times 3}\end{array}\right]R_{bc}^T{\widehat{\Omega }}_b{P}_{bc},\hfill \\ \hfill B_1\left(s_1\right)=& \left[\begin{array}{cc}L(u_c,n_c)M(u_c,n_c)& -{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)\\ Z_{ct}{e}_3^{T}M(u_c,n_c)& -e_3^T\\ -e_3^T{R}_{wc}{\widehat{e}}_1& 0_{1\times 3}\\ 0_{1\times 3}& e_3^T{R}_{wc}\end{array}\right],\hfill \end{array}$$

(9)

and ${[{\alpha }^T,{\beta }^T]}^T$ is related to ${[{\Omega }_c^T,V_{wb}^T]}^T$, which is given by

$$\begin{array}{cc}\hfill \alpha & ={\Omega }_c+R_{bc}^T{\Omega }_b\in {\mathbb{R}}^3,\hfill \\ \hfill \beta & =R_{bc}^T{R}_{wb}^T{V}_{wb}\in {\mathbb{R}}^3.\hfill \end{array}$$

(10)

Proof. 

The first two variables of $s_1$ are given by the pinhole camera model [18] as

$$\left[\begin{array}{c}{u}_c\\ n_c\end{array}\right]={\displaystyle \frac{1}{Z_{ct}}}\left[\begin{array}{c}{e}_1^T\\ e_2^T\end{array}\right]K{P}_{ct},$$

(11)

where K, $Z_{ct}$, and $P_{ct}$ are given in (3) and (5), respectively. By differentiation and the chain rule, we get

$$\left[\begin{array}{c}{\dot{u}}_c\\ {\dot{n}}_c\end{array}\right]={\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c){\dot{P}}_{ct},$$

(12)

where $L(u_c,n_c)$ is given in (2). To obtain ${\dot{P}}_{ct}$ in (12), we express $P_{ct}$ as follows:

$$P_{ct}=R_{bc}^T{R}_{wb}^T(P_{wt}-R_{wb}{P}_{bc}-P_{wb}),$$

(13)

where $P_{wt}$ and $P_{bc}$ are the position of the target in the world frame $\left\{w\right\}$ and the position of the camera in the body frame $\left\{b\right\}$, respectively, and are both constant. By differentiating (13) in t and using Lemma 1, (1), and (10), we get

$$\begin{array}{cc}\hfill {\dot{P}}_{ct}=& {\dot{R}}_{bc}^T{R}_{wb}^T(P_{wt}-R_{wb}{P}_{bc}-P_{wb})+R_{bc}^T{\dot{R}}_{wb}^T(P_{wt}-R_{wb}{P}_{bc}-P_{wb})\hfill \\ \hfill & +R_{bc}^T{R}_{wb}^T(-{\dot{R}}_{wb}{P}_{bc}-V_{wb})\hfill \\ \hfill =& Z_{ct}M(u_c,n_c)\alpha -\beta -R_{bc}^T{\widehat{\Omega }}_b{P}_{bc},\hfill \end{array}$$

(14)

where $M(u_c,n_c)$ is the evaluation of $M(u,n)$ given in (4) at $(u,n)=(u_c,n_c)$, which can be simplified to

$$M(u_c,n_c)={\displaystyle \frac{1}{Z_{ct}}}{\widehat{P}}_{ct},$$

(15)

which can be verified with (4) and (11). By substituting (14) into (12), we obtain the time derivative of the first two variables of $s_1$, which is

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\dot{u}}_c\\ {\dot{n}}_c\end{array}\right]=& L(u_c,n_c)M(u_c,n_c)\alpha -{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)\beta -{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)R_{bc}^T{\widehat{\Omega }}_b{P}_{bc}.\hfill \end{array}$$

(16)

The time derivative of the third variable of $s_1$ is

$${\dot{Z}}_{ct}=e_3^T{\dot{P}}_{ct}=Z_{ct}{e}_3^{T}M(u_c,n_c)\alpha -e_3^T\beta -e_3^T{R}_{bc}^T{\widehat{\Omega }}_b{P}_{bc}$$

(17)

by (5) and (14). The time derivative of the fourth variable of $s_1$ is computed as

$$\dot{\eta }=e_3^T({\dot{R}}_{wb}{R}_{bc}+R_{wb}{\dot{R}}_{bc})e_1=-e_3^T{R}_{wc}{\widehat{e}}_1\alpha$$

(18)

by (1), (6), (10) and Lemma 1. The time derivative of the fifth variable of $s_1$ is given by

$$\dot{h}=e_3^T{\dot{P}}_{wb}=e_3^T{R}_{wc}\beta$$

(19)

by (1), (7), and (10). By incorporating (16)–(19), we get the dynamics of the state $s_1$ in (8). □

 Remark 1. 

Since the position and yaw angle of the multirotor can be controlled independently [19], we control the yaw angle to be fixed in this paper. Additionally, the orientation of the gimbal can also be controlled independently. Therefore, we choose the linear velocity $V_{wb}$ of the multirotor and the angular velocity ${\Omega }_c$ of the gimbal as the control inputs in our proposed system. Lower-level control for thrust and torque can be handled using a commercial flight controller or a backstepping technique.

 Lemma 2. 

Assume $Z_{ct}>0$. The matrix $B_1\left(s_1\right)$ in (9) has full rank if and only if the following condition holds:

$$(e_1\times \left(R_{wc}^T{e}_3\right))·P_{ct}\ne 0.$$

(20)

 Proof. 

The matrix $B_1\left(s_1\right)$ satisfies

$$V{B}_1\left(s_1\right)W=\left[\begin{array}{cc}{0}_{3\times 3}& -I_{3\times 3}\\ {(e_1\times R_{wc}^T{e}_3)}^T& 0_{1\times 3}\\ -Z_{ct}{\left(M(u_c,n_c)R_{wc}^T{e}_3\right)}^T& 0_{1\times 3}\end{array}\right],$$

where

$$V=\left[\begin{array}{ccc}{\left[\begin{array}{c}{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)\\ e_3^T\end{array}\right]}^{-1}& 0_{3\times 1}& 0_{3\times 1}\\ 0_{1\times 3}& 1& 0\\ e_3^T{R}_{wc}{\left[\begin{array}{c}{\displaystyle \frac{1}{Z_{ct}}}L(u_c,n_c)\\ e_3^T\end{array}\right]}^{-1}& 0& 1\end{array}\right]$$

and

$$W=\left[\begin{array}{cc}{I}_{3\times 3}& 0_{3\times 3}\\ Z_{ct}M(u_c,n_c)& I_{3\times 3}\end{array}\right].$$

The matrices V and W are both invertible because ${\left[\begin{array}{c}{\displaystyle \frac{1}{Z_{ct}}}L{(u_c,n_c)}^T{e}_3\end{array}\right]}^T$ is always invertible. Hence, the matrix $B_1\left(s_1\right)$ has full rank if and only if

$$(e_1\times \left(R_{wc}^T{e}_3\right))\times \left(M(u_c,n_c)R_{wc}^T{e}_3\right)\ne 0_{3\times 1}.$$

(21)

By using $\parallel R_{wc}^T{e}_3\parallel \ne 0$, (15), and the triple product expansion, we can show that (21) is equivalent to (20). □

We now derive the dynamics of the state, $s_1$, and devise a controller that stabilizes at the desired state given by

$$s_{1,d}={[c_x,c_y,Z_{ct,d},0,h_d]}^T,$$

where $Z_{ct,d}$ is the desired depth of the feature from the camera, and $h_d$ is the desired altitude of the multirotor.

 Theorem 2. 

Define the error state as follows:

$${ϵ}_1=s_1-s_{1,d}.$$

Suppose that (20) holds. Then ${ϵ}_1$ converges to zero as $\tau \to \infty$ if the control law is given by

$$\begin{array}{cc}\hfill {\Omega }_c& ={\alpha }_1-R_{bc}^T{\Omega }_b,\hfill \\ \hfill V_{wb}& =R_{wc}{\beta }_1,\hfill \end{array}$$

where

$$\left[\begin{array}{c}{\alpha }_1\\ {\beta }_1\end{array}\right]=B_1{\left(s_1\right)}^{♯}(-{\lambda }_1{ϵ}_1-A_1\left(s_1\right)),$$

(22)

with $B_1{\left(s_1\right)}^{♯}=B_1{\left(s_1\right)}^T{\left(B_1\left(s_1\right)B_1{\left(s_1\right)}^T\right)}^{-1}$ the pseudo-inverse of $B_1\left(s_1\right)$ and ${\lambda }_1\in {\mathbb{R}}^{5\times 5}$ a positive-definite diagonal matrix.

Proof. 

The dynamics of the error state ${ϵ}_1$ becomes

$${\dot{ϵ}}_1=A_1\left(s_1\right)+B_1\left(s_1\right)\left[\begin{array}{c}{\alpha }_1\\ {\beta }_1\end{array}\right],$$

(23)

by Theorem 1. Since (20) holds, we obtain

$${\dot{ϵ}}_1=-{\lambda }_1{ϵ}_1$$

by Lemma 2 and (22), so that ${ϵ}_1$ converges to zero as $\tau \to \infty$. □

 Remark 2. 

Calculation of the horizontal approaching controller in (22) needs the values of $u_c$, $n_c$, h, $R_{wb}$, ${\Omega }_b$, $R_{bc}$, $Z_{ct}$, and η. The values of $u_c$ and $n_c$ can be obtained from a camera image. The value of h can be obtained from a range finder. The values of $R_{wb}$ and ${\Omega }_b$ can be obtained from an IMU or a commercial state estimator. The value of $R_{bc}$ can be obtained from a gimbal. The value of $Z_{ct}$ can be obtained from a depth camera. The value of η is calculated by (6).

2.3. The Second Step: Vertical Approaching

We adapt the virtual camera definition in [6,20] as depicted in the left part of Figure 1 to avoid the ill-conditioned problem of Theorem 2. Rewrite (20) as follows:

$$(e_1\times \left(R_{wc}^T{e}_3\right))·P_{ct}=-e_3^T{R}_{wc}{\widehat{e}}_1{R}_{wc}^T\overline{R}{P}_{vt}=e_3·(\left(\overline{R}{P}_{vt}\right)\times \left(R_{wc}{e}_1\right)),$$

(24)

where

$$P_{vt}={[X_{vt},Y_{vt},Z_{vt}]}^T={\overline{R}}^T{R}_{wc}{P}_{ct}$$

(25)

is the position of the target in the virtual camera frame $\left\{v\right\}$ and

$$\overline{R}=\left[\begin{array}{ccc}0& -1& 0\\ -1& 0& 0\\ 0& 0& -1\end{array}\right].$$

There are three conditions under which (24) becomes zero. The first is $e_3=R_{wc}{e}_1$, but we can ignore this condition because the two vectors are not close to each other in the horizontal approaching step. The second is $\overline{R}{P}_{vt}=k{R}_{wc}{e}_1$ for some $k\in \mathbb{R}$, which is equivalent to $P_{ct}=k{e}_1$ by (25), but this condition can not occur during image feature tracking. The third is $e_3=k\overline{R}{P}_{vt}$ for some $k\in \mathbb{R}$, which is equivalent to $u_v=c_x$ and $n_v=c_y$, where $u_v$ and $n_v$ are the pixel coordinates of the target in the virtual camera image, which has the same intrinsic matrix K as the actual camera. The pixel coordinates ${[u_v,n_v]}^T$ are derived as follows:

$$\left[\begin{array}{c}{u}_v\\ n_v\\ 1\end{array}\right]={\mu }^{-1}K{\overline{R}}^T{R}_{wc}{K}^{-1}\left[\begin{array}{c}{u}_c\\ n_c\\ 1\end{array}\right],$$

(26)

where

$$\mu =e_3^{T}K{\overline{R}}^T{R}_{wc}{K}^{-1}\left[\begin{array}{c}{u}_c\\ n_c\\ 1\end{array}\right].$$

(27)

Hence, we switch from the horizontal approaching to the vertical approaching when the following condition is satisfied:

$$\left|u_v-c_x\right|\le {\ell }_{u,1},\left|n_v-c_y\right|\le {\ell }_{n,1},$$

(28)

where ${\ell }_{u,1}$ and ${\ell }_{n,1}$ are the threshold values corresponding to the absolute values of errors so that we can avoid the condition (20). When the condition (28) holds, we anchor the actual camera so that it is always looking downward relative to the body frame $\left\{b\right\}$ and start the vertical approaching step that will be derived here. In this step, the multirotor descends while staying close to the center of the target. The state for the controller in this second step is defined as

$$s_2={[u_v,n_v,h]}^T\in {\mathbb{R}}^3.$$

 Theorem 3. 

The dynamics of the state $s_2={[u_v,n_v,h]}^T\in {\mathbb{R}}^3$ is given by

$${\dot{s}}_2=A_2\left(s_2\right)+B_2\left(s_2\right)\gamma ,$$

(29)

where

$$\begin{array}{cc}\hfill & A_2\left(s_2\right)=\left[\begin{array}{c}-{\displaystyle \frac{1}{Z_{vt}}}L(u_v,n_v)\left({\overline{R}}^T{R}_{wb}{\widehat{\Omega }}_b{P}_{bc}\right)\\ 0\end{array}\right],\hfill \\ \hfill & B_2\left(s_2\right)=-\left[\begin{array}{c}{\displaystyle \frac{1}{Z_{vt}}}L(u_v,n_v)\\ e_3^T\end{array}\right],\hfill \end{array}$$

and γ is related to $V_{wb}$ as

$$\gamma ={\overline{R}}^T{V}_{wb}\in {\mathbb{R}}^3.$$

(30)

Proof. 

The first two variables of the $s_2$ satisfy:

$$\left[\begin{array}{c}{\dot{u}}_v\\ {\dot{n}}_v\end{array}\right]={\displaystyle \frac{1}{Z_{vt}}}L(u_v,n_v){\dot{P}}_{vt}$$

(31)

as for the derivation of (12), where $Z_{vt}$ and $P_{vt}$ are from (25). To obtain ${\dot{P}}_{vt}$ in (31), we express $P_{vt}$ as

$$P_{vt}={\overline{R}}^T(P_{wt}-R_{wb}{P}_{bc}-P_{wb}),$$

(32)

whose time derivative is

$$\begin{array}{cc}\hfill {\dot{P}}_{vt}& ={\overline{R}}^T(-R_{wb}{\widehat{\Omega }}_b{P}_{bc}-V_{wb})=-{\overline{R}}^T{R}_{wb}{\widehat{\Omega }}_b{P}_{bc}-\gamma \hfill \end{array}$$

(33)

by Lemma 1, (1), and (30). By substituting (33) into (31), we obtain the time derivatives of the first two variables of $s_2$ as

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\dot{u}}_v\\ {\dot{n}}_v\end{array}\right]=-{\displaystyle \frac{1}{Z_{vt}}}L(u_v,n_v)({\overline{R}}^T{R}_{wb}{\widehat{\Omega }}_b{P}_{bc}+\gamma ).\end{array}$$

(34)

The time derivative of the third variable of $s_2$ is given by

$$\dot{h}=e_3^T\overline{R}\gamma =-e_3^T\gamma .$$

(35)

By incorporating (34) and (35), we get (29). □

The desired state is given as

$$s_{2,d}={[c_x,c_y,h_{2,d}]}^T,$$

where $h_{2,d}$ is a sufficiently small distance between the multirotor and the landing site.

 Theorem 4. 

Define the error state as follows:

$${ϵ}_2=s_2-s_{2,d}.$$

Then ${ϵ}_2$ converges to zero as $\tau \to \infty$ if the control law is given by

$$V_{wb}=\overline{R}{\gamma }_2,$$

where

$${\gamma }_2=B_2{\left(s_2\right)}^{-1}(-{\lambda }_2{ϵ}_2-A_2\left(s_2\right))$$

(36)

with a positive-definite diagonal matrix ${\lambda }_2\in {\mathbb{R}}^{3\times 3}$.

Proof. 

The dynamics of the error state ${ϵ}_2$ becomes

$${\dot{ϵ}}_2=A_2\left(s_2\right)+B_2\left(s_2\right){\gamma }_2,$$

(37)

by Theorem 3. Since the matrix $B_2\left(s_2\right)$ is always invertible, we obtain

$${\dot{ϵ}}_2=-{\lambda }_2{ϵ}_2$$

by (36), so that ${ϵ}_2$ converges to zero as $\tau \to \infty$. □

Through Theorem 2, Theorem 4, and condition (28), our control algorithm operates continuously, driving the defined error to converge to zero, thereby reaching the position for landing. The method in [14] considers the roll angle of the gimbal to be small to avoid singularity cases. Still, our algorithm theoretically identifies the singularity case and addresses this issue by implementing the two-step controller. Additionally, recent methods [7,14,15,16] implement gimbal controllers without considering UAV movement or treating it merely as a disturbance. In contrast, we design the controller based on integrated dynamics that ensure every component of the error simultaneously and exponentially converges to zero.

 Remark 3. 

Calculation of the vertical approaching controller in (36) needs the values of $u_v$, $n_v$, h, $Z_{vt}$, $R_{wb}$, and ${\Omega }_b$. The way to obtain the values of h, $R_{wb}$, and ${\Omega }_b$ is explained in Remark 2. The values of $u_v$, $n_v$ can be obtained by (26). Normally, $Z_{vt}$ can be obtained by $Z_{vt}=\mu Z_{ct}$ with μ defined in (27). If $P_{bc}$ is small, we can approximate $Z_{vt}$ with h.

The comprehensive workflow for regulating the multirotor is depicted in Figure 2. Initially, the multirotor surveys the surrounding area to identify the target landing site within the field of view of the camera. Subsequently, the horizontal approaching step outlined in Section 2.2 steers the multirotor. Once the multirotor approaches the target and fulfils the requirements specified in (28), the controller shifts to the vertical approaching step as discussed in Section 2.3. When the multirotor reaches a vicinity of the target vertically and satisfies

$$\left|u_v-c_x\right|\le {\ell }_{u,2},\left|n_v-c_y\right|\le {\ell }_{n,2},\left|h-h_{2,d}\right|\le {\ell }_h,\parallel V_{wb}\parallel \le V_{wb}^{th},$$

(38)

where ${\ell }_{u,2}$, ${\ell }_{n,2}$, ${\ell }_h$, and $V_{wb}^{th}$ are the threshold values corresponding to the absolute values of errors, the landing process is initiated by directing the multirotor to descend and powering off the multirotor.

3. Results

To evaluate the effectiveness of our two-step VBAL controller, we first conducted experiments in the simulation using Gazebo on Ubuntu 18.04. The controller was implemented in C++ and ROS [21], and a control input for both the multirotor and gimbal was sent via the PX4 autopilot. We gave $V_{wb}$ as a control command. On the other hand, our gimbal controller only gets three-axis angle values as control commands, so we calculate a rotation matrix $R_{\mathrm{cmd}}$ using ${\Omega }_c$ as follows:

$$R_{\mathrm{cmd}}=R_{bc}\left(\tau \right)e^{{\widehat{\Omega }}_c\Delta \tau },$$

where $\Delta \tau$ is a sampling time. Then, we convert $R_{\mathrm{cmd}}$ into Euler angles in Z-X-Y sequence to provide yaw, roll, and pitch angle commands.

We utilized the center pixel of the Apriltag as the target landing site, and the detection algorithm [22] was used by the multirotor solely for the specific target detection. The initial hover position of the multirotor was $P_{wb}={[0,0.05,2.65]}^T$, as illustrated in Figure 3a, and the target position was $P_{wt}={[5.5,3.5,0]}^T$, the center of a $1\mathrm{m}\times 1\mathrm{m}$ Apriltag. We initiated the horizontal approaching step with a desired state of $s_{1,d}={[c_x,c_y,2.5,0,2.5]}^T$, followed by the vertical approaching step with a desired state of $s_{2,d}={[c_x,c_y,0.5]}^T$. The gain matrices for the controllers were set as ${\lambda }_1=diagonal(30,30,20,1,1)$ and ${\lambda }_2=I_{3\times 3}$ based on experience. The camera intrinsic parameters are given in

Table 1. Camera intrinsic parameters.

	Parameter: $({\mathit{f}}_{\mathit{x}},{\mathit{f}}_{\mathit{y}},{\mathit{c}}_{\mathit{x}},{\mathit{c}}_{\mathit{y}})$
Simulation	(205.47, 205.47, 320.50, 180.50)
Real experiment	(207.86, 205.42, 217.39, 112.67)

. We set $({\ell }_{u,1},{\ell }_{n,1})=(160,80)\mathrm{pixels}$, $({\ell }_{u,2},{\ell }_{n,2})=(20,20)\mathrm{pixels}$, ${\ell }_h=0.05\mathrm{m}$, and no condition was specified in $V_{wb}^{th}$. The sampling time $\Delta \tau$ is 0.1 $\mathrm{s}$. As seen in Figure 3b, the multirotor landed securely on the landing site.

The three-dimensional trajectory of the multirotor is shown in Figure 4, with the blue line and red line representing the trajectory by the horizontal approaching step and the vertical approaching step, respectively, while the green pyramid triangles indicate the camera orientations. No unnecessary rolling of the gimbal camera was observed.

Figure 5 shows the measurement plots beginning when the gimbal camera of the multirotor initially detects the target. The intervals for vertical approaching, horizontal approaching, and landing are indicated by dotted blue, red, and purple boxes, respectively. The trajectory of the pixel errors $u_c-c_x$ and $n_c-c_y$ between the target feature and the camera principal point in the image are plotted in Figure 5a, demonstrating that the gimbal camera was directed to place the target feature near the principal point of the camera. Moreover, the gimbal camera consistently focused on the target throughout the experiment, as demonstrated in Figure 4. The horizontal distance $∥{[X_{wb},Y_{wb}]}^T-{[X_{wt},Y_{wt}]}^T∥$ between the multirotor and the target is depicted in Figure 5b, illustrating that the multirotor flew horizontally towards the target during the horizontal approaching step. The altitude h of the multirotor is presented in Figure 5c, indicating a small decrease in altitude during the horizontal approaching step, while the multirotor descended vertically towards the target during the vertical approaching step. In Figure 5d, the yaw, roll, and pitch angles of the gimbal camera trajectories are shown.

Our method was also validated in a real-world setting using a multirotor in Figure 6a. This multirotor was equipped with an onboard computer (NVIDIA’s Jetson TX2) for algorithm processing and a flight controller with PX4 firmware for controlling the actuators. We attached a gimbal (STorM32) to the multirotor bottom and mounted a camera (GoPro Hero 4) on the gimbal. The diagonal size and height of the multirotor are 37 cm and 21 cm, respectively. We defined the center of the multirotor as being 16.8 cm from the bottom of the multirotor. The total weight of the multirotor with onboard computer, gimbal camera, and battery is 2.46 $\mathrm{k}\mathrm{g}$. We used the state estimator of the PX4 controller, which fused data from a motion capture system (OptiTrack), an IMU, and a range finder, so we were able to get the altitude, attitude, and angular velocity of the multirotor center and give linear velocity and yaw commands. The landing site consisted of two Apriltags, as shown in Figure 6b. The experimental site was an indoor area with dimensions of 6 $\mathrm{m}$ × 7 $\mathrm{m}$ × 3 $\mathrm{m}$ (width × depth × height), with the center of the ground defined as the origin using the motion capture system.

When the multirotor was far from the target, it detected the larger Apriltag first, and as it approached the target, it detected the smaller Apriltag. The initial position of the multirotor was $P_{wb}={[-1.69,0.94,1.77]}^T$, and the target position was $P_{wt}={[2,-1,0]}^T$. The desired states for the horizontal approaching step and the vertical approaching step were $s_{1,d}={[c_x,c_y,1.8,0,1.8]}^T$ and $s_{2,d}={[c_x,c_y,0.5]}^T$, respectively. The gain matrices of the controller were selected as ${\lambda }_1=diagonal(60,60,80,5,5)$ and ${\lambda }_2=diagonal(0.5,0.5,1.0)$ from experience. The camera intrinsic parameters are given in Table 1. We set $({\ell }_{u,1},{\ell }_{n,1})=(108,60)\mathrm{pixels}$, $({\ell }_{u,2},{\ell }_{n,2})=(20,20)\mathrm{pixels}$, ${\ell }_h=0.2\mathrm{m}$, and $V_{wb}^{th}=0.3\mathrm{m}·{\mathrm{s}}^{-1}$. The sampling time $\Delta \tau$ is 0.033 $\mathrm{s}$. We captured four images of the multirotor flight as shown in Figure 7. The initial and final states of the multirotor are depicted in Figure 7a,d, respectively.

The three-dimensional trajectory of the multirotor is shown in Figure 8 using the same representation as in Figure 4. We plotted the pixel distance between the target feature and the principal point of the camera in the camera image, the horizontal distance between the multirotor and the target, the altitude of the multirotor, and the three-axis angles of the gimbal camera in the real environment in Figure 9 using the same representation as in Figure 5. The multirotor safely landed on the target landing site during the real experiment, confirming the feasibility of our two-step VBAL controller. We also conducted ten experiments for each of the three different tests, keeping all the real experimental setup parameters mentioned previously, except for the initial position. We plotted all three-dimensional position trajectories of the multirotor in Figure 10. The red trajectories represent the first test, the blue trajectories represent the second test, and the green trajectories represent the third test. In

Table 2. Initial positions and quantitative results for the three tests.

	Test 1	Test 2	Test 3
Initial position ($\mathrm{m}$)	$[-1,1.5,1.8]$	$[-1.5,-1,1.8]$	$[-1.5,0.5,1.8]$
Success rate (%)	100	100	100
Landing position error ($\mathrm{m}$)	$[0.027,0.042]$	$[0.003,0.046]$	$[0.057,0.042]$

, the three-dimensional initial positions, percentages of success rate, and mean final horizontal landing position errors $[X_{wb},Y_{wb}]-[X_{wt},Y_{wt}]$ for the three tests are shown. The landing was successful in all experiments, and the total mean and standard deviation of final landing position errors were $[0.029,0.043]\pm [0.044,0.036]$. Compared to the landing site size, the total mean distance error is less than $10\%$, demonstrating that stable and precise landings are possible. Our successful landing results from various initial multirotor positions are shown in the video available at https://youtu.be/MtBXix4g638 (accessed on 6 August 2024) (see Supplementary Video S1).

4. Conclusions

This study proposes a novel two-step controller for achieving vision-based autonomous landing of a multirotor equipped with a gimbal camera. We define state variables for each step, derive integrated dynamics of the multirotor and the gimbal camera, and formulate controllers using feedback linearization. The first step involves horizontal approaching of the multirotor towards the landing site, while the second step is comprised of vertical descent. The controller requires only one feature point and effectively eliminates unnecessary camera rolling. The efficacy of the controller is demonstrated in both simulation and real environment, exhibiting desirable multirotor control. We expect our control algorithm to be applicable when a multirotor needs to explore a wide area for surveillance and detection, then approach a target closely, or when a delivery drone needs to land at a site where its location is not precisely known. Our proposed algorithm, based on a velocity-level controller, will be further developed after a thorough consideration of the dynamics of the multirotor, to ensure its robustness against target movement and disturbances caused by external forces in future work. Additionally, we will conduct experiments under various conditions to derive metrics for quantitative performance, allowing for comparison with other state-of-the-art VBAL methods.