Vision-Based Autonomous Landing of a Quadrotor on the Perturbed Deck of an Unmanned Surface Vehicle

Abstract

Autonomous landing on the deck of an unmanned surface vehicle (USV) is still a major challenge for unmanned aerial vehicles (UAVs). In this paper, a fiducial marker is located on the platform so as to facilitate the task since it is possible to retrieve its six-degrees of freedom relative-pose in an easy way. To compensate interruption in the marker’s observations, an extended Kalman filter (EKF) estimates the current USV’s position with reference to the last known position. Validation experiments have been performed in a simulated environment under various marine conditions. The results confirmed that the EKF provides estimates accurate enough to direct the UAV in proximity of the autonomous vessel such that the marker becomes visible again. Using only the odometry and the inertial measurements for the estimation, this method is found to be applicable even under adverse weather conditions in the absence of the global positioning system.

1. Introduction

In the last few years, significant interest has grown towards unmanned aerial vehicles (UAVs), as described in [1]. The applications involving UAVs range from scientific exploration and data collection [2,3,4], to commercial services, military reconnaissance and law enforcement [5,6], search and rescue [7,8], patrolling [9] and even entertainment [10].

Among different UAVs topologies, helicopter flight capabilities such as hovering or vertical take-off and landing (VTOL) represent a valuable advantage over fixed-wing aircraft. The ability of autonomously landing is very important for unmanned aerial vehicles, and landing on the deck of an un-/manned ship is still an open research area. Landing a UAV on an unmanned surface vehicle (USV) is a complex multi-agent problem [11], and solutions to this can be used for numerous applications such as disaster monitoring [12], coastal surveillance [13,14] and wildlife monitoring [15,16]. In addition, a flying vehicle can also represents an additional sensor data source when planning a safe collision-free path for USVs [17].

Flying a UAV in the marine environment, one encounters rough and unpredictable operating conditions due to the influence of wind or waves in the manoeuvre compared to land. Apart from the above, there are various other challenges associated with the operation of UAVs; for example, the inaccuracy of low-cost GPS units mounted on most UAV and the influence of the electrical noise generated by the motors and on-board computers on magnetometers. In addition to this, the estimation of the USV’s movements is a difficult task due to natural disturbances (e.g., winds, sea currents, etc.). This poses difficulty for a UAV to land on a moving marine vehicle with low quality pose information. To overcome these issues, the camera mounted on the UAV and commonly used during surveillance missions [18] can also be used to increase the accuracy of the relative-pose estimates between the aerial vehicle and the landing platform [19]. The adoption of fiducial markers on the vessel’s deck is proposed as a solution to further improve the estimate results. To increase the robustness of the approach, a state estimation filter is adopted for predicting the six degrees-of-freedom (DOF) pose of the landing deck, which is not perceived by the UAV’s cameras. This work can be considered as the natural consequence of [20], in which the developed algorithm has been tested against a mobile ground robot, without any pitch and roll movements of the landing platform.

In terms of the paper’s organisation, Section 2 presents the method existing in the literature about autonomous landing for UAVs, while Section 3 introduces the quad-copter model, the image processing library used for the deck identification, the UAV controller and the pose estimation filter. In Section 4, three experiments, each with a different kind of perturbation acting on the landing platform, are presented and discussed. Finally, conclusions and future works are shown in Section 5.

2. State of the Art

Autonomous landing has until now been one of the most dangerous challenges for UAV. Inertial navigation systems (INS) and global navigation satellite systems (GNSS) are the traditional sensors of the navigation system. On the other hand, INS accumulates error while integrating position and velocity of the vehicle, and the GNSS sometimes fails when satellites are occluded by buildings. At this stage, vision-based landing became attractive because it is passive and does not require any special equipment other than a camera (generally already mounted on the vehicle) and a processing unit. The problem of accurately landing using vision-based control has been well studied. For a detailed survey about autonomously landing, please refer to [21,22,23]. Here, only a small amount of works is presented.

In [24,25], an IR-LED helipad is adopted for robust tracking and landing, while a more traditional T-shaped and H-shaped helipad are used respectively in [26,27,28,29]. The landing site is searched for an area whose pixels have a contrast value below a given threshold in [30]. In [31], a light imaging, detection and ranging (LIDAR) sensor is combined with a camera, and the approach has been tested with a full-scale helicopter. Bio-inspired by honeybees that use optic flow to guide landing, [32] follow the approach for fixed-wing UAV. The same has been done in [33,34] showing that by maintaining constant optic flow during the manoeuvre, the vehicle can be easily controlled. Hovering and landing control of a UAV on a large textured moving platform enabled by measuring optical flow is achieved in [35]. In [36], a vision algorithm based on multiple view geometry detects a known target and computes the relative position and orientation. The controller is able to control only the x and y positions to hover on the platform. In a similar work [37], the authors were also able to regulate the UAV’s orientation to a set point hover. In [38], an omnidirectional camera has been used to extend the field of view of the observations.

Four light sources have been located on a ground robot, and homography is used to perform autonomous take-off, tracking and landing on a UGV [39]. In order to land on a ground robot, [40] introduces a switching control approach based on optical flow to react when the landing pad is out of the UAV’s camera field of view. In [41], the authors propose the use of an IR camera to track a ship from long distances using its shape, when the ship-deck and rotorcraft are close. Similarly, [42] address the problem of landing on a ship moving only on a 2D plane without its motion known in advance.

The work presented in this paper must be collocated among vision-based methods. Differently from most of them, given the platform used, it relies on a pair of low resolution fixed RGB cameras, without requiring the vehicle to be provided with other sensors. Furthermore, instead of estimating the current pose of the UAV, in order to land on a moving platform, we employ an extended Kalman filter to predict the current position of the vessel on the deck of which the landing pad is located. The estimate is forwarded as input to our control algorithm that updates the last observed USV’s pose and sends a new command to the UAV. In this way, even if the landing pad is no longer within the camera’s field of view, the UAV can start a recovery manoeuvre that, differently from other works, takes the drone in proximity to its final destination. In this way, it can compensate interruptions in the tracking due to changes in attitude of the USV’s deck on which the pad is located.

3. Methods

In this section, all the components used to accomplish the autonomous landing on an USV are introduced. Initially, the aerial vehicle, together with its mathematical formulation, is described. Successively, the ar_pose computer vision library is presented. In the end, the controller and the pose estimation filter are discussed. A graphical representation of these components is depicted in Figure 1, and a video showing the overall working principle is available online (video showing the working principle of the algorithm: https://youtu.be/J1ib9PIsr-8).

3.1. Quad-Copter Model

The quad-copter in this study is an affordable ($250 USD in 2017) AR Drone 2.0 built by the French company Parrot. and it comprises multiple sensors such as two cameras, a processing unit, gyroscope, accelerometers, magnetometer, altimeter and pressure sensor. It is equipped with an external hull for indoor navigation, and it is mainly piloted using smart-phones and tablets through the application released by the producer over a WiFi network. Despite the availability of an official software development kit (SDK), the Robot Operating System (ROS) [43] framework is used to communicate with it, using in particular the ardrone-autonomy package developed by the Autonomy Laboratory of Simon Fraser University and the the tum-ardrone package [44,45,46] developed within the TUM Computer Vision Group in Munich. These packages run within ROS Indigo on a GNU/Linux Ubuntu 14.04 LTS machine. The specifications of the UAV are as follows:

• Dimensions: 53 cm × 52 cm (hull included);
• Weight: 420 g;
• IMU, including gyroscope, accelerometer, magnetometer, altimeter and pressure sensor;
• Front-camera with a high-definition (HD) resolution (1280 × 720), a field of view (FOV) of $73.5°\times 58.5°$ and video streamed at 30 frames per second (fps);
• Bottom-camera with a Quarted Video graphics Array (QVGA) resolution (320 × 240), a FOV of $47.5°\times 36.5°$ and video streamed at 60 fps;
• Central processing unit running an embedded version of the Linux operating system;

The downward-looking camera is mainly used to estimate the horizontal velocity, and the accuracy of the estimation highly depends on the ground texture and the quad-copter’s altitude. Only one of the two video streams can be streamed at the same time. Sensors’ data are generated at 200 Hz. The on-board controller (closed-source) is used to act on the roll $\Phi$ and pitch $\Theta$, the yaw $\Psi$ and the altitude of the platform z. Control commands u = ($\Phi$, $\Theta$,$\Psi$, z) ∈ [−1,1] are sent to the quad-copter at a frequency of 100 Hz.

While defining the UAV dynamics model, the vehicle must be considered as a rigid body with 6-DOF able to generate the necessary forces and moments for moving [47]. The equations of motion are expressed in the body-fixed reference frame $\mathcal{B}$ [48]:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}m\dot{V}+\Omega \times mV=F\hfill \\ J\dot{\Omega }+\Omega \times J\Omega ={\Gamma }^b\hfill \end{array}\right.\end{array}$$

(1)

where $V={[u,v,w]}^T$ and $\Omega ={[p,q,r]}^T$ represent, respectively, the linear and angular velocities of the UAV in $\mathcal{B}$. F is the translational force combining gravity, thrust and other components, while $J\in R^{3\times 3}$ is the inertial matrix subject to F and torque vector ${\Gamma }^b$.

The orientation of the UAV in air is given by a rotation matrix R from $\mathcal{B}$ to the inertial reference frame $\mathcal{I}$:

$$\begin{array}{cc}\hfill R& =R_{\psi }{R}_{\theta }{R}_{\varphi }\hfill \\ & =\left(\begin{array}{ccc}c\theta c\psi & s\varphi s\theta c\psi -c\varphi s\psi & c\varphi s\theta c\psi +s\varphi s\psi \\ c\theta s\psi & s\varphi s\theta s\psi +c\varphi c\psi & c\varphi s\theta s\psi -s\varphi c\psi \\ -s\theta & s\varphi c\theta & c\varphi c\theta \end{array}\right)\hfill \end{array}$$

(2)

where $\eta ={[\varphi ,\theta ,\psi ]}^T$ is the Euler angles vector and $s.$ and $c.$ are abbreviations for $sin(.)$ and $cos(.)$.

Given the transformation from the body frame $\mathcal{B}$ to the inertial frame $\mathcal{I}$, the gravitational force and the translational dynamics in $\mathcal{I}$ are obtained in the following way:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\dot{\xi }=v\hfill \\ m\dot{v}=R{F}^b-mg{e}_3^i\hfill \end{array}\right.\end{array}$$

(3)

where g is the gravitational acceleration, $F^b$ is the resulting force in $\mathcal{B}$ and $\xi ={[x,y,z]}^T$ and $v={[\dot{x},\dot{y},\dot{z}]}^T$ are the UAV’s position and velocity in $\mathcal{I}$.

3.2. Augmented Reality

The UAV’s body frame follows the right-handed z-up convention such that the positive x-axis is oriented along the UAV’s forward direction of travel. Both camera frames are fixed with respect to the UAV’s body frame, but translated and rotated in such a way that the positive z-axis points out of the camera lens, the x-axis points to the right from the image centre and the y-axis points down. The USV’s frame also follows the same convention and is positioned at the centre of the landing platform. Finally, a local frame fixed has been defined with respect to the world and initialized by the system at an arbitrary location. In Figure 2, the coordinate systems previously described are depicted.

The pose of frame j with respect to frame i is now defined as the 6-DOF vector:

$$\begin{array}{c}\hfill x_{i,j}={{[}^i{t}_{i,j}^T,{\Theta }_{i,j}^T]}^T={[x_{i,j},y_{i,j},z_{i,j},{\varphi }_{i,j},{\theta }_{i,j},{\psi }_{i,j}]}^T\end{array}$$

(4)

composed of the translation vector from frame i to frame j and the Euler angles $\varphi$, $\theta$, $\psi$.

Then, the homogeneous coordinate transformation from frame j to frame i can be written as:

$$\begin{array}{c}\hfill {}_j^{i}H=\left[\begin{array}{cc}{}_j^{i}R& {}^i{t}_i^{T}j\\ 0& 1\end{array}\right]\end{array}$$

(5)

where ${}_j^{i}R$ is the orthonormal rotation matrix that rotates frame j into frame i and is defined as:

$$\begin{array}{c}\hfill {}_j^{i}R=rotxyz\left({\Theta }_{i}j\right)=rotz{\left({\psi }_{i}j\right)}^{T}roty{\left({\theta }_{i}j\right)}^{T}rotx{\left({\varphi }_{i}j\right)}^T\end{array}$$

(6)

Figure 3 offers a graphical representation of the problem studied: retrieving the homogeneous matrix H offers the possibility to calculate the UAV’s pose with reference to the USV expressed as translation and rotation along and around three axes, respectively.

In this work, augmented reality (AR) visual markers are adopted to identify the landing platform. As described in [49], “in a AR virtual objects super-imposed upon or composited with the real world. Therefore, AR supplements reality”.

The ar_pose ROS package [50], a wrapper for the ARToolkit library widely used in human computer interaction (HCI) [51,52], is used to achieve this task. The ar_pose markers are high-contrast 2D tags designed to be robust to low image resolution, occlusions, rotations and lighting variation. For this reason, it is considered suitable for a possible application in a marine scenario, where the landing platform can be subject to adverse conditions that can affect its direct observation.

In order to use this library, the camera calibration file, the marker’s dimension and the proper topic’s name must be defined inside a configuration file. The package subscribes to one of the two cameras. Pixels in the current frame are clustered based on the similar gradient, and candidate markers are identified. The direct linear transform (DLT) algorithm [53] maps the tag’s coordinate frame to the camera’s frame, and the candidate marker is searched for within a database containing pre-trained markers. The points in the marker’s frame and camera’s frame are respectively denoted as ^MP and ^CP. Therefore, the transformation from one frame to the other is defined as follows:

$$\begin{array}{c}\hfill {}^{M}P={}_C^{M}H{}^{C}P={}_C^{M}H{}^{-1C}P={}_M^{C}H{}^{C}P\end{array}$$

(7)

where ${}_C^{M}H$ and ${}_M^{C}H$ represent the transforms from the marker to the camera frame and vice versa, respectively.

Using the camera’s calibration file and the actual size of the marker of interest, the 6-DOF relative-pose of the marker’s frame with respect to the UAV camera is estimated at a frequency of 1 Hz. For the current and the last marker’s observation, the time stamp and the transformation are recorded. This information is then used to detect if the marker has been lost and to actuate a compensatory behaviour.

3.3. Controller

In order to control the drone in a less complex way, the PID controller offered by the tum_ardrone package has been replaced with a (critically) damped spring one.

In the original work of [46], for each of the four degrees of freedom (roll $\overline{\Phi }$, pitch $\overline{\Theta }$, the yaw $\overline{\Psi }$ and the altitude $\overline{z}$), a separate PID controller is employed. Each of them is used to steer the quad-copter toward a desired goal position $p=(\widehat{x},\widehat{y},\widehat{z},\widehat{\psi })\in {\mathbb{R}}^4$ in a global coordinate system. The generated controls are then transformed into a robotic-centric coordinate frame and sent to the UAV at 100 Hz.

In this paper, in order to simplify the process of tuning the controller’s parameters, a damped spring controller has been adopted. In the implementation, only two parameters, $K\_direct$ and $K\_rp$, were used to modify the spring strength of the directly controlled dimensions ($yaw$ and z) and the leaning ones (x and y). An additional one, $xy\_damping\_factor$, is responsible for approximating a damped spring and accounting for external disturbances such as air resistance and wind.

The controller inputs are variations in the angles of roll, pitch, yaw and altitude, respectively denoted as $u_{\Phi }$, $u_{\Theta }$, $u_{\Psi }$ and $u_z$, defined as follows:

$$\begin{array}{cc}\hfill u_{\Phi }& =-K\_rp(\widehat{x}-x)+c\_rp(\widehat{\dot{x}}-\dot{x})\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill u_{\Theta }& =-K\_rp(\widehat{y}-y)+c\_rp(\widehat{\dot{y}}-\dot{y})\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill u_{\Psi }& =-K\_direct(\widehat{\psi }-\psi )+c\_direct(\widehat{\dot{\psi }}-\dot{\psi })\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill u_z& =-K\_direct(\widehat{z}-z)+c\_direct(\widehat{\dot{z}}-\dot{z})\hfill \end{array}$$

(11)

where $c\_rp$ and $c\_direct$ are the damping coefficients calculated in the following way:

$$\begin{array}{cc}\hfill c\_rp& =xy\_damping\_factor·2\sqrt{K\_rp·droneMass}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill c\_direct& =2\sqrt{K\_direct·droneMass}\hfill \end{array}$$

(13)

Therefore, instead of controlling nine independent parameters (three for the yaw, three for the vertical speed and three for roll and pitch paired together), the control problem is reduced to the three described above (namely $K\_direct$, $K\_rp$ and $xy\_damping\_factor$).

The remaining controller parameters are platform-dependent variables, and they are kept always constant during all the trials. Ignoring $droneMass$, which does not require an additional description more than its name, $max\_yaw$, $max\_gaz\_rise$ and $max\_gaz\_drop$ limit the rotation and linear speed on the yaw and z-axis, respectively. In the end, $max\_rp$ limits the maximum leaning command sent.

The controller’s parameters are the same across all the experiments performed, and they are shown in

Table 1. The controller parameters used in the simulation performed.

Parameter	Value	Parameter	Value
K_direct	5.0	K_rp	0.3
droneMass (kg)	0.525	max_yaw (rad/s)	1.0
xy_damping_factor st19	0.65	max_gaz_rise (m/s)	1.0
max_gaz_drop (m/s)	−0.1	max_rp	1.0

. The $K\_rp$ parameter, responsible for controlling the roll and pitch behaviour, is kept small in order to guarantee smooth movements along the leaning dimensions. In the same way, $max\_gaz\_drop$ has been reduced to a value of $0.1$ to decrease the descending velocity. On the other hand, the $max\_yaw$ parameter, used to control the yaw speed, has been set to its maximum value because the drone must align with the base in the minimum amount of time possible. The others have been left at their default values.

3.4. Pose Estimation

To increase the robustness and efficiency of the approach, an extended Kalman filter (EKF) has been adopted here to estimate the pose of the landing platform [54]. In fact, it may happen that the UAV loses track of the fiducial marker while approaching and descending on it. In order to redirect the flying vehicle in the right direction, the EKF estimates the USV’s current pose, which is then processed and forwarded to the controller. For estimation purposes, the odometry and inertial data are fused together to increase the accuracy [55,56]. The state vector is defined as $\mathbf{x}=[x,y,z,\varphi ,\theta ,\psi ,\dot{x},\dot{y},\dot{z},\dot{\varphi },\dot{\theta },\dot{\psi }]$, with $x,y,z$ and $\dot{x},\dot{y},\dot{z}$ representing respectively the global positions and velocity and $\varphi ,\theta ,\psi$ and $\dot{\varphi },\dot{\theta },\dot{\psi }$ the attitude of the vessel. Considering the sensor readings, the estimation process satisfies the following equations:

$$\begin{array}{cc}\hfill {\widehat{\mathbf{x}}}_{k|k-1}& ={\mathbf{F}}_k{\widehat{\mathbf{x}}}_{k-1|k-1}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathbf{P}}_{k|k-1}& ={\mathbf{F}}_k{\mathbf{P}}_{k-1|k-1}{{\mathbf{F}}_k}^T+{\mathbf{Q}}_k\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathbf{K}}_k& ={\mathbf{P}}_{k|k-1}{{\mathbf{H}}_k}^T{({\mathbf{H}}_k{\mathbf{P}}_{k|k-1}{{\mathbf{H}}_k}^T+{\mathbf{R}}_k)}^{-1}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\widehat{\mathbf{x}}}_{k|k}& ={\widehat{\mathbf{x}}}_{k|k-1}+{\mathbf{K}}_k({\mathbf{z}}_k-{\mathbf{H}}_k{\widehat{\mathbf{x}}}_{k|k-1})\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mathbf{P}}_{k|k}& =(\mathbf{I}-{\mathbf{K}}_k{\mathbf{H}}_k){\mathbf{P}}_{k|k-1}\hfill \end{array}$$

(18)

where k represents a discrete time instant, ${\mathbf{F}}_k$ is a kinematic constant velocity model, ${\mathbf{H}}_k$ is the observation model, ${\mathbf{z}}_k$ is the measurements vector, $\mathbf{I}$ is an identity matrix, ${\mathbf{Q}}_k$ is the process covariance matrix and ${\mathbf{R}}_k$ is the measurement covariance matrix.

The working principle of the EKF in this case is detailed below:

• the filter estimates the USV’s pose at 50 Hz, and its encoding is saved in a hash table using the time stamp as the key;
• when the UAV loses the track, the hash table is accessed, and the last record inserted (the most recent estimate produced by the filter) together with the one having as the key the time stamp of the last recorded observation are retrieved;
• the deck’s current position with reference to the old one is calculated using the geometric relationship;
• the controller commands are updated including the new relative position.

The procedure described above is iterated until the UAV is redirected above the visual marker and can perceive it through its bottom camera.

3.5. Methodology

The following section explains how the Algorithm 1 works. The code is publicly available on our repository (Github repository: https://github.com/pulver22/ardrone_tf_controller).

Algorithm 1 Landing Algorithm.

1: while not $landed$ do 2:     $last\_known\_pose=$ NULL 3:     if $marker\_visible$ then 4:         $last\_known\_pose\leftarrow detect\_landing\_marker\left(\right)$ 5:         if $last\_known\_pose<user\_defined\_threshold$ then 6:            $controller.send\_commands\left(land\right)$ 7:            $landed\leftarrow true$ 8:         end if 9:     else 10:         $usv\_pose\leftarrow ekf.estimates\_pose\left(\right)$ 11:         $last\_known\_pose\leftarrow last\_known\_pose+usv\_pose$ 12:     end if 13:     $trajectory\leftarrow calculate\_trj(last\_known\_pose)$ 14:     $attitude\_cmd\leftarrow controller.calculate\_cmd\left(trajectory\right)$ 15:     $controller.send\_commands(attutude\_cmd)$ 16: end while

The quad-copter flies using its fixed non-tilting frontal camera, approaching the landing site on the USV’s deck identified only by a fiducial marker. This, the scope of which is to outline the landing area, has to be perceived during the entirety of the landing manoeuvre. This is a requirement for precise landing despite the state estimator being able to compensate an interruption in observation. When a visual marker is detected, the image processing library computes the 6-DOF relative-pose between the marker itself and the UAV. The result is used to make the quad-copter approach the marker with the right orientation. To obtain this result, a damped spring controller reduces the error on the x-, y- and z-axis and on the quad-copter’s $yaw$. On attaining close proximity to the marker, the marker leaves the field of view of the frontal camera. This is due to hardware limitation of fixed non-tilting cameras. To overcome this problem, the video stream from the frontal camera is interrupted and acquired from the one located under the UAV and downward-looking. The quad-copter continues the landing manoeuvre keeping the marker at the centre of the second camera’s FOV. Otherwise, a compensatory behaviour is adopted: the EKF estimates the actual position of the USV, and the drone is redirected close to it while increasing its altitude. Increasing the altitude allows enlarging the field of view of the bottom camera, which is quite limited. In this way, it is guaranteed that the marker will soon be perceived and centred by the aerial vehicle. When an experimentally-defined distance from the marker is reached, the drone lands safely. This distance depends on the side length of the marker used. In fact, with a smaller marker, it would be possible to decrease this value, but it would become impossible to perceive the marker at a longer distance. We found that a marker side length of approximately 0.30 m represents a good trade-off for making the marker visible at a long and a close distance at the same time. As a consequence, we decide to use 0.75 m as the distance for starting the touchdown phase of the descending manoeuvre, during which the power of the motors is progressively reduced until complete shut-down. The use of visual markers allows the estimation of the full 6-DOF pose information of the aerial and surface vehicles. In this way, landing operations in a rough sea condition with a significant pitching and rolling deck can still be addressed.

4. Results and Discussion

All the experiments have been conducted inside a simulated environment built on Gazebo 2.2.X and offering a 3D model of the AR Drone 2.0. In the scope of this work, the existing simulator has been partially rewritten and extended to support multiple different robots at the same time. The Kingfisher USV, produced by Clearpath Robotics, has been used as the floating base. It is a small catamaran with dimensions of 135 × 98 cm, which can be deployed in an autonomous or teleoperated way. It is equipped with a flat plane representing a versatile deck for UAVs of small dimensions. On this surface, a square visual marker is placed. Previous research demonstrated that a linear relationship exists between the side length of the marker and its observability. Therefore, we opted for a side length of 0.3 m, which represents a good compromise, making the marker visible in the range [0.5, 6.5] meters.

The algorithm has been tested under multiple conditions, namely three. In the first scenario, the USV is subjected only to a rolling movement, while floating in the same position for the entire length of the experiment; in the second scenario, the USV is subjected only to a pitching movement; while in the last scenario, the USV is subject to both rolling and pitching disturbances at the same time. Figure 4 illustrates the rotation angles around their corresponding axis. In all the simulations, the disturbances are modelled as a signal having a maximum amplitude of five degrees and a frequency of 0.2 Hz. Rolling and pitching of a vessel generate upward and downward acceleration forces directed tangentially to the direction of rotation, which cause linear motion known as swaying and surging along the transverse or longitudinal axis, respectively [57].

4.1. Rolling Platform

In this subsection, the results of a landing manoeuvre on a rolling floating base are reported. In particular, Figure 5 illustrates the UAV’ and the USV’s trajectory, respectively in blue and red, in the UAV’s reference frame; while Figure 6 and Figure 7 show the controller commands and the salient moments of the manoeuvre, respectively.

The marker has been successfully recognised at a distance of 3.74 m in front of the UAV and at 0.09 m to its left. The displacement on the z-axis, used as a reference for the altitude, was of 0.84 m instead. The UAV, with the parameters reported in the previous Table 1, was able to complete the landing in 25 s.

The quad-copter approaches the landing base trying to keep it at the centre (in a range of ±10 degrees) of its camera’s FOV. In the case that the marker leaves this interval of tolerance, the UAV would rotate around its z-axis in order to centre it again. The approach continues until the UAV’s low altitude prevents the marker from being seen by the frontal camera, as shown in Figure 7a ($t=10$ s). At this point, the video stream is switched from the frontal camera to the one located at the bottom of the quad-copter and looking down, and new commands are generated and sent. The UAV is instructed to move towards the last known position of the landing platform, but increasing its altitude in order to enlarge the area covered by its bottom camera. At $t=15$ s, as represented in Figure 7b, the UAV is located exactly above the marker, and it can now complete the landing phase: it descends while trying to keep the marker at the centre of its FOV, as shown in Figure 7c. Small velocity commands are sent on the leaning direction (x and y, respectively) in order to approach the final position with high accuracy. Finally, at $t=25$ s, the UAV reaches the minimum altitude required to shut-down its motors and land on the platform (Figure 7f).

The commands generated from the relative-pose between the UAV and the landing platform’s frame are illustrated in Figure 5. Here, the controller’s commands are plotted against the perception from the camera. As is possible to see in the figure, for most of the travel, the two curves of the commands and of the observations overlap perfectly. When they do not, the marker is lost, and the UAV actuates the compensatory behaviour: the estimation filter’s output, namely the USV’s predicted pose, is combined with the latest visual observation in order to generate new commands for the UAV. In this way, it is possible to explain the change in roll, pitch and altitude in the graph. Since the UAV has the same yaw of the floating base, namely they have the same orientation along the z-axis, no rotation commands are issued for this degree of freedom.

A few words are reserved for the pitch’s data between $t=18$ s and $t=22$ s and the gaz’s data between $t=5$ s and $t=8$ s. In this case, the offsets are below a user-defined threshold, and a null command is sent instead. The use of a threshold has been introduced for speeding up the landing phase: while testing the controller, it was noticed that the UAV spent much time trying to align perfectly on the three axes with the centre of the landing plane, sometimes without any success. This has been identified as a limitation of controllers with fixed value parameters, and a new, more versatile solution is already planned as future work.

4.2. Pitching Platform

In this subsection, an experiment with a pitching floating platform is reported. As before, the time for completing the landing manoeuvre is not considered as a key factor, but the attention is on the ability of the UAV to approach and land on the USV with high precision.

As in the previous experiments, the two vehicles’ 3D trajectories are reported in Figure 8 in the UAV’s reference frame, the controller commands in Figure 9 and example frames in Figure 10. The quad-copter, with the same controller parameters as before, was able to follow and land on the visual marker in almost 34 s after identifying it 4.46 m ahead and 0.12 m on its left.

As in the case of a rolling base, Figure 10a shows that the UAV starts moving in order to keep the visual marker at the centre of its frontal camera’s field of view. This is what happens at time $t=26$ s and shown in Figure 10b. At $t=6$ s, the UAV reaches its minimum altitude, and it is now impossible for it to see the visual marker, as illustrated in Figure 10c. At this point, the video stream starts to be acquired from the bottom camera, and the USV’s estimated position is sent to the controller; at the same time, instructing the UAV to increase its altitude to augment the total area covered with its downward-looking camera. Doing this, at $t=13$ s, the UAV is located exactly above the USV. The landing base is at the centre of the camera’s FOV; therefore, a null velocity command is sent to stop the USV. Figure 10e,f shows that the UAV can then descend slowly to centre the marker properly and, in the end, land on it.

Further analysis can be done with the results reported in Figure 9. In the same way as the experiment with a rolling deck, the curve of the controller’s commands and the one related to the offsets overlap for most of the time. All the considerations made before still hold: while the marker is lost, the EKF is able to estimate the landing platform’s current pose with reference to the instant of time when the marker has been lost. This relative-pose is added to the last observation in order to produce a new command.

This is what is possible to see in the plot between $t=21$ s and $t=25$ s. Here, the two curves differ: while all the offsets remain constant because no new marker observations have been done by the UAV, the commands (gaz and roll) slightly change. The plot is now discussed in more details. While the yaw and the pitch commands remain identical to zero because the UAV is already aligned with the landing base (within the predefined bounds), the UAV’s roll command is changed including at every instant the new relative-pose (changing in the longitudinal direction) of the USV.

4.3. Rolling and Pitching Platform

The last simulation has been done with a floating platform that is subject to both rolling and pitching stresses. The goal of this experiment is to test the developed landing algorithm against simulated harsh marine conditions.

The results are reported in Figure 11, showing both vehicles trajectories along a 23-s operation. The UAV successfully accomplishes the landing manoeuvre starting from an initial marker’s identification 3.71 m in front of it and 0.30 m to its left. Figure 12 shows the comparison between the offsets obtained through the vision algorithm and the commands sent to the controller. It is possible to see that, as in the previous experiments, the curve of the offsets and the one related to the commands mainly overlap. All the analyses made before are still valid, but it is interesting to notice how the framework proposed is able to react properly also when the landing platform is subject to complex disturbances. The salient moments of the flight are illustrated in Figure 13.

5. Conclusions and Future Directions

In this paper, a solution to make an unmanned aerial vehicle autonomously land on the deck of a USV is presented. It relies only on the UAV’s on-board sensors and on the adoption of a visual marker on the landing platform. In this way, the UAV can estimate the 6-DOF landing area position through an image processing algorithm. The adoption of a pose estimation filter (in this case, an extended Kalman filter) allows overcoming issues with fixed non-tilting cameras and the image processing algorithm. Not involving GPS signals in the pose estimation and in the generation of flight commands allows the UAV to land also in situations where this signal is not available (indoor scenarios or adverse weather conditions).

The validation of the approach has been done in a simulation with a quad-rotor and an unmanned surface vehicle as the platform on which to land. Three different experiments were performed, each of them with a different type of disturbance acting on the landing base. In all scenarios, successful results were obtained.

The future research is two-fold. From a practical point of view, the proposed approach needs to be tested in a real environment with an unmanned surface vehicle in order to test its robustness against real wind and sea currents. The second aspect is more related to the identified limitation of the algorithm itself. Therefore, it is suggested to develop an adaptive controller, possibly based on an intelligent solution such as artificial neural networks or fuzzy logic, where the gain of the controller change depends on the distance to the landing base.