A Novel Fuzzy Image-Based UAV Landing Using RGBD Data and Visual SLAM

Abstract

In this work, an innovative perception-guided approach is proposed for landing zone detection and realization of Unmanned Aerial Vehicles (UAVs) operating in unstructured environments ridden with obstacles. To accommodate secure landing, two well-established tools, namely fuzzy systems and visual Simultaneous Localization and Mapping (vSLAM), are implemented into the landing pipeline. Firstly, colored images and point clouds acquired by a visual sensory device are processed to serve as characterizing maps that acquire information about flatness, steepness, inclination, and depth variation. By leveraging these images, a novel fuzzy map infers the areas for risk-free landing on which the UAV can safely land. Subsequently, the vSLAM system is employed to estimate the platform’s pose and an additional set of point clouds. The vSLAM point clouds presented in the corresponding keyframe are projected back onto the image plane on which a threshold fuzzy landing score map is applied. In other words, this binary image serves as a mask for the re-projected vSLAM world points to identify the best subset for landing. Once these image points are identified, their corresponding world points are located, and among them, the center of the cluster with the largest area is chosen as the point to land. Depending on the UAV’s size, four synthesis points are added to the vSLAM point cloud to execute the image-based visual servoing landing using image moment features. The effectiveness of the landing package is assessed through the ROS Gazebo simulation environment, where comparisons are made with a state-of-the-art landing site detection method.

1. Introduction

In recent years, Unmanned Aerial Vehicles (UAVs) have played a pivotal role not only in daily domestic applications but also in the rapid development of industrial applications. UAVs have been long-established in diverse areas, including ecological monitoring [1,2], smart agriculture [3,4], high-voltage power transmission [5,6], and radar tomography [7]. The forthcoming transportation technology may be contingent upon multirotor devices owing to their lack of interaction with the environment, which is currently being addressed by proposing aerial manipulation systems equipped with a variety of sensors for controlling the UAV’s pose and speed or collecting data. Regardless of the task at hand, the issue of autonomous landing must be tackled, particularly when the system encounters harsh weather conditions or technical difficulties that necessitate an emergency landing to avoid crashing the aerial vehicle.

Autonomous landing of UAVs comprises landing zone detection, path planning, obstacle avoidance, and target tracking. To achieve this, incorporating visual data is suggested to be a promising solution, especially in GPS-denied environments [8]. There are two key factors that affect the design and testing of the landing approach: the type of environment in which the drone aims to land and the technique that is employed for Safe Landing Zone (SLZ) detection. The former can be classified as either an indoor or outdoor environment with the variation of using a static (fixed) or dynamic (moving) landing zone [9]. Furthermore, visual markers can be used to facilitate the landing procedure; in other words, the landing location can be either known or unknown. Readers can find the upsides and downsides of each of these scenarios described thoroughly in [10]. Generally speaking, SLZ approaches can be categorized as vision-based, non-vision-based, or hybrid approaches. Most of existing research focuses on addressing the issue of autonomous landing in known environments in which UAVs detect predefined marks or man-made structures; nonetheless, in real-world applications of the UAVs, these techniques are perceived as unacceptable since unknown or extreme scenes can pose a major setback.

Safe landing site detection approaches for unforeseen environments have enjoyed continuously growing attention these decades. Three-dimensional reconstruction of the surroundings plays a significant role in a safe landing pipeline. Techniques such as structure from motion, LiDAR, and stereo-ranging contribute to generating a geometry map. Localization plays a pivotal role in the safe landing process. To this end, a dynamic localization framework with asynchronous LiDAR–camera fusion was designed in [11] and utilized to estimate the absolute attitude and position observations of UAVs in outdoor, unknown environments. Recently, fusing the information from a swarm of UAVs using the iterative closest point algorithm and a graph-based SLAM was performed to enhance the localization and mapping [12]. The semantic segmentation using deep neural networks [13] was utilized to discriminate the areas that are most likely suitable for landing. The main advantage of semantic maps over traditional geometry maps is the high-level perception of the scene in cases where the scene is flat but hazardous for landing, such as rivers or lakes; nonetheless, the semantic segmentation is prone to error when there are small objects, like debris, close to the segmented areas. To address these issues, feature extraction and matching were used to robustly localize the landing area in [14]. Visual SLAM and deep semantic segmentation were integrated to offer a comprehensive landing framework for realizing autonomous navigation and landing in unknown indoor environments [15]. The suggested model processes the RGB and depth images and then employs the ORB-SLAM2 to construct the sparse map. Afterward, a 3D semantic map was developed by combining a deep learning model and a SLAM system. Providing poor information about uniform and low-contrast surfaces due to the sparse point cloud is the main drawback of the SLAM system. To remedy this, Chatzikalymnios et al. [16] suggested a method that utilizes the scene’s disparity map and point cloud from stereo processing to evaluate terrain factors and quantify them into map metrics.

Visual clues can not only be used to create the map of the environment but are also utilized to perform the landing as well. An example is the approach proposed by Dougherty et al. in [17], where a CMOS camera was utilized to tackle the problem of landing on a flat inclined surface without prior information regarding the orientation of the landing surface. Visual servoing is a technique that employs visual features such as corner points, lines, or ovals to control the camera’s pose. The position-based visual servoing (PBVS) utilizes a calibrated camera and the geometric model of the target to estimate its pose with respect to the camera [18,19]. Image-based visual servoing (IBVS), on the other hand, calculates the feature errors and uses an interaction matrix (a.k.a. image jacobian) to find the camera’s Cartesian velocity. Furthermore, depending on the configuration of the camera, two eye-in-hand (or end-point closed-loop) and eye-off-hand (end-point open-loop) groups are available [20].

Previous research had several shortcomings. First, the vast majority of these studies relied on GPS information, which is inadequate in areas where signals are blocked. Second, human expertise has not been incorporated into the landing systems as a fuzzy system, despite its potential as a tremendous asset. Third, in most outdoor environments, employing man-made markers is not feasible. Lastly, in studies that used stereo-ranging, the limited range was not considered as a constraining factor. This motivates us to propose a novel landing pipeline that addresses these problems.

To tackle the above issues, a combination of ORB-SLAM3, stereo range point cloud, and fuzzy inference is utilized to detect the safe zone site for landing. First, depth images are processed to provide information on flatness and depth variation. The stereo-ranging point clouds are used to acquire data regarding the steepness and inclination of the terrain. Then a fuzzy inference engine that performs a pixel-wise operation on the four images characterizing the terrain data is incorporated to create a novel fuzzy landing score map. Nevertheless, owing to the limited resolution of the radar sensors, the 3D constructed point cloud is not complete; to remedy this, the sparse ORB-SLAM3 point cloud is used. By using the thresholded fuzzy landing score map as a mask for the back-projected ORB-SLAM3 point cloud, a subset of world points that are the best for landing is detected. By using clustering techniques and calculating the local flatness, the best world point for landing is identified. Afterward, a set of synthetic point clouds will be created by considering the area of the UAV. Finally, an IBVS using the image moments is performed to complete the autonomous landing. The main contributions of this study are summarized as follows:

• A novel fuzzy landing score map to reliably characterize the proper landing site is proposed.
• To address the problem of the limited resolution of stereo range cameras, the monocular ORB-SLAM point cloud is exploited to identify the best subset of the point cloud for landing.
• A novel technique for adding synthetic world points to serve as the corner features for IBVS is suggested.
• A new method that integrates the ORB-SLAM3 localization with image moment-based IBVS is offered to perform the landing.
• The close-to-real-world ROS Gazebo simulations for the indoor/outdoor, unknown environments are conducted to verify the effectiveness of the landing pipeline.

The content of this paper is as follows: In Section 2, the landing zone detection structure, including its subsystems, will be thoroughly discussed. The images characterizing the terrain considered for landing are also investigated in this section. Furthermore, the IBVS controller design and stability analyses are carried out. The simulation results using the Gazebo simulator are presented next in Section 3, where a comparison is made between the proposed method and the state-of-the-art method to demonstrate its effectiveness. The results and potential future contributions have been briefly discussed in Section 4. Eventually, the concluding remarks are expressed in Section 5.

2. Methodology

The system structure is illustrated in Figure 1. As shown, the landing pipeline comprises four main subsystems. First, the sensors integrated into the RGB-D camera acquire the frames needed for SLAM system initialization and tracking. Afterward, a 3D sparse map of the environment is generated using the ORB-SLAM3 system. Likewise, there exists a secondary set of map points and a depth image computed using stereo-ranging. The depth information and stereo-ranging point clouds are used to generate four images that characterize the terrain’s appropriateness for landing. To achieve a 2D landing map, a rule-based fuzzy system is utilized to infer where is safe for landing by assigning a score to each corresponding pixel. The last subsystem filters the undesirable ORB-SLAM 3D points, identifying a single cluster with the best flatness. Finally, a vision-based landing is executed based on the SLAM pose estimation and moment of an artificially created image.

2.1. Dense Stereo Reconstruction

The estimation of the world points using two images taken by two different (or similar) cameras with distinct poses is known either as sparse or dense reconstruction. The former involves three major steps: feature extraction, stereo matching, and matching refinement. The disparity is calculated for each matched point in the pair of images and can be converted into three-dimensional Cartesian coordinates. However, in the latter, the depth is estimated for all pixels in the image pair using similarity measures in template matching techniques given a range of disparity. Assuming that the disparity is calculated for all points in the image space, one may use the following equations to derive the coordinates of each world point $(X,Y,Z)$ corresponding to each pixel [18]:

$$X={\displaystyle \frac{b({}^{l}u-u_0)}{d}},Y={\displaystyle \frac{b({}^{l}v-v_0)}{d}},Z={\displaystyle \frac{fb}{{\rho }_{u}d}},$$

(1)

where $(u_0,v_0)$ is the principal point coordinate (in pixels), b is the baseline of the stereo camera, usually reported in $mm$, f is the focal length of the camera in pixels, ${\rho }_u$ is the effective pixel size, $({}^{l}v,{}^{l}u)$ and $({}^{r}v,{}^{r}u)$ are the coordinates of the left and right conjugate points, respectively, $d={}^{l}u-{}^{r}u\ge 0$ is the disparity. One issue with infrared cameras is their limited range for optimal performance. For instance, the well-established RealSense D435 and D415 cameras are reported to yield the best performance within a limited range, according to [21].

2.2. Visual Monocular ORB-SLAM3 System

The vSLAM system realizes a simultaneous estimation of the body pose and point cloud generation. The core components of the visual ORB-SLAM consist of initialization, tracking, local mapping, and loop closure [22]. It is well acknowledged that the depth information cannot be recovered with a single image, with which one may encounter at the beginning stage of the algorithm. To cope with this issue, the initialization subsystem starts extracting the ORB features for each keyframe and matching them with subsequent frames. When the number of matched features exceeds a pre-specified threshold, the initial point cloud is reconstructed solely based on these two frames. The concept of the epipolar pencils rotating around the baseline is afterward utilized to recover the pose information. Based on (2), the image points ${\mathbf{x}}_r$ and ${\mathbf{x}}_l$ can be projected onto each other using a 2D homography matrix $\mathbf{H}$. Furthermore, Equation (3) indicates the epipoles and the image points are co-planar, and the position of ${\mathbf{x}}_l$ is constrained by the epipolar lines:

$${\mathbf{x}}_r=\mathbf{H}{\mathbf{x}}_l,$$

(2)

$${\mathbf{x}}_r\mathbf{F}{\mathbf{x}}_l=0.$$

(3)

The planar homography and fundamental matrices are obtained in parallel threads. Depending on the scene’s parallax and the existence of planar elements, one of the mentioned models is selected to solve the above equations with the normalized direct linear transformation and eight-point algorithms leveraging RANSAC serving as a remedy to eliminate outliers.

2.3. Landing Zone Detection Algorithm

There are several key factors hindering the safety of landing that one can encounter in real-world scenarios. The flatness of the terrain of landing, for instance, holds crucial significance in the context of landing where the environment is perceived as unknown. Furthermore, there is a possibility of crashing the landing gear or UAV overthrow if the slope of the terrain exceeds a limit. Lastly, the area should be sufficiently large for the UAV to land—an area twice the size of the UAV, as suggested in some studies like [16].

The proposed landing algorithm is initiated when the UAV is hovering above the area approximately considered for landing. A prerequisite of the autonomous landing is that the pitch and roll angles of the UAV are kept small and close to zero just before the landing point is detected [16]. However, it is emphasized that this assumption does not limit this approach owing to the fact that the concept of the virtual camera has been incorporated into the control system formulation, compensating for the effects of the non-zero roll and pitch angles. In addition, the camera should be oriented downward with a pitch angle of approximately 90 degrees.

The first map metric established in the processing steps relies on obtaining the local variance of the depth image using a fixed window applied to each pixel. The depth discontinuities like edges can account for high intensity levels. Once the variance image is obtained, a pixel-intensity transformation is utilized to map the variance values to a gray-scale image where the higher variance is represented by black (zero intensity) and the lower variances with white (full intensity). The same pixel-wise transformation $f_1(.)$ as proposed by [16] is adopted:

$$f_1\left(i_0\right)=exp(-i_0^2),$$

(4)

Therein, $i_0$ represents the pixel intensity of the depth image variance image. A min–max normalization (5) is applied to the image after each intensity transformation described as follows:

$$f_n\left(i\right)={\displaystyle \frac{i-i_{min}}{i_{max}-i_{min}}},$$

(5)

where $i_{min}$ and $i_{max}$ are the minimum and maximum pixel intensities, respectively. The details of the algorithm’s implementation are given in Algorithm 1. In the second line of this algorithm, the element-wise product, denoted by ⊙, is used to calculate the square of mean image.

Algorithm 1 Algorithm for obtaining depth variance map

Input:   depth image, window size Output: DepthVarMap   1: ${\mu }_1\leftarrow$ Blur(depth image, window size)   2: MeanImgSquare ← depth image ⊙ depth image   3: ${\mu }_2\leftarrow$ Blur(MeanImgSquare, window size)   4: LocalVariance ← SqaureRoot(AbsoluteValue(${\mu }_2-{\mu }_1^2$))   5: DepthVarMap ← exp(-LocalVariance)   6: DepthVarMap ← Min–Max Normalization(DepthVarMap)   7: return DepthVarMap

To obtain the flatness image, edges in the depth image are detected using the Canny edge detector with the pixel intensity $i_c$, and spatial variables of the output and input images are denoted by p and q, respectively. Next, the minimum Euclidean distance of each pixel in the generated binary image to the non-zero intensity pixels is calculated as follows [23]:

$$\begin{array}{c}\hfill f_2\left(p\right)=\underset{p,q\in {\mathbb{R}}^2}{min}\left|\right|p-q\left|\right|,\mathrm{subject}\mathrm{to}{i}_c\left(q\right)=1.\end{array}$$

(6)

The elements of how to derive the flatness image from the depth image are presented in Algorithm 2.

Algorithm 2 Algorithm for obtaining the flatness map

Input:  depth image Output: FlatnessMap   1: BinImg ← Canny(depth image)   2: FlatnessMap ← DistanceTransform(BinImg) DistanceTransform Calculates the Euclidean distance between each non-zero pixel in an image and the nearest zero pixel.   3: FlatnessMap ← Min–Max Normalization(FlatnessMap)   4: return FlatnessMap

Being flat is not solely sufficient for landing, and how inclined the surface is should also be taken into consideration. To achieve this, the three-space points generated by dense stereo reconstruction are established. The principal curvature of the surfaces rendered by the point cloud can serve as a measure indicating the inclination. A k-means clustering algorithm is employed on the candidate point set to identify the dense landing sites within the scene. The covariance of each patch is calculated, and the eigenvectors of each yield the principal curvature components. The third component of the principal curvature vector indicates the inclination of the patch. The following relation describes how the inclination image is obtained:

$$f_3(C_z,C_{max})=exp(-\sqrt{{\displaystyle \frac{C_z}{C_{max}}}}),$$

(7)

Therein, $\mathbf{C}=(C_x,C_y,C_z)$ is the vector of principal curvature, and $C_{max}$ is the maximum value of the vector. A min-max normalization (5) is essential after this operation. Algorithm 3 shows the overall steps to compute the inclination map.

Algorithm 3 Algorithm for computing the inclination map

Input: Point Cloud Patches Output: InclinationMap   1: EigenVectors ← ComputeCurvature(Point Cloud Patches)   2: for  $i=0$ to N do   3:    ReprojectedImg ← Reproject(Point Cloud Patches [i])   4:    InclinationMap[ReprojectedImg] = Compute_f3(EigenVectors [i])   5: end for   6: InclinationMap ← Min-Max Normalization(InclinationMap)   7: return InclinationMap

Steepness of each pixel in the depth map cannot be obtained in the image space; instead, cloud point normal vectors are used to determine this metric. The angle between the normal and the z-axis of the world reference frame is a measure showing how steep the surface is. Accordingly, the dot product of the normal and the unit vector yields the angle. Nonetheless, the angle corresponding to each pixel should be mapped to a pixel intensity. As a result, the following formulation can be used to generate the steepness image [23]:

$$\psi =arccos({\mathbf{e}}_3.\mathbf{n}),$$

(8)

$$f_4\left(\psi \right)=exp(-{\psi }^2),$$

(9)

where ${\mathbf{e}}_3=(0,0,1)$, expressed in the inertial frame, $\mathbf{n}$ is the normal vector, and $\psi$ is the angle between the normal and world frame coordinate z-axis. Algorithm 4 outlines the procedure to obtain the steepness of the image.

Algorithm 4 Procedure for obtaining the steepness map

Input:  Point Cloud Patches Output: SteepnessMap   1: Normals ← ComputeNormals(Point Cloud Patches)   2: for $i=0$ to N do   3:    $\psi$← ComputePsi (Normals [i])   4:    ReprojectedImg ← Reproject(Point Cloud Patches [i])   5:    SteepnessMap[ReprojectedImg] = Compute_f4($\psi$)   6: end for   7: SteepnessMap ← Min-Max Normalization(InclinationMap)   8: return SteepnessMap

2.4. Fuzzy Landing Map

Once the four map metrics are obtained using the image processing techniques implemented on the depth image and point cloud, a final map that combines these maps is required to ascertain safe landing site detection. A fuzzy inference engine is utilized to generate a fuzzy landing score image in which the intensity values of each pixel in each of the four images are considered the input vector of the fuzzy system, and the output is the intensity of the landing image. Despite the wide variety of fuzzy systems, the Mamdani-type system is chosen in this paper. This model has been used widely to express the relationship between inputs and outputs.

2.5. Finding the Best Points to Land

Once the fuzzy landing map is constructed, binary thresholding may be used to create a binary mask serving as a filter to eliminate a subset of active ORB-SLAM3 world points. Nevertheless, the world points corresponding to the current keyframe should be projected back onto the image plane so that a grayscale image with the active keypoints as pixels with one intensity (white-colored pixels) is generated. Once a subset of the image points is identified, reverse indexing is utilized to recover the three-space ORB points. As a result, a k-means clustering technique is applied to the candidate point set to detect the best set of the dense landing points. The points organized in a k-d tree are searched to find their corresponding normal vectors, and if they are close to each other, they are classified as a unique group. The centroid of the cluster with the minimum normal variance is selected as the safest landing point. The number of clusters can impact the identification of the flattest patch if its area does not meet the minimum dimensions $(L_1,L_2)$ constraint, resulting in discarding the selection by the algorithm. Conversely, setting the number of clusters to a small value can lead to inaccuracies and the detection of unsafe landing zones. For the image-based control to be performed, there should be at least three corner points. To achieve this, the best point to land is considered as $\tilde{\mathbf{P}}=({\tilde{p}}_x,{\tilde{p}}_y,{\tilde{p}}_z)$, and the dimensions of the rectangular area on which the UAV aims to land are $(L_1,L_2)$, so the coordinates of the synthesis points are selected to be as follows:

$${\mathbf{P}}^{*}=\left(\begin{array}{c}{\tilde{p}}_x+L_1,{\tilde{p}}_y+L_2,{\tilde{p}}_z\\ {\tilde{p}}_x+L_1,{\tilde{p}}_y-L_2,{\tilde{p}}_z\\ {\tilde{p}}_x-L_1,{\tilde{p}}_y+L_2,{\tilde{p}}_z\\ {\tilde{p}}_x-L_1,{\tilde{p}}_y-L_2,{\tilde{p}}_z\end{array}\right).$$

(10)

2.6. Reference Frame Attachment

The coordinate reference frames are depicted in Figure 2. Before presenting the governing dynamics of the UAV and the image features, the coordinate systems should be determined to facilitate transformations between frames. The inertial frame is shown with $I=\{O_I,X_I,Y_I,Z_I\}$ where the $Z_I$ axis is parallel to the gravitational acceleration vector $\mathbf{g}={[0,0,g]}^{\top }$. The $X_I$ and $Y_I$ axes point North and East, respectively, and should be treated as positive directions when deriving the system’s equations of motion. $B=\{O_B,X_B,Y_B,Z_B\}$ indicates the fixed-body frame, which is initially aligned with the inertial reference frame. To discuss the dynamics of visual data, the real and virtual camera frames are defined as $C=\{O_C,X_C,Y_C,Z_C\}$ and $V=\{O_V,X_V,Y_V,Z_V\}$, respectively. The latter is an imaginary coordinate frame coinciding with the camera frame with a different orientation. More specifically, while the yaw angles are the same, the roll and pitch angles of the virtual frame differ in the sense that they are always kept zero with respect to the inertial frame, ensuring that two $X_V{Y}_V$ and $X_I{Y}_I$ planes remain parallel. The Z–Y–X Tait–Bryan angles (intrinsic rotation) are utilized here to obtain the rotation of a coordinate frame.

2.7. Quadrotor’s Equations of Motion

In this section, the UAV’s translational and rotational equations of motion, expressed in the fixed-body frame, are derived employing the Newton–Euler equation of rigid body dynamics [24], as follows:

$$\begin{array}{cc}\hfill {}^B\dot{\mathbf{P}}& ={}^B\mathbf{V},\hfill \\ \hfill {}^B\dot{\mathbf{V}}& ={}^B\mathbf{V}{\left[{}^B\mathbf{\omega }\right]}_{\times }-{\displaystyle \frac{1}{m}}{U}_1{\mathbf{E}}_3+g{{}^B\mathbf{R}}_I{\mathbf{e}}_3,\hfill \\ \hfill {{}^I\dot{\mathbf{R}}}_B& ={{}^I\mathbf{R}}_B{\left[{}^B\mathbf{\omega }\right]}_{\times },\hfill \\ \hfill {}^B\dot{\mathbf{\omega }}& ={\mathbf{J}}^{-1}(-{}^B\mathbf{\omega }\times \mathbf{J}{}^B\mathbf{\omega }+\mathbf{\tau }),\hfill \end{array}$$

(11)

where m is the mass of the UAV, $\mathbf{J}\in {\mathbb{R}}^{3\times 3}$ is the moment of inertia of the UAV, $\mathbf{\tau }\in {\mathbb{R}}^3$ is the moment vector expressed in body frame, and $U_1\in \mathbb{R}$ is the total thrust force generated by the four propellers. g is the acceleration due to gravity, $\mathit{\omega }\in {\mathbb{R}}^3$ is the angular velocity vector, $\mathit{P}\in {\mathbb{R}}^3$ is the position of the UAV, and $\mathit{V}\in {\mathbb{R}}^3$ is the linear speed of the UAV. Nevertheless, if there is a transformation between the UAV and the camera, then the speed of the camera is a function of the speed of the UAV. Since there is no prismatic or revolute joint between the camera and UAV, the linear speed of the camera and the UAV is the same in the virtual frame; thus, ${{}^V\mathbf{V}}_c={{}^V\mathbf{V}}_b$. Hence, the UAV’s translational dynamics must be expressed in the virtual camera frame. With this frame attachment, the following rotation matrices are obtained:

$${{}^I\mathbf{R}}_V={\mathbf{R}}_z\left(\psi \right),$$

(12)

$${{}^I\mathbf{R}}_B={\mathbf{R}}_z\left(\psi \right){\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right),$$

(13)

$${{}^I\mathbf{R}}_B={{}^I\mathbf{R}}_V{{}^V\mathbf{R}}_B,$$

(14)

$${{}^V\mathbf{R}}_B={{}^V\mathbf{R}}_I{{}^I\mathbf{R}}_B={\mathbf{R}}_z{\left(\psi \right)}^T{\mathbf{R}}_z\left(\psi \right){\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)={\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right).$$

(15)

Using the transformations (12) to (15), the thrust force in the virtual frame can be expressed as follows:

$$-{{}^V\mathbf{R}}_B{U}_1{\mathbf{E}}_3=-{\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)U_1{\mathbf{E}}_3.$$

(16)

The force exerted by the gravity is calculated as follows:

$${{}^V\mathbf{R}}_{I}mg{\mathbf{e}}_3={\mathbf{R}}_z{\left(\psi \right)}^{T}mg{\mathbf{e}}_3=mg{\mathbf{e}}_3.$$

(17)

The following equation describing the UAV’s translational dynamics can be expressed in the virtual reference frame as follows:

$${}^V\dot{\mathbf{V}}={}^V\mathbf{V}{\left[{}^B\dot{\mathit{\psi }}{\mathbf{e}}_3\right]}_{\times }-{\displaystyle \frac{1}{m}}{\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)U_1{\mathbf{E}}_3+{\displaystyle \frac{1}{m}}{}^V{\mathbf{F}}_d.$$

(18)

The attitude dynamics equations are derived with respect to the body-fixed frame.

2.8. Visual Data Dynamics

To derive the image feature dynamics, first, the world points ${}^I\tilde{\mathbf{P}}$ are expressed w.r.t the camera coordinate frame as follows:

$${}^C\mathbf{P}={{}^C\mathbf{R}}_I{}^I\mathbf{P}-{{}^I\mathbf{t}}_C={\left({{}^I\mathbf{R}}_C\right)}^T({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(19)

By expanding the rotation matrix ${{}^C\mathbf{R}}_I$ in terms of the basic rotations, one can obtain the following:

$${}^C\mathbf{P}={\left({\mathbf{R}}_z\left(\psi \right){\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)\right)}^T({}^I\mathbf{P}-{{}^I\mathbf{t}}_C),$$

(20)

$${}^C\mathbf{P}=\left({\mathbf{R}}_x{\left(\varphi \right)}^{\top }{\mathbf{R}}_y{\left(\theta \right)}^T{\mathbf{R}}_z{\left(\psi \right)}^T\right)({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(21)

As it is seen in Figure 2, the pitch and roll angles are zero for the virtual camera frame. Hence,

$${\mathbf{R}}_x\left(\varphi \right)={\mathbf{R}}_y\left(\theta \right)={\mathbf{I}}_3.$$

(22)

Thus, the coordinates of the world point in the virtual frame are as follows:

$${}^V\mathbf{P}={\mathbf{R}}_z{\left(\psi \right)}^T({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(23)

Differentiating (23) with respect to time results in the following:

$${}^V\dot{\mathbf{P}}={\displaystyle \frac{d}{dt}}{\left({\mathbf{R}}_z\left(\psi \right)\right)}^T({}^I\mathbf{P}-{{}^I\mathbf{t}}_C)+{\mathbf{R}}_z{\left(\psi \right)}^T{\displaystyle \frac{d}{dt}}({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(24)

Given that the time derivative of the rotation matrix is equal to the product of a skew-symmetric matrix of angular velocity and the rotation matrix itself, it follows that

$${\displaystyle \frac{d}{dt}}{\mathbf{R}}_z\left(\psi \right)=\omega \times {\mathbf{R}}_z\left(\psi \right)={\left[\omega \right]}_{\times }{\mathbf{R}}_z\left(\psi \right),$$

(25)

$${\left({\displaystyle \frac{d}{dt}}{\mathbf{R}}_z\left(\psi \right)\right)}^T={\left({\left[\omega \right]}_{\times }\right)}^T{\mathbf{R}}_z{\left(\psi \right)}^T=-{\left[\omega \right]}_{\times }{\mathbf{R}}_z{\left(\psi \right)}^T,$$

(26)

where $\mathit{\omega }={[{\omega }_x,{\omega }_y,{\omega }_z]}^T$. Because the pitch and roll angles of the virtual plane are zero, $\mathit{\omega }={[0,0,{\omega }_z]}^T={\dot{\psi }}_z{\mathbf{E}}_3$. Therein ${\mathbf{E}}_3$ represents the unit vector aligned with the z-axis of the inertial frame. Substituting $\mathit{\omega }={\dot{\psi }}_z{\mathbf{E}}_3$ into (26) yields:

$${\left({\displaystyle \frac{d}{dt}}{\mathbf{R}}_z\left(\psi \right)\right)}^T=-{\left({\left[{\dot{\psi }}_z{\mathbf{E}}_3\right]}_{\times }\right)}^T{\mathbf{R}}_z{\left(\psi \right)}^T=-{\left[{\dot{\psi }}_z{\mathbf{e}}_3\right]}_{\times }{\mathbf{R}}_z{\left(\psi \right)}^T,$$

(27)

where ${\mathbf{e}}_3$ is the unit vector describing the z-axis of the virtual frame. In (27), since the rotation occurs around the z-axis of the virtual frame, it can be inferred that ${\mathbf{E}}_3$ and ${\mathbf{e}}_3$ are aligned, and the rotation does not affect the orientation of the z-axis of the virtual frame. Therefore, (24) can be rewritten as follows:

$${}^V\dot{\mathbf{P}}=-{\left[{\dot{\psi }}_z{\mathbf{e}}_3\right]}_{\times }{\mathbf{R}}_z{\left(\psi \right)}^T({}^I\mathbf{P}-{{}^I\mathbf{t}}_C)+{\mathbf{R}}_z{\left(\psi \right)}^T{\displaystyle \frac{d}{dt}}({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(28)

Based on (23), the coordinates of a point in the image plane will be represented as in (28), which gives

$${}^V\dot{\mathbf{P}}=-{\left[{\dot{\psi }}_z{\mathbf{e}}_3\right]}_{\times }{}^V\mathbf{P}+{\mathbf{R}}_z{\left(\psi \right)}^T{\displaystyle \frac{d}{dt}}({}^I\mathbf{P}-{{}^I\mathbf{t}}_C).$$

(29)

The equation above can be reformulated in terms of the linear velocities of the camera, ${{}^V\mathbf{v}}_C$ as follows:

$${}^V\dot{\mathbf{P}}=-{\left[{\dot{\psi }}_z{\mathbf{e}}_3\right]}_{\times }{}^V\mathbf{P}+{\mathbf{R}}_z{\left(\psi \right)}^T{}^I\dot{\mathbf{P}}-{\mathbf{R}}_z{\left(\psi \right)}^T{{}^I\mathbf{v}}_C,$$

(30)

$${}^V\dot{\mathbf{P}}=-{\left[{\dot{\psi }}_z{\mathbf{e}}_3\right]}_{\times }{}^V\mathbf{P}-{{}^V\mathbf{v}}_C.$$

(31)

Let ${}^V\mathbf{P}=[{}^{V}x,{}^{V}y,{}^{V}z]$, the virtual image plane coordinates are reported as follows:

$${}^{V}u=f{\displaystyle \frac{{}^{V}x}{{}^{V}z}},{}^{V}v=f{\displaystyle \frac{{}^{V}y}{{}^{V}z}},$$

(32)

where f is the focal length of the camera identified by calibration. To obtain the dynamics of the virtual image plane coordinates, the derivative of (32) should be taken, incorporating (31) in the process:

$$\left[\begin{array}{c}{}^V\dot{u}\\ {}^V\dot{v}\end{array}\right]=\left[\begin{array}{ccc}-{\displaystyle \frac{f}{{}^{V}z}}& 0& {\displaystyle \frac{{}^{V}u}{{}^{V}z}}\\ 0& -{\displaystyle \frac{f}{{}^{V}z}}& {\displaystyle \frac{{}^{V}v}{{}^{V}z}}\end{array}\right]\left({{}^V\mathbf{v}}_C\right)+\left[\begin{array}{c}{}^{V}v\\ -{}^{V}u\end{array}\right]\dot{\psi }.$$

(33)

The features chosen to control the motion of the UAV are defined as follows:

$$q_z=\sqrt{{\displaystyle \frac{a_d}{a}}},q_x=q_z{\displaystyle \frac{{{}^{V}u}_g}{f}},q_y=q_z{\displaystyle \frac{{{}^{V}v}_g}{f}}.$$

(34)

Therein, ${{}^{V}u}_g$ and ${{}^{V}u}_g$, are the central gravity coordinates, and the moments $I_{ij}$ are defined as follows:

$$I_{ij}(u,v)=\sum _{k=1}^N{u}_k^i{v}_k^j,{{}^{V}u}_g={\displaystyle \frac{I_{10}}{I_{00}}},{{}^{V}v}_g={\displaystyle \frac{I_{01}}{I_{00}}}.$$

(35)

Furthermore, a is defined leveraging the second central moments:

$$M_{ij}(u,v)=\sum _{k=1}^N{(u_k-u_g)}^i{(v_k-v_g)}^j,$$

(36)

$$a=M_{02}+M_{20}.$$

(37)

The dynamics of the image features in the virtual image plane is represented by $a_d$, which serves as the setpoint for a:

$$\dot{\mathbf{q}}=-sk\left(\dot{\psi }{\mathbf{e}}_3\right)\mathbf{q}-{\displaystyle \frac{1}{z_d}}{{}^V\mathbf{v}}_C,$$

(38)

where $z_d$ is the depth value when the camera is at the desired pose.

2.9. IBVS Control Design

The block diagram of the control system is illustrated in Figure 3. The best corner points for landing, ${\mathbf{P}}^{*}$, were obtained in the previous section in (10). Considering the camera projection matrix $\mathbf{P}=\mathbf{K}\left[{\mathbf{R}}_i\right|{\mathbf{t}}_i]$, the projected points on the camera frame are generated by the estimated camera pose ${\mathbf{T}}_i$ using ORB-SLAM3 in Section 2.2. Using a calibrated camera with the intrinsic parameters $\mathbf{K}$, the pixel coordinates can be obtained. The control law is to minimize the image feature error, ensuring that the image features reach their predetermined values in finite time. The vector of image feature error can be defined as follows:

$${\mathbf{e}}_q={\mathbf{q}}_d-\mathbf{q},$$

(39)

Therein, ${\mathbf{q}}_d={\left[\begin{array}{ccc}{q}_{x_d}& q_{y_d}& q_{z_d}\end{array}\right]}^T\in {\mathbb{R}}^3$ and $\mathbf{q}={\left[\begin{array}{ccc}{q}_x& q_y& q_z\end{array}\right]}^T\in {\mathbb{R}}^3$ are the pre-specified and current values of the image feature vectors, respectively. An acceptable error for each element of ${\mathbf{e}}_q$ can be considered to be less than 1 pixel. By substituting (39) into (33), the dynamics of the feature error are obtained as follows:

$${\dot{\mathbf{e}}}_q=-sk\left(\dot{\psi }{\mathbf{e}}_3\right)\mathbf{q}-{\displaystyle \frac{1}{z_d}}\left({{}^V\mathbf{v}}_C\right).$$

(40)

With the assumption that the desired feature vector is constant, differentiating (40) with respect to time yields the following relation:

$${\ddot{\mathbf{e}}}_q=-sk\left(\dot{\psi }{\mathbf{e}}_3\right)\dot{\mathbf{q}}-sk\left(\ddot{\psi }{\mathbf{e}}_3\right)\mathbf{q}-{\displaystyle \frac{1}{z_d}}\left({{}^V\dot{\mathbf{v}}}_C\right).$$

(41)

By utilizing the equations that describe the dynamics of the UAV in the virtual image frame, the above equation can be expressed in terms of the feature error vector. Consequently, combining the UAV dynamics with the image dynamics results in

$${\ddot{\mathbf{e}}}_q+2{\mathbf{d}}_1{\dot{\mathbf{e}}}_q+{\mathbf{d}}_2{\mathbf{e}}_q-{\displaystyle \frac{1}{m}}{\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)U_1{\mathbf{E}}_3=\mathbf{0},$$

(42)

where

$${\mathbf{d}}_1=-sk\left(\dot{\psi }{\mathbf{e}}_3\right)\in {\mathbb{R}}^{3\times 3},$$

(43)

$${\mathbf{d}}_2=-sk\left(\ddot{\psi }{\mathbf{e}}_3\right)+{\mathbf{d}}_1^2\in {\mathbb{R}}^{3\times 3}.$$

(44)

It is required to determine the control vector ${\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)U_1{\mathbf{E}}_3$ that can globally and asymptotically stabilize both the feature error ${\mathbf{e}}_q$ and its derivative ${\dot{\mathbf{e}}}_q$. To accomplish this, a cascaded controller is introduced. In this setup, the feed-forward open-loop block provides the desired values for $U_1$, $\varphi$, and $\theta$, which are computed by the outer loop controller. The thrust is applied directly to the UAV, while the desired values are used in the attitude control loop due to the UAV’s under-actuated characteristics. Let $\mathbf{u}$ be the control effort capable of stabilizing the closed-loop system. As a result, one can obtain the following:

$${\mathbf{R}}_y\left(\theta \right){\mathbf{R}}_x\left(\varphi \right)U_1{E}_3=\mathbf{u},$$

(45)

$$\left[\begin{array}{c}{u}_x\\ u_y\\ u_z\end{array}\right]=\left[\begin{array}{ccc}cos\left(\theta \right)& 0& sin\left(\theta \right)\\ 0& 1& 0\\ -sin\left(\theta \right)& 0& cos\left(\theta \right)\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\left(\varphi \right)& sin\left(\varphi \right)\\ 0& sin\left(\varphi \right)& cos\left(\varphi \right)\end{array}\right]\left[\begin{array}{c}0\\ 0\\ U_1\end{array}\right],$$

(46)

$$\left[\begin{array}{c}{u}_x\\ u_y\\ u_z\end{array}\right]=\left[\begin{array}{c}sin\left(\theta \right)cos\left(\varphi \right)U_1\\ -sin\left(\varphi \right)U_1\\ cos\left(\theta \right)cos\left(\varphi \right)U_1\end{array}\right].$$

(47)

The attitude loop employs a straightforward PD controller to track the desired pitch and roll angles as follows:

$${\theta }_{des}=arctan\left({\displaystyle \frac{u_x}{u_z}}\right),{\varphi }_{des}=arctan(-{\displaystyle \frac{u_{x}cos{\theta }_{des}}{u_z}}).$$

(48)

The magnitude of the thrust vector is described as follows:

$$U_1=\left|\right|\mathbf{u}\left|\right|.$$

(49)

The attitude inner loop control laws can be stated as follows:

$${\tau }_1={\beta }_1({\theta }_{des}-\theta )+{\beta }_2{\displaystyle \frac{d}{dt}}({\theta }_{des}-\theta ),$$

(50)

$${\tau }_2={\mu }_1({\varphi }_{des}-\varphi )+{\mu }_2{\displaystyle \frac{d}{dt}}({\varphi }_{des}-\varphi ).$$

(51)

A new variable, namely filter error, is introduced here:

$${\mathbf{e}}_f={\mathbf{K}}_e{\mathbf{e}}_q+{\dot{\mathbf{e}}}_q,$$

(52)

Based on what is stated in [25], if one can prove that the filtered error is bounded, the image feature error and its time differentiation will also be limited. According to [26], the following relations are satisfied:

$$\left|\right|{\mathbf{e}}_q\left|\right|\le \left|\right|{\mathbf{e}}_f\left|\right|/{\sigma }_{min}\left({\mathbf{K}}_e\right),$$

(53)

wherein ${\sigma }_{min}\left({\mathbf{K}}_e\right)$ represents the smallest singular value of ${\mathbf{K}}_e$. Moreover, the feature error is upper bound by

$$\left|\right|{\dot{\mathbf{e}}}_q\left|\right|\le \left|\right|{\mathbf{e}}_f\left|\right|.$$

(54)

The first derivative of filtered feature error, ${\dot{\mathbf{e}}}_f$, can be derived as follows using (42):

$${\dot{\mathbf{e}}}_f=({\mathbf{K}}_e-2{\mathbf{d}}_1){\dot{\mathbf{e}}}_q-{\mathbf{d}}_2{\mathbf{e}}_q+{\displaystyle \frac{1}{m{z}_d}}\mathbf{u}.$$

(55)

Theorem 1.

The image feature error dynamics stated in (55) is globally asymptotically stable under the following closed-loop control law:

$${\mathbf{u}}_c=m{z}_d(-({\mathbf{K}}_e-2{\mathbf{d}}_1)({\mathbf{e}}_f-{\mathbf{K}}_e{\mathbf{e}}_q)+{\mathbf{d}}_2{\mathbf{e}}_q-{\mathbf{K}}_p{\mathbf{e}}_f-{\mathbf{K}}_i{\int }_0^t{\mathbf{e}}_{f}dt).$$

(56)

Proof. 

To prove the stability of the error dynamics, the control law suggested in (56) is substituted into (55). Consequently, after simplification, the following error dynamics are derived:

$${\dot{\mathbf{e}}}_f=-{\mathbf{K}}_p{\mathbf{e}}_f-{\mathbf{K}}_i{\int }_0^t{\mathbf{e}}_{f}dt.$$

(57)

Therein, ${\mathbf{K}}_p$ and ${\mathbf{K}}_i$ are selected to be positive definite matrices, representing the control gains in diagonal matrix configuration. Let the following be the Lyapunov candidate in which ${\int }_0^t{\mathbf{e}}_{f}dt={\mathbf{e}}_i$:

$$V={\displaystyle \frac{1}{2}}({\mathbf{e}}_f^T{\mathbf{e}}_f+{\mathbf{e}}_i^T{\mathbf{K}}_i{\mathbf{e}}_i).$$

(58)

By taking the derivative of V with respect to time:

$$\dot{V}=({\mathbf{e}}_f^T{\dot{\mathbf{e}}}_f+{\mathbf{e}}_i^T{\mathbf{K}}_i{\mathbf{e}}_f).$$

(59)

By substituting (55) into (59), one can get:

$$\dot{V}=({\mathbf{e}}_f^T{\dot{\mathbf{e}}}_f+{\mathbf{e}}_i^T{\mathbf{K}}_i{\mathbf{e}}_f),$$

(60)

$$\dot{V}=-({\mathbf{e}}_f^T({\mathbf{K}}_p{\mathbf{e}}_f-{\mathbf{K}}_i{\mathbf{e}}_i)+{\mathbf{e}}_i^T{\mathbf{K}}_i{\mathbf{e}}_f).$$

(61)

Expanding the above equation, one can obtain:

$$\dot{V}=-{\mathbf{e}}_f^T{\mathbf{K}}_p{\mathbf{e}}_f+{\mathbf{e}}_f^T{\mathbf{K}}_i{\mathbf{e}}_i-{\mathbf{e}}_i^T{\mathbf{K}}_i{\mathbf{e}}_f.$$

(62)

Lemma 1.

The following equation holds whenever $\mathbf{K}\in {\mathbb{R}}^{N\times N}$ is a diagonal matrix and $\mathbf{A}\in {\mathbb{R}}^{N\times 1}$ and $\mathbf{B}\in {\mathbb{R}}^{N\times 1}$, consequently:

$${\mathbf{A}}^T\mathbf{K}\mathbf{B}={\mathbf{B}}^T\mathbf{K}\mathbf{A}.$$

(63)

Proof. 

Readers may refer to [27] for the proof.

□

Based on Lemma 1, (62) can be simplified as follows:

$$\dot{V}=-{\mathbf{e}}_f^T{\mathbf{K}}_p{\mathbf{e}}_f\le 0.$$

(64)

By selecting ${\mathbf{K}}_p$ as a positive definite matrix, it is guaranteed that $\dot{V}$ is negative semidefinite, and the filtered error always remains bounded. As a result, the closed-loop system is always stable, completing the proof.

□

To ensure the stability of the dynamics of the yaw controller, the following image feature based on the second-order moments is defined:

$$q_{\psi }={\displaystyle \frac{1}{2}}arctan\left({\displaystyle \frac{2M_{11}}{M_{20}-M_{02}}}\right).$$

(65)

The scalar feature in (65) yields information regarding the orientation of the target in the virtual image plane for the UAV heading control. Nevertheless, one can take advantage of the fact that the target is not skewed and use the angle between the line connecting any two features in the image plane and the positive u-axis of the pixel coordinate frame. Accordingly, the image feature tracking error related to the heading angle is as follows:

$$e_{q_{\psi }}=q_{{\psi }_{des}}-q_{\psi },$$

(66)

where $q_{{\psi }_{des}}$ is the desired angle of the line with respect to the positive u-axis of the pixel coordinates frame. The following proportional control is used here to track the desired heading angle:

$${\tau }_3=k_{\psi }{e}_{q_{\psi }},$$

(67)

therein, $k_{\psi }$ is a positive real number.

3. Results

To demonstrate the effectiveness and efficiency of the landing algorithm in an outdoor and indoor, unknown environment, three worlds were built in the Gazebo (Gazebo simulation software: [online]. Available: https://gazebosim.org, accessed on 1 June 2024) simulator software shown in Figure 4. A simulated Parrot Bebop Drone equipped with an Intel RealSense Depth Camera D435 facing downwards, capturing the depth images and point cloud. The specifications of the drone used in the simulations are outlined in

Table 1. Simulated Parrot Bebop drone and camera parameters.

Parameter	Value	Unit
UAV Mass (with hull)	$0.399$	kg
UAV Moment of Inertia	$i_{xx}=i_{yy}=0.01152$$i_{zz}=0.0218$	$\mathrm{kg}.{\mathrm{m}}^2$
Color Camera Focal Length	$462.137$	pixel
Color Camera Principal Point	$(320,240)$	pixel
Radial/Tangential Distortion	ignored	-

[28]. The dimensions of the multirotor are $328\times 328\times 89$ (mm), which is considered in the landing strategy as a parameter. Based on the dimensions of the UAV, the values of $L_1=L_2$ are set to 656 mm. The ROS publisher and subscriber nodes are developed in C++14 and Python 3.8.10 and executed by a 12th Gen Intel(R) Core(TM) i7-12700 CPU. The entire landing operation is completed within a 20 s interval. Four different scenarios are considered, where in the first two the Gazebo world is the same but the initial position of the UAV differs, while in the third and last scenarios, two different environments are created to assess the robustness of the algorithm.

The pixel intensities are normalized to ensure a consistent domain. Triangular membership functions are then adopted for both the input and output fuzzy spaces, each with an equal support set representing the degree of appropriateness for landing. Three membership functions within the universe of discourse for each input variable are defined. In addition, the output space is fuzzified with seven membership functions that indicate the safety of landing at specific pixel coordinates. The construction of the rule base underwent two major phases. In the first phase, the general form of the rules was designed based on human approximate reasoning. For instance, if flatness and steepness are appropriate but the other two metrics have large degrees in the intermediate membership functions, the point is highly suitable for landing. The second stage involved fine-tuning the input and output membership functions, which was achieved through numerous trials and errors.

The Visual Monocular ORB-SLAM3 package (ORB-SLAM3: [online]. Available: https://github.com/UZ-SLAMLab/ORB_SLAM3, accessed on 1 June 2024) with the ROS-enabled branch is adopted in the simulations. The realsnese RGB image topic in ROS has been mapped to the input frame topic of the ORB-SLAM3. Additionally, the frame attachment is required to be adjusted according to the frames defined for the camera. The configuration file is also updated to match the intrinsics of the camera. For example, the focal length and principal values are set to $462.137$ and $(320,240)$ pixels, respectively. The distortion parameters are ignored in the simulations, and the frame per second is set to 30. The ORB feature extractor finds 1000 ORB points along with their corresponding descriptors for each frame. The number of levels and the scale factor between them in the image pyramid are reported to be 8 and $1.2$, respectively.

3.1. The First Scenario

The initial position of the quadrotor is $(0,0,10)$ $\mathrm{m}$ in the world reference frame, and the goal is to reach an altitude of approximately 2 $\mathrm{m}$ at some safe landing location in the world. Figure 5 illustrates the four landing metrics characterizing the terrain when the fuzzy landing image is applied. The appropriateness of the landing site is illustrated through a reversed heat map convention where the zero-intensity pixels are the best, and conversely, the light-colored pixels indicate the least suitable locations. For a more straightforward visual representation, heat maps were used to display the complement of the original pixel values rather than the values themselves. The depth variance map (Figure 5a) is constructed with a window size of 3 and shows the existing depth discontinuities of the scene; however, the surface of the trees, for example, is counted as safe regions for landing. Compared with the depth variance image, the flatness image introduces more reliable information, which has been incorporated into the rule base of the fuzzy system. The inclination map metric (Figure 5c) determines how inclined each surface is and is of high importance since it can enhance the performance of the landing site detection by assigning a low pixel intensity to the non-inclined surfaces. Nonetheless, some small objects, such as the postbox, are missed due to the large value of $C_z$, expressed in (7). Lastly, Figure 5d illustrates the steepness map constructed using the point cloud based on (8) and (9). Even though it contributed to the identification of the areas with large steepness, it failed to eliminate the pine tree from the suitable spots for landing.

Figure 6 shows the fuzzy landing score map and other SLAM-based images used for finding the landing zone. The fuzzy landing map shows a good performance in combining the four landing map metrics by excluding all of the present objects and humans from the landing sites. Nonetheless, a minimum filter is applied to this fuzzy image to further enhance the safety of this landing map, and the result is shown in Figure 6Aa. Sparse SLAM world keypoints, which are active for this keyframe, back-projected into the image plane depicted in Figure 6Ab. The fuzzy image is then transformed into a binary image using global thresholding and then used as a mask on the SLAM back-projected keypoints to select a subset of them that are the best for landing (see Figure 6Ac). Among these points, the best points for landing that meet the quadrotor area constraint are selected as shown in (see Figure 6Ad).

The suggested method was compared with the state-of-the-art landing method in [16], where Bayesian classification of landing points is suggested. The results of this method are highlighted in Figure 6Ba. Figure 6Bd demonstrates the back-projected active SLAM keypoints in the image plane. The filtered subset of these points and the corresponding image points of the SLAM flattest 3D patch are shown in Figure 6Bc and Bd, respectively. Areas such as the back of the pick-up truck are detected as good sites for landing, whereas in the proposed method that area is marked as areas not recommended for landing. An additional advantage is the safety of the landing, as the suggested method avoids landing close to areas where there is an object or person. However, the time taken to construct the fuzzy landing image is about 984 ms, while the same quantity is reported to be 454 ms for the Bayesian method. Despite these differences, the final detected landing mean of the world points is very close to each other. The coordinates of the world point are computed to be $(3.24,-3.32,0.4)$ and $(4.48,-2.74,0.45)$ meters for the proposed method and Bayesian method, respectively. Readers may refer to

Table 2. Comparison of experimental indicators for all scenarios.

Experimental Indicator	Value	Unit
Average landing zone detection time	$1367.43$	ms
IBVS stabilization time	$7.81$	s
Landing success rate	$100\%$	Not Applicable
Distance error	$1.249$	m

to view the key experimental indicators for all scenarios. The fuzzy inference accounts for approximately 71.9% of the total average time required for landing zone detection, due to its high complexity. Moreover, the algorithm did not lead to any quadrotor crashes or risky landings across various scenarios. Finally, the mean Euclidean distance between the best landing points of the proposed method and the state-of-the-art method was 1.249 m across all scenarios.

The key parameters used in the IBVS control system and image processing blocks are listed in

Table 3. The parameters values used in the simulations.

Parameter	Value	Unit
${\mathbf{K}}_e$	diag($[0.5,0.5,0.5]$)	dimensionless
${\mathbf{K}}_i$	diag($[0.1,0.1,0.001]$)	dimensionless
${\mathbf{K}}_p$	diag($[5,5,5]$)	dimensionless
$k_{\psi }$	$0.7$	dimensionless
${\beta }_1$	2	dimensionless
${\beta }_2$	$0.2$	dimensionless
${\mu }_1$	2	dimensionless
${\mu }_2$	$0.2$	dimensionless
Depth image variance kernel size in Algorithm 1	$(3,3)$	pixel
Canny lower and upper values in hysteresis thresholding in Algorithm 2	$(50,220)$	dimensionless
Distance transformation’s mask size in Algorithm 2	$(5,5)$	pixel
KNN search radius in Algorithms 3 and 4	$0.5$	m

. The parameter ${\mathbf{K}}_e$ controls the upper and lower limits, ${\mathbf{K}}_i$ eliminates the steady-state offset, and ${\mathbf{K}}_p$ controls the convergence rate of the image feature tracking error. ${\beta }_i$ and ${\mu }_i$, where $i=1,2$, affect the transient response characteristics of roll and pitch angles, respectively. The window size in Algorithm 1 impacts the sensitivity to depth variations. By using two threshold values, Algorithm 2, one can identify and link edges. Narrower bounds result in more edges but can also be affected by image noise. The kernel size in the distance transformation controls the accuracy of the distance to the nearest nonzero pixel. The KNN search radius can influence the number of patches in point clouds.

The mean of the detected cluster of the world points is considered the center of the square with sides obtained from the quadrotor dimensions that will be used to conduct IBVS landing for both techniques. The results of the perception-guided landing using image moment features presented in (11)–(62) are shown Figure 7a and b for the proposed method and the state-of-the-art method, respectively. The target pose of the camera is considered to be a fronto-parallel view in both cases. The feature error development is outlined in Figure 8, where the feature errors defined in (39) finally converge to a small value, indicating that the camera reached the desired pose. It is worth highlighting that there is a static transformation between the camera optical frame and the quadrotor frame; consequently, the twist applied to the camera is mapped to the quadrotor local frame.

3.2. The Second Scenario

In the second scenario, the same environment is adopted as in the first scenario. However, the quadrotor is spawned in a different position compared with the first experiment to showcase the robustness of the method against the initial position variations. The UAV is initially at the $(-3,12,9)$ $\mathrm{m}$ expressed in the inertial frame. Heat maps illustrated in Figure 9 report the previously discussed metrics. The edges are assigned low scores in Figure 9a, while the flatness map discarded has proved highly effective by excluding the trees and the cars despite eliminating safe space as shown in Figure 9b. The inclination map also obtains the vast majority of hazardous locations for landing with minor inaccuracies such as misclassifying the back of the pickup truck located on the bottom left corner of the map, as it can be observed in Figure 9c. Figure 9d indicates the steepness of the terrain, which lacks precision in detecting small objects like debris. The results of the fuzzy and Bayesian inference, which combine all of the maps, are illustrated in Figure 10A and B, respectively. It is evident that the best image points to land are obtained in the bottom right corner, where there are no objects. Lastly, the image plane trajectories for both methods and the evolution of the image feature error are depicted in Figure 11 and Figure 12, respectively. These two figures indicated that the features will converge to their desired values in less than 10 s.

3.3. The Third Scenario

The third scenario employs a different world in comparison to the other two. A playground has been selected as an environment to show the versatility of the suggested approach. The results of the simulation are presented in Figure 13, Figure 14, Figure 15 and Figure 16. The environment is more cluttered and unstructured than in the first scenario. The left part of the map, as seen in Figure 13, is occupied by trees and an electricity pole, both of which are correctly identified by the steepness and inclination map. Figure 14A,B show the results for the designed approach and the state-of-the-art method, respectively. It can be generally deduced that the suggested approach is more reliable since there is more space between the points for landing an object with depth variations. The rest of the figures, Figure 15 and Figure 16, show how the IBVS control is successfully established, similar to the other scenarios.

3.4. The Fourth Scenario

An indoor office environment is considered to assess the performance of the landing technique. The quadrotor is initially positioned at $(0,0,10)$ $\mathrm{m}$ expressed in the global frame. Figure 17 illustrates the map metrics for the initial frame. As shown in this figure, the edges are assigned with high-intensity values in the depth variation, and the flat areas are detected correctly. Furthermore, the stairs are identified as regions with high inclination and steepness. Figure 18A,B indicates the fuzzy and Bayesian maps for this scene, where a subset of SLAM keypoints are selected as the best landing point. The image-plane features and tracking error are shown in Figure 19 and Figure 20, respectively. These figures validate that the image feature tracking error converges to zero in a finite time.

3.5. Sensitivity Analysis Environmental Conditions

In this subsection, the effect of changing ground materials and illumination variations was investigated on the landing metrics and fuzzy map. The experiments were conducted while the initial position of the UAV and scenery elements, such as the solar panels (except for the ground), remained stationary. Three distinct ground types were considered: grassy, hard, and sandy, to cover a reasonable range of materials. The results, along with the maps, are illustrated in Figure 21, Figure 22 and Figure 23 for grassy, hard, and sandy ground, respectively. Depth variation and flatness demonstrated high sensitivity to these factors; however, inclination and steepness remained almost unchanged, showing tolerance toward these variations. The final fuzzy map shows similar outputs, demonstrating the method’s effectiveness and strong generalization, particularly in areas with a high risk of landing. Additionally, the proposed algorithm was tested under both low and high illumination conditions and observed no changes in the fuzzy landing map. This can be attributed to the fact that this work relies on dense stereo reconstruction using depth data.

4. Discussion

The performance of the proposed landing algorithm has been assessed for various conditions, including different initial conditions and distinct Gazebo worlds. Despite the absence of all man-made markers and the environment being cluttered with a diverse range of objects exhibiting different dimensions, the designed method demonstrated not only its applicability in detecting the best landing zone if there exists any, but it also performed an IBVS control method based on the image moments enabling autonomous landing. The image feature tracking error remained bound in all experiments and converged to zero. It is worth highlighting that large tracking errors can lead to the UAV landing in hazardous zones or even crashing during operation. Potential future works could be devoted to integrating the semantic maps into the designed algorithm to enhance the perception of the environment. Another direction could involve improving the robustness of the method through multi-sensor fusion frameworks, leading to superior pose estimation. Optimizing the fuzzy system’s rules and membership functions using the ground-truth landing points and iterative offline learning can lead to the enhancement of the system’s accuracy. Lastly, integrating finite-time IBVS methods can lower the convergence time, leading to a more time-efficient landing process. Incorporating this landing approach addresses new challenges, such as emergency landings in GPS-denied, unknown environments. The parallel computation made feasible by the emergence of powerful onboard GPUs may potentially reduce the fuzzy system computations, resulting in a more time-efficient algorithm.

5. Conclusions

This work introduces a novel algorithm for safe landing zone detection based on a fuzzy landing map and ORB-SLAM3. Four distinct images are constructed using RGB-D images and the constructed dense stereo point cloud. Based on fuzzy inference, a landing map is then generated, capable of eliminating the areas hazardous for landing. Next, the point cloud generated by the active sparse ORB-SLAM3 is incorporated into the algorithm to alleviate the limited range of the RGB-D camera by identifying a subset of keypoints that are optimal for landing. Third, four synthetic world points are added to the SLAM world points, and an IBVS technique is then performed to complete the landing using the estimated pose from the SLAM system. Eventually, an IBVS controller is suggested to stabilize the image feature error. Stability analyses showed that the control method is able to converge to the predefined desired image. Exemplary ROS Gazebo simulations are provided to demonstrate the effectiveness of the methods in four different scenarios. The results ensured that the landing algorithm can not only tolerate various initial UAV positions but also the change in the environment. The proposed solution was lastly compared with the state-of-the-art method to verify the superiority and effectiveness.