Visual Navigation Algorithm for Night Landing of Fixed-Wing Unmanned Aerial Vehicle

Abstract

In the recent years, visual navigation has been considered an effective mechanism for achieving an autonomous landing of Unmanned Aerial Vehicles (UAVs). Nevertheless, with the limitations of visual cameras, the effectiveness of visual algorithms is significantly limited by lighting conditions. Therefore, a novel vision-based autonomous landing navigation scheme is proposed for night-time autonomous landing of fixed-wing UAV. Firstly, due to the difficulty of detecting the runway caused by the low-light image, a strategy of visible and infrared image fusion is adopted. The objective functions of the fused and visible image, and the fused and infrared image, are established. Then, the fusion problem is transformed into the optimal situation of the objective function, and the optimal solution is realized by gradient descent schemes to obtain the fused image. Secondly, to improve the performance of detecting the runway from the enhanced image, a runway detection algorithm based on an improved Faster region-based convolutional neural network (Faster R-CNN) is proposed. The runway ground-truth box of the dataset is statistically analyzed, and the size and number of anchors in line with the runway detection background are redesigned based on the analysis results. Finally, a relative attitude and position estimation method for the UAV with respect to the landing runway is proposed. New coordinate reference systems are established, six landing parameters, such as three attitude and three positions, are further calculated by Orthogonal Iteration (OI). Simulation results reveal that the proposed algorithm can achieve 1.85% improvement of AP on runway detection, and the reprojection error of rotation and translation for pose estimation are $0.{675}^{\circ }$ and 0.581%, respectively.

1. Introduction

The broader applications of unmanned aerial vehicles (UAVs) in various industries have increased the requirements for intelligence and autonomy on unmanned aircraft. Autonomous UAV flights are often classified into three stages: autonomous takeoff, cruise, and landing. Among them, landing is considered to be one of the most critical and dangerous stages during the whole operation [1]. Therefore, studying the autonomous landing of UAVs is particularly important to improve their level of autonomy.

Existing navigation systems for aircraft landing include the Global Positioning System (GPS), Instrument Landing System (ILS), and Radio Navigation System (RNS) [2]. However, these systems have several disadvantages. (1) GPS signals are easily disrupted and frequently lost. If a UAV loses the GPS signal during flight, it will be unable to complete the established landing mission. (2) The ILS can only guide the UAV to the decision height. It cannot guide the UAV to land on the runway; hence, the final landing process is manually completed by the pilot. (3) The RNS needs to deal with information interaction between airborne and ground equipment—a process that is easily interfered by other electromagnetic signals. In short, these navigation technologies are unsuitable for autonomous UAV landing.

Because vision provides rich information, machine vision has become a promising technology to guide UAV landings. Visual navigation has the following advantages, such as visual sensors are lighter in weight, low energy consumption, and easier to deploy on UAVs. On the other hand, GPS, ILS, RNS, and other systems require information interaction with other equipment, the working process of visual sensor is completely passive; hence, it avoids signal interference, is less conspicuous, and enhances information security. Visual navigation can directly obtain the attitude and position of the UAV with respect to the runway, which is the essential information for landing control. With the development of hardware, it is becoming faster to process image information with an onboard processor, such as the Parallel processors arrays (PPAS) [3], a new class of vision sensor devices, can enable the UAV to track target, detect and avoid an obstacle. In addition, Benini et al. [4] developed a real-time GPU-based UAV pose estimation system which can process the monocular video stream at least 30 fps. The progress of onboard processors makes it possible for fixed-wing UAVs to utilize visual sensors for navigation. A large number of studies have shown that the accuracy of visual navigation can reach the centimeter level [5]. Moreover, machine vision is a key area of focus to achieve UAS autonomy [6]. Because of these advantages, visual navigation has been studied for several years and successfully applied to rotary and fixed-wing UAVs, such as autonomous landing on mobile platforms [7] and landing fixed-wing UAVs using a vision system [8].

The difference in flight characteristics between the rotary and fixed-wing UAVs has made the landing process of the latter more challenging for the following reasons. (1) Fixed-wing UAVs have higher requirements for robustness and real-time visual navigation algorithms because, unlike rotary UAVs, they cannot hover during the landing process. (2) Because rotary UAVs are equipped with a down-looking visual sensor and signs with obvious characters that are set on the landing platform (e.g., ARTag [9], AprilTag [10], ARUco [11]), these are easier to detect from visual images. In contrast, the runway characteristics of fixed-wing UAVs are not obvious, which makes them difficult to detect. Furthermore, the malfunction runway light, such as one pixel color change, could lead to an erroneous runway detection. The authors of [12] studied the robustness of Bayesian neural networks, proving that Bayesian methods are effective in improving the robustness of image-based deep learning systems, providing a feasible idea for runway recognition in interference scenarios. (3) Most studies on visual landing were conducted under good lighting conditions. However, there are many tasks that have to be performed at night. A low-light image can reduce the performance of high-level vision algorithms; sometimes, the object detection algorithm cannot detect any object at all, as shown in Figure 1. Our observation of visual navigation for fixed-wing UAVs leads us to conclude that (1) although the topic of visual navigation for fixed-wing UAVs has been widely researched, it still cannot cover overall fixed-wing UAVs autonomous landing scenarios, such as landing at night; (2) the properties of fixed-wing UAVs landing runway are difficult to be detected by normal vision detection algorithm. These conclusions motivated us to develop a novel visual navigation approach to achieve fixed-wing UAVs autonomous landing at night.

To solve the three problems described above, the vision-based navigation algorithm of a fixed-wing UAV is divided into three parts. First, the image fusion algorithm based on the objective function is used to fuse the visible and infrared images. Second, a runway detection algorithm based on an improved Faster region-based convolutional neural network (Faster R-CNN) is proposed. Finally, the relationships between the world coordinate system, camera coordinate system, and image coordinate system are established, and the attitude and position of the UAV relative to the landing runway are calculated by an Orthogonal Iteration (OI) algorithm. The general scheme of the vision-based navigation algorithm for fixed-wing UAVs for night landing is shown in Figure 2. The main novelty of this work includes three aspects.

(1)

For fixed-wing UAV autonomous landing at night, an image brightness enhancement method based on visible and infrared images fusion is proposed. While effectively enhancing brightness, more texture information is preserved.

(2)

Since it is difficult for ordinary visual detection algorithms to detect runway targets from the enhanced image, a runway detection algorithm based on an improved Faster R-CNN is proposed. The ResNet101 model that is most suitable for the runway detection scene is selected, and the size of the anchor is tailored specifically according to the statistical analysis results of the runway ground-truth box label.

(3)

As an indirect result, this manuscript presents a complete system for fixed-wing UAVs landing at night by means of visual navigation.

The remainder of this paper is organized as follows. Section 2 describes the visible and infrared image fusion algorithm based on the objective function. Section 3 introduces the runway detection algorithm based on the improved Faster R-CNN. In Section 4, the relative attitude and position between the UAV and landing runway are calculated. Section 5 presents and analyzes the experimental results. Finally, the results of this study are summarized in Section 6.

2. Illuminance Enhancement Based on Visible and Infrared Image Fusion

The first step in vision-based fixed-wing UAV landing at night is image preprocessing, which aims to enhance the illuminance of the airborne image when the UAV lands at night to enable the subsequent runway detection algorithm to detect the runway. This section adopts the strategy of visible and infrared image fusion to realize low-illumination airborne image enhancement.

2.1. Establishment of Objective Function

Airborne image fusion has been utilized to generate an image with enhanced illumination and rich texture information. Therefore, the fused image should retain as much brightness and detailed appearance texture information as possible from the two images. The infrared, visible, and fused images are denoted as I, V, and $F\in R^{m\times n\times 1}$, respectively, $m\times n$ represents the size of image.

The texture information in the visible image is mainly characterized by the gradients. The area with a larger gradient value indicates that it contains richer detailed texture information. The thermal radiation information in an infrared image has pixel intensity characteristics. Runway objects usually have a high pixel intensity, which means that these have a high brightness in the infrared image. To improve the recognition ability of the runway detection algorithm, it is necessary to retain more feature information in the fused image. Therefore, as described in Equation (1), an empirical error formula for ${\ell }^p$ norm measurement is established to constrain the fused image such that the fused image has a pixel gradient similar to the visible image, where ∇ is the gradient.

$${\epsilon }_1\left(F\right)=\frac{1}{p}{∥\nabla F-\nabla V∥}_p^p$$

(1)

To ensure that the runway object in the fused image has a large brightness value, it is necessary to retain more radiation information (pixel intensity) from the infrared image so that the fused image has a brightness distribution similar to that of the infrared image. Therefore, the fused image is constrained by the following equation:

$${\epsilon }_2\left(F\right)=\frac{1}{q}{∥F-I∥}_q^q$$

(2)

Combining Equations (1) and (2), the fusion problem can be transformed into minimizing the following objective function:

$$\epsilon \left(F\right)=\eta {\epsilon }_1\left(F\right)+{\epsilon }_2\left(F\right)=\eta \frac{1}{p}{∥\nabla F-\nabla V∥}_p^p+\frac{1}{q}{∥F-I∥}_q^q$$

(3)

The first term in Equation (3) ensures that the fused image F retains as much gradient information of the visible image F as possible to retain more texture details. The second item ensures that the fused image F has a pixel intensity similar to that of the infrared image I. $\eta$ is a positive parameter that controls the trade-off between the two items.

The gradient/texture in the visible image can be fused with the corresponding position in the infrared image through the objective Function (3). This ensures that the fused image retains the brightness of the infrared image visually as well as more runway appearance details. Therefore, we can obtain a brightness-enhanced fused image with a more complex scene performance and richer detailed features.

2.2. Solving Objective Function

This section solves the objective function (3). To enhance the brightness of the image, the fused image must retain the brightness information of I, i.e., most entries of $F-I$ should be zero. Simultaneously, to preserve the texture information (gradient) of the visible image V, a small number of entries may be significant. Therefore, the difference between F and I should be Laplace or pulsed, where the parameter q is set to 1. To maintain the sparsity of the gradient of the visible image, p = 0 is selected. Since, p = 0 is NP-hard, we set p = 1. Let $Z=F-V$, then the fusion problem is transformed into the solution in Equation (4).

$$\begin{array}{c}\hfill Z^{\ast }=arg\underset{Z}{min}\{\sum _{i=1}^{mn}|Z_i-(I_i-V_i)|+\eta J\left(Z\right)\}\\ \hfill J\left(Z\right)=\sum _{i=1}^{mn}\left|{\nabla }_{i}Z\right|=\sum _{i=1}^{mn}\sqrt{{\left({\nabla }_i^{h}Z\right)}^2+{\left({\nabla }_i^{v}Z\right)}^2}\end{array}$$

(4)

where ${\nabla }_i=({\nabla }_i^h,{\nabla }_i^v)$ represents the image gradients ∇ at pixel i, ${\nabla }_i^{h}F=F_i-F_{r\left(i\right)}$, and ${\nabla }_i^{v}F=F_i^{ﬞ}–F_{b\left(i\right)}$; and $r\left(i\right)$ and $b\left(i\right)$ represent adjacent points below and to the right of pixel i. The objective function (4) is convex; hence, it has the optimal solution. The gradient descent method in [13] is used to solve $Z^{\ast }$, and the optimal fused image is obtained from $F^{\ast }=Z^{\ast }+V$. In this way, the enhanced image can retain more feature information while increasing the illuminance, thus improving the accuracy of runway recognition at night.

The process of the proposed visible and infrared image fusion algorithm is described below in Algorithm 1.

Algorithm 1 Visible and infrared image fusion algorithm.

Input: visible image I, infrared image V, parameter $\eta$ Output: fused image F 1: Calculating the error between F and V and I respectively: ${\epsilon }_1\left(F\right)=\frac{1}{p}{∥\nabla F-\nabla V∥}_p^p$ and ${\epsilon }_2\left(F\right)=\frac{1}{q}{∥F-I∥}_q^q$. 2: Combining the two errors by $\epsilon \left(F\right)=\eta \frac{1}{p}{∥\nabla F-\nabla V∥}_p^p+\frac{1}{q}{∥F-I∥}_q^q$. 3: Transferring the fusion problem to solve $Z^{\ast }=arg\underset{Z}{min}\{{\displaystyle \sum _{i=1}^{mn}}|Z_i-(I_i-V_i)|+\eta J\left(Z\right)\}$. 4: Computing $Z^{\ast }$ using the gradient descent method [13].

3. Runway Detection Based on Improved Faster R-CNN

Traditional target-detection technologies have three steps: region selection, feature extraction, and feature classification [14]. The most commonly used region selection algorithm is the sliding window algorithm. However, its high computational complexity makes it difficult to detect the runway in real time. Therefore, this method is unsuitable for the vision-based autonomous landing of fixed-wing UAVs. In the feature extraction stage, commonly used algorithms include scale-invariant feature transform (SIFT) and histogram of oriented gradients (HOG). These types of extractors have fewer parameters, low robustness of manual design, and low quality of feature extraction. Therefore, these are also unsuitable for detecting runways whose point-line characteristics are not very obvious. The learning-based object detection algorithm uses a neural network for feature extraction. It can extract features with improved robustness and semantics, whereas the classifier has better performance and detection results. Due to such advantages, a learning-based object detection algorithm is used for runway detection in this study.

There are two types of learning-based object detection: single-step detection, such as Yolo [15] and SSD [16]; and two-step detection, such as Fast R-CNN [17], Faster R-CNN [18], and R-FCN [19]. Compared with the single-step detection algorithm, the two-step detection algorithm has a high detection accuracy and favorable outcomes. Since the Faster R-CNN directly uses a region proposal network (RPN) to generate detection boxes, the algorithm speed is significantly improved. Therefore, the runway detection algorithm proposed in this study is designed based on the Faster R-CNN architecture.

The Faster R-CNN has achieved excellent performance in multi-class object detection in many fields, such as multi-class fruit detection [20], multi-object recognition in the complex traffic environment [21], aerial target detection [22], and aircraft dents detection [23]. However, it has not been applied to runway detection in airborne images. This paper proposes an improved Faster R-CNN to detect runways, as shown in Figure 3. The framework consists of two modules: an RPN and a Fast R-CNN detector. Firstly, the RPN processes feature maps to generate multiple region proposals that may contain airports. Secondly, the Fast R-CNN module takes the feature maps and region proposals as inputs, which are then processed by a full connection layer, loss function, and boundary box regression function. Finally, the accurate object category and object box are generated.

3.1. Region Proposal Network

When an image is input into the Faster R-CNN, it first utilizes a set of basic CNNs to extract features from the input image. Then, the CNN generates a feature map for the RPN module and region of interest (ROI) pooling layer, as shown in Figure 3. CNNs frequently used in Faster R-CNN include VGGNet [24], GoogLeNet [25], and ResNet [26]. In the experimental section, VGG19, Inception V4, and ResNet101 were selected as CNN models for feature extraction; their runway detection performance was compared and analyzed.

The RPN is a full CNN that takes the feature map output by the CNN as the input, generates multiple candidate regions with different scales and aspect ratios, and transmits them to the Fast R-CNN module. The RPN structure, shown in Figure 4, adopts the sliding window strategy. A $3\times 3$ window is used to slide on the feature map generated by the last layer, and a short feature vector is generated and input into two full connection layers: the classification layer and boundary box regression layer. The number of candidate boxes generated at each location is k. Each candidate box is encoded with four coordinates; hence, the boundary box regression layer has 4k outputs. The classification layer outputs 2k scores to estimate the probability that the candidate region is or is not the object. These k parameterized candidate boxes are known as anchors.

The default parameter of the anchor in the Faster R-CNN is $[{128}^2,{256}^2,{512}^2]$, and the number of anchors is 9. The efficiency and accuracy of the detector are closely related to the shape, aspect, and number of anchors [27]. Through the statistical analysis of the runway ground-truth box in the airborne image dataset with a resolution of $1224\times 520$, the relationship between the number of anchors and mean Intersection over Union (IoU) is shown in Figure 5. It can be seen that when the number of anchors is 1–4, the mean IoU significantly increased with the increase in the number of anchors. When the number of anchors is 4–15, the rate of increase of the mean IoU slowed down, and decreased when the number of anchors is 8 and 12, respectively. Considering the speed and accuracy of detection, the number of anchors was set as 9 in this study.

Figure 6 shows the statistical analysis results for the width–height ratio and size of the runway ground-truth box in the dataset. It can be seen that the aspect ratios of the ground-truth box in the dataset ranged from 0.5 to 1.5, and 2.3 to 3.5, while the size ranged from 0 to 10,000. To improve the generalization performance and robustness of the Faster R-CNN for runway detection, nine anchors were designed that met the requirements of a runway detection application scenario. Those anchors were designed by using a K-means cluster analysis algorithm which uses the intersection-over-union (IoU) distance metric to cluster together boxes of similar aspect ratios and sizes, which ensure that the estimated anchors fit the ground-truth size of the runway. When the number of anchors has been determined, the algorithm outputs the specified number of anchor sizes. The anchor sizes are [60, 50], [195, 263], [49, 144], [100, 122], [79, 76], [44, 118], [54, 167], [49, 41], and [140, 169].

3.2. Fast R-CNN Detector

As shown in Figure 3, multiple ROIs were input into the Fast R-CNN detector. For each ROI, the ROI pooling layer extracts a fixed-length feature vector from the convolutional layer. Each feature is input into the fully connected layer. The final outputs of the Fast R-CNN detector through the softmax layer and bounding box regression layer are the object category and the bounding box. The object category calculates the probability score of proposal A that belongs to the A category; the bounding box uses bounding box regression to correct the position of each proposal detection box to obtain a more accurate bounding box position. In this study, the object class contains only one object, ‘runway’, A = 1.

The position regression loss and classification loss functions of an object [18] are expressed as

$$L\left(\{p_j,t_j\}\right)=\frac{1}{N_{cls}}\sum _j{L}_{cls}(p_j,p_j^{\ast })+\lambda \frac{1}{N_{reg}}\sum _j{p}_j^{\ast }{L}_{reg}(t_j,t_j^{\ast })$$

(5)

where j is the index of an anchor in a mini batch. $N_{cls}$ and $N_{reg}$ are the normalized parameters and balanced weighted by $\lambda$. The values of $N_{cls}$, $N_{reg}$, and $\lambda$ are 256, 2400, and 10, respectively, which indicate that the weights of cls and reg are approximately equal.

$$L_{cls}(p_j,p_j^{\ast })=-log[p_j{p}_j^{\ast }+(1-p_j)(1-p_j^{\ast })]$$

(6)

$p_j$ is the prediction probability of anchor j as the object. If the anchor is positive, then the ground-truth label $p_j^{\ast }$ = 1; otherwise, it is 0.

$$\begin{array}{c}\hfill L_{cls}(t_j,t_j^{\ast })=\sum _{j\in \{x,y,w,h\}}smoot{h}_{L1}(t_j-t_j^{\ast })\\ \hfill smoot{h}_{L1}\left(x\right)=\left\{\begin{array}{cc}0.5x^2,\hfill & \mathrm{if}\left|x\right|<1\hfill \\ \left|x\right|-0.5,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(7)

$$\left\{\begin{array}{c}{t}_x=\frac{x-x_a}{w_a},t_x=\frac{y-y_a}{h_a}\hfill \\ t_w=log(\frac{w}{w_a}),t_h=log(\frac{h}{h_a})\hfill \\ t_x^{\ast }=\frac{x^{\ast }-x_a}{w_a},t_y^{\ast }=\frac{y^{\ast }-y_a}{h_a}\hfill \\ t_w^{\ast }=log(\frac{w^{\ast }}{w_a}),t_h^{\ast }=log(\frac{h^{\ast }}{h_a})\hfill \end{array}\right.$$

(8)

where x and y are the central coordinates of the bounding box; w and h are the width and height of the bounding box; and x, $x_a$, and $x^{\ast }$ are the prediction box, anchor box, and ground-truth box, respectively. The definitions of y, w, and h are similar.

4. Landing Parameter Calculation

As mentioned in Section 1, a vision-based night-landing system should solve three problems: (1) low-light airborne image enhancement, (2) landing runway detection, and (3) estimation of the relative attitude and position between the UAV and runway. Since the problems associated with the low-light airborne image enhancement and landing runway detection have been addressed in Section 2 and Section 3, the current section focuses on estimating the relative attitude and position between the UAV and runway.

4.1. Construction of Reference Coordinate System

The first step in the estimation of vision-based navigation parameters is to establish the mapping relationship between the position of a point in space and its position in the image. Figure 7 shows the four coordinate systems, elaborated below, that were constructed.

(1)

Earth-fixed coordinate system $(O_w-X_w{Y}_w{Z}_w)$: the origin of Earth-fixed coordinate system can be fixed anywhere. Since the study only focuses on the attitude and position of the UAV relative to the runway, the Earth-fixed coordinate system was set at the runway plane.

(2)

Camera reference coordinate system $(O_b-X_b{Y}_b{Z}_b)$: the camera was fixed at the center of the UAV body. Thus, the body coordinate system and camera reference coordinate system are the same.

(3)

Image coordinate system $(O_I-X_I{Y}_I)$: the origin $O_I$ of Image coordinate system is the intersection of the camera optical axis $O_b{X}_b$ and the image plane, usually the image center point.

(4)

Pixel coordinate system $(O-uv)$: the Pixel coordinate system is established on the pixel plane, and the unit is pixel. The digital image is stored in the form of $M\times N$, so the pixel coordinate $(u,v)$ represents the position of the pixel in the array $M\times N$.

According to the pinhole imaging model, the homogeneous relationship between position $(X_b,Y_b,Z_b)$ of a space point P in the camera reference coordinate system and the corresponding point $p(X_I,Y_I)$ in the image coordinate system is

$$\alpha \left[\begin{array}{c}{X}_I\\ Y_I\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\\ 1\end{array}\right]$$

(9)

where $\alpha$ is the scale factor and f is the focal length of the camera.

The homogeneous relationship between the point $p(u,v)$ in the pixel coordinate system and the point $p(X_I,Y_I)$ in the image coordinate system is

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ 0& \frac{1}{d_y}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_I\\ Y_I\\ 1\end{array}\right]$$

(10)

The conversion relationship between the pixel coordinate system and camera reference coordinate system can be obtained from Equations (9) and (10) as

$$\alpha \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{f}{d_x}& 0& u_0\\ 0& \frac{f}{d_y}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\\ 1\end{array}\right]=K\left[\begin{array}{cc}I& 0\end{array}\right]\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\\ 1\end{array}\right]$$

(11)

where K is the calibration matrix of the camera, describing the intrinsic parameters of the camera, which is a necessary parameter for relative attitude and position estimation.

The coordinate correspondence of point P in the camera reference coordinate system $(X_b,Y_b,Z_b)$ and the Earth-fixed coordinate system $(X_w,Y_w,Z_w)$ can be expressed by the rotation matrix R and translation vector T.

$$\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\\ 1\end{array}\right]=\left[\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(12)

Substituting Equation (12) into Equation (11) yields a homogeneous mapping relationship between the point p in the pixel coordinate system and the point P in the Earth-fixed coordinate system.

$$\alpha \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=K\left[\begin{array}{cc}I& 0\end{array}\right]\left[\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=K\left[\begin{array}{cc}R& T\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$$

(13)

From Equation (13), the relationship between the pixel coordinate system and the Earth-fixed coordinate system can be determined by K, R, and T. Therefore, the core problem of determining the relative attitude and position between the UAV and landing runway is solved by calculating the R and T.

4.2. Calculation of Relative Attitude and Position

The main problem discussed in this section is the calculation of the R and T. In general, calculation methods for the R and T can be classified into two types: linear and nonlinear. The linear method has poor robustness and accuracy, making it is unsuitable for the autonomous landing of fixed-wing UAVs. Nonlinear methods, such as the Gauss–Newton and Levenberg–Marquardt methods, can overcome such problem. However, the hundreds of iterations required by these methods result in low efficiency and poor real-time performance. To optimize the number of iterations and improve the convergence speed, Lu et al. [28] proposed an OI algorithm that transformed the attitude estimation problem into an object-space collinear error minimization problem that could globally converge in five to ten iterations. This paper proposes a relative attitude and position calculation method based on OI.

We designed three feature-point layout schemes on the runway, as shown in Figure 8a–c. These feature points can be detected by image segmentation technology.

In the above three schemes, the coordinates of the feature points in the Earth-fixed coordinate system and camera reference coordinate system are $P_r^w=\left[\begin{array}{ccc}{X}_r^w& Y_r^w& Z_r^w\end{array}\right]$ and $Q_r^b=\left[\begin{array}{ccc}{X}_r^b& Y_r^b& Z_r^b\end{array}\right]$, respectively, where $r=1,2,,,s;s\ge 5$. According to the relationship between these two coordinate systems, the relationship between $P_r^w$ and $Q_r^b$ is

$$Q_r^b=R{P}_r^w+T$$

(14)

The estimation of the rotation matrix R and translation vector T can be converted into a least-squares problem.

$$\underset{R,T}{min}\sum _{r=1}^n{\parallel R{P}_r^w+T-Q_r^b\parallel }^{2}s.t.R{R}^T=I,det\left(R\right)=I$$

(15)

Through the OI algorithm, the solution of Equation (15) is transformed into an iterative process to calculate R and T. Then, the relative attitude and position between the UAV and runway can be obtained.

5. Simulation Results and Discussion

This section discusses the capabilities of the proposed algorithm, including visible and infrared image fusion, landing runway detection, and UAV landing parameter calculation.

5.1. Simulation Dataset Setup

In this study, three real-world airports in different environments were set in the flight simulation software FlightGear 2020.3 [29]: Data 1—Livermore Municipal Airport (Livermore, California, USA) located in an urban environment; Data 2—George Bush Intercontinental Airport (Houston, Texas, USA) located in an open environment; and Data 3—San Francisco International Airport (San Francisco, CA, USA) located along the coast. A fixed-wing UAV, i.e., MQ-9, which contains a visible/infrared dual light pod module was selected to land at the three airports. The visible and infrared images are collected during MQ-9 autonomous landing process at night. Figure 9 shows the MQ-9. The MQ-9 properties are presented in

Table 1. The MQ-9 properties.

UAV	Length	Wingspan	Height	Max Takeoff Weight	Cruising Speed	Range	Endurance
MQ-9	11 m	20 m	3.81 m	4760 kg	313 km/h	1852 km	14 h

. A total of 1500 visible images and 1500 infrared images were collected. Images of the three airports are shown in Figure 10. The characteristics of the three airports are listed in

Table 2. Characteristics of three different airports.

Airport	ICAO	Latitude	Longitude	Elevation	Runway Length
Livermore Municipal Airport	KLVK	37.693	−121.821	400 ft	5248 ft
George Bush Intercontinental Airport	IAH	29.983	−95.342	97 ft	11,995 ft
San Francisco International Airport	KSFO	37.619	−122.375	13 ft	11,867 ft

.

The simulation was implemented using the Flightgear and MATLAB 2020b on a computer equipped with an Intel Core i7–9700KF CPU and NVIDIA GeForce RTX2070Super GPU.

5.2. Fusion Experiment when $\eta$ Changes

In the proposed fusion algorithm, the weight parameter $\eta$ controls the information preservation trade-off between the visible and infrared images. By setting different $\eta$ values, the brightness and texture information of the fused image can be adjusted. Therefore, the weight parameter $\eta$ affects the fusion results. This section examines the influence of different $\eta$ values on the fused image. To quantitatively measure the quality of the fused image, three commonly used image fusion metrics were utilized for the quantitative evaluation: entropy (EN) [30], mutual information (MI) [31], and gradient-based fusion measurement ($Q^G$) [32]. These three indicators are defined as follows:

• EN evaluates the amount of image information after fusion.
• MI evaluates the image fusion performance by measuring the amount of information a given variable contains about another variable. In other words, it measures the dependence between the infrared and visible images.
• $Q^G$ evaluates the amount of detail texture information in the fused image that was transferred from the infrared and visible images.

The larger the values of these three parameters, the higher the quality of the fused image.

Figure 11 shows the fusion results of Data 1 at different $\eta$ values, where (a) is the visible image and (b) is the infrared image. (c–m) are the results when $\eta =0.1,1,2,3,4,5,6,7,8,9,10$, respectively. Figure 12 and Figure 13 show the fusion results of Data 2 and 3 at different $\eta$ values. The quantitative evaluation results of the fused images at different $\eta$ values of Data 1, Data 2, and Data 3 using EN, MI, and $Q^G$ are shown in Figure 14, Figure 15 and Figure 16, respectively.

It can be seen from Figure 11, Figure 12 and Figure 13 that the smaller the value of $\eta$, the greater the amount of brightness information from the infrared image saved in the fused image. The larger the value of $\eta$, the greater the amount of runway texture information is saved in the fused image. When $\eta$ gradually increased from 0.1, the fused image has a high brightness, the texture information of the runway in the image gradually increased, and the visual effect of the fused image continuously improved. When $\eta$ increased to 4, the improvement of airport texture details in the fused image gradually slowed down, and the improvement of visual effect of the fused image was not very significant. In terms of quantitative evaluation metrics, it can be seen from Figure 14, Figure 15 and Figure 16, increasing $\eta$ gradually raises the EN and $Q^G$, while the MI indicator decreases. Considering these three objective evaluation metrics comprehensively, to obtain high EN and $Q^G$ values and good MI results, the $\eta$ was kept constant at 5 in this study to achieve good visual effects in most cases.

5.3. Runway Detection

The runway detection dataset in this section was obtained from the fused image in the previous section.

5.3.1. Influence of Different Feature Extraction Networks on Runway Detection

In the Faster R-CNN, different feature extraction networks have different effects on the target detection performance [33]. VGGNet, ResNet, and GoogLeNet have achieved good results in remote sensing image classification and airport detection [34,35,36,37,38,39,40,41]. Therefore, in this section, VGG19, ResNet101, and Inception V4 models are used as the airport feature extraction networks. The default anchor size is used, and AP (Average Precision is a metric which indicates the performance of the detector on two aspects of Precision and Recall) is used as the evaluation metric to analyze the impact of different feature extraction networks on the runway detection. The comparison results of the three models are summarized in

Table 3. Comparison of different feature extraction models.

Test Data	Feature Extraction Model	AP Value
Airport	VGG19	79.23%
Airport	ResNet101	82.52%
Airport	Inception V4	80.95%

, where ResNet101 has the highest AP value, meaning that it has the best runway detection performance.

5.3.2. Influence of Different Anchor Size on Runway Detection

The results from the previous section showed that the ResNet101 network has the best runway detection performance. To compare and analyze the effects of the anchor size before and after modification on the detection performance of the Faster R-CNN, this study used the ResNet101 to detect the runway in the fused image with a resolution of $1224\times 520$. Figure 17 and Figure 18 show the detection results before and after the anchor size modification. It can be seen that the Faster R-CNN with the modified anchor size can locate the runway more accurately than the Faster R-CNN with the default anchor size.

Table 4. Quantitative comparison of different anchor sizes.

Anchor	Feature Extraction Model	AP Value
Default	ResNet101	82.52%
Improved	ResNet101	84.37%

presents the quantitative comparison results of the AP values before and after the anchor size modification. It can be seen that the AP value of the Faster R-CNN improved after the anchor size was modified.

5.4. Relative Attitude and Position Calculation

This section conducts the calculation experiment on the UAV landing attitude and position parameters. The influence of the number of feature points and feature point extraction error on the accuracy of the calculated attitude and position is analyzed. Three layout schemes with different numbers of feature points are shown in Figure 8. By using the camera calibration toolbox in MATLAB, we can obtain the camera intrinsic matrix

$$K=\left[\begin{array}{ccc}571& 0& 254\\ 0& 571& 206\\ 0& 0& 1\end{array}\right]$$

(16)

5.4.1. Error of Different Numbers of Feature Points

The error results of the UAV landing parameter estimation are shown in Figure 19, Figure 20 and Figure 21 when five, seven, and nine feature points were set on the airport, respectively. The results have shown that the attitude and position errors decreased with the increase of the number of feature points. For all three schemes, the attitude estimation error is larger than the position estimation error.

5.4.2. Feature Point Extraction Error

Different standard deviations (SD) of white noise were added to the three feature-point-setting schemes to analyze the influence of the feature point position extraction error on the UAV attitude and position calculation. Noise values of 0.1, 0.5 and 1.0 were added to the five, seven, and nine feature-point-setting schemes, respectively. The error results are shown in Figure 22, Figure 23 and Figure 24. The correlation results show that the estimation error increased with the increase in the feature point extraction error. In the five-feature-point scheme, the calculated landing parameters had the largest error when SD = 1, making it unsafe for UAV landing.

Therefore, in subsequent engineering applications, we can consider increasing the number of feature points and improving the accuracy of feature point position extraction to improve the calculation accuracy of the attitude and position.

5.4.3. Computational Cost as Feature Points Increase

To evaluate the relationship between the number of feature points and computational complexity and the pose estimation accuracy, we calculate the pose estimation accuracy and the computation time as the number of feature points increases. The related results are shown in

Table 5. The influence on pose estimation accuracy and computation time when feature-point number increases.

Feature-Point Number	5	6	7	8	9	10	11
Mean rotational reprojection error/${}^{\circ }$	15.731	6.879	5.106	3.354	0.675	0.702	0.693
Mean translational reprojection error/%	13.905	11.127	4.914	2.632	0.581	0.587	0.546
Mean computational time $ms$	1.597	3.964	4.156	6.097	9.572	10.484	15.803

.

The results shown in Table 5 illustrate two things. (1) When the number of feature points increases initially, the computational accuracy improves rapidly, but then tends to be flat; (2) the computational complexity always increases steadily with the increase of the number of feature points. Considering the balance of computational complexity and accuracy, the number of 9 feature points was selected.

5.5. Comparison with Other Methods

To demonstrate the superiority of the proposed algorithm, several state-of-the-art algorithms for image fusion, runway detection, and pose estimation are compared in this section.

5.5.1. Fusion Comparison

The proposed image enhancement method was compared with existing image fusion methods, including CBF [36], HMSDGF [37], IFEVIP [38], MSTNO [39], and MGFF [40]. The EN, MI, and $Q^G$ were used to quantitatively compare the fusion results. For the purpose of fairness, open source code, i.e., publicly available, was utilized and we did not make any amendment to the parameters. Two pairs of visible and infrared images, Image1 and Image2, were randomly selected in the image dataset, as shown in Figure 25a,b and Figure 26a,b. Figure 25 and Figure 26 show the intuitive results of the compared methods, and the quantitative comparison of these results is summarized in

Table 6. Quantitative comparison between different fusion methods.

Method	EN	MI	${\mathit{Q}}^{\mathit{G}}$
CBF	4.519	2.166	0.415
HMSDGF	4.229	1.857	0.437
IFEVIP	3.908	2.576	0.280
MSTNO	4.721	2.889	0.355
MGFF	4.213	2.628	0.408
Our method	4.823	3.033	0.467

.

The image enhancement method proposed in this study achieved higher EN, MI, and $Q^G$ compared with the other similar techniques, indicating that the method can achieve better fused image quality. As shown in Figure 25 and Figure 26, the CBF images have an unrealistic visual effect. The HMSDGF images have large blocks of unexpected color errors. The IFEVIP and MSTNO images have a white area at the boundary between the sky and earth. The MGFF images are not sufficiently bright. In conclusion, the results of the proposed model have better contrast, clearer details, and improved noise suppression.

5.5.2. Runway Detection Comparison

In this section, two algorithms (MSF [41] and DCNN [42]) that performed well in airport detection in remote sensing images are compared with the algorithm proposed in this study. Figure 27 presents a visual comparison of the detection results of the three algorithms: (a–c) improved Faster R-CNN, (d–f) DCNN, and (g–i) MSF. It can be seen from Figure 27e that the DCNN algorithm could mistakenly identify roads as runways in some scenarios, which may be attributed to the generation of candidate regions based on prior knowledge of the airport segment distribution.

Table 7. Quantitative comparison between different algorithms.

Method	AP Value
Improved Faster R-CNN	84.37%
DCNN	80.81%
DSF	82.19%

presents a quantitative comparison of the three algorithms. It can be seen that the proposed algorithm has the highest AP value. Figure 27 shows that the proposed algorithm accurately detected the runway because the anchor size was modified according to the runway detection scenario. In short, the proposed algorithm is one of the best-performing algorithms till date.

5.5.3. Pose Calculation Comparison

To verify the superiority of the proposed landing parameter estimation algorithm, several state-of-the-art algorithms for R and T calculation were selected for comparison, including DLT [43], DLS [44], UPnP [45] and OPnP [46], with the reprojection error as the index. The rotational and translational reprojection errors were calculated, and the results are listed in

Table 8. Pose calculation comparison.

Method	Rotational Reprojection Error/${}^{\circ }$	Translational Reprojection Error/%
DLT	2.594	2.386
DLS	1.164	0.953
UPnP	0.776	0.627
OPnP	0.693	0.603
Our method	0.675	0.581

. It can be seen that the proposed algorithm has the lowest R and T reprojection errors, which means that it has the highest solution accuracy.

6. Conclusions

This paper proposed a visual navigation scheme for fixed-wing UAV autonomous landing at night. It has considered three key problems: enhancing the illumination of UAV night-airborne images, detecting the landing runway from an enhanced image, and calculating the relative attitude and position between the UAV and runway. To solve the first problem, the objective functions of the fused and visible image, and fused and infrared image were established. Then, the fusion problem was transformed into an optimal problem. To solve the second problem, we compared and analyzed the performance of three different CNNs. After that, in the candidate region extraction network part, we statistically analyzed the size of the runway ground-truth box label and improved the number and size of the anchors. To solve the third problem, we designed three feature-point layout schemes on the runway surface and adopted the OI algorithm to calculate the relative attitude and position parameters between the UAV and landing runway. The main contributions of this study are as follows:

• For the problem of the low brightness of airborne images taken at night, which make it difficult to extract runway information and identify runway objects, an image fusion method based on gradient descent to obtain the enhanced images was proposed. This method ensures the retention of more texture details and other feature information while enhancing the brightness.
• For the problem in which the original Faster R-CNN could not be effectively applied to the runway detection scene, the anchors were redesigned and improved. Additionally, the ResNet101 network was selected, which is the most suitable for runway feature extraction. The simulation results show that, for runway detection, we can achieve 1.85% improvement of AP by the improved Faster R-CNN.
• For the low efficiency and poor real-time performance of nonlinear algorithms in calculating the R and T, a relative attitude and position estimation method based on the OI algorithm was proposed. The simulation results show that the reprojection error of rotation and translation for pose estimation can be as little as $0.{675}^{\circ }$ and 0.581%, respectively.

The experimental results demonstrated the effectiveness and superiority of the newly developed algorithm.