Target Positioning for Complex Scenes in Remote Sensing Frame Using Depth Estimation Based on Optical Flow Information

Abstract

UAV-based target positioning methods are in great demand in fields, such as national defense and urban management. In previous studies, the localization accuracy of UAVs in complex scenes was difficult to be guaranteed. Target positioning methods need to improve the accuracy with guaranteed computational speed. The purpose of this study is to improve the accuracy of target localization while using only UAV information. With the introduction of depth estimation methods that perform well, the localization errors caused by complex terrain can be effectively reduced. In this study, a new target position system is developed. The system has these features: real-time target detection and monocular depth estimation based on video streams. The performance of the system is tested through several target localization experiments in complex scenes, and the results proved that the system can accomplish the expected goals with guaranteed localization accuracy and computational speed.

1. Introduction

In recent years, with the development of UAV technology, the research direction of remote sensing images has gradually increased, such as remote sensing image registration [1,2,3,4,5,6], image fusion [7,8], etc. UAVs are increasingly used in complex scenes or special perspectives, such as environment monitoring [9,10], search and rescue [11,12], surveying and mapping [13,14,15], power inspection [16], and intelligent agriculture [17,18]. The target positioning method has high practical value in UAVs for Earth observation missions, such as national defense, emergency management, and urban management.

Unlike vehicle-mounted lenses, UAVs have a higher degree of freedom in spatial location, which makes it difficult to use a stable scale standard for UAV remote sensing images. This problem leads to the fact that when target localization methods are applied in UAV remote sensing images, more information needs to be obtained to determine the scale information of the remote sensing images. Common localization methods include laser ranging, point cloud modeling, and binocular localization. Both laser ranging and point cloud modeling require specialized sensors on the UAV. This brings more challenges to the range of UAVs and limits the application scenarios of UAVs to some extent. Using fewer sensors to obtain more and more accurate remote sensing information as much as possible is the trend of UAV civilization development.

Currently, binocular localization methods are mostly used in the fields of target tracking [19], Simultaneous Localization and Mapping (SLAM) [20], and autonomous driving [21]. The principle of binocular positioning is to calculate the relative depth using parallax information and the absolute depth information from the baseline to achieve the effect of localization. Ma et al. [22] uses the UAV binocular positioning method to locate insulators.

The monocular localization method mostly relies on spatial triangulation. Sun et al. [23] uses the flight height of UAV on the internal reference of camera to achieve the calculation of target localization. Madhuanand et al. [24] proposes the depth estimation of tilted remote sensing images from UAV.

The binocular positioning method increases the hardware cost and the amount of remote sensing data due to the addition of a video acquisition unit, which shortens the UAV endurance. In addition, binocular localization relies on parallax information, which leads to a baseline length that limits the maximum depth range that can be trusted. The baseline length of binocular cameras can be limited by the size of the UAV. On the other hand, the size of the UAV limits the maximum depth range of binocular localization methods, which imposes significant limitations on the use scenarios for UAV localization.

Monocular vision target positioning method relies mostly on the establishment of spatial triangles. Currently, in addition to constructing spatial triangles by assuming the ground level, depth estimation is mostly used to determine the depth of the target for target localization calculation. Currently, monocular depth estimation methods allow prediction of relative depth. These methods are mostly used in fields, such as the autonomous driving of cars. Since the in-vehicle camera height is stable to the ground, a more accurate scale factor can be obtained by predicting the camera height, which is used to obtain the mapping relationship from relative depth to absolute depth. However, this method has difficulty producing good results for obtaining the scale factor of UAV remote sensing images. This is due to the difficulty of determining a stable reference plane as the ground in remote sensing images, especially in complex scenes with undulating heights, multiple planes or no planes. This makes it a challenge to obtain the absolute depth of UAV remote sensing images.

A new solution is proposed to address these problems. This solution uses the motion of the UAV as a scaling criterion and combines optical flow estimation with the UAV position information. The optical flow estimation model predicts the motion relationship of each pixel point, and then solves the depth information to achieve absolute depth estimation of monocular remote sensing images. In order to solve the problem of UAV target localization, we also build a UAV target positioning system, which takes the monocular UAV as the sensor, and the ground equipment takes up all the computational work, with open access to the target detection module and the absolute depth estimation module. We also constructed two datasets for remote sensing images, which are used for training the target detection model and optical flow estimation model, respectively.

The main contributions of this work are as follow:

• We propose a solution for estimating the absolute depth of monocular remote sensing images. It combines the optical flow estimation model with the UAV motion information, and solves the problem of not being able to obtain accurate absolute depth information in complex scenes, such as no-plane and multi-plane.
• A UAV targeting system is proposed. This system deploys the components in a distributed manner, with the monocular UAV acting as a sensor. The device on the ground combines a target detection module with an absolute depth estimation module to perform real-time operations on the received remote sensing image sequences.
• We constructed two datasets for training the target detection network and the optical flow estimation network, respectively.

This paper is structured as follows. The Section 2 is an overview of the related work in our research process. The Section 3 describes our proposed methodology in detail. The description of data used in the experiments and experimental results are described in Section 4, and finally, the Conclusion.

2. Related Work

We review several currently used methods for target position, as well as a selection of well-performing depth estimation models using self-supervised or ground truth readily available supervised training, which includes monocular and stereo-based training.

2.1. Target Positioning Methods on UAV

Target positioning methods on UAVs are mainly divided into laser ranging [25,26], point cloud modeling [27], and visual position [28]. Laser ranging and point cloud modeling both rely on specialized sensors to directly acquire the relative position of the target and the UAV.

Visual positioning methods can be further divided into stereo and monocular vision Stereo vision generally refers to synchronized stereo image pairs, which are acquired by binocular cameras, and the depth information is predicted by calculating the parallax relationship between the binocular images to achieve target position. Since the baseline of binocular cameras directly limits the calculation of the parallax relationship, binocular cameras with shorter baselines are generally only used in indoor environments to ensure the accuracy of the calculation.

Monocular cameras lack the baseline as a constraint on the scale information compared to binocular cameras, so some kind of more stable parameter is usually used to constrain the scale information. For example, a camera on the ground will use the camera height as a constraint to convert the relative depth information predicted by the monocular depth estimation network into absolute depth information. UAVs cannot find the correct datum to complete the constraint in complex environments, such as multiplanes, mountains, and cliffs during flight.

In this work, we propose a new benchmark for real-time depth estimation during UAV flight based on the motion information of UAVs.

2.2. Monocular Depth Estimation with Self-Supervised Training

Unsupervised learning-based monocular depth estimation methods have become a hot topic in monocular depth estimation research because they do not rely on the depth truth during network training [29,30,31].

In the absence of truth depth information, the depth estimation model can be trained using image reconstruction as a supervised signal based on the geometric relationship between image pairs. During the training process, the input images can be stereo image pairs acquired by a multi-ocular camera or image sequences acquired by a monocular camera. The reprojection of images are calculated based on the predicted depth, and then the training of the model is completed by minimizing the reprojection error.

2.2.1. Stereo Training

The ability to use stereo image pairs for supervised training of monocular depth estimation networks is due to the ability to obtain parallax information of stereo images by predicting pixel differences between image pairs, thus obtaining depth values that can be used as supervised information. Stereo-based approaches have now been extended for semi-supervised data, generative adversarial networks, additional consistency, temporal information, etc.

The production of datasets requires binocular cameras with fixed relative positions, mostly mounted on ground vehicles, such as cars. Such remote sensing datasets are difficult to produce and few public datasets are available.

The baseline length of the binocular camera is the main factor limiting the maximum depth information by acquiring surveillance information through stereo image pairs. When performing a mission, the UAV flies at an altitude of about 40 m. When the baseline length is too short, it is difficult to predict the pixel differences between stereo image pairs, and thus no effective supervision information can be obtained. Additionally, too long baselines make the flight cost and flight safety of UAVs increase dramatically. Therefore, using stereo images for training supervision of the monocular depth estimation network is a less feasible option in the task of this scenario.

2.2.2. Monocular Training

In the absence of sufficient constraints, the more common form of self-supervised training today uses video streams, or image sequences, captured by monocular cameras. Along with the depth prediction, the camera’s pose must be estimated. The pose estimation model is only used in training to constrain the depth estimation network by participating in reprojection calculation.

In 2019, proposed methods such as minimum reprojection loss and full-resolution multi-scale sampling to significantly improve the quality of depth estimation through self-supervised monocular training. On this basis, in 2021, ref. [32] proposed the ManyDepth, an adaptive approach to dense depth estimation that can make use of sequence information at test time, when it is available.

In 2021, Madhuanand et al. [24] first proposed a self-supervised monocular depth estimation model for oblique UAV videos. In that study, they used two consecutive time frames to generate feature maps as a way to generate the inverse depth, and added a contrast loss term in the training phase, which is the image produced by the model closer to the original video image.

3. Materials and Methods

3.1. Positioning System

The complete system is deployed in a distributed manner on three types of devices, 4G/5G devices for controlling the UAV and sending remote sensing images with UAV location information, computing devices, usually computers, for mission planning and monitoring to target location computational tasks, and cloud servers for message forwarding between the first two types of endpoints. We recommend using a multi-rotor UAV as a sensor for the system. A multi-rotor UAV can take off and land vertically in complex scenarios without a runway, and it can fly at a controlled speed. This is ideal for flight operations in complex scenarios. The computing device contains a target detection module and an absolute depth estimation module. The sensors, the target detection module and the depth estimation module are all connected to the system through an open interface, and any sensor or model that satisfies the interface is able to replace the module units in the system.

The YOLOv5 model is divided into several versions according to the complexity of the network structure, including YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The complexity of network structures of these five versions increases and the operation speed decreases in order. When connecting YOLOv5 to the system, a combination of computing speed and target detection recall and accuracy is required. The target detection module takes key frames divided into equal time intervals as input, one frame at a time, detects the target of the current frame and outputs the pixel coordinates.

The absolute depth estimation module is used to calculate the absolute depth of the current frame. Unlike the target detection module, monocular depth estimation method combining optical flow estimation with UAV motion information requires, in addition to the current frame, the key frame of the previous frame and the UAV displacement information corresponding to both frames of the current frame together as the input of the module. The target detection module and the absolute depth estimation module are executed in parallel, and when the two modules complete the calculation of the same frame, the pixel coordinates, the absolute depth information of the current frame and the corresponding UAV position of the current frame will be used as a set of inputs for target localization calculation, and the GPS coordinates of the target are subsequently output. The flow of the calculation is shown in Figure 1.

Target positioning accuracy is influenced by various aspects such as UAV position accuracy, flight altitude, and speed. Since the remote sensing image and UAV flight information are transmitted separately, we mark the two kinds of information separately. The alignment of the two types of information is achieved in milliseconds by tagging and linear calculation. This reduces the impact of the UAV flight speed on the positioning error. Additionally, in the mission, the flight speed of the UAV should be proportional to the height relative to the scanned area. This is to ensure that the IOU of the area corresponding to the front and back frames at the same time interval remains within a more stable range. In order to take into account the flight safety and the clarity of the image, we generally position the flight height around 40 m and the flight speed is 8 m/s.

3.2. Depth Estimate

The main idea is to establish the function relationship between optical flow information and depth information by converting optical flow information into parallax information.

Considering that the rotation of the camera causes a significant change in the optical flow information, the optical flow information is corrected using the rotation information of the UAV before the calculation. Then by transforming the optical flow information through the camera coordinate system, the scaling factor is obtained as the relative depth of the camera displacement length. In the following, we will explain the method in several key steps.

3.2.1. Optical Flow Correct

The pixel coordinate transformation caused by camera rotation is independent of the depth information. The optical flow noise caused by rotation can be obtained by back-projecting the pixel coordinates to the camera coordinate system, then reprojecting them to the pixel coordinate system after the coordinate rotation transformation. By subtracting the optical flow noise from the result of the optical flow estimation model, we can obtain the optical flow information in the same directional view.

The inverse projection calculation requires the parameters of the camera. After calibrating the camera, we obtain the internal reference matrix, denoted as $\mathbf{K}$. This matrix represents the projection of the camera coordinate system with respect to the pixel coordinate system.

$$\begin{array}{c}{[u,v,1]}^{\top }=\mathbf{K}\times {[\frac{x}{z},\frac{y}{z},1]}^{\top }\end{array}$$

(1)

where $(u,v)$ denotes the pixel coordinates and $(x,y,z)$ is the spatial position in the current camera coordinate system corresponding to the pixel coordinates.

The inverse matrix of $\mathbf{K}$ is denoted as $\mathbf{inv}\_\mathbf{K}$. The formula for the inverse projection is expressed as:

$$\begin{array}{c}{[x_0,y_0,1]}^{\top }=\mathbf{inv}\_\mathbf{K}\times {[u,v,1]}^{\top }\end{array}$$

(2)

$(x_0,y_0,1)$ denotes the corresponding point of the pixel point in the plane of $z=1$ m in the camera coordinate system.

The calculation also involves the rotational change of the spatial coordinate system. If the angles of rotation around the three axes are set to ${\theta }_x$, ${\theta }_y$, and ${\theta }_z$, then the rotation matrices around each of the three axes are

$$\left\{\begin{array}{cc}\hfill R_x\left({\theta }_x\right)& =\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\left({\theta }_x\right)& -sin\left({\theta }_x\right)\\ 0& sin\left({\theta }_x\right)& cos\left({\theta }_x\right)\end{array}\right]\hfill \\ \hfill R_y\left({\theta }_y\right)& =\left[\begin{array}{ccc}cos\left({\theta }_y\right)& 0& sin\left({\theta }_y\right)\\ 0& 1& 0\\ -sin\left({\theta }_y\right)& 0& cos\left({\theta }_y\right)\end{array}\right]\hfill \\ \hfill R_z\left({\theta }_z\right)& =\left[\begin{array}{ccc}cos\left({\theta }_z\right)& -sin\left({\theta }_z\right)& 0\\ sin\left({\theta }_z\right)& cos\left({\theta }_z\right)& 0\\ 0& 0& 1\end{array}\right]\hfill \end{array}\right.$$

(3)

The rotation can be decomposed into three steps. (1) The camera coordinate system rotates around the x-axis $p_1$, so that the z-axis is horizontal. (2) Rotates around the y-axis $y_1-y_2$, so that the projection of both z-axis on the horizontal plane is in the same direction. (3) Rotates around the x-axis again $-p_2$, so that the two coordinate systems are in the same direction of the three axes. $p_1$ and $p_2$ are the corresponding pitch angle of the two frames, while $y_1$, $y_2$ represent the yaw angle of the camera. The rotation matrix $\mathbf{R}$ is expressed by the equation as:

$$\begin{array}{c}\mathbf{R}=R_x(-p_2)\times R_y(y_1-y_2)\times R_x\left(p_1\right)\end{array}$$

(4)

Combining the above formulas, the angle correction is calculated as follows:

$$\left\{\begin{array}{cc}\hfill {}^{\top }& =\mathbf{K}\times \mathbf{R}\times \mathbf{inv}\_\mathbf{K}\times {[u,v,1]}^{\top }\hfill \\ \hfill flo{w}_c& ={[u,v]}^{\top }+flow-{[u^{\prime },v^{\prime }]}^{\top }\hfill \end{array}\right.$$

(5)

$flow$ represents the optical flow information estimated by the model for the two frames, and $flow\_c$ is what we need, after rotation correction.

3.2.2. Depth Computing

The main idea is to combine the optical flow, $flo{w}_c$, with the displacement of the camera to construct similar triangles. The depth information of the current frame is derived by equiproportional calculation. We choose a plane in the 3D coordinate system to illustrate the construction of the triangle, as shown in Figure 2.

To facilitate the calculation, we use inverse projection to transform the pixel coordinates so that all coordinate calculations can be placed under the same camera coordinate system. The point P is the real position of any point in the current frame to one, and $P^{\prime }$ is the position of point P in the previous frame relative to the camera. After point P make a parallel line parallel to $z=1$ and intersect the line where $O{P}^{\prime }$ is at the point $P^{″}$. Thus, we obtain a set of similar triangles, $▵OAB$ with $▵OP{P}^{″}$. The absolute depth of point P, denoted as $abs\_d$, is

$$\begin{array}{c}abs\_d=\frac{|{\mathit{PP}}^{″}|}{\left|\mathit{AB}\right|}\times 1\end{array}$$

(6)

The length of $\mathit{AB}$ can be found by $flow\_c$ through the inverse projection. $\mathit{PD}$ is perpendicular to ${\mathit{PP}}^{″}$ with the vertical point D. ${\mathit{PP}}^{″}$ is divided into two parts. $\mathit{PD}$ is the projection of camera displacement in the direction of ${\mathit{PP}}^{″}$ and ${\mathit{DP}}^{″}$ is the correction to the previous value. Figure 2 shows two different positions of the points in relation to the camera, corresponding to the cases where the correction value is greater than zero and less than zero, respectively. The correction value is influenced by $\mathit{OB}$ and $\mathit{AB}$, and is opposite in sign to the cosine of the angle between these two vectors.

$$\left\{\begin{array}{cc}\hfill \left|\mathit{AB}\right|& =\mathbf{inv}\_\mathbf{K}\times {[flow\_u,flow\_v,0]}^{\top }\hfill \\ \hfill |{\mathit{PP}}^{″}|& =\left|\mathit{PD}\right|+sign\times |{\mathit{DP}}^{″}|\hfill \\ \hfill sign& =\left\{\begin{array}{c}\hfill -1,cos(\mathit{AB},\mathit{OB})>0\\ \hfill 0,cos(\mathit{AB},\mathit{OB})=0\\ \hfill 1,cos(\mathit{AB},\mathit{OB})<0\end{array}\right.\hfill \end{array}\right.$$

(7)

The length of $\mathit{DP}$ can be found by the Rt$▵DP{P}^{\prime }$. Set the coordinates of point C as $(0,0,1)$, which is the projection point of O on the plane $z=1$. We achieve the solution for the length of ${\mathit{DP}}^{″}$ by constructing the second pair of equivalence relations as follows:

$$\left\{\begin{array}{cc}\hfill \left|\mathit{DP}\right|& =|{\mathit{PP}}^{\prime }|\times cos(\mathit{AB},{\mathit{PP}}^{\prime })\hfill \\ \hfill |{\mathit{DP}}^{″}|& =|{\mathit{PP}}^{\prime }|\times sin(\mathit{AB},{\mathit{PP}}^{\prime })\times \left|\mathit{BC}\right|\hfill \end{array}\right.$$

(8)

Finally, combining the above equations, the absolute depth is solved by:

$$abs\_d=|{\mathit{PP}}^{\prime }|\times \frac{cos(\mathit{AB},{\mathit{PP}}^{\prime })+sign\times sin(\mathit{AB},{\mathit{PP}}^{\prime })\times \left|\mathit{BC}\right|}{\left|\mathit{AB}\right|}$$

(9)

3.3. World Coordinate Calculation

The function of this method is to convert pixel coordinates to world coordinates. The main process is divided into two parts. First, converting pixel to camera coordinates, and then continuing the conversion to world coordinates through the both spatial coordinate system conversion relationship.

Record the longitude of the camera as L, the latitude as B, and the elevation as H. The Z-axis of camera coordinate system coincides with the camera optical axis, and the direction is outward, so it only needs to be positively rotated around the X-axis by a pitch angle, noted as p, and the Z-axis direction is vertical to the horizontal plane. Then, rotate B around the Y-axis, and finally rotate the Z-axis ${270}^{°}-L$, the three axis direction and the geocentric coordinate system to maintain the same. Finally, the formula of rotation matrix ${\mathbf{R}}_W$ is:

$${\mathbf{R}}_W=R_z({270}^{°}-L)\times R_y\left(B\right)\times R_x({90}^{°}+p)$$

(10)

The origin of the camera coordinate system can be regarded as the position of the UAV, which is written as $O_C$, and the center of the circle of the geocentric coordinate system is written as $O_W$. Then $O_W{O}_C$ is the geocentric coordinate of the UAV, which is written as $(x_{OC},y_{OC},z_{OC})$. The conversion method through latitude, longitude and elevation is:

$$\left\{\begin{array}{cc}\hfill x_{OC}& =(N+H)\times cosB\times cosL\hfill \\ \hfill y_{OC}& =(N+H)\times cosB\times sinL\hfill \\ \hfill z_{OC}& =(N\times (1-E^2)+H)\times sinB\hfill \\ \hfill E^2& ={\displaystyle \frac{a^2-b^2}{a^2}}\hfill \\ \hfill N& ={\displaystyle \frac{a}{\sqrt{1-E^2{sin}^{2}B}}}\hfill \end{array}\right.$$

(11)

where the equatorial radius of the reference ellipsoid is noted as a and the polar radius of the reference ellipsoid is b.

Finally, the conversion process of the target pixel coordinates is summarized as:

$$\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\end{array}\right]={\mathbf{R}}_W\times \mathbf{inv}\_\mathbf{K}\times \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]\times abs\_d+{\mathit{O}}_W{\mathit{O}}_C$$

(12)

4. Results

In this section, we will show the positioning effect of our method in practical applications. The experiments are divided into four parts: (1) optical flow model training, (2) depth calculation, and (3) target localization experiments in complex environments.

4.1. Models Training

4.1.1. Optical Flow Model

The purpose of introducing the optical flow estimation model is to find the correspondence between the pixel coordinates in two frames. We select the currently well-performing model, RAFT [33], and train it. It takes the optical flow estimation problem and estimates the motion of all pixels end-to-end using deep neural networks and achieves higher accuracy and robustness than other optical flow algorithms. It has strong generalization over many datasets, so we think it can also have good performance in remote sensing images. RAFT performs well in terms of number of parameters, inference time, which can meet the real-time requirements well. We reprojected remote sensing images of complex terrain based on depth information with random orientation camera positional changes to form the dataset used for network training. We show part of the dataset, as well as the test results in Figure 3.

The whole dataset consists of 20 videos with a frame rate of 30 frames/s. The UAVs fly at 20–60 m and have a flight speed of 1 m/s. The scene contents of the videos mainly include forests, steep cliffs, mountains, etc. We obtained 7201 images by extracting key frames, and modeled the scene through SLAM method. The modeling results serve as the reference value source for depth information. The scene with point cloud is displayed in Figure 4. We amplified all images by performing multiple random direction reprojection operations on each one, and finally obtained a dataset containing 36,005 images. The training set and verification set are randomly allocated according to the ratio of 8:2.

4.1.2. Target Detection Model

We trained each of the five models with different specifications and complexity of YOLOv5, in descending order of network complexity, YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The datasets used for training and testing are from the same source as the datasets used for training the optical flow model. To enlarge the dataset size, we used data enhancement methods including rotation, scaling, and single-shoulder transformation. The dataset contains 3080 images and 45,096 target labels. The ratio of training set to validation set is about 8:2. The trends of precision and recall in training are shown in Figure 5. The performance effect of each model is shown in the

Table 1. Performance of different models on the RTX 2070 Super.

Model	Precision	Recall	[email protected]	s (ms)
YOLOv5n	0.823	0.606	0.687	3.6
YOLOv5s	0.848	0.673	0.726	5.5
YOLOv5m	0.881	0.693	0.752	7.1
YOLOv5l	0.904	0.698	0.763	10.0
YOLOv5x	0.889	0.726	0.778	16.1

. The $s\left(ms\right)$ represents the time required to process an image when the target detection model is invoked alone. The comparison results show that YOLOv5 performs the best in terms of the combined evaluation criteria of accuracy and recall. The YOLOv5x model is the preferred choice for the experiments provided that the real-time requirements are met. Considering the need to access two neural network models at the same time and to ensure the real-time performance of the operation, we used YOLOv5l for the subsequent experiments.

4.2. Depth Calculation

In the experiment, we used the pictures collected in the places shown in Figure 4 that did not appear in the training set to form the test set. The test set contains 604 images, which are composed of three video key frames. Considering that our method requires two adjacent frames for computation, we compared the results of $601sets$ except the first key frame of each video.

We evaluated the effectiveness of our method by comparing it with the reference depth theywere born using the SLAM method. We likewise compare with three other methods, Monodepth2 [34], Madhuanand et al. [24], and CADepth [35]. Monodepth2 [34] uses a joint training approach to train both PoseNet and DepthNet using consecutive image sequences for self-supervised training. Madhuanand et al. [24] proposes for the first time to train a depth estimation model using tilted drone videos. All these models are trained under the same environment, dataset, and resolution as our method to make them comparable.

To evaluate the performance, we compared the Madhuanand et al. [24] according to a series of metrics. These include Absolute Relative difference (Abs Rel), given in Equation (13), used to calculate the average difference between the reference and corresponding pixel position of the method’s predicted depth, Squared Relative difference (Sq Rel) as given in Equation (14) which is used to represent the squared difference between reference and method predicted depth, Root Mean Square Error (RMSE), given in Equation (15), accuracy as given in Equation (16).

$$\begin{array}{c}AbsRel=\frac{1}{N}\sum _{i=1}^N\frac{|d\left(x_i\right)-d^{\prime }\left(x_i\right)|}{d\left(x_i\right)}\end{array}$$

(13)

$$\begin{array}{c}SqRel=\frac{1}{N}\sum _{i=1}^N\frac{{(d\left(x_i\right)-d^{\prime }\left(x_i\right))}^2}{d\left(x_i\right)}\end{array}$$

(14)

$$\begin{array}{c}RMSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{(d\left(x_i\right)-d^{\prime }\left(x_i\right))}^2}\end{array}$$

(15)

$$\begin{array}{c}accuracy\left({\delta }_{\theta }\right)=\frac{1}{N}\sum _{i=1}^{N}max(\frac{d\left(x_i\right)}{d^{\prime }\left(x_i\right)},\frac{d^{\prime }\left(x_i\right)}{d\left(x_i\right)})<\theta \end{array}$$

(16)

where $d\left(x_i\right)$ is the reference depth of each pixel at the $i_{th}$ position and $d^{\prime }\left(x_i\right)$ is the method predicted depth at the $i_{th}h$ position. The accuracy of Equation (16) is the percentage of pixels within a certain threshold $\theta$. Based on the standard benchmarks of KITTI quantitative evaluation, the thresholds are chosen as $5\%$, $15\%$, and $25\%$. The predicted depths of our method with these depth estimation models are visualized in Figure 6. The quantitative evaluation results are shown in

Table 2. Comparison of assessment results.

Method	Abs Rel	Sq Rel	RMSE	${\mathit{\delta }}_{1.05}$	${\mathit{\delta }}_{1.15}$	${\mathit{\delta }}_{1.25}$
Monodepth2	0.479	3.961	6.001	0.715	0.881	0.967
Madhuanand et al.	0.460	3.506	5.816	0.727	0.893	0.973
CADepth	0.431	2.769	5.479	0.744	0.906	0.983
Ours	0.425	2.563	5.315	0.732	0.915	0.983

. In steep and rugged scenes, our method has higher accuracy. It is also obvious from the depth information visualization images that the depth distribution predicted by our method is more consistent with the real depth distribution and is not limited by the inherent depth distribution trend at any angle, in any terrain.

Areas with larger depth values are colored blue-purple, and smaller ones are yellow. Our method has clearer edges in a variety of scenes including cliffs, woods, slopes, etc. Additionally, in a variety of depth distribution trends, our method can better and more accurately reflect the changes in depth. However, in the edge region, our method sometimes has errors, especially when the true depth value of the image edge varies widely. Since there are no moving objects involved in the test set, the effect of depth prediction for moving objects is not reflected in the test images. We also performed a quantitative evaluation to compare the effects between several methods more accurately.

The quantitative metrics between the methods are shown in Table 2. The data in the table are the evaluation metrics calculated by calculating the ratio of the reference depth to the mean value of the predicted depth of each method, after scaling the predicted depth. From the table, we can observe that our method achieves the best results for all three evaluation metrics, Abs Rel, Sq Rel, and RMSE. At a threshold of 1.05, the accuracy of our method is second only to CADepth and obtains the best results with the same effect as CADepth at a threshold of 1.25.

4.3. Positioning in Complex Scenes

To demonstrate the effectiveness and accuracy of the method in complex scenes, we designed several field positioning experiments. Experimenters were dispersed into scenes as positioning targets. These scenes included hills, woods, cliffs, etc. The experiments were conducted in the Xishan Forest Park (${24}^{\circ }{57}^{\prime }{6}^{″}$N ${102}^{\circ }{38}^{\prime }{24}^{″}$E) and the Gudian Wetland Park (${24}^{\circ }{46}^{\prime }{34}^{″}$N ${102}^{\circ }{44}^{\prime }{57}^{″}$E). The target localization calculation was partially run on a laptop with an i7-10875H CPU, RTX 2070 SUPER GPU and 16GB RAM. The computation speed can reach more than 25 frames per second, which meets the real-time requirement. Finally, the calculated points are displayed in the form of coordinates in Figure 7. The error results of target localization are shown in

Table 3. Target positioning result.

Target	True Longitude	True Latitude	Positioning Lng	Positioning Lat	Error (m)
a1	${102.63915721}^{°}$	${24.95188318}^{°}$	${102.63912727}^{°}$	${24.95188277}^{°}$	3.76
a2	${102.63913654}^{°}$	${24.95188509}^{°}$	${102.63910836}^{°}$	${24.95188470}^{°}$	3.54
a3	${102.63912274}^{°}$	${24.95190336}^{°}$	${102.63909213}^{°}$	${24.95190294}^{°}$	3.84
b1	${102.74805046}^{°}$	${24.77594520}^{°}$	${102.74808081}^{°}$	${24.77594815}^{°}$	3.88
b2	${102.74807838}^{°}$	${24.77597145}^{°}$	${102.74811375}^{°}$	${24.77597488}^{°}$	4.53
c1	${102.63912085}^{°}$	${24.95211931}^{°}$	${102.63912903}^{°}$	${24.95207762}^{°}$	5.70
d1	${102.63921830}^{°}$	${24.95203052}^{°}$	${102.63926834}^{°}$	${24.95200349}^{°}$	6.26
d2	${102.63922935}^{°}$	${24.95202471}^{°}$	${102.63926816}^{°}$	${24.95200375}^{°}$	4.855
e1	${102.63908945}^{°}$	${24.95147445}^{°}$	${102.63905687}^{°}$	${24.95142916}^{°}$	7.56
f1	${102.63908748}^{°}$	${24.95178806}^{°}$	${102.63906224}^{°}$	${24.95177477}^{°}$	3.09
g1	${102.75100980}^{°}$	${24.77648635}^{°}$	${102.75100103}^{°}$	${24.77647137}^{°}$	1.96
g2	${102.75107292}^{°}$	${24.77642159}^{°}$	${102.75106357}^{°}$	${24.77640562}^{°}$	2.09
h1	${102.81578700}^{°}$	${24.85020296}^{°}$	${102.81584440}^{°}$	${24.85018642}^{°}$	6.48
i1	${102.74980807}^{°}$	${24.77701205}^{°}$	${102.74981547}^{°}$	${24.77700765}^{°}$	1.265
i2	${102.74989316}^{°}$	${24.77702797}^{°}$	${102.74990084}^{°}$	${24.77702340}^{°}$	1.31
i3	${102.74997273}^{°}$	${24.77696585}^{°}$	${102.74998578}^{°}$	${24.77695809}^{°}$	2.23

.

The error distance of localization is derived by calculating the spatial distance between the true and predicted coordinates. As can be seen from the settlement results in the table, $75\%$ of positioning accuracy errors remain within 5 m in complex scenarios. The overly steep environment is still generally lower than the positioning accuracy in other environments. However, the error can still be guaranteed to be within 8 m. Overall, this method can meet the positioning requirements in complex scenes.

We cite different depth estimation methods involved in positioning for comparing the effect of depth estimation methods on positioning errors. Since the exact distance from the camera to a plane is not available in a complex environment, the scale factor of the depth map cannot be calculated by predicting the camera height during the experiment. We designed a computational method for the calculation of scale factor. This method derives the scale factor corresponding to two depth maps by the different representations of two depth maps with different UAV positions at the same spatial point. The formula is as follows:

$${\alpha }_1\times dept{h}_1\times inv\_K\times \left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]-{\alpha }_2\times dept{h}_2\times inv\_K\times R\times \left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]+b\times \left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]=\left[\begin{array}{c}x\\ y\\ z\end{array}\right]$$

(17)

where $(u_i,v_i)$ is a set of determined corresponding points, $dept{h}_i$ is the relative depth information corresponding to this pair of points, and $\mathbf{R}$ is the rotation matrix of the UAV. The equation can be solved for three unknowns. ${\alpha }_i$ is the scale factor corresponding to the two depth maps. b represents the error distance, and the scaling factor is more accurate only when the value of B is smaller. In the experiment, the absolute value of b is limited to less than 0.2, which represents the selected corresponding point at a distance less than 0.2 m in space.

The analysis of the positioning errors after plugging different depth estimation models into the target positioning method is shown in

Table 4. Target positioning result with different depth estimate method.

	Monodepth2	Madhuanand et al.	CADepth	Error (m)
$Erro{r}_{min}$	2.47	2.66	1.73	1.265
$Erro{r}_{max}$	68.33	57.145	19.31	7.56
$Erro{r}_{mean}$	22.6872	20.1564	12.2137	3.8969
${\delta }_3$	0.125	0.125	0.125	0.3125
${\delta }_5$	0.25	0.1875	0.5625	0.75
${\delta }_8$	0.6875	0.6875	0.875	1.0

. We calculated the minimum, maximum, and average values of the errors, and counted the proportion of samples with errors within 3 m, 5 m, and 8 m of the total samples, respectively. From the table, we can see that the target localization results using our depth estimation method have smaller errors overall and more stable results.

5. Discussion

In this paper, a new method is proposed for estimating the depth information of UAV videos in complex scenes. This method is used to improve the accuracy of target localization in complex scenes. The method we propose requires a progressive depth calculation based on the pixel coordinate relationship between frames based on the motion information of the UAV. The pixel motion used in the computation is predicted by the trained optical flow estimation model. Although supervised training is performed, the supervised signal can be obtained by reprojection calculation, which is less difficult to obtain and more accurate. Moreover, the trained model is not limited by the original terrain type because it is detached from the original scene of the terrain, and can be used for a variety of multi-angle complex scenes.

In terms of target positioning, the computational process that introduces depth estimation is detached from the dependence on elevation information and assumed planes, which allows for much higher positioning accuracy in complex terrain, especially in scenes with large elevation changes. The calculation of depth information is related to the pixel motion distance. The smaller the pixel motion distance is, the larger the depth estimation error is. When the displacement of the UAV is parallel to the imaging plane of the camera, the pixel motion distance corresponding to the same spatial point reaches the maximum and the depth estimation is the most accurate.

In our future work, we will further explore the depth estimation methods for remote sensing videos in complex scenes, improve the depth estimation accuracy for reflective and dynamic objects, and further improve the accuracy of target localization methods.