Remote Target High-Precision Global Geolocalization of UAV Based on Multimodal Visual Servo

Abstract

In this work, we propose a geolocation framework for distant ground targets integrating laser rangefinder sensors with multimodal visual servo control. By simulating binocular visual servo measurements through monocular visual servo tracking at fixed time intervals, our approach requires only single-session sensor attitude correction calibration to accurately geolocalize multiple targets during a single flight, which significantly enhances operational efficiency in multi-target geolocation scenarios. We design a step-convergent target geolocation optimization algorithm. By adjusting the step size and the scale factor of the cost function, we achieve fast accuracy convergence for different UAV reconnaissance modes, while maintaining the geolocation accuracy without divergence even when the laser ranging sensor is turned off for a short period. The experimental results show that through the UAV’s continuous reconnaissance measurements, the geolocalization error of remote ground targets based on our algorithm is less than 7 m for 3000 m, and less than 3.5 m for 1500 m. We have realized the fast and high-precision geolocalization of remote targets on the ground under the high-altitude reconnaissance of UAVs.

1. Introduction

Unmanned Aerial Vehicles (UAVs) are widely utilized in Earth observation due to their scalability and integration with various sensor configurations [1,2]. Particularly, the combination of UAVs with multimodal vision sensors (including visible and infrared spectra) enables video-image information acquisition across various scenarios, meeting application requirements such as urban intelligent transportation systems, environmental field monitoring, and 3D scene reconstruction [3,4,5,6]. To accomplish these tasks, knowing the mapping relationship between targets in video sequences and their actual geospatial positions is often necessary, which necessitates visual-based target geolocation [7].

Visual-based target localization primarily falls into monocular vision [8,9,10] and multi-view vision approaches [11,12,13]. Monocular vision systems, lacking direct depth measurement capability, typically require geometric features of targets and reference objects for depth estimation [14]. Kim et al. implemented inertial measurement unit-based camera pose correction, transforming corrected frontal images to bird’s-eye view through inverse perspective mapping for target position, size, and orientation estimation [15]. You et al. generated simulated LiDAR point clouds from image data to obtain accurate pixel-level depth information [16]. Sun et al. developed an unsupervised learning-based depth estimation module combined with monocular image input for target localization [17]. While these methods demonstrated effectiveness on the KITTI dataset, their applicability remains constrained to targets within a 30 m range as per the KITTI dataset characteristics [18]. With further research, monocular vision has made significant breakthroughs in target localization in outdoor long-distance environments. Li et al. designed an outdoor monocular localization device achieving target positioning at 135 m through the fusion of multi-focal-length measurements [19]. Gao et al. proposed a soft correspondence constraint, weakening the coupling between detection and localization to increase the effective target localization distance of monocular vision outdoor targets to 150 m [20]. However, precise focal length calibration remains critical for monocular systems. Adjusting sensor focal length to enhance distant target features significantly amplifies localization errors, fundamentally limiting monocular vision’s capability for accurate long-range target localization.

In contrast, multi-view vision systems employ multiple vision sensors to observe targets simultaneously. These systems leverage pixel correspondences of identical targets across multiple images combined with sensor pose information for relative position deduction [21,22]. Xin et al. proposed a multi-UAV collaborative geolocalization method based on inertial navigation position information, which achieves the relative position inference of the target even when GPS information is unavailable, so that the localization results are not global, and the computed relative position information gradually loses its reliability as the IMU drifts [23]. Pan et al. proposed a moving target geolocalization method based on dual UAV vision, where they used two navigation-capable UAVs combined with multi-view geometry to estimate the target altitude to achieve target geolocalization within the UAV’s overhead field of view at a flight altitude of 100 m [24]. Sun et al. created an air–ground collaborative geolocation system (AGCG) that employs ground-based stereo vision to guide UAV alignment, achieving 4.4 m average accuracy at 300 m range through GPS-based tracking [25]. Liu et al. used the target position information as the control signal, servo-controlled two ground cameras with known geographic locations to align with the target, solved the precise geographic location of the target based on the camera attitude rotation angle, and achieved geolocalization of a 1200 m long-distance target with an error of 3.5% [26]. While multi-view methods overcome focal length constraints of long-range geolocation, they typically require precise calibration before measurements.

The vast majority of the visual localization methods mentioned above use a local coordinate system to represent the target location, which greatly limits the connection between observations. The UAV’s own GPS information is readily available, so UAV-based visual localization methods can achieve global geolocation [27]. In recent years, aerial remote sensing image small target recognition technology and multimodal image target recognition technology have made great progress [28,29,30,31]. The wide application of the electro-optical stabilization and tracking platform (EOSTP) and Micro-Electro-Mechanical System (MEMS) inertial attitude measurement system in UAV platforms [32,33,34] has led to significant improvement in the attitude control accuracy of UAV payloads. In this paper, the UAV payload is servo-controlled to keep the center of the multimodal image always aligned with the target. Global geolocalization of remote ground targets is performed by combining the UAV moment-knowable GPS information and UAV payload attitude information. The main innovations of this paper are as follows:

• A UAV visual geolocalization framework based on multimodal visual servoing is proposed, which is combined with UAV GPS information to achieve global geo-localization of remote targets on the ground;
• The hardware framework requires only one initial calibration after the assembly is completed and does not need to be calibrated again before each subsequent measurement, which greatly improves the positioning efficiency.
• The stepped localization convergence model improves the robustness to abnormal measurement interference based on ensuring the convergence speed of localization accuracy.

2. Materials and Methods

2.1. Materials

Based on our previous related research [35], we use the same hardware framework in this paper. As shown in Figure 1, we use an ASN-216 vertical take-off and landing (VTOL) UAV for continuous reconnaissance of ground targets, employing image-based visual servo control of the UAV payload to always keep the ground target in the center of the field of view. We systematically record UAV GPS coordinates, payload attitude parameters, and laser-measured target distances throughout this process. Notably, the UAV’s maximum operational altitude reaches 5000 m with an effective control range of 8000 m. The computer platforms used in this paper are CPU I7-9700 (Intel, City of Santa Clara, CA, USA) and GPU GTX2070S (NVIDIA, City of Santa Clara, CA, USA), and the errors of each sensor of the UAV platform are shown in

Table 1. Error of each sensor of the UAV platform.

Parameter Name	Error Value
UAV navigational yaw angle	1°
UAV navigational pitch angle	0.2°
UAV navigational roll angle	0.2°
Optoelectronic load pitch angle	0.1°
Optoelectronic load rotation angle	0.1°
UAV GPS	10 m
Laser measurement distance	10 m

. The frequency of visual image and other measurement data acquisition is 25 times/second.

We use the UAV multimodal vision servo module [32] to control the UAV payload to continuously measure remote targets on the ground. The binocular vision measurement of the VSG algorithm requires binocular alignment before each measurement, whereas our method requires only one calibration to obtain the attitude correction parameters after the hardware assembly is completed. No additional calibration is required for each measurement on the premise that the sensors of the UAV do not undergo drastic collision displacement, which greatly improves the efficiency of mission execution. Continuous monocular visual servo measurements provide sufficient data for more accurate remote target geolocation, and to take advantage of continuous measurements more efficiently, we design a stepped-convergence target geolocation optimization method. In this section, our proposed method is described in detail.

2.2. Global Geolocation Model

In the study of Liu et al. [26], the transformation relationship between the global geographic coordinate system and the local geographic coordinate system, denoted as $C_{BLH2NED}$ in this paper, is derived in detail. Based on that, we introduce a laser ranging sensor to measure the distance from the target to the visual sensor, based on which we get rid of the dependence of visual geolocalization on the accurate focal length information of the visual sensor.

We can obtain the position of the UAV under the localized geographic coordinate system from the UAV’s GPS information ${\left[\begin{array}{ccc}{B}_u& L_u& H_u\end{array}\right]}^T$:

$$\left[\begin{array}{c}{x}_u\\ y_u\\ z_u\end{array}\right]=C_{BLH2NED}·\left[\begin{array}{c}{B}_u\\ L_u\\ H_u\end{array}\right]$$

(1)

The position of a ground target in the local coordinate system can be defined by the laser-measured distance transformed by two attitude rotations:

$$\left[\begin{array}{c}{x}_t\\ y_t\\ z_t\end{array}\right]=R_u^{-1}·R_p^{-1}·\left[\begin{array}{c}L\\ 0\\ 0\end{array}\right]+\left[\begin{array}{c}{x}_u\\ y_u\\ z_u\end{array}\right]$$

(2)

where $R_u^{-1}$ is the inverse transformation of the UAV attitude rotation matrix. $R_p^{-1}$ is the inverse transformation of the UAV payload attitude rotation matrix. $L$ is the laser measurement distance. The UAV attitude and payload attitude rotation matrices are:

$$R_u=R_{\gamma \theta \phi }=\left[\begin{array}{ccc}{\phi }_c{\theta }_c& {-\phi }_s{\gamma }_c+{\phi }_c{\theta }_s{\gamma }_s& {\phi }_s{\gamma }_s+{\phi }_c{\theta }_s{\gamma }_c\\ {\phi }_s{\theta }_c& {\phi }_c{\gamma }_c+{\phi }_s{\theta }_s{\gamma }_s& {-\phi }_c{\gamma }_s+{\phi }_s{\theta }_s{\gamma }_c\\ -{\theta }_s& {\theta }_c{\gamma }_s& {\theta }_c{\gamma }_c\end{array}\right]$$

(3)

$$R_p=R_{\beta \alpha }=\left[\begin{array}{ccc}{\alpha }_c{\beta }_c& {-\alpha }_s& {\alpha }_c{\beta }_s\\ {\alpha }_s{\beta }_c& {\alpha }_c& {\alpha }_s{\beta }_s\\ -{\beta }_s& 0& {\beta }_c\end{array}\right]$$

(4)

$\phi$ is the UAV heading angle, $\gamma$ is the UAV pitch angle, and θ is the UAV roll angle. $\alpha$ is the rotational azimuth of the photoelectric load; $\beta$ is the pitch angle of the photoelectric load. The lower right corner labels represent the sine function (s) and cosine function (c), respectively. The latitude, longitude, and altitude of the target can be obtained by converting the results calculated in Equation (2) to the global geographic coordinate system.

$$\left[\begin{array}{c}{B}_t\\ L_t\\ H_t\end{array}\right]=C_{BLH2NED}^{-1}·\left[\begin{array}{c}{x}_t\\ y_t\\ z_t\end{array}\right]$$

(5)

2.3. Attitude Error Calibration

The accuracy of a UAV’s attitude is critical for ground target localization. We calibrate the system installation error by tracking measurements of known target points to improve the system’s localization accuracy for unknown targets. The exact location of the target is known and converted to a local geographic coordinate system expressed as ${\left[\begin{array}{ccc}{x}_t& y_t& z_t\end{array}\right]}^T$. The system’s estimate of the position of this target point at moment $i$ is:

$$\left[\begin{array}{c}{x}_{ti}\\ y_{ti}\\ z_{ti}\end{array}\right]=R_u^{-1}·R_p^{-1}·\left[\begin{array}{c}{L}_i\\ 0\\ 0\end{array}\right]+\left[\begin{array}{c}{x}_{ui}\\ y_{ui}\\ z_{ui}\end{array}\right]$$

(6)

Simultaneous Taylor series expansion of the position estimates for moments $i$ and $i+1$ can be obtained:

$$T=\left[\begin{array}{c}{x}_{ti}\\ y_{ti}\\ z_{ti}\\ x_{ti+1}\\ y_{ti+1}\\ z_{ti+1}\end{array}\right]-\left[\begin{array}{c}{x}_{ti}\left(0\right)\\ y_{ti}\left(0\right)\\ z_{ti}\left(0\right)\\ x_{ti+1}\left(0\right)\\ y_{ti+1}\left(0\right)\\ z_{ti+1}\left(0\right)\end{array}\right]=\left[\begin{array}{ccc}{\displaystyle \frac{\partial x_{ti}}{\partial \phi }}& {\displaystyle \frac{\partial x_{ti}}{\partial \gamma }}& {\displaystyle \frac{\partial x_{ti}}{\partial \theta }}\\ {\displaystyle \frac{\partial y_{ti}}{\partial \phi }}& {\displaystyle \frac{\partial y_{ti}}{\partial \gamma }}& {\displaystyle \frac{\partial y_{ti}}{\partial \theta }}\\ {\displaystyle \frac{\partial z_{ti}}{\partial \phi }}& {\displaystyle \frac{\partial z_{ti}}{\partial \gamma }}& {\displaystyle \frac{\partial z_{ti}}{\partial \theta }}\\ {\displaystyle \frac{\partial x_{ti+1}}{\partial \phi }}& {\displaystyle \frac{\partial x_{ti+1}}{\partial \gamma }}& {\displaystyle \frac{\partial x_{ti+1}}{\partial \theta }}\\ {\displaystyle \frac{\partial y_{ti+1}}{\partial \phi }}& {\displaystyle \frac{\partial y_{ti+1}}{\partial \gamma }}& {\displaystyle \frac{\partial y_{ti+1}}{\partial \theta }}\\ {\displaystyle \frac{\partial z_{ti+1}}{\partial \phi }}& {\displaystyle \frac{\partial z_{ti+1}}{\partial \gamma }}& {\displaystyle \frac{\partial z_{ti+1}}{\partial \theta }}\end{array}\right]·\left[\begin{array}{c}∆\phi \\ ∆\gamma \\ ∆\theta \end{array}\right]$$

(7)

Solving the matrix equation gives a one-step estimate of the attitude angle ${\left[\begin{array}{ccc}\Delta \phi & \Delta \gamma & \Delta \theta \end{array}\right]}^T$ and corrects the attitude measurements at moment $i$ and moment $i+1$.

$$\left[\begin{array}{c}\begin{array}{c}{\phi }_{i,2}\\ {\gamma }_{i,2}\\ {\theta }_{i,2}\end{array}\\ \begin{array}{c}{\phi }_{i+1,2}\\ {\gamma }_{i+1,2}\\ {\theta }_{i+1,2}\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}{\phi }_{i,1}\\ {\gamma }_{i,1}\\ {\theta }_{i,1}\end{array}\\ \begin{array}{c}{\phi }_{i+1,1}\\ {\gamma }_{i+1,1}\\ {\theta }_{i+1,1}\end{array}\end{array}\right]+\left[\begin{array}{c}\begin{array}{c}\Delta \phi \\ \Delta \gamma \\ \Delta \theta \end{array}\\ \begin{array}{c}\Delta \phi \\ \Delta \gamma \\ \Delta \theta \end{array}\end{array}\right]$$

(8)

Substitute the corrected attitude angle into Equation (6) until the $j$th iteration lets $T^{T}T<200$. At this point, an estimate of the attitude angle mounting error at moment $i$ is obtained ${\left[\begin{array}{ccc}{\phi }_{ei}& {\gamma }_{ei}& {\theta }_{ei}\end{array}\right]}^T$. The attitude angle mounting error of the system is obtained by averaging the error estimates from multiple moments over a period of time.

$$\left[\begin{array}{c}{\phi }_{ei}\\ {\gamma }_{ei}\\ {\theta }_{ei}\end{array}\right]=\left[\begin{array}{c}{\phi }_{i,j}\\ {\gamma }_{i,j}\\ {\theta }_{i,j}\end{array}\right]-\left[\begin{array}{c}{\phi }_{i,1}\\ {\gamma }_{i,1}\\ {\theta }_{i,1}\end{array}\right]$$

(9)

The calibration parameters obtained here are applicable to each measurement process, and the whole sensor system does not need to be calibrated repeatedly when geolocalizing different targets, which greatly exploits the convenience of UAVs performing tasks in the air.

2.4. Fast Step Convergence Model

Gaussian Newton method [36] has strong local convergence ability in solving least squares problems, and target geolocalization based on continuous reconnaissance measurements by UAVs is a typical least squares problem. Least squares is a mathematical optimization technique. It searches for the best function match of the data by minimizing the sum of squares of the errors. In the target positioning, three-dimensional coordinate information error optimization solution has a similarity. The selected target positioning with the x-axis information requires the following analysis, and the error function in the least squares problem in solving the target coordinates can be defined as

$$\underset{x}{min}F\left(x_i+\Delta x_i\right)=\Vert f\left(x_i+\Delta x_i\right){\Vert }^2$$

(10)

Here, the first order Taylor expansion of the error function is performed, and the optimization objective function after the expansion is

$$\underset{x}{min}F\left(x_i+\Delta x_i\right)\approx \Vert f\left(x_i\right)+J{\left(x_i\right)}^T\Delta x_i{\Vert }^2$$

(11)

where $J\left(x_i\right)$ is the Jacobi matrix of $f\left(x\right)$, the only variable in Equation (11) is $\Delta x_i$, and $x_i$ is a definite value, so Equation (11) is a quadratic function with $\Delta x_i$ as a variable. Therefore, this function has a minimum value, and the condition for its minimum value is that the derivative of this equation with respect to $\Delta x_i$ is zero.

$${\displaystyle \frac{\partial \Vert f\left(x_i\right)+J{\left(x_i\right)}^T\Delta x_i{\Vert }^2}{\partial \Delta x_i}}=0$$

(12)

$$J\left(x_i\right)J{\left(x_i\right)}^T\Delta x_i=-J\left(x_i\right)f\left(x_i\right)$$

(13)

Let $H\left(x_i\right)=J\left(x_i\right)J{\left(x_i\right)}^T$, $g\left(x_i\right)=-J\left(x_i\right)f\left(x_i\right)$. Then, the incremental equation for the Gaussian Newton method is:

$$H\left(x_i\right)\Delta x_i=g\left(x_i\right)$$

(14)

In solving $\Delta x_i$, it is necessary to compute $H{\left(x_i\right)}^{-1}$. $H\left(x_i\right)$ has only semi-positive definite properties, and in the actual computation $H\left(x_i\right)$ may be a singular matrix, when $\Delta x_i$ is unstable and the algorithm fails to converge.

However, the Gaussian Newton method is extremely sensitive to the initial guess value, and when the initial prediction seriously deviates from the true position of the target, the Gaussian Newton method will not be able to converge to the global optimal solution. In the face of possible anomalous measurements, the Gaussian Newton method will fall into the local optimum due to the lack of global search capability, resulting in large target localization errors. In order to cope with the problem of the non-convergence of the Gaussian Newton method due to local data anomalies encountered in practical calculations, while retaining the characteristics of fast local convergence of the Gaussian Newton method, this paper is inspired by the Gaussian Newton method using the gradient to predict the minimum coordinate value of the error and proposes a fast stepwise convergence method for the geolocalization of the target.

First, we sampled 100 sets of measurements at equal intervals from the continuous measurement global data, where the $k$th set of the data is denoted as ${\left[\begin{array}{ccc}{x}_{tk}& y_{tk}& z_{tk}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{uk}& y_{uk}& z_{uk}\end{array}\right]}^T, L_k, R_{pk}, R_{uk}$. $R_{pk}$ and $R_{uk}$ denote the UAV attitude and payload attitude of the extracted measurements. The initial target localization estimate and UAV measurements of the system at moment $i$ are ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}& z_{ti}\end{array}\right]}^T, L_i$. The distance error function and angle error function are defined as follows:

$$d_k={\displaystyle \frac{L_i-\sqrt{{\left(x_{tk}-x_{ui}\right)}^2+{\left(y_{tk}-y_{ui}\right)}^2+{\left(z_{tk}-z_{ui}\right)}^2}}{\epsilon }}$$

(15)

$${\omega }_k={\displaystyle \frac{\rho -L_k}{\eta }}$$

(16)

$$\left[\begin{array}{c}\rho \\ 0\\ 0\end{array}\right]=R_{pk}·R_{uk}· \left[\begin{array}{c}{x}_{ti}-x_{uk}\\ y_{ti}-y_{uk}\\ z_{ti}-z_{uk}\end{array}\right]$$

(17)

$$\eta ={\displaystyle \frac{1}{\ln \left(\frac{1}{99}\left({{\displaystyle \sum }}_{k=1}^{100}{({\phi }_k-\overline{\phi })}^2+{{\displaystyle \sum }}_{k=1}^{100}{({\theta }_k-\overline{\theta })}^2+{{\displaystyle \sum }}_{k=1}^{100}{({\gamma }_k-\overline{\gamma })}^2\right)\right)}}$$

(18)

$$\epsilon ={\displaystyle \frac{1}{log\left(\frac{1}{99}{{\displaystyle \sum }}_{k=1}^{100}{(L_k-\overline{L})}^2\right)}}$$

(19)

where $\epsilon$ is the distance error scale factor, and $\eta$ is the angle error scale factor. They are represented by the attitude measurement variance and the laser measurement distance variance of the extracted measurements. The distance error function and angle error function are combined to construct the error loss function.

$${\mathcal{L}}_{cost}={\displaystyle \sum }_{k=1}^{100}{d_k}^2+{\displaystyle \sum }_{k=1}^{100}{{\omega }_k}^2$$

(20)

Calculate the cost function for the original predicted coordinates and the stepping coordinates ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}& z_{ti}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{ti}+\lambda & y_{ti}& z_{ti}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{ti}-\lambda & y_{ti}& z_{ti}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}+\lambda & z_{ti}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}-\lambda & z_{ti}\end{array}\right]}^T$, ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}& z_{ti}\end{array}+\lambda \right]}^T$, and ${\left[\begin{array}{ccc}{x}_{ti}& y_{ti}& z_{ti}\end{array}-\lambda \right]}^T$, respectively. Compare the coordinate cost function values of each step and take the coordinate value with the smallest cost function value as the final predicted coordinate value.

We set a fixed iteration step size so that the target geolocalization error can only dissipate in iteration steps at the fastest rate in the case of anomalous data measurements, and the system still has the ability to converge quickly after new valid data inputs. Such an iterative approach makes full use of the available historical observations to predict the geographic location of the target.

3. Results

In the previous section, we presented a fast stepwise convergence algorithm for high-precision geolocalization of UAV remote targets. In this section, in order to evaluate the performance of our proposed algorithm in an all-around way, we use three different UAV reconnaissance modes to geolocalize targets at different geographic locations, respectively.

3.1. Experiments’ Setup

As shown in Figure 2, our experimental site is located at an airfield in the northern region of Shaanxi Province, China. We set up a total of six sets of experiments involving three UAV reconnaissance modes and two different imaging sensors. The differential GPS localization value of the target is used as the reference value in the experiment. The DGPS accuracy is <0.5 m horizontal, <0.5 m vertical with an update rate of five times per second. In static target geolocalization scenarios, the target position is considered to be unchanging over time, so the differential GPS reference can be interpolated to ensure that the RMSE is based on consistent time-aligned truth values. We compare the current publicly available optimal algorithm for outdoor long-range geolocalization, VSG [24], but this method is only applicable to a single measurement. We improved the VSG using the Gaussian Newton method [34] so that it can effectively utilize the continuous measurement data to make the target geolocation results more accurate. In six extensive sets of experiments, we compared VSG, VSG + G-N, and different preset parameters of our method.

We set up Experiment A and Experiment B as a free circling reconnaissance of ground targets by the UAV using a visible vision sensor, during which the laser ranging sensor was briefly turned off. We used the UAV to perform free circling reconnaissance of ground targets at an altitude of 2400 m, with a hovering radius of 1100 m. The UAV flight trajectory and laser-measured distance in Experiment A are shown in Figure 3a and Figure 4a. The UAV flight trajectory and laser-measured distance in Experiment B are shown in Figure 3b and Figure 4b.

Experiments C and D use visible vision sensors and infrared vision sensors, respectively, to geolocalize targets and simulate potential manned–unmanned cooperative scenarios. Manned–unmanned cooperative scenarios are prevalent in the military domain. In this case, the UAV is between a ground target and a manned aircraft and uses a laser to continuously illuminate the target to provide guidance to the manned aircraft. During the mission, the UAV also provides cover for the manned aircraft, which limits the effective activity area of the UAV. The UAV flight trajectory and laser-measured distance in Experiment C are shown in Figure 3c and Figure 4c. The UAV flight trajectory and laser-measured distance in Experiment D are shown in Figure 3d and Figure 4d.

Experiments E and F used infrared vision sensors and visible vision sensors to geolocalize ground targets, respectively. In both sets of experiments, the UAV performs reconnaissance measurements of the target in a fly-over mode directly over the target at an altitude of 4300 m. The UAV flight trajectory and laser-measured distance in Experiment E are shown in Figure 3e and Figure 4e. The UAV flight trajectory and laser-measured distance in Experiment F are shown in Figure 3f and Figure 4f.

3.2. Experiments’ Results

In order to evaluate the performance of our proposed algorithm in all aspects, we present four sets of step sizes in each set of experiments. Figure 5 and Figure 6 show the results for each of the six sets of experiments. Our method achieves the highest target geolocalization accuracy in all six sets of experiments. Moreover, our method has the fastest accuracy convergence rate as well as higher stability.

As shown in Figure 5a and Figure 6a, our algorithm performs optimally with the preset parameter of $\lambda =0.25$ in Experiment A. After tracking measurements on ground targets for 40 s (the 1000th set of time series), the geolocalization accuracy of our method for remote targets on the ground converges to 5 m and stably maintains itself within this accuracy thereafter. As shown in Figure 6b, before and after briefly turning off and then on the laser ranging sensor in Experiment B, the experimental results exhibit two different convergence results. The experimental results before the 3000th set of time series are similar to those in Experiment A. During the second convergence process, the different preset parameters of our algorithm exhibit convergence again, where the optimal preset parameter is $\lambda =0.05$. This is due to the fact that the smaller convergence step size greatly limits the target geolocalization drift when experiencing measurement disturbances. The VSG + G-N method, which is affected by Gaussian Newton algorithms, is trapped in a local optimum and cannot be corrected back to a reasonable range in a short period of time. The VSG method does not use laser range sensors to participate in the geolocalization of the ground targets and therefore is not disturbed by laser range sensor errors.

As shown in Figure 5c and Figure 6c, our algorithm performs optimally with the preset parameter of $\lambda =0.25$ in Experiment C. After tracking measurements on ground targets for 24 s (the 600th set of time series), the geolocalization accuracy of our method for remote targets on the ground converges to 7 m and stably maintains itself within this accuracy thereafter. As shown in Figure 5d and Figure 6d, the VSG method exhibits greater volatility in Experiment D. This is due to the fact that Experiment D uses the infrared vision sensor as a benchmark for target geolocalization. Compared to the target recognition task based on visible sensors, the target recognition task based on infrared vision sensors is more difficult at long distances. At this time, the laser distance measurement is more reliable in target geolocalization than UAV payload attitude angle. The experimental results show that the optimal preset parameter of our algorithm is $\lambda =0.25$.

In both Experiment E and Experiment F, the UAV is in a reconnaissance mode flying directly above the target, simulating the potential military bombing scenario. Combining Figure 4f and Figure 6f, we can find that the geolocalization accuracy of the target fluctuates significantly when the UAV is directly above the target, i.e., near the 1800th set of time series. This is due to the Eulerian angle deadlock of the UAV payload in the positive downward view, when the reliability of the attitude angle information of the UAV payload is weaker than that of the laser measurement of the distance. In Experiment E, our algorithm performs best with the preset parameter of $\lambda =0.25$. As shown in Figure 4e and Figure 6e, the results of Experiment E present the same phenomenon.

Table 2. Comparison of the target localization error of each algorithm.

Experiment Name		A	B(a)	B(b)	C	D	E	F
VSG one-shot		38.13	39.29	38.13	40.13	41.61	57.19	59.99
VSG + G-N		13.45	17.88	290.61	23.47	21.85	24.48	27.78
Ours	λ = 0.05	2.79	2.43	1.62	9.45	5.74	15.86	22.13
	λ = 0.25	2.12	1.58	3.55	3.30	3.34	4.38	6.24
	λ = 0.45	2.48	2.77	7.14	3.96	3.86	5.93	8.17
	λ = 0.65	2.57	4.05	10.33	4.22	4.12	6.75	9.37

counts the target geolocalization errors of each algorithm in the six sets of experiments. In these six sets of experiments, the preset parameter that reflects the optimal performance of our algorithm is $\lambda =0.25$. We use the root mean square error (RMSE) to characterize the target geolocation error. The statistical interval for Experiment A is from the 1001st frame to the 4500th frame of data. It is worth noting that Experiment B has two convergence processes, and the statistical interval of Experiment B(a) is from the 1001st frame to the 3000th frame of data. The statistical interval of Experiment B(b) is from frame 3001 to frame 5000 of data. Due to the presence of anomalous measurements interfering, the best performance is exhibited with the preset parameter $\lambda =0.05$. The statistical interval of Experiment C is from frame 601 to frame 2000 of the data. The statistical intervals for Experiment D are from the 201st frame to the 2000th frame of data. The statistical interval of Experiment E is from frame 1501 to frame 4000. The statistical interval of Experiment F is from frame 1501 to frame 3300. The results of Experiment D and Experiment E show that our method is still advantageous in infrared vision sensor systems.

4. Discussion

Although our method showed the best performance in all six sets of experiments, analyzing the effects of different preset parameter combinations on the target geolocation accuracy is crucial for our subsequent research work.

Among the preset parameters we set, the distance error scaling factor $\epsilon$ and the angle error scaling factor $\eta$ affect the weights of the distance error function and the angle error function in the loss cost function, respectively. When the error proportion coefficient $\epsilon$ is larger and the angular error proportion coefficient $\eta$ is smaller, the angular error function in the loss cost function has a larger weight. It can be seen from Figure 7. At this time, the localization convergence process presents vertical priority over the horizontal position. The system is suitable for scenarios with better attitude-angle reliability, such as Experiments A, B, C and D. Correspondingly, when the error proportion coefficient $\epsilon$ is smaller and the angular error proportion coefficient $\eta$ larger, the distance error function in the loss cost function has a larger weight, and then the system is suitable for scenarios with better reliability of laser ranging, such as Experiments E and F.

The results in Figure 6 and Figure 8 and Table 2 show that the larger step size target geolocalization accuracy converges faster but is less stable when anomalous measurements interfere than the smaller step size control group. Such a characteristic suggests that we can develop a step-size adaptive strategy in engineering applications to achieve robust maintenance of the converged accuracy by changing the step-size after converging to a certain accuracy for target geolocalization.

5. Conclusions

In this paper, we propose a multimodal visual servo-based framework for UAV geolocalization of remote targets on the ground. By introducing a laser ranging sensor, we get rid of the dependence on the accurate focal length of the visual sensor. The experimental results show that we achieve global geolocalization of remote targets above 3000 m with a localization error of less than 7 m.

However, our current high-precision multimodal visual servo-based geolocation framework remains applicable solely to stationary targets. Practical mission scenarios further require geolocation and motion state estimation for moving ground targets at extended ranges. Based on our framework, new challenges will be faced if geolocalizing ground-based remote motion targets, such as visual servo tracking of remote motion targets, is more difficult. How to realize high-precision geo-localization and motion state estimation for ground-based remote motion targets will be our future research direction.