Vision-Only Localization of Drones with Optimal Window Velocity Fusion

Abstract

Drone localization is essential for various purposes such as navigation, autonomous flight, and object tracking. However, this task is challenging when satellite signals are unavailable. This paper addresses database-free vision-only localization of flying drones using optimal window template matching and velocity fusion. Assuming the ground is flat, multiple optimal windows are derived from a piecewise linear segment (regression) model of the image-to-real world conversion function. The optimal window is used as a fixed region template to estimate the instantaneous velocity of the drone. The multiple velocities obtained from multiple optimal windows are integrated by a hybrid fusion rule: a weighted average for lateral (sideways) velocities, and a winner-take-all decision for longitudinal velocities. In the experiments, a drone performed a total of six medium-range (800 m to 2 km round trip) and high-speed (up to 14 m/s) maneuvering flights in rural and urban areas. The flight maneuvers include forward-backward, zigzags, and banked turns. Performance was evaluated by root mean squared error (RMSE) and drift error of the GNSS-derived ground-truth trajectories and rigid-body rotated vision-only trajectories. Four fusion rules (simple average, weighted average, winner-take-all, hybrid fusion) were evaluated, and the hybrid fusion rule performed the best. The proposed video stream-based method has been shown to achieve flight errors ranging from a few meters to tens of meters, which corresponds to a few percent of the flight length.

1. Introduction

The applications of unmanned aerial vehicles (UAVs) have expanded dramatically [1,2,3]. It is crucial to determine drones’ position to perform high-level tasks. Drones typically estimate their position by integrating external sensors, such as global navigation satellite systems (GNSSs) and internal sensors, such as inertial measurement units (IMUs) [4,5]. However, GNSS signals can be absent, blocked, or intentionally jammed. In these GNSS-denied environments, accurate localization is very challenging [5]. Without absolute position reference, drones must rely on alternative sensing methods to infer their positions, often requiring additional onboard sensing or external infrastructure [6,7]. One prominent solution to this problem is visual-inertial odometry (VIO), which fuses data from cameras and IMUs [8,9,10].

One vision-based approach is the template-based method, which determines the drone’s movements by matching the current frame with a template from a previous frame or reference frame. Classical template matching methods are suitable for estimating small translational displacements between frames when lighting is stable and there are no rotational or scale changes [11]. Since template selection is typically scene- or object-dependent, incorrect template updates can lead to accumulated errors [12]. A robust illumination-invariant localization method was proposed for UAV navigation [13]. In [14], a multi-region scene matching-based method is proposed for automated navigation of UAV. Fourier-based image phase correlation was adopted to estimate absolute velocity estimation of UAVs [15]. However, [13,14] utilize pre-stored reference images, and [15] fuses data from onboard sensors. Various matching techniques are surveyed for UAV navigation in [16].

This paper addresses database-free vision-only localization of drones for medium-range (800 m to 2 km round trip) and high-speed (up to 14 m/s) maneuvering flights. The database-free video stream-only localization method proposed in this paper offers the advantages of low cost, light weight, and energy efficiency without using multiple sensor devices and infrastructure. Furthermore, it does not require the collection or updating of maps or reference data. When the camera is pointed at a flat surface, pixel coordinates can be converted to unique real-world coordinates using ray optics [17,18,19]. However, this conversion produces non-uniform spacing in the real-world coordinate system. The optimal windows were proposed to overcome this non-uniform spacing [19]. A piecewise linear segment (regression) model [20,21] determines the vertical side of the optimal window. This model minimizes total least-squares error over all linear segments that approximate the conversion function. As a result, a frame is divided into several non-overlapping windows, which function as fixed area templates. The optimal window template matching is performed between consecutive frames to minimize the normalized sum of squared differences (NSSD) [22], which estimates the instantaneous velocity. In [19], this technique combined with a state estimator achieves errors of several meters on short-range (about 150 m) flights.

This approach is scene- or object-independent and requires no template updates, as the optimal window is a fixed region spanning the entire frame. Furthermore, this fixed-region template matching method searches for only a small, consistent region, rather than the entire image, providing a simple and fast hardware solution.

In the paper, the optimal window template matching technique is significantly improved through velocity fusion. A hybrid fusion rule is proposed considering minimum detectable velocity (MDV) as the velocity resolution. The weights used in hybrid fusion are based on the improved conversion function. This hybrid fusion rule applies a weighted average to lateral velocity and a winner-take-all method to longitudinal velocity. In addition, a zero-order hold scheme is applied to reduce computational load.

In the experiments, a multirotor drone (DJI Mavic 2 Enterprise Advanced [23]) performed medium-range flights including high-speed maneuvers in rural and urban areas. The rural paths have flat and simple terrain, while the urban paths have more dynamic and complex terrain. In rural areas, four paths consist of a straight-forward-backward flight, a zigzag forward-backward flight (snake path), a squared path with three banked turns, and a free flight including both banked turns and zigzags. In urban areas, two paths are a straight outbound flight and a straight-forward-backward flight.

For performance evaluation, GNSS-derived trajectories are used as ground-truths. The GNSS trajectories are projected into a local North-East-Down (NED) Cartesian frame using the WGS-84 ellipsoid model for spatial consistency [24,25]. To compensate for the initial orientation offset between the vision-only trajectory and the global reference, a rigid-body rotation is applied to align the vision-only trajectory with the GNSS-derived ground-truth trajectory. Root mean square error (RMSE) and drift error are calculated at three intermediate points and one final point. Four fusion rules (simple average, weighted average, winner-take-all, and hybrid fusion) are evaluated, with the hybrid fusion rule demonstrating the best performance. It will be shown that the proposed method achieves errors ranging from a few meters to tens of meters. Errors mainly come from the constraint of the flat ground assumption. They also originate from the drone’s mechanical system, which is thoroughly discussed in the discussion section. Figure 1 shows a block diagram of the proposed method. In the block diagram, ellipses indicate signals/data; rectangles indicate processing blocks; arrows indicate direction of signal flow. The block diagram distinguishes two primary processes by separate encompassing boxes: localization and performance evaluation.

The contributions of this study are as follows: (1) The process of converting images to real-world coordinates has been improved, enabling more precise velocity resolution analysis. (2) Multiple velocities observed through multiple window matching were fused based on velocity resolution. A hybrid fusion rule was proposed that applies a weighted average to sideways velocities and a winner-take-all decision to longitudinal velocities. (3) The robustness of the system was verified through high-speed maneuvering flights. Furthermore, the estimated trajectory was verified with the ground-truth derived from GNSS signals.

The rest of this paper is organized as follows: the optimal window template matching is presented in Section 2. The velocity fusion rules and performance evaluation are described in Section 3. Section 4 demonstrates experimental results. Discussion and Conclusions follow in Section 5 and Section 6, respectively.

2. Optimal Windows for Template Matching

This section describes how the optimal windows are derived from the improved image-to-position conversion function.

2.1. Improved Image-to-Position Conversion

The image-to-position conversion [18,19] applies trigonometry to compute real-world coordinates from pixel coordinates when the camera’s angular field of view (AFOV), elevation, and tilt angle are known. It is assumed that the ground is flat and the camera rotates only around the pitch axis. The improved conversion function is as follows:

$$x_i\left(j\right)=\surd (h^2+y_j^2)·\mathrm{t}\mathrm{a}\mathrm{n}\left[\left(i-{\displaystyle \frac{W}{2}}+1\right){\displaystyle \frac{a_x}{W}}\right], i=0,\dots ,W-1,$$

(1)

$$y_j=h ·\mathrm{t}\mathrm{a}\mathrm{n}\left[{\theta }_T+\left({\displaystyle \frac{H}{2}}-j\right){\displaystyle \frac{a_y}{H}}\right] , j=0,\dots ,H-1,$$

(2)

where W and H are the image sizes in the horizontal and vertical directions, respectively, h is the altitude of the drone or the elevation of the camera, $a_x$ and $a_y$ are the AFOV in the horizontal and vertical directions, respectively, and ${\theta }_T$ is the tilt angle of the camera. It is noted that the horizontal and vertical directions in the image are the same with the lateral (sideways) and longitudinal directions of the drone body frame, respectively. Figure 2a,b visualize the coordinate conversion function in the horizontal and vertical directions, respectively: W and H are set to 3840 and 2160 pixels, respectively; $a_x$ and $a_y$ are set to 68° and 42°, respectively; h is set to 40 m; ${\theta }_T$ is set to 60°. In Figure 2a, eight nonlinear conversion functions are plotted varying the vertical index j from 1 to 2101 by 300-pixel intervals. In [18,19], $\surd (h^2+y_j^2)$ was approximated by $\surd (h^2+y_{H/2}^2)$; thus, the horizontal conversion function was independent of the column index of the image. As shown in Figure 2b, the vertical direction conversion function shows a rapidly decreasing nonlinearity, causing non-uniform spacing in the real coordinate system. Due to this nonlinearity, pixel-domain displacement cannot accurately measure real-world displacement.

2.2. Optimal Windows

The optimal window aims to ensure uniform spacing in the real-world coordinates. A piecewise linear segment model is adopted to approximate the nonlinear conversion function as multiple line segments. Therefore, the discrete pixel domain is partitioned into multiple intervals, and the breakpoints between each interval are determined to minimize total least-squares error (total sum of squared errors) as follows:

$${\widehat{s}}_1,\dots ,{\widehat{s}}_{N-1}={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}_{s_1,\dots ,s_{N-1}}\left[\sum _{n=0}^{N-1}\sum _{j=s_n}^{s_{n+1}-1}{{\mathrm{m}\mathrm{i}\mathrm{n}}_{a_n,b_n}\left[y_j-\left(a_{n}j+b_n\right)\right]}^2\right],$$

(3)

where $s_1,\dots ,s_{N-1}$ are $N-1$ break points for $N$ segments, and $s_0$ and $s_N$ are equal to 0 and the image size in the vertical direction, respectively, and $a_n$ and $b_n$ are the coefficients of the n-th linear regression line [20,21]. It is noted that $N$ is equal to the number of windows. The length of each interval is equal to the height of the corresponding optimal window. The number of windows can be determined empirically. Too many windows (i.e., too few sampling points (pixels) per window) can result in inaccurate matching, while too few windows cannot compensate for the non-uniform spacing of the nonlinear conversion function. In the experiments, the frame was cropped by 90 pixels near the edges to remove distortions that might occur during image capture; thus, optimal windows tiled in an area of 3660 × 1980 pixels. The size of the n-th window is 3660 × $\left(s_n-s_{n-1}\right)$ pixels. Figure 3 shows three piecewise linear segment models when $N$ = 10, 12, 15. Figure 4 shows the corresponding optimal windows to Figure 3 in a sample frame. Total sum of squared errors which are minimized according to Equation (3) are 55.30, 26.85, and 11.11. The experimental results will be presented along with the 12 optimal windows that produced the best results in terms of the number of windows.

The computational complexity of exhaustively searching all possible piecewise linear models grows exponentially with the number of segments. Dynamic programming [26,27] efficiently determines the optimal linear segments while significantly reducing computational burden. This approach first precomputes the least-squares fitting error for all candidate line segments. It then determines the optimal n segment solution when n = 2. During this step, all possible least partial sum of squared errors for two segments are calculated at $j\in \left[\mathrm{0,2}\right],\left[\mathrm{0,3}\right],\dots ,\left[0,H-1\right]$. Then n is increased by one, and the optimal solution for n segments is calculated using the least partial sum of squared errors obtained from the previous step. This process continues until n reaches $N$.

Each window region in the image serves as a fixed-region template to generate pixel displacement between consecutive frames. The velocities in lateral and longitudinal directions of the drone are obtained by minimizing the NSSD between frames [19,22]. The MDV at pixel (i,j) is calculated as ${\mathsf{\delta }}_{xi}(j)={\displaystyle \frac{{|x}_{i+1}\left(j\right)-x_i\left(j\right)|}{T}}$ and ${\mathsf{\delta }}_{yj}={\displaystyle \frac{{|y}_{j+1}-y_j|}{T}}$ in lateral and longitudinal directions, respectively, where T is the frame period. Figure 5a,b show horizontal and vertical MDV components, respectively, when T =1/30 sec and $N=12$. The j values in Figure 5a are the center of 12 optimal windows in the vertical direction. As shown in Figure 5b, the MDV in the vertical direction decreases sharply.

3. Velocity Fusion and Performance Evaluation

This section describes how to fuse the velocities of multiple optimal windows, and a zero-order hold scheme to reduce the matching counts. Performance evaluation using GNSS trajectories is also presented at the end of the section.

3.1. Hybrid Fusion Rule

It is assumed that MDV is equal to the velocity resolution. The lateral MDV of the optimal window is set to the MDV of the window center, while the longitudinal MDV of the optimal window remains constant for each window because the vertical conversion function is linearized by the piecewise linear segment model. The weight on each window is calculated based on its resolution as

$$w_{nx}={\displaystyle \frac{\frac{1}{{\sigma }_{nx}^2}}{{\sum }_{n=1}^N\frac{1}{{\sigma }_{nx}^2}}}={\displaystyle \frac{\frac{1}{{\Delta }_{nx}^2}}{{\sum }_{n=1}^N\frac{1}{{\Delta }_{nx}^2}}}\approx {\displaystyle \frac{\frac{1}{{\delta }_{nx}^2}}{{\sum }_{n=1}^N\frac{1}{{\delta }_{nx}^2}}},$$

(4)

$$w_{ny}={\displaystyle \frac{\frac{1}{{\sigma }_{ny}^2}}{{\sum }_{n=1}^N\frac{1}{{\sigma }_{ny}^2}}}={\displaystyle \frac{\frac{1}{{\Delta }_{ny}^2}}{{\sum }_{n=1}^N\frac{1}{{\Delta }_{ny}^2}}}\approx {\displaystyle \frac{\frac{1}{{\delta }_{ny}^2}}{{\sum }_{n=1}^N\frac{1}{{\delta }_{ny}^2}}},$$

(5)

where ${\sigma }_{nx}^2$ and ${\sigma }_{ny}^2$ are the variance of velocity of the n-th optimal window in the horizontal (lateral) and vertical (longitudinal) directions, respectively, ${\Delta }_{nx}$ and ${\Delta }_{ny}$ are the velocity resolution of the n-th optimal window in the horizontal and vertical directions, respectively, and ${\delta }_{nx}$ and ${\delta }_{ny}$ are the MDV of the n-th optimal window in the horizontal and vertical directions, respectively. Assuming the quantization error is uniformly distributed, the variance is proportional to the resolution squared as ${\sigma }_{nx}^2={\displaystyle \frac{{\Delta }_{nx}^2}{12}}$ and ${\sigma }_{ny}^2={\displaystyle \frac{{\Delta }_{ny}^2}{12}}.$

The lateral (sideways) velocities are fused by a weighted average as follows

$$v_x\left(k\right)={\sum }_{n=1}^N{w}_{nx}{v}_{nx}\left(k\right),$$

(6)

where $v_{nx}\left(k\right)$ is the lateral velocity obtained from the n-th optimal window template matching between k and $k-1$ frames. Equation (6) is the weighted least-squares estimator [28,29]. As shown in Figure 5b, the resolution of longitudinal velocity varies significantly. Therefore, longitudinal velocity fusion adopts a winner-takes-all strategy.

$$v_y\left(k\right)={\sum }_{n=1}^N{w}_{ny}^{*}{v}_{ny}\left(k\right)=v_{Ny}\left(k\right),$$

(7)

$$w_{ny}^{*}=1\left({n=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{i=1,\dots ,N}{w}_{iy}\right),$$

(8)

where $v_{ny}\left(k\right)$ is the longitudinal velocity obtained from the n-th optimal window template matching between k and $k-1$ frames, and $1\left(·\right)$ denotes the indicator function, which returns 1 if the condition inside the parentheses is true and 0 otherwise. This winner-take-all decision rule is meaningful when the best resolution is significantly better than others [30]. Figure 6 shows the lateral and longitudinal velocity weights for 12 optimal windows. The last longitudinal weight is almost 40% of the total. Therefore, this weight was selected as a winner, and the remaining weights were excluded from the hybrid fusion.

3.2. Vision-Only Traejctory

The vision-only trajectory is generated as

$${\mathit{x}}_v\left(k+1\right)={\mathit{x}}_v\left(k\right)+\mathit{v}\left(k\right)T=\left[\begin{array}{c}{x}_v\left(k\right)+v_x\left(k\right)·T\\ y_v\left(k\right)+v_y\left(k\right)·T\end{array}\right], k=1,\dots ,l-1$$

(9)

where ${\mathit{x}}_v\left(k\right)={\left[x_v\left(k\right) y_v\left(k\right)\right]}^t$, $\mathit{v}\left(k\right)={\left[v_x\left(k\right) v_y\left(k\right)\right]}^t$, t denotes matrix transpose, $l$ is the trajectory length, and the initial position y is set to the zero vector (origin). To reduce the computational cost of template matching, a zero-order hold scheme can be applied. This scheme maintains the velocity constant across multiple frames; thus, the fused velocity $\mathit{v}\left(k\right)$ is replaced by ${\mathit{v}}_{zoh}\left(k\right)$ as follows

$${\mathit{v}}_{zoh}\left(k\right)=\left\{\begin{array}{c}\mathit{v}\left(1\right), 1\le k<M-1\\ \mathit{v}\left(M\right), M\le k<2M-1\\ .\\ .\\ .\end{array}\right.$$

(10)

where $M$ is the number of frames between consecutive template-matching operations; thus, the frame-matching speed becomes frame rate (frame capture speed) divided by $M$.

3.3. Perofrmance Evaluation

Since the vision-only method yields a different orientation from the ground-truth trajectory derived from the GNSS signals, a rigid-body rotation is applied to the vision-only trajectory as

$${\mathit{x}}_g\left(k\right)=R\left({\theta }_l\right){\mathit{x}}_v\left(k\right)+{\mathit{ϵ}}_k, k=1,\dots ,l,$$

(11)

$$R\left(\theta \right)=\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right],$$

(12)

where ${\mathit{x}}_g\left(k\right)$ is the ground-truth trajectory at frame k, the initial position is also set to the origin, ${\theta }_l$ is the counterclockwise angle from the vision-only trajectory to the ground-truth trajectory when the frame number of the trajecoty is l, and $\mathit{ϵ}\left(k\right)$ is the 2D noise vector generating the discrepancy between the ground-truth and the rotated vision-only trajectories. The optimal rotation angle that aligns two trajectories in the least-square sense is obtained as [31,32]

$${\widehat{\theta }}_l={argmin}_{{\theta }_l}\sum _{k=1}^l{‖{\mathit{x}}_g\left(k\right)-R\left({\theta }_l\right){\mathit{x}}_v\left(k\right)‖}^2 ,$$

(13)

$$R\left({\widehat{\theta }}_l\right)={VU}^t,$$

(14)

where $UD{V}^t$ is a singular value decomposition of the cross-covariance matrix $X_v{X}_g^t$, $X_v=\left[{\mathit{x}}_v\left(1\right)\dots {\mathit{x}}_v\left(l\right)\right],$ and $X_g=\left[{\mathit{x}}_g\left(1\right)\dots {\mathit{x}}_g\left(l\right)\right].$ It is noted that ${\widehat{\theta }}_l$ varies with the trajectory length l. The RMSE and drift error are defined, respectively, as

$$E_{RMSE}(l)=\sqrt{{\displaystyle \frac{1}{l}}{\sum }_{k=1}^{l }{‖{\mathit{x}}_g\left(k\right)-R\left({\widehat{\theta }}_l\right){\mathit{x}}_v\left(k\right)‖}^2},$$

(15)

$$E_D\left(l\right)=‖{\mathit{x}}_g\left(l\right)-R\left({\widehat{\theta }}_l\right){\mathit{x}}_v\left(l\right)‖.$$

(16)

The RMSE evaluates the overall accuracy of the entire trajectory, whereas the drift error measures the accuracy at a specific point. In the experiments, RMSE and drift error are caluated at three intermediate points and one final point.

4. Results

4.1. Flight Paths

A multi-rotor drone (DJI Mavic 2 Enterprise Advanced) [23] flew along six different paths in rural and urban areas. All videos were captured at 30 frames per second (FPS) with a frame size of 3840 × 2160 pixels. The altitude of the drone was maintained at 40 m, and the camera tilt angle was set to 60°. The AFOV was assumed to be 68° and 42° in the horizontal and vertical directions, respectively.

Figure 7 shows the GNSS trajectories of six flight paths (Paths 1 to 6) obtained from the drone. The GNSS trajectory is generated through the onboard sensor fusion of absolute GNSS and relatively high-frequency IMU data [23]. This process produces a spatially consistent and smooth path. The trajectories are plotted in the latitude (north–south) and longitude (east–west) coordinates. The starting point is marked ‘O’, three intermediate points or waypoints are marked ‘A’, ‘B’, ‘C’, and the final point is marked ‘D’. Figure 7a–d show four flight paths (Paths 1 to 4) in rural areas, while Figure 7e,f show two flight paths (Paths 5 to 6) in urban areas. Rural areas are mostly flat agricultural land. Urban areas, on the other hand, have complex and irregular terrain due to various artificial structures. Figure 7a,b show a forward-backward straight path and a forward-backward zigzag path, respectively. Figure 7c is a squared path with three banked turns. Figure 7d is a free path with multiple banked turns and zigzags. Figure 7e,f are an outbound straight path and a forward-backward straight path, respectively. No flat turn (yaw-only rotation) was included for all paths to observe the errors accurately. In all flights, the drone starts from a stationary hovering state and accelerates to fly at its highest possible speed. For the forward-backward flight in Figure 7a,f, an abrupt joystick reversal causes the drone to decelerate, stop, and move in the opposite direction. Figure 8 shows the corresponding ground-truth trajectories for Paths 1 to 6. The GNSS trajectory is projected into a local Cartesian frame (north–east) using the WGS-84 ellipsoidal model [24,25]. These projected trajectories are assumed to be the ground-truths for the performance evaluation.

Figure 9a–f show video frames when the drone passes Points O, A, B, C, and D in six paths. These frames show only part of the path, but all frames can be seen in the MP4 videos uploaded to the database site.

Table 1. Frame number and path lengths of six paths.

Point	Path 1		Path 2		Path 3		Path 4		Path 5		Path 6
	Frame Number	Path Length (m)	Frame Number	Path Length (m)	Frame Number	Path Length (m)	Frame Number	Path Length (m)	Frame Number	Path Length (m)	Frame Number	Path Length (m)
A	1175	500.5	1063	398.7	678	268.35	866	302.19	476	182.05	1131	346.97
B	2350	1033.1	2126	824.6	1482	611.45	1731	613.65	951	403.22	2261	720.68
C	3686	1539.4	3367	1284.0	2242	891.84	2596	928.49	1426	624.12	3291	1031.44
D	5021	2066.7	4560	1725.6	3013	1225.90	3462	1231.27	1901	810.99	4320	1440.1

shows the frame number and the ground-truth trajectory length of Paths 1 to 6. The shortest length is Path 5, at 810.99 m, and the longest length is Path 1, at 2066.7 m.

4.2. Vision-Only Trajectories

Table 2. RMSE and drift error for hybrid fusion.

Point	Path 1		Path 2		Path 3		Path 4		Path 5		Path 6
	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)
A	2.85	5.45	6.23	5.23	4.07	5.91	3.18	2.45	3.06	7.32	7.96	6.99
B	11.93	26.45	7.68	10.95	11.82	28.19	5.59	5.27	5.67	6.29	8.35	9.42
C	15.7	17.89	7.78	7	19.98	33.62	11.41	32.99	8.67	18.22	10.27	11.69
D	16.44	21.72	9.88	19.45	19.65	9.11	24.89	40.28	11.26	5.92	16.75	42.47
Avg.	11.73	17.88	7.89	10.66	13.88	19.21	11.27	20.25	7.17	9.44	10.83	17.64

shows the RMSE and drift error for six trajectories at Points A, B, C, D for the hybrid fusion. The RMSE reflects overall trajectory accuracy, whereas drift error quantifies the deviation at a specific point. The RMSE and drift error at Points A, B, C, D range from 2.85 m to 24.89 m and from 2.45 m to 42.47 m, respectively. The average RMSE ranges from 7.17 m to 13.88 m, and the average drift error ranges from 9.44 m to 20.25 m. In percentage terms, the RMSE and drift error at Points A, B, C, D range from 0.57% to 2.29% and from 0.55% to 4.61%, respectively. In terms of percentage, the lowest RMSE (0.57%) is obtained both at Point A on Path 1 and Point D on Path 2 while the lowest drift error (0.55%) is obtained at Point C on Path 2. The highest RMSE and drift error are obtained at Point A on Path 6 and Point B on Path 3, respectively.

Table 3. Average RMSE and drift errors for various fusion rules.

Rule	Path 1		Path 2		Path 3		Path 4		Path 5		Path 6
	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)	RMSE (m)	Drift (m)
Simple Avg.	54.97	69.54	19.24	19.32	15.66	20.5	23.16	27.11	5.52	6.77	17.05	31.82
Weighted Avg.	31.71	38.92	14.55	17.14	14.72	19.72	12.87	23.32	13.01	19.88	12.68	24.16
Winner-take-all	12.35	19.55	9.59	13.46	14.09	18.93	18.33	32.64	7.17	9.76	15.93	30.56
Hybrid	11.73	17.88	7.89	10.66	13.88	19.21	11.27	20.25	7.17	9.44	10.83	17.64

shows the RMSE and drift error averaged over four points to compare various fusion rules. Four fusion rules were tested: simple average, weighted average, winner-take-all, and hybrid fusion. The hybrid fusion rule produced the smallest error in all cases except for Path 5, followed by the winner-take-all decision. Simple average produced the largest error, but the lowest error for Path 5.

Figure 10 shows the vision-only trajectory aligned with the ground-truth trajectory. Alignment is achieved through the rigid-body rotation of the vision-only trajectory, as shown in Equations (11)–(14).

Figure 11 shows the ground-truth and the rigid-body rotated vision-only velocities in the GNSS-projected coordinate system. The ground-truth velocity is the average of the forward and backward velocities over one second. In Figure 11a–f, the north–south and east–west velocity components for six paths are shown separately. As shown in Figure 10 and Figure 11, higher errors often appear at higher velocities; errors were the lowest when the drone was moving forward, followed by backward flight and then sideways flight. Several outliers were observed from the vision-only rotated velocity. These isolated outliers can be removed using a statistical tool such as the Hampel filter [33] or by considering the drone’s actual maximum speed, but the performance improvement was minimal.

4.3. Zero-Order Hold Scheme

Figure 12 shows the RMSE and drift error at the final point, as well as the average RMSE and drift error when applying the zero-order hold scheme. The frame matching rate varies from 30 Hz to 1 Hz. A lower frame matching rate reduces the computational burden.

As the frame matching rate decreases, the performance of Paths 1, 2, 5 degrades, but the performance of Paths 3, 4, 6 remains relatively stable.

A total of 12 supplementary files are available online. Six MP4 videos for six flights and their corresponding GNSS files (captions).

5. Discussion

The optimal window aims to ensure a uniform spacing in the real-world coordinate system. The number of optimal windows was heuristically set to 12, and the same number of velocities were obtained through template matching. These velocities were fused with the hybrid fusion rule. Two constraints were imposed on the optimal window derivation. One is the planar ground assumption, and the other is that the camera rotates only in the pitch direction. Localization errors in the same environments primarily originate from the drone’s mechanical system including its camera stabilizer (gimbal), which is optimized for flying forward. The multi-rotor tilts its whole body in the traveling direction to increase its speed [23]. This tilting effect hinders the gimbal to maintain a stable alignment at the intended angle. Experimental results indicate that these errors are most significant during sideways flights due to the gimbal’s physical limitations in the roll direction. Backward flight generates fewer errors than sideways, but more than forward flight. The hardware optimization for forward flight creates directional performance bias. This bias is proportional to speed and generates a persistent lag in the calculated location compared to the ground-truth. Errors caused by this directional bias and the flat ground assumption can be alleviated by a practical post-processing solution studied in [19]. The flat ground assumption may be more appropriate for high-altitude drone localization, a topic that remains for future research.

All experiments are conducted at 40 m altitude, 60° camera tilt, and 30 FPS, and the AFOV is determined approximately. Evaluating robustness to different acquisition settings (altitude/tilt/AFOV/FPS) including maximum permissible speeds is left for future work.

The optimal window templates are non-overlapping and tile the image; thus, each template contains approximately K/N pixels, where K denotes the total image pixel and N the number of templates. Each template restricts its search to a local region, S = (2 × maximum speed / minimum resolution)². The total operations per frame are approximately N∙K/N∙S= K∙S, yielding 3 × 10¹⁰ operations for K = 3840 × 2160 pixels, maximum speed = 15 m/s, and minimum resolution = 0.5 m/s. Approximately, the floating-point operations (FLOPs) for NSSD are a few times the number of pixel-level operations [34]. In consequence, the computational demand reaches 9 TFLOPS at a 30 Hz frame matching rate. Advanced embedded drone platforms can reach up to 10 TFLOPS; thus, a 9 TFLOPS workload is feasible for real-time onboard processing [35]. It is noted that applying a zero-order hold scheme can reduce the computational burden by several tens of times, at the cost of potential performance degradation.

6. Conclusions

This vision-only technique allows drones to estimate their location using only visual sensors. A novel optimal window template-matching technique was significantly improved in the paper. The optimal window is derived from the piecewise linear segment model that linearizes the nonlinear image-to-real world conversion function. The optimal window template is scene-independent and requires no additional processes for updating.

Multiple velocities are fused based on the MDV of the optimal window. The RMSE and drift error were found to range from several meters to a few tens of meters during high-speed maneuvers ranging from about 800 m to a round trip of 2 km.

Although the application of this method may be limited due to constraints on the optimal window, it has great potential in a wide range of applications, from commercial and industrial sectors to military missions. Thermal imaging cameras [36] could be used in environments with low illumination, a possibility also being explored in future research.