Automated Two-Step Seamline Detection for Generating Large-Scale Orthophoto Mosaics from Drone Images

Abstract

To generate an orthophoto mosaic from a collection of aerial images, the original images are first orthorectified individually using a Digital Surface Model (DSM). Then, they are stitched together along some determined seamlines to form the orthophoto mosaic. Determining appropriate seamlines is a critical process, as it affects the visual and geometric quality of the results. The stitching process can usually be done in frame-to-frame or multi-frame modes. Although the latter is more efficient, both still involve a lot of pre-processing, such as creating individual orthophotos, image registration, and overlap extraction. This paper presents a novel coarse-to-fine approach that directly determines the seamline network without such pre-processing. Our method has been specifically applied for UAV photogrammetry projects where, due to the large number of images and the corresponding overlaps, the orthophoto mosaic generation can be very challenging and time-consuming. We established the seamlines simultaneously for all the images through a two-step process. First, a DSM was generated, and a low-resolution grid was overlayed. Then, for each grid point, an optimal image was selected. Then, the grid cells are grouped into polygons based on their corresponding optimal image. Boundaries of these polygons established our seamline network. Thereafter, to generate the orthophoto mosaic, we overlayed a higher/full resolution grid on the top of the DSM, the optimal image of each point of which was quickly identified via our low-resolution polygons. In this approach, not only seamlines were automatically generated, but also were the need for the creation, registration, and overlap extraction of individual orthophotos. Our method was systematically compared with a conventional frame-to-frame (CF) technique from different aspects, including the number of double-mapped areas, discontinuities across the seamlines network, and the amount of processing time. The outcomes revealed a 46% decrease in orthophoto generation time and a notable reduction in the number of double-mapped areas, sawtooth effects, and object discontinuities within the constructed orthophoto mosaic.

1. Introduction

The generation of orthophoto mosaics using drone images is an important application that has attracted a lot of interest in recent years [1,2,3,4,5,6,7]. To create an orthophoto mosaic from a set of images, individual orthophotos are initially produced through orthorectification using direct or indirect methods [8,9,10]. Then, they are registered, and the overlap areas between the neighboring orthophotos are determined [11]. Next, the border lines (so-called seamlines network) at which the individual orthophotos need to be cut and stitched together are detected [12]. The network of seamlines, which is crucial for ensuring the visual and geometric quality of the orthophoto mosaic, needs to be accurately defined using frame-to-frame or multi-frame methods [13,14]. These seamlines guide the stitching of individual orthophotos, culminating in the creation of the final, seamless orthophoto mosaic [15]. However, there are challenges and complications at every step of mosaic generation, particularly in drone datasets.

In most cases, a drone image covers only a small area on the ground. Therefore, thousands or even tens of thousands of images are needed to create the orthophoto of a large area. This means that although we need only a small portion of each image, the complete formation of the orthophoto mosaic requires all of them to be orthorectified [12]. As shown in Figure 1, only a small part of the individual orthophotos 1, 2, 3, 4, 10, 11, 12, and 13 are used to create the orthophoto of the red rectangle. However, all related individual orthophotos should be registered with respect to each other, and seamlines in overlapping areas of individual orthophotos should be determined. As a result, the orthophoto mosaic generation can become time-consuming in processing the data, especially when the number of images is large [16].

At the image registration stage, there is a further challenge. Image registration mainly involves key point extraction, image matching, and solving image-to-image transformations [17]. Image matching methods can include matching errors which affect the continuity of the resulting orthophoto mosaic where individual orthophotos are stitched together [18]. Many studies have been conducted to analyze and compare the performance of image-matching algorithms for image-stitching tasks [19].

Moreover, seamline detection in the image space is another challenging task. Although detecting seamlines is automatic, it often requires post-processing to improve the results. This is because the difference in the perspectives of the images is high at high-rise objects like buildings, trees, cars, and extruded roads and can lead to clear discontinuities in the resulting orthophoto mosaic [20]. This issue is particularly evident with drone imagery since the images are captured from close proximity to the ground. To avoid this problem, seamlines undergo an optimization process to find the best seamlines that pass over the minimum number of high-rise objects possible [21]. To optimize the seamlines, the existing optimization methods can be grouped into frame-to-frame (local) [22,23,24] and multi-frame (global) [25,26,27]. Frame-to-frame methods determine the optimized seamlines image by image, whereas in the multi-frame approaches this is done simultaneously. In both approaches, the main intention is to reduce the processing time and/or the number of high-rise objects passing through the seamlines.

For example, aiming to reduce the computation time, ref. [12] introduced a stitching method for generating orthophoto mosaics from UAV images. Initially, individual ortho images are generated, followed by the creation of initial seamlines using Voronoi diagrams. Subsequently, the watershed segmentation algorithm is applied to the individual orthophotos, and the resulting edges are utilized to optimize seamlines, avoiding buildings. While processing time was reduced compared to Dijkstra’s method, more buildings intersected with the seamlines. As another example [22], they applied their method to two pairs of individual UAV orthophotos. Initially, seamlines were generated in overlapping areas using the Voronoi diagram. Auxiliary data from the DSM helped identify buildings and optimize the initial seamlines. This approach reduced processing time and the number of buildings intersecting with seamlines compared to the Dijkstra’s method. Additionally, a stitching-based method was developed to generate a UAV orthophoto mosaic [16], using both image information and ground data. The result was fewer buildings intersecting with the seamlines compared to OrthoVista 5.6.3 software. They recommended exploring a multi-scale strategy for improved processing efficiency. Furthermore, ref. [28] introduced a super pixel approach to improve calculation time efficiency. They extracted overlapping areas from adjacent orthophotos and determined seamlines using the Dijkstra method. Seamline optimization occurred in low-resolution image space before being refined in the original image space, resulting in reduced processing time compared to their pixel-based method. However, the method proved to be slower compared to directly applying the Dijkstra method.

As can be seen, creating a method for generating orthophoto mosaics through image stitching requires several common steps. These include generating individual orthophotos, accurately registering them, and detecting and optimizing seamlines on the orthophotos. However, the process can be time-consuming [16], especially with drone images covering small ground areas. It can also result in discontinuities at the borders of adjacent orthophotos [22]. Two main reasons account for these discontinuities. Firstly, errors in the image matching method may lead to inaccuracies in tie point matching, causing boundary jumps between neighboring orthophotos. Secondly, seamlines may cross high-rise objects, especially buildings, due to differences in perspective between neighboring images.

We introduced a novel approach [29] that suggested a global DSM-based mosaic generation to increase the speed and reduce the number of discontinuities. Instead of creating individual orthophotos, this method involves generating a comprehensive orthophoto of the entire area using DSM cells. Each DSM cell was projected onto one of the visible images, and the pixel intensity from the corresponding image filled the orthophoto mosaic pixel of that DSM cell. Compared to the existing methods, this approach eliminated the need for matching, image registration, and overlap extraction. This leads to a reduction in discontinuities and the processing time. However, processing all DSM cells still requires a considerable amount of time. Therefore, in this paper, we propose a new method to enhance processing speed without sacrificing orthophoto continuity. We initially employ a low-resolution DSM grid, for each cell of which we identify the closest image (referred to as the optimal image). Subsequently, the grid cells are organized into polygons based on the corresponding optimal images, defining our seamline network. To generate the orthophoto mosaic, we overlay a higher/full-resolution grid onto the DSM, swiftly identifying the optimal image for each orthophoto mosaic pixel using the seamline polygons. Compared to our previous work, we expect this approach to reduce processing time even more compared to the existing ones. In the meantime, similar to [29], it eliminates the need for orthorectification, registration, or overlaps the area extraction of individual orthophotos. Furthermore, due to its pixel-by-pixel formation, we anticipate reduced discontinuities in the orthophoto mosaic.

Overall, the main contributions of this study are: (a) The development of a fast and automatic seamline generation technique within the DSM space. (b) Constructing the large-scale orthophoto mosaic with no need for the generation of individual orthophotos, nor their registration and overlap areas extraction and the rest of this paper is organized as follows. Section 2 reviews the works related to seamline detection methods. In Section 3, details of our method are given. We show how seamlines are automatically generated in the object space and how the final orthophoto mosaic is constructed without needing individual orthophoto creation. Section 4 evaluates the algorithm and compares the results with those obtained using a common image mosaicking method. Finally, this paper concludes the study with a conclusion in Section 5.

2. Related Work

Two types of orthophoto mosaic generation approaches are usually employed to construct an orthophoto mosaic of a set of drone images: frame-to-frame and multi-frame (or so-called local and global). The general workflow of a typical frame-to-frame approach [30] is shown in Figure 2. As can be seen, the orthophotos of the first and second pairs of images are generated. The resulting orthophotos are then registered (a), and the overlap between them is identified (b). Next, the features within this overlap are searched in order to identify the most appropriate pixels for stitching the orthophotos (c). These pixels together establish a seamline segment at which the two orthophotos are stitched (d). As shown in Figure 2, the process is repeated to join the subsequent orthophotos until the complete orthophoto mosaic is constructed. In contrast to the frame-to-frame methods, the multi-frame methods register all individual orthophotos and identify the seamlines in a single phase. These methods, mostly presented in recent years, are more time-efficient than the frame-to-frame methods as they eliminate some of the intermediate steps [27].

As can be seen, the creation of orthophoto mosaics through image stitching, whether employing frame-to-frame or multi-frame methods, involves distinct yet interrelated steps. These steps include producing individual orthophotos, registering them, extracting overlapping areas, identifying seamlines, and ultimately integrating them into a complete orthophoto mosaic. When working with UAV imagery, which typically captures only a small ground area, creating an orthophoto mosaic of a large area demands thousands, if not tens of thousands, of images. Thus, employing existing methods can give rise to the following challenges:

• Pre-processing overhead: both frame-to-frame and multi-frame methods require extensive pre-processing. This consumes substantial time and computational resources, particularly in large-scale drone mapping projects.
• Matching errors and discontinuities: Image stitching processes can suffer from matching errors, leading to inaccuracies in orientation parameters and alignment of neighboring orthophotos. This results in noticeable discontinuities near seamlines, especially around complex features like buildings and roads.
• Viewing angle disparities: variations in viewing angles among UAV-captured images contribute to visible discontinuities along seamlines, particularly around high-rise objects. Detecting and optimizing seamlines to minimize disruptions is challenging and time-consuming, especially in drone-based projects.

The challenges mentioned are considerably amplified compared to scenarios involving aerial imagery. In the following, the literature relating to the above challenges is briefly discussed.

2.1. Pre-Processing Overhead

Unfortunately, it is not easy to compare and assess related works in terms of processing time. This is due to the fact that each study only looked at a certain step in the production of orthophoto mosaic. For example, some studies concentrated on the generation of individual orthophotos [31,32]. Some researchers looked at image registration methods [18,33]. Some just generated and optimized seamlines [24,28]. Furthermore, the datasets used are different in terms of the number of images, orthophoto resolution, Ground Sampling Distance (GSD), processing system specifications, software, and so on. Nevertheless, time savings are addressed at all stages of orthophoto generating. For example, reference [16] highlighted the significant amount of time required to construct seamlines in UAV data and recommended a multiscale or top-down pyramid method for future study. Another example [22] demonstrated significantly reduced processing times for seamline detection compared to Dijkstra’s algorithm. By leveraging DSM, certain pixels are excluded from calculations, increasing speed. Meanwhile, some studies, such as [12], proposed a multi-frame strategy, rather than frame-to-frame, to enhance algorithm efficiency during the seamline determination step. In addition, ref. [28] proposed a super pixel technique to increase processing time efficiency. Compared to their pixel-based approach, the processing time decreased by the use of seamline optimization in low-resolution image space before being improved in the original space.

2.2. Matching Errors and Discontinuities

Another important challenge is the discontinuity/differences of features near the seamlines, resulting from matching errors. Image matching errors lead to inaccurate estimation of orientation parameters, which subsequently results in inaccurate alignment of neighboring orthophotos. Image matching can be performed based on feature extraction methods like Oriented FAST and Rotated BRIEF (ORB), Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), KAZE, Accelerated-KAZE (A-KAZE), and BRISK (Binary Robust Invariant Scalable Keypoints) [33,34]. The effectiveness of feature extraction algorithms for image-stitching tasks is the subject of numerous studies. Researchers have attempted to improve the image-matching accuracy to generate a mosaic with higher precision [11,18,19,21]. For example, after registering two neighboring orthophotos, ref. [21] utilized the NCC criterion to measure the similarity degree between matched tie points in overlapping areas. Subsequently, matched points were filtered based on normalized cross correlation (NCC), where a value closer to zero indicates a lower mismatching error. The mosaic results demonstrate reduced discontinuity in the produced mosaic. To reduce the computation time, ref. [19] applied feature extraction methods like FAST and SIFT to image blocks instead of individual pixels, reducing computation time and discontinuity in the produced mosaics. Ref. [18] employed the Random Sample Consensus (RANSAC) method to delete generated SIFT blunder points in order to improve image registration accuracy and create a seamless mosaic. Also, ref. [11] introduced a two-step method for registering UAV images. Initially, two adjacent images are registered using the SURF method. Subsequently, after extracting the overlapping areas, optimized seamlines are determined. The left image is re-registered to the overlapping areas, followed by repeating the image stitching based on new orientation parameters. This method effectively reduces errors in UAV image registration.

2.3. Viewing Angle Disparities

Typically, each image possesses its unique viewing angle, leading to noticeable discontinuities along the sides of seamlines, particularly evident when passing over buildings, extruded roads, and moving objects [35,36]. To address this problem, researchers have tried to detect optimized seamlines so they do not cross over such objects. For example, ref. [25] registered single orthophotos and established initial seamlines using the multi-frame method employed the Area Voronoi Diagrams algorithm. Subsequently, they optimized the initial seamlines using radiometric information, resulting in a 37% reduction in processing time compared to the frame-to-frame method. Later, ref. [26] highlighted significant discontinuities along seamlines when passing through buildings and roads. Consequently, optimizing seamlines to avoid intersecting buildings and roads became imperative. They introduced a multi-frame method to optimize seamlines, reducing computing time. Their approach involved optimizing the nodes of the seamlines generated by the Voronoi Diagrams algorithm, aiming to minimize intensity differences along the seamlines. Accordingly, ref. [37] employed the traditional frame-to-frame method for mosaicking images, where the images are stitched consecutively, one after another. They have used a DSM to direct a seamline to avoid passing through the buildings. In another study, ref. [27] proposed a semi-automated multi-frame method for guiding seamlines using human–computer interaction. They recommended exploring multi-resolution or super pixel strategies to decrease computational time for future research. Moreover, ref. [38] optimized a frame-to-frame strategy for stitching individual orthoimages. Seamlines are optimized using a combination of marker-based watershed segmentation and Dijkstra’s algorithm. Ref. [23] developed a traditional frame-to-frame algorithm to mosaic UAV images. They used the edge detection method to limit the seamline along the roadways. Ref. [12] suggested a stitching-based method for generating orthophoto mosaics from UAV data. Individual orthophotos are initially produced, followed by initial seamlines generated using the Voronoi diagram technique. Watershed segmentation is then applied to each orthophoto, and the resulting edges are utilized to optimize seamlines while avoiding buildings. Furthermore, ref. [39] employed a histogram equalization strategy to prevent optimal seamlines from crossing over moving objects like cars. Another example [22] applied their mosaic generation approach to two pairs of UAV orthophotos. The Voronoi diagram are employed to create initial seamlines in overlapping regions. DSM helped locate buildings and detect optimized seamlines. In addition, ref. [16], a common orthophoto mosaic generation method, employed extra information from DSM to direct the seamlines so that they do not pass through the buildings. They employed both aerial and drone-based orthophotos to test their method, and recommended using large-scale pixels or a multi-scale strategy to reduce processing time. A traditional frame-to-frame seamline optimization method is proposed by [24]. The seamlines are optimized to pass over fewer buildings and follow the roads. Compared to Ortho Vista software (INPHO, 2005), processing time increased; however, compared to Dijkstra’s algorithm, processing time decreased.

To address the above problems, and to enhance the efficiency of existing methods, in previous research [29], we developed a global method that generates the orthophoto mosaic pixel by pixel. Our method did not rely on image stitching; as such, the matching errors that cause discontinuity issues were kept to a minimum. Furthermore, when seamlines cross over buildings and extruded roads, the discontinuity was minimal since the orthophoto pixels were filled using the best images having the least amount of relief displacement. Although this led to greater continuity in the resulting orthophoto, a substantial number of pixels still need to be processed. Thus, in this study, we provide a novel way to improve the efficiency method developed by [29], while maintaining its benefits. Our approach determines the optimal images at a low-resolution scale, and the seamlines are determined based on the grouped optimal images. Then, the production of orthophoto mosaics at full scale is conducted. As shown later, like our previous method, the results contain fewer double-mapping discontinuity effects. In addition, as no individual orthophotos need to be generated, the image registration and overlap area extraction steps are removed from the process. Thus, the process is more efficient, especially when the number of drone images is large.

3. Method

In this section, the method proposed in this paper is described. Instead of creating individual orthophotos, our method generates the complete orthophoto mosaic covering an entire region, using the DSM of the whole area. Each DSM cell is back-projected onto one of the images (reference image) where it is visible. Subsequently, the pixel intensity from the corresponding image fills the orthophoto mosaic pixel of that DSM cell. Figure 3 illustrates key components of the proposed algorithm, primarily tailored for drone imagery. Our algorithm comprises three core modules: initial processing, low-resolution grid-based seamline detection, and full-resolution orthophoto mosaic generation.

The initial processing stage commences with the input of drone images and their corresponding ground control points, initiating orientation procedures. This phase yields essential internal and external orientation parameters alongside a sparse point cloud, subsequently refined into a dense point cloud and a Digital Surface Model (DSM).

Subsequently, a coarse network of seamlines is established to expedite the determination of reference images for each point in the full-resolution orthophoto mosaic. This is achieved by generating a 3D grid with significantly spaced intervals (10–20 times the original DSM interval) whose points are interpolated from the dense cloud computed earlier. An optimization procedure assigns a reference image to each grid point, and these coarse grid points are clustered to define seamlines based on their associated reference images. The resulting polygon boundaries delineate our seamline network.

Given that the resolution of the orthophoto mosaic matches that of the DSM, the pixels within it can seamlessly inherit values from their respective reference images. Retrieving these reference images for each full-resolution DSM cell is facilitated through a straightforward point-in-polygon method. Here, the polygons are the ones established based on the low-resolution grid, streamlining the process of associating reference images with specific areas of the DSM. In the following, the above steps are discussed in detail.

3.1. Initial Processing

To generate a full orthophoto mosaic, we need the camera orientation parameters, a full-resolution dense cloud, and DSM for the area of interest. This data is all produced at the initial processing stage, using photogrammetric software (in our study, Pix4DMapper 4.5.6). Given the UAV images and the corresponding ground control points, the images are first oriented. The results are the camera orientation parameters and a sparse point cloud, which includes the 3D coordinates of the tie points, from which the full-resolution dense cloud is created. The dense point cloud is then used to construct a DSM with a resolution equal to that of the orthophoto mosaic; thus, points on the final orthophoto mosaic have a one-to-one relation with the DSM cells. Alternatively, the location of its points can be interpolated from the underlying DSM.

3.2. Low-Resolution Grid-Based Seamline Detection

Our seamline detection algorithm is a two-step coarse-to-fine process in which the seamlines are delineated in the object space. As explained above, despite the traditional techniques that work by stitching individual orthophotos, we generate the orthophoto mosaic of the whole area on a pixel-by-pixel basis. For each pixel, we identify a reference (optimal) image which is used to define its value on the final mosaic. This means that when the number of images is large, identifying the reference images of all the mosaic pixels can become a lengthy process. Thus, in this paper, we do the job in two steps. At first, we generate a low-resolution grid, called a coarse grid (in this paper with a 30 cm interval) in the object space. Then, we only identify the reference images of points on this grid. This will greatly increase the speed of reference image selection when pixels of the complete orthophoto mosaic are being processed.

There are several ways to select the reference images which are mostly used for occlusion removal [32,40,41,42,43,44,45,46]. Examples are the image viewing angle [32,47], the distances from the occluded pixel [42], and the distance from the principal or nadir points of the nearby images [48]. In our work, we use a similar approach to determine the optimal images, with the grid-to-camera distance as the deciding factor [49]. The process is simple and fast, and is performed for all grid points of the low-resolution DSM. We developed the below pseudo code (Algorithm 1) for optimal image determination using the grid-to-camera (projection center (PC)) distance principle.

Using (T_X, T_Y, T_Z), the distances between each grid point and the relevant cameras/projection centers are calculated. The image with the shortest distance to the grid point is considered as its corresponding optimal image. Figure 4 shows an example. As can be seen, to find the optimal image for point P, the distances between the point and projection centers of all six cameras are measured (D₁–D₆). As can be seen, the camera PC₅ (green color) has the smallest value (D₅) and is, thus, selected as the optimal image. The process is repeated for all points of the low-resolution grid.

Algorithm 1: Optimal image determination

Inputs:

(a) CGP = (CGPi) 3d Coarse Grid Points (Xi, Yi, Zi)

(b) DI = (DIj)   Drone Images

(c) EO      External Orientations parameters (TX, TY, TZ)

Output: OI = OIi   Optimal Images

for CGPi ϵ CGP do

for DIj ϵ DI do

Cij = compute distance to the projection center ((Xi, Yi, Zi), EOj,)

End

OIi = DIj (Cij == min)

End

Return OI

Secondly, coarse DSM points are grouped based on their optimal image to detect seamlines network. Each optimal image is assigned a color. The result is a colored mosaic image (Figure 5), on which pixels with the same color refer to the same optimal image. At this stage, the 3D coarse grid points can be grouped into polygons based on their optimal images. In such a way that the points with the same optimal image (same color) are placed in the same group. The boundaries of the resulting polygons will form our seamline network, which is required to form the overall orthophoto mosaic. However, despite the most currently available techniques, they are determined directly in the object space, i.e., the low-resolution DSM. We used this set of polygons to generate the full-resolution orthophoto mosaic, as described in the following sub-section.

3.3. Full-Resolution Orthophoto Mosaic Generation

As mentioned before, currently available techniques work by stitching single orthophotos to generate a complete orthophoto mosaic. However, in our technique, we create the orthophoto of the whole area in one go, using the points of the full-resolution DSM. This approach can result in smoother and more seamless results than those created using traditional techniques. We use the low-resolution seamlines formed in the previous step to determine the optimal image corresponding to any point on the full-resolution DSM.

Figure 6 illustrates the overall view of our orthophoto mosaic generation method, which is performed pixel-by-pixel.

Consider point A with (X, Y, Z)^T coordinates as a sample. The first step is to define its corresponding DSM point. This is trivial, as the relation between the DSM and orthophoto mosaic points is one-to-one. Therefore, the main goal now is to define the pixel value of A, which should be derived from its corresponding optimal image. This is where our low-resolution seamlines/polygons are used. With a simple in-polygon check, the optimal image is quickly identified. Then, we back-project the point onto its image to extract its pixel value.

As we know, $\mathrm{A} \to =\left(\mathrm{X},\mathrm{Y}, \mathrm{Z}\right)$^T is a three-dimensional point in the object space whose coordinates ($\overrightarrow{{\mathrm{A}}^{\prime }}=\left({\mathrm{X}}^{\prime },{\mathrm{Y}}^{\prime },{\mathrm{Z}}^{\prime }\right)$) in each of the camera coordinate systems is given by:

$$\overrightarrow{{\mathrm{A}}^{\prime }}={\mathrm{R}}^{\mathrm{T}}\left(\overrightarrow{\mathrm{A}}-\overrightarrow{\mathrm{T}}\right),$$

(1)

where R = R_ω, R_φ, R_κ and T = (T_x, T_y, T_z) are external orientation parameters in the world coordinate system.

Set:

$$\left(\begin{array}{c}{\mathrm{x}}_{\mathrm{h}}\\ {\mathrm{y}}_{\mathrm{h}}\end{array}\right)=\left(\begin{array}{c}\frac{{\mathrm{X}}^{\prime }}{{\mathrm{Z}}^{\prime }}\\ \frac{{\mathrm{Y}}^{\prime }}{{\mathrm{Z}}^{\prime }}\end{array}\right) \mathrm{and} {\mathrm{r}}^2={\mathrm{x}}_{\mathrm{h}}^{2 }+{ \mathrm{y}}_{\mathrm{h}}^{2 }$$

(2)

Using the following formula, we can then identify the image coordinates of the point corresponding to A:

$$\left(\begin{array}{c}{\mathrm{x}}_{\mathrm{oi}}\\ {\mathrm{y}}_{\mathrm{oi}}\end{array}\right)=-\mathrm{f}\left(\begin{array}{c}{\mathrm{x}}_{\mathrm{hd}}\\ {\mathrm{y}}_{\mathrm{hd}}\end{array}\right)+\left(\begin{array}{c}{\mathrm{c}}_{\mathrm{x}}\\ \mathrm{cy}\end{array}\right),$$

(3)

where (f, c_x, c_y) are the internal camera parameters in pixel coordinates, x_hd and y_hd are the corresponding x and y distortion models estimated by:

$$\left(\begin{array}{c}{\mathrm{x}}_{\mathrm{hd}}\\ {\mathrm{y}}_{\mathrm{hd}}\end{array}\right)=\left(\begin{array}{c}\left(1+{\mathrm{K}}_1{\mathrm{r}}^2+{\mathrm{K}}_2{\mathrm{r}}^4+{\mathrm{K}}_3{\mathrm{r}}^6\right){\mathrm{x}}_{\mathrm{h}}+2{\mathrm{P}}_1{\mathrm{x}}_{\mathrm{h}}{\mathrm{y}}_{\mathrm{h}}+{\mathrm{P}}_2\left({\mathrm{r}}^2+2{\left({\mathrm{x}}_{\mathrm{h}}\right)}^2\right)\\ \left(1+{\mathrm{K}}_1{\mathrm{r}}^2+{\mathrm{K}}_2{\mathrm{r}}^4+{\mathrm{K}}_3{\mathrm{r}}^6\right){\mathrm{y}}_{\mathrm{h}}+2{\mathrm{P}}_2{\mathrm{x}}_{\mathrm{h}}{\mathrm{y}}_{\mathrm{h}}+{\mathrm{P}}_1\left({\mathrm{r}}^2+2{\left({\mathrm{y}}_{\mathrm{h}}\right)}^2\right)\end{array}\right),$$

(4)

K₁, K₂, K₃, and P₁, P₂ are the radial and the tangential distortion coefficients, respectively.

The RGB values of the pixel at (x_oi, y_oi) are utilized to fill the orthophoto mosaic pixel of point A (x_om, y_om). The above process is repeated for all 3D full-resolution DSM points until the large-scale complete orthophoto mosaic of the entire area is generated. In the following, the tests carried out to evaluate the effectiveness of the proposed method are discussed.

4. Datasets and Experiments

4.1. Datasets

Two drone datasets were used in this experiment. The first dataset (Figure 7a) was obtained in an area near Qazvin city in Iran, and included a low-level hilly terrain covered with light grass. The second dataset (Figure 7b) corresponds to Golgir, a small city in southern Iran, a relatively flat area with various objects like buildings, roads, and some vegetation.

Table 1. Datasets used for evaluations.

	Dataset 1	Dataset 2
Area Covered (ha)	7.3	17.8
Camera model	DJI FC6310	DJI FC6520
UAV type	Phantom 4 Pro	Inspire 2
Sensor dimensions (mm)	12.833 × 8.556	17.5 × 13.112
Number of images	38	199
Image resolution (pixels)	5472 × 3648	5280 × 3956
Average Ground Sampling Distance (cm)	2.98	2.48
Overlap (%)	Forward overlap: 85 Lateral overlap: 85	Forward overlap: 80 Lateral overlap: 80
Flight height (m)	120	100
Number of ground points	18	6

shows further details about the datasets.

We used MATLAB R2020b “https://www.mathworks.com (accessed on 20 February 2024)” for the implementations and Pix4Dmapper v 4.5.6 photogrammetric software “https://www.pix4d.com (accessed on 20 February 2024)” for the initial computations. All experiments were performed on a computer running a 64-bit Windows 10 operating system, with an Intel(R) Core (TM) i9-9900K CPU at 3.60 GHz, 32GB RAM, 1TB SSD, and 8TB hard drive.

4.2. Experimental Results and Discussion

This section describes the tests carried out to evaluate our method’s effectiveness and compares it with other common techniques. We evaluated our method by studying the double-mapped areas, the discontinuities along the seamlines, and the processing times. Our results are compared with those obtained with our previous work [29]. In addition, our work is compared with a traditional frame-to-frame method [30], which was well-documented and was not difficult to implement. As part of the frame-to-frame method, we had to use a matching algorithm and a mismatch removal technique. In this study, we used SURF for image matching and the RANSAC algorithm to eliminate and optimize mismatching. For both parts, we used the techniques developed by [17]. The large-scale orthophoto mosaic from dataset 1 and dataset 2, generated by our algorithm, is shown on the left side of Figure 8 and Figure 9, respectively. The corresponding seamline networks (the colored mosaic/polygons shown on the right sides of Figure 8 and Figure 9) are also overlayed on the mosaics (the red/yellow dotted lines). In both datasets, all the images are used to create the orthophoto mosaics. To generate seamlines, as mentioned in Section 3.2, an optimal image is first determined for each point of the low-resolution grid. The dimensions of a low-resolution grid are 10 times the original DSM interval (in our experiments 30 cm). Each optimal image is assigned a color. The grid points are grouped based on the optimal image so that the points that have the same optimal image (same color) are placed in the same group. The boundaries of the different groups define the seamlines on the DSM.

In the following section, the tests carried out are described.

4.2.1. Double-Mapped Areas

When generating an orthophoto using a DSM, there would be an occluded area close to the objects with sudden relief change [50,51]. As illustrated in Figure 10, this causes an unpleasant doubling of RGB values in orthorectified images (Point C) [52]. The size of the occluded area grows when point C is reprojected with the second image (PC₂), which is farther away. That means the occluded area can be larger for a DSM point that is farther from the camera. Researchers have offered many approaches for automatically detecting and removing occlusions [31], which are not covered in this study. Nevertheless, we expect our proposed method to result in less occluded areas, using the closest camera to any point to the intensity value of the relevant orthophoto mosaic pixel.

As dataset 1 did not include significant elevation changes, it did not have many occlusions to study in this experiment. So, we concentrated on the second dataset for evaluating the number of double-mapped areas. In the second dataset, there were many areas where the double mapping effect appeared. Some examples are shown in Figure 11, where we compare the results obtained by our method with the frame-to-frame technique (CF). As can be seen in Figure 11b and d, the double-mapped areas in orthophotos produced by the CF method (Figure 11b areas 1, 2, and 3) are larger than those by the proposed method (Figure 11d areas 1, 2, and 3). This follows our assumption that fewer double-mapped areas are produced with our method.

4.2.2. Discontinuity along the Seamlines

Ideally, there should be no discontinuity at the junction of two individual orthophotos. However, this is not possible due to reasons such as perspective differences in the images and errors in matching and positioning [12,16]. The perspective difference of neighboring images is usually larger when a seamline passes over a high-rise object like a building. Figure 12 shows several examples of the seamlines that passed over buildings in the two research areas (1) and (2) using the CF-based method (a) and the proposed method (b). As seen in Figure 12(a1,a2), there are some discrepancies along the seamlines generated using the CF-based method, where they cross buildings. These errors are 1–4 pixels, which are equivalent to 3–12 cm, respectively. In the same areas (1) and (2), the proposed method (Figure 12(b1,b2) kept the differences along the seamlines to a minimum (less than 1 pixel) where they cross the buildings. As can be seen in Figure 12(a3,b3), the proposed method and the CF-based method both performed even when the seamlines crossed over the buildings.

These results suggest that both methods performed well considering the crossing seamlines with the roadways, as shown in Figure 13(a1,b1) for dataset 1, which represented a hilly region. However, differences are seen in CF-based seamlines in dataset 2 (Figure 13(a2,a3)). As we can see in these figures, along the blue lines (Figure 13(a2)) and blue-white curve lines (Figure 13(a3)), the extruded roads, there are discontinuities as big as 2 pixels. Both the factors of image matching errors during the composition of individual orthophotos and reprojection errors during image rectification are involved in creating differences along the seamlines. The image blending technique can also reduce the remaining differences around the peripheries of seamlines [53].

In addition to the large double-mapped areas and discrepancies along the seamlines, the CF-based method produced orthophotos with more sawtooth effects. Some examples are indicated with white arrows in Figure 11(a2,a3) and Figure 12(a1,a2). Here, the sawtooth effect is likely impacted by changes in the reprojection error. The reprojection error of each DSM point during image rectification is affected by the distance to the camera, as well as the accuracy of the images’ external orientation parameters. In fact, when the image utilized in orthophoto generation is closer to the perpendicular direction of the point, the reprojection error is the lowest. This is guaranteed through the optimization method used in the proposed method. Also, the accuracy of the external orientation parameters of UAV images can be very different, since the quality and precision of UAV images may differ for reasons such as being more influenced by environmental factors, sensor fluctuations, and low flight height [54]. As a result, a point’s reprojection error differs depending on the UAV image.

4.2.3. Processing Time

The total processing times of the proposed and CF-based methods for large-scale orthophoto mosaic generation are recorded and listed in

Table 2. The total processing times of the proposed method and CF-based method for large-scale orthophoto mosaic generation.

Method	Dataset 1	Dataset 2
	Time (min)	Time (min)
CF-based	32	93
Research [29]	25	79
Proposed	17	51

. Since the proposed method is developed to improve the efficiency of research [29], the processing times of this research for generating orthophoto mosaics are also included in this table for comparison. The times given for the proposed method are for a 3D coarse grid with a 30 cm × 30 cm dimension and the full orthophoto with a 3 cm × 3 cm resolution. Also, the CF-based orthophotos are produced with a resolution of 3 cm × 3 cm. Orthophoto mosaics by [29] are generated with a resolution of 3 cm × 3 cm as well. The dimensions of the orthophoto mosaic of dataset 1 are 11,530 × 9982, and the second dataset is 9739 × 8436.

As shown in Table 2, compared to the CF-based method and research [29], in the first dataset, the processing times are decreased by 15 min and 8 min, respectively, representing a 46% and 32% reduction. In the second dataset, the processing times are reduced by 42 min and 28 min, respectively, which means approximately 45% and 35% reductions. According to Table 2, in the first dataset (38 images), the average processing time (per image) for the CF-based, method by [29], and the proposed method are 51 s, 39 s, and 27 s, respectively. In the second dataset (199 images), the times are 28, 24, and 15 s, respectively. As can be seen, the method proposed in this paper is faster than the one we developed in [29]. However, it can be noted that the average time does not have a linear relation to concerning the number of images. This is because, although the number of images in the first dataset is less than that of the second dataset, it includes more orthophoto pixels. Thus, its average time per image is higher. Indeed, in our technique, the factors influencing the processing time are mainly the number of images, the dimensions of the orthophoto mosaic, and the overlap between images.

4.3. Discussion

As mentioned above, two drone datasets were used to represent diverse terrains—a hilly area near Qazvin city (Dataset 1) and the relatively flat Golgir city in southern Iran (Dataset 2). The proposed method was evaluated against a common frame-to-frame (CF) method, focusing on double-mapped areas, discontinuities across the seamlines, and processing times. The proposed method effectively reduces double-mapped areas, particularly in areas with significant elevation changes, considering a closer distance to the camera during optimal image determination minimizes occluded areas, leading to superior results compared to the CF-based method. Seamlines generated by the proposed method exhibit fewer discrepancies, especially when crossing high-rise objects like buildings. In comparison, the CF-based method introduces errors along seamlines, with larger discrepancies and sawtooth effects.

The proposed method demonstrates a notable reduction in processing time compared to the CF-based method and Research [29]. In Dataset 1, a 46% and 32% reduction, respectively (15 min and 8 min), and in Dataset 2, a 45% and 35% reduction (42 min and 28 min) highlight the efficiency gains, making it a more time-effective solution. In summary, the proposed method offers a substantial improvement over traditional techniques, effectively addressing issues related to double-mapped areas, seamline discontinuities, and processing time. Its pixel-by-pixel approach and low-resolution grid-based seamline detection contribute to a more efficient and seamless orthophoto mosaic generation process. The experimental analysis validates its superiority, showcasing its potential for various applications in drone-based mapping applications.

5. Conclusions

The generation of large orthophoto mosaics from drone images is an essential task in most photogrammetry projects. For this, the original images are usually orthorectified individually using a DSM and then stitched together along some determined seamlines. Either frame-to-frame or multi-frame joint method can determine these seamlines. Unfortunately, such methods require substantial pre-processing, such as individual orthophoto generation, image registration, and overlap area extraction. In this paper, we proposed a simple yet effective and strong method that requires no additional pre-processing. The developed method focuses on generating a global pixel-by-pixel approach, enhancing the efficiency of existing methods while minimizing the processing time. It leverages optimal image determination at a low-resolution scale, forming seamlines from grouped optimal images and creating full-scale orthophoto mosaics. The proposed algorithm comprises three main stages: initial processing, low-resolution grid-based seamline detection, and full-resolution orthophoto mosaic generation. The initial step involves processing drone images, obtaining orientation parameters, sparse and dense point clouds, and a DSM. The low-resolution grid-based seamline detection involves determining optimal images for DSM points, forming seamline polygons, and establishing a seamline network. This network guides the creation of a complete orthophoto mosaic at full resolution. Optimal images are determined using a grid-to-camera distance principle, which results in reducing double-mapped areas and minimizing occlusion effects. The algorithm categorizes 3D grid points into polygons based on optimal images to form the seamlines network crucial for orthophoto mosaic generation. Unlike traditional methods, which work by stitching individual orthophotos, the proposed method generates the entire orthophoto mosaic in a single process, ensuring smoother, seamless results. The process involves identifying optimal images for full-resolution DSM points, with pixel values extracted through back-projection. This pixel-by-pixel approach enhances efficiency and eliminates the need for image registration and overlap extraction.

According to the assessments, our proposed method produces orthophoto mosaics with far less double mapped areas than Common frame-to-frame (CF)-based methods. Differences along the seamlines are negligible, even where they pass through buildings and extruded roads. Also, the sawtooth effect of the edges is considerably less than those of CF-based orthophoto mosaics. These improvements can enhance the quality of orthophoto mosaics. The proposed method is also more time-efficient than traditional algorithms. However, our methods used grid-to-camera distance to select optimum images for orthophoto mosaic construction. In this regard, we suggest studying other criteria such as viewing angle, distance from the occluded area, image quality, and reprojection error. It is also important to take into account the accuracy of the exterior orientation parameters when selecting the optimal image, which was not addressed in previous studies and is also not included in our research. This is particularly significant in areas that are forested and near water. Considering that exterior orientation parameters of UAV images can have highly varying accuracy due to their poor texture. So, the authors are considering providing a multi-criteria method for generating orthophoto mosaics in such datasets with higher accuracy and quality in future research.

Also, as mentioned in the text, we considered the distance between the low-resolution grid points to be between 10 to 20 times the original DSM interval. Here, we did not suggest a specific number, since our main goal was only to show that the smaller the low-resolution grid is, the faster the computations will be. This issue can also be explored further in future studies.