Next Article in Journal
Road Extraction from VHR Remote-Sensing Imagery via Object Segmentation Constrained by Gabor Features
Previous Article in Journal
Share Our Cultural Heritage (SOCH): Worldwide 3D Heritage Reconstruction and Visualization via Web and Mobile GIS
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts

1
State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
2
Collaborative Innovation Center of Geospatial Technology, Wuhan University, Wuhan 430079, China
3
School of Resource and Environmental Science, Wuhan University, Wuhan 430079, China
*
Authors to whom correspondence should be addressed.
ISPRS Int. J. Geo-Inf. 2018, 7(9), 361; https://doi.org/10.3390/ijgi7090361
Submission received: 13 July 2018 / Revised: 17 August 2018 / Accepted: 31 August 2018 / Published: 31 August 2018

Abstract

:
Image mosaicking is one of the key technologies in data processing in the field of computer vision and digital photogrammetry. For the existing problems of seam, pixel aliasing, and ghosting in mosaic images, this paper proposes and implements an optimal seam-line search method based on graph cuts for unmanned aerial vehicle (UAV) remote sensing image mosaicking. This paper first uses a mature and accurate image matching method to register the pre-mosaicked UAV images, and then it marks the source of each pixel in the overlapped area of adjacent images and calculates the energy value contributed by the marker by using the target energy function of graph cuts constructed in this paper. Finally, the optimal seam-line can be obtained by solving the minimum value of target energy function based on graph cuts. The experimental results show that our method can realize seamless UAV image mosaicking, and the image mosaic area transitions naturally.

1. Introduction

It is of great significance to integrate multiple small images with overlapped scenes into one large scene through image mosaicking technology, which can expand the field of vision and content of the image scene. This technology is widely used in aviation remote sensing, medical imaging diagnostics, military target tracking, disaster emergency management, and virtual reality or augmented reality [1,2,3,4]. Currently, being economical, concise, lightweight, and fast, unmanned aerial vehicles (UAV) equipped with inexpensive cameras have become increasingly popular in the tasks of people’s daily life entertainment and professional departments’ surveying. In particular, virtual earth browsing, small-scale field mapping, forest resources investigation, and urban land planning have all created demand for UAV image mosaic technology. However, there are still many problems in the image mosaicking of UAV images, such as ghosting, stitching dislocation, blurriness, aliasing, and time efficiency [5,6,7,8,9]. How to realize the seamless mosaicking of UAV images has always been an ultimate goal pursued by a large number of researchers. However, the current methods are still not satisfactory, especially in some applications such as real-time UAV image mosaicking. Therefore, with the gradual expansion of the application of UAV photogrammetry, UAV image mosaic technology has become a research hotspot in computer vision and image processing in recent years, and how to achieve seamless stitching of UAV images from different perspectives, different times, and even different cameras has become a major problem in these applications.
The complete UAV image mosaic process mainly consists of three parts. (1) UAV image preprocessing, including noise elimination, and image distortion correction. (2) UAV image registration mainly includes feature extraction, feature matching, and transformation model solving. (3) UAV image fusion, which mainly includes the selection of projection model, seam-line detection, and even light uniformity. The main innovative work done in this paper can be summarized as follows. On the basis of the existing graph cuts algorithm, the concept of global optical flow is introduced to constrain the geometric displacements caused by perspective imaging, geometric distortion, and especially the projection errors caused by buildings, and then the optimal seam-line for image fusion is determined so that there is no problem such as seams, aliasing, and ghosting in the mosaicked UAV images.

2. Related Work

Internationally, image mosaic technology has developed rapidly, and many algorithms and software have emerged. They benefit from a great deal of research advances in computer vision, computer photogrammetry, and computer graphics. Specific to the UAV image mosaic problem studied in this paper, although existing methods can achieve good image mosaic effects under certain conditions, there are still some problems that need to be solved and overcome. Especially for the problems and characteristics of aerial images captured by inexpensive non-measurement cameras and small UAV platforms, it is necessary to study a highly efficient and reliable image mosaic method for UAV remote sensing images. Therefore, in order to solve these problems, this paper mainly focuses on the optimal seam-line search in UAV image mosaicking based on previous research work, in order to eventually give rise to an economic, efficient, and reliable UAV image mosaicking method.
Image preprocessing is a mature technology in the research of image mosaicking. Camera calibration and image sampling are two basic tasks. There are traditional camera calibration methods, active vision camera calibration methods, and camera self-calibration methods for different camera lenses. Moreover, image-sampling methods are classified into nearest neighbor interpolation, bilinear interpolation, bi-cubic convolution, and B-spline interpolation according to different interpolation methods [10,11]. Image registration is a key step in image mosaicking. In fact, image registration is also an independent hot research direction, it can be applied to many fields other than image mosaicking. At present, there are many effective and reliable image matching methods such as point feature matching algorithms, line matching algorithms, and patch matching algorithms: ORB (Oriented FAST and Rotated BRIEF) algorithm, SIFT (Scale Invariant Feature Transform) algorithm, and SURF (Speeded Up Robust Features) algorithm, for example [12,13,14]. Image fusion determines how to produce the final stitched image. Its work mainly includes two parts, one is to determine what kind of fusion surface to use, and the other is to determine whether the pixels of final mosaic images should be taken from the reference image or the pre-mosaicked image, or how to fuse these pixels in some way to achieve the best vison in order to eliminate the presence of seams, aliasing, and ghosting [15,16,17]. This paper mainly studies how to find the best seam-line, which is the key to the visual effect of image fusion. The research in this filed can be summarized into two major categories. One is the mosaic methods based on pixel fusion, they integrate pixels in overlapped areas in to new pixels of mosaicked image through some algorithms (such as pixel averaging method, weighted average method and so on) to eliminate the mosaic lines and illumination errors. However, due to the influence of image matching errors, image interpolation errors and intrinsic factors such as illumination and noise, it is difficult to avoid misplacement, ghosting, and aliasing. The other is the mosaic methods based on seam-line, their solution is different from the mosaic methods based on pixel fusion [18,19,20,21,22]. The mosaic methods based on seam-line only take the pixels of the reference image or pre-mosaicked image in the mosaicked result, that is, the overlapped area is divided into two blocks, one for taking pixels from the reference image and the other for taking pixels from the pre-mosaicked image. In this way, there will be no ghosting and aliasing due to image fusion, and this division is achieved using a seam-line. Therefore, how to divide is a problem that needs to be solved by finding the optimal seam-line. Many researchers have done a lot of research on the search for the best seam-line. Among them, the method based on dynamic programming is a symbolic method. This method is based on the optimization theory and is the best search based on the basic principle of dynamic programming. That is, regardless of the initial state of a process or the last decision, the next state or decision must be optimal. However, this kind of search process based on dynamic programming is very easy to fall into the local optimal solution and cannot obtain the global optimal solution. In addition, there are uncertain factors in the choice of the adjustment coefficient and the neighborhood range. Because using dynamic programming to obtain a local optimal seam-line cannot reach a good result. Many scholars have studied how to use a search method to obtain the global optimal seam-line [20,21,22]. Among them, the optimal seam-line search method based on graph cuts is the most representative. It uses graph cuts to obtain the best mosaic texture on both sides of the seam-line to obtain the optimal seam-line, which can solve the shortcomings of dynamic programming in the image mosaicking (such as no need to specify additional energy end points, etc.). In addition, compared with the annealing algorithm and ant colony algorithm, it has the advantage of fast calculation [20,21,22,23,24,25,26]. In these optimal seam-line search methods based on graph cuts, they can be divided into a color domain-based method, a gradient-based method, and a color domain and gradient domain-based method according to the difference of the energy function smoothing term. These methods use either the color space pixel difference information or gradient information separately for the construction of the graph cuts energy function, either considering the influence of the color domain and the gradient domain at the same time, this can make the energy function have more information. Therefore, when the latter is looking for the seam-line, it not only tends to have less gray information difference, but also hopes to find the seam-line at the edge of the drastic change of the building to be able to cover the later splicing traces. However, in the later feathering process, this will have a fuzzy and degrading effect on the edge of the building. This effect is extremely unfriendly to the mosaicking and application of high-resolution UAV images.
In summary, the optimal seam-line search method based on graph cuts is a global and extremely promising image mosaic method. However, any existing mosaic method based on graph cuts pays little attention to how to use geometric information constraints, and often focuses on more limited pixel level difference information (such as pixel color, gray scale, and gradient) to construct the smoothing term of an energy function, without looking at this issue form a global perspective. This often results in these methods not getting ideal, robust algorithm parameters, and ultimately affects the applicability of these methods. Therefore, the energy function of most current optimal seam-line search methods based on graph cuts only consider the image local information, and finally only the pseudo global solution of the target energy function can be obtained. In this paper, aiming at this problem, we propose to introduce the concept of global optical flow filed based on the existing graph cuts method to constrain the geometrical displacement of pixels due to perspective imaging, geometric distortion, and especially the projection errors caused by buildings, to achieve seamless UAV image mosaicking.

3. Method

The experimental UAV images used in this paper have been pre-processed in advance. In UAV image registration, the image registration method based on SURF is used in this paper, and RANSAC (Random Sample Consensus) algorithm is used to eliminate and optimize mismatching. Furthermore, the coarse-to-fine matching strategy is used in the transformation model solution. That is, all the matching feature points are used to estimate the transformation model parameters of adjacent images, and then the new transformation model parameters are accurately solved based on the elimination of the mismatching feature points. Finallt, the coordinate transformation between different images is implemented so that all the pre-mosaicked images are in a unified coordinate system [12,13,14]. After completing image registration and coordinate conversion, the next step is to determine how to produce the final stitched image. This is the innovative work of this paper. In the following sections, we will introduce in detail on the optimal seam-line search method based on graph cuts proposed by this paper and verify the effectiveness and superiority of our proposed method through experiments.

3.1. Classic Graph Cuts-Based Optimal Seam-Line Search Method from OpenCV Library

If the reference image and the image to be registered have been completely registered, any mosaic method based on graph cuts can achieve a good image mosaic effect. However, due to the influence of external factors, the two images cannot be fully registered. Therefore, it is required to obtain an optimal seam-line to segment the overlapped area, so that the visual difference between the both sides of the optimal seam-line is the smallest, thereby realizing seamless mosaicking of UAV images. Image mosaicking based on seam-lines is one of dominant methods for UAV image mosaicking. It does not need to strictly align the entire overlapped area, but only processes the area near the seam-line, and it can splice large parallax scenes. In the famous open-source computer vision algorithm library (OpenCV), the graph cuts-based seam-line search methods are mainly divided into two types. One is to use the difference information of color space pixels for the construction of graph cuts energy function, and the other is based on the previous method, the constraint of gradient information is added. From the review and discussion of the previous literature, it can be seen that the mosaic results of any method are not satisfactory, and these methods have room for improvement in the algorithmic effectiveness and efficiency. Since the OpenCV open-source algorithm library is a very versatile and representative of the latest image processing integrated function library. Therefore, this paper will use the two representative graph cuts-based optimal seam-line search method in OpenCV as control experimental methods to evaluate the effectiveness and efficiency of the corresponding method proposed in this paper. Equations (1) and (2) are two typical construction strategies for smoothing terms of graph cuts energy function (1) only considers the gray value of the image pixel, and (2) added the gradient constraint of the image pixel.
E s m o o t h   = p , q V p , q ( f p , f q )
E s m o o t h   = p , q V p , q ( f p , f q ) + p , q g r a d p , q ( f p , f q )
In (1) and (2), p and q are adjacent pixels in the overlapped area, V(*) is the gray-scale difference function produced by the image pixel difference, grad(*) is the gradient change function generated by the pixel gradient, and f is the image. When the gradient value is not considered, the post-processing of the seam-line can easily make the edge of the ground building blurry. After considering the gradient value, the influence of the illumination change can be effectively avoided.

3.2. Improved Optimal Seam-Line Search Method Based on Graph Cuts

In general, adjacent UAV images will have the same local information, such as the same style buildings, but the scale information of adjacent images may not be exactly the same. At the same time, there will be differences in the perspective of different UAV images. If you do not consider the above situation in the image information, it is very likely that there will be geometrical misalignment or building information missing due to occlusion in the stitched images. Since many mosaic methods still rely on local information to solve the pseudo global solution of target energy function, the inaccuracy of the image splicing problem itself in the mathematical modeling process is neglected, and this leads to the target energy function created cannot fully express the problem that needs to be solved. Therefore, on the basic of the existing methods, this paper designs a more complete UAV image mosaic method with more global constraints. The new method introduces the concept of global optical flow field based on the existing graph cuts method to constrain the geometrical displacement of pixels due to perspective imaging, geometric distortion, and especially the projection errors caused by buildings to achieve seamless UAV image stitching.

3.2.1. Optical Flow Field

Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and scene [27,28]. Optical flow fields are dense correspondence fields for accurate displacement optical flow estimation, they contain rich three-dimensional structural information of feature objects in images. This information of pixels in the overlapped area of adjacent images can effectively provide the structural information of images, which is conducive to search for the optimal seam-line [19]. In this paper, in order to take into account the image structure information better in the overlapped area of the adjacent images, we use the optical flow value of pixels in the overlapped area as a constraint condition for constructing the energy function based on graph cuts. The Horn–Schunck method of estimating optical flow is a global method [27,28]. This paper calculates the optical flow fields of experimental UAV remote sensing images based on this method. Equation (3) is the function of the Horn–Schunck algorithm, it is given for two-dimensional image streams.
F = [ ( I x u + I y v + I t )   2 + a 2 ( | | u | | 2 + | | v | | 2 ) ] d x d y
In (3), F is the value of optical flow, Ix, Iy, and It are the derivatives of image intensity values along x, y, and time dimensions respectively; u, v are the displacements along x, y axis; and а is a regularization constant. Figure 1 is a schematic of the calculated optical flow field.

3.2.2. Graph Construction

A graph is a directed structure with weights, which is usually represented by some nodes and directed edges. In this paper, these nodes are image pixels, and a graph usually has some special nodes called endpoints. In computer vision, these endpoints are used for pixel segmentation labels. As shown in Figure 2a, it is a directed graph structure with two endpoints, one of these is usually called a source node (s) and the other is called a sink node (t). The others are called nodes in the graph, and there is a weighted edge with direction between the terminal node and each node. In addition, there are also weighted edges between each node. Figure 2b shows an example of segmentation of a 3 × 3 directed graph structure. The red dashed line is a cut of the graph. The left pixels of the cut line are labeled as originating from the source node, and the right pixels of the cut line are labeled as originating from the sink node. In the graph, a and b are two common nodes. The cut cost of different segmentation methods will be calculated by the weighted edge. In image processing applications, many of these energy minimization problems can be approximated by solving a max-flow and min-cut problem in a graph.
Specific to this paper, graph cuts-based optimal seam-line search is based on the graph structure. Therefore, it is necessary to convert the image pixels to the desired graph structure before searching for the seam-line. According to our UAV image mosaic theory and the definition of the graph above, the adjacent UAV images and graph structure can be illustrated in Figure 3. Figure 3a is a schematic diagram of UAV image mosaic, in which the pre-mosaicked image has a certain overlapped area with the reference image after the coordinate transformation, and the seam-line will be positioned in the overlapped area. The red line in Figure 3a is an example seam-line. According to this layout in Figure 3a, this paper constructs a graph structure and a part of the graph is shown in Figure 3b. Assuming that there are only 12 pixels in the overlapped area between the reference image and the pre-mosaicked image, and each pixel in the overlapped area is regarded as a node in the graph structure. Then, we can use the 12 circles from 1 to 12 in Figure 3b to represent 12 nodes in the graph, and the two adjacent nodes forming an edge which have weight, as shown in the green lines in Figure 3b. There are also weighted edges between the nodes in the non-overlapped area (that is, box portion of the reference image and the pre-mosaicked image) and the nodes in the overlapped area, as shown by the black lines in Figure 3b. The concept of the optimal seam-line detection is to search for a seam-line in the overlapped of adjacent images to divide the overlapped area into two parts, one part from the reference image, the other is from the pre-mosaicked image. This seam-line will pass through nodes in the overlapped area and directly cut off the weighted edges between the nodes. This is the concept of graph cuts in this paper. In Figure 3b, the red dash line is an example best seam-line. It divides nodes in the overlapped area into two parts. In the mosaicked image, pixels of 1, 5, 6, 9, 10, and 11 come from the pre-mosaicked image, and other pixels come from the reference image. In this way, the source of two partial pixels in the overlapped area are determined, and the purpose of seamless image mosaicking is achieved.

3.2.3. Energy Function Definition

According to the above, the overlapped area of two large adjacent images can construct a large graph structure. In the graph structure, the most critical part is the definition of the weighted edges, and the concrete form of energy function is determined by the definition of the weighted edges. The general energy function of graph cuts consists of data item and smooth item, which correspond to the weights of the edges in the cutting of graph cuts. Equation (4) is the general form of the graph cuts energy function definition, where f is the label assigned by the node, that is, the cost of a node from the source or sink is labeled as f.
E ( f ) = E d a t a   ( f ) + E s m o o t h ( f )
Among (4), Edata is a data item, which is used to express the difference between the value of the node and the value of the label f. Esmooth is a smoothing term, it is used to measure the segment smoothness. Data item generally use a distance measurement method to determine the size of the difference, while the smoothing item calculation is more complex. Because the smoothing item must not only tolerate certain gradual transitions and noise disturbances, but also must be detected in boundaries or mutations, the property is called discontinuity preserving. In [29], a Bayesian first-order partial derivative Markov random field is used to express the energy function of Equation (4) in Equation (5).
E ( f ) = p O   D p ( f p ) + { p , q } N V p , q ( f p , f q )
Among (5), N represents adjacent pixels, O is all pixels in the overlapped area. fp is a label assigned to pixel p. Therefore, the Equation (5) is the cost of the numerical difference caused by allocating labels for all the pixels in the overlapped area and the cutting for adjacent pixels near the seam-line. In [29], the data item is defined as non-negative by Boykov. At the same time, in order to make the smoothing item have the property of discontinuity preserving, Boykov specifies that the value of the smoothing item must be a metric or a semimetric.
So far, the energy function model proposed is clear. Equation (6) is the energy function definition based on graph cuts proposed in this paper. The biggest difference compared to the other energy functions described above is that the definition of our energy function is added to the global optical flow field. In this way, based on the existing graph cuts algorithm, the geometric information constraint can be better considered. The geometric displacement of the pixel caused by the perspective imaging and geometric distortion in the image mosaic can be solved.
E = p O   g r a d ( f 1 ( p ) , f 2 ( p ) ) + p O { V [ f 1 ( p ) , f 2 ( p ) ] } + p O f l o w ( p ) + p O N ( p )
where V(*) is the grayscale difference function generated by the pixel difference, grad(*) is the gradient change function generated by the gradient difference, flow(*) is the optical flow field function of the overlapped area between the adjacent UAV images, N(p) is the energy value of the invalid area (the invalid area refers to the non-pixel area in the minimum enclosing rectangle of the overlapped area), and its value is generally much large than the value of the overlapped area. In this paper, the value of N(p) is constant. f1 and f2 are two pre-mosaicked images. O is the overlapped area. The pixel grayscale difference function not only needs to reflect the difference information of pixels in the same position of adjacent images, but also needs to introduce local information to make it have locality information to enhance the robustness of the algorithm. Combined with the particularity of this problem, it is not appropriate to add individual difference data items to each node of a graph. At the same time, since the essence of searching the seam-line is the difference information, not the direct probability problem of a certain label. Therefore, all the energy terms of the algorithm in this paper are reflected in the weight of the neighboring edge in the graph. However, the weight function of the node and the endpoint (s or t) is a binary function, and its value range is {0,constant1}. The value of Constant1 is that much larger than the adjacent edge weight. Therefore, the functions constructed in this paper need to consider the information the nodes at both ends of the edge. The grayscale difference function of the adjacent pixels (p, p + 1) can be expressed as the form of Equation (7).
V ( f 1 ( p ) , f 2 ( p ) ) = | | f 1 ( p ) f 2 ( p ) | | + | | f 1 ( p + 1 ) f 2 ( p + 1 ) | |  
If the image to be mosaicked is an RGB image, the three-dimensional vector is used for calculation, and | | * | | is the Euclidean distance of one RGB color value of the pixel. Similarly, the gradient change function can be expressed as the form of Equation (8).
g r a d ( f 1 ( p ) , f 2 ( p ) ) = d l ( f 1 ( p ) ) + d l ( f 1 ( p + 1 ) ) + d l ( f 2 ( p ) ) + d l ( f 2 ( p + 1 ) )  
where dl is a gradient operator in a certain direction and can be calculated using four-neighbor or eight-neighbor. Because of the small difference, this paper uses a four-neighbor gradient operator for computational efficiency.
Equation (9) is the optical flow field function herein.
f l o w ( p ) = F ( p ) + F ( p + 1 )  
F is the optical flow calculation function, that is, the displacement field obtained by the difference after the sparse matching of the adjacent images, and the value records the sum of the squares of the pixel displacements.

3.2.4. Solving Minimum Energy Value

Through the definition of the energy function based on graph cuts above, a realistic image seam-line search problem is expressed by a mathematical model, and the minimization process of the energy function value is the search process of the optimal seam-line. In the specific solution process, we use the maximum flow minimum cut algorithm to solve the minimization problem of the energy function. Among them, α-β swap and α-expansion are two good practical solution strategies. In this paper, the α-β swap optimization strategy is used to solve the energy function. It is an effective graph cuts binary optimization algorithm, which can not only optimize the segmentation of the initial labeled data set, but also transform the multi-dimensional directed graph into two-dimensional simple directed graph, Thereby, the limited uncertainty of the multi-dimensional directed graph on the capacity values connecting the different node edges is avoided [29,30].
Figure 4 shows the construction way of α-β graph. α and β are connected through the Pα set and Pβ set in the α-β graph, where Pαβ = PαPβ to define the weighted value of the corresponding sides that link with labeled nodes and sides that link with source node, sink nodes, or unlabeled nodes. Np is the neighborhood of node p, and the dissimilarity degree is labeled by Dp(·), it guides the energy function space of graph cuts; V(·) is the diversity measure of the neighborhood nodes, it has a smoothing effect.
In the actual implementation process, we also need to consider the solvable nature of the graph cuts problem, and we need to adjust the value range of each term appropriately according to some convention [31]. The energy function constructed in this paper satisfies the following three constraints. (1) The value of each function term in the graph cuts has upper and lower limits. (2) Besides the boundary range of the overlapped area, the values of the data term and smoothing term and their range of values should be roughly the same. For example, the value in the non-overlapped area of this paper can be larger than the ratio in the overlapped area, and it is equal to constant2. (3) As to the necessary and sufficient condition of the graph cuts problem, only if this condition is met, the graph cuts problem can be equivalent to max-flow min-cut theorem. Among them, the constraints (1) and (2) are weak constraints, by which the function solution range can be reduced and the solution efficiency is accelerated, and the constraint (3) is a strong one. In addition, the functions of V(*) and grad(*) in this paper use the truncation function. This is because, for a position with a large difference, it is only necessary to know that it is desirable, but not its value. If it is too large, it will destroy the constraint condition (3). Equation (10) is a truncation function, γ is a threshold. Table 1 shows some of the parameters selected in this paper.
ψ ( x ) = { x ; x < γ   y ; x γ

4. Experiments and Evaluation

4.1. Experimental Data and Computing Environment

So as to verify the effectiveness of our method (graph cuts-based seamless image mosaicking in in the color domain and gradient domain with optical flow, GCCGOF), this paper utilized the UAV image pairs from different regions with different flight heights and cameras, and compared the experimental results with representative methods in Open Source Computer Vision Library. In the comparison methods, one is a graph cuts-based seamless image mosaicking in the color domain (GCC), and the other is a graph cuts-based seamless image mosaicking in the color domain and gradient domain (GCCG). In this paper, we used Visual C++ based on OpenCV to program the proposed new method. The experimental UAV image pairs are divided into four groups. Among them, the data in Figure 5a were acquired by Canon IXUS 220HS (Canon, Oita, Japan) in Paris which the height of the UAV was approximately 250 m. The data in Figure 5b were acquired by DJ-Phantom4 (DJ, Shenzhen, China) at Wuhan University Square which the height of the UAV was approximately 115 m. The data in Figure 5c were acquired by ILCE-QX1 (Sony, Chonburi, Thailand) in Jiashan County, China which the height of the UAV was approximately 150 m. The data in Figure 5d were acquired by NEX-7 (Sony, Chonburi, Thailand) at Tacheng city which the height of the UAV was approximately 200 m. The experimental computing environment is Windows 7 operating system, CPU: Intel (R) Core (TM) i7-7700K, RAM: 32 GB. Figure 6 is a flow chart of our method, it listed the key steps.

4.2. Experimental Results and Analysis

4.2.1. Comparison of UAV Image Mosaic Results Using Different Methods

In order to verify the effect and superiority of the method that this paper proposed, methods of GCC and GCCG are used to search the optimal seam-lines for UAV image pairs with arbitrary overlapped regions indicated in Figure 5. In this paper, the non-regular overlapped area of image pair is extended to the regular area by the minimum circumscribed rectangle for the convenience of calculation, and the extended area is the invalid area (N). Figure 7, Figure 8, Figure 9 and Figure 10 are the respective results of image mosaicking in different UAV image data. It can be seen from Figure 7, Figure 8, Figure 9 and Figure 10 that the optimal seam-lines are obviously different in three test methods. From the local zoom view of Figure 7, Figure 8, Figure 9 and Figure 10, we can easily find that there are some great differences between the Figure 7a–c on both sides of the optimal seam-lines in the colored oval boxes, the same is true for other experimental results of Figure 8, Figure 9 and Figure 10. More specifically, in Figure 7, there are two roads with obvious image mosaic dislocations in the colored oval boxes on both sides of the optimal seam-lines which searched by method of GCC, and there is one road with distinct dislocation and a loss of housing structure information in Figure 7b. As opposed to Figure 7a,b, the above problems do not appear in Figure 7c, which shows that our method has almost achieved seamless stitching of UAV images in building-intensive areas. In Figure 8, we also can easily find that there are two apparent mosaic misplacements on the ground parking space. Meanwhile, we also found there are fewer cars in the overlapped areas of pre-mosaicked image pair in Figure 5b. It is worth pointing out that our method ensures the image information integrity of the overlapped area in mosaicked image, and there is almost no ground object dislocation.
In Figure 9a,b, the image stitching marks are very obvious. We can see the problems of ground object misplacements and structural information missing in the colored oval boxes in the partial enlargement drawings, which shows that their optimal seam-lines do not properly segment the overlapped areas of adjacent images. In Figure 9c, these problems have cured, and the image mosaic effect is very natural. In Figure 10, there are many high-rise buildings and low-rise factory buildings, and these buildings are densely distributed. In Figure 5d, we can discover that the height distribution in the overlapped area is higher in the middle and lower in both sides, and there is a large difference in the middle segment of the optimal seam-lines in Figure 10. From the colored oval boxes in Figure 10a,b, the housing structure information on the corner and roof is incomplete on both sides of the optimal seam-lines, not to mention there is a dislocation in the ground sidewalk. On the contrary, the ground objects on both sides of the optimal seam-line in Figure 10c achieve good alignment and the housing structure information is complete. Through the above analysis, we can see that our method can achieve seamless mosaicking of UAV remote sensing images, and the effect is better than the experimental results of comparison methods.

4.2.2. Efficiency Comparison of Different Methods

The improved method process not only found the better seam-line but also the more efficient one. As shown in Table 2. In order to speed up the computation time, the resolution of this experimental image is not same as the original one. From the statistical results analysis of Table 2, we can see that the time efficiency of our method (GCCGOF) is faster than the methods of GCC and GCCG in almost all experiments. Compared to the method of GCCG, in particular, the time efficiency of our method is almost one order of magnitude higher.

5. Conclusions

Image mosaicking is one of the key technologies in the field of UAV image processing, and it has a fundamental and vital role in many UAV remote sensing applications. This paper mainly focuses on the problem existing in the mosaicking of small UAV remote sensing images. Based on the previous related research, we manage to bring up a method for searching the best seam-line based on optical flow field and graph cuts for adjacent UAV images. Experiments prove the effectiveness of this proposed method. Specifically, this paper illustrated the following. (1) the four groups of pre-processed UAV images used in the experiments were image-registered using the mature and reliable SURF algorithm, and then the pre-mosaicked images were unified into the same coordinate system. (2) In the construction of graph cuts energy function, by considering the image pixel color domain, gradient domain information, together with the overlapped area global optical flow field information, a global seam-line detection method that takes into account image geometry information come into being. (3) The experimental results of this paper are compared with that of two different graph cuts-based seam-line search methods in OpenCV library used in the image mosaic experiments. Due to the intrinsic features of seam-line based image mosaicking method, aliasing and ghosting can be successfully avoided. At the same time, compared with the other two experimental methods, there is almost no problem of stitching misalignment and housing structure information missing, to sum up the method used in this paper is thus more effective and superior. The UAV mosaic images leaves no splicing visual traces.

Author Contributions

M.L. and D.L. proposed the methodology and wrote the paper; B.G. and L.L. conceived and designed the experiments; W.Z. and T.W. performed the experiments.

Funding

This research was supported by the National Key R&D Program of China (NO. 2018YFB0505400), the Fundamental Research Funds for the Central universities (NO. 2042018kf0012), the China Postdoctoral Science Foundation (NO. 2017M622520), and the LIESMARS Special Research Expenses.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Ghosh, D.; Kaabouch, N. A Survey on Image Mosaicking Techniques. Vis. Commun. Image Represent. 2016, 34, 1–11. [Google Scholar] [CrossRef]
  2. Byrne, J.; Keeffe, E.; Lenon, D.; Laefer, D. 3D reconstructions using unstabilized video footage from an unmanned aerial vehicle. J. Imaging 2017, 15, 1–15. [Google Scholar] [CrossRef]
  3. Pravenaa, S.; Methodical, R. A Methodical Review on Image Stitching and Video Stitching Techniques. Int. J. Appl. Eng. Res. 2016, 11, 3442–3448. [Google Scholar]
  4. Huang, Y.; Li, J.; Fan, N. Image Mosaicing for UAV Application. In Proceedings of the International Symposium on Knowledge Acquisition and Modeling, Wuhan, China, 21–22 December 2008; pp. 663–667. [Google Scholar]
  5. LI, D.; Li, M. Research advance and application prospect of unmanned aerial vehicle remote sensing system. Geomat. Inf. Sci. Wuhan Univ. 2014, 39, 505–513. [Google Scholar]
  6. Sun, H.; Li, L.; Ding, X.; Guo, B. The precise multimode GNSS positioning for UAV and its application in large scale photogrammetry. Geo-Spat. Inf. Sci. 2016, 19, 188–194. [Google Scholar] [CrossRef] [Green Version]
  7. Li, M.; Li, D.; Fan, D. A study on automatic UAV image method for paroxysmal disaster. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2012, XXXIX-B6, 123–128. [Google Scholar] [CrossRef]
  8. Zhang, Y.; Fu, H. A Method for Woodland Aerial Image Sequences Mosaic based on Iacunarity. J. Beijing For. Univ. 2017, 107–115. [Google Scholar]
  9. Chen, Z.; Zhang, L. An Aerial Image Mosaic Method Based on UAV Position and Attitude Information. Acta Geod. Cartogr. Sin. 2016, 45, 698–705. [Google Scholar]
  10. Sun, G.; Huang, W.; Chen, P.; Gao, X.; Wang, X. Advances in UAV-based Multispectral Remote Sensing Applications. Trans. Chin. Soc. Agric. Mach. 2018, 3, 1–17. [Google Scholar]
  11. Wang, H.; Zhou, L.; Zhang, X.; Gao, S.; Chu, Y.; Qiao, Y. Calibration Parameters of Long Focal Aerial Mapping Camera by Means of Calibration Filed. Opt. Precis. Eng. 2018, 26, 435–441. [Google Scholar] [CrossRef]
  12. Tan, X.; Mao, H.; Zhi, X.; Hu, X.; Ma, A.; Yan, L. Research on Image Data Matching Method Based on Infrared Spectrum Technology of UAV. Spectrosc. Spectr. Anal. 2018, 38, 413–417. [Google Scholar]
  13. Jia, Y.; Xu, Z.; Su, Z.; Qin, S.; Arshad, M. Mosaic of Crop Remote Sensing Images from UAV Based on Improved SIFT Algorithm. Trans. Chin. Soc. Agric. Mach. 2017, 33, 123–129. [Google Scholar]
  14. Balaji, G.; Subashini, T.; Chidambaram, N. Cardiac View Classification Using Speed up Robust Features. India J. Sci. Technol. 2015, 8, 1–5. [Google Scholar] [CrossRef]
  15. Tartavel, G.; Gousseau, Y.; Peyre, G. Variational Texture Synthesis with Sparsity and Spectrum Constraints. J. Math. Imaging Vis. 2015, 52, 124–144. [Google Scholar] [CrossRef]
  16. Guillemot, C.; Meur, O. Image Inpainting: Overview and Recent Advances. IEEE Signal Process. Mag. 2013, 31, 127–144. [Google Scholar] [CrossRef]
  17. Domadia, A.; Gautam, P. An Effective and Efficient Techniques of Image Inpainting: A Review. Int. J. Comput. Appl. 2015, 112, 29–32. [Google Scholar]
  18. Li, M.; Chen, R.; Zhang, W.; Li, D.; Liao, X.; Wang, L.; Pan, Y.; Zhang, P. A Stereo Dual-Channel Dynamic Programming Algorithm for UAV Image Stitching. Sensors 2017, 17, 2060. [Google Scholar] [CrossRef] [PubMed]
  19. Zhang, W.; Guo, B.; Li, M.; Liao, X.; Li, W. Improved Seam-Line Searching Algorithm for UAV Image Mosaic with Optical Flow. Sensors 2018, 18, 1214. [Google Scholar] [CrossRef] [PubMed]
  20. Li, N.; Liao, T.; Wang, C. Perception-based seam cutting for Image Stitching. Signal Image Video Process. 2018, 12, 967–974. [Google Scholar] [CrossRef]
  21. Li, L.; Yao, J.; Liu, Y.; Yuan, W.; Shi, S. Optimal Seamline Detection for Orthoimage Mosaicking by Combining Deep Convolution Neural Network and Graph Cuts. Remote Sens. 2017, 9, 701. [Google Scholar] [CrossRef]
  22. Li, L.; Yao, J.; Li, H.; Xia, M.; Zhang, W. Optimal Seamline Detection in Dynamic Scenes via Graph Cuts for Image Mosaicking. Mach. Vis. Appl. 2017, 28, 819–837. [Google Scholar] [CrossRef]
  23. Amrita, N. A Novel Approach for Medical Image Stitching Using Ant Colony Optimization. J. Eng. Res. Appl. 2014, 4, 21–28. [Google Scholar]
  24. Shashank, K.; Sivachaiutanya, N.; Mainkanta, G.; Balaji, C.; Murthy, V. A Survey and Review over Image Alignment and Stitching Methods. J. Electron. Commun. Technol. 2014, 5, 50–52. [Google Scholar]
  25. Wang, D.; Cao, W.; Xin, X.; Shao, Q.; Brolly, M. Using Vector Building Maps to Aid in Generating Seams for Low-attitude Aerial Orthoimage Mosaicking: Advantages in Avoiding the Crossing of Buildings. ISPRS J. Photogramm. Remote Sens. 2017, 125, 207–224. [Google Scholar] [CrossRef]
  26. Levin, A.; Zomet, A.; Peleg, S.; Weiss, Y. Seamless Image Stitching in the Gradient Domain. Proc. ECCV 2004, 3024, 377–389. [Google Scholar] [Green Version]
  27. Chen, J.; Cai, Z.; Lai, J.; Xie, X. Fast Optical Flow Estimation Based on the Split Bergman Method. IEEE Trans. Citcuits Syst. Video Technol. 2018, 28, 664–678. [Google Scholar] [CrossRef]
  28. Liu, B.; Wei, W.; Pan, Z.; Wang, S. Fast Algorithms for large Displacement Variation Optical Flow Computation. J. Image Graph. 2017, 22, 66–74. [Google Scholar]
  29. Boykov, Y.; Veksler, O.; Zabih, R. Fast Approximate Energy Minimization via Graph Cuts. IEEE Trans. Pattern Anal. Mach. Intell. 2001, 23, 1222–1239. [Google Scholar] [CrossRef]
  30. Schmidt, M.; Alahari, K. Generalized Fast Approximate Energy Minimization via Graph cuts: α-Expansion β-Shrink Moves. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, Barcelona, Spain, 14–17 July 2011. [Google Scholar]
  31. Zhang, W.; Li, M.; Guo, B.; Li, D.; Guo, G. Rapid Texture Optimization of Three-Dimensional Urban Model Based on Oblique Images. Sensors 2017, 17, 911. [Google Scholar] [CrossRef] [PubMed]
Figure 1. An UAV image pair and their optical flow field rendering. (a) UAV image pair. (b) Rendering of optical flow field.
Figure 1. An UAV image pair and their optical flow field rendering. (a) UAV image pair. (b) Rendering of optical flow field.
Ijgi 07 00361 g001
Figure 2. Structure of graph. (a) A directed graph structure. (b) A cut of the graph.
Figure 2. Structure of graph. (a) A directed graph structure. (b) A cut of the graph.
Ijgi 07 00361 g002
Figure 3. Images and construct of graph. (a) An analog seam-line in the overlapping area of adjacent images. (b) Diagram of a graph structure.
Figure 3. Images and construct of graph. (a) An analog seam-line in the overlapping area of adjacent images. (b) Diagram of a graph structure.
Ijgi 07 00361 g003
Figure 4. The α-β graph.
Figure 4. The α-β graph.
Ijgi 07 00361 g004
Figure 5. Four UAV image pairs. (a) The first pair. (b) The second pair. (c) The third pair. (d) The fourth pair.
Figure 5. Four UAV image pairs. (a) The first pair. (b) The second pair. (c) The third pair. (d) The fourth pair.
Ijgi 07 00361 g005
Figure 6. Flow chart of our method.
Figure 6. Flow chart of our method.
Ijgi 07 00361 g006
Figure 7. Optimal seam-lines of three test methods for Figure 5a. (a) GCC. (b) GCCG. (c) GCCGOF.
Figure 7. Optimal seam-lines of three test methods for Figure 5a. (a) GCC. (b) GCCG. (c) GCCGOF.
Ijgi 07 00361 g007
Figure 8. Optimal seam-lines of three test methods for Figure 5b. (a) GCC. (b) GCCG. (c) GCCGOF.
Figure 8. Optimal seam-lines of three test methods for Figure 5b. (a) GCC. (b) GCCG. (c) GCCGOF.
Ijgi 07 00361 g008
Figure 9. Optimal seam-lines of three test methods for Figure 5c. (a) GCC. (b) GCCG. (c) GCCGOF.
Figure 9. Optimal seam-lines of three test methods for Figure 5c. (a) GCC. (b) GCCG. (c) GCCGOF.
Ijgi 07 00361 g009
Figure 10. Optimal seam-lines of three test methods for Figure 5d. (a) GCC. (b) GCCG. (c) GCCGOF.
Figure 10. Optimal seam-lines of three test methods for Figure 5d. (a) GCC. (b) GCCG. (c) GCCGOF.
Ijgi 07 00361 g010
Table 1. The parameters.
Table 1. The parameters.
V(*) functionUse the truncation function first, then remapping to [0,1], truncation threshold is 120.
grad(*) functionUse the truncation function first, then remapping to [0,1], truncation threshold is 120.
flow(*)Remapping to [0,3], so that the global impact factor is greater than the sum of the local impact factors, which serves as a guidance.
Constant110,000
Constant21000
Table 2. Time efficiency comparison of three test methods
Table 2. Time efficiency comparison of three test methods
DataFigure 5aFigure 5bFigure 5cFigure 5d
Size of Overlapped Area956 × 522 (Pixels)635 × 362 (Pixels)1473 × 522 (Pixels)881 × 429 (Pixels)
LocationParis, FranceWuhan, ChinaJiashan, ChinaTacheng, China
Time of GCC2419 ms1042 ms3356 ms4552 ms
Time of GCCG11,736 ms3828 ms25,736 ms11,002 ms
Time of GCCGOF2150 ms3119 ms2848 ms3283 ms

Share and Cite

MDPI and ACS Style

Li, M.; Li, D.; Guo, B.; Li, L.; Wu, T.; Zhang, W. Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts. ISPRS Int. J. Geo-Inf. 2018, 7, 361. https://doi.org/10.3390/ijgi7090361

AMA Style

Li M, Li D, Guo B, Li L, Wu T, Zhang W. Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts. ISPRS International Journal of Geo-Information. 2018; 7(9):361. https://doi.org/10.3390/ijgi7090361

Chicago/Turabian Style

Li, Ming, Deren Li, Bingxuan Guo, Lin Li, Teng Wu, and Weilong Zhang. 2018. "Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts" ISPRS International Journal of Geo-Information 7, no. 9: 361. https://doi.org/10.3390/ijgi7090361

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop