Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts

: Image mosaicking is one of the key technologies in data processing in the ﬁeld of computer vision and digital photogrammetry. For the existing problems of seam, pixel aliasing, and ghosting in mosaic images, this paper proposes and implements an optimal seam-line search method based on graph cuts for unmanned aerial vehicle (UAV) remote sensing image mosaicking. This paper ﬁrst uses a mature and accurate image matching method to register the pre-mosaicked UAV images


Introduction
It is of great significance to integrate multiple small images with overlapped scenes into one large scene through image mosaicking technology, which can expand the field of vision and content of the image scene.This technology is widely used in aviation remote sensing, medical imaging diagnostics, military target tracking, disaster emergency management, and virtual reality or augmented reality [1][2][3][4].Currently, being economical, concise, lightweight, and fast, unmanned aerial vehicles (UAV) equipped with inexpensive cameras have become increasingly popular in the tasks of people's daily life entertainment and professional departments' surveying.In particular, virtual earth browsing, small-scale field mapping, forest resources investigation, and urban land planning have all created demand for UAV image mosaic technology.However, there are still many problems in the image mosaicking of UAV images, such as ghosting, stitching dislocation, blurriness, aliasing, and time efficiency [5][6][7][8][9].How to realize the seamless mosaicking of UAV images has always been an ultimate goal pursued by a large number of researchers.However, the current methods are still not satisfactory, especially in some applications such as real-time UAV image mosaicking.Therefore, with the gradual expansion of the application of UAV photogrammetry, UAV image mosaic technology has become a research hotspot in computer vision and image processing in recent years, and how to achieve seamless stitching of UAV images from different perspectives, different times, and even different cameras has become a major problem in these applications.dynamic programming is a symbolic method.This method is based on the optimization theory and is the best search based on the basic principle of dynamic programming.That is, regardless of the initial state of a process or the last decision, the next state or decision must be optimal.However, this kind of search process based on dynamic programming is very easy to fall into the local optimal solution and cannot obtain the global optimal solution.In addition, there are uncertain factors in the choice of the adjustment coefficient and the neighborhood range.Because using dynamic programming to obtain a local optimal seam-line cannot reach a good result.Many scholars have studied how to use a search method to obtain the global optimal seam-line [20][21][22].Among them, the optimal seam-line search method based on graph cuts is the most representative.It uses graph cuts to obtain the best mosaic texture on both sides of the seam-line to obtain the optimal seam-line, which can solve the shortcomings of dynamic programming in the image mosaicking (such as no need to specify additional energy end points, etc.).In addition, compared with the annealing algorithm and ant colony algorithm, it has the advantage of fast calculation [20][21][22][23][24][25][26].In these optimal seam-line search methods based on graph cuts, they can be divided into a color domain-based method, a gradient-based method, and a color domain and gradient domain-based method according to the difference of the energy function smoothing term.These methods use either the color space pixel difference information or gradient information separately for the construction of the graph cuts energy function, either considering the influence of the color domain and the gradient domain at the same time, this can make the energy function have more information.Therefore, when the latter is looking for the seam-line, it not only tends to have less gray information difference, but also hopes to find the seam-line at the edge of the drastic change of the building to be able to cover the later splicing traces.However, in the later feathering process, this will have a fuzzy and degrading effect on the edge of the building.This effect is extremely unfriendly to the mosaicking and application of high-resolution UAV images.
In summary, the optimal seam-line search method based on graph cuts is a global and extremely promising image mosaic method.However, any existing mosaic method based on graph cuts pays little attention to how to use geometric information constraints, and often focuses on more limited pixel level difference information (such as pixel color, gray scale, and gradient) to construct the smoothing term of an energy function, without looking at this issue form a global perspective.This often results in these methods not getting ideal, robust algorithm parameters, and ultimately affects the applicability of these methods.Therefore, the energy function of most current optimal seam-line search methods based on graph cuts only consider the image local information, and finally only the pseudo global solution of the target energy function can be obtained.In this paper, aiming at this problem, we propose to introduce the concept of global optical flow filed based on the existing graph cuts method to constrain the geometrical displacement of pixels due to perspective imaging, geometric distortion, and especially the projection errors caused by buildings, to achieve seamless UAV image mosaicking.

Method
The experimental UAV images used in this paper have been pre-processed in advance.In UAV image registration, the image registration method based on SURF is used in this paper, and RANSAC (Random Sample Consensus) algorithm is used to eliminate and optimize mismatching.Furthermore, the coarse-to-fine matching strategy is used in the transformation model solution.That is, all the matching feature points are used to estimate the transformation model parameters of adjacent images, and then the new transformation model parameters are accurately solved based on the elimination of the mismatching feature points.Finallt, the coordinate transformation between different images is implemented so that all the pre-mosaicked images are in a unified coordinate system [12][13][14].After completing image registration and coordinate conversion, the next step is to determine how to produce the final stitched image.This is the innovative work of this paper.In the following sections, we will introduce in detail on the optimal seam-line search method based on graph cuts proposed by this paper and verify the effectiveness and superiority of our proposed method through experiments.

Classic Graph Cuts-Based Optimal Seam-Line Search Method from OpenCV Library
If the reference image and the image to be registered have been completely registered, any mosaic method based on graph cuts can achieve a good image mosaic effect.However, due to the influence of external factors, the two images cannot be fully registered.Therefore, it is required to obtain an optimal seam-line to segment the overlapped area, so that the visual difference between the both sides of the optimal seam-line is the smallest, thereby realizing seamless mosaicking of UAV images.Image mosaicking based on seam-lines is one of dominant methods for UAV image mosaicking.It does not need to strictly align the entire overlapped area, but only processes the area near the seam-line, and it can splice large parallax scenes.In the famous open-source computer vision algorithm library (OpenCV), the graph cuts-based seam-line search methods are mainly divided into two types.One is to use the difference information of color space pixels for the construction of graph cuts energy function, and the other is based on the previous method, the constraint of gradient information is added.From the review and discussion of the previous literature, it can be seen that the mosaic results of any method are not satisfactory, and these methods have room for improvement in the algorithmic effectiveness and efficiency.Since the OpenCV open-source algorithm library is a very versatile and representative of the latest image processing integrated function library.Therefore, this paper will use the two representative graph cuts-based optimal seam-line search method in OpenCV as control experimental methods to evaluate the effectiveness and efficiency of the corresponding method proposed in this paper.Equations ( 1) and ( 2) are two typical construction strategies for smoothing terms of graph cuts energy function (1) only considers the gray value of the image pixel, and (2) added the gradient constraint of the image pixel.
In ( 1) and (2), p and q are adjacent pixels in the overlapped area, V(*) is the gray-scale difference function produced by the image pixel difference, grad(*) is the gradient change function generated by the pixel gradient, and f is the image.When the gradient value is not considered, the post-processing of the seam-line can easily make the edge of the ground building blurry.After considering the gradient value, the influence of the illumination change can be effectively avoided.

Improved Optimal Seam-Line Search Method Based on Graph Cuts
In general, adjacent UAV images will have the same local information, such as the same style buildings, but the scale information of adjacent images may not be exactly the same.At the same time, there will be differences in the perspective of different UAV images.If you do not consider the above situation in the image information, it is very likely that there will be geometrical misalignment or building information missing due to occlusion in the stitched images.Since many mosaic methods still rely on local information to solve the pseudo global solution of target energy function, the inaccuracy of the image splicing problem itself in the mathematical modeling process is neglected, and this leads to the target energy function created cannot fully express the problem that needs to be solved.Therefore, on the basic of the existing methods, this paper designs a more complete UAV image mosaic method with more global constraints.The new method introduces the concept of global optical flow field based on the existing graph cuts method to constrain the geometrical displacement of pixels due to perspective imaging, geometric distortion, and especially the projection errors caused by buildings to achieve seamless UAV image stitching.

Optical Flow Field
Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and scene [27,28].Optical flow fields are dense correspondence fields for accurate displacement optical flow estimation, they contain rich three-dimensional structural information of feature objects in images.This information of pixels in the overlapped area of adjacent images can effectively provide the structural information of images, which is conducive to search for the optimal seam-line [19].In this paper, in order to take into account the image structure information better in the overlapped area of the adjacent images, we use the optical flow value of pixels in the overlapped area as a constraint condition for constructing the energy function based on graph cuts.The Horn-Schunck method of estimating optical flow is a global method [27,28].This paper calculates the optical flow fields of experimental UAV remote sensing images based on this method.Equation (3) is the function of the Horn-Schunck algorithm, it is given for two-dimensional image streams.
In (3), F is the value of optical flow, I x , I y , and I t are the derivatives of image intensity values along x, y, and time dimensions respectively; u, v are the displacements along x, y axis; and a is a regularization constant.Figure 1 is a schematic of the calculated optical flow field.

Optical Flow Field
Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and scene [27,28].Optical flow fields are dense correspondence fields for accurate displacement optical flow estimation, they contain rich three-dimensional structural information of feature objects in images.This information of pixels in the overlapped area of adjacent images can effectively provide the structural information of images, which is conducive to search for the optimal seam-line [19].In this paper, in order to take into account the image structure information better in the overlapped area of the adjacent images, we use the optical flow value of pixels in the overlapped area as a constraint condition for constructing the energy function based on graph cuts.The Horn-Schunck method of estimating optical flow is a global method [27,28].This paper calculates the optical flow fields of experimental UAV remote sensing images based on this method.Equation ( 3) is the function of the Horn-Schunck algorithm, it is given for two-dimensional image streams.

Graph Construction
A graph is a directed structure with weights, which is usually represented by some nodes and directed edges.In this paper, these nodes are image pixels, and a graph usually has some special nodes called endpoints.In computer vision, these endpoints are used for pixel segmentation labels.As shown in Figure 2a, it is a directed graph structure with two endpoints, one of these is usually called a source node (s) and the other is called a sink node (t).The others are called nodes in the graph, and there is a weighted edge with direction between the terminal node and each node.In addition, there are also weighted edges between each node.Figure 2b shows an example of segmentation of a 3 × 3 directed graph structure.The red dashed line is a cut of the graph.The left pixels of the cut line are labeled as originating from the source node, and the right pixels of the cut line are labeled as originating from the sink node.In the graph, a and b are two common nodes.The cut cost of different segmentation methods will be calculated by the weighted edge.In image processing applications, many of these energy minimization problems can be approximated by solving a max-flow and min-cut problem in a graph.

Graph Construction
A graph is a directed structure with weights, which is usually represented by some nodes and directed edges.In this paper, these nodes are image pixels, and a graph usually has some special nodes called endpoints.In computer vision, these endpoints are used for pixel segmentation labels.As shown in Figure 2a, it is a directed graph structure with two endpoints, one of these is usually called a source node (s) and the other is called a sink node (t).The others are called nodes in the graph, and there is a weighted edge with direction between the terminal node and each node.In addition, there are also weighted edges between each node.Figure 2b shows an example of segmentation of a 3 × 3 directed graph structure.The red dashed line is a cut of the graph.The left pixels of the cut line are labeled as originating from the source node, and the right pixels of the cut line are labeled as originating from the sink node.In the graph, a and b are two common nodes.The cut cost of different segmentation methods will be calculated by the weighted edge.In image processing applications, many of these energy minimization problems can be approximated by solving a max-flow and min-cut problem in a graph.Specific to this paper, graph cuts-based optimal seam-line search is based on the graph structure.Therefore, it is necessary to convert the image pixels to the desired graph structure before searching for the seam-line.According to our UAV image mosaic theory and the definition of the graph above, the adjacent UAV images and graph structure can be illustrated in Figure 3. Figure 3a is a schematic diagram of UAV image mosaic, in which the pre-mosaicked image has a certain overlapped area with the reference image after the coordinate transformation, and the seam-line will be positioned in the overlapped area.The red line in Figure 3a is an example seam-line.According to this layout in Figure 3a, this paper constructs a graph structure and a part of the graph is shown in Figure 3b.Assuming that there are only 12 pixels in the overlapped area between the reference image and the pre-mosaicked image, and each pixel in the overlapped area is regarded as a node in the graph structure.Then, we can use the 12 circles from 1 to 12 in Figure 3b to represent 12 nodes in the graph, and the two adjacent nodes forming an edge which have weight, as shown in the green lines in Figure 3b.There are also weighted edges between the nodes in the non-overlapped area (that is, box portion of the reference image and the pre-mosaicked image) and the nodes in the overlapped area, as shown by the black lines in Figure 3b.The concept of the optimal seam-line detection is to search for a seam-line in the overlapped of adjacent images to divide the overlapped area into two parts, one part from the reference image, the other is from the pre-mosaicked image.This seam-line will pass through nodes in the overlapped area and directly cut off the weighted edges between the nodes.This is the concept of graph cuts in this paper.In Figure 3b, the red dash line is an example best seam-line.It divides nodes in the overlapped area into two parts.In the mosaicked image, pixels of 1, 5, 6, 9, 10, and 11 come from the pre-mosaicked image, and other pixels come from the reference image.In this way, the source of two partial pixels in the overlapped area are determined, and the purpose of seamless image mosaicking is achieved.

Energy Function Definition
According to the above, the overlapped area of two large adjacent images can construct a large graph structure.In the graph structure, the most critical part is the definition of the weighted edges, and the concrete form of energy function is determined by the definition of the weighted edges.The Specific to this paper, graph cuts-based optimal seam-line search is based on the graph structure.Therefore, it is necessary to convert the image pixels to the desired graph structure before searching for the seam-line.According to our UAV image mosaic theory and the definition of the graph above, the adjacent UAV images and graph structure can be illustrated in Figure 3. Figure 3a is a schematic diagram of UAV image mosaic, in which the pre-mosaicked image has a certain overlapped area with the reference image after the coordinate transformation, and the seam-line will be positioned in the overlapped area.The red line in Figure 3a is an example seam-line.According to this layout in Figure 3a, this paper constructs a graph structure and a part of the graph is shown in Figure 3b.Assuming that there are only 12 pixels in the overlapped area between the reference image and the pre-mosaicked image, and each pixel in the overlapped area is regarded as a node in the graph structure.Then, we can use the 12 circles from 1 to 12 in Figure 3b to represent 12 nodes in the graph, and the two adjacent nodes forming an edge which have weight, as shown in the green lines in Figure 3b.There are also weighted edges between the nodes in the non-overlapped area (that is, box portion of the reference image and the pre-mosaicked image) and the nodes in the overlapped area, as shown by the black lines in Figure 3b.The concept of the optimal seam-line detection is to search for a seam-line in the overlapped of adjacent images to divide the overlapped area into two parts, one part from the reference image, the other is from the pre-mosaicked image.This seam-line will pass through nodes in the overlapped area and directly cut off the weighted edges between the nodes.This is the concept of graph cuts in this paper.In Figure 3b, the red dash line is an example best seam-line.It divides nodes in the overlapped area into two parts.In the mosaicked image, pixels of 1, 5, 6, 9, 10, and 11 come from the pre-mosaicked image, and other pixels come from the reference image.In this way, the source of two partial pixels in the overlapped area are determined, and the purpose of seamless image mosaicking is achieved.Specific to this paper, graph cuts-based optimal seam-line search is based on the graph structure.Therefore, it is necessary to convert the image pixels to the desired graph structure before searching for the seam-line.According to our UAV image mosaic theory and the definition of the graph above, the adjacent UAV images and graph structure can be illustrated in Figure 3. Figure 3a is a schematic diagram of UAV image mosaic, in which the pre-mosaicked image has a certain overlapped area with the reference image after the coordinate transformation, and the seam-line will be positioned in the overlapped area.The red line in Figure 3a is an example seam-line.According to this layout in Figure 3a, this paper constructs a graph structure and a part of the graph is shown in Figure 3b.Assuming that there are only 12 pixels in the overlapped area between the reference image and the pre-mosaicked image, and each pixel in the overlapped area is regarded as a node in the graph structure.Then, we can use the 12 circles from 1 to 12 in Figure 3b to represent 12 nodes in the graph, and the two adjacent nodes forming an edge which have weight, as shown in the green lines in Figure 3b.There are also weighted edges between the nodes in the non-overlapped area (that is, box portion of the reference image and the pre-mosaicked image) and the nodes in the overlapped area, as shown by the black lines in Figure 3b.The concept of the optimal seam-line detection is to search for a seam-line in the overlapped of adjacent images to divide the overlapped area into two parts, one part from the reference image, the other is from the pre-mosaicked image.This seam-line will pass through nodes in the overlapped area and directly cut off the weighted edges between the nodes.This is the concept of graph cuts in this paper.In Figure 3b, the red dash line is an example best seam-line.It divides nodes in the overlapped area into two parts.In the mosaicked image, pixels of 1, 5, 6, 9, 10, and 11 come from the pre-mosaicked image, and other pixels come from the reference image.In this way, the source of two partial pixels in the overlapped area are determined, and the purpose of seamless image mosaicking is achieved.

Energy Function Definition
According to the above, the overlapped area of two large adjacent images can construct a large graph structure.In the graph structure, the most critical part is the definition of the weighted edges, and the concrete form of energy function is determined by the definition of the weighted edges.The

Energy Function Definition
According to the above, the overlapped area of two large adjacent images can construct a large graph structure.In the graph structure, the most critical part is the definition of the weighted edges, and the concrete form of energy function is determined by the definition of the weighted edges.The general energy function of graph cuts consists of data item and smooth item, which correspond to the weights of the edges in the cutting of graph cuts.Equation ( 4) is the general form of the graph cuts energy function definition, where f is the label assigned by the node, that is, the cost of a node from the source or sink is labeled as f.
Among ( 4), E data is a data item, which is used to express the difference between the value of the node and the value of the label f.E smooth is a smoothing term, it is used to measure the segment smoothness.Data item generally use a distance measurement method to determine the size of the difference, while the smoothing item calculation is more complex.Because the smoothing item must not only tolerate certain gradual transitions and noise disturbances, but also must be detected in boundaries or mutations, the property is called discontinuity preserving.In [29], a Bayesian first-order partial derivative Markov random field is used to express the energy function of Equation (4) in Equation ( 5).
Among ( 5), N represents adjacent pixels, O is all pixels in the overlapped area.f p is a label assigned to pixel p.Therefore, the Equation ( 5) is the cost of the numerical difference caused by allocating labels for all the pixels in the overlapped area and the cutting for adjacent pixels near the seam-line.In [29], the data item is defined as non-negative by Boykov.At the same time, in order to make the smoothing item have the property of discontinuity preserving, Boykov specifies that the value of the smoothing item must be a metric or a semimetric.
So far, the energy function model proposed is clear.Equation ( 6) is the energy function definition based on graph cuts proposed in this paper.The biggest difference compared to the other energy functions described above is that the definition of our energy function is added to the global optical flow field.In this way, based on the existing graph cuts algorithm, the geometric information constraint can be better considered.The geometric displacement of the pixel caused by the perspective imaging and geometric distortion in the image mosaic can be solved.
where V(*) is the grayscale difference function generated by the pixel difference, grad(*) is the gradient change function generated by the gradient difference, flow(*) is the optical flow field function of the overlapped area between the adjacent UAV images, N(p) is the energy value of the invalid area (the invalid area refers to the non-pixel area in the minimum enclosing rectangle of the overlapped area), and its value is generally much large than the value of the overlapped area.In this paper, the value of N(p) is constant.f 1 and f 2 are two pre-mosaicked images.O is the overlapped area.The pixel grayscale difference function not only needs to reflect the difference information of pixels in the same position of adjacent images, but also needs to introduce local information to make it have locality information to enhance the robustness of the algorithm.Combined with the particularity of this problem, it is not appropriate to add individual difference data items to each node of a graph.At the same time, since the essence of searching the seam-line is the difference information, not the direct probability problem of a certain label.Therefore, all the energy terms of the algorithm in this paper are reflected in the weight of the neighboring edge in the graph.However, the weight function of the node and the endpoint (s or t) is a binary function, and its value range is {0,constant1}.The value of Constant1 is that much larger than the adjacent edge weight.Therefore, the functions constructed in this paper need to consider the information the nodes at both ends of the edge.The grayscale difference function of the adjacent pixels (p, p + 1) can be expressed as the form of Equation (7).
If the image to be mosaicked is an RGB image, the three-dimensional vector is used for calculation, and ||*|| is the Euclidean distance of one RGB color value of the pixel.Similarly, the gradient change function can be expressed as the form of Equation (8).
where d l is a gradient operator in a certain direction and can be calculated using four-neighbor or eight-neighbor.Because of the small difference, this paper uses a four-neighbor gradient operator for computational efficiency.Equation ( 9) is the optical flow field function herein.
F is the optical flow calculation function, that is, the displacement field obtained by the difference after the sparse matching of the adjacent images, and the value records the sum of the squares of the pixel displacements.

Solving Minimum Energy Value
Through the definition of the energy function based on graph cuts above, a realistic image seam-line search problem is expressed by a mathematical model, and the minimization process of the energy function value is the search process of the optimal seam-line.In the specific solution process, we use the maximum flow minimum cut algorithm to solve the minimization problem of the energy function.Among them, α-β swap and α-expansion are two good practical solution strategies.In this paper, the α-β swap optimization strategy is used to solve the energy function.It is an effective graph cuts binary optimization algorithm, which can not only optimize the segmentation of the initial labeled data set, but also transform the multi-dimensional directed graph into two-dimensional simple directed graph, Thereby, the limited uncertainty of the multi-dimensional directed graph on the capacity values connecting the different node edges is avoided [29,30].
Figure 4 shows the construction way of α-β graph.α and β are connected through the P α set and P β set in the α-β graph, where P αβ = P α ∪ P β to define the weighted value of the corresponding sides that link with labeled nodes and sides that link with source node, sink nodes, or unlabeled nodes.N p is the neighborhood of node p, and the dissimilarity degree is labeled by D p (•), it guides the energy function space of graph cuts; V(•) is the diversity measure of the neighborhood nodes, it has a smoothing effect.If the image to be mosaicked is an RGB image, the three-dimensional vector is used for calculation, and || * || is the Euclidean distance of one RGB color value of the pixel.Similarly, the gradient change function can be expressed as the form of Equation ( 8). ( ( ), ( )) ( ( )) ( ( 1)) ( ( )) ( ( 1)) where dl is a gradient operator in a certain direction and can be calculated using four-neighbor or eight-neighbor.Because of the small difference, this paper uses a four-neighbor gradient operator for computational efficiency.Equation ( 9) is the optical flow field function herein.

( ) ( ) ( 1) flow p F p F p
F is the optical flow calculation function, that is, the displacement field obtained by the difference after the sparse matching of the adjacent images, and the value records the sum of the squares of the pixel displacements.

Solving Minimum Energy Value
Through the definition of the energy function based on graph cuts above, a realistic image seam-line search problem is expressed by a mathematical model, and the minimization process of the energy function value is the search process of the optimal seam-line.In the specific solution process, we use the maximum flow minimum cut algorithm to solve the minimization problem of the energy function.Among them, α-β swap and α-expansion are two good practical solution strategies.In this paper, the α-β swap optimization strategy is used to solve the energy function.It is an effective graph cuts binary optimization algorithm, which can not only optimize the segmentation of the initial labeled data set, but also transform the multi-dimensional directed graph into two-dimensional simple directed graph, Thereby, the limited uncertainty of the multi-dimensional directed graph on the capacity values connecting the different node edges is avoided [29,30].
Figure 4 shows the construction way of α-β graph.α and β are connected through the Pα set and Pβ set in the α-β graph, where Pαβ = Pα ∪ Pβ to define the weighted value of the corresponding sides that link with labeled nodes and sides that link with source node, sink nodes, or unlabeled nodes.Np is the neighborhood of node p, and the dissimilarity degree is labeled by Dp(•), it guides the energy function space of graph cuts; V(•) is the diversity measure of the neighborhood nodes, it has a smoothing effect.In the actual implementation process, we also need to consider the solvable nature of the graph cuts problem, and we need to adjust the value range of each term appropriately according to some convention [31].The energy function constructed in this paper satisfies the following three constraints.( 1  In the actual implementation process, we also need to consider the solvable nature of the graph cuts problem, and we need to adjust the value range of each term appropriately according to some convention [31].The energy function constructed in this paper satisfies the following three constraints. (1) The value of each function term in the graph cuts has upper and lower limits.(2) Besides the boundary range of the overlapped area, the values of the data term and smoothing term and their range of values should be roughly the same.For example, the value in the non-overlapped area of this paper can be larger than the ratio in the overlapped area, and it is equal to constant2.(3) As to the necessary and sufficient condition of the graph cuts problem, only if this condition is met, the graph cuts problem can be equivalent to max-flow min-cut theorem.Among them, the constraints (1) and ( 2) are weak constraints, by which the function solution range can be reduced and the solution efficiency is accelerated, and the constraint ( 3) is a strong one.In addition, the functions of V(*) and grad(*) in this paper use the truncation function.This is because, for a position with a large difference, it is only necessary to know that it is desirable, but not its value.If it is too large, it will destroy the constraint condition (3).Equation ( 10) is a truncation function, γ is a threshold.Table 1 shows some of the parameters selected in this paper.

flow(*)
Remapping to [0,3], so that the global impact factor is greater than the sum of the local impact factors, which serves as a guidance.Constant1 10,000 Constant2 1000

Experimental Data and Computing Environment
So as to verify the effectiveness of our method (graph cuts-based seamless image mosaicking in in the color domain and gradient domain with optical flow, GCCGOF), this paper utilized the UAV image pairs from different regions with different flight heights and cameras, and compared the experimental results with representative methods in Open Source Computer Vision Library.In the comparison methods, one is a graph cuts-based seamless image mosaicking in the color domain (GCC), and the other is a graph cuts-based seamless image mosaicking in the color domain and gradient domain (GCCG).In this paper, we used Visual C++ based on OpenCV to program the proposed new method.The experimental UAV image pairs are divided into four groups.Among them, the data in Figure 5a were acquired by Canon IXUS 220HS (Canon, Oita, Japan) in Paris which the height of the UAV was approximately 250 m.The data in Figure 5b were acquired by DJ-Phantom4 (DJ, Shenzhen, China) at Wuhan University Square which the height of the UAV was approximately 115 m.The data in Figure 5c were acquired by ILCE-QX1 (Sony, Chonburi, Thailand) in Jiashan County, China which the height of the UAV was approximately 150 m.The data in Figure 5d were acquired by NEX-7 (Sony, Chonburi, Thailand) at Tacheng city which the height of the UAV was approximately 200 m.The experimental computing environment is Windows 7 operating system, CPU: Intel (R) Core (TM) i7-7700K, RAM: 32 GB. Figure 6 is a flow chart of our method, it listed the key steps.

Comparison of UAV Image Mosaic Results Using Different Methods
In order to verify the effect and superiority of the method that this paper proposed, methods of GCC and GCCG are used to search the optimal seam-lines for UAV image pairs with arbitrary overlapped regions indicated in Figure 5.In this paper, the non-regular overlapped area of image pair is extended to the regular area by the minimum circumscribed rectangle for the convenience of calculation, and the extended area is the invalid area (N).we can easily find that there are some great differences between the Figure 7a-c on both sides of the optimal seam-lines in the colored oval boxes, the same is true for other experimental results of Figures 8-10.More specifically, in Figure 7, there are two roads with obvious image mosaic dislocations in the colored oval boxes on both sides of the optimal seam-lines which searched by method of GCC, and there is one road with distinct dislocation and a loss of housing structure information in Figure 7b.As opposed to Figure 7a,b, the above problems do not appear in Figure 7c, which shows that our method has almost achieved seamless stitching of UAV images in building-intensive areas.In Figure 8, we also can easily find that there are two apparent mosaic

Comparison of UAV Image Mosaic Results Using Different Methods
In order to verify the effect and superiority of the method that this paper proposed, methods of GCC and GCCG are used to search the optimal seam-lines for UAV image pairs with arbitrary overlapped regions indicated in Figure 5.In this paper, the non-regular overlapped area of image pair is extended to the regular area by the minimum circumscribed rectangle for the convenience of calculation, and the extended area is the invalid area (N).we can easily find that there are some great differences between the Figure 7a-c on both sides of the optimal seam-lines in the colored oval boxes, the same is true for other experimental results of Figures 8-10.More specifically, in Figure 7, there are two roads with obvious image mosaic dislocations in the colored oval boxes on both sides of the optimal seam-lines which searched by method of GCC, and there is one road with distinct dislocation and a loss of housing structure information in Figure 7b.As opposed to Figure 7a,b, the above problems do not appear in Figure 7c, which shows that our method has almost achieved seamless stitching of UAV images in building-intensive areas.In Figure 8, we also can easily find that there are two apparent mosaic

Comparison of UAV Image Mosaic Results Using Different Methods
In order to verify the effect and superiority of the method that this paper proposed, methods of GCC and GCCG are used to search the optimal seam-lines for UAV image pairs with arbitrary overlapped regions indicated in Figure 5.In this paper, the non-regular overlapped area of image pair is extended to the regular area by the minimum circumscribed rectangle for the convenience of calculation, and the extended area is the invalid area (N).we can easily find that there are some great differences between the Figure 7a-c on both sides of the optimal seam-lines in the colored oval boxes, the same is true for other experimental results of Figures 8-10.More specifically, in Figure 7, there are two roads with obvious image mosaic dislocations in the colored oval boxes on both sides of the optimal seam-lines which searched by method of GCC, and there is one road with distinct dislocation and a loss of housing structure information in Figure 7b.As opposed to Figure 7a,b, the above problems do not appear in Figure 7c, which shows that our method has almost achieved seamless stitching of UAV images in building-intensive areas.In Figure 8, we also can easily find that there are two apparent mosaic misplacements on the ground parking space.Meanwhile, we also found there are fewer cars in the overlapped areas of pre-mosaicked image pair in Figure 5b.It is worth pointing out that our method ensures the image information integrity of the overlapped area in mosaicked image, and there is almost no ground object dislocation.misplacements on the ground parking space.Meanwhile, we also found there are fewer cars in the overlapped areas of pre-mosaicked image pair in Figure 5b.It is worth pointing out that our method ensures the image information integrity of the overlapped area in mosaicked image, and there is almost no ground object dislocation.In Figure 9a,b, the image stitching marks are very obvious.We can see the problems of ground object misplacements and structural information missing in the colored oval boxes in the partial enlargement drawings, which shows that their optimal seam-lines do not properly segment the overlapped areas of adjacent images.In Figure 9c, these problems have cured, and the image mosaic effect is very natural.In Figure 10, there are many high-rise buildings and low-rise factory buildings, and these buildings are densely distributed.In Figure 5d, we can discover that the height distribution in the overlapped area is higher in the middle and lower in both sides, and there is a large difference in the middle segment of the optimal seam-lines in Figure 10.From the colored oval misplacements on the ground parking space.Meanwhile, we also found there are fewer cars in the overlapped areas of pre-mosaicked image pair in Figure 5b.It is worth pointing out that our method ensures the image information integrity of the overlapped area in mosaicked image, and there is almost no ground object dislocation.In Figure 9a,b, the image stitching marks are very obvious.We can see the problems of ground object misplacements and structural information missing in the colored oval boxes in the partial enlargement drawings, which shows that their optimal seam-lines do not properly segment the overlapped areas of adjacent images.In Figure 9c, these problems have cured, and the image mosaic effect is very natural.In Figure 10, there are many high-rise buildings and low-rise factory buildings, and these buildings are densely distributed.In Figure 5d, we can discover that the height distribution in the overlapped area is higher in the middle and lower in both sides, and there is a large difference in the middle segment of the optimal seam-lines in Figure 10.From the colored oval In Figure 9a,b, the image stitching marks are obvious.We can see the problems of ground object misplacements and structural information missing in the colored oval boxes in the partial enlargement drawings, which shows that their optimal seam-lines do not properly segment the overlapped areas of adjacent images.In Figure 9c, these problems have cured, and the image mosaic effect is very natural.In Figure 10, there are many high-rise buildings and low-rise factory buildings, and these buildings are densely distributed.In Figure 5d, we can discover that the height distribution in the overlapped area is higher in the middle and lower in both sides, and there is a large difference in the middle segment of the optimal seam-lines in Figure 10.From the colored oval boxes in Figure 10a,b, the housing structure information on the corner and roof is incomplete on both sides of the optimal seam-lines, not to mention there is a dislocation in the ground sidewalk.On the contrary, the ground objects on both sides of the optimal seam-line in Figure 10c achieve good alignment and the housing structure information is complete.Through the above analysis, we can see that our method can achieve seamless mosaicking of UAV remote sensing images, and the effect is better than the experimental results of comparison methods.boxes in Figure 10a,b, the housing structure information on the corner and roof is incomplete on both sides of the optimal seam-lines, not to mention there is a dislocation in the ground sidewalk.On the contrary, the ground objects on both sides of the optimal seam-line in Figure 10c achieve good alignment and the housing structure information is complete.Through the above analysis, we can see that our method can achieve seamless mosaicking of UAV remote sensing images, and the effect is better than the experimental results of comparison methods.boxes in Figure 10a,b, the housing structure information on the corner and roof is incomplete on both sides of the optimal seam-lines, not to mention there is a dislocation in the ground sidewalk.On the contrary, the ground objects on both sides of the optimal seam-line in Figure 10c achieve good alignment and the housing structure information is complete.Through the above analysis, we can see that our method can achieve seamless mosaicking of UAV remote sensing images, and the effect is better than the experimental results of comparison methods.

Efficiency Comparison of Different Methods
The improved method process not only found the better seam-line but also the more efficient one.As shown in Table 2.In order to speed up the computation time, the resolution of this experimental image is not same as the original one.From the statistical results analysis of Table 2, we can see that the time efficiency of our method (GCCGOF) is faster than the methods of GCC and GCCG in almost all experiments.Compared to the method of GCCG, in particular, the time efficiency of our method is almost one order of magnitude higher.

Conclusions
Image mosaicking is one of the key technologies in the field of UAV image processing, and it has a fundamental and vital role in many UAV remote sensing applications.This paper mainly focuses on the problem existing in the mosaicking of small UAV remote sensing images.Based on the previous related research, we manage to bring up a method for searching the best seam-line based on optical flow field and graph cuts for adjacent UAV images.Experiments prove the effectiveness of this proposed method.Specifically, this paper illustrated the following.(1) the four groups of pre-processed UAV images used in the experiments were image-registered using the mature and reliable SURF algorithm, and then the pre-mosaicked images were unified into the same coordinate system.(2) In the construction of graph cuts energy function, by considering the image pixel color domain, gradient domain information, together with the overlapped area global optical flow field information, a global seam-line detection method that takes into account image geometry information come into being.(3) The experimental results of this paper are compared with that of two different graph cuts-based seam-line search methods in OpenCV library used in the image mosaic experiments.Due to the intrinsic features of seam-line based image mosaicking method, aliasing and ghosting can be successfully avoided.At the same time, compared with the other two experimental methods, there is almost no problem of stitching misalignment and housing structure information missing, to sum up the method used in this paper is thus more effective and superior.The UAV mosaic images leaves no splicing visual traces.

Figure 1 .
Figure 1.An UAV image pair and their optical flow field rendering.(a) UAV image pair.(b) Rendering of optical flow field.

Figure 1 .
Figure 1.An UAV image pair and their optical flow field rendering.(a) UAV image pair.(b) Rendering of optical flow field.

15 Figure 2 .
Figure 2. Structure of graph.(a) A directed graph structure.(b) A cut of the graph.

Figure 3 .
Figure 3. Images and construct of graph.(a) An analog seam-line in the overlapping area of adjacent images.(b) Diagram of a graph structure.

Figure 2 .
Figure 2. Structure of graph.(a) A directed graph structure.(b) A cut of the graph.

15 Figure 2 .
Figure 2. Structure of graph.(a) A directed graph structure.(b) A cut of the graph.

Figure 3 .
Figure 3. Images and construct of graph.(a) An analog seam-line in the overlapping area of adjacent images.(b) Diagram of a graph structure.

Figure 3 .
Figure 3. Images and construct of graph.(a) An analog seam-line in the overlapping area of adjacent images.(b) Diagram of a graph structure.
) The value of each function term in the graph cuts has upper and lower limits.(2) Besides the boundary range of the overlapped area, the values of the data term and smoothing term

15 Figure 5 .
Figure 5. Four UAV image pairs.(a) The first pair.(b) The second pair.(c) The third pair.(d) The fourth pair.

Figure 6 .
Figure 6.Flow chart of our method.

Figures 7 -
10 are the respective results of image mosaicking in different UAV image data.It can be seen from Figures 7-10 that the optimal seam-lines are obviously different in three test methods.From the local zoom view of Figures 7-10 ,

Figure 5 . 15 Figure 5 .
Figure 5. Four UAV image pairs.(a) The first pair.(b) The second pair.(c) The third pair.(d) The fourth pair.

Figure 6 .
Figure 6.Flow chart of our method.

Figures 7 -
10 are the respective results of image mosaicking in different UAV image data.It can be seen from Figures 7-10 that the optimal seam-lines are obviously different in three test methods.From the local zoom view of Figures 7-10 ,

Figure 6 .
Figure 6.Flow chart of our method.

Figures 7 -
10 are the respective results of image mosaicking in different UAV image data.It can be seen from Figures 7-10 that the optimal seam-lines are obviously different in three test methods.From the local zoom view of Figures 7-10 , ISPRS Int.J. Geo-Inf.2018, 7, x FOR PEER REVIEW 11 of 15

Table 2 .
Time efficiency comparison of three test methods