Geometric Primitive-Guided UAV Path Planning for High-Quality Image-Based Reconstruction

Abstract

Image-based refined 3D reconstruction relies on high-resolution and multi-angle images of scenes. The assistance of multi-rotor drones and gimbal provides great convenience for image acquisition. However, capturing images with manual control generally takes a long time. It could easily lead to redundant or insufficient local area coverage, resulting in poor quality of the reconstructed model. We propose a surface geometric primitive-guided UAV path planning method (SGP-G) that aims to automatically and quickly plan a collision-free path to capture fewer images, based on which high-quality models can be obtained. The geometric primitives are extracted by plane segmentation on the proxy, which performs three main functions. First, a more representative evaluation of the reconstructability of the whole scene is realized. Second, two optimization strategies for different geometric primitives are executed to quickly generate a near-global optimized set of viewpoints. Third, regularly arranged viewpoints are generated to improve the efficiency of image acquisition. Experiments on virtual and real scenes demonstrate the remarkable performance of our method. Compared with the state of the art, we accomplish the planning of the photographic path with higher efficiency in a relatively simple way, achieving equivalent and even higher quality of the reconstructed model with fewer images.

1. Introduction

There is an increasing demand for 3D reconstruction of large scenes in areas such as urban planning, autonomous driving, virtual reality and gaming. Currently, considering data sources, the 3D reconstruction methods can be divided into two types: laser scanning-based and image-based [1,2]. Generally, the laser scanning-based model is most costly, and the reconstructed model suffers from a lack of texture. Image-based 3D reconstruction methods, on the other hand, are less expensive and more effective even with only a monocular camera [3]. Many fine-grained reconstruction efforts are carried out using image-based 3D reconstruction methods. These works mainly revolve around the SFM (Structure from Motion) and MVS (Multi-View Stereo) theories [4,5], observing the target from different perspectives and capturing the 3D information to obtain better reconstruction results. The core idea of the image-based 3D reconstruction method is the efficient use of geometric information on multi-view images [6,7,8].

To obtain high-quality models, current research focuses on two aspects. One is to improve the accuracy and computational performance of reconstruction algorithms. There are already many mature algorithms and software that can achieve high-quality image 3D reconstruction. Open-source algorithms include Colmap, OpenMVG, VisualSfM, etc., while commercial software includes Context Capture, MetaShape, Reality Capture, Pix4D, etc. In addition, some studies such as MVSNet [9] and R-MVSNet [10] have used an end-to-end depth estimation framework based on deep learning to obtain 3D dense point clouds by estimating depth directly from images to improve accuracy in scenes with repeated or missing textures and drastic changes in illumination. The reconstruction algorithms have been maturely developed relatively. The other is to improve the quality of the reconstructed model by acquiring or selecting high-quality images. Unlike the optimization of reconstruction algorithms, this is mainly applied in the data acquisition phase to input high-quality images to the reconstruction system. The quality of the images determines the reconstructed model quality [11], while the number and resolution of the images determine the time cost of the reconstruction process. Inadequate and insufficient coverage can result in mismatches between images or holes in reconstructed models. On the other hand, excessively redundant images would increase the time and calculation cost during image acquisition and reconstruction processes, and even lead to poor reconstruction quality [7,12]. Image collection is increasingly becoming an essential issue in 3D reconstruction work [13].

UAVs are widely used in the image acquisition process for 3D reconstruction to acquire images from multiple views, including different orientations and positions. However, most of the flight paths of UAVs are performed under manual control or some predefined flight modes in practical operations [14], in which situation a greater number of images is prone to be captured and thus cause redundancy and long-time consumption. An efficient path planning solution for UAVs to capture images autonomously which can ensure flight safety and reconstructability is urgently required [15,16]. This paper focuses on the planning of UAV photographic viewpoints and paths to achieve high-quality image acquisition and ultimately high-quality model reconstruction. The multi-rotor UAV with RTK (Real Time Kinematic) module and gimbal provides the hardware implementation basis for high-quality image collection [17,18,19], so that the one-to-one correspondence of captured images and the planned viewpoint can be met.

The common planning process is explore-then-exploit, which requires obtaining a priori coarse model of the scene in advance, which is called proxy hereafter. Sample points are then generated on the surface of the proxy and the reconstructability of each sample point is calculated based on the reconstructability heuristics, which are used to measure the reconstructability of the scene and to optimize the viewpoint poses or select the optimal set of viewpoints [20,21]. However, most of them adopted a holistic optimization approach to achieve the optimization of photographic viewpoints, which brings the problem of long optimization time. We propose a UAV path planning method guided by geometric primitives to realize close-up photography for 3D refined reconstruction. The contributions of our work are:

• We pioneer the preprocessing method of planar segmentation and primitive extraction to divide the reconstructed scene into independent geometric primitives, and we simplify the viewpoint optimization of the whole scene to the reconstructable optimization problem of each independent primitive.
• We establish two mathematical models from the traditional aerial photogrammetry to measure the reconstructability of polygon and line primitives, based on which suitable overlap ratios are calculated to quickly generate an initial set of viewpoints for approximating the global optimum.
• We construct an objective function that satisfies submodularity to accelerate the iterative selection of optimal viewpoints for the point primitives by measuring the expected gain instead of the actual reward.

2. Related Works

2.1. Priori Geometry Proxy

Current viewpoint selection methods for UAV path planning can be divided into two categories depending on whether an initial proxy is required: estimating the viewpoints in an unknown environment iteratively and determining the viewpoints based on an initial coarse model [20]. The former estimates new viewpoints through iterative computation to increase information gain without prior knowledge [22,23,24,25,26,27,28,29]. Ref. [24] dynamically estimated 3D boundary boxes of buildings to guide online path planning for scene exploration and building observation. Ref. [23] estimated building height and captured close-up images through the SLAM framework to reveal architectural details. Generally referred to as the next-best-view, it is challenging for this approach to meet the full coverage requirement of refined reconstruction and it relies on the real-time computing power of the UAV.

The latter solution is also called the priori proxy-based viewpoint selection approach [15,16,20,22,30,31,32,33,34,35], which is often referred to as explore-then-exploit. It requires an initial model, based on which analysis and planning are carried out to identify viewpoints that satisfy the reconstruction requirements. The proxy can be some existing 3D data of the scene with height information, or a low-precision model obtained from an initial flight. Ref. [31] used a 2D map of the scene to estimate the building height from the shadows, then generated a 2.5D coarse model. Ref. [32] took the reconstructed dense point cloud as the initial model and determined the viewpoints covering the whole scene after preprocessing and quality evaluation of the point cloud. Some studies used a 3D mesh as the proxy and planned the photographic viewpoint based on the surface geometric information [15,16,20]. Considering safety and robustness, we choose the latter, that the rough proxy of the scene to be reconstructed is utilized to plan the flight path. For generality, a triangular mesh is utilized as the proxy. Any 3D data that can be converted to it can also be used, including DEM (Digital Elevation Model), point cloud, BIM model and 3D mesh of the large scene.

2.2. Viewpoint Optimization

To meet the requirements of 3D reconstruction and ensure the efficiency of the UAV, it is necessary to optimize the viewpoints set continuously and generate fewer viewpoints to complete the image acquisition and 3D reconstruction. Ref. [36] selected viewpoints covering the whole scene considering visibility and self-occlusion, but did not consider the observation of particular parts and limitations on the number of viewpoints and flight time. Refs. [15,16,17] applied submodular optimization methods to select candidate viewpoints, considering factors such as the number of viewpoints, camera angle, flight time and space obstacle avoidance, etc., hoping to obtain more information with the fewest viewpoints under the given constraints. Ref. [20] applied a reconstruction heuristic to plan the location and orientation of photographic viewpoints in a continuous optimization problem, intending to produce a more accurate and complete reconstruction result with fewer images. The above methods require many iterations, and each iteration requires traversing every viewpoint. In addition, methods such as [15,20,24] would generate many viewpoints, resulting in extremely long optimization time and local optimum [34].

We adopt two different optimization strategies based on the primitives extracted from the proxy. For polygon and line primitives, we quickly generate an initial set of optimized viewpoints at an appropriate overlap ratio that roughly covers the scene. The initial set occupies most of the final viewpoints so that we can meet the requirements of refined reconstruction with only a few viewpoints added for point primitives.

3. Methodology

In this section, our SGP-G method for refined 3D reconstruction is introduced in detail. First, a rough geometric proxy of the scene is required to generate the SDSM (Shaped Digital Surface Model) for safe flight and extract the primitives (Section 3.1). Two different optimization methods are then performed to obtain optimized viewpoints (Section 3.2). Finally, a collision-free appropriate optimal path is connected for image acquisition (Section 3.3). The overall workflow is shown in Figure 1.

3.1. Priori Information and Pre-Processing

3.1.1. Geometric Proxy Preparation

Similar to the method in [20,30,31], SGP-G requires an initial proxy that would serve as the data basis for the entire path planning process, including sample point and viewpoint generation, obstacle avoidance, and occlusion judgment. A relatively accurate absolute position is required for the initial proxy to ensure flight safety. Common proxies include point cloud [32], DEM [20], 2.5D coarse model [30,31], BIM, 3D mesh obtained by oblique photography [35], etc. We usually adopt the coarse mesh as the initial proxy, as shown in Figure 1a, which can be reconstructed by capturing images over the scene following a regular path. In Section 4.2.1, we have also conducted experiments on proxies with varying levels of detail.

3.1.2. SDSM Generation

Considering flight safety, the generated viewpoint and the final path must be outside the obstacles. In addition, the occlusion of sight between the viewpoint and its corresponding target point must also be considered. We already have an initial proxy as the prior information of the scene to help obstacle avoidance and occlusion judgment. Therefore, we only need to judge whether it is occluded according to whether the sight or path intersects with the triangular faces of the proxy. However, the number of triangular faces is relatively large, especially in large-scale scenes, making the calculation highly time-consuming.

To accelerate the calculation, we generate a SDSM (a simplified Octomap) to represent the obstacle, as shown in Figure 1b. We generate m × n equally spaced grid points on the XY plane with the sampling interval set to 1 m. For each grid, the maximum elevation of the four corners is taken as the elevation of the grid. When judging whether a viewpoint is within the obstacle, we only need to calculate whether its elevation is higher than the elevation of the corresponding point with the same plane coordinates on SDSM. For sight and path occlusion, sampling is performed on them first to compare elevations.

3.1.3. Geometric Primitives Extraction

Based on the initial proxy, basic geometric primitives are extracted to decompose the viewpoint optimization of the whole scene reconstruction into the reconstructable optimization problem of each primitive. Plane segmentation [37] is performed on the proxy surface to obtain each planar region that is composed of triangular faces. After that, polygon primitives are extracted by tracing edges of each planar region [38,39]. The edges of the polygon that can perfectly describe the segmented planar area are obtained, as shown in the colored block in Figure 2c. The normal vector of the polygon is the same as the original planar region. Considering that the connecting edges of adjacent polygon primitives are not smooth [15], we track the edges of the adjacent plane regions and obtain the line primitive as a supplement [40,41]. The extracted line primitives are as shown as the red dotted line in Figure 2c. The normal vector of the line primitive is equal to the average normal vector of the two adjacent planar regions. In addition, curved surfaces or fragmented details inevitably form a tiny target during mesh segmentation. We split the small areas into point primitives for subsequent processing according to the area threshold.

With extracted primitives, a more representative measurement of reconstructability is realized subsequently. Then, different optimization strategies are performed on the primitives, which realize an efficient optimization of photography viewpoints.

3.2. Primitive-Guided Viewpoint Generation and Optimization

3.2.1. Sample on Primitives

For consistency with [16,20,24,30,31,32,34,35] and convenient distinguishing, the terms “target point” and “sample point” are introduced in our work to represent the point with corresponding normal vector sampling on the initial coarse proxy or the extracted primitives. However, the target and sample points serve two different purposes, respectively.

• Viewpoint Generation and Adjustment.

The target point is used to calculate and adjust the corresponding viewpoint position and orientation. For target point s_i, the corresponding viewpoint v_i is generated as shown in Figure 3. The pose of the viewpoint v_i can be transformed from (x, y, z, −nx_i, −ny_i, −nz_i) into (x, y, z, yaw, pitch) by the Rodriguez formula and angle decomposition, as shown in Equation (1), which is then provided for UAV to capture images. Rod (a, b) represents the rotation matrix of vector a rotated around the a × b axis to vector b. n_i = (nx_i, ny_i, nz_i) is the normal vector of the target point s_i. sight_before and y_before are the original sight vector and the y-axis of the image coordinate system when (yaw, pitch, roll) = 0, respectively. Sight_after is the photographic sight which is −n_i. Note that the equation roll ≡ 0.0 can be adopted to judge whether the angle yaw and pitch are correctly calculated.

$$\begin{gathered} v_i=\left(x_i,y_i,z_i\right)=s_i+d\ast n_i \\ {R}_1=Rod\left(sigh{t}_{before},sigh{t}_{after}\right) \\ {R}_2=Rod\left(R_1\ast y_{before},sigh{t}_{after}\times Z\right) \\ yaw,pitch,roll=angledecompostion(R_2\ast R_1), roll\equiv 0.0 \end{gathered}$$

(1)

When the shooting distance is far, the generated viewpoint may be inside an obstacle, as shown in Figure 4a, and there may be obstacles between the sight of the viewpoint and the target point, as shown in Figure 4b. To avoid this situation, we fix the distance from the viewpoint to the target point, controlling the viewpoint orientation and the normal vector of the target point within a certain angle range. Then, we sample candidate viewpoints within a certain spherical range of area and search for other unobstructed viewpoints, as shown in Figure 4.

• Reconstruction Heuristics

The sample point is used to evaluate the reconstructability of the scene by calculating its reconstructable value which indicates whether the neighbor region can be reconstructed well. Usually, a greater number of sample points would be generated on the proxy surface to better represent the reconstructability of the scene. Note that there is no viewpoint corresponding to the sample points.

A mathematical model that can evaluate the reconstructable value is needed. Some related works [4,15,16,20,22,34,42] have studied this problem, considering factors including shooting distance, observation angle, parallax angle, multi-view observation, etc. Considering the computational complexity and the theoretical integrity, the heuristics proposed by [20] are adopted, as shown in Equation (2). H(s, V) represents the reconstructable value (hereinafter referred to as H) of sample point s calculated from all viewpoints V. $c\left(s,v_i,v_j\right)$ represents contribution of two viewpoints $v_i,v_j$ to s, and $v\left(s,v_i\right)$ is a binary function that evaluates the visibility of sample to v_i.

$$H\left(s,V\right)={\displaystyle \sum }_{i=1\dots \left|V\right|,j=i\dots \left|V\right|}v\left(s,v_i\right)v\left(s,v_j\right)c\left(s,v_i,v_j\right)$$

(2)

Many researchers have studied the reconstruction mechanism. According to [4,31], the reconstruction can be satisfied when the three rays observe the same point at the same angle of 15 degrees, based on which the minimum threshold H_T is calculated at about 1.30. According to [20], when H > 3, the depth error of the sample point hardly changes. According to [24], when H > 5, the Spearman correlation between the H and the accuracy of the sample point decreases sharply, floating around 0.1. Therefore, in the subsequent optimization, we keep the H of each sample point between 1.3 and 5.0 and set the minimum threshold to 1.3 to satisfy the reconstruction. If it exceeds 5.0, redundancy is considered to exist in the visible viewpoints.

For polygon primitives, we calculate its two-dimensional enveloping rectangle, then generate sample points and target points with a grid pattern according to different intervals within the rectangle, and judge whether the point falls within the polygon range. For line primitives, we directly sample equidistantly, and when the extracted edge is short, its center with normal vector is directly taken as the sample point and target point, which is the same as the operation of point primitives.

3.2.2. Optimization Objective Function

The internal normal vectors of the extracted polygon primitives are consistent. When we generate viewpoints according to certain rules, such as the overlap ratio, if the images captured at the viewpoints can realize the high-quality reconstruction of the internal polygon region, we can generate viewpoints quickly with a preset overlap ratio to realize the coverage of the polygon region. The time complexity of this step is O(n), where n is the number of target points. In most scenes, especially in urban scenes, when we segment the plane on rough proxies, polygon and line primitives can represent most surfaces and edges, respectively. In this way, we can speed up the viewpoint optimization process significantly. We construct an objective function, as shown in Equation (3), which divides viewpoint optimization into two steps.

$$\begin{gathered} V=\underset{V_1}{argmin}{\displaystyle \sum }_{s_i\in S_{polygon}+S_{line}}[H\left(s_i,V_1\right)-H_T] \\ +\underset{V_2}{argmin}{\displaystyle \sum }_{s_j\in \lambda S_{unrecon}}[H\left(s_j,V_1+V_2\right)-H_T] \\ s.t.H>H_T, \lambda \in \left(85\%, 95\%\right) \end{gathered}$$

(3)

The viewpoint set V₁ for the polygon and line primitives is generated first. We consider these viewpoints an initial set for achieving the global optimum, which realizes the coverage of most parts needed for reconstruction in the urban scene, especially. Subsequent viewpoint optimization is performed to obtain V₂ for point primitives, that is, fragmentary details in the scene. S_polygon represents the sample points generated in polygon primitives, and the same for lines and points. S_unrecon represents the unreconstructable sample points after adding viewpoints generated for polygon and line primitives, which is composed of S_point and a small proportion of S_polygon and S_line. H_T is set at 1.3. In addition, λ ∈ (85%,95%) is set, which means that we do not require that all fragmentary details, especially the trees, satisfy the reconstructability, which is difficult to achieve.

3.2.3. Initial Optimal Set for Polygon and Line Primitives

To guarantee reconstructability, we need to calculate a proper forward and side overlap based on Equation (2) to generate aligned target points and viewpoints. We construct a mathematical model for measuring the reconstructability of polygon primitives, as shown in Figure 5a, where $d_{side}=\left(1-A\right)\ast h\ast GSD, d_{forward}=\left(1-B\right)\ast w\ast GSD, d_m=f\ast GSD$. GSD (Ground Sample Distance) represents the length on the ground corresponding to a pixel with the unit of cm/pixel. A and B represent the side and forward overlap ratio, respectively, and f is the focal length in pixels. Furthermore, w and h are the pixels of the image in width and height. We convert the first half of the objective function into Equation (4) and calculate the optimal overlap ratio which keeps the H of each sample point in S exceeding the minimum threshold H_T. S is shown as the blue area in Figure 5a which corresponds to the central range of v₄, while other regions in the polygon primitives would correspond to other viewpoints. The viewpoint set V(A,B) only includes 9 viewpoints since the outer 16 viewpoints only contribute to the reconstruction of region S when the overlap exceeds 66%.

$$\begin{gathered} A,B=argmin{\displaystyle \sum }_{s_i\in S}\left[H\left(s_i,V\left(A,B\right)\right)-H_T\right], s.t.H>H_T \\ V\left(A,B\right)=\left\{v_i|\begin{array}{c}v{x}_i\in \left\{-d_{forward},0,d_{forward}\right\},\\ v{y}_i\in \left\{-d_{side},0,d_{side}\right\},\\ v{z}_i=d_m\\ i=1,2,\dots \dots 9\end{array}\right\}, \\ S=\left\{s_i|\begin{array}{c}s{x}_i\in \left(-\frac{d_{forward}}{2},\frac{d_{forward}}{2}\right),\\ s{y}_i\in \left(-\frac{d_{side}}{2},\frac{d_{side}}{2}\right), \\ s{z}_i=0\end{array}\right\} \end{gathered}$$

(4)

We calculate the optimal forward and side overlap with parameters of the camera mounted on commonly used DJI P4RTK, with a resolution of 4864 × 3648, a focal length of 8.8 mm and a pixel size of 2.4 microns. We set the search range from 66% to 33%. Iterative calculation finds that 50% for both forward and side overlap ratio is the optimal solution of Equation (4). The H distribution in area S is shown in Figure 6. It can be found that the distribution of H is relatively uniform, and the H of every point in S exceeds H_T. When the overlap is 50%, each sample point in the blue area in Figure 5a can be uniformly covered by four images.

However, according to Figure 5a and Equation (4), the relative distribution of the viewpoint set V(A, B) and the sample point set S is related to the three factors: w, h, and f of the camera. The GSD only affects the scale of the scene model in Figure 5a. In fact, when calculating the parallax angle α between two viewpoints, it is found that for the same GSD, the FOV (Field of View) and image frame would have an impact on α, which in turn affects the H. Thus, we consider two independent variables, w/f and w/h, to further verify the effectiveness of the 50% forward and side overlap. w/f corresponds to the FOV, while w/h corresponds to the image frame which is commonly 4:3 and 3:2. We calculate the coordinates of sample points in area S and the nine viewpoints in V(A, B) with different w/f and w/h, and then obtain the average and the minimum reconstructable value of sample points in the region S, as shown in Figure 7. For commonly used UAVs and cameras like DJI P4RTK, DJI M300, and Nex7 used by [20], a 50% overlap ratio can ensure that each sample point inside the polygon primitives can be reconstructed. Moreover, from Figure 7, it can be found that the reconstructable value is maintained above 2.0 when the FOV is in the range of 30–55 degrees which is considered to be the appropriate camera FOV with an overlap ratio of 50% for acquiring images.

The mathematical model in Figure 5a only considers the interior of polygon primitives. However, there are still uncovered edge regions, as shown in the blue area in Figure 5b. The same calculation is also employed to generate target points and viewpoints on line primitives, which proves that all sample points in the blue area can be reconstructed when θ is between (75°, 270°) with a 50% overlap ratio. If the reconstructability is still not satisfied, two additional viewpoints are generated according to the normal vector of its adjacent polygons to ensure the reconstructability. Through the complementation of adjacent polygons and line primitives, all points on the edges of polygon primitives can be reconstructed.

The overlap ratio of 50% Is Ideal without considering the influence of viewpoint adjustment for obstacle avoidance. For cameras with wider FOV, the 50% overlap ratio may not guarantee that every sample point could be well reconstructed. While images from other viewpoints would compensate for the reconstructability of unreconstructable areas, we still recommend choosing an overlap ratio between 53% and 60% for polygon and line primitives to generate target points and viewpoints. The overlap range is set for three reasons: (1) when the overlap is more than 50% in both forward and side directions, each sample point can be covered by at least four images and meet the reconstructability; (2) considering the effect of camera distortion, the closer to the edge, the greater the distortion, so the overlap is set to a minimum of 53%; (3) to avoid redundancy from high overlap, the upper limit is set to 60%. However, this is an empirically derived range, and we strongly recommend no less than 53%, but the upper limit can be improved appropriately according to the complexity of the scene.

Note that a dense sampling whose density is much higher than [16,20,24,30,31,32,34] is performed in S, resulting in a more representative measurement of the reconstructability of polygon and line primitives. However, densely sampling is only performed to compute the suitable overlap ratio and verify its effectiveness. Since the internal normal vector of each primitive is consistent, each generated sample point can represent a certain area around it compared with random uniform sampling. In addition, their reconstructability has been almost satisfied by the preset overlap ratio. Hence, fewer sample points need to be generated on polygon and line primitives to judge their reconstructability. In subsequent submodular optimization, the sample points for judging reconstructability including unreconstructable sample points are directly transformed from target points generated with preset overlap ratio on polygon and line primitives, whose amount is relatively low.

3.2.4. Submodular Optimization for Point Primitives

For polygon and line primitives, we have generated an optimized initial set of viewpoints that already cover most surfaces in the scene. However, for small details in the scene, if all point primitives are performed as target points, a large number of viewpoints would be generated. It would result in redundancy, especially when the proxy tends to be more detailed, such as the proxy obtained by oblique photography. We also only need fewer images to cover the details of the scene while maintaining a high-quality reconstruction result. To better restore the details in the scene and maximize the gain from each viewpoint, we construct a submodular formulation as shown in Equation (5). The point primitives are converted to sample points and target points simultaneously to measure reconstructability and generate viewpoint set $V_p$ for optimization. Through submodular optimization, we can select one viewpoint each time with the largest information gain, so that fewer viewpoints are produced to cover the details.

$$\begin{gathered} v_i=argmax EH, v_i\in V_p \\ EH={\displaystyle \sum }_{s\in S_{v_i}}[\mathit{exp}\left(H_T-H\left(s,V\right)\right)-1]\ast A_s \end{gathered}$$

(5)

Unlike [15] directly calculating the actual H gain of each viewpoint during iterations, we select the viewpoint expected to obtain the maximum gain EH calculated by Equation (5). From Equation (5), the optimal viewpoint whose sum reconstructability of its visible sample points is the smallest is selected for the current iteration. It is considered that adding this viewpoint will bring a greater expected gain. A_s refers to the area of the neighbor region the sample point represents, which is a weight parameter. Sv_i is the set of unreconstructable sample points visible from viewpoint v_i. H(s,V) is the reconstructable value of s calculated from the current optimized viewpoint set V. Compared to the O$\left(m\widehat{m}\widehat{n}\right)$ time complexity of [20,31] with the same heuristics, ours is only O($\widehat{m}\widehat{n}$) in each iteration, where $\widehat{m}$ is the average number of viewpoints visible to a sample point, $\widehat{n}$ is the opposite, and $m$ is the number of viewpoint points. We can easily find that this formula satisfies the condition of submodularity, as when the H of every sample point increases, the EH decreases. After each new viewpoint is added, H is updated for the sample points seen by the viewpoint, and a new viewpoint is recalculated until the income EH is less than 0. In this way, the process of viewpoint optimization is very fast. Moreover, we can determine the required number of viewpoints to ensure details coverage without adding too many viewpoints.

3.3. Shortest Path Connection

To maximize the effectiveness of the UAV and evaluate the quality of the final generated path, we construct a mathematical model to measure energy consumption and flight time based on the UAV flight dynamics theory [43,44]. The mathematical model is shown in Equation (6). The following loose assumptions are set to simplify the complexity of the mathematical model. We consider that the air density at different heights remains the same. Statistics have found that the air density change is minimal when the height difference is within 300 m. Energy consumption only considers work against gravity and work against resistance. When overcoming the resistance, it is considered that the UAV always keeps a stable speed without acceleration and deceleration. In addition, we assume that the energy consumption of the UAV orientation change is minimal, and the most power consumption is used to keep the UAV hovering when changing the orientation. Since gravity is a conservative force, we consider that the energy consumed to overcome gravity when the drone hovers is the same as when the drone rises and then descends to the same position [44].

$$\begin{gathered} P_{drag}=\frac{1}{2}{C}_{D}AD{v}^3, Q_{drag}=P_{drag\ast }{T}_{drag} \\ {P}_{lift}=\frac{W^{\frac{3}{2}}}{\sqrt{2DB}}, Q_{lift}=P_{lift\ast }{T}_{lift} \end{gathered}$$

(6)

T_lift is the time from takeoff to landing, and T_drag is the time for drone flight motion. C_D is the wind resistance coefficient; A is the front-facing area of UAV in m²; B is the area of the rotor disc, which refers to the plane formed by the rotating rotor blades of rotorcraft. D is the air density, v is the flying speed of the drone, and W is the weight of the drone itself. To achieve high-precision positioning and attitude shooting, the drone needs to hover and adjust the orientation of the drone to control the yaw, and adjust the gimbal to control the pitch. The time that the drone hovers to change the orientation is considered. We define the relationship between hovering angle and time according to the experiments performed by [30].

According to the above model, we can realistically simulate the energy consumption and time of the UAV capturing images along the path. When we input the viewpoint sequence $V=\left\{V_1,\dots ,V_i\dots ,V_n\right\} ,n\ge 2$, the time and energy consumed during a flight can be obtained. For the follow-up calculation, the parameters of commonly used M300 UAV are substituted into Equation (6) for calculation.

After obtaining the optimized set of viewpoints, we need to connect these viewpoints into a flight path and provide it to the UAV, which is a typical TSP problem. The method of [45,46] is adopted to calculate an approximately optimal path connection scheme through the ACO (Ant Colony Optimization) algorithm. Each viewpoint is a node in the graph, and we calculate the energy consumption between two viewpoints as the cost between the nodes based on Equation (6). When calculating the cost, we must determine whether the path between the two nodes passes through an obstacle, in which situation we consider the cost infinite. To speed up the subsequent path calculation process, we adopt the k-proximity method and only calculate the cost of the 20 nodes closest to each node.

4. Results

To comprehensively evaluate the effectiveness of our SGP-G method, some metrics are set to evaluate the quality of the path and the final reconstructed model. Experiments are performed both on real and virtual scenes, including the UK, Goth, and NY scenes of [20], the School, Town scenes of [47], and our Zhlp scene constructed from Urban City, as shown in Figure 8. Three types of proxy are adopted, including 2.5D box, 2.5D coarse and 3D inter. The proxy of 3D inter retains more details of the scene with more triangular faces, and 2.5D coarse retains only the range and elevation of buildings, while 2.5D box is composed of simpler bounding boxes.

The commonly used Context capture software is used to recover 3D models from images. Considering the camera distortion and the different detailed proxies, the forward and side overlap ratio of 53–60% is adopted to generate viewpoints of polygon and line primitives.

4.1. Evaluation Measures

We comprehensively evaluate the methods from two aspects: path quality and model quality. For the former, the number of viewpoints, path length, ideal flight time and energy consumption calculated based on [43,44] are evaluated. As for the model quality, the average GSD and RMS meter of the aerial triangulation process is measured, as well as the precision and recall of the reconstructed model. In addition, the visual fidelity of the reconstructed model on some details is also compared.

Especially, the metrics of precision and recall are similar to error and completeness brought up by [20]. We uniformly sample the recon (Reconstructed) and gt (Ground Truth) models at 400 pts/square meters to obtain recon points and gt points [48], so that the distributions in the two point clouds are appropriately the same. For each point in the recon points, we calculate the closest distance to gt points. Given a certain distance threshold d, we calculate the percentage of recon points whose distance is less than d. We set the distance threshold d = 0.1 m, which can better reflect the poor-quality area in the model. That is, the area where the distance with the nearest corresponding point is more than 0.1 m is considered poorly reconstructed. The percentage is then called precision (0.1). In turn, we calculate the percentage of gt points whose distance from the closest point in recon points is less than d, which is then called recall (0.1). A higher percentage means better quality of the reconstructed model. $FScore=2\ast \frac{precision\left(0.1\right)\ast recall\left(0.1\right)}{precision\left(0.1\right)+recall\left(0.1\right)}$ is also calculated, which is a common measure for binary classification and suitable for the comprehensive evaluation of model quality. In addition, we count the distances at which 90% of the points in the recon and gt models are smaller, and precision(90%) and recall (90%) are obtained, respectively.

4.2. Self-Evaluation

4.2.1. Impact of Different Detailed Proxy

Three virtual scenes, including UK, Town and School, are used to measure the difference of plane segmentation in different detailed proxies. Among them, UK mainly includes relatively regular buildings; Town includes irregular buildings, bridges and trees; School includes suspended buildings with repeated texture of ground areas. Plane segmentation is performed on proxies containing varying levels of detail, and then geometric primitives are extracted, as shown in Figure 9.

For regular and simple proxies, such as 2.5D box and 2.5D coarse, the segmentation is neat because only the range and elevation information are retained with fewer details. There are more details in the proxy of 3D inter, leading to messier segmentation results. However, regardless of the type of proxy used, the extracted primitives can reflect the geometric composition of the proxy well.

The impact of different detailed proxies on the reconstruction results is also evaluated on Town and School scenes. Both scenes contain some details that the 2.5D model cannot represent. We generate a photographic path with a GSD of 0.7 cm/pixel based on the extracted primitives, and then reconstruct the model. The calculated precision and recall of the reconstructed model are shown in

Table 1. Comparison of model quality between using different proxies. The arrows represent the trend of the values with better results on the indicators, and the following tables are the same.

Scene	Proxy	#Imgs	Precision		Recall
			0.1 m ↑%	90% ↓m	0.1 m ↑%	90% ↓m
Town 0.7 cm 53%	2.5D Box	183	91.47	0.066	70.51	0.806
	2.5D Coarse	220	93.04	0.056	71.07	0.778
	3D Inter	242	94.00	0.053	71.31	0.769
School 0.7 cm 53%	2.5D Box	198	67.83	30.103	44.27	2.308
	2.5D Coarse	220	85.79	0.212	46.47	2.041
	3D Inter	379	85.63	0.253	50.53	1.663

.

It is straightforward to see that precision and recall are on the rise in general for proxy from rough to fine, which aligns with our expectations, especially on School scene. The precision drops with more images on School from 2.5D coarse to 3D inter. It is considered normal, as when the reconstructability of the scene is already guaranteed by sufficient coverage, adding additional images may lead to decreased precision [30].

The overall model quality is satisfying, proving that our method is relatively robust for proxies with different detailed proxies. Although the precision is low for the 2.5D box proxy of School, the recall is not. Moreover, a higher overlap ratio, such as 60%, can make up for the impact of the proxy being too simple. Experiments show that when we adopt an overlap ratio of 60% for the 2.5D box proxy to generate 263 images, the precision and recall are improved to 85.60% and 49.30%, respectively.

4.2.2. Impact of Overlap Ratio

We have theoretically verified the rationality of the 50% forward and side overlap ratio by calculating H in Section 3.2.3. In addition, we generate viewpoints from polygon and line primitives by combining multiple overlap ratios to verify rationality by calculating precision and recall. To avoid the influence of point primitives generated by the details, the experiment is performed on Zhlp scene. Zhlp includes three regular buildings with a few non-planar details, which can better reflect the effect of the overlap ratio on reconstruction. We generate viewpoints with forward and side overlap ratios of 45–45%, 53–53%, 60–60%, and 66–66%, respectively. In particular, specific overlap ratios of 66–33% and 66–8% are adopted, which also ensure that at least three viewpoints cover the sample points.

The quantitative evaluation of the reconstructed model is shown in

Table 2. Comparison of model quality between using different overlap ratios.

Overlap Ratio	#Imgs	Reconstructable Percent ↑%	Precision		Recall
			0.1 m ↑%	90% ↓m	0.1 m ↑%	90% ↓m
45–45%	66	64.8	aerial triangulation failed
53–53%	76	94.4	93.50	0.060	79.54	0.296
56–56%	79	96.8	94.51	0.055	80.53	0.257
60–60%	86	99.1	93.76	0.055	81.07	0.174
66–66%	97	99.6	94.33	0.052	81.18	0.211
66–33%	80	89.0	92.01	0.068	78.28	0.307
66–8%	59	73.7	aerial triangulation failed

. Reconstructable percent represents the proportion of the uniform sample points [48] that satisfy the reconstructability according to its H and the threshold H_T. From Table 2, the aerial triangulation is prone to failure when the reconstructable percent is relatively low, such as ratios of 45–45% and 66–8%. Higher overlap rates do not significantly improve recall and accuracy when the overlap rate is greater than 53%. For the 66–33% overlap ratio, the precision and recall decrease and the number of images increases. It is clear that our preset overlap ratio range is reasonable, and a 53% overlap ratio can achieve good results with fewer viewpoints. Increasing the overlap ratio can also lead to some improvement in precision and recall if the limitation on the number of images is not considered.

4.2.3. Effectiveness of Submodular Formulation

Submodular optimization is performed on viewpoints from point primitives to cover fragmented details with little redundancy. To verify the effectiveness of the submodular optimization strategy, the 3D inter proxies of School and Town are adopted to generate optimal viewpoint sets Φ, Φ + 15, Φ + 30, Φ + 50, and viewpoint set when the automatic iteration terminates, respectively. Φ is the viewpoint set generated from polygon and line primitives. A 2.5D proxy is not suggested, as it would generate fewer point primitives. We calculate the estimated gain EH by Equation (5) and ΔH which is the actual H gain of unreconstructable sample points for each iteration after adding the viewpoint with maximum EH, as shown in Figure 10. Based on images captured from viewpoints, we reconstruct the 3D model and calculate the recall, as shown in

Table 3. Evaluation of model quality obtained with different numbers of images.

Scene	Optimization End	#Images	Reconstructable Percent ↑%	EH	ΔH	Recall
						0.1 m ↑%	90% ↓m
Town Inter 0.7 cm	Φ	236	98.30			70.62	0.782
	convergence	242	98.90	3532.52	36.63	71.31	0.769
	Φ + 15	251	98.96	−486.44	2.20	71.34	0.765
	Φ + 30	266	99.01	−1835.38	2.65	71.32	0.772
	Φ + 50	286	99.01	−3377.84	0.02	71.48	0.752
School Inter 0.7 cm	Φ	359	97.29			49.36	1.768
	Φ + 15	374	99.60	8071.96	120.32	50.27	1.669
	convergence	379	99.67	160.64	11.48	50.53	1.663
	Φ + 30	389	99.73	−283.32	1.6	50.67	1.637
	Φ + 50	409	99.74	−1425.27	0	50.79	1.630

.

From Figure 10 and Table 3, the EH calculated by the submodular function can almost represent the actual H gain. Their trends are consistent, which verifies that submodular optimization is reasonable. Furthermore, when the set contains viewpoints generated for polygon and line primitives, the recall has already reached a high level. As the number of viewpoints increases, the recall increases slowly, reflecting the diminishing marginal effect of submodular optimization. When the convergence condition we set is reached, the recall hardly increases, which shows that our convergence conditions are reasonable. In addition, we find that the reconstructability can already be generally satisfied by only the viewpoint set generated for polygon and line primitives, which proves that it is a suitable initial value to approximate the global optimum.

4.3. Comparison to State of the Art

• Comparison with [20]

To verify the effectiveness of our SGP-G method, we compare it with [20] on NY, Goth and UK. Since its gt model cannot be obtained, three metrics based on its evaluation tool are evaluated, including Precision, Local Completeness (hereinafter referred to as LComp) and Global Completeness (hereinafter referred to as GComp). The evaluation from these three metrics is completed similarly to precision and recall. The difference is that the distance threshold is set as 0.05 m and 0.075 m for LComp. For GComp, we modified the parameters and set the distance threshold to 0.5 m and 0.75 m to measure the percentage of the unreconstructed part. For a fair comparison, the same camera with pixel size of 3.9 microns, resolution of 6000 × 4000 and focal length of 14.16 mm are adopted to generate the flight path and capture virtual images. In addition, we set the GSD of the captured image to be consistent with its average GSD.

Table 4. Comparison with [20] on scenes of UK, Goth and NY. The bolded values represent the best results for the current indicator, and the following tables are the same.

Scene	Method	#Plan ↓mins	#Imgs	Length ↓m	Time ↓s	Energy ↓J	RMSE ↓mm	GSD ↓mm	Precision		LComp		GComp
									90% ↓m	95% ↓m	0.05 m ↑%	0.075 m ↑%	0.50 m ↑%	0.75 m ↑%
UK	[20]	7.93	923	7819	6795	58,162	5.70	13.72	0.04	0.069	35.99	38.25	69.20	73.21
	Ours-53%	1.45	479	6923	2962	25,924	5.47	13.57	0.039	0.060	37.35	39.46	69.31	73.26
NY	[20]	4.12	433	2807	2960	25,258	4.20	9.86	0.064	0.218	45.09	48.89	83.61	85.57
	Ours-53%	0.75	234	2639	1189	10,366	4.57	10.17	0.045	0.247	44.98	48.49	83.99	85.88
GOTH	[20]	5.52	588	4213	4206	35,917	4.50	12.02	0.034	0.084	52.83	56.69	84.77	88.77
	Ours-53%	0.92	334	4558	1927	16,870	5.34	12.04	0.039	0.191	52.54	56.35	85.09	89.42

shows the comparison with [20] on UK, Goth and NY with a 2.5D coarse proxy.

On all three scenes, our precision is close to the [20]. For LComp, it is ahead of us except for the UK scene. Considering the large number of images it generates, it is reasonable that it achieves better quality on some details. We obtain slightly better results on UK scene which is relatively regular. For GComp, our completeness is slightly higher on all three scenes, indicating that our method is close to [20] in model quality and even slightly better. Note that under the same GSD, its images are nearly two times ours, resulting in a long-time reconstruction process. Furthermore, because the viewpoints from polygon primitives are regularly arranged, our path length, time and energy consumption are smaller, even less than half by comparison. Compared with [20], we consume less time during path generation, UAV flight and reconstruction with comparable model quality.

• Comparison with [30,31]

Additionally, School and Town in Urban Scene3D are adopted to compare with the methods of [30,31], respectively. For a fair comparison, we use the same camera with a pixel size of 3.9 microns, resolution of 6000 × 4000 and focal length of 20.30 mm, and we keep the average GSD consistent with them when generating the path. We choose two overlap ratios of 53% and 60%. In the aerial triangulation stage, the capturing positions of the images are all imported.

Figure 11 shows the viewpoint distribution and reconstructed scene resolution generated on Town scene with two proxies. It is directly visualized by Context Capture after the aerial triangulation, where color represents the size of one pixel on the ground, that is, the GSD. For the viewpoint distribution, that of [30] as well as its path are very regular, but the shooting distance is significantly farther. Therefore, with a 3D inter proxy, the same number of images results in a much lower resolution than [31] and ours. Our image distribution is more regular than that of [31], as polygon and line primitives are utilized to generate regular viewpoint arrays, especially with a 2.5D coarse proxy. As for the resolution, ours is significantly more uniform than the other two methods and maintains an excellent level, especially with 3D inter proxy, as shown in Figure 11f. Our resolution value looks higher with a 2.5D coarse proxy, as shown in Figure 11a–c. However, switching the perspective, as shown in Figure 12, our resolution is still close to [31] or even better. In addition, the number of our images is much smaller than that of [30,31].

Table 5. Qualitative comparison with [30] on School and Town.

Proxy	Method	#Imgs	Length	Time	Energy	RMSE	GSD	Precision		Recall		F-Score	Index
								0.1 m	90%	0.1 m	90%
			↓m	↓s	↓J	↓mm	↓mm	↑%	↓m	↑%	↓m	↑%
2.5D Coarse (School)	[30]-high	570	4485	2120	18,504	6.05	12.86	82.62	0.316	48.20	1.814	60.89	1
	[30]-low	330	4294	1587	14,013	5.97	11.98	82.96	0.325	46.64	1.893	59.71	2
	Ours-53%-0.7	220	3404	1300	11,434	3.68	8.13	85.79	0.212	46.47	2.041	60.29	3
3D Inter (School)	[30]-high	570	4521	2105	18,385	8.73	15.70	78.74	0.525	48.43	1.678	59.97	4
	[30]-low	330	4239	1560	13,776	8.70	15.96	79.95	0.413	46.81	1.853	59.05	5
	Ours-53%-0.8	328	4238	1545	13,474	3.71	8.03	86.55	0.191	50.37	1.710	63.68	6
2.5D Coarse (Town)	[30]-high	511	3638	1840	16,024	5.44	12.52	90.11	0.097	69.32	0.805	78.36	7
	[30]-low	217	3457	1197	10,608	6.08	12.76	90.09	0.098	68.50	0.841	77.83	8
	Ours-53%-0.7	220	2860	1085	9540	3.87	8.51	93.04	0.056	71.07	0.778	80.58	9
3D Inter (Town)	[30]-high	428	3502	1641	14,331	5.66	12.65	89.76	0.106	68.95	0.825	77.99	10
	[30]-low	258	3115	1212	10,683	6.04	12.87	89.75	0.106	68.85	0.800	77.92	11
	Ours-53%-0.7	242	2626	1098	9615	3.50	7.32	94.00	0.053	71.31	0.769	81.10	12

and

Table 6. Qualitative comparison with [31] on School and Town.

Proxy	Method	#Imgs	RMSE	GSD	Precision		Recall		F-Score	Index
					0.1 m	90%	0.1 m	90%
			↓mm	↓mm	↑%	↓m	↑%	↓m	↑%
2.5D Coarse (School)	[31]-high	570	4.18	9.16	83.97	0.304	51.2	1.602	63.61	1
	[31]-low	330	4.24	8.68	86.85	0.179	48.46	1.963	62.21	2
	Ours-60%-0.7	272	3.72	8.15	85.69	0.226	49.39	1.744	62.67	3
3D Inter (School)	[31]-high	595	3.79	8.03	15.39	1.819	9.86	2.136	12.02	4
	[31]-low	342	3.99	8.26	87.14	0.157	48.2	1.992	62.07	5
	Ours-60%-0.7	400	3.15	7.02	87.50	0.163	50.52	1.639	64.06	6
2.5D Coarse (Town)	[31]-high	511	4.02	8.77	92.01	0.061	72.02	0.721	80.80	7
	[31]-low	217	4.36	8.61	92.74	0.059	70.77	0.763	80.28	8
	Ours-60%-0.7	241	3.81	8.56	92.85	0.057	71.70	0.747	80.92	9
3D Inter (Town)	[31]-high	428	2.78	4.48	95.29	0.050	72.02	0.727	82.04	10
	[31]-low	259	2.46	4.11	84.32	2.07	66.75	0.926	74.51	11
	Ours-60%-0.45	426	2.33	4.79	94.90	0.050	73.28	0.700	82.70	12

show the comparison between ours and [30,31] on School and Town scenes. The designation Ours-53%-0.7 refers to generating a path with an overlap ratio of 53% and a distance corresponding to a GSD of 0.7 cm. For the coarse proxy, the difference between the designed GSD and the average GSD calculated by aerial triangulation is relatively large due to its simplicity, as shown in rows (3, 9) of Table 5 and Table 6. However, the difference in GSD is tiny for the inter proxy, as shown in rows (6, 12) of Table 5 and Table 6, indicating that the path we designed can reach the expected resolution with a detailed proxy.

From Table 5, regardless of scene and proxy, the precision and recall of [30] are significantly behind ours with even more images. Due to its flight height, the average GSD is significantly larger, resulting in low resolution of the building side, as shown in Figure 11 and Figure 12. Moreover, the length of its path is not ahead and even consumes more time and energy with a further shooting distance, even if a continuous path is connected.

From Table 6, compared with [31], the RMSE of our reprojection is the lowest in both scenes, keeping close cases in terms of precision and recall. Overall, except for the 2.5D coarse proxy on School scene, our method is slightly ahead in the F-score. Figure 13 visually shows the comparison between the details of the reconstructed model with [31]. Compared to it, we obtain better visual fidelity of these details, since we generate sample points for building corners and fragmented details in a more targeted manner.

Overall, considering the evaluation of path quality and model quality, we obtain a higher quality with less energy consumption and fewer images, which illustrates that our method reaches the state of the art.

4.4. Field Test

Experiments on three real scenes are also conducted. The viewpoints generated from different primitives and unreconstructable sample points during the optimization process are shown in Figure 14. Regularly arranged viewpoints generated from polygon and line primitives, which can basically realize the reconstructability of flat areas, are obviously seen in Figure 14a,b. With only viewpoints generated from polygon and line primitives, the unreconstructable sample points mainly concentrate in fragmented areas. After the whole optimization, there are only a few sample points that cannot satisfy the reconstructability, which are mainly located in narrow and slender areas.

We compare our method with the regular baseline pattern in oblique photogrammetry on the Museum scene, which flies at a fixed altitude with a five-direction camera over the scene to capture the images. The DJI P4RTK is adopted to capture images, which is a UAV intended for the mapping field mounted with a camera with a pixel size of 2.4 microns, resolution of 5472 × 3648 and focal length of 8.8 mm. To keep nearly the same number of images, the desired GSD for the oblique photography and ours is set as 1.2 cm and 0.6 cm, respectively. The point clouds captured from 21 sites by FARO FOECUS S350 are adopted as ground truth to evaluate the precision and recall. Figure 15 shows the visual fidelity comparison of some details.

Table 7. Quantitative comparison with oblique on Museum scene.

Scene	Method	#Imgs	GSD	Precision		Recall
				0.1 m	0.5 m	0.1 m	0.5 m
			mm	↑%	↑%	↑%	↑%
Museum	Oblique-1.2 cm	270	15.23	50.43	70.46	54.56	72.24
	Ours-0.6 cm	274	6.15	62.75	79.16	70.52	91.64

lists the quantitative comparison in the final reconstructed quality. We can see that our method is far ahead of oblique photography in precision and recall. It is mainly because we adopt a close photographic way that can adjust the orientation to obtain more information about the details in the scene.

5. Discussion

Compared with other methods, the most prominent feature of our method is the efficient utilization of geometric primitives extracted from the proxy, based on which the viewpoint optimization for the reconstruction of the whole scene is transferred into the reconstructable optimization problem of each primitive. Moreover, the geometric primitive has various positive effects on subsequent steps.

First, a more representative evaluation of reconstructability is realized by primitives. Every triangle on the proxy surface is segmented into a special primitive, and its reconstructability is measured. Compared with uniform sampling of [16,20,24,30,31,32,34] which easily ignores fragmented details and edges, we fully consider the coverage of large areas in the scene based on polygon primitives, and then consider the reconstructability of every fragmented detail in the scene based on line and point primitives.

Second, two viewpoint optimization strategies are performed on different primitives, which greatly accelerates the optimization process, as shown in Table 4. An initial set of viewpoints that can almost satisfy the reconstructability of polygon and line primitive is generated directly at a preset overlap ratio, which is considered as an initial value for global optimum with no further optimization, as shown in Table 3 and Figure 14b. For point primitives, we establish a submodular formula to augment a few viewpoints to improve the reconstruction of details and minimize redundancy. The optimization is only performed on the viewpoints generated by the point primitives. Changing the GSD of the desired model and images does not affect the number of viewpoints for point primitives, thus the computation time consumed is generally unchanged. Compared with the methods of [20,30,31,34], our SGP-G method realizes a near-global optimum with almost no optimization.

Finally, for flat areas in the scene, such as building facades and ground, regularly arranged viewpoints with the same orientation are generated from polygons and line primitives, as shown in Figure 11c and Figure 14a. Compared with [20,31], our viewpoints prefer a more regular arrangement and smoother angle change of the final connected path, resulting in less consumption of time and energy, as shown in Table 4 and Table 5.

In all, our method achieves viewpoint selection and optimization in a relatively simple way and exhibits extraordinary performance in all aspects. There are also some limitations to our method. First, the mesh segmentation method is limited in many situations. We currently use the bottom-up method of plane segmentation to extract primitives, which requires correct triangular information of proxy. In future work, we hope to use a top-down segmentation method and iteratively split the bounding box to obtain a more regular representation of the proxy, which is similar to the LOD2 model in GIS. Second, cross-direction and multi-resolution combined photography is not considered. Although the workload of optimization computation increases when considering them, there is lower information entropy in the image of the cross-direction shooting, which can better capture the side information of the scene. Furthermore, low-resolution images are also more suitable for reconstructing weak texture areas. We hope to conduct in-depth research on how to capture images with multi-resolution and multi-orientation to reconstruct higher-quality models in future work.

6. Conclusions

In this article, we propose a surface geometric primitive-guided UAV path planning method (SGP-G) for high-quality image-based 3D reconstruction. There are significant improvements in three aspects of our method based on the extracted primitives. First, the generated primitives with consistent normal vector assist in comprehensively assessing the reconstructability of the whole scene. Second, an initial viewpoint set which is considered a better initial set for global optimum is generated to satisfy the reconstructability of most areas with an appropriate overlap ratio, which greatly speeds up the optimization process. Third, a well-aligned path is generated with smaller turns and less time and energy consumption. The experiments show that our method is applicable to different detailed proxies. Compared with the state of the art, we can plan photography paths with higher efficiency, and finally achieve 3D reconstruction with equal or higher quality with fewer images. Our method provides a fast UAV 3D path planning method for image-based 3D reconstruction, which is useful for the acquisition of high-quality models in engineering.