Multi-Constrained Optimization Method of Line Segment Extraction Based on Multi-Scale Image Space

Image-based line segment extraction plays an important role in a wide range of applications. Traditional line segment extraction algorithms focus on the accuracy and efficiency, without considering the integrity. Serious line segmentation fracture problems caused by image quality will result in poor subsequent applications. To solve this problem, a multi-constrained line segment extraction method, based on multi-scale image space, is presented. Firstly, using Gaussian down-sampling with a classical line segment detection method, a multi-scale image space is constructed to extract line segments in each image scale and all line segments are projected onto the original image. Then, a new line segment optimization and purification strategy is proposed with the horizontal and vertical distances and angle geometric constraint relationships between line segments to merge fracture line segments and delete redundant line segments. Finally, line segments with adjacent positions are optimized using the grayscale constraint relationship, based on normalized cross-correlation similarity criterion for realizing the second optimization of fracture line segments. Compared with mainstream line segment detector and edge drawing lines methods, experimental results (i.e., indoor, outdoor, and aerial images) indicate the validity and superiority of our proposed methods which can extract longer and more complete line segments.


Introduction
The line feature is an important part of an image's geometric information and plays a crucial role in photogrammetry and remote sensing [1], three-dimensional (3D) urban modeling [2,3], computer vision and robot navigation positioning [4,5], and so on.Simultaneously, the importance of line elements is emphasized in cartography and the even more user-oriented discipline, spatial cognition [6,7].The line feature has the following advantages as a more advanced feature than the point feature: (1) It has rich structural information for expressing edge information of 3D objects, such as structured long line segments, which causes more processing errors (error matching and error reconstruction).Currently, there are few methods to solve this line segmentation fracture problem.To alleviate this common problem, this paper proposes a multi-constrained line segment extraction optimization algorithm based on multi-scale image space, namely MSLines, including MSLSD and MSEDLines (MS is an abbreviation of Multi-Scale), which reduces the influence of grayscale distortion on feature extraction using the idea of fuzzy processing.
Figure 1 shows the overall methodology presented in this paper, which is divided into three main modules.The first module consists of constructing the line segment extraction model.A multi-scale image pyramid was constructed based on the Gaussian down-sampling and line segments of each image layer were extracted by the traditional line segment detection algorithm.All line segments were projected onto the original image and stored in line vector spaces.The second module was the optimization of the purification strategy.A set of line segment optimization and purification methods were designed with geometric constraints; line segments were merged or deleted according to the distance and angle relationships between line segments.The third module was second optimization.The virtual straight line was constructed and the secondary merge optimization of the line segment was realized based on the grayscale constraint relationship of the normalized cross correlation (NCC) measurement.Compared with several mainstream line segment extraction algorithms, i.e., LSD and EDLines, the proposed method could alleviate the segmental fracture effect of line segments and eliminate line segments' redundant problem.
ISPRS Int.J. Geo-Inf.2019, 8, 183 3 of 25 common problem, this paper proposes a multi-constrained line segment extraction optimization algorithm based on multi-scale image space, namely MSLines, including MSLSD and MSEDLines (MS is an abbreviation of Multi-Scale), which reduces the influence of grayscale distortion on feature extraction using the idea of fuzzy processing.
Figure 1 shows the overall methodology presented in this paper, which is divided into three main modules.The first module consists of constructing the line segment extraction model.A multiscale image pyramid was constructed based on the Gaussian down-sampling and line segments of each image layer were extracted by the traditional line segment detection algorithm.All line segments were projected onto the original image and stored in line vector spaces.The second module was the optimization of the purification strategy.A set of line segment optimization and purification methods were designed with geometric constraints; line segments were merged or deleted according to the distance and angle relationships between line segments.The third module was second optimization.The virtual straight line was constructed and the secondary merge optimization of the line segment was realized based on the grayscale constraint relationship of the normalized cross correlation (NCC) measurement.Compared with several mainstream line segment extraction algorithms, i.e., LSD and EDLines, the proposed method could alleviate the segmental fracture effect of line segments and eliminate line segments' redundant problem.The remainder of this study is organized as follows.Section 2 describes the principle of multiscale image space line segment extraction method.This section also introduces a new line segment optimization and purification method based on geometric constraints.Also, a second optimization method of grayscale constraint is presented.In Section 3, three sets of real data, i.e., indoor, outdoor, and aerial images are tested to analyze superiority of our proposed line segment extraction method compared with the mainstream method.Section 4 discusses the selection of parameters in the model The remainder of this study is organized as follows.Section 2 describes the principle of multi-scale image space line segment extraction method.This section also introduces a new line segment optimization and purification method based on geometric constraints.Also, a second optimization method of grayscale constraint is presented.In Section 3, three sets of real data, i.e., indoor, outdoor, and aerial images are tested to analyze superiority of our proposed line segment extraction method compared with the mainstream method.Section 4 discusses the selection of parameters in the model construction for various resolution images and analyses the accuracy of the proposed method.Section 5 presents conclusions and possible further studies.

Line Segment Detection Method
At present, Von Gioi's LSD line segment extraction algorithm is one of the most widely used methods.The principle of several mainstream line segment extraction algorithms is similar to LSD. Figure 2 shows the workflow of the LSD algorithm.
ISPRS Int.J. Geo-Inf.2019, 8, 183 4 of 25 construction for various resolution images and analyses the accuracy of the proposed method.Section 5 presents conclusions and possible further studies.In the line detection process, the input image was first down-sampled to 80% of the original image by the Gaussian down-sampling method to weaken the sawtooth effect of image, as well as to maintain a balance between the sawtooth effect and image blur.Next, a 2 × 2 gradient template was given to calculate the gradient direction (level-line angle (LLA)) and amplitude (intensity) of all pixels after image down-sampling.Depending on the pixel gradient amplitude, 1024 containers were opened and all image pixels were pseudo sorted from high to low order.Simultaneously, a state list was established and the initial state of all pixels was set to UNUSED.The state of pixel labels whose gradient amplitude was smaller than the threshold "ρ" was changed to USED, to represent an area where the image is flat or the gradient changes slowly; this means it did not participate in the line support region (LSR) and rectangle construction.Then, an UNUSED pixel from the sorted list was selected as a seed point.The region growing algorithm was used to generate LSR with the seed point.The seed point and LLA were continuously updated until the current point could not meet the threshold requirement.The final regional direction angle θreg was achieved.The rectangle and the number of false alarms (NFA) was calculated according to θreg and finally, line segments were determined after traversing all UNUSED seed points.

Model Construction
A point can only belong to certain line segment with a traditional LSD algorithm where extracted line segments cannot intersect with each other.If two line segments intersect, they break and split into four short line segments.Additionally, the algorithm is susceptible to pixel grayscale gradient mutation during the regional growth period, so that the difference between the current point LLA and the regional direction angle exceeds the growth threshold, resulting in segmentation fracture of the line segments.The EDLines algorithm uses the Edge Drawing operator for feature extraction.The speed of this method is fast, but the line segmentation fracture problem is not improved.Therefore, traditional line segment extraction algorithms should be further optimized to solve the segmentation In the line detection process, the input image was first down-sampled to 80% of the original image by the Gaussian down-sampling method to weaken the sawtooth effect of image, as well as to maintain a balance between the sawtooth effect and image blur.Next, a 2 × 2 gradient template was given to calculate the gradient direction (level-line angle (LLA)) and amplitude (intensity) of all pixels after image down-sampling.Depending on the pixel gradient amplitude, 1024 containers were opened and all image pixels were pseudo sorted from high to low order.Simultaneously, a state list was established and the initial state of all pixels was set to UNUSED.The state of pixel labels whose gradient amplitude was smaller than the threshold "ρ" was changed to USED, to represent an area where the image is flat or the gradient changes slowly; this means it did not participate in the line support region (LSR) and rectangle construction.Then, an UNUSED pixel from the sorted list was selected as a seed point.The region growing algorithm was used to generate LSR with the seed point.The seed point and LLA were continuously updated until the current point could not meet the threshold requirement.The final regional direction angle θ reg was achieved.The rectangle and the number of false alarms (NFA) was calculated according to θ reg and finally, line segments were determined after traversing all UNUSED seed points.
A point can only belong to certain line segment with a traditional LSD algorithm where extracted line segments cannot intersect with each other.If two line segments intersect, they break and split into four short line segments.Additionally, the algorithm is susceptible to pixel grayscale gradient mutation during the regional growth period, so that the difference between the current point LLA and the regional direction angle exceeds the growth threshold, resulting in segmentation fracture of the line segments.The EDLines algorithm uses the Edge Drawing operator for feature extraction.The speed of this method is fast, but the line segmentation fracture problem is not improved.Therefore, traditional line segment extraction algorithms should be further optimized to solve the segmentation fracture effect.

Multi-Scale Image Space Line Segment Extraction Method
In order to overcome the segmentation fracture problem of line segment extraction, the proposed method combines the idea of fuzzy processing by down-sampling the original image to produce a pyramid-like model of multi-scale image space.
To perform this method, the specific procedure was as follows: (1) According to the actual image resolution and processing requirements, the standard deviation factor (scale space factor) of Gaussian filter kernel function σ was set and the maximum down-sampling number t max was set; (2) Gaussian kernel convolution on image I to achieve Gaussian blur of the image was performed; (3) image-based down-sample on image-scale space factor σ (σ is generally set to 0.5) was carried out; (4) to detect line segment, the traditional method was used for this layered image and the line segments were stored in the corresponding line vector spaces; (5) steps (2 -4) were repeated until the maximum number of down-sampling was reached.Specifically, the Gaussian-blurred image I (u, v, σ) is first generated with Equation (1).
where G(u, v, σ) is the Gaussian kernel function, I(u, v) is the image I, and * is the convolution operation on the image pixel (u, v).
Based on scale space factor σ, the original image was down-sampled to obtain a low-resolution image J.The above operation was repeated to continue down-sampling of image J until the maximum down-sampling number t max was reached.Finally, an N-layer image model was created (N = t max + 1), forming a pyramid-like multi-scale image space model.
To detect line segments, traditional line segment detection method was used for each image layer in scale space and line segment vector spaces corresponding to the number of images were opened.Line segments extracted in each scale image were stored in a line segment vector LineVec (k) = {}, where k represents the k-scale image of line segment vector space and the maximum value of k is N.Moreover, line segment endpoints' coordinates and length information were stored in line segment class line end , length (i) , where k represents kth line segment vector space and i represents ith line in the image.Line segment extraction and storage process is shown in Figure 3.

Multi-Scale Image Space Line Segment Extraction Method
In order to overcome the segmentation fracture problem of line segment extraction, the proposed method combines the idea of fuzzy processing by down-sampling the original image to produce a pyramid-like model of multi-scale image space.
To perform this method, the specific procedure was as follows: (1) According to the actual image resolution and processing requirements, the standard deviation factor (scale space factor) of Gaussian filter kernel function σ was set and the maximum down-sampling number tmax was set; (2) Gaussian kernel convolution on image I to achieve Gaussian blur of the image was performed; (3) image-based down-sample on image-scale space factor σ (σ is generally set to 0.5) was carried out; (4) to detect line segment, the traditional method was used for this layered image and the line segments were stored in the corresponding line vector spaces; (5) steps (2)(3)(4) were repeated until the maximum number of down-sampling was reached.Specifically, the Gaussian-blurred image ( ) ' , , ( )/2 where ( ) G u v σ is the Gaussian kernel function, ( ) , I u v is the image I , and * is the convolution operation on the image pixel ( ) Based on scale space factor σ, the original image was down-sampled to obtain a low-resolution image J.The above operation was repeated to continue down-sampling of image J until the maximum down-sampling number tmax was reached.Finally, an N-layer image model was created (N = tmax + 1), forming a pyramid-like multi-scale image space model.
To detect line segments, traditional line segment detection method was used for each image layer in scale space and line segment vector spaces corresponding to the number of images were opened.Line segments extracted in each scale image were stored in a line segment vector , where k represents kth line segment vector space and i represents ith line in the image.Line segment extraction and storage process is shown in Figure 3. Figure 3 shows that the traditional line segment detector extracts line segments for each scale image and stores segments in vector space.If detector extracts m1 and m2 line segments in the first and second layers of the image, respectively, then the line segment method extracts mN line segments  in the Nth layer of the image.The expression of multi-scale image space line feature vector can be represented with Equation (2).

Line Segment Projection
Line segments extracted from different scale images were all projected to the original image and stored in a new opened memory vector space.Particularly, line segments extracted from the original image were directly stored in a new vector.For a line segment extracted after down-sampling of image, the line (k) * i needed to be projected with Equation (3) in case of known scaling factor σ and the number of the image down-sampled times t (t = k − 1).Finally, a vector set SumLineVec of all line segments extracted by traditional method in the multi-scale image space was achieved with Equation (4).The vector dimension of SumLineVec is end , length (i) , where, * m 2 . . .

Optimization and Purification Based on Geometric Constraints
After line segments extracted by the multi-scale image space were projected to the original image, severe redundancy.Overlapping.and clutter problems of line segments occur.Many line segments are meaningless, so the entire set of line segments needs to be optimized and purified.As a result, the down-sampled image with low resolution can only extract a few segments, which brings extraction error.In this subsection, a line segment optimization and purification strategy is proposed, which optimizes line segment extraction results through line segment geometric constraint relationships.The optimized line segments have longer length and higher integrity, while alleviating the redundancy problem.
Before optimization and purification, it is necessary to describe and determine constraint relationships between line segments.As shown in Figure 4, the geometric constraint relationships between line segments are defined as follows: the down-sampled image with low resolution can only extract a few segments, which brings extraction error.In this subsection, a line segment optimization and purification strategy is proposed, which optimizes line segment extraction results through line segment geometric constraint relationships.The optimized line segments have longer length and higher integrity, while alleviating the redundancy problem.
Before optimization and purification, it is necessary to describe and determine constraint relationships between line segments.As shown in Figure 4, the geometric constraint relationships Based on the geometric constraint relationships in Figure 4, we define horizontal, vertical, and angle thresholds to determine whether line segments need to be optimized and purified.We define 2 , length (1) and 2 , length (2) , respectively, as reference and determined line segments that needed to be judged.The distance between midpoints of two and the average distance between the endpoints of two segments is The angle between two lines is θ.

Distance Geometric Constraint
The distance geometric constraint relationships can be divided into horizontal and vertical distance relationships.Considering the horizontal distance relationship (Figure 4a), the distance D 1 between the midpoints of two line segments is used to judge whether they are related and whether l 2 needs to be retained or deleted.If D 1 > length (1) + length (2) /2, l 2 and l 1 are independent.l 2 is retained and stored as one of the reference line segments in the next judgment process.If D 1 ≤ length (1) + length (2) /2, l 2 is related to l 1 and l 2 continues to be judged by the vertical distance constraint relationship.Considering vertical distance constraint (Figure 4b), if D 2 ≤ ξ d (where ξ d is the vertical threshold), l 2 is strongly correlated with l 1 and we then merge l 1 and l 2 .If ξ d < D 2 ≤ 3ξ d , l 2 is a redundant or invalid segment and needs to be deleted.If D 2 > 3ξ d , l 2 and l 1 are independent and l 2 is retained and stored as one of the reference line segments in the next judgment.Here we set ξ d = 1(pixel), because the resolution of the human eye is 1 pixel, and the error of more than 1 pixel is discernible.
In this paper, as a part, we also propose an optimization method for merging short line segments based on endpoint projection transformation.The reason for selecting optimization method is that the least squares fitting method has larger extraction errors for extracted line segments from low-resolution images than from original images by using the multi-scale image space model.The least squares fitting is an error-averaging method to minimize the sum of squared errors.If the least squares fitting strategy was used here to merge line segments, it would bring more merging errors.The line segment merging method based on endpoint projection transformation is shown in Figure 5. Based on the geometric constraint relationships in Figure 4, we define horizontal, vertical, and angle thresholds to determine whether line segments need to be optimized and purified.We define , respectively, as reference and determined line segments that needed to be judged.The distance between midpoints of two segments is and the average distance between the endpoints of two segments is ( ) . The angle between two lines is θ.

Distance Geometric Constraint
The distance geometric constraint relationships can be divided into horizontal and vertical distance relationships.Considering the horizontal distance relationship (Figure 4a), the distance D1 between the midpoints of two line segments is used to judge whether they are related and whether 2 l needs to be retained or deleted.If retained and stored as one of the reference line segments in the next judgment process.If In this paper, as a part, we also propose an optimization method for merging short line segments based on endpoint projection transformation.The reason for selecting optimization method is that the least squares fitting method has larger extraction errors for extracted line segments from lowresolution images than from original images by using the multi-scale image space model.The least squares fitting is an error-averaging method to minimize the sum of squared errors.If the least squares fitting strategy was used here to merge line segments, it would bring more merging errors.The line segment merging method based on endpoint projection transformation is shown in Figure 5.The origin O of an image coordinate system is located in the upper left corner of image.l 1 was a reference line segment extracted from the original image and the two endpoints were p 1 = (u 1 , v 1 ) and p 3 = (u 3 , v 3 ).l 2 was an error-containing line segment in the multi-scale image space and its two endpoints were p 2 = (u 2 , v 2 ) and p 4 = (u 4 , v 4 ), whereas p = (u, v, 1) were homogeneous coordinates corresponding to these endpoints.We projected error-containing line segments onto the reference line segment, as well as obtained projected coordinate values p * 2 = u * 2 , v * 2 and p * 4 = u * 4 , v * 4 at both endpoints of the error-containing line segment.Equation (6) gives the endpoint projection formula: where A, B, and C are the reference line equation Au + Bv + C = 0 parameters, which can be taken by known two endpoints image coordinates.Thereby, the projected coordinate value p * = (u * , v * ) of endpoints for the error-containing line segment can be achieved.Due to difference between positive and negative slopes of the reference line segment, the method of line segment merging has two possibilities regarding endpoint transformations.We can finally get endpoints p f irst = u f irst , v f irst and p last = (u last , v last ) of the merged line segment using Equation (7).

Angle Geometric Constraint
When l 2 and l 1 intersect each other at the extension line (Figure 4c), there is no overlap between l 1 and l 2 due to that line segments are not infinitely long lines.Moreover, l 1 and l 2 don't need to be judged by the angle geometric constraint relationship, but only the vertical distance geometric relationship is used for judgment.When l 2 intersects with l 1 at the non-extended line (Figure 4d), the angle constraint relationship is also judged for the deletion of the pseudo-line after completing the distance constraint judgment.If θ < ξ θ (where ξ θ is the angle threshold, ξ θ = 5 • ), we think that l 2 is a false line and delete it.Otherwise, we retain l 2 .The expression of angle θ is given in Equation (8).

Optimization Based on Grayscale Constraint
After completing the geometric constraint relationship judgment, we can get optimized line segments.Grayscale constraint was performed on the first optimized line segment to realize the second optimization (Figure 6).
Furthermore, it must be mentioned that direction information of all line segments is traverse after geometric optimization.When any two line segments, l 1 and l 2 , have the same direction (the angle between the two lines θ < 1 • ) and the distance between them is less than 20 pixels, which is suitable for the current typical megapixel/medium-resolution image in the experiment section of paper, these two line segments are temporarily merged into one virtual line segment.Furthermore, it must be mentioned that direction information of all line segments is traverse after geometric optimization.When any two line segments, 1 l and 2 l , have the same direction (the angle between the two lines θ < 1º) and the distance between them is less than 20 pixels, which is suitable for the current typical megapixel/medium-resolution image in the experiment section of paper, these two line segments are temporarily merged into one virtual line segment.
In addition to this, grayscale points on the central positions of AB (midpoint of 1 l ), BC, and CD segments (midpoint of 2 l ) on virtual line segments are used as center grayscale to open a 3 × 3 matching window with NCC similarity judgment.NCC is often used to compare the similarity of two images and realize the matching features.Here, we propose an extended NCC that compares the similarities of the three windows (small patches) opened on one image.
(where ψ is grayscale threshold among three matching windows, ψ = 0.8), the line segment has high grayscalelevel similarity that means a virtual line segment is an optimized line segment.In this case, we combine 1 l and 2 l to get a real line segment.If N C C ψ < , the virtual line segment is not a real merging line segment.In this case, 1 l and 2 l are retained and the virtual line segment is released.The specific expression of NCC is given in Equation ( 9). ( where Gp and Gq represent grayscale windows at positiond p and q on virtual line segment, respectively, and N represents the size of the grayscale window, which is 3 here.

Overall Procedure of Optimization
When the kth line segment in the SumLineVec line feature set was processed (k < dim), the previous k-1 line segments were used as reference lines for judgment.In this article, vertical distance constraint, angle constraint, and grayscale constraint thresholds were set respectively by the empirical method decision above.
According to the constraint relationships between line segments as described above, the specific In addition to this, grayscale points on the central positions of AB (midpoint of l 1 ), BC, and CD segments (midpoint of l 2 ) on virtual line segments are used as center grayscale to open a 3 × 3 matching window with NCC similarity judgment.NCC is often used to compare the similarity of two images and realize the matching features.Here, we propose an extended NCC that compares the similarities of the three windows (small patches) opened on one image.If NCC ≥ ψ (where ψ is grayscale threshold among three matching windows, ψ = 0.8), the line segment has high grayscale-level similarity that means a virtual line segment is an optimized line segment.In this case, we combine l 1 and l 2 to get a real line segment.If NCC < ψ, the virtual line segment is not a real merging line segment.In this case, l 1 and l 2 are retained and the virtual line segment is released.The specific expression of NCC is given in Equation (9).
where G p and G q represent grayscale windows at positiond p and q on virtual line segment, respectively, and N represents the size of the grayscale window, which is 3 here.

Overall Procedure of Optimization
When the kth line segment in the SumLineVec line feature set was processed (k < dim), the previous k-1 line segments were used as reference lines for judgment.In this article, vertical distance constraint, angle constraint, and grayscale constraint thresholds were set respectively by the empirical method decision above.
According to the constraint relationships between line segments as described above, the specific procedure of line segment optimization and purification was as follows: (1) The line feature vector space R 1 and R 2 were opened for storing line segments that were optimized by geometric and grayscale constraints.The initial state was R 1 = {} and R 2 = {}.
(2) Line segments extracted from the original image in SumLineVec were taken as reference lines and directly taken into the optimized set R 1 .
(3) Line segments extracted from other scale image space in SumLineVec were judged as determined lines with reference lines in the optimized set based on the geometric distance and angle constraint relationships.If the determined line satisfied the optimization and purification situation, R 1 would be updated.Otherwise, the determined line would be deleted.
(4) When all lines in SumLineVec were processed, an optimized set R 1 of line segments was obtained.
(5) For any two lines in R 1 , virtual line segments were constructed by directional characteristics, where the grayscale constraint relationship was used for judgement.If two lines satisfied the grayscale constraint optimization situation, R 2 was optimized.Otherwise, those lines that do not need to be optimized were directly stored to R 2 .
(6) The optimization set R 2 was purified by geometric constraint as a purification criterion.When geometric distance between line segments satisfies the culling condition, it was considered that there was a redundant line segment and we deleted the error.Also, when the angle distance between lines satisfied the culling condition, it was considered that there was a pseudo-line, which we deleted.Finally, we achieved the resulted line segment feature set R 2 * .

Experiments and Analysis
Experiments were carried out using three data sets, each with three images, i.e., indoor, outdoor, and aerial images, to verify the effectiveness of the proposed method, as shown in Figure 7.Our method was implemented in C++ and OpenCV.It was executed on a personal computer with Intel (R) Core (TM) i7-7500U 2.70 GHz CPU and 8.0 GB RAM.
ISPRS Int.J. Geo-Inf.2019, 8, 183 10 of 25 (2) Line segments extracted from the original image in SumLineVec were taken as reference lines and directly taken into the optimized set 1 R .
(3) Line segments extracted from other scale image space in SumLineVec were judged as determined lines with reference lines in the optimized set based on the geometric distance and angle constraint relationships.If the determined line satisfied the optimization and purification situation, 1 R would be updated.Otherwise, the determined line would be deleted.
(4) When all lines in SumLineVec were processed, an optimized set 1 R of line segments was obtained.
(5) For any two lines in 1 R , virtual line segments were constructed by directional characteristics, where the grayscale constraint relationship was used for judgement.If two lines satisfied the grayscale constraint optimization situation, 2 R was optimized.Otherwise, those lines that do not need to be optimized were directly stored to When geometric distance between line segments satisfies the culling condition, it was considered that there was a redundant line segment and we deleted the error.Also, when the angle distance between lines satisfied the culling condition, it was considered that there was a pseudo-line, which we deleted.
Finally, we achieved the resulted line segment feature set * 2 R .

Experiments and Analysis
Experiments were carried out using three data sets, each with three images, i.e., indoor, outdoor, and aerial images, to verify the effectiveness of the proposed method, as shown in Figure 7.Our method was implemented in C++ and OpenCV.It was executed on a personal computer with Intel (R) Core (TM) i7-7500U 2.70 GHz CPU and 8.0 GB RAM.(g) (h) (i) As for the nine selected megapixel resolution data sets, we set the maximum down-sampling number tmax = 2 and scale space factor σ = 0.5 based on the parameter setting of discussion section.Therefore, a three-layered multi-scale image space model was formed.For the three sets of nine images, each image consists of three images with different resolutions arranged from high to low.Moreover, the merits of the proposed line segment extraction method (referred to as MSLSD and MSEDLines for better description) are mainly discussed from the aspect of number of extracted line segments, extraction time, accuracy rate, total length, average length, and visualization effect of the line segment extraction.Here, the accuracy rate is the ratio of the number of correctly extracted line segments to the total number of line feature extraction [29].Generally, a longer average length of a line segment indicates a higher integrity of the extracted line, whereas a shorter extraction time means the method is better.However, this paper firstly uses the traditional method to extract line segments for multiple image scales and, as a result, extraction time is generally longer than the original method.The proposed method needs a short optimization time, but the overall extraction speed is not greatly affected.This study mainly solves the segmentation fracture problem of line segments and redundant or invalid feature problems.Therefore, it mainly focuses on the average and total lengths of line segments and the visualization effect of extracted line segments on the image.The proposed method is compared with LSD and EDLines methods from the above perspectives in this section.
For these test data, the line segment extraction results in Figure 8-16 are, respectively, shown for conventional LSD, EDLines, and our proposed method, MSLSD, MSEDLines.Moreover, a quantitative index of traditional and proposed methods for line segment extraction in three sets of nine images is presented in Table 1.As for the nine selected megapixel resolution data sets, we set the maximum down-sampling number t max = 2 and scale space factor σ = 0.5 based on the parameter setting of discussion section.Therefore, a three-layered multi-scale image space model was formed.For the three sets of nine images, each image consists of three images with different resolutions arranged from high to low.Moreover, the merits of the proposed line segment extraction method (referred to as MSLSD and MSEDLines for better description) are mainly discussed from the aspect of number of extracted line segments, extraction time, accuracy rate, total length, average length, and visualization effect of the line segment extraction.Here, the accuracy rate is the ratio of the number of correctly extracted line segments to the total number of line feature extraction [29].Generally, a longer average length of a line segment indicates a higher integrity of the extracted line, whereas a shorter extraction time means the method is better.However, this paper firstly uses the traditional method to extract line segments for multiple image scales and, as a result, extraction time is generally longer than the original method.The proposed method needs a short optimization time, but the overall extraction speed is not greatly affected.This study mainly solves the segmentation fracture problem of line segments and redundant or invalid feature problems.Therefore, it mainly focuses on the average and total lengths of line segments and the visualization effect of extracted line segments on the image.The proposed method is compared with LSD and EDLines methods from the above perspectives in this section.
For these test data, the line segment extraction results in Figures 8-16 are, respectively, shown for conventional LSD, EDLines, and our proposed method, MSLSD, MSEDLines.Moreover, a quantitative index of traditional and proposed methods for line segment extraction in three sets of nine images is presented in Table 1.As for the nine selected megapixel resolution data sets, we set the maximum down-sampling number tmax = 2 and scale space factor σ = 0.5 based on the parameter setting of discussion section.Therefore, a three-layered multi-scale image space model was formed.For the three sets of nine images, each image consists of three images with different resolutions arranged from high to low.Moreover, the merits of the proposed line segment extraction method (referred to as MSLSD and MSEDLines for better description) are mainly discussed from the aspect of number of extracted line segments, extraction time, accuracy rate, total length, average length, and visualization effect of the line segment extraction.Here, the accuracy rate is the ratio of the number of correctly extracted line segments to the total number of line feature extraction [29].Generally, a longer average length of a line segment indicates a higher integrity of the extracted line, whereas a shorter extraction time means the method is better.However, this paper firstly uses the traditional method to extract line segments for multiple image scales and, as a result, extraction time is generally longer than the original method.The proposed method needs a short optimization time, but the overall extraction speed is not greatly affected.This study mainly solves the segmentation fracture problem of line segments and redundant or invalid feature problems.Therefore, it mainly focuses on the average and total lengths of line segments and the visualization effect of extracted line segments on the image.The proposed method is compared with LSD and EDLines methods from the above perspectives in this section.
For these test data, the line segment extraction results in Figure 8            It can be seen from the visualization results in Figures 8-16 that the proposed methods MSLSD and MSEDLines alleviated the segmentation fracture effect of line segments.Also, it can be seen from the local area marked by the yellow frames in the nine figures that the line extraction results of conventional LSD and EDLines method were affected by image factors such as grayscale distortion, causing a relatively severe fracture phenomenon.MSLSD and MSEDLines had the effect of optimization and could merge multiple broken short line segments into one complete long segment.Additionally, we can clearly see from Figure 9, Figure 10, and Figure 13 that the proposed methods, MSLSD and MSEDLines, played a role in removing redundancy, eliminating the redundant lines while retaining the key geometric contour lines.From the comparison of Figure 10a,b, it was found that the proposed method MSEDLines could remove some pseudo-lines extracted by algorithm mistakes.
Table 1 shows the quantitative results of line extraction with nine sets of experimental data of indoor, outdoor, and aerial images.Regardless of indoor or outdoor environments or aerial scenes, MSLSD and MSEDLines extract longer lines than the corresponding traditional methods, LSD and EDLines.The longer the average length of line segments, the higher the integrity of extracted line segments on the images.For Indoor 1, the line segment average length of MSEDLines and MSLSD was 85.3 pixels and 85.2 pixels respectively, which is much higher than that of EDLines (63.1 pixels) and LSD (51.3 pixels).Since our methods had the effect of removing line segment redundancy, the number of extracted line segments was less than that of conventional methods.MSEDLines and MSLSD extracted 215 and 238 line segments respectively from Indoor 1, while EDLines and LSD extracted 327 and 445 line segments, respectively.However, the reduction in the number of line segments does not result in the sharp drop in the total length of line segments.Because proposed methods in this paper preserve and optimize the key geometric contour information on the image while removing the redundant line segments, they ensure that the total length does not change much.The extraction time of the four methods of MSEDLines, MSLSD, EDLines, and LSD is not much different, which indicates that the optimization steps in this paper's algorithm did not take much time.The quantification results of line segment extraction of the other eight sets of data have the same trend as Indoor 1.
Generally, line segments extracted by MSLSD and MSEDLines were more complete than that extracted by traditional methods and they also alleviated the line segmentation fracture problem.Many broken line segments were optimized to be combined into a complete line feature, which completely expressed the geometric profile information of the target.Meanwhile, the proposed method eliminated redundant clutter in the line feature detection process.Quantitative results showed that the average length of optimized line segments extracted by the proposed method was greatly improved compared with that extracted by traditional methods.Also, it was found that the accuracy of traditional methods and proposed methods was very high, except for images including the presence of trees and reflections, for example, such as Indoor 1 and Outdoor 2, and the accuracy of the proposed method was generally higher than that of the classical method.The line feature extraction time of all four methods in the paper was similar.The proposed method does not affect the extraction efficiency.

Parameter Setting
This part discusses the selection of problem-oriented parameters in the model construction process, providing the basis for constructing the most suitable multi-scale image space model for various resolution images.The image resolution is different, as is as the maximum number of down-sampling setting, which causes the number of multi-scale image space model layers formed to be different.
In order to get a suitable multi-scale image space for a line feature extraction, the corresponding and reasonable maximum down-sampling number t max should be set according to the image resolution.Regarding the construction of multi-scale image space, this article is consistent with the traditional empirical method.The scale space factor σ is 0.5 and the whole model has a total of N-layered images.To discuss the selection of model parameteres, an outdoor image (front view of the Grand Auditorium) with a resolution of approximatly 10 million pixels (3500 × 2625) was used as an example to down-sample from one to four times.The corresponding multi-scale image space model was constructed and the line features of the image were extracted by MSLSD.The results are shown in Figure 17 and Table 2.As we can see from Figure 17a, a line segment extracted from the original image suffers a severe fracture effect.Using our proposed method to construct a multi-scale image space model, we can effectively alleviate the fracture problem of extracted line segments, as can be seen from Figure 17b-e and Table 2, and the average length of the line segments is greatly improved.The continuity of line segment extracted by MSLSD with fusion of down-sampled images becomes better, but with the increase of down-sampling times, images, particularly those down-sampled four times, do not change the effect of extracted segments much.Moreover, the average length of line segments changed from 103.3 to 110.4 pixels, which shows that the growth trend was not obvious.Therefore, to construct the multi-scale image space model, we set model parameters according to the proposed rules of Table 3.The super-resolution images of more than 10 million pixels were down-sampled at least three times to form more than four layers of the multi-scale image space model.High-resolution images with megapixel resolution were down-sampled three times to form an image pyramid model of four-layered multi-scale image space.Mid-resolution images were down-sampled twice to form a three-layered pyramid-like model.Low-resolution images with a pixel resolution of less than 100,000 pixels only needed to be down-sampled once to form a two-layered scale image space.

Extraction Error Analysis
Taking the indoor corridor image as an example, when MSLSD constructs a multi-scale image space model and uses LSD algorithm to extract line features from down-scaled space, the extraction error of line features on the down-sampling image is larger than that on the original image.As is shown in Figure 18, Gaussian blur makes the image resolution lower; this is an indispensable step of model construction, causing the extraction error to be amplified.
As we can see from Figure 17(a), a line segment extracted from the original image suffers a severe fracture effect.Using our proposed method to construct a multi-scale image space model, we can effectively alleviate the fracture problem of extracted line segments, as can be seen from Figure 17be and Table 2, and the average length of the line segments is greatly improved.The continuity of line segment extracted by MSLSD with fusion of down-sampled images becomes better, but with the increase of down-sampling times, images, particularly those down-sampled four times, do not change the effect of extracted segments much.Moreover, the average length of line segments changed from 103.3 to 110.4 pixels, which shows that the growth trend was not obvious.Therefore, to construct the multi-scale image space model, we set model parameters according to the proposed rules of Table 3.The super-resolution images of more than 10 million pixels were down-sampled at least three times to form more than four layers of the multi-scale image space model.High-resolution images with megapixel resolution were down-sampled three times to form an image pyramid model of fourlayered multi-scale image space.Mid-resolution images were down-sampled twice to form a threelayered pyramid-like model.Low-resolution images with a pixel resolution of less than 100,000 pixels only needed to be down-sampled once to form a two-layered scale image space.

Extraction Error Analysis
Taking the indoor corridor image as an example, when MSLSD constructs a multi-scale image space model and uses LSD algorithm to extract line features from down-scaled space, the extraction error of line features on the down-sampling image is larger than that on the original image.As is shown in Figure 18, Gaussian blur makes the image resolution lower; this is an indispensable step of model construction, In the optimization and purification process, the proposed MSLSD method always takes the line segment on the original image as the reference, ensuring the extraction error from the low-resolution image is not introduced.Figure 19 is the result of MSLSD optimization, where we can see the method In the optimization and purification process, the proposed MSLSD method always takes the line segment on the original image as the reference, ensuring the extraction error from the low-resolution image is not introduced.Figure 19

The Impact of Image Noise
Image noise tends to affect line segment extraction and causes serious line segmentation fracture problems.In order to analyze the influence of the proposed method on noise, we selected an aerial image with a resolution of 1440 × 920 for discussion.In the experimental analysis, the image was down-sampled twice, forming a three-layer multi-scale image space.Simultaneously, we added Gaussian white noise with different mean and variance to the image.We found that, when the noise variance was greater than 0.02, the image quality was seriously affected.Based on this, the traditional and proposed methods under different image noises with means of 0, 0.1, and 0.2 and variances of 0.005, 0.01, and 0.02 were analyzed.The partial line segment extraction result of the noisy image is shown in Figure 20.

The Impact of Image Noise
Image noise tends to affect line segment extraction and causes serious line segmentation fracture problems.In order to analyze the influence of the proposed method on noise, we selected an aerial image with a resolution of 1440 × 920 for discussion.In the experimental analysis, the image was down-sampled twice, forming a three-layer multi-scale image space.Simultaneously, we added Gaussian white noise with different mean and variance to the image.We found that, when the noise variance was greater than 0.02, the image quality was seriously affected.Based on this, the traditional and proposed methods under different image noises with means of 0, 0.1, and 0.2 and variances of 0.005, 0.01, and 0.02 were analyzed.The partial line segment extraction result of the noisy image is shown in Figure 20.

The Impact of Image Noise
Image noise tends to affect line segment extraction and causes serious line segmentation fracture problems.In order to analyze the influence of the proposed method on noise, we selected an aerial image with a resolution of 1440 × 920 for discussion.In the experimental analysis, the image was down-sampled twice, forming a three-layer multi-scale image space.Simultaneously, we added Gaussian white noise with different mean and variance to the image.We found that, when the noise variance was greater than 0.02, the image quality was seriously affected.Based on this, the traditional and proposed methods under different image noises with means of 0, 0.1, and 0.2 and variances of 0.005, 0.01, and 0.02 were analyzed.The partial line segment extraction result of the noisy image is    Without image noise, LSD can extract 1613 line segments and the average length of line segments is 33.2 pixels, while MSLSD can detect 998 line segments and the average length of line segments is 43.4 pixels.In Figure 20, we find that the number of line segments extracted by the conventional methods and proposed methods is significantly reduced due to the influence of image noise.
Meanwhile, it can be firstly found from Figure 21 that LSD is greatly affected by image noise.The larger the noise, the more obvious the line segmentation fracture phenomenon, resulting in the shorter average length of the line segment.Secondly, MSLSD is robust to noise and is less affected by noise.Under different image noise environments, the average length of line segment extraction is basically unchanged.The average length of the MSLSD line segment extraction without noise is 43.4 pixels and the average length of the MSLSD line segment extraction under various noise conditions is maintained at 41.2-45.3pixels.Thirdly, EDLines is less affected by image noise than LSD, but it still produces severe line segmentation fracture problems in noisy environments.In this article, MSEDLines can also alleviate the influence of image noise on line segment extraction and the average length of the line segment is greatly improved compared to EDLines.Through the discussion, MSLSD/MSEDLines can make the average length of lines longer than LSD/EDLines under different Gaussian noise conditions, respectively.

Conclusions
In this article, our proposed line detection method, MSLines, including MSLSD and MSEDLines, solves the line segmentation fracture problem.This novel method has the following advantages and characteristics: (1) Using advantages of down-sampling fuzzy processing, the influence of grayscale distortion on line segment extraction is reduced and the segmentation fracture effect of the traditional line feature extraction algorithm is alleviated.The continuity of line segments is improved.
(2) Multi-scale image space line features are combined to construct an optimization and purification strategy with multiple constraints.The average length of extracted line segments is longer with higher integrity.Longer and accurate line segments provide a good research basis for line-based camera calibration, image matching, and 3D reconstruction.(3) Ineffective lines with partial redundancy and pseudo-lines are removed, as well as key line segments, such as geometric contours of the main object are optimized, which makes edge structures of the image be described more intuitively and clearly.(4) Only one parameter needs to be controlled manually, which makes the novel method more automated and parameter-less.
We experimentally verified the validity and superiority of our proposed method.Compared with LSD and EDLines, our proposed method has a longer average length and higher line segment integrity, which alleviates the line segmentation fracture effect to some extent.Particularly, our approach can extract more complete and comprehensive details for structured buildings.The presented improvements also can provide better structural features for SLAM navigation and other line-based applications.However, the proposed method does not completely solve the segmentation fracture effect of line segment extraction.Later, we can consider the fusion of LiDAR (Light Detection And Ranging) point cloud data based on our proposed method to further optimize and eliminate the segmentation fracture effect of line segment extraction.It can also be considered from the classical LSD algorithm perspective to improve problems where the internal gradient direction angle of the algorithm is abruptly affected by the grayscale mutation of individual pixels, which leads to the interruption of region growing and insufficient density of the same points.

Figure 1 .
Figure 1.Overall methodology of the proposed method.

Figure 1 .
Figure 1.Overall methodology of the proposed method.

2. 1 . 1 .
Line Segment Detection Method At present, Von Gioi's LSD line segment extraction algorithm is one of the most widely used methods.The principle of several mainstream line segment extraction algorithms is similar to LSD.

Figure 2 .
Figure 2. Workflow of LSD line segment extraction algorithm.

Figure 2 .
Figure 2. Workflow of LSD line segment extraction algorithm.
k represents the k-scale image of line segment vector space and the maximum value of k is N.Moreover, line segment endpoints' coordinates and length information were stored in line segment class

Figure 3 .
Figure 3. Line segment extraction and storage model in multi-scale image space.

Figure 3 .
Figure 3. Line segment extraction and storage model in multi-scale image space.

Figure 3
Figure 3 shows that the traditional line segment detector extracts line segments for each scale image and stores segments in vector space.If detector extracts m 1 and m 2 line segments in the first and second layers of the image, respectively, then the line segment method extracts m N line segments

and 2 l 2 ll and 1 l are independent and 2 l
continues to be judged by the vertical distance constraint relationship.Considering vertical distance constraint (Figure 4b), if ξ ≤ 2 d D (where ξ d is the vertical threshold), 2l is strongly correlated with 1 l and we then merge 1 l and or invalid segment and needs to be deleted.If is retained and stored as one of the reference line segments in the next judgment.Here we set ξd = 1(pixel), because the resolution of the human eye is 1 pixel, and the error of more than 1 pixel is discernible.

Figure 5 .
Figure 5. Line segment merging method based on endpoint projection transformation.(a) The slope of the reference line segment is > 0. (b) The slope of the reference line segment is ≤0.

Figure 6 .
Figure 6.The grayscale constraint relationship is based on NCC measurement.Two arbitrary line segments with the same direction information are merged into a virtual line segment and 3 × 3 matching windows 1 G , 2 G , and 3 G are opened grayscale points at the central positions of AB, BC, CD segments on a virtual line segment.NCC is the grayscale normalized cross-correlation index between line segments.

Figure 6 .
Figure 6.The grayscale constraint relationship is based on NCC measurement.Two arbitrary line segments with the same direction information are merged into a virtual line segment and 3 × 3 matching windows G 1 , G 2 , and G 3 are opened grayscale points at the central positions of AB, BC, CD segments on a virtual line segment.NCC is the grayscale normalized cross-correlation index between line segments.

2 R
was purified by geometric constraint as a purification criterion.

Figure 7 .
Figure 7. Nine images of three test sets.(a) Indoor 1: The inner corridor of one building with an image resolution of 1050 × 1400.(b) Indoor 2: Teaching building indoor environment with an image resolution of 640 × 480.(c) Indoor 3: Office indoor environment with an image resolution of 640 × 480.(d) Outdoor 1: The outdoor library with an image resolution of 1750 × 820.(e) Outdoor 2: The outdoor environment of a teaching building with an image resolution of 640 × 480.(f) Outdoor 3: Auditorium building with an image resolution of 640 × 480.(g) Aerial 1: The aerial image had an image resolution of 800 × 600.(h) Aerial 2: The aerial image with an image resolution of 765 × 763.(i) Aerial 3: The aerial image with an image resolution of 937 × 735.

Figure 7 .
Figure 7. Nine images of three test sets.(a) Indoor 1: The inner corridor of one building with an image resolution of 1050 × 1400.(b) Indoor 2: Teaching building indoor environment with an image resolution of 640 × 480.(c) Indoor 3: Office indoor environment with an image resolution of 640 × 480.(d) Outdoor 1: The outdoor library with an image resolution of 1750 × 820.(e) Outdoor 2: The outdoor environment of a teaching building with an image resolution of 640 × 480.(f) Outdoor 3: Auditorium building with an image resolution of 640 × 480.(g) Aerial 1: The aerial image had an image resolution of 800 × 600.(h) Aerial 2: The aerial image with an image resolution of 765 × 763.(i) Aerial 3: The aerial image with an image resolution of 937 × 735.

Figure 7 .
Figure 7. Nine images of three test sets.(a) Indoor 1: The inner corridor of one building with an image resolution of 1050 × 1400.(b) Indoor 2: Teaching building indoor environment with an image resolution of 640 × 480.(c) Indoor 3: Office indoor environment with an image resolution of 640 × 480.(d) Outdoor 1: The outdoor library with an image resolution of 1750 × 820.(e) Outdoor 2: The outdoor environment of a teaching building with an image resolution of 640 × 480.(f) Outdoor 3: Auditorium building with an image resolution of 640 × 480.(g) Aerial 1: The aerial image had an image resolution of 800 × 600.(h) Aerial 2: The aerial image with an image resolution of 765 × 763.(i) Aerial 3: The aerial image with an image resolution of 937 × 735.

Figure 17 .
Figure 17.Line segment extraction results of different multi-scale image space models.(a) Downsampled zero times.(b) Down-sampled one time.(c) Down-sampled two times.(d) Down-sampled three times.(e) Down-sampled four times.

Figure 17 .
Figure 17.Line segment extraction results of different multi-scale image space models.(a) Down-sampled zero times.(b) Down-sampled one time.(c) Down-sampled two times.(d) Down-sampled three times.(e) Down-sampled four times.

Figure 18 .
Figure 18.Multi-scale image space LSD line feature extraction of an indoor corridor.The left image is the original image.The middle image is the image with down-sampling once.The right image is the image with down-sampling twice.

Figure 18 .
Figure 18.Multi-scale image space LSD line feature extraction of an indoor corridor.The left image is the original image.The middle image is the image with down-sampling once.The right image is the image with down-sampling twice.
is the result of MSLSD optimization, where we can see the method of this paper solved the line segmentation problem and ensured the accuracy of the final extracted line segments.Therefore, selecting the line segment extracted from the original image as the benchmark in the optimization and purification process can ensure the line segment extraction accuracy.It does not introduce additional error generated by the low-resolution image line segment extraction when constructing the multi-scale image space.ofthis paper solved the line segmentation problem and ensured the accuracy of the final extracted line segments.

Figure 19 .
Figure 19.The line features of MSLSD without introducing low-resolution image line segment extraction error.

Figure 19 .
Figure 19.The line features of MSLSD without introducing low-resolution image line segment extraction error.
ISPRS Int.J. Geo-Inf.2019, 8, 183 21 of 25 of this paper solved the line segmentation problem and ensured the accuracy of the final extracted line segments.

Figure 19 .
Figure 19.The line features of MSLSD without introducing low-resolution image line segment extraction error.Therefore, selecting the line segment extracted from the original image as the benchmark in the optimization and purification process can ensure the line segment extraction accuracy.It does not introduce additional error generated by the low-resolution image line segment extraction when constructing the multi-scale image space.

Table 1 .
Results of line segment extraction for indoor, outdoor, and aerial images using traditional and proposed methods.

Table 2 .
MSLSD line segment extraction results under different down-sampling times.

Table 2 .
MSLSD line segment extraction results under different down-sampling times.

Table 3 .
The setting rules of model parameters.

Table 3 .
The setting rules of model parameters.