Statistical Evaluation and Analysis of Road Extraction Methodologies Using a Unique Dataset from Remote Sensing

Abstract

In the scientific literature, multiple studies address the application of road extraction methodologies to a particular cartographic dataset. However, it is difficult for any study to perform a more reliable comparison among road extraction methodologies when their results come from different cartographic datasets. Therefore, aiming to enable a more reliable comparison among different road extraction methodologies from the scientific literature, this study proposed a statistical evaluation and analysis of road extraction methodologies using a common image dataset. To achieve this goal, we setup a dataset containing remote sensing images of three different road types, highways, cities network and rural paths, and a group of images from the ISPRS (International Society for Photogrammetry and Remote Sensing) dataset. Furthermore, three road extraction methodologies were selected from the literature, in accordance with their availability, to be processed and evaluated using well-known statistical metrics. The achieved results are encouraging and indicate that the proposed statistical evaluation and analysis can allow researchers to evaluate and compare road extraction methodologies using this common dataset extracting similar characteristics to obtain a more reliable comparison among them.

1. Introduction

Road extraction methodologies, based on the digital processing of images from remote sensing, have been extensively studied by cartography researchers to help update important graphical representations, for example maps, for several different purposes that are useful for many research areas. Those methodologies use digital processing to extract road characteristics from remote sensing images. Remote sensing deals with information, such as images of land surface targets, that is remotely acquired by sensors placed in, for example, airplanes or satellites. The information is collected by cartography to compose cartographic datasets of multiple geographical data and earth measurements and to later be graphically represented as maps, globes, and cartograms, among other representations. In the scientific literature, many studies addressed the application of road extraction methodologies to different cartographic datasets and their respective statistical evaluations and analysis. However, it is difficult for any digital processing method to perform a more reliable comparison among road extraction methodologies when their results come from different cartographic datasets. Therefore, there remains a need for more studies on the comparison of road extraction methodologies by performing statistical evaluations and analysis based on the same dataset.

There is a significant number of studies published by the scientific literature presenting methodologies for road extraction based on the digital processing of images from remote sensing. For example, it is possible to find papers that presented road extraction methodologies based on many different approaches, such as GPS data [1], texture filters [2], mathematical morphology [3,4,5], and directional filters [6,7], among others. Additionally, due to the importance of this field to cartography, it is easy to find some research reviewing the results of road extraction methodologies [8,9,10]. In particular, these three last papers presented different reviews of several road extraction methodologies explaining their concepts and comparing advantages and disadvantages of each one. Despite being very studied in the cartographic area, other research areas also have interest in the road extraction methodologies, such as civil engineering and social studies, due to the importance of the road extraction studies for traffic management, vehicle navigation system, location based services, tourism and industrial development [1].

Although the scientific literature has many published studies presenting methodologies for road extraction based on the digital processing of images from remote sensing, unfortunately, most, to the knowledge of the authors, focused only on a specific type of road and quite often with a limited set of characteristics or conditions. In this sense, even though road extraction methodologies have attracted much attention from the scientific community, they have also been a challenge due to the difficulty reproducing the complexity of roads characteristics as a general model [8]. Thus, there is a need for more studies that could tend toward generalizations of those problems presented by the scientific literature. Moreover, some review papers compared road extraction methodologies with the results presented by their own authors, instead of performing a statistical evaluation and analysis of the road extraction methodologies using a unique image dataset. The comparison of road extraction methodologies, by performing statistical evaluations and analysis based on a unique common dataset, could bring some advantages for researchers enabling a more reliable ranking of future extraction methodologies with the same image dataset and guiding researchers to perform studies that tend to be generalizations of those problems presented by the scientific literature. Thus, Mayer et al. [11] grouped seven different road extraction methodologies and applied them to the same dataset. The dataset created contains only six images, of which three are aerial images and the other three are images acquired by the satellite Ikonos. In this way, the created dataset does not allow a great variety of images with different scenes that can exist in road extraction cases.

Considering the importance of studies on road extraction methodologies and additionally the advantages of using a unique image dataset to statistically evaluate and compare them, this study proposed a statistical evaluation and analysis of road extraction methodologies applied to the same image dataset. The study aimed to enable a more reliable comparison among different road extraction methodologies from the scientific literature. To achieve this goal, a dataset containing remote sensing images of three different road types, highways, city networks and rural paths, was proposed containing twenty-five images for each road type. In addition, the dataset was incremented with sixteen images from a public dataset (Vaihingen dataset) [12] from the International Society for Photogrammetry and Remote Sensing (ISPRS), which totals ninety-one images in the dataset. Furthermore, three road extraction methodologies were selected from the literature, in accordance with their availability, to be processed and evaluated using well-known statistical metrics defined in the scientific literature for road extraction evaluation [13,14]. The experiment was performed in three steps. The achieved results have pointed out that the proposed statistical evaluation and analysis could allow researchers to evaluate and compare road extraction methodologies using the same dataset and characteristics to obtain a more reliable comparison among them.

The rest of the paper is organized as follows. Section 2 summarizes some related work on road extraction and segmentation. Section 3 provides the necessary definitions and concepts on road extraction methodologies. Section 4 presents the image dataset and the methodology applied in this paper. Section 5 shows the extraction results achieved by the three selected methodologies and their respective statistical evaluation. Section 6 presents a brief discussion about the results obtained. Section 7 presents a conclusion about the results achieved by this study.

2. Related Works

Due to the importance of road extraction methodologies, several works in the scientific literature proposed many different methods to perform road extraction. The significant variability of methods can be explained by the fact that each road or image type has its own characteristics, which directly interfere in the extraction process and, consequently, hinder the development of a unique extraction methodology for all possible situations presented in static images of roads. In this sense, the scientific literature also includes some review papers that presented extraction methodology concepts and compared their results.

Extraction methods can be classified in accordance with the following different road aspects, according to Wang et al. [8]: geometric, radiometric or photometric, topological, functional and texture features. The geometric features are based on the geometric characteristics and road shapes, such as the fact that the width does not change suddenly and the ratio between the length and width is very large. Another common feature of roads is the color does not change much along the course, while it is different from those of the neighboring areas, which are considered as the radiometric features. The topological features are related to the fact that roads have intersections and they are not suddenly interrupted. On the other hand, the functional features consist of specific functions related to roads that must have some constraints conditions to perform those functions. Lastly, the texture features are based on finding the spatial distribution of pixel colors along the road course or on the neighborhood [8,15]. Wang et al. [8] presented extraction methodologies based on classification, knowledge, mathematical morphology and active contour models, for example. The authors described those methodologies and presented their results in accordance with the information contained in each paper. Additionally, the extraction methodologies were only compared using the information provided by the authors, without using the same characteristics to perform the comparison. Analyzing the papers results, in [10], the difficulty obtaining high accuracy values using only one extraction algorithm was shown, and that it required a study combining different extraction algorithms in accordance with the application.

Aiming to present a summary of the study area, Kaur and Singh [10] presented several automatic road extraction methodologies. The authors presented a comparison table containing the papers’ information, such as the basic extraction algorithm, the image source, statistical results when available, advantages and disadvantages. However, the methodology analysis was done using only the information provided by the authors. In this sense, the comparison was not direct because the extraction results were obtained using different image dataset and characteristics in each extraction methodology compared.

Image segmentation is an important step present in all road extraction methodologies, according to Kaur and Kaur [16]. In their study, segmentation was used to partition an image into meaningful parts having similar features and properties. Considering the large number of segmentation methods, Kaur and Kaur [16] reviewed various segmentation algorithms classifying them as structural, stochastic (normally known as spectral) or hybrid techniques. The structural techniques are based on the structure information of the interest feature in the image. On the other hand, the stochastic segmentation methods are only based on the pixel values, while the hybrid ones combine the structural information with the pixel values. After doing a comparison of several segmentation techniques, Kaur and Kaur [16] concluded that a unique extraction methodology is not sufficient to correctly extract the road networks from all the different image types and that some methods are not suitable for a particular image type. This fact motivated this work to perform a systematic analysis of different road extraction methodologies using a unique image dataset and, consequently, enable a real comparison among them.

After performing a review of the literature on road extraction methodologies, it is possible to realize that several methodologies used the mathematical morphology (MM) theory. MM is widely used due to the large number of functions and the ability to maintain the global geometry of the interest feature during the image processing. The most common use for MM is for post-processing steps aiming to improve the segmentation results by removing noises and linking some interest targets due to the capacity of processing the image while preserving the geometric structures of interest targets [17].

A methodology for road extraction based on the Otsu segmentation and the mathematical morphology theory was presented by Ma et al. [4]. After obtaining a gray scale image, Otsu algorithm was applied to convert it to a binary image highlighting the road network. However, the segmentation created a lot of noise (such as salt and pepper noise) that was removed using some mathematical morphology operations. Since the steps performed were based on the gray level of pixels, some other features could be part of the image together with the road network. Thus, the methodology proposed by Ma et al. [4] calculated a ratio between the target area and perimeter, removing targets that do not have an elongated shape to obtain the final extraction result.

In addition, using the MM and before applying the segmentation, a smoothing step using Partial Differential Equation (PDE) was applied by Leonardi et al. [18]. The PDE was used to remove noise, smoothing the image for the segmentation step. The segmentation was performed automatically using the Otsu method to obtain a binary image, which was post-processed using some mathematical morphology functions to obtain only the interest road.

The mathematical morphology to post-process the segmentation result was used by Wang and Shan [5]. They classified the images into four groups (linear, curvilinear, crossings and breakages), creating a different post-processing for each road type.

A methodology based on the growing region to segment the image using some road samples provided by the user was proposed by Cardim et al. [19]. Using the road samples, the algorithm calculated some statistical values to determine a range of values belonging to the roads to apply the growing region method and to obtain the segmentation result. After the segmentation, a post-processing step was performed using the mathematical morphology theory, to obtain the extraction result.

An extraction methodology, for which mathematical morphology and Gabor filter are used in parallel as pre-processing steps to enhance the contrast between road and non-road pixels, was proposed by Alshehhi and Marpu [20]. The second methodology step was based on the construction of a graph representation of the image in accordance with the initial segmentation and a hierarchical merging and splitting of the images segments in accordance with the color and shapes. Finally, the post-processing step removed the irregular segments.

The use of directional morphological operators for road extraction, which eliminates the needs of a previously defined structure element that is usually necessary in traditional MM filters, was introduced by Valero et al. [7]. Directional filters, path openings and path closings were used to create a morphological profile, enabling the extraction of linear geometrical information to classify each pixel as road or non-road pixels.

Another road extraction methodology, proposed by Courtrai and Lefèvre [6], applied a pre-extraction of roads segments, filtering the image with background knowledge, which was then analyzed and connected, whenever necessary, using a region-based path closing. This last step reconstructed the unconnected road segments producing a better result.

A road extraction methodology that can be divided into two main steps was presented by Sghaier and Lepage [2]. The first step, aiming to prepare the image for the road extraction, used a texture filter to extract linear surfaces, a morphological dilation operator to fill the holes caused by shadows, cars or other occlusions and finally applied the optimal canny edge detector to extract the road candidates. Afterwards, the second step was based on the beamlet transformation, which allowed the identification of the most appropriate scale for each road segment. The beamlet transformation allowed the methodology to use local information for each road segmentation. In this step, only the road candidates with the energy associated with a beamlet greater than a threshold were considered [2].

A methodology applying a Hough transform, after a canny edge detector, to identify possible roads lines was proposed by Ibtissam et al. [21]. As a parallel step, their methodology applied a local binary pattern algorithm to analyze the image texture. Afterwards, they assumed that a road is somewhat uniform, with similar values in the local binary pattern and with different values in the road edges detected by Hough transformation. In this sense, the methodology obtained the result of the extraction, taking the lines obtained by Hough transformation that have similar value in the local binary pattern result.

Two road features, the distinct spectral contrast and locally linear trajectory, wre used to present a multistage road extraction methodology exploited by Das et al. [22]. To achieve the desired results, four probabilistic support vector machines (SVM) were trained using samples of four different categories. The methodology was divided into two stages. During the first stage, a probabilistic support vector machine (P-SVM) was applied in parallel with a dominant singular measure (DSM). When performed alone, the steps cited produced errors that were not verified when they were applied together in a constraint satisfaction neural network (CSNN), which was modified to have a complementary information integration step (CSNN-CII). The CSNN-CII was used to integrate the information of the edge-based processing (DSM) and the region-based processing (P-SVM). In the second stage, a post-processing was performed to improve the accuracy of the road detection by removing some false alarms as well as recovering a few short road segments neglected due to false rejection.

Using convolutional neural network (CNN), Li et al. [23] proposed a road extraction methodology. The algorithm predicted whether a pixel belongs to the road, creating a rough map. To improve the previous results, a line integral convolution (LIC) based processing was used to connect the gaps and cracks, refining the rough road map obtained before.

According to Zhong et al. [24], most published CNNs are not designed for remote sensing images of high resolution, because they are adapted from natural scene classification, using thousands of training samples. In this sense, a CNN architecture based on the recent improvements of CNNs was proposed using more efficient convolutional layers and smaller kernels. Nevertheless, the architecture used only the CNN developed to predict all pixels of the image in different classifications and not just for road extraction.

As is shown in the review of state of the art papers, the classification of remote sensing images is an active topic of research, which aims to categorize scene images into a discrete set of meaningful land use and land cover classes according to the image contents [25]. In this way, it is very common to find people confounding classification techniques with road extraction methodologies. However, while the classifications aim to evaluate multiple spectral-bands to classify every pixel in a predetermined class, the road extraction methodology focuses only on the roads and, therefore, can apply techniques beyond spectral data, such as geometric and context objects techniques. In this sense, classification methods may be used in a preprocess step of road extraction identifying possible roads regions for a better and faster identification of the roads segments using some road extraction methodology. In this way, a classification method could improve the road extraction result, performing an initial and fast scene analysis and checking the need to perform a road extraction in the evaluated image scene in case of road candidates were selected. If a classification method does not find road characteristics in an image scene, the road extraction algorithm does not need to be used in this scene, making the process faster. Conversely, if the classification finds road characteristics in the scene, then the classification can be considered as a good candidate to perform the road extraction methodology, because it detects the interest features and its characteristics.

3. Background

3.1. Review Summary

According to the above discussion, we realized that each remote sensing satellite/sensor has its own features, such as image size, pixel size resolution and acquisition geometry, which makes the definition of a unique road extraction methodology difficult. Furthermore, the roads structures are too complex to be completely modeled by only one methodology. Those facts contribute to the high number of road extraction methodologies. Table 1 presents the main characteristics and results obtained by the road extraction methodologies presented before.

Table 1. Comparison of road extraction methodologies.

Methodology	Based on	Statistics
		Completeness	Correctness
Ma et al. [4]	Otsu thresholding and MM for post-processing.	Data not provided.
Leonardi et al. [18]	Barcelos and Chen PDE model.	0.927	0.918
	Otsu thresholding and MM for post-processing.
Wang and Shan [5]	Mathematical morphology.	Data not provided.
Cardim et al. [19]	Growing region algorithm and MM for post-processing.	0.953	0.945
Alshehhi and Marpu [20]	Gabor and MM filtering.	0.925	0.910
	Graph-based segmentation.
Valero et al. [7]	Directional morphological filters and morphological profiling.	Data not provided.
Courtrai and Lefèvre [6]	Directional morphological filters.	0.93	0.85
Sghaier and Lepage [2]	Texture filter and beamlet transformation.	0.825	0.875
Ibtissam et al. [21]	Hough transformation and local binary patterns	Data not provided.
Das et al. [22]	Multistage framework using SVM	0.89	0.93
Li et al. [23]	Convolutional neural network	0.771–0.839	0.912–0.974
Zhong et al. [24]	Convolutional neural network	Data not provided.

The road extraction methodologies presented before were tested, by their own authors, using different remote sensing image datasets and processing characteristics. This fact precludes a more reliable comparison between the data presented in the papers. Thus, it makes sense to perform the extraction methodologies using a unique common dataset framework, which allows a more reliable comparison of the obtained results.

Aiming to verify and analyze the performance of some road extraction methodologies, some authors, from the extraction methodologies presented, were contacted to make their methodologies available. However, only the codes of two methodologies were made available, the Sghaier and Lepage [2] and Cardim et al. [19] methodologies, and a third, that of Ma et al. [4], was implemented using the paper description. These three methodologies are distinct approaches and are based on different theories, being representative of the variety of methodologies available in the literature. In the sequence, these last three methodologies are detailed.

3.2. Road Extraction from Very High Resolution Remote Sensing Optical Images Based on Texture Analysis and Beamlet Transform [2]

Sghaier and Lepage [2] performed a multiscale decomposition using the beamlet transformation that allows the identification of the most appropriate scale for each road segment calculation, and the segments’ position and direction. Summarizing, the beamlet transformation consists of successive divisions of the image by four in a recursive way until it obtains blocks of 2 × 2 pixels. The former division of the image leads to the requisite of processing images or subimages with a size equal to a number power of two.

The Sghaier and Lepage [2] methodology can be divided in two main steps. The first step is responsible for preparing the initial image for the processing and the extraction of local information. During this step, the methodology uses the image texture to extract linear surface; applies the morphological dilation operator to fill the holes caused by shadows, cars or other occlusions; and applies the optimal canny edge detector to extract the road candidates. The second step is based on the beamlet transformation, allowing the introduction of spatial relationship rules between the road candidates segments at different scales. During the beamlet transformation, the roads candidates are analyzed and only those with energy $C_S$ greater than a threshold are maintained. The energy $C_S$ is obtained in accordance with the following equation:

$$C_S=\frac{T_f(b)}{\sqrt{l(b)}},$$

(1)

where, $C_S$ is the beamlet energy; $T_f(b)$ is the line integral of the beamlet; and $\sqrt{l(b)}$ is the Euclidean distance between two vertices (endpoints) $v_1$ and $v_2$.

Figure 1 shows the flowchart presented by the Sghaier and Lepage [2] describing this methodology, where SFS-SD is a structural feature set using the standard deviation.

Figure 1. Sghaier and Lepage [2] methodology flowchart. Adapted from Sghaier and Lepage [2].

The work of Sghaier and Lepage [2] presented the results of the methodology applied to four subimages of a GeoEyes 1 satellite image acquired in 2010 with spatial resolution of 0.58 m and 2.21 m for panchromatic and multispectral bands, respectively. The obtained results were statistically verified using the metrics of completeness and correctness defined in the literature [13]. The completeness result values obtained by this methodology were between 75% and 90%, while the correctness values were between 82% and 93%. However, the parameters used in the statistical calculation were not mentioned.

3.3. Semiautomatic Methodology for Cartographic Features Extraction Using High-Resolution Remote Sensing Images [19]

Cardim et al. [19] described a semiautomatic methodology for road extraction based on the road radiometric features. It is considered a semiautomatic method because it requires user interaction. The user must provide some samples of the interested road and then the algorithm tries to delimitate all of the roads based on the given sample information. Using the road samples, provided by the user, the algorithm calculates some statistical values to determine a range of values belonging to the roads. Once the range of values is determined, the growing region method, exemplified in Figure 2, is applied to delimitate the road edges. The growing region technique uses the samples as initial points by which a search begins in their neighborhood checking if there are other pixels belonging to the defined range. It is a recursive idea, since the same search procedure is done for all pixels detected as part of the interest feature.

Figure 2. The growing region method. Acquired from Cardim et al. [19].

Summarizing, the growing region method uses the samples to segment the image delimitating the road edges and it can be considered as the first step of the methodology. The second and final step of the methodology is based on the mathematical morphology theory to close small holes generated during the segmentation, which could be produced by cars present in the road. Figure 3 shows the flowchart presented by Cardim et al. [19] describing that methodology.

Figure 3. Cardim et al. [19] methodology flowchart. Acquired from Cardim et al. [19].

Cardim et al. [18] obtained the results of the methodology applied to one subimage of a Quickbird Satellite panchromatic band with spatial resolution of 0.6 m. Even though it was not presented in the paper, the authors mentioned that the methodology was tested using a set of remote sensing images. The results presented were statistically verified [13,14], achieving values of 95% for completeness and 94% for correctness for the image presented in that paper.

3.4. Road Information Extraction from High Resolution Remote Sensing Images Based on Threshold Segmentation and Mathematical Morphology (Ma et al., 2013) [4]

Ma et al. [4] proposed a methodology for road extraction based on the Otsu segmentation and mathematical morphology theory to remove noise and to improve the segmentation result. Firstly, the authors convert an RGB image into a grayscale one using the equation:

$$GrayTone=(0.229R+0.587G+0.114B),$$

(2)

where $R$, $G$ and $B$ are the pixels colors for the red, green and blue bands, respectively.

Once the image was converted to gray scale, the Otsu thresholding method is applied to segment the image trying to identify the road network. The segmentation process highlights the road network; however, it creates a lot of noise in the image. To reduce the noise, the methodology detects and subtracts the boundary of the target of the segmented image from itself, removing very small targets. To reduce the holes inside the interest targets, the methodology proposed by the authors applies a morphological dilation using an eight-neighborhood structural element. Despite all those steps to highlight the interest road network, other structures can also be highlighted, because all steps before are based on the pixel color. To remove the non-interest features detected, the paper proposes the following equation:

$$t=\frac{\sqrt{S}}{C}>n,$$

(3)

where $S$ is the target area; $C$ is the target perimeter; and $t$ is the ratio of the area of square root to the perimeter. The $t$ value is calculated for each object in the image and only those with $t$ greater than a determined $n$ value are kept as road in the resulting image.

Figure 4 shows the flowchart created according to the descriptions presented by the authors for the proposed methodology.

Figure 4. Ma et al. [4] methodology flowchart according to the description.

Ma et al. [4] presented the results of the methodology that was applied to a small set of remote sensing images, however the satellite and sensors characteristics used to acquire the images were not mentioned. Despite the visual results presented, there is no quantitative evaluation of the results presented in that paper.

4. Methodology

In our paper, aiming to compare road extraction methodologies, which were chosen from the scientific literature in accordance with their availability, three different steps were performed. First, a diversified and very useful dataset containing images from remote sensors was created. Next, a digital processing method was individually performed to statistically evaluate and analyze each of the three different methodologies, applying all of them to the same image dataset and processing setup. Finally, the same three road extraction methodologies were carefully compared each other based on their results and some well-known evaluation metrics.

4.1. The Creation of the Dataset

In the first step, a dataset containing images from remote sensing was created to supply input data to the next step of this experiment, i.e., it is used by the methodologies to perform road extractions. Different from other studies in which each evaluated methodology was experimented using a different dataset (e.g., [2,10,17]), this study performed all experiments using a unique dataset of images from remote sensing satellites and photogrammetric flights. This dataset is important because, by applying all road extraction methodologies to the same dataset, the experiment can enable a more reliable comparison among different road extraction methodologies than those often presented in the scientific literature. All the images were carefully selected to compose the dataset and they contained at least one road as the interest feature. The roads were categorized into three groups in accordance with the interest feature: highways; city roads networks; and rural paths. The groups were created because it is important to evaluate extraction methodologies on each road type. Each group has twenty-five images, which were supplied by the Unesp’s research group database. In addition, for each image of the dataset, a ground truth image was manually created. Furthermore, another group containing sixteen images from the Vaihingen dataset, one of the ISPRS (International Society for Photogrammetry and Remote Sensing) datasets, were added. Considering that the ISPRS datasets are for classification studies, the ground truth available were not suitable for methodologies dedicated to the road extraction process. Thus, the ground truth for those images had to be manually detected by an expert. In addition, since the road extraction methodologies evaluated here need images with sizes of multiples of 512 pixels, the images had to be cut to fit that need. Figure 5 presents four different images that compose the dataset, each of them exemplifying one road group of the dataset: (a) highways; (b) cities network; (c) rural area; and (d) ISPRS dataset. The end-user of the dataset might consider reducing the images considering their evaluation; for example, a subset of images of western roads not including road images taken from undeveloped countries, because they are particularly interested in detection of these particular roads. The dataset used, except ISPRS original images, can be found at the following link: https://goo.gl/e33K74.

Figure 5. Dataset examples: (a) Highways; (b) cities network; (c) rural area; and (d) ISPRS dataset.

In Figure 5, it is possible to observe that the dataset was carefully created to be composed of images containing different road types and structures to obtain the maximum diversity and real situations regarding to road networks. This is further explained in Table 2, which contains some information about the dataset used, such as the spatial resolutions and images sizes.

Table 2. Dataset specifications.

Image Group	Number of Images		Satellite	Spatial Resolution	Width	Height
Highways	25	4	Quickbird	0.6 m	512–1536	512–2048
		11	WorkdView-2	0.5 m
		2	Pléiades	0.5 m
		8	Photogrammetric flights	0.45 m
Cities	25	2	GeoEyes-1	0.58 m	512–1024	512–1024
		12	Quickbird	0.6 m
		1	WorldView-2	0.5 m
		5	Pléiades	0.5 m
		5	Photogrammetric flights	0.45 m
Rural	25	1	Quickbird	0.6 m	512–2048	512–2048
		10	WorkdView-2	0.5 m
		14	Photogrammetric flights	0.45 m
ISPRS Dataset	16		Photogrammetric flights	0.08 m	1024–2560	1024–2560

4.2. The Digital Processing Method

In the second step, a digital processing method was performed to statistically evaluate and analyze three methodologies, which were chosen from the scientific literature in accordance with their availability, applying them to the same image dataset and processing setup. These three road extraction methodologies were obtained from: Sghaier and Lepage [2], Cardim et al. [19] and Ma et al. [4]. The original version of the methodology from Sghaier and Lepage [2] was developed using an older version of the Orfeo Toolbox, OTB 4.0 [26]. Therefore, it was adapted to a newer version of the Orfeo Toolbox because it was incompatible with the available Orfeo Toolbox. The methodology from Cardim et al. [19] was made available as an executable file making possible the evaluation without modifications. Considering that only the codes of two methodologies were supplied by their authors, the methodology from Ma et al. [4] was reproduced in accordance with the description found in Ref. [4]. Thus, some tasks of the methodology were deduced empirically because they was insufficient detail in the authors’ description. All three road extraction methodologies were carefully processed and the extraction results were rigorously evaluated in accordance with the schematic plan of the present proposal presented in Figure 6.

Figure 6. Schematic plan of the present proposal.

Figure 6 exemplifies the implementation of the digital processing method to statistically evaluate each methodology. According to the schematic plan presented in Figure 6, while performing the digital processing method, each road extraction methodology is applied to every image that composes the dataset. Next, each road extraction result is statistically evaluated based on the ground truth dataset. The evaluation metrics used by the digital processing method are described in the next subsection and the results achieved by their applications are presented and discussed in Section 5 and Section 6.

4.3. The Evaluation Metrics

To evaluate the results of a road extraction methodology, the simplest method is to overlay the results over the original images. It is common to find some papers presenting this type of verification, because it allows verifying the position of the extracted road in relation to the road position in the original image. However, it is a visual-based and subjective method.

To perform a statistical evaluation, the most widespread method is the one described by Wiedemann [13]. That paper described some metrics that are commonly used by many studies as reference to compare their results with results from other studies. Those metrics perform comparisons based on the use of vector data of the main axis of the interest road. Another evaluation method, described by Cardim et al. [14], is based on the first method cited [13] but adopted for high-resolution images performing the evaluation by comparing the pixels values instead of the vector data and using the roads edges instead of the main axis. In this way, since two of the evaluated methodologies return road segment areas, instead of the road edges, the edges of the segmented area were acquired applying the Laplacian filter. It is done to make possible the comparison of all three methodologies using the same evaluation methodology.

The reference data must be considered as an ideal solution being compared with the automatically detected data. The comparison is performed pixel-by-pixel using an acceptance buffer (tolerance), which is separately disposed for both images. To consider the pixel as a coincident one, it must be in the acceptance buffer of the other data during the comparison [13,14,27]. Figure 7 presents how this comparison is performed.

Figure 7. Comparison scheme. Adapted from Cardim et al. [14]: (a) comparison of the extracted image; and (b) comparison of the reference image.

The comparison of the extracted image, illustrated in Figure 7a, results the number of pixels matched in this comparison and the number of false positives errors obtained by the extraction methodology. Moreover, the comparison of the reference image, illustrated in Figure 7b, results the number of pixels matched in this comparison and the number of false negatives errors obtained by the evaluated methodology. Note that false positive errors are the pixels extracted as part of the interest road by the extraction methodology applied but do not correspond to the interest road when checked in the reference road image. Conversely, false negative errors are pixels that correspond to the interest road in the reference road image but were not extracted as part of the interest road by the extraction methodology applied.

Additionally, two metrics, completeness and correctness, are considered the most important metrics for some statistical evaluations when the total of false positives and false negatives errors are known. Completeness and correctness are described next.

Completeness can be summarized as the percentage of the reference image that was correctly extracted by the proposed methodology, i.e., this metric describes how completely the roads were detected. The optimal value for this metric is 1 and it is calculated, as described in [13,14], using the following equation:

$$completeness=\frac{matchedpixelsofreferenceimagecomparison}{totalofpixelsofthereferenceimage},$$

(4)

Correctness is the percentage of the extracted image that corresponds to the reference data, i.e., this metric describes how correct is the image extracted by the evaluated methodology. The optimal value for this metric is 1 and it is calculated, as described in [13,14], using the following equation:

$$correctness=\frac{matchedpixelsofextractedimagecomparison}{totalofpixelsoftheextractedimage},$$

(5)

Completeness and correctness are the two most important statistical values for the evaluation of road extraction results because they include the false positives and false negatives errors. Therefore, this study rigorously calculated the completeness and correctness metrics for all results obtained by the methodologies in the experiment.

It was hard to add other images to this dataset, create new groups and experiment on them because this study was limited by the fact that most images were obtained from the private domain. However, a dataset composed on ninety-one images and divided into four different groups is considered acceptable to perform this experiment and provide a common dataset for the research community. Future research will make the effort to join other possible available images to add them to this dataset, create new groups and perform more experiments on them.

5. Results

As mentioned before, aiming to enable a more reliable comparison among different road extraction methodologies from the scientific literature, the purpose of this study was to perform a statistical evaluation and analysis of three road extraction methodologies by applying them to a unique image dataset. Three different steps were performed to achieve this purpose. First, a dataset of images from remote sensing was created. Second, it was necessary to obtain the extraction result of the interest roads for each extraction methodology that should be evaluated. That is, the second step is a digital processing method (see Figure 6) that was performed to process the three methodologies to obtain the extraction results for each image of the dataset. Thrid, the results were statistically evaluated in accordance with a consolidated method [13], for which the results are compared with a reference image, which is considered as an ideal result, as explained before in the Methodology Section.

In this section, the statistical evaluation was performed calculating the metrics of completeness and correctness, presented, respectively, by Equations (4) and (5). As mentioned in the equation definitions, an acceptance buffer is created according to a specified size, which is empirically defined. Since we have the spatial resolution for every image used, we defined the buffer size as 2 m and the algorithm calculate the buffer size in pixels for each image.

Figure 8 presents the results obtained in the second step by processing and applying the extraction methodologies of Sghaier and Lepage [2], Cardim et al. [19] and Ma et al. [4] to the images presented in Figure 5.

Figure 8. Examples of results obtained by the tested methodologies.

Table 3 presents the average values for completeness and correctness metrics obtained in the third step for each methodology related to each road type. The bold values are the best values obtained during the experiment.

Table 3. Statistical values obtained for each extraction methodology.

	Sghaier and Lepage [2]		Cardim et al. [19]		Ma et al. [4]
	Completeness	Correctness	Completeness	Correctness	Completeness	Correctness
Highways	0.204	0.129	0.582	0.492	0.121	0.106
Cities	0.100	0.223	0.361	0.184	0.061	0.072
Rural	0.197	0.218	0.753	0.733	0.072	0.032
ISPRS	0.321	0.173	0.561	0.384	0.383	0.092
Average	0.194	0.188	0.564	0.455	0.128	0.062

Table 3 presents an average value obtained by each methodology when applied to the created dataset, but Table 3 does not show the individual values for each image from the dataset. Aiming to provide a visualization of the statistical values obtained for each image from the dataset, Figure 9 and Figure 10 present, respectively, the completeness and correctness variation in the experimented dataset. It is worth mentioning that Images 1–25 correspond to the highways; Images 26–50 correspond to the cities networks; Images 51–75 correspond to the rural paths; and Images 76–91 correspond to the images from the ISPRS dataset.

Figure 9. Evaluation of the completeness metric obtained for the entire dataset. Images 1–25 correspond to the highways; Images 26–50 correspond to the cities networks; Images 51–75 correspond to the rural paths; and Images 76–91 correspond to the images from the ISPRS dataset.

Figure 10. Evaluation of the correctness metric obtained for the entire dataset. I Images 1–25 correspond to the highways; Images 26–50 correspond to the cities networks; Images 51–75 correspond to the rural paths; and Images 76–91 correspond to the images from the ISPRS dataset.

6. Discussion

It is possible to verify, in Table 3 and Figure 9 and Figure 10, that the statistical values obtained for city road networks extraction, Images 26–50, were the lowest obtained values. The low values obtained for city scenes are due to the difficulty faced by the extraction methodologies to differentiate the interest road from city components. City scenes are very complex containing a lot of similar surrounding information that confuses the extraction methodologies. It seems that, in this case, the results are in good agreement with the observation made by Maboudi et al. [28] that the spectral and spatial characteristics of non-road structures can be similar to roads, making road extraction more difficult in urban areas. Moreover, the second methodology applied to highways and rural paths achieved high completeness and correctness average results are over 80%, while for some images the statistics surprisingly achieved 100%. As opposed to city roads networks, highways and rural paths are well defined related to the surrounding scene but they still have some interruptions in the road course.

Observing the completeness values obtained by the three methodologies evaluated and presented in Figure 9, it is possible to verify that the second methodology (Cardim et al., [19]) obtained the best values for almost all images, except for Images 21 and 26, for which the third methodology (Ma et al., [4]) and the first one (Sghaier and Lepage [2]) obtained, respectively, the best values of completeness. Meanwhile, the correctness values obtained, presented in Figure 10, do not distinguish a methodology as the best one. Actually, the Cardim et al., [19] methodology achieved the best correctness values for almost all figures containing highways and rural paths, whereas the Sghaier and Lepage [2] methodology achieved the best correctness values for the majority of images containing city roads. All those facts confirm the information presented in Table 2, where the best average values of completeness and correctness are in bold.

Regarding to the three methodologies and the dataset analyzed in this paper, it is possible to claim that city roads networks are the most difficult road type to extract, because there are a lot of similar information around them. Moreover, the best average values obtained with the statistical evaluation are results from rural paths images. It could be explained because of the large contrast between the rural paths and their surrounding information. Comparing the results of the methodologies, the Cardim et al. [19] methodology achieved better statistical results than the other two methodologies for highways and rural paths. It can be explained by the fact that the last methodology is semiautomatic and, therefore, it needs some road samples before the beginning of the extraction process. However, for images containing city road networks, the Sghaier and Lepage [2] methodology achieved the best correctness statistical results because the detected segments belong to the interest road.

The achieved results indicate that the proposed statistical evaluation and analysis could allow researchers to evaluate and compare road extraction methodologies using the same dataset and characteristics to obtain a more reliable comparison among them.

7. Conclusions

In the scientific literature, many studies addressed the application of road extraction methodologies to different cartographic datasets and their respective statistical evaluations and analysis. However, it is difficult for any study to perform a more reliable comparison among road extraction methodologies when their results come from different cartographic datasets. Facing the importance of studies on road extraction methodologies and the need for more studies to statistically evaluate and compare the application of these methodologies to a unique image dataset, this study proposed a statistical evaluation and analysis of road extraction methodologies applied to the same image dataset. The study aimed to enable a more reliable comparison among different road extraction methodologies from literature.

This study contributed by creating an important dataset of images from remote sensing containing three different types of roads: highways, city road networks and rural paths. Moreover, the achieved results demonstrate that the proposed study also contributed by providing a useful digital processing method able to perform statistical evaluation and analysis of road extraction methodologies applied to the same dataset of images from remote sensing. It could be concluded that the results indicate that the proposed study could provide a more realistic, therefore a more reliable, comparison among the three road extraction methodologies that were selected from the literature.

The proposed study expands prior studies [8,10,16] applying the evaluated methodologies to a unique dataset containing miscellaneous images of different resolutions and context, which are available for future works and analysis. Therefore, this study provided considerable improvements over the state-of-the-art studies applied for remote sensing and significant support for future research to improve the process of comparison of road extraction methodologies using a unique dataset of images from remote sensing.

However, a limitation is worth noting. Although the findings of this study are promising, the difficulty to reproduce the complexity of roads characteristics as a general model remains a challenge. Future work should therefore consider the need for more studies that could be generalizations of the problems presented by the scientific literature.

In the future, it is expected that this study could be applied without difficulty to bring some advantages for researchers enabling a more reliable comparison of other road extraction methodologies with the same image dataset and to support researchers to perform studies that generalize the problems presented by the scientific literature.