Around View Monitoring-Based Vacant Parking Space Detection and Analysis

Abstract

Accelerated urbanization and the ensuing rapid increase in urban populations led to the need for a tremendous number of parking spaces. Automated parking systems coupled with new parking lot layouts can effectively address the need. However, most automated parking systems available on the market today use ultrasonic sensors to detect vacant parking spaces. One limitation of this method is that a reference vehicle must be parked in an adjacent space, and the accuracy of distance information is highly dependent on the positioning of the reference vehicle. To overcome this limitation, an around view monitoring-based method for detecting parking spaces and algorithms analyzing the vacancy of the space are proposed in this study. The framework of the algorithm comprises two main stages: parking space detection and space occupancy classification. In addition, a highly robust analysis method is proposed to classify parking space occupancy. Two angles of view were used to detect features, classified as road or obstacle features, within the parking space. Road features were used to provide information regarding the possible vacancy of a parking space, and obstacle features were used to provide information regarding the possible occupancy of a parking space. Finally, these two types of information were integrated to determine whether a specific parking space is occupied. The experimental settings in this study consisted of three common settings: an indoor parking lot, an outdoor parking lot, and roadside parking spaces. The final tests showed that the method’s detection rate was lower in indoor settings than outdoor settings because lighting problems are severer in indoor settings than outdoor settings in around view monitoring (AVM) systems. However, the method achieved favorable detection performance overall. Furthermore, we tested and compared performance based on road features, obstacle features, and a combination of both. The results showed that integrating both types of features produced the lowest rate of classification error.

1. Introduction

In traffic-congested cities, parking can be extremely challenging. Firstly, finding a vacant parking space may require considerable time. After finding a space, the driver’s parking skill is tested. An unskilled driver can cause traffic congestion on busy streets or, worse, accidentally collide with the parked cars adjacent to the vacant spot. Because of this situation, automated parking systems became popular driver assistance systems in recent years [1,2]. Most automated parking systems available on the market are semiautomated systems based on ultrasonic sensors. Multiple ultrasonic sensors mounted on the vehicle provide two-dimensional scanning and positioning of a parking space and nearby obstacles. After detecting relative positions of the space and obstacles, the system uses geometric parameters to plot a path for parking and displays a virtual track for this path. The system then controls the steering wheel to assist the driver with parking in a parking space, but the driver must manually shift gears and apply brakes as necessary. However, ultrasonic sensor-based parking assistance systems are highly dependent on the positions of adjacent vehicles. If the parking assistance system uses a reference vehicle that is parked improperly, the car with the parking assistance system will likely be parked improperly (Figure 1). Therefore, human intervention is often required to ensure the car is properly parked within a parking space. Furthermore, the parking assistance system cannot function if no vehicles are parked next to the desired parking space. Therefore, we hope to develop a parking system in the future that is truly and fully automated by using image sensors to detect the marked boundaries of a parking space and ultrasonic sensors to classify occupancy. The integration of multiple types of sensors will increase the robustness of the overall system.

In this paper, we propose a parking space detection method based on an around view monitoring (AVM) system and related vacancy analysis algorithms. The images from AVM systems are primarily produced when individual images from four wide-angle lenses are converted into bird’s-eye view images and then merged. The advantages of an AVM system are its wider visual range compared to monocular cameras and its ability to help drivers identify surrounding parking space markings and obstacles. The vacancy or occupancy of parking spaces was detected solely through AVM image sensors. The paper provides two contributions. Firstly, it presents a high-level fusion of parking space detection and related vacancy analysis to facilitate indoor parking, where inadequate light sources are present, and outdoor parking, where environmental variations can be extensive. The proposed approach performed well in both settings. Secondly, it proposes a space occupancy classification via a low-level fusion of road and obstacle features. For road features, a region growing method was used to search for usable space within the parking space. For obstacle features, an edge detection method was used to determine whether a car is parked in the parking space. Finally, features were output to a naïve Bayes (NB) classifier to classify the current status of the parking space.

Section 1 provides a brief introduction to the research motivations, and an overview of the development progress of the present study is provided as follows: Section 2 contains an in-depth discussion of previous studies relevant to vacant parking space detection and includes an analysis of the advantages and disadvantages of the sensors and techniques used in the present study. Section 3 introduces the algorithm to detect vacant parking spaces and the methods used in each stage of the process. For example, features of parking space lines were extracted to detect corners, a random sampling algorithm was used to fit lane boundary lines, road features were extracted using region growing, obstacle features were extracted using edge detection, and the naïve Bayes (NB) classifier was used to determine the vacancy of a given parking space. Section 4 explains the experimental process and the parameters set for each stage. The recall and precision rates of parking space detection and the error rate of space occupancy classification for spaces with different features were assessed in both indoor and outdoor settings. Section 5 presents the conclusion of the present study as an algorithm to detect vacant parking spaces. Future developments, such as the use of supplementary ultrasonic sensors and instantaneous detection performance, are discussed.

2. Literature Review

Figure 2 illustrates the classification of the existing literature based on the automated parking cycle. The concept of a control cycle was employed to classify the literature into three major categories: measurement, analysis, and execution. Firstly, in a parking process, sensors mounted on a car detect information about the surrounding environment. Next, algorithms are used to analyze these readings. Finally, the results are transmitted to the control end to execute commands.

2.1. The Measurement Stage

The primary function of the measurement stage of the control cycle is to receive sensor information for analysis in the next step. Two types of sensors are commonly used in current automated parking systems: free space-based sensors and parking space marking-based sensors. Free space-based sensors primarily provide distance information, because this type of information allows systems to calculate whether the current space is large enough to park a car. In contrast, the purpose of parking space marking-based sensors is to recognize a parking space, because places marked by parking space lines are more representative of areas where parking is legally allowed. The two types of sensors are introduced and discussed below.

2.1.1. Free Space-Based Sensors

These types of sensors primarily use distance information to calculate the space needed for parking. Once a space large enough to accommodate a car is detected, the system initiates the parking process. In the current market, free space-based sensors are most commonly ultrasonic sensors [3,4]. Because of their low price and ease of installation, ultrasonic sensors are widely used in parking assistance systems. However, the precision of ultrasonic sensors is highly dependent on the positioning of adjacent vehicles, and ultrasonic sensors cannot function without reference vehicles. Laser scanners [5] can measure distance with a high degree of precision, but their high cost hinders their commercialization as an automobile accessory. Depth cameras [6] refer to binocular stereo vision or moving stereo vision sensors. These sensors provide not only image information but also depth information. However, because generating a depth map is computationally demanding, depth cameras are difficult to use in real-time settings.

2.1.2. Parking Space Marking-Based Sensors

These types of sensors are all image sensors. Their primary function is to recognize the markings that define a parking space. Along with other rapid advances in automotive electronics, vehicle-based imaging systems advanced from monocular cameras [7,8,9,10,11] to AVM systems [12,13,14,15,16]. As a result, the advantages of an AVM system are its wider visual range compared to monocular cameras and its ability to help drivers identify surrounding parking space markings and obstacles. However, because both monocular cameras and AVM systems are image sensors, they are relatively sensitive to lighting problems.

2.2. The Analysis Stage

After obtaining measurements and readings, the analysis stage of the control cycle primarily analyzes the information using various algorithms; these analyses facilitate decision-making in the next stage. Automated parking systems contain two core technologies: parking space detection-based technology and space occupancy classification-based technology. The algorithms and methods used in these two types of technologies are introduced and discussed below.

2.2.1. Parking Space Detection-Based Technology

Image-based parking space detection is intuitively associated with line detection. For example, Jung et al. [6] used the Hough transform on bird’s-eye view images from monocular cameras to detect line segments through a combination of Hough space and marking line width restrictions. However, this method is unsuitable for AVM systems because AVM images are produced by the merging of four bird’s-eye view images. This merging causes an inevitable curvature in line segments near the spliced boundaries. Hamada et al. [12] used the probabilistic Hough transform to detect line segments and inferred the parking space. The detection rate of the approach depends on the robustness of the line detector. In addition, some divider lines are blocked by previously parked cars, leaving only a short visible segment. These two commonly seen variations in line segments can both affect line detection in the Hough space.

To resolve these problems, Lee et al. [8,13] used directional density-based spatial clustering of applications with noise to cluster line segments with different orientations. The type of parking space is identified using the geometric features of various types of parking spaces. However, because the clustered line segments lack parking space information, this method often mistakenly detects road markings. Suhr et al. [14] divided lane boundary lines into a pair of parallel lines with opposite gradient directions. Next, random sample consensus (RANSAC) was used to detect lane boundary lines. To detect divider lines between parking spaces, they used a distance transformation in combination with template matching to identify lines that could be divider lines. However, their method included only a single template; thus, it could be applied only to rectangular parking spaces. In addition to line detection, others used corners as the feature for parking space detection. Suhr and Jung [8] proposed applying the concept of hierarchical tree diagrams to parking space detection by using hierarchies to represent different types of parking spaces. Corner detection is the lowest level of the hierarchy; gradual progression up the hierarchy results in the identification of a corresponding parking space. Suhr and Jung [15,16] used the Harris corner detector to extract junction patterns to infer the valid parking space. However, the overdependence on corner features can result in an inability to construct parking spaces because corners cannot be detected in dim indoor environments. For leaning-based parking space detection, Li et al. [17,18] proposed a machine learning approach in the field of parking space detection. They adopted boosting decision trees to train a marking-point detector to infer the parking space. In addition, Zhang et al. [19] proposed a deep convolutional neural network (DCNN) to formulate the parking space detection problem as two sub-problems—local image pattern classification and marking-point detection—to efficiently detect parking spaces.

2.2.2. Space Occupancy Classification-Based Technology

Space occupancy classification is a core technology of automated parking systems. Occupancy analysis comprises primarily three methods: ultrasonic sensors, image analysis, and a combination of the two. Ultrasonic sensor-based methods [3,4] are the most common automated parking systems available on the market today. Multiple ultrasonic sensors mounted on the vehicle provide two-dimensional scanning and positioning of a parking space and nearby obstacles. After detecting relative positions of the space and obstacles, the system uses geometric parameters to plot a path for parking and displays a virtual track for this path. The system then controls the steering wheel to assist the driver with parking in a parking space, but the driver must manually shift gears and apply brakes as necessary.

The image analysis-based method primarily targets and analyzes visible features within the parking space. Wang et al. [11] determined whether a car was parked within a space by analyzing the ratio of edges in the parking space. The red area represents the region of interest for calculating edge ratios. Although a parking space characterized by a high ratio of edges could represent a space occupied by a car, basing occupancy classification on a single feature seems inadequate. For example, the features of roadside parking spaces are inevitably affected by noise, such as concrete surfaces, storm drain covers, or manhole covers, because of their locations. These environmental factors can greatly influence the performance of classification methods. Lee and Seo [8] proposed using a histogram of oriented gradient and fast Fourier transform magnitude to analyze parking spaces. A support vector machine was used to classify parking space occupancy. Suhr and Jung [14,16] proposed a method to detect parking spaces by fusing the sensors of an ultrasonic-based automatic parking system and an AVM system. Firstly, distance information obtained from ultrasonic sensors is transformed and mapped to an AVM image. Next, each parking space is viewed as one cell of a grid. Finally, the posteriori probability of the grid is used to determine whether a parking space is occupied. For leaning-based space occupancy classification, Amato et al. [20,21] proposed a decentralized solution for space occupancy classification using a deep CNN and smart cameras. This deep learning approach demonstrates high classification accuracy in the presence of light variation, partial occlusions, and shadows. In addition, Xiang et al. [22] proposed a Haar–AdaBoost cascade-based approach combined with a deep CNN to detect parked cars and reported an accuracy of greater than 95% for space occupancy classification.

2.3. The Execution Stage

In the final stage of the automated parking cycle, actuators in the car perform corresponding controls based on the results of the prior analyses. In the automated parking system, when a system identifies a vacant parking space, the system begins path planning and tracking. Not until the car is parked is the automated parking cycle considered complete. For path planning, because parking space detection was completed in the previous stage, this stage uses the positioning information obtained from the parking space to plan the path traveled by the car. Lin et al. [23] proposed a multi-turn path planning method that effectively decreases the space required for parking: the space only needs to be 1.28 times the length of the car. The purpose of path tracking is to compare any deviation between the expected and actual paths and to control and track the turning of the car to assist the driver in executing the expected path. Wang et al. [11] proposed a preview control strategy and a double circular trajectory planning to realize automatic parking. Suhr and Jung [14,16] used directional chamfer matching to compare parking space markings and the odometer to track the path between the car and the parking space.

2.4. Summary

In free space-based sensors, the precision of ultrasonic sensors is highly dependent on the positioning of adjacent vehicles, and ultrasonic sensors cannot function without reference vehicles. The high cost of laser scanners hinders their commercialization as an automobile accessory. Depth cameras are difficult to use in real-time settings because generating a depth map requires an enormous amount of computing power for embedded systems. In parking space marking-based sensors, monocular cameras were gradually replaced because their field of vision is too narrow. In parking space detection technology, detection methods based on line and corner features perform equally well. The two methods differ by the type of technology used for space occupancy classification. In space occupancy classification technology, ultrasonic sensor-based methods have broad applications in the current market. However, these methods still rely on the existence and positioning of adjacent cars. Image analysis-based methods that analyze the edge ratios within parking spaces perform well; however, basing occupancy classification on a single feature seems somewhat inadequate. Based on these issues, we chose the AVM system as a sensor for the surrounding parking environment. This system was paired with corner feature detection to detect parking spaces in the system’s field of vision. Finally, in space occupancy classification, the proposed method simultaneously analyzes both road and obstacle features to improve the robustness of the algorithm. The proposed method is introduced in the next chapter.

3. Algorithm to Detect Vacant Parking Spaces

3.1. Algorithm Flowchart and Framework

Figure 3 illustrates the framework of the algorithm to detect vacant parking spaces. The two major stages of the algorithm are parking space detection and space occupancy classification. In the parking space detection stage, corner features of parking space markings are extracted first. The extracted features are filtered by the circular intensity proportion and the maximum vertical intensity profile in the x-axis. After filtration, a random sampling algorithm is adopted to identify the lane boundary line. Next, the lane boundary line is used as a template to scan and record the intensity proportion of the divider lines; this allows the system to determine the type of parking space. The widths of the parking space and parking space markings are then obtained. These widths, in conjunction with scanning and sampling of the lane boundary lines, are used to identify the divider lines, signaling the completion of parking space detection. In the space occupancy classification stage, region growing and edge detection are applied to extract road and obstacle features within the parking space. Finally, the features are output to an NB classifier to classify whether the parking space is occupied.

3.2. Parking Space Detection Stage

Accurate and precise parking space detection is critical. Such detection data serve as the input for the subsequent stage (i.e., space occupancy classification) and as the reference for later planning of the car’s path into the parking space. In addition, because nearly 70% of all nations worldwide (including Taiwan) drive on the righthand side of the road, the proposed parking space detection method targets only the area to the right of the car.

3.2.1. Feature Extraction

As shown in Figure 4, parking space markings typically comprise a lane boundary line (the red line in Figure 4) and multiple divider lines (the yellow lines in Figure 4) that are parallel to the lane boundary line. Using this characteristic, we can construct an entire parking space by detecting the junctions between the lane boundary line and the divider lines. For corner detection, we chose the features from accelerated segment test (FAST) detector [24] instead of the traditional Harris corner detector. The key difference between the FAST corner detector and other corner detectors is that the FAST detector is an accelerated method that does not need to calculate image pixel gradients; instead, it only needs to determine the relationships between a specific pixel and 16 pixels in a circular mask. Because the circular mask around the corner candidate in the segment test of FAST is operated by examining only four pixels (the four compass directions) for rejecting non-corner points, through sampling, four of the masked pixels are chosen to predict the result. Therefore, this method is substantially faster.

Figure 5 shows the results using the FAST detector; green dots indicate corners. In addition, to reduce outliers (i.e., corners that are not part of the parking space) and to reduce the processing load in the next stage of feature classification, we defined a region of interest to limit the corner detection field (the area outlined in brown in Figure 5).

3.2.2. Feature Classification

Because the features we intend to extract are corners at junctions between parking space lines, we defined two separate criteria to eliminate corners that are not part of the parking space lines: circular intensity proportion and maximum vertical intensity profile in the x-axis.

(a) Circular Intensity Proportion

Figure 6a illustrates the circular intensity proportion of a given corner. The green dots indicate the corners, and the red circle indicates the area of intensity proportion. Firstly, to clearly distinguish parking space lines from the road, binary processing of the image is performed. Next, using the corner as the center of a circle, we use the radius r to calculate its circular intensity proportion. From this, it can be observed that 75% of the circular area is high intensity (i.e., parking space markings) and 25% of the circular area is low intensity (i.e., road). Therefore, we used the criteria to eliminate corners that are not part of the parking space lines. The process for this elimination is shown in Figure 6b.

(b) Maximum Vertical Intensity Profile

Lane boundary lines are represented as long, straight lines of high brightness in AVM images. We used this characteristic to project intensity values onto the x-axis, as shown in Figure 7, in which red dotted lines mark the region of interest. The areas of maximum vertical intensity profile fall approximately near the lane boundary line; therefore, we can eliminate corners that are not near the lane boundary line as they are not part of the parking space. Figure 8 shows the results of feature classification.

3.2.3. Lane Boundary Line Recognition

Although we could simply connect classified corners to form the lane boundary line, the final set of classified corners inevitably includes outliers. Therefore, we chose to use the random sample consensus (RANSAC) algorithm [25], which is robust at detecting outliers, to identify the lane boundary line. The RANSAC algorithm is shown in Figure 9a. Because the most time-consuming portion of the algorithm is the K reiterations of the entire process, we added a criterion to accelerate the entire algorithm (the yellow rhombus in Figure 9a). The criterion is that the calculated line model must be oriented within θ° of the direction of travel. If a calculated line model does not meet this criterion, testing of that particular model is skipped. This not only increases calculation speed but also decreases errors in detection. Figure 9b shows the results of the lane boundary line recognition; the red line indicates the lane boundary line.

3.2.4. Space Type Determination

The type of parking space must be identified before a parking strategy can be determined. Divider lines must be connected to the lane boundary line. Using this characteristic, we can assume a virtual lane boundary line in a binary image, scan d pixels to the right along the x-axis, and record accumulated pixels along the y-axis. The results of this scan are shown in Figure 10a and Figure 11a. Because the scanning process can contain noise, such as parking space serial numbers, we firstly filtered accumulated pixels under a threshold ($0.75d$)). Next, the line width, $W_{line}$ (10 pixels = 13 cm) served as a criterion to eliminate errors in detection. The filtered results are shown in Figure 10b and Figure 11b. By observing the distance between peaks $W_{slot}$ (178 pixels = 231 cm in Figure 10b, 435 pixels = 566 cm in Figure 11b), the parking space can be classified as a perpendicular or parallel space.

3.2.5. Divider Line Recognition

After determining the type of parking space, we obtained three samples along the x-axis starting from the virtual lane boundary line, as shown in Figure 12a. Each color in the figure indicates the results from one sampling. The intensities obtained from the three samplings were recorded. Next, we searched for white-to-black (W2B) and black-to-white (B2W) interface pairs along the markings. The line width, $W_{line}$, was then used to select candidate points that may be located on a divider line. Finally, the parking space width, $W_{slot}$, served as a criterion to eliminate erroneous candidates, as shown in Figure 12b. In this figure, the crosses and rhombi represent B2W and W2B interfaces, and the circles represent candidate points that may be located on a divider line. During the three samplings along the virtual lane boundary lane, if a candidate point on a divider line is classified incorrectly or is accidentally omitted, a voting mechanism still allows for the choice of possible locations among the three samplings. Consequently, the least squares method [26] was adopted to fit the three points into a line, which is then identified as a divider line. The divider line recognition process is illustrated in Figure 13a, and Figure 13b shows the results of the process; yellow lines indicate divider lines.

3.3. Space Occupancy Classification Stage

Once the parking space detection stage is complete, we need to test the vacancy of each parking space because we cannot park in a space already occupied by another car. Here, we view each parking space as occupying a cell in a grid and determine its vacancy by extracting the features within the parking space.

3.3.1. Parking Space Feature Extraction

Unlike traditional methods that only analyze the obstacle features within a parking space, the proposed method is a highly robust analysis that observes parking space features from two perspectives. The red area in Figure 14 represents the results of viewing the space from the perspective of an occupied space. The parking space contains many features of a vehicle, such as edges and symmetry. The green area in Figure 14 represents the results of viewing the space from the perspective of a vacant space. The parking space contains only features of the road, such as consistency in texture. By integrating both perspectives, the features within a parking space can be classified as road or obstacle. Road features were used to provide information regarding the possible vacancy of a parking space, and obstacle features were used to provide information regarding the possible occupancy of a parking space. Finally, these two types of information were integrated to determine whether a specific parking space is occupied. For road features, a region growing method was used to search for usable space within the parking space. For obstacle features, an edge detection method was used to determine whether a car is parked in the parking space.

(a) Vacant Space Detection Based on Region Growing Method

One of the essential features of road is that all areas of the road have a similar texture. Therefore, we used the region growing method to search for paved areas within the parking space. Key parameters set within the region growing algorithm were initial positions of seed points, growth characteristics, and similarity threshold values. Firstly, we selected seed points (red crosses in Figure 15b) near two corner features in the parking space. Two seed points were selected to prevent a single seed point from randomly being located near a road stain, which would inhibit its growth. Next, we chose standard deviation as the growth characteristic instead of the more typical grayscale because grayscale often undergoes dramatic changes caused by lighting. In addition, when the car color is similar to the road color, overgrowth often occurs. In contrast, standard deviation produces more optimal growth in both of these situations. The standard deviation feature map is shown in Figure 15a. Finally, the similarity threshold values for region growing were dependent on lighting. The threshold values were lower for settings with dimmer lighting. Figure 15b shows the results after region growing. White areas indicate detected vacant spaces. To quantify road features into numerical data, we calculated the region growing ratio within the parking space using Equation (1).

$$Region growing ratio=\frac{Growing pixel numbers in the slot}{Total pixel numbers in the slot}.$$

(1)

(b)  Vehicle Detection Based on Edge Features

Obstacle objects in parking spaces refer to cars. Features that indicate the existence of a car include symmetry, as well as complex line structures. However, objects in AVM images are distorted based on the angular relationship between a specific object and the center of the camera (e.g., the car in the first parking space in Figure 16). Therefore, symmetry is unsuitable for use as a feature. Instead, we chose Canny’s edge detection approach [27] to extract edge features within the parking space. Although the field of vision of AVM systems only covers approximately half of the depth of a parking space, a vehicle typically occupies at least three-quarters of a parking space. Thus, the front of the vehicle, except in the case of motorcycles, is always clearly within the field of vision. In addition, the front of the vehicle always contains more edges than any other part of the vehicle. Therefore, we used the number of edges within a parking space as the obstacle feature. Figure 16 shows the edge detection results; cyan lines represent edge features detected within the parking space.

3.3.2. Naïve Bayes Classifier

After extracting the features within the parking space, we need to classify the features to determine whether the space is vacant. Regarding our choice of classifier, although support vector machines have excellent classification performance, they involve relatively complex models and require manual manipulations of numerous parameters. Furthermore, classifying the status of a parking space is a simple binary problem. Thus, we chose the NB classifier, which is a simple model with satisfactory performance [28].

NB classifiers are based on Bayes theorem and the assumption of independence among features. Bayes theorem can be stated as follows:

$$p\left(C|F_1,\dots ,F_n\right)=\frac{p\left(C\right)p(F_1,\dots ,F_n|C)}{p\left(F_1,\dots ,F_n\right)},$$

(2)

where C and F are the types and features of the training set, respectively. Because of the assumption that features are independent, Equation (2) can be rewritten as Equation (3), which can be simplified to Equation (4), which is the Bayesian probability model.

$$p\left(C|F_1,\dots ,F_n\right)\propto p\left(C\right)p\left(F_1|C\right)p\left(F_2|C\right)p\left(F_3|C\right)\dots ;$$

(3)

$$\propto p\left(C\right){\displaystyle \prod }_{i=1}^{n}p\left(F_i|C\right).$$

(4)

The NB classifier is primarily composed of the model in Equation (4) and a corresponding decision rule. This rule states that the most likely possibility should be chosen. In the learning stage, the input is the training set and the outputs are the parameters that are required to be estimated for the classifier. To select appropriate parameters for the classifier from the training set, one can use maximum a posteriori (MAP) or maximum likelihood estimation (MLE). In this paper, MAP is used to estimate parameters for the decision rule of the Bayes classifier. The corresponding classifier is shown in Equation (5).

$$classify\left(f_1,\dots ,f_n\right)=\underset{c}{\mathrm{argmax}}p\left(C=c\right){\displaystyle \prod }_{i=1}^{n}p\left(F_i=f_i|C=c\right),$$

(5)

where $C$ can be occupied or vacant, and $F$ is the region growing ratio and number of edges. Figure 17a is the scatter plot of features for the perpendicular parking space training set. This figure shows that vacant parking spaces are characterized by high region growing ratios and low numbers of edges. By contrast, occupied parking spaces are characterized by region growing ratios that are not high and a more scattered distribution of numbers of edges. Figure 17b,c present the Gaussian distributions of region growing ratio and number of edges, respectively, for the perpendicular parking space training set. These figures show that parking spaces can be generally classified as vacant or occupied by these two features. Figure 18a is the scatter plot of features for the parallel parking space training set. The key difference between these results and the results of the perpendicular training set is that vacant parking spaces are no longer characterized by low numbers of edges. This is because edges in roadside parking spaces are inevitably affected by noise, such as concrete surfaces, storm drain covers, or manhole covers. Figure 18b,c present the Gaussian distributions of region growing ratio and number of edges, respectively. These figures show that the number of edges feature cannot be used to classify parking spaces as vacant or occupied.

Finally, Figure 19 shows the classification results using the NB classifier; red frames indicate occupied parking spaces and green frames indicate vacant parking spaces.

For the space occupancy classification stage, two perspectives were used to detect features within the parking space, which can be classified as road or obstacle features. For road features, a region growing method was used to search for usable space within the parking space. For obstacle features, an edge detection method was used to determine whether a car is parked in the parking space. Finally, features were output to an NB classifier to classify the current status of the parking space.

4. Experimental Results

This section introduces the experiment to test the system, including the equipment, process, and conditions, as well as the results of the experiment. Examples of failed detection are analyzed and discussed.

4.1. Experimental Equipment

The AVM system is shown in Figure 20. Individual images from wide-angle lenses, located at the front and back of the car and the left and right rear-view mirrors, are converted into bird’s-eye view images and merged to produce the output image.

4.2. Experimental Process

The experimental process was divided into two stages: parking space detection and space occupancy classification. In the parking space detection stage, the FAST corner detector was used to extract features. To reduce outliers and processing load, we designated a 700 × 100-pixel region of interest to limit the detection field. Next, a circle was defined by choosing a detected corner as the center and by setting the radius to one-half of the marking line width (five pixels). The circular intensity proportion of this circle and the maximum vertical intensity profile in the x-axis were used to classify features. Because the set of classified corners can include outliers, we used RANSAC to estimate the model with the best fit; this model indicates the lane boundary line. We set the number of iterations to 50 in consideration of the small number of feature points. In addition, we defined the criterion that the detected lane boundary line must be within 10° of the direction of travel to improve the accuracy of the lane boundary line. Divider lines must connect to the lane boundary line. Using this characteristic, we assumed a virtual lane boundary line, scanned 20 pixels to the right along the x-axis, and recorded accumulated pixels along the y-axis. Accumulated pixels under the threshold (15 pixels) were filtered out, and the width of the marking lines (10 pixels) was used to further eliminate errors in detection. After doing so, the type of parking space could be determined by observing the distance between peaks. After identifying the type of parking space, we obtained three samples perpendicular to the lane boundary line using criteria based on widths of the marking lines and the parking space. Finally, the least squares method was used to fit the three points into a divider line. This concluded the parking space detection stage.

In the space occupancy classification stage, we viewed each parking space as occupying a grid in a grid and used road and obstacle features to determine whether the parking space was vacant. Because the texture of the road is homogeneous, we used region growing to search for road information. Because the seed points must accurately record the road, we chose locations near two corner features of the parking space. Standard deviation was used as the growth characteristic. The road feature was defined as the ratio of region growing area to total area. For the obstacle analysis, we observed that cars have complex structures and, therefore, must have a high number of edges. Therefore, we chose to use Canny’s edge detector and defined the number of edges as the obstacle feature. Because the occupancy or vacancy of a parking space is a simple binary problem, we chose the less complex NB classifier for classification. Firstly, we needed to calculate the a priori probability of the existing data. The a priori probabilities of an occupied space and a vacant space are 0.5 each. Next, we individually calculated the probability that the features indicate an occupied or vacant slot; high-probability classifications are regarded as classified results. The entire experimental process and framework is depicted in Figure 21.

4.3. Experimental Results

4.3.1. Indoor Experimental Results

Because of their uneven or dim lighting and pillars, manmade indoor parking structures are one of the most challenging settings. Figure 22a shows the B5 parking structure at the National Taipei University of Technology. Figure 22b shows the detection process. In this figure, the first parking space has very few edges. In combination with the high region growing ratio, this space was classified as vacant. The second space was occupied by a parked car; thus, a large number of edges were detected. In addition, the region growing ratio was relatively low. Thus, this space was classified as occupied. The third space was vacant, but because the angle between the car in the second parking space and the camera was not absolutely perpendicular, part of that car was projected into the third parking space. This resulted in an increased number of edges. However, with the assistance of road features, the space was correctly classified as vacant.

Table 1. Parking space features at the National Taipei University of Technology.

Header	Growing Ratio	Edges (Pixels)	Status
First slot	0.894	44	Vacant
Second slot	0.199	1556	Occupied
Third slot	0.845	230	Vacant

lists the quantified data for these results. Figure 22c shows the detected results.

4.3.2. Outdoor Experimental Results

Because outdoor settings contain more variables, we chose three different sites for the outdoor experiments: an outdoor parking lot, a parking lot under a bridge, and roadside parking spaces. Figure 23a shows the parking lot at Guang Hua Digital Plaza. Figure 23b shows the detection process. In this figure, the cars in first and third parking spaces are severely distorted. This is an imaging problem caused by the non-perpendicular angles between the cars and the camera. Regardless, the first and third parking spaces contained clear obstacle features such as tires or other parts of a car; therefore, they were classified as occupied. The second parking space exhibited features characteristic of a vacant parking space, i.e., high region growing ratio and low number of edges; thus, it was classified as vacant.

Table 2. Parking space features at Guang Hua Digital Plaza.

Header	Growing Ratio	Edges (Pixels)	Status
First slot	0.394	1654	Occupied
Second slot	0.908	0	Vacant
Third slot	0.325	1325	Occupied

lists the quantified data for these results. Figure 23c shows the detected results.

Figure 24a shows the parking lot underneath the overpass at Xinsheng North Road. Because the parking spaces are covered by the bridge above but open to sunlight on both sides, the problems caused by uneven lighting and shadows are rather severe. Figure 24b shows the detection process. In this figure, the first parking space is unevenly lit on the left and right sides. This decreased the contrast between road and obstacle features and caused erroneous region growth. However, a high number of edges were detected within the parking space; thus, it was classified as occupied. Because of the road stain in the second parking space, region growing did not successfully fill the entire parking space. However, too few edges were detected within the parking space, thereby rendering a classification of vacancy. The car in the third parking space was severely distorted and very few obstacle features were visible. However, with the assistance of road features, the space was correctly classified as occupied.

Table 3. Parking space features at Xinsheng North Road.

Header	Growing Ratio	Edges (Pixels)	Status
First slot	0.473	696	Occupied
Second slot	0.744	5	Vacant
Third slot	0.339	480	Occupied

lists the quantified data for these results. Figure 24c shows the detected results.

Figure 25a shows an urban roadside parking setting. The only option for outdoor parking, other than a parking lot, is roadside parallel parking. One key difference between perpendicular and parallel parking is that the AVM’s field of vision can only detect one parking space at a time. Figure 25b shows the detection process when a parking space is occupied. Unlike the obstacle features of a perpendicular parking space, which contain the line structures of the front of the car and the license plate, the obstacle features of the parallel parking space primarily contain the line structures on the lateral side of the car and tires. The two types of parking spaces are similar in that an occupied parking space has a lower region growing ratio and a higher number of edges. Figure 25d shows the detection process when a parking space is vacant. Because the parking space is located at the side of the road, its features are affected by unavoidable noise such as concrete surfaces, storm drain covers, or manhole covers. These environmental factors can affect the extraction of obstacle features; therefore, we are more reliant on the information provided by road features. Furthermore, the training set for parallel parking differed from the training set for perpendicular parking.

Table 4. Parking space features for urban roadside parking setting.

Header	Growing Ratio	Edges (Pixels)	Status
First slot	0.25	2036	Occupied
Second slot	0.928	867	Vacant

lists the quantified data for these results. Figure 25c,e show the detected results.

4.4. Performance Evaluation

To evaluate the performance of the algorithm, we examined parking space detection and space occupancy classification separately using recall and precision rates, which are defined as follows:

$$Recall=\frac{No.ofcorrectlydetectedslots}{No.ofexistingslots},$$

(6)

$$Precision=\frac{No.ofcorrectlydetectedslots}{No.of\mathrm{detected}slots}.$$

(7)

There are four relationships between classification results and actual conditions, defined as follows:

• True positive (TP): Classified as occupied, and is actually occupied.
• False positive (FP): Classified as occupied, but is actually vacant.
• True negative (TN): Classified as vacant, and is actually vacant.
• False negative (FN): Classified as vacant, but is actually occupied.

We chose the error rate to evaluate classification, as defined in Equations (8)–(10):

$$FN Rate=\frac{No.of\mathrm{occupied}\mathrm{slots}\mathrm{being}detected as vacant}{No.ofoccupiedslots},$$

(8)

$$FP Rate=\frac{No.of\mathrm{vacant}\mathrm{slots}\mathrm{being}detected as \mathrm{occupied}}{No.ofvacantslots},$$

(9)

$$Error Rate=\frac{No.of\mathrm{in}correctdetection}{No.ofcorrectlydetectedslots}.$$

(10)

4.4.1. Parking Space Detection Rates

Table 5. Indoor and outdoor parking space detection rates.

Scene	Indoor	Outdoor
Computing time (s)	0.17	0.16
Parking type	Perpendicular parking	Perpendicular parking	Parallel Parking	Total
# of existing slots	311	545	233	778
# of detected slots	274	521	224	745
# of correctly slots	263	519	224	743
Recall (%)	84.6	95.2	96.1	95.5
Precision (%)	96	99.6	100	99.7

lists the parking space detection rates for both indoor and outdoor settings. This table shows that the number of detected spaces was substantially lower than the number of actual spaces in indoor settings. The primary reason for this is uneven lighting. This problem causes a lower contrast between parking space marking lines and the road and leads to the omission of some parking spaces. In addition, columns in parking lots often cause the detected parking spaces to include errors in detection. In outdoor settings, a direct relationship was found between the detection rate of parking spaces and the integrity of the parking space markings. Because the marking lines are constantly exposed to weather, they are likely to be fragmented. This disrupts the parking space structure and affects parking space detection. In addition, if sunlight casts shadows on the junctions between marking lines, the system is unable to detect the corner and results in a detection failure.

4.4.2. Space Occupancy Classification Rates

Table 6. Indoor and outdoor space occupancy classification rates. FNR—false negative rate; FPR—false positive rate.

Scene	Indoor			Outdoor
Feature type	Edge-based [10]	Growing -based	Fusion	Edge-based [10]	Growing -based	Fusion
Computing time (s)	0.14	0.13	0.27	0.13	0.1	0.24
# of occupied slots (positive)	175	173	179	394	333	333
# of vacant slots (negative)	88	90	84	349	410	410
# of false negative	17	21	12	6	11	7
# of false positive	4	6	3	77	20	18
FNR (%)	9.7	12.1	6.7	1.5	3.3%	2.1
FPR (%)	4.5	6.7	3.6	22	4.9	4.4
Error Rate (%)	7.1	9.4	5.2	11.8	4.1	3.3

lists the space occupancy classification rates for both indoor and outdoor settings. We provide an in-depth discussion of the classification rates obtained from different features. In indoor parking settings, we observed that the region growing method produced higher error rates than the edge detection method in detecting features. This was caused by dim lighting in indoor parking structures. Inadequate lighting decreased the contrast between the road and cars and led to errors in region growing. In outdoor parking settings, particularly for roadside parking, the edge detection method produced higher error rates than the region growing method. This is because a vacant parking space may contain objects with a high number of edges and a high amount of noise such as manhole covers or storm drain covers. This noise is easily mistaken for a car in the parking space. These results showed that road features and obstacle features are each advantageous in different settings. Thus, both types of features were included to offset the inadequacies of using only one type of features. The final results confirmed that the fusion of both feature methods produced the lowest error rates in both indoor and outdoor settings.

4.5. Analysis of Detection Failures

4.5.1. Parking Space Detection Failures

Although the proposed algorithm performed favorably in both indoor and outdoor settings, relying solely on image-based information was inadequate in certain situations. Some examples of failures and the reasons for the failures are discussed below. Figure 26 shows examples of parking space detection failures. Figure 26a shows an example where dark shadows were caused by the lighting. Because the shadows covered the parking space markings, the corner detector was not able to successfully extract features. Figure 26b shows an example where the parking space markings are severely damaged. Because the paint at the line junctions was faded, the system was unable to detect corners. Figure 26c shows an example involving standing water. The divider lines were not detected because of reflections in the water.

4.5.2. Space Occupancy Classification Failures

Figure 27 shows examples of space occupancy classification failures. Figure 27a shows an example involving standing water. Reflections in the water resulted in a large number of edges in the parking space, and the space was mistakenly classified as occupied. Figure 27b shows an example involving a severely distorted car. The car in the parking was distorted because it was too close to the camera. This caused the image of the car to be extended to the adjacent parking spaces, and the spaces were mistakenly classified as occupied. Figure 27c shows an example involving a motorcycle. According to regulations, heavy motorcycles must be parked in a parking space. However, because motorcycles are substantially smaller than cars, they are often outside the AVM’s field of vision if they are parked too deep in the parking space. Thus, this space was mistakenly classified as vacant.

4.6. Summary

In this paper, an AVM-based method to detect vacant parking spaces was proposed. To test its viability in real-life settings, we performed experiments in indoor parking lots, outdoor parking lots, and roadside parking spaces. Table 5 shows that the parking space detection algorithm had high recall and precision rates in outdoor settings. In comparison, the algorithm was less effective at detecting parking spaces in indoor settings because of factors such as lighting and columns. However, the algorithm had a high precision rate for detected spaces. Table 6 shows that the proposed space occupancy classification algorithm achieved favorable classification rates in both indoor and outdoor settings. In particular, integrating both types of feature extraction methods produced the lowest error rates. Because our algorithms are based on visual detection of vacant parking spaces, lighting can cause serious problems that must still be resolved. Using the system at night is another obstacle that awaits solutions. In the future, we hope to use active sensors to improve the robustness of the system.

5. Conclusions and Prospects

This study proposed a parking space detection method based on an AVM system and related vacancy analysis algorithms to facilitate indoor parking, where various directional lighting sources are present, and outdoor parking, where environmental variations can be extensive. The proposed method performed well in both settings. An AVM system, with its wider field of vision, was chosen as the sensor instead of traditional monocular cameras because the AVM system produces overhead images that look like they were shot from a camera mounted above the car roof. These images eliminate blind spots and provide clear views of nearby obstacles. This is the same reason why major car manufacturers began to invest in similar systems [29] and why these systems gradually became standard equipment in most cars. In addition, a highly robust analysis method was proposed to classify parking space occupancy. Two perspectives were used to detect features, classified as road or obstacle features, within the parking space. Road features were used to provide information regarding the possible vacancy of a parking space, and obstacle features were used to provide information regarding the possible occupancy of a parking space. Finally, these two types of information were integrated to determine whether a specific parking space is occupied. For road features, a region growing method was used to search for usable space within the parking space. For obstacle features, an edge detection method was used to determine whether a car is parked in the parking space. In indoor parking settings, we observed that the region growing method produced higher error rates than the edge detection method in detecting features. This was caused by dim lighting in indoor parking structures. Inadequate lighting decreased the contrast between the road and cars and led to errors in region growing. In outdoor parking settings, particularly for roadside parking, the edge detection method produced higher error rates than the region growing method. This was because a vacant parking space may contain distracting objects such as manhole covers or storm drain covers. These distractors were easily mistaken for a car in the parking space. These results showed that road features and obstacle features each had their advantages in different settings. Thus, both types of features were included to offset the inadequacies of using only one type of features. Compared to traditional image-based methods of analyzing parking space occupancy, the proposed method demonstrated superior performance. In addition, with the exception of the two types of parking slot markings, the other slot markings can be classified into long T, short T, slanted, slanted long T, and slanted short T types. We also tested the proposed approach with the five types of parking slot markings (virtual images) representative of real conditions. The experimental results showed that our proposed approach can also successfully be used in these five types of parking lots.

By integrating both road and obstacle feature analysis, this method improves upon traditional image-based methods of parking space analysis. The performance evaluation detailed in Section 4 showed that the method’s detection rate was lower in indoor settings than outdoor settings because lighting problems are more severe in indoor settings than outdoor settings. However, the method achieved favorable detection performance overall. Furthermore, we tested and compared performance based on road features, obstacle features, and a combination of both. The results showed that integrating both types of features produced the lowest classification error rates. Future research into vacant parking space detection algorithms can consider one of three directions. Firstly, because our detection of vacant parking spaces was entirely based on image sensors, the system may encounter numerous issues at night or in settings where lighting has a serious impact. Therefore, we hope to integrate active sensors in the future to enhance the robustness of the system and achieve truly automated parking. Secondly, currently, the time needed for real-time detection of a vacant parking space is approximately 0.4 s/frame. In the future, porting the system from MATLAB to the C programming language can improve computational efficiency (less than 0.1 s). Thirdly, for parking space detection failures (dark shadows, severely damaged markings, and puddles), we think that using a CNN to detect whether a car is present or absent (instead of a naïve Bayes with edge data) in the future will enhance the robustness of the system and achieve truly automated parking.