A Fast and Noise Tolerable Binarization Method for Automatic License Plate Recognition in the Open Environment in Taiwan

Abstract

License plate recognition is widely used in our daily life. Image binarization, which is a process to convert an image to white and black, is an important step of license plate recognition. Among the proposed binarization methods, Otsu method is the most famous and commonly used one in a license plate recognition system since it is the fastest and can reach a comparable recognition accuracy. The main disadvantage of Otsu method is that it is sensitive to luminance effect and noise, and this property is impractical since most vehicle images are captured in an open environment. In this paper, we propose a system to improve the performance of automatic license plates reorganization in the open environment in Taiwan. Our system uses a binarization method which is inspired by the symmetry principles. Experimental results showed that when our method has a similar time complexity to that of Otsu, our method can improve the recognition rate up to 1.30 times better than Otsu.

1. Introduction

Recently, IT technology has been applied symmetrically in our daily life and intelligent transport system (ITS) has become more and more important. License plate recognition (LPR) is a normal and essential core technology of ITS, and it has been a positive research area in the past few years [1,2,3,4]. LPR is used widely in our daily life, such as traffic law enforcement [5], electronic toll collection, automatic payment of tolls on highways, and parking management systems [6]. LPR also helps vehicle tracking and provides a reference for activity analysis of vehicles [7]. Because of the different environments, LPR technologies vary from application to application and many related methods have been proposed. Ashtari, Nordin, and Fathy [8] proposed a method to detect the Iranian license plates by using a template on connected target pixels with the specific color. However, this method is invalid when there are areas whose color information is similar to that of the license plate. Türkyılmaz and Kaçan [9] proposed a LPR method using artificial neural networks. This method improved the recognition rate, but it spent more time on recognizing characters in license plates. Al-Ghaili et al. [10] proposed a fast vertical edge detection algorithm to improve the speed of license plate detection. The authors also compared their method to the Sobel operator in terms of accuracy and processing time. Du et al. [11] presented a survey on existing LPR methods, classified them according to the features used in each stage, and compared them in terms of accuracy and processing speed.

The main challenge of LPR is the large variety of license plates, which differ with respect to shape [12], color, size, and the ratio of height to width [13]. Other difficulties of LPR include severe poor lighting [14], weather conditions [15], the distance between vehicle and camera, and prescribed driveways [8]. The development of an accurate and efficient license plate recognition system (LPRS) that solves these problems mentioned above is important. In general a typical LPRS consists of three main stages [9]: license plate detection, character segmentation, and character recognition, as shown in Figure 1. The goal of license plate detection is to detect vehicle’s license plates in a captured image. Then in the stage of character segmentation [16], the LPRS aims at finding the segmentation of the characters on license plates. Finally, in the character recognition stage, different methods such as support vector machine [17], artificial neural networks [9], or template matching [18] is used to recognize the characters in a license plate.

Figure 1. Exampled flowchart of a typical license plate recognition system (LPRS).

For a typical LPRS, an image processing technique called image binarization is used to convert gray level images into binary ones. In general, image binarization uses grayscale pixels information to calculate a threshold value, and this threshold is then used to classify the image pixels as belonging to the background or foreground. A good image binarization algorithm helps a LPRS to achieve better efficiency and recognition accuracy. There are many proposed binarization methods such as iterative thresholding [19], peak-and-valley method [20], Otsu method [21], and so on. Proposed image binarization methods can be divided into six groups [22]: histogram shape-based methods, clustering-based methods, entropy-based methods, object attribute-based methods, spatial methods, and local thresholding methods. For a LPRS, real-time recognition is very important and the image background is usually very complicated. Therefore, spatial, entropy-based, and object attribute-based methods were rarely adopted in a LPRS since they require more computational time or have a poor performance when the image background is too complicated. Local thresholding methods were usually used in documents recognition because they have better performance under illumination conditions but also need more computational time.

The most famous and commonly binarization method adopted by a LPRS is Otsu, which is a clustering-based method [21,23,24,25]. According to our experimental analyses, we found that Otsu has two main drawbacks. First, Otsu has a poor performance when a license plate is in a bright or dark environment. That is, Otsu is very sensitive to the ambient light. An example is shown in Figure 2. Figure 2a shows a license plate captured in a bright environment and Figure 2b is the corresponding binary image generated by Otsu. As shown in Figure 2b, when we used Otsu to generate the binary image and then inputted the image into a LPRS, the LPRS was obviously unable to detect and recognize the license plate. Second, Otsu is easily affected by noise such as dirt or rusty screws on license plates. For a license plate in Taiwan, it is usually fixed by two or more screws. Since the climate in Taiwan is hot and humid, these screws get rusty easily. An example is shown in Figure 3. Figure 3a shows a license plate on which two screws got rusty and Figure 3b is the corresponding binary image generated by Otsu. As you can see in Figure 3b, after binarization the rusty screws adhere to the Character “E.” This condition made a LPRS difficult to recognize the character “E” on the license plate.

Figure 2. (a) A license plate captured in a bright environment, (b) the binary image generated by Otsu, where the license plate disappears.

Figure 3. (a) A license plate on which two screws get rusty, (b) the binary image generated by Otsu, where rusty screws adhere to the Character “E.”

In order to solve the above-mentioned problems, we propose a LPRS named FANS (FAst and Noise tolerable automatic license plate recognition System). FANS cuts the image into small regions and then calculates a threshold for each region. This property enables FANS to be insensitive to ambient light and noise, and therefore produce better binary images to improve the recognition rate of license plates. Besides, FANS uses a strategy named summed area table [26], which will be discussed in Section 3, to reduce the binarization time. Experimental results showed that FANS needed less execution time than the LPRS using Otsu.

The rest of this paper is organized as follows. In Section 2 we discuss the related works. In Section 3 we propose our LPRS. In Section 4, experimental results and analyses are provided. Finally Section 5 presents the conclusions.

2. Related Works

In this section, we briefly discuss the three main stages of a typical LPRS and the proposed image binarization techniques.

2.1. License Plate Detection

The main goal of license plate detection is to identify some local patches containing license plates in an image. Because a license plate can exist anywhere in an image with different sizes, it is impractical to check every pixel in the image to locate the license plate. A preferable approach is to extract some useful features, such as the ratio of height to width or color and gray level information from images, and then use the obtained information to locate license plates.

Proposed license plate detection methods can be roughly classified into three categories [27]: texture-based, edge-based, and color-based. Among the three kinds of methods, edge-based method is the most popular because of its reliable performance. In general, a license plate is characterized by a rectangular shape with a specific aspect ratio. As a result, license plates can be theoretically extracted by checking all possible rectangles in the image. Kamat and Ganesan [28] used the Hough transform to find the boundaries of license plates and detected the license plates regions. However, the Hough transform’s computational complexity is very high and it is valid only when the image background is simple. Zheng, Zhao, and Wang [29] combined edge detection with the morphological operations to find rectangles of license plate location. The disadvantage of this method is that it is sensitive to the brightness change in the license plate region and the complicated background in images. There are two main problems of edge-based methods [23]. First, edge-based methods may be confused by some objects with rich texture and similar shape. Second, when a license plate changes its location in an image compared to observation, edge-based methods may only detect partial license plates or even miss them.

Color-based methods are based on the observation that many countries have specific colors in their license plates. Therefore we can intuitively extract license plates by locating colors in the images. For example, Shi, Zhao, and Shen [30] used a classifier of color model to detect the plates. Yang et al. [31] proposed a method based on fixed color collocation to detect license plates. Their method can eliminate the disturbances of the fake plate’s region and locate the plate efficiently by the color collocation of the plates’ background and characters combined with the plates’ structure and texture. Zhu, Hou, and Xing [32] proposed a method based on color feature and mathematical morphology to locate license plates in a complicated background. In their method, the color of each pixel in the image is identified by some characteristic functions. Then candidates of license plate areas are generated by a series of mathematical operations, and the license plate is extracted from these candidates. The main advantage of license plate detection using color information is that it can locate the deformed and inclined license plates. However, color-based approaches are sensitive to the illumination changes and complicated backgrounds.

Methods in the third category are focused on texture features. There are many texture features in license plates, such as color, shape, size, and so on. Kim, Jung, and Kim [33] presented a color texture-based method for license plate detection. They used support vector machine to analyze the color textural properties of license plate, and then locate license plates. Porikli and Kocak [34] used a covariance matrix and neural network to detect license plates. Their method converts an image into a feature covariance matrix, which can capture not only the license plate appearance but also the statistical properties of image regions.

2.2. Character Segmentation

Character segmentation is the procedure of extracting characters from license plates in an image [35]. The main objective of character segmentation is to obtain clean characters to improve the recognition rate of license plates. Many character segmentation methods have been proposed such as horizontal and vertical projections, mathematical morphology, and contour tracking. Among them, projection methods are commonly used in LPRS [36]. The main principle behind vertical projection is to calculate the number of pixels of each column on a license plate and use this information to separate characters. Shi, Zhao, and Shen [30] separated characters from each other using vertical projection at binary level. Nomura et al. [37] proposed a morphological method based on the histogram for character segmentation. Their method separates characters by searching for segmentation points in a histogram and then these points are combined to find characters.

Connected component analysis is another important technique in binary image processing. This technique was commonly used in license plate detection and character segmentation. It scans a binary image and labels its pixels into components based on pixel connectivity. Once all groups of pixels are determined, each pixel is labeled with a value according to the component to which it is assigned. According to the number of components used in connected component analysis, this method can further be classified into 4-connectivity and 8-connectivity [36].

2.3. Character Recognition

There are also many proposed character recognition methods such as support vector machine [17], artificial neural network [9], and template matching [18]. Türkyılmaz and Kaçan [9] used artificial neural network to recognize license plates. This method improves the recognition rate but spends more computational time to recognize license plates. Khan and others [17] proposed support vector machine to deal with the problems of light variations, occlusion, and multi-views. Their method can reach good recognition rate under these conditions. Gazcón, Chesñevar, and Castro [18] proposed an approach named intelligent template matching to solve the LPRS problem of Argentinean license plates. Experimental results showed that their method is superior to artificial neural network in classification accuracy and training time.

2.4. Image Binarization

Image binarization is a process to convert an image to white and black. In general, a certain threshold is chosen to classify certain pixels as white and certain pixels as black. Proposed image binarization method can be divided into six groups according [22]: histogram shape-based methods, clustering-based methods, entropy-based methods, object attribute-based methods, spatial methods, and local thresholding methods. Histogram shape-based methods achieve thresholding based on the shape properties of the histogram. The main idea is to calculate the number of pixels in an image to obtain the corresponding histogram. Then the threshold is selected by searching for peaks and valleys in the histogram or by searching overlapping peaks via curvature analysis. For example, Rosenfeld and De La Torre [38] analyzed the concavities of histogram to choose a threshold. When the convex hull of the histogram is calculated, the deepest concavity points became candidates for a threshold. Sezan [17] proposed an adaptive peak detection algorithm. The algorithm uses a peak detection signal derived from the histogram to locate the peaks and chooses a threshold from peaks. The main problem of histogram shape-based methods is that they are easily affected by pixel distribution.

In clustering-based methods, the gray level data undergoes a clustering analysis in which the number of clusters is always set to two. Accordingly, pixels from gray level data are classified to two clusters: background and foreground. Since the two clusters correspond to the two lobes of a pixel histogram, some authors search for the midpoint of the peaks and treat the midpoint as a threshold [22]. Ridler and Calvard [19] proposed an iterative thresholding method. At each iteration, a new threshold is obtained by calculating the average of the foreground and background pixels. Otsu [18] minimized the weighted sum of within-class variances of the background and foreground pixels to establish a best threshold. Otsu has proved that the minimization of within-class variances is tantamount to the maximization of between-class scatter. When the numbers of pixels in each class were close to each other, Otsu method reaches satisfactory results.

Entropy-based methods utilize the entropy of the distribution of the gray levels to binarize images. For example, Kapur, Sahoo, and Wong [39] considered the image foreground and background as two different signal sources, so that when the sum of the two class entropies reaches its maximum, the image got an optimal threshold. The main drawback of entropy-based method is that it requires more computational time to calculate the entropy. Object attribute-based methods select the threshold based on similarity measure or some attribute quality between the original image and the binary image. These object attributes can take the form of shape compactness, gray level moments, edge matching, and so on. Hertz and Schafer [40] proposed a multi-thresholding technique in which a thinned edge field, obtained from the gray level image, was compared with the edge field obtained from the binary image. The threshold was given by the value that maximizes the coincidence of the two edge fields.

Spatial thresholding methods utilize the gray value distribution and dependency of pixels in a neighborhood, for example, in the form of correlation functions, context probabilities, co-occurrence probabilities, and so on. Chanda and Majumder [41] used gray level co-occurrence matrices to get a threshold. This method considers dependency of pixels in a neighborhood in images, and it is effective for segmenting noisy images. Local thresholding methods get a threshold by considering some local statistics like variance or range. White and Rohrer [42] proposed a local contrast method to binarize documents. This method compares the gray value of the pixel with the average of the gray values in the neighborhood. If the pixel is darker than the average, it is denoted as character; otherwise, it is classified as background. Local thresholding methods were usually used in documents recognition, because they have better performance under illumination conditions.

The most famous and commonly binarization method adopted by a LPRS is Otsu method since it is the fastest and can reach a comparable recognition rate [22,24]. For a gray level image, Otsu calculates the frequency and the probability distribution of each pixel and generate a corresponding histogram. Suppose that each pixel in a gray level image is represented by a gray level value i (0 ≤ i ≤ 255, i ∈ Z), the probability distribution p(i) is calculated by Equation (1).

p(i) = n_i/N

(1)

In Equation (1), n_i denotes the number of pixels at gray scale i, N denotes the total number of pixels in a gray level image, Σp(i) = 1, and N = Σn_i. Otsu checks all gray level values to obtain an optimal threshold and transfers a gray level image into binary image by this threshold. For a threshold t, Otsu divides 256 gray scales in an image into two classes: foreground pixel C₁ and background pixel C₂, where C₁ = {0, 1, …, t} and C₂ = {t + 1, t + 2, …, 255}. Then the probability of class occurrence ω₁(t) and ω₂(t) are calculated by Equation (2) and class mean μ₁(t) and μ₂(t) are calculated by Equation (3).

$$\{\begin{array}{c}{\omega }_1\left(t\right)=p\left({\mathrm{C}}_1\right)={\mathsf{\Sigma }}_{i\le t}p\left(i\right),\\ {\omega }_2\left(t\right)=p\left({\mathrm{C}}_2\right)={\mathsf{\Sigma }}_{i>t}p\left(i\right).\end{array}$$

(2)

$$\{\begin{array}{c}{u}_1\left(t\right)={\mathsf{\Sigma }}_{i\le t}\left(p\left(i\right)\ast i\right)/{\omega }_1,\\ u_2\left(t\right)={\mathsf{\Sigma }}_{i>t}\left(p\left(i\right)\ast i\right)/{\omega }_{2.}\end{array}$$

(3)

For any threshold t, the following relation is valid.

ω₁(t) + ω₂(t) = 1

(4)

Otsu uses the within-class variance σ_w²(t) and between-class variance σ_b²(t) as shown in Equations (5) and (6) to evaluate the goodness of a threshold t.

σ_w²(t) = ω₁ × σ₁² + ω₂ × σ₂².

(5)

σ_b²(t) = ω₁(t) × ω₂(t) × [μ₁(t) − μ₂(t)]².

(6)

Otsu has proved that minimization of within-class variances is equivalent to the maximization of between-class scatter. Finally, the optimal threshold t^* is obtained by finding the value that maximizes σ_b²(t) as shown in Equation (7).

σ_b²(t^*) = max σ_b²(t)

(7)

3. Proposed System

3.1. Gray Level Processing

For a LRPS, an input image may consist of many objects and colors. Input images may also be captured in different lighting environment. The gray level conversion can enhance the features of input images and reduce the luminance effect by separating the darker values in images more differently. As in most LPRSs, the first step of FANS is gray level processing. The common method to transform color images into gray level ones is shown in Equation (8), which is defined by the National Television System Committee [43].

Gray = 0.299 × Red + 0.587 × Green + 0.114 × Blue.

(8)

Since floating-point operations spend more computational cost than integer operations, FANS reduces the number of floating-point operations by using Equation (9) to improve system performance. Figure 4 shows an example of gray level conversion in FANS.

Gray = (3 × Red + 6 × Green + Blue)/10.

(9)

Figure 4. (a) An input image, (b) the result of gray scale conversion in our LPRS.

3.2. Fast and Noise Tolerable Binarization Algorithm

As discussed in Section 1 and Section 2, Otsu is the most famous and commonly used image binarization method in a LPRS because it is the fastest and can reach a comparable recognition rate. However, Otsu has two main drawbacks, i.e., it is sensitive to luminance effect and noise. To solve this problem, FANS uses a local threshold method. FANS uses an b × b mask to calculate a threshold T_ij for each pixel p_ij as shown in Equations (10) and (11), where 1 ≤ i ≤ W, 1 ≤ j ≤ H, and W and H respectively denotes the width and height of the image.

T_ij = Σ p_xy/b

(10)

$$\{\begin{array}{c}i-\left(b-1\right)/2 x i+\left(b-1\right)/2,\\ j-\left(b-1\right)/2 y j+\left(b-1\right)/2.\end{array}$$

(11)

However, when grayscale pixels are close to each other in a mask, the pixels of this region may be wrongly classified. For example, when there are night grayscale pixels in a 3 × 3 mask region, i.e., one pixel is 255 and the others are 250, the threshold of this region is 250.55. In this situation, the eight pixels are classified as black because the threshold is a bit higher. However, these pixels should be classified as white since they are actually white pixels but are affected by shadow. In a real application, this situation often occurs when license plates fall under luminance effect. To solve the above situation, FANS subtracts a constant C from threshold T_ij as shown in Equation (12). This strategy was commonly used in many proposed binarization methods [44,45]. The experimental analysis of C in our LPRS is discussed in Section 4.

T_ij − C = ^*T_ij.

(12)

Consequently, FANS uses the threshold ^*T_ij to binarize a pixel p_ij as shown in Equation (13). The main idea behind FANS is that if we use only one threshold to binarize the whole image, the results would be sensitive to luminance effect and noise. FANS finishes the binarization until the mask finishes scanning a whole image.

$$\{\begin{array}{c} \mathrm{if} p_{ij}\le {}^{*}{T}_{ij} , p_{ij}=0, \\ \mathrm{if} p_{ij}> {}^{*}{T}_{ij} , p_{ij}=255.\end{array}$$

(13)

However, similar to the proposed local binarization methods, FANS spends time on calculating thresholds for each pixel. The computational time is affected by the mask size. The larger the mask size, the more binarization time FANS needs. To solve this problem, FANS uses a summed area table (SAT) [26] as defined in Definition 1 to reduce the computational cost.

Definition 1.

Suppose that the mask size is b × b. For a given matrix P[] which records the grayscale p_ij of pixels in an image, a SAT is a matrix in which each element a_ij stores the summation of p_xy as shown in Equation (14), where 1 ≤ x ≤ i, 1 ≤ y ≤ j, and x, y ∈ Z.

a_ij = Σp_xy

(14)

In other words, each element a_ij in a SAT is the summation of p_xy which locates at the top-left corner of p_ij. The illustration of a SAT is shown in Figure 5, where Figure 5a is the pixels distribution matrix of a grayscale image consisting of W × H pixels and Figure 5b is the corresponding SAT generated by FANS. For example, the element a_ij in Figure 5b stores the sum of grayscales within the red region in Figure 5a. It is worth noting that before the image binarization starts, the first row of SAT is initialized to zero to avoid the program crashing.

Figure 5. (a) The pixels distribution matrix of a grayscale image consists of W × H pixels, (b) the summed area table.

FANS uses SAT to reduce the computational time. Without SAT, FANS needs b² calculation steps in a double loop to calculate a threshold T_ij, where b² is the mask size. When FANS uses SAT, the number of steps is reduced from b² to 1. That is, FANS needs only one step to calculate a local threshold. We prove this property in Lemma 1.

Lemma 1.

For a given pixels distribution matrix of a grayscale image, suppose that the mask size is b × b, SAT enables FANS to reduce the number of calculation steps from b² to 1 when calculating a local threshold.

Proof. 

As shown in Equation (10), FANS needs b² steps to obtain the sum Σp_xy to calculate a threshold T_ij. However, for any pixel p_ij, we can divide an image into four regions S1, S2, S3, and S4 as shown in Figure 6a. Since the mask size is b × b, we can get that Σp_xy is the summation of pixels located at Region S4. Since every element a_ij in SAT stores the summation of p_xy which locates at the top-left corner of p_ij, we can get

a_ij = S1 + S2 + S3 + S4,

(15)

as shown in Figure 6b. Similarly, we can get

a_{(I − b)(j − 1)} = S1 + S2,

(16)

a_{(I − 1)(j − b)} = S1 + S3,

(17)

a_{(I − b − 1) (j – b − 1)} = S1.

(18)

Figure 6. (a) An image can be separated into four regions; (b) four corresponding points in a summed area table (SAT).

Combine Equations (15) to (18), we can get

a_ij − a_{(i − b)(j − 1)} − a_{(i − 1)(j − b)} + a_{(i − b − 1) (j − b − 1)} = S4 = Σp_xy

(19)

That is, with SAT, FANS needs only one calculation step, i.e., a_ij – a_{(i − b)j} – a _{i(j − b)} + a _{(i − b) (j − b)}, to calculate a threshold.  □

The pseudocode of binarization in FANS is shown in Figure 7. After inputting the grayscale image in Line 1, FANS generates SAT in Line 2. In Line 4, FANS calculates the local threshold for each pixel and utilizes the SAT to speed up the calculation time. In Lines 5 to 9, FANS binarizes a pixel. FANS finishes the binarization of an image until all pixels are binarized. Finally, FANS outputs the binary image in Line 11. We analyze and compare the computational complexity of Otsu and FANS as shown in Table 1 and Table 2. For a grayscale image consisting of W × H pixels, Otsu needs W × H calculation steps to obtain the histogram and probability distribution. Then, Otsu calculates a global threshold form the histogram, and this procedure needs 256 steps. Finally, Otsu uses the threshold to binaries the image, which needs W × H steps. As a result, the total computational complexity of Otsu is 2WH + 256. As to FANS, the time complexity of building a SAT is W × H. FANS also needs W×H steps to binarize each pixel. It is worth noting that without SAT, the computational complexity of this step is W × H × b². Consequently, the total computational complexity of FANS is 2WH. From the viewpoint of big O analysis, Otsu and FANS have the same time complexity O(WH).

Figure 7. The pseudocode of FANS.

Table 1. The computational complexity of Otsu.

Procedure	Computation
Generate the histogram	WH
Threshold calculation	256
Image binarization	WH
Total	2WH + 256

Table 2. The computational complexity of FANS.

Procedure	Computation
Generate the SAT	WH
Threshold calculation and image binarization	WH
Total	2WH

3.3. License Plate Detection and Character Segmentation

FANS uses 4-connectivity to label connected components and remove noise and then uses an edge-based method that uses a mask to detect license plates. The mask locates the license plate by a fixed license plate ratio of height to width. If the number of black pixels within this mask is close to that of a real license plate, this area covered by the mask is regarded as a license plate. A standard license plate ratio of height to width is 2.375 in Taiwan as shown in Figure 8. After license plate detection, FANS uses the projection method to segment characters.

Figure 8. A standard license plate in Taiwan.

3.4. Character Recognition

License numbers consist of single characters, including letters and numbers. In the final step, FANS uses a template matching method to recognize the characters in a license plate. Template matching is a simple and straightforward method in character recognition. The recognition is achieved by comparing each extracted character with the templates in a pre-defined database. The template with highest matching value is regarded as the recognized character. Since the sizes of license plates in input images may be different, we need to normalize the character sizes before recognizing them. Templates size used by Türkyılmaz and Kaçan is 20 × 30 pixels [9]. However, the characters size in Taiwan’s license plates is 45 × 90 mm and consequently the ratio of height to width is 0.5. Therefore, FANS normalized templates to 15 × 30 pixels, which has a same ratio of height to width 0.5. Since the size of our experimental images is 4032 × 3024, which is too big for a LPRS, we normalized all images to 800 × 600. Both aspect ratios are 1.333. The license plates have twenty-six letters and ten numbers, so the minimum number of templates in a database is thirty-six. Our database has 854 templates. In order to avoid the problem of overfitting, the templates were created by 246 captured images that were different to those used in our experimental analysis.

4. Experimental Results and Discussions

4.1. Experimental Environment and Data

We implemented five LPRSs by Java. The five LPRSs are the same except for the image binarization method they adopted. In order to facilitate the following explanation, we use the term “Otsu” to represent the LPRS in the rest of this paper. The interfaces of the LPRSs of FANS and Otsu are shown in Figure 9, in which the gray buttons were used to observe the experimental results, such as gray level, detection rate, and recognition rate. All experiments were done on a laptop with Intel(R) Core(TM) i7-6700 3.50 GHz quad-core CPU, 8.0 GB RAM memory, and Windows 10 home version OS. All experimental images were taken by an iPhone 8 which has a built-in camera with 1.22 μm pixel size. The resolution of all captured experimental images is 4032 × 3024. The camera’s flash was set to “automatic.” With regard to the number of experimental images and the distance from the camera to the license plate, we followed the experimental setups in [8] and [9]. The distance from the camera to license plates ranged from 1.5 to 3 m and 340 images were captured from the real environment in Taiwan. In order to compare the detection rate and recognition rate in different illumination conditions, we collected 170 experimental images in the daytime and the other 170 images in the nighttime in Taiwan. Among the 340 experimental images, 44 images contained dirt and these images were used to make comparisons with recognizing dirty license plates.

Figure 9. (a) The interface of LPRS uses FANS, (b) the interface of LPRS uses Otsu.

Table 3 is the analysis of parameter C used in FANS in Equation (12). In Table 3, detection rate denotes the percentage of license plates that were correctly detected and recognition rate denotes the percentage of license plates in which all characters were correctly recognized. According to our experimental analysis, we found that when C was 4, FANS reached the best detection rate and recognition rate. From Table 3 we also found that the recognition rate and detection rate fell when C was larger 10. The reason is that when C became too large, the threshold ^*T_ij became smaller. Consequently, there were more pixels classified as to white, but these pixels possibly belong to black in this mask region. This binarization error reduced the recognition rate of a LPRS. Table 4 is the analysis of mask size and we found that when the mask size was 9 × 9, FANS reached the best performance. The reason is that a small mask would make the characters in a license plate too thin while a big mask would make the characters too thick, so a moderate mask size is more appropriate. According to Table 3 and Table 4, the default mask size of FANS was set to 9 × 9 and C was set to 4 in the following experiments.

Table 3. The analysis of parameter C used in FANS, where the row in bold denotes FANS reaches the best recognition rate.

Value C	# of Images	# of Correct Detection	# of correct Recognition	Detection Rate	Recognition Rate
1	340	335	275	98.52%	80.88%
2	340	340	289	100%	85.00%
3	340	340	288	100%	84.70%
4	340	340	297	100%	87.35%
5	340	340	285	100%	83.82%
6	340	340	295	100%	86.76%
7	340	337	281	99.11%	82.64%
8	340	340	286	100%	84.11%
9	340	340	285	100%	83.82%
10	340	339	283	99.70%	83.23%
20	340	337	271	99.11%	79.70%
30	340	335	261	98.52%	76.76%
40	340	325	248	95.58%	72.94%
50	340	318	246	93.52%	72.35%

Table 4. The analysis of mask size used in FANS, where the row in bold denotes FANS reaches the best recognition rate.

Mask Size	# of Images	# of Correct Detection	# of Correct Recognition	Detection Rate	Recognition Rate
3 × 3	340	317	220	93.23%	64.70%
5 × 5	340	306	219	90.00%	64.11%
7 × 7	340	336	271	98.82%	79.70%
9 × 9	340	340	297	100%	87.35%
11 × 11	340	340	272	100%	80%
13 × 13	340	338	283	99.41%	83.23%
15 × 15	340	340	278	100%	81.76%
17 × 17	340	340	269	100%	79.11%
19 × 19	340	340	273	100%	80.29%

4.2. Comparisons between FANS and Otsu in the Daytime

The comparisons between FANS and Otsu in the daytime are shown in Table 5. From Table 5 we found that FANS reached 100% detection rate and 92.94% recognition rate, and Otsu reached 89.41% detection rate and 73.52% recognition rate. Obviously, FANS is superior to Otsu when images were captured in the daytime. After analyzing our experimental data, we found some plates were detected correctly but recognized unsuccessfully. This problem is caused by the template matching method used in our two LPRSs. Some characters are too similar in our template database, such as 2 and Z, D and 0, G and C, and so on. This situation made both LPRSs recognize the characters unsuccessfully.

Table 5. The comparisons of detection rate and recognition rate between FANS and Otsu in the daytime, where the results marked with boldface depict the best results.

Method	# of Images	# of Correct Detection	# of Correct Recognition	Detection Rate	Recognition Rate
FANS	170	170	158	100%	92.94%
Otsu	170	152	125	89.41%	73.52%

Otsu performed worse than FANS because of the two main reasons. First, Otsu performed badly when images were taken in a high brightness environment, as shown in Figure 10a–c where the binary images were generated by Otsu and FANS respectively. In Figure 10a, because the sun was strong, the characters in the license plate were fuzzy, adhering, and fractured. Consequently Otsu generated a bad binary image as shown in Figure 10b, in which Characters “0” and “X” were fractured and therefore resulted in failed detection and recognition. Obviously, the binary image generated by FANS in Figure 10c did not have this problem. Another problem of Otsu is the noise interference as shown in Figure 11. In Figure 11a, the Character “9” is fractured since it is covered with dirt. Figure 11b,c shows the binary images generated by Otsu and FANS respectively. After binarization, the Character “9” in Figure 11b was very similar to Character “3.” This problem made the Otsu recognize plate characters unsuccessfully. On the contrary, the binary image generated by FANS did not have this problem as shown in Figure 11c.

Figure 10. (a) An image taken in a high brightness environment in the daytime; (b) the binary image generated by Otsu, where characters “0” and “X” were fractured; (c) the binary image generated by FANS.

Figure 11. (a) An image contains noise and was taken in the daytime; (b) the binary image generated by Otsu, where the Character “9” was very similar to Character “3”; (c) the binary image generated by FANS.

4.3. Comparisons between FANS and Otsu in the Nighttime

Table 6 shows the comparisons between FANS and Otsu in the nighttime. FANS reached 100% detection rate and 81.76% recognition rate, and Otsu reached 81.17% detection rate and 60.58% recognition rate. Obviously, FANS outperforms Otsu in the nighttime. However, since the light is darker and license plates are often covered by shadow in the nighttime, the detection rates and recognition rates of both FANS and Otsu became worse than those in the daytime.

Table 6. The comparisons of detection rate and recognition rate between FANS and Otsu at the nighttime, where the results marked with boldface depict the best results.

Method	# of Images	# of Correct Detection	# of Correct Recognition	Detection Rate	Recognition Rate
FANS	170	170	139	100%	81.76%
Otsu	170	138	103	81.17%	60.58%

There are many factors that interfere with the performance of a LPRS in the nighttime, and two main factors are shadow and light. Figure 12a is an image taken in the nighttime. Since the license plate in Figure 12a was covered by a shadow, the binary image generated by Otsu in Figure 12b was difficult to be detected and recognized. Figure 12c is the binary image generated by FANS. It is obvious that Figure 12c is much better than Figure 12b. Another factor that interferes with the performance of LPRS in the nighttime is light. For example, Figure 13a is an image captured with a car headlight. In the binary image generated by Otsu in Figure 13b, some parts of Characters “8,” “R,” and “H” disappeared. On the contrary, the binary image generated by FANS in Figure 13c was not affected by the car headlight. It is worth noting that in a real application, license plates are usually affected by the headlights of cars or motorcycles in the nighttime.

Figure 12. (a) A license plate that was covered by a shadow and taken in the nighttime; (b) the binary image generated by Otsu, where the license plate disappears; (c) the binary image generated by FANS.

Figure 13. (a) An image captured with a camera flash in the nighttime; (b) the binary image generated by Otsu, where some parts of Characters “8,” “R,” and “H” disappeared. (c) The binary image generated by FANS.

4.4. Comparisons between FANS and Otsu in Dealing with Dirty License Plates

The comparisons between FANS and Otsu in dealing with dirty license plates are shown in Table 7. FANS reached 100% detection rate and 93.18% recognition rate, and Otsu reached 70.45% detection rate and 50% recognition rate. Obviously, FANS performed better than Otsu in dealing with dirty plates. This result corresponds to our discussion in Section 1, i.e., Otsu has poor performance when there is dirt on license plates. After analyzing our experimental data, we also found that since some characters are covered with too much dirt, template matching method could not recognize them successfully even FANS reaches 100% detection rate.

Table 7. The comparisons of detection rate and recognition rate between FANS and Otsu in dealing with dirty license plates, where the results marked with boldface depict the best results.

Method	# of Images	# of Correct Detection	# of Correct Recognition	Detection Rate	Recognition Rate
FANS	44	44	41	100%	93.18%
Otsu	44	31	22	70.45%	50.00%

For license plates in Taiwan, this problem is very important since the climate in Taiwan is usually hot and humid and license plates in Taiwan are fixed by two or more screws. These screws gather rust easily and therefore become dirty hence degrading the recognition rate. Figure 14a shows an image contained rusty screws in Taiwan. As shown in Figure 14a, the two screws on the plate were rusty and their locations were very close to Characters “E” and “4.” In the binary image generated by Otsu as shown in Figure 14b, one rusty screw adhered to Character “E” and this situation made the LPRS difficult to detect and recognize the license plate. The binary image generated by FANS is shown in Figure 14c. Obviously, Character “E” and the rusty screw were separated, so the LPRS could successfully detect and recognize the license plate. Figure 15a is an image with dirty plate in which there was silt between Characters “M” and “Q.” The binary image generated by Otsu is shown in Figure 15b, in which Characters “M” and “Q” adhered to each other. Although Otsu can detect the license plate correctly, it could not successfully recognize all characters on this plate. On the other hand, the binary image generated by FANS had no problem. As shown in Figure 15c, the Characters “M” and “Q” were obviously separated. Consequently, FANS can successfully recognize all the characters on this plate.

Figure 14. (a) A license plate contained rusty screws, (b) the binary image generated by Otsu, where the rusty screw adhered to Character “E,” (c) the binary image generated by FANS.

Figure 15. (a) An image contained dirty plates, (b) the binary image generated by Otsu, where Characters “M” and “Q” adhered to each other. (c) The binary image generated by FANS.

4.5. Summary and Discussion

Please note, there are some new binarization methods proposed in recent years. However, as we discussed in Section 1, although these new methods improved the binarization results, they required a higher computational cost. When a LPRS is used in a real-time application, the computational cost is a very important factor. Suppose that we implement a LPRS to recognize the license plates captured by a 120 fps camera, if the recognition time is more than 0.01 s, the LPRS cannot recognize all images in real time since there are 120 images in a second. Consequently, Otsu is still one of the most commonly used binarization method in a LPRS, as reported in many recently published papers. The main purpose of this paper is to propose a binarization method that will not only improve the binarization results of license plates taken in the open environment in Taiwan, but also keeps the time complexity as simple as Otsu.

Table 8 shows of the summary comparisons between FANS and Otsu in dealing with all 340 images. We also included three newer binarization methods in Table 8. In Table 8, FANS reached 100% detection rate and 87.35% recognition rate, and Otsu reached 85.29% detection rate and 67.05% recognition rate. Although FANS does not reach the best recognition rate, it improved the detection rate up to 1.17 times better than Otsu and improved the recognition rate up to 1.30 times better than Otsu. The comparisons of average execution time of the five LPRSs is shown in Table 9. From Table 9 we found that FANS is the fastest.

Table 8. Summary comparisons of detection rate and recognition rate between five binarization methods, where the results marked with boldface depict the best results.

Method	# of Images	# of Correct Detection	# of Correct Recognition	Detection Rate	Recognition Rate
Otsu (1979) [21]	340	290	228	85.29%	67.05%
Halabi et al. (2009) [46]	340	340	292	100%	85.88%
Phansalkar et al. (2011) [47]	340	340	302	100%	88.88%
Saxena (2014) [48]	340	340	315	100%	92.65%
FANS (2020)	340	340	297	100%	87.35%

Table 9. Comparisons of average execution time between five binarization methods, where the results marked with boldface depict the best results.

Method	Average Execution Time (s)
Otsu (1979) [21]	0.7066
Halabi et al. (2009) [46]	2.2984
Phansalkar et al. (2011) [47]	1.6467
Saxena (2014) [48]	3.1550
FANS (2020)	0.6817

5. Conclusions

License plate recognition is a positive research area in the past few years and is used widely in our daily life. In this paper, we propose a LPRS named FANS for the automatic license plate recognition in the open environment in Taiwan. Experimental results in Section 4 show that FANS reached a better performance than Otsu in the daytime, in the nighttime, and in dealing with noise. FANS improved the detection rate up to 1.17 times better than Otsu and improved the recognition rate up to 1.30 times better than Otsu. With regard to the efficiency, in a real application, the FANS needs less execution time than that of Otsu.