Intelligent System for Vehicles Number Plate Detection and Recognition Using Convolutional Neural Networks

Abstract

Vehicles on the road are rising in extensive numbers, particularly in proportion to the industrial revolution and growing economy. The significant use of vehicles has increased the probability of traffic rules violation, causing unexpected accidents, and triggering traffic crimes. In order to overcome these problems, an intelligent traffic monitoring system is required. The intelligent system can play a vital role in traffic control through the number plate detection of the vehicles. In this research work, a system is developed for detecting and recognizing of vehicle number plates using a convolutional neural network (CNN), a deep learning technique. This system comprises of two parts: number plate detection and number plate recognition. In the detection part, a vehicle’s image is captured through a digital camera. Then the system segments the number plate region from the image frame. After extracting the number plate region, a super resolution method is applied to convert the low-resolution image into a high-resolution image. The super resolution technique is used with the convolutional layer of CNN to reconstruct the pixel quality of the input image. Each character of the number plate is segmented using a bounding box method. In the recognition part, features are extracted and classified using the CNN technique. The novelty of this research is the development of an intelligent system employing CNN to recognize number plates, which have less resolution, and are written in the Bengali language.

1. Introduction

Vehicle Number Plate Recognition (VNPR) is an exoteric and effective research modality in the field of computer vision [1]. As there are an increasing number of vehicles on the road, it is highly challenging to monitor and control the vehicles using existing systems (such as manual monitoring and monitoring by traffic police). An intelligent system can be used to overcome this problem in a convenient and efficient way. Real time detection of number plates from moving vehicles is needed, not only for monitoring traffic systems, but also for traffic law enforcement. However, development in this area is slow and very challenging to implement from a practical point of view [2].

Recognizing vehicle number plates can help with authorization (for example, when a vehicle enters into an impervious premise). VNPR can develop a security policy while the issues are more crucial. This research work aims to detect and recognize number plates in an intelligent way. The tests were carried out on vehicles in Dhaka city, in Bangladesh, although the work can be extended to any country. Dhaka is a densely populated city with huge amounts of traffic—and people frequently break the traffic rules. In Bangladesh, Bangladesh Road Transport Authority (BRTA) has the authority to register vehicles. According to the annual report of BRTA [3], the number of vehicles are increasing rapidly every year in Bangladesh (Figure 1). This could be accounted for the increasing road traffic accidents and related deaths. Bengali alphabet and Bengali numerals are used in Bangladeshi vehicle number plates. The international vehicle registration code for Bangladesh is BD. The two types of vehicles are used in Bangladesh are civil vehicles and army vehicles. In Figure 2, the sample of Bangladeshi vehicle number plates is shown.

Authorized letters and numeric numbers for the vehicle number plates in Bangladesh with the corresponding English equivalent figures are presented in

Table 1. Bangla letters and their corresponding English equivalents used in the vehicle number plates.

Bangla letter

অ

ই

উ

এ

ক

খ

গ

ঘ

ঙ

চ

ছ

জ

ঝ

ত

থ

ড

English letter

a

i

u

e

ka

kha

ga

gha

na

ca

cha

ja

jha

ta

tha

da

Bangla letter

ঢ

ট

ঠ

দ

ধ

ন

প

ফ

ব

ভ

ম

য

র

ল

শ

স

English letter

dha

ta

tha

da

dha

na

pa

pha

ba

bha

ma

ya

ra

la

sha

sa

Bangla letter

০

১

২

৩

৪

৫

৬

৭

৮

৯

-

English number

0

1

2

3

4

5

6

7

8

9

-

.

The format of vehicle number plates in Bangladesh is “city name—class letter of a vehicle and its number—vehicle number”. For example, “DHAKA METREO—GA 0568”. Here, “Dhaka” represents city name, “GA: represents vehicle class in Bangla alphabets. The second line (number line) contains six digits, where the first two digits (15) denote vehicle class number and the last four digits (0568) represent the vehicle registration number in Bangla numeral. Figure 3 shows the representation of the above in the number plate.

This research work developed an intelligent system that is efficient to recognize vehicle number plates using Convolutional Neural Networks (CNN). The recognition system consists of five major steps: image pre-processing, detection of the number plate from the captured image, learning based super resolution technique to produce a high-resolution image, segmentation, and recognition of each character. Segmentation is the most important task as it provides the result of the entire number plate analysis. The main objective of segmentation is to determine each region according to vehicle city, type, and number. However, segmentation of blurred number plates was more challenging, and this was overcome by using the super resolution method that transformed the blurred number plate image into a clear image.

To get a perfect segmentation result, the bounding box method was used. Then the system used CNN for extracting features of the number plate and recognizing the vehicle number.

The main steps of this system are organized as follows:

• Localization of the number plate region: template matching algorithm is used for extracting the number plate region from the input image frame of the vehicle.
• Super resolution and segmentation techniques: the super resolution technique is used to get a clear number plate with good resolution and the bounding box method is used for segmenting each character of the number plate. The method segments the vehicle city, type, and number from the plate region.
• Feature extraction: the system used 700 number plate images for training by using CNN, and it provided 4096 features for each character to recognize correctly. The number plate images used in this investigation was collected from Bangladesh Road Transport Authority (https://service.brta.gov.bd/).

The paper is organized as follows: Section 2 describes the related work; Section 3 analyzes the methodology; Section 4 represents the simulation of this work; finally, the conclusion is drawn in Section 5.

2. Related Works

In the literature, a large number of systems were proposed and applied for vehicle number plate recognition: digital image enhancement, detection of the number plate area from the captured image, segmentation of each character, and recognition of the character form the core steps in the recognition systems.

Cheokman et al. [4] showed morphological operators to pre-process the image. After preprocessing, the template matching approach was used for recognition of each character. It was issued for the vehicle registration plate (Macao-style). In [5], scaling and cross-validation was applied for removing outliers and finding the clear parameters, using the Support Vector Machine (SVM) method. Recognizing characters via the SVM method, the rate of accuracy was higher from the Neural Network (NN) system.

Prabhakar et al. [6] proposed a webcam for capturing images. This system can localize several sizes of number plates from the captured images. After localizing the plate, characters are segmented and recognized using several NNs.

Sobel color detector for detecting vertical edges was used in [7], where the ineffective edge was removed. The plate region was discovered by using the template matching approach. Mathematical morphology and connected component analysis were used for segmentation. Chirag Patel [8] proposed mathematical morphology and connected component analysis for segmentation and recognition of characters using the radial basis function of the neural network.

The number plate detection system in [9] used plate background and character color to find the plate location. For segmentation, the column sum vector was adopted. Artificial Neural Network (ANN) was used for character recognition.

The system in [10] is used for Chinese number plate recognition. The number plate image converts into a binary image, and noises of the image are removed. Then, the feature is extracted from the image and the image is normalized in an 8 × 16 pixel. After normalization, the back-propagation neuronal network is used for recognition.

Ziya et al. proposed Fuzzy geometry to locate the number plate, and segmented the plate by using Fuzzy C-Means [11]. The segmentation technique, by using blob labeling and clustering, provides a segmentation accuracy of 94.24% [12]. In [13], Gabor filter, threshold, and connected component labeling were used for finding the number plate. A self-organizing map (SOM) neural network wass used for character recognition after segmentation [13]. In [14], a two-layer Marko network was used for segmentation and character recognition. Similar works on number plate detection are published in [15,16].

Maulidia et al. [17] presented a method where the accuracy of Otsu and K-nearest neighbor (KNN) were obtained for converting an RGB image into a binary image, extracting characteristics of the image. Feature extraction in pattern recognition was used for converting pixels into binary form. Feature extraction was performed by the Otsu method where KNN classified the image by comparing the neighborhood test data to the training data. Test data were determined by using the learning algorithm through a classification process, which groups the test data into classes. The Otsu method was developed based on a pattern recognition process with a binary vector without influencing the threshold value. Adjustment of distribution of the pixel values of the image was performed to obtain binary segmentation. KNN classification proved to be a great boon in recognizing the vehicle number plate. However, the authors did not provide the recognition capability of the system under extreme weather conditions.

Liu et al. [18] presented a supervised K-means machine learning algorithm to segregate the characters of the number plate into subgroups, which were classified further by the Support Vector Machine (SVM). Their system recognized blurred number plate images and improved the classification accuracy. This system differentiated the obstacles in character recognition due to the angle of the camera, speed of the vehicle, and surrounding light and shadow. The camera captured faint and unrecognizable character images. A huge number of samples increased the workload of SVM classifiers; thus, affecting the accuracy.

Quiros et al. [19] used the KNN algorithm for classifying characters from number plates. An image processing camera was installed on a highway in their proposed system and analyzed the feed received, capturing the images of vehicles. Contours within the number plates were computed as if they were valid characters, along with their sizes, and afterwards, the plates were segmented from the detected contours. Each contour was classified using the KNN algorithm, which was trained using different sets of data, containing 36 characters, comprised of 26 alphabets and 10 numerical digits. The algorithm was tested on previously segmented characters and compared with the character recognition technique, such as artificial neural network. Their proposed system did not provide the character recognition performance compared to the literature.

Thangallapally et al. [20] implemented a technique to recognize the characters on number plates and to upload details into a server. This, in turn, was segregated to extract the image of the vehicle number plate. The process led to compartmentalizing the characters from the number plate, where KNN was applied to extract the characters uploaded in the server. The hindrance of this process was recognizing the number plates from blurred or ambiguous images.

Singh and Roy [21] proposed a vehicle number plate recognition system in India, where various issues were observed, including a plethora of font sizes, different colors, double line number plates, etc. Artificial neural network (ANN) and SVM were employed to recognize characters, and to detect plate contours, respectively. Although the number of algorithms were employed in literature to remove noise, and to enhance plate recognition, ANN showed good results with easing camera constraints.

Sanchez [22] implemented a recognition system of vehicle number plates in the UK using machine learning algorithms, including SVM, ANN, and KNN. The system received the car image, processed and analyzed with the Machine Learning (ML) algorithm, and computer vision techniques. The results from the investigation showed that the system could identify the number plate of the car from the images.

Panahi and Gholampour [23] proposed a system to detect unclear number plates during rough weather and high-speed vehicles in different traffic situations. The image data of the vehicle number plates were collected from different roads, streets, and highways during the day and night. The proposed system was robustly receptive to variations in light, size, and clarity of the number plates. The aforementioned techniques helped in compiling a dedicated set of solutions to problems and challenges involved in the formation of a number plate recognition system in various intelligent transportation system applications.

Subhadhira et al. [24] used the deep learning method, which was used for training processes, to classify vehicle number plates accurately. This system consisted of two parts: (1) pre-processed and extracted features using the histogram of oriented gradients (HOG) and (2) the second part classified each number and alphabetical character that appeared on the number plate to be analyzed and segregated. The extreme learning machine (ELM) was used as a classifier, whereas HOG extracted important features from the plate to recognize Thai characters on the number plate. The ELM system performed better due to its high speed and acceptable testing and training tenets.

In previous works, different techniques, such as template matching, and several classifiers, SVM, ANN, were used to recognize characters. The ambiguous characters were not dealt with concerning template matching techniques. For illumination ambiences, orientation of each character of the damaged number plates, SVM could not be supported. Considering these issues, in this research, the number plate region is extracted from the captured vehicle image by using a template matching method and the super resolution technique is applied to improve the resolution quality. Then, segmentation of characters from the number plate region, and feature extraction to recognize the characters, are performed using CNN. CNN uses the gradient-based learning algorithm with modernized activation functions, including Rectified Linear Unit (ReLU), which can deal with diminishing gradient problem [25]. Gradient descent-based method performs training and creates models to minimize the errors and update the weights accordingly. Thus, better prediction accuracy is obtained through producing highly optimized weights during training. CNN is capable of representing two-dimensional (2D) or 3D images with meaningful features, which can help achieve superior recognition performance. In particular, the max-pooling layer of CNN can deal with shape, as well as scale invariant image problems. In addition, the algorithm uses a considerably low number of network parameters compared to traditional neural networks with similar sizes.

3. Proposed Methodology

This proposed system has the capability of detecting and recognizing vehicle number plates in any language. To detect each character of a number plate, this system trains all alphabetic letters from the plate using machine learning. Number plate characters for the majority countries across the world are from A to Z and 0 to 9, so that it can easily be detected, whereas Bangla number plate detection and recognition is very challenging, due to the complex alphanumeric characters. Therefore, Bangla number plates have been considered as a case study in this research.

The basic steps of the proposed methodology are (a) pre-processing; (b) localization of the number plate region; (c) super resolution techniques to get clear images; (d) segmentation of characters; (e) feature extraction; and (f) recognition of characters, shown in Figure 4.

3.1. Pre-Processing

In this system, there are two parts: one part is for number plate detection of moving vehicles and another part is for number plate recognition. The first part extracts the number plate region from the captured image of the vehicle by using the template matching technique. The number plate recognition part consists of three activities. (1) The super resolution method is used in the number plate region for converting a low-resolution image to a high-resolution image. Then it converts RGB into a gray image. (2) The bounding box method is used for segmenting characters. (3) Finally, features from the authorized alphabets and numbers are extracted using CNN. The CNN model provides 4096 features for recognizing each character.

3.2. Localization of the Number Plate Region

The template matching technique was applied to recognize the plate region from the vehicle image, which is identical to the template from the target image. In this method, the target image is governed by the template and calculates the measures of similarity.

The localization process traverses the template image to each position in the vehicle image and computes the numeral indexes that ensure the template matches the image pixel-by-pixel in that portion. Finally, the strongest similarities are identified as efficient pattern positions. Figure 5 illustrates the template matching procedure. The various portions of the input image are matched to the predefined template image and the naive template matching approach is performed in order to extract the template. Then, the extracted plate region is resized to 127 × 127 pixels. An example of the extracted plate region from the vehicle image frame using the template matching technique is shown in Figure 6.

3.3. Super Resolution Technique

The imaging chips and optical components are highly expensive and, practically, not used in surveillance cameras. The quality of the surveillance camera and the configuration of the hardware components limit the captured image resolution. It cannot detect character of the number plate from the vagueness plate region. In this research, a spatial super resolution (SR) technique is used to overcome the limitation successfully.

The SR techniques construct the high-resolution images from various accomplished low-resolution images. The concept of SR is to organize the non-redundant data contained in abundant low-resolution frames to produce a high-resolution image. The main objective of super resolution is to acquire the high-resolution image from the multiscale low-resolution image through the spatial resolution approach. The spatial super resolution is applied to get a high-resolution number plate image. First, the system used the downsampling approach and converted the image samples, such as local gradient, alignment vector, and local statistics. Then, the kernel formed the downsampling image and constructed the high-resolution RGB image. The high-resolution image was converted from RGB to a gray-scale image (I), according to Equation (1).

I = Wr * R + Wg * G + Wb * B

(1)

The R, G, and B are values of monochrome colors (red, green, and blue, respectively) of the RGB color image, which are linear in luminance. Wr, Wg, and Wb are the coefficient (fixed weight) of red, green, and black colors, with a value of 0.299, 0.587, and 0.114. Summation of all three weights (R, G, and B) is equal to 1.

Figure 7 describes the process to reconstruct images using the super resolution (SR) technique. SR techniques are a good choice to get clear images to segment from blurred images. There are various super resolution techniques. Among these, better results are achieved for the spatial super resolution technique. This work calculates peak signal-to-noise ratio (PSNR) value for different super resolution techniques, but spatial super resolution technique archives better PSNR value. Multiple low-resolution frames are downsampled, shifting subpixels from the high-resolution scene between one another. The construction of SR aligns with the low-resolution observances and combines subpixels into high-resolution image grids to overcome the limitation of a camera’s image processing. The proposed super resolution is summarized in Algorithm 1. This algorithm combines many low-resolution images from the original image [26]. Then, a kernel is added to the combined image to get a clearer image.

Algorithm 1: Super Resolution of Plate Region

Super_resolution $\hspace{1em}\hspace{1em}\hspace{1em}\left(\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}:\mathit{O}\mathit{r}\mathit{i}\mathit{g}\mathit{i}\mathit{n}\mathit{a}\mathit{l} \mathit{L}\mathit{o}\mathit{w} \mathit{R}\mathit{e}\mathit{s}\mathit{o}\mathit{l}\mathit{u}\mathit{t}\mathit{i}\mathit{o}\mathit{n} \mathit{F}\mathit{r}\mathit{a}\mathit{m}\mathit{e}\mathit{s} \left[{\mathit{O}}_{1, \dots \dots \dots ,}{\mathit{O}}_{\mathit{k}}\right];\mathit{o}\mathit{u}\mathit{t}\mathit{p}\mathit{u}\mathit{t}:\mathit{S}{\mathit{R}}_{\mathit{F}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}\right)$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathit{H}=\mathit{B}\mathit{i}\mathit{c}\mathit{u}\mathit{b}\mathit{i}{\mathit{c}}_{\mathit{I}\mathit{n}\mathit{t}\mathit{e}\mathit{r}\mathit{p}\mathit{o}\mathit{l}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}\left(\mathit{A}\mathit{v}\mathit{r}\left({\mathit{O}}_{1,\dots \dots \dots ,}{\mathit{O}}_{\mathit{k}}\right)\right)}$ For i = 1 to N    For j = 1 to K       SumFrame = SumFrame + Fk−1(Fk (H, Ok) − Ok);    End loop;    H = H – SumFrame;    SR_Frame = H; Return SR_Frame;

In Algorithm 1, an enlarged hypothesis frame, H, was first created from several sequential low-resolution frames. The initial hypothesis frame can be either a simple bicubic interpolation of one of the frames, or a bicubic interpolation of the average of all of the low-resolution frames. Afterwards, this hypothesis frame was iteratively altered and adjusted with the information from all of the low-resolution frames. Algorithm 1 runs for N iterations specified by the user. The hypothesis frame, H, was adjusted in each of the iterations of the algorithm. $F_n$ was a function used to align the hypothesis frame with the original low-resolution frames. These were then subtracted with each of the original, individual low-resolution frames, $O_n$. $F_n^{ -1}$ was a function that reversed the alignment and enlarged the error frame. These were summed over the number, n, of low-resolution frames, so that it could be used to adjust the hypothesis frame, H.

This algorithm shows that H is a hypothesis frame that is first created from different pursuant low-resolution frames. The primary hypothesis frame can be either a light bicubic interpolation of one of the frames or a bicubic interpolation of the average of all low-resolution frames. This hypothesis frame is altered and adjusted with the information from all of the low-resolution frames. The images, after applying the super resolution algorithm in the plate region shown in Figure 8, clearly demonstrate that image quality was significantly improved with the correct balance of brightness and contrast.

3.4. Segmentation of Character

The character segmentation process will partition the number plate image into multiple sub-images; each sub-image offers one character. In this work, segmentation is the most significant part, as the successful recognition of each character relies on accurate segmentation. If segmentation is not performed correctly, then recognition will not be accurate.

The bounding box method is used to segment the exact region of each character. This method is efficient at identifying the boundaries of each character. It surrounds the labeled region with a rectangular box, as shown in Figure 9. Then, it determines the upper left and lower left corner of the rectangle by the x and y coordinates, and it provides a label of each character. The bounding box method is described in Equation (2) [27] and Algorithm 2. This algorithm is a bounding box algorithm to describe the target location. The bounding box is a rectangular box that can be determined by the x- and y-axis coordinates in the upper-left corner, and the x- and y-axis coordinates in the lower-right corner of the rectangle.

$$E\left(x\right)={\displaystyle \sum }_{p\in \beta }{U}^p\times x_p+{\displaystyle \sum }_{\left\{p,q\right\}ϵ\epsilon }{V}^{pq}\times \left|x_p-x_q\right|,x_pϵ\left\{0,1\right\}$$

(2)

where β is an image with a set of pixels p ∈ β. x_p as an individual pixel label assumes values of 1 and 0 for foreground and background, respectively. Pairs of adjacent pixels, unary potentials, and pairwise potentials are defined by ε, U^p, and V^pq.

Algorithm 2: Bounding Box Method of Plate Region

For $i=1 :N \left(each video segment\right)$ do For $t=1 : L_i \left(each frame\right)$ do   Detect character as foreground object with a target bounding box;    Form the target image region $R_i$ defined by the target bounding box; Normalization Region of Interest (ROI) with preservation of aspect ratio;  Compute shaper description $p_i\in ℳ$ from normalized ROI, where $ℳ$ is the underlying Riemannian manifold;  End  Collect the set of manifold points ${\left\{p_i\right\}}_{t=1}^{L_i};$   Compute final feature vector $x_i$  End  Construct training set $\mathcal{X}={\left(\left\{x_i,y_i\right\}\right)}_{i=1}^N;$  Train CNN classifier using $\mathcal{X}$ with cross-validation

The algorithm selects a frame and detects characters as a foreground object with a target bounding box. Then, the target image region defined by the target bounding box and normalization ROI (preserving the aspect ratio) are formed. Afterwards, feature vector was extracted for the target image. The training data were matched with the features of the target image to segment the image. This process was continued until all of the sections were segmented.

The features were extracted according to three classes for the segmented number plates. The first class consisted of 64 categories (64 districts of Bangladesh). In the second class, there were 19 categories according to the vehicle type. The last class consisted of 10 categories (0–9), which identified the vehicle number. A deep learning technique, CNN, was employed to train the classes of the vehicle city, type number using 700 number plate images of different angles, and resolutions to introduce a high-resolution trainable image. The CNN extracted features from different categories and stored the features in three separate vectors [28]. Figure 10 shows the proposed CNN AlexNet architecture. The AlexNet model consisted of five convolutional layers, three max-pooling layers, two fully connected layers, and one Softmax layer. Each convolutional layer took on convolutional filters and a nonlinear activation function ReLU. The pooling layers were used to perform max pooling. The first convolution layer contained 96 filters with a filter size of 11 × 11 and a Stride of 4. On the other hand, Layer 2 had a size of 55 × 55 and the number of filters was 256. Layers 3 and 4 contained filter sizes of 13 × 13, with 384 filters. The last convolutional layer contained a filter size of 13 × 13, with 256 filters. The two fully connected layers provided 1 × 4096 features to classify the output by the Softmax layer.

After, segmentation of the number plate images for vehicle city, type, and number were found. The CNN had extracted features of these images to recognize vehicle city, type, and number, separately, by transforming image text to characters. The 224 × 224 × 1 image size was used for feature extraction. The system used AlexNet as the CNN model. Figure 11 shows the recognition process of each character.

Table 2. Detection, segmentation, and recognition techniques of the applied method and existing methods.

Reference	Detection	Segmentation	Recognition
Proposed system	For extracting plate region, bounding box method was used Super Resolution techniques were used to get clear images	Template matching	CNN
[29]	Sobel edge detection with additional morphological operations	Line segmentation, word segmentation based on area filtering	Feed forward neural network
[30]	Connected component technique	Template matching
[31]	Sobel edge detector	Line segment orientation (LSO) algorithm	Template matching
[25]	CNN	CNN	CNN

presents a comparison of detection, segmentation, and recognition techniques used in this investigation and in other relevant literature. The key novelty in the proposed system compared to the literature was the capability of producing high-resolution images from blur images, by using the super resolution method to recognize the number plate characters with high accuracy. Furthermore, the employment of machine learning based on the AlexNet model can identify text from images correctly, to obtain a recognition rate higher than the other techniques.

4. Simulation Results and Discussions

An experiment was carried out on MATLAB R2018a simulator (Image Processing Toolbox™) for recognition of vehicle number plates. This detection and recognition process consisted of the following steps:

• Step 1: the system captured a video of the vehicle. Then, the system extracted the vehicle frame from the video and localized the number plate from the vehicle image. The number plate images were converted to high-resolution images to perform accurate segmentation. For extracting the number plate, the template matching method was used.
• Step 2: for segmentation, the system used the bounding box method to segment each character. Each letter or word was mapped with a box value and extracted groups of characters. Figure 12 illustrates the segmented characters from the vehicle number plate images.
• Step 3: the system used CNN for extracting features and tested number plates on the VLPR vehicle dataset. In order to evaluate the experiment results, 700 vehicle images were appointed. The AlexNet model was employed for training the CNN. The system accomplished a maximum of 70 iterations for each input set. The iterations were confined when the minimum error rate was clarified by the user. The error rate for this system was 1.8%. After training, the CNN acquired 98.2% accuracy based on the validation set, and attained 98.1% accuracy based on the testing set.

The error rate was calculated as E_i of an individual program; I depends on the number of samples incorrectly classified (false positives plus false negatives), and is evaluated by Equation (3):

$$E_i=\frac{f}{n}\times 100$$

(3)

where f is the number of sample cases incorrectly classified, and n is the total number of sample cases.

Table 3. Comparative accuracy of the applied system and the existing systems for image processing.

Reference	Sample Size	Localization	Accuracy	Processing Time
Proposed system	Training: 500 Testing: 200	100%	98.2%	111 milliseconds for the whole process
[29]	Testing: 300	84%	80%	-
[30]	Testing: 120	-	-	1.3 s
[31]	Testing: 119	95.8%	84.87%	-
[25]	Training: 450Testing: 50	88.67%	-	-

shows comparative accuracies and computational processing time between the proposed system and the systems employed in relevant literature. In [30], the authors estimated that 1.3 s was taken to complete the image processing, whereas the proposed system in this experiment took, on average, only 111 milliseconds to complete the whole process. A Graphics Processing Unit (GEFORCE RTX 2070 super) with 16 GB RAM was used in this system to perform quick computation. With a comparable sample size for training and testing, the accuracy of the proposed system was found much higher than the others available in the literature.

A comparison of prediction accuracies for detecting number plates between different CNN models, such as the scratch model [32], ResNet50 [33], VGG 16 [34], and the model employed in this work (AlexNet) [35] are presented in Figure 13. Total of number of true positives (TP) and true negatives (TN) were divided by the total number of TP, TN, false positives (FP) and false negatives (FN) to measure the accuracy (Equation (4)).

$$\mathrm{Accuracy}\left(\mathrm{ACC}\right)=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(4)

It can be concluded that AlexNet outperforms (98.2%) the other models for image processing. Figure 14 shows the performance comparison in terms of peak signal to noise ratio (PSNR) using different super resolution techniques, such as interpolation-based SR [36] and reconstruction-based SR [37]. The spatial super resolution technique used in this study showed high PSNR (33.7845 dB) compared to other related works, such as interpolation-based (32.3676 dB) and reconstruction-based (32.4787 dB) super resolution techniques, indicating a better quality of the compressed or reconstructed images.

In addition, 700 number plates collected in different backgrounds and different illumination conditions were used to test the number plate detection. Among them, 605 number plates were correctly identified equating to a recognition rate of 86.50%. A total of 133 valid characters were checked and 121 of them were correctly identified, which showed that the character recognition rate was significantly high (90.9%). The average recognition time for a single number plate was calculated as 707 ms from a sample of 700 number plates. The recognition rates and times for characters are summarized in

Table 4. Identification performance of each character.

Object	Parameter	Value
Letters	Recognition Rate (%) Recognition Times (ms)	86.5 48.6
Numbers	Recognition Rate (%) Recognition Times (ms)	97.8 48.9
Characters (Letters & Numbers)	Recognition Rate (%) Recognition Times (ms)	90.9 52.3

.

A system presented in [38] might have failed if the texture of the plate was not clear and the number plate region was short of the threshold of the projection operation. To solve this problem, this proposed system brought novelty in employing the super resolution technique to organize the non-redundant data contained in abundant low-resolution frames, and to produce a high-resolution image. The technique converted blur images to clear images to recognize the texts correctly. A vehicle number plate detection system presented in [39] was unable to properly recognize some characters and numbers, such as 2, 0, 7, and more, due to having a lower recognition rate compared to the other characters. In this system, the machine learning based CNN model was employed to recognize all characters and numbers correctly. Thus, the recognition rate was higher than the other techniques reported in the literature. Capturing images by directly pointing the camera towards the number plates was not feasible for real-world usage due to complex background and environment [40,41]. This system employed a novel template matching technique to detect vehicle number plate images from the video in order to extract the target region more accurately.

Furthermore, a similar system was proposed for Bangla number plate detection in [42], which showed a limitation of correctly recognizing the characters from large blur images. However, the proposed system could solve this issue by employing the super resolution technique to get high-resolution images for easy detection. The previous system recognized only Bangla number plates in contrast to this system having the capability of recognizing the vehicle number plates from other countries. Furthermore, the previous system used only 200 number plate images in comparison to 700 number plate images containing very high and very low resolutions for training the proposed system. Therefore, low-resolution images during testing were detected and recognized accurately by the system.

Vehicle number plate recognition system (VNPRS) can play a vital role in implementing technologies for smart cities, such as traffic control, smart parking, toll automation, driverless car, air quality monitoring, security, etc. [43,44]. A conceptual framework for integrating VNPRS with the smart city systems is presented in Figure 15. The key theme in the smart city is about collecting data from the individual systems by sensors and cameras, communicating within different systems, and taking action from the hidden information within the data. Automatic vehicle number identification could provide a base data to the network of the smart city. For example, in case of monitoring and managing security for a particular location within a city, VNPRS can serve as the tracking aid for the security authority. The VNPRS can be connected to a cloud-based system where all registered vehicle numbers will be stored. A vehicle number plate recognized by the system will be directed to the cloud system for matching with the database, and identifying the vehicle user information, for taking further action by the relevant authority.

5. Conclusions

In this research, a system is proposed for detecting and recognizing vehicle number plates in Bangladesh, which are written in the Bengali language. In this system, the images of the vehicles are captured and then the number plate regions are extracted using the template matching method. Then, the segmentation of each character is performed. Finally, a convolutional neural networks (CNN) is used for extracting features of each character that classifies the vehicle city, type, and number, to recognize the characters of the number plate. The CNN provides a large number of features to help with accurate recognition of characters from the number plate. This research used super resolution techniques to recognize characters with high resolution. In order to evaluate the experiment results, 700 vehicle images (with 70 iterations of each input set) were appointed. After training, the CNN acquired 98.2% accuracy based on the validation set, and attained 98.1% accuracy based on the testing set. This system can also be used for the number plates written in other languages in the same way.