License Plate Recognition Under the Dual Challenges of Sand and Light: Dataset Construction and Model Optimization

Abstract

License plate recognition in sandstorm conditions faces challenges such as image blurriness, reduced contrast, and partial information loss, which result in significant limitations in the feature extraction and recognition accuracy of existing methods. To address these challenges, this study proposes a license plate recognition method based on an improved AlexNetBN network. By introducing Batch Normalization (BN) layers, the model achieves greater training stability and generalization in complex environments. A dedicated dataset tailored for license plate recognition in sandstorm conditions was constructed, and data augmentation techniques were used to simulate real-world scenarios for model training and testing. Experimental results demonstrate that, compared to the traditional AlexNet model, AlexNetBN achieves higher recognition accuracy and robustness in environments with frequent sandstorms and significant variations in lighting intensity. This study not only effectively enhances license plate recognition performance under sandstorm conditions but also offers new insights and references for applying CNN-based methods in low-visibility scenarios.

1. Introduction

In recent years, target recognition under adverse weather conditions has gained widespread attention due to its practical applications in autonomous driving [1], intelligent surveillance, and traffic management [2]. Meanwhile, dust storms, as a common and challenging environment, exist in a variety of classifications [3]; except during strong and very strong dust storms, people still carry out their normal life activities including but not limited to the use of vehicles in the dust storm weather; therefore, license plate recognition and traffic management under dust storm weather are still greatly affected. Airborne dust particles cause severe image degradation, resulting in blurring, reduced contrast, and partial occlusion of targets. These effects make it increasingly difficult for traditional recognition methods to achieve accurate performance. Developing a robust recognition system that can function effectively under such conditions is critical for ensuring public safety and improving traffic efficiency.

Currently, various target recognition algorithms have been developed for object detection and classification in standard environments. Traditional methods rely heavily on manual feature extraction techniques such as SIFT [4] and HOG [5]. While these methods perform adequately under normal conditions, their ability to describe features degrades significantly when faced with blurred or low-contrast images. Moreover, sliding window-based object localization is computationally expensive and struggles to meet real-time requirements.

With the advent of deep learning and the fact that convolutional neural networks (CNNs) are characterized by local receptive fields [6] and weight sharing [7], CNNs have become a core technology for modern target recognition. CNN-based methods, especially the R-CNN family [8,9] (R-CNN, Fast R-CNN, Faster R-CNN, and Libra R-CNN), have made significant advances in detection efficiency and accuracy through the use of Regional Proponent Networks (RPNs) [10]. However, these methods still face some challenges when applied to dust storm weather conditions. The R-CNN family has limited feature extraction capability for low-quality images (e.g., blurred and low-contrast images), which makes it difficult to deal with the case of unclear object contours. First, the presence of sandstorms can lead to image blurring, uneven lighting, and contrast reduction, which can interfere with features extracted from the convolutional layer, resulting in an unstable model training process and slow or even failed convergence. Secondly, in such complex environments, the traditional pooling operation of convolutional neural networks may not be able to adequately preserve the key features, thus leading to a decrease in recognition accuracy.

The YOLO (You Only Look Once) family of algorithms has made significant progress in real-time object detection by combining object localization and classification into a single-stage detection network, resulting in extremely high detection speeds. However, although the fast detection mechanism of YOLO improves the speed, lower versions of YOLO (e.g., YOLO V1 [11], YOLO V2 [12], YOLO V3 [13], YOLO V4 [14], and YOLOv7 [15]) still face similar challenges as the R-CNN family in applications in sandstorm environments: image blurring and low contrast co-lead to degraded detection accuracy, especially when dealing with partially occluded or missing object information. While they are very robust, they struggle to capture detailed features under such conditions. While the advanced YOLO version (YOLOv8 [16]) effectively addresses some of the issues present in the lower versions of YOLO, the multi-scale detection used in YOLOv8 requires feature extraction and inference at multiple scales, which increases the computational cost and the post-processing complexity, and may lead to slower inference, especially when dealing with high-resolution images.

DSSD [17] (Deconvolutional Single Shot Detector) improves the detection of multi-scale objects by introducing deconvolution layers, which enhance detection performance on low-resolution targets. However, DSSD still faces performance instability in adverse weather conditions, especially in sandstorm environments, where image degradation is significant. Its ability to describe blurred features remains limited, leading to suboptimal detection results.

Currently, there are many reliable results on license plate detection in harsh environments, such as the techniques studied by Rio-Alvarez, A. et al. [18], which were tested based on descriptors and classifiers with different textures and are highly adaptable to machine learning-based algorithms, and Azam, S. et al. [19], who worked on a new ALPD method that can efficiently detect license plate regions from complex environments. Therefore, in this study, we will focus on the work of license plate number recognition and classification after the completion of license plate detection segmentation, and the optimization of license plate detection techniques under sandstorm conditions will be studied in detail in future work. At the same time, taking into account the different advantages and disadvantages of the networks described above, in this research, we adopt the AlexNetBN network to address the performance bottlenecks of existing methods in sandstorm weather. AlexNetBN, an enhanced version of the classic AlexNet [20], introduces Batch Normalization (BN) layers to improve model training stability and generalization [21], particularly when handling degraded images. Compared to other methods, AlexNetBN excels at feature extraction in challenging conditions, such as blurred and low-contrast images. By optimizing the network structure to focus on key feature regions that are less affected by blur and contrast reduction, our approach improves target recognition accuracy and robustness in sandstorm environments.

The main contributions of this paper are as follows: (1) We propose a robust target recognition algorithm tailored for sandstorm conditions, optimizing detection accuracy and resilience through the AlexNetBN network. (2) We develop and release a custom dataset of license plates affected by sandstorms to validate the effectiveness of our method, while also comparing its performance with existing advanced algorithms. In the following sections, we will first provide a detailed discussion of the AlexNetBN architecture, explaining why it is suitable for target recognition in sandstorm weather. Next, we will describe the construction of our custom dataset and the pre-processing techniques used to simulate the real-world conditions of sandstorms. Finally, we will present our experimental results, comparing the performance of our method with alternative approaches in terms of accuracy, efficiency, and robustness.

This work not only offers a new solution for target recognition under adverse weather conditions but also provides valuable insights into the application of CNN-based algorithms in scenarios where visibility is significantly compromised.

2. Materials and Methods

2.1. Introduction to the Model Structure

To address the issue that the feature extraction of the convolutional layer is disturbed by the sandstorm environment and the traditional pooling operation fails to meet the requirement of retaining key features, we made improvements based on the AlexNet model, specifically introducing BN layers after each convolutional layer. The BN layer helps to reduce the perturbation of input features, ensuring that before feature extraction by the convolutional layers, the image characteristics are already normalized [21]. This mitigates the impact of uneven illumination distribution, accelerates model convergence, and improves training stability. Additionally, we utilized max pooling operations to further enhance the model’s resistance to interference, as max pooling retains the most prominent features when noise levels are high [22], thereby ensuring the effective extraction of key information from the license plates.

Furthermore, considering the sensitivity of fully connected layers to noise, lighting, and rotation in complex environments, we retained only one fully connected layer in the model to reduce the number of parameters and avoid overfitting [23]. Ultimately, the license plate images are processed through five convolutional layers, each followed by a BN layer, and then further refined through pooling layers, before being integrated by the fully connected layer, and finally, character recognition is achieved via the SoftMax layer [24], and the model structure is shown in Figure 1.

This improved AlexNetBN architecture effectively handles license plate recognition tasks under extreme conditions such as sandstorms, enhancing the model’s robustness and training efficiency.

2.2. Analysis of Optimization Principles

In the process of license plate recognition, the input image first undergoes convolutional layer processing [25], where convolutional kernels extract feature information from the image. The feature information processed by the ReLU activation function is then fed into the next convolutional layer for similar operations. The feature maps generated by the convolutional operations are input into the pooling layer for further feature extraction. This study uses Max Pooling to enhance the ability of convolutional kernels to extract image information [26]. Finally, the features are integrated through a fully connected layer [27], and the final classification result is obtained through the classifier.

By observing the heat maps generated during the convolutional operations, as shown in Figure 2b, in complex environments, the convolutional layers have difficulty effectively extracting the key features of license plates. To address the nonlinearity issue in convolutional neural networks, this study employs the ReLU activation function [28], which is defined by the following formula:

$$\mathrm{Re}LU(\mathrm{w})=\left\{\begin{array}{cc}w& w\ge 0\\ 0& w\le 0\end{array}\right.$$

(1)

$w$ represents the input value of the activation function, which specifically refers to the output of the previous convolutional layer in this study.

However, relying solely on the ReLU activation function is insufficient to effectively address the challenges faced by the convolutional layers when processing license plates in complex environments. To overcome these issues, BN is introduced into the traditional AlexNet model. The BN layer normalizes each feature channel and introduces two learnable parameters to enhance the model’s expressive capability.

During normalization, the BN layer computes the mean and variance of the input features within each mini-batch [21]. Let ${\mu }_{batch}$ represent the mean, and the formula for calculating the mean is as follows:

$${\mu }_{batch}={\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=1}^m{X}_i}$$

(2)

where $m$ is the number of samples in the mini-batch, and $X_{\mathrm{i}}$ represents the input feature of the i sample. Let ${{\sigma }^2}_{batch}$ represent the variance, and the formula for calculating the variance is as follows:

$${{\sigma }^2}_{batch}={\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=1}^m{\left(X_i-{\mu }_{batch}\right)}^2}$$

(3)

Each input feature is normalized according to the following formula [21]:

$${\widehat{X}}_i={\displaystyle \frac{X_i-{\mu }_{batch}}{\sqrt{{{\sigma }^2}_{batch}+\epsilon }}}$$

(4)

where ${\widehat{X}}_i$ is the normalized input, and $\epsilon$ is a small constant added to prevent division by zero.

The BN layer normalizes the input of each layer to reduce the impact of internal covariate shift. This mechanism keeps the mean and variance relatively stable during training, effectively reducing disturbances in the input features [29]. This stability allows the network to better handle conditions like sandstorm blurring and strong light reflections, resulting in more uniform illumination of license plate images, effectively resisting noise interference, and facilitating subsequent feature extraction.

In addition, the BN layer introduces two learnable parameters, whose formulas are as follows:

$$y_i=\gamma {\widehat{X}}_i+\beta$$

(5)

where the scaling parameter $\gamma$ adjusts the scale of the features, allowing the normalized features to be adapted to the range suitable for the current task. The shift parameter $\beta$ translates the normalized features, further adjusting their distribution. The introduction of these parameters enables the network to flexibly recover the original feature distribution during training, reducing sensitivity to the initial values of the network weights [21].

Figure 2 shows the attention levels of different networks to various regions of the input image [30,31]. Figure 2a represents the input image, while Figure 2b illustrates that the network without the BN layer excessively focuses on the image’s edge information. Figure 2c visually reveals that the network with the BN layer focuses more on the area containing the license plate number. A comparison of Figure 2a–c shows that the addition of the BN layer enables the network to concentrate more on extracting the core information of the license plate while paying less attention to distracting edge information and background noise. In complex environments, it can effectively filter out some unnecessary information in the image, such as lighting variations and cluttered backgrounds, ensuring that the attention remains focused on the license plate number. Thus, adding a BN layer helps to address the limitations of the convolutional layers, significantly improving their feature extraction efficiency, which is crucial for license plate recognition in complex environments. We attempted to integrate the BN layer into the convolutional layer to further enhance the model’s convergence speed and recognition performance, addressing the challenges of license plate recognition in environments with frequent sandstorms and dramatic changes in light intensity.

3. Results

3.1. Construction of the Dataset

Choosing a suitable dataset is especially critical when training a license plate recognition model suitable for special environments such as semi-arid regions with frequent sand and dust and intense light [32]. Datasets targeting specific environmental conditions can significantly improve the performance of the model in these scenarios, especially in regions with drastic changes in light and frequent sand and dust weather. However, the coverage of existing mainstream datasets at home and abroad in this area is still insufficient to provide adequate support for the needs of this study.

Table 1. Comparison of license plate datasets.

Dataset	Year	Country	Amount	Resolution	Description
CCPD (2019) [33]	2019	China	250 k	720 × 1280	Unevenly bright license plates, tilted license plates, etc., but not enough of them
UFPR-ALPR [34]	2018	Brazil	4500	1920 × 1080	License plates from different national regions, lack of dusty and light-variable license plates
GAP-LP [34]	2019	Tunisia	9175	Includes multiple resolutions	Lack of license plates with lots of dust and strong light variations; labels may be inaccurate and inconsistent
OpenALRP-EU [35]	2016	Europe	108	Includes multiple resolutions	License plates from various EU countries with poor generalization when used
USCD-still [36]	2005	America	291	640 × 480	Multiple license plate styles and image capture conditions, fewer license plates in complex environments
CD-HARD [37]	2016	Involving multiple countries	102	Includes multiple resolutions	Higher number of difficult-to-identify samples, but lack of photographs of license plates that are sandy and have strong light variations
CSCL	2024	China	25 k	240 × 80	Includes a large number of license plate photos for areas with frequent dust storms and drastic changes in lighting

demonstrates a comparative analysis of some public datasets; however, we find that most public datasets do not contain a large number of license plate images from sandy and dusty environments with strong lighting. Therefore, constructing a more targeted dataset that can effectively fill this gap is a top priority for this study.

The following data enhancement strategies were used in this study:

• Noise addition: By introducing additional ‘granular’ interference into the image, we simulate the image quality problems that may occur in the real environment. This strategy effectively trains the robustness of the model in dealing with multiple noise interferences, so that it can maintain stable recognition performance in the face of complex real-world scenarios, providing the model with ‘real-world experience’ in dealing with various uncertain environments.
• Light intensity variations: By simulating different lighting conditions, such as strong sunlight and deep shadows, the model is able to learn how to accurately recognize license plates in diverse lighting environments, thus effectively reducing recognition errors due to lighting variations.
• Rotation and tilting: By rotating and tilting the image to simulate the presentation of the license plate under different viewing angles and perspectives, the model is enhanced to adapt to changes in the position of the license plate, thus improving its recognition accuracy in real application scenarios. This enhancement allows the model to better cope with the challenges posed by changes in angles and optimizes its performance in complex scenes.
• Masking: By adding a large amount of noise or increasing the local brightness in the image, some or all characters of the license plate are masked to simulate the state of the license plate in complex environments such as sand and dust coverage or direct sunlight, which further improves the robustness and adaptive ability of the model.

With these data enhancement strategies, we constructed a more targeted dataset CSCL (license plates in China with a lot of sand and dust and strong changes in light variations). Some of the license plate images are shown in Figure 3. This dataset more accurately reflects the characteristics of the license plate in areas with frequent sand and dust and strong light, and significantly improves the performance of the license plate recognition model in semi-arid areas. The comparison of this dataset with the existing dataset is shown in Table 1.

By adding Gaussian noise, salt-and-pepper noise, random noise, and adjusting the image colors, the images are made to appear as if they have more ‘dust,’ with a more yellowish hue, thus more realistically simulating a dust-heavy environment. The result is shown in Figure 3a. Randomly adjusting image brightness simulates conditions of strong lighting, as shown in Figure 3b. Rotation techniques are employed for data augmentation to enhance the robustness of model training and prevent issues such as overfitting. The results are shown in Figure 3c. By covering the license plate surface with a significant amount of ‘dust’ and greatly increasing local brightness, real-world scenarios are simulated where license plates are covered by dust or directly exposed to sunlight, as shown in Figure 3d.

3.2. Experimental Results and Analyses

The evaluation metrics for the license plate recognition network model in this study mainly included accuracy (acc), PFC (Probability of Fully Identifying the Correct License Plate Number), and Loss (value of the loss function). Among them, acc is the ratio of the number of symbols correctly recognized by the model in the prediction to the total number of symbols; by comparing the accuracy rates of different models or different datasets, the optimal model could be selected; by observing the trend of the accuracy rate during the training process, we could judge whether the model converged or overfitted; by analyzing the accuracy rate on the test set, we could evaluate the model’s generalization ability. The PFC refers to the model’s ability to completely recognize all the characters correctly on a license plate. It is more stringent than the accuracy rate, as PFC requires that every character on the license plate must be recognized correctly; it is usually lower than the accuracy rate because it is more sensitive to errors. Loss is a metric that measures the difference between the model’s prediction and the true value; observing the trend of the loss value during the training process can determine whether the model learns effectively or not; comparing the loss value on the training and validation sets can help us detect overfitting. Here, A represents the model trained using the augmented dataset, N represents the model with an added BN layer, O represents the model trained using the unaugmented dataset, val indicates the model’s performance on the test set, and train indicates the model’s performance on the training set.

In this study, we trained both the AlexNetBN model and the AlexNet model using the original dataset and the augmented dataset, respectively. After completing the training, we conducted an in-depth analysis of the performance metrics of the different models on these two datasets, aiming to validate the effectiveness of the optimization measures proposed in this study and to explore their impact on model performance.

Figure 4 shows the performance of different models on the test set. From Figure 4, it can be observed that during the fifth and sixth epochs, the recognition accuracy of each model reached its peak. Subsequently, as the number of training epochs increased, the improvement in recognition accuracy gradually diminished and tended to converge. Furthermore, when the number of training epochs exceeded six, some models began to exhibit overfitting. The second part of this section will focus on analyzing this phenomenon. Therefore, this study will focus on the recognition performance of each model around the sixth epoch, as well as the overfitting that occurs when the number of training epochs increases, to gain a deeper understanding of the performance characteristics of the models and their potential influencing factors.

3.2.1. Comparative Experiments on Datasets

In this section, we analyze the performance metrics of the AlexNetBN model after training with different datasets to validate the effectiveness and importance of data augmentation techniques in model training.

Figure 5a shows that the enhancement of the dataset greatly improved the model performance, and it can be seen that the model trained using the enhanced dataset had a reduced change in accuracy after the fifth round of training, and tended to converge with a recognition accuracy of more than 90%; whereas the model trained using the original dataset began to converge only after the ninth round, and had a recognition accuracy of only about 80%. As can be seen from Figure 5b, in the model training after adding the BN layer, the number of completely correctly recognized license plate numbers (PFC) of the model trained with the enhanced dataset was significantly larger than that of the model trained with the original dataset, and at the same time, the PFC value was very sensitive to the occurrence of recognition errors, which suggests that it was less likely to recognize license plate numbers incorrectly in the model trained with the enhanced dataset. From the analysis of Figure 5a,b, it can be concluded that, under the same model architecture, the model trained with the augmented dataset outperformed the model trained with the original dataset in terms of recognition accuracy, convergence speed, and PFC. This indicates that the diversity and relevance of the dataset are crucial to the model’s performance. Data augmentation enhances the effectiveness of training data by introducing diversified sample transformations, allowing the model to better learn feature diversity and robustness. This not only improves the generalization ability of the model but also achieves better performance within a shorter training period [38]. Therefore, this study clearly demonstrates the effectiveness of data augmentation techniques.

3.2.2. Ablation Experiment

In this section, we analyze the performance metrics of different models after training with different datasets to illustrate the effectiveness and importance of the BN layer.

The comparison of accuracy curves in Figure 6a shows that the model with the BN layer converged faster around the fourth epoch, and its recognition accuracy was significantly higher than that of the model without the BN layer. Figure 6b demonstrates that at the sixth epoch, the model with the BN layer had a license plate symbol recognition accuracy of more than 90% and a probability of recognizing the correct license plate number of about 56%, which is much higher than the model without the BN layer, which had a recognition accuracy and a PFC value of about 80% versus 20%. These results clearly show that the BN layer makes deep network training more efficient by reducing the phenomena of gradient vanishing and gradient explosion [39], thereby effectively improving the overall performance of the model, greatly improving the model’s recognition accuracy in areas with high sand and dust and large illumination variations.

Observing Figure 7, we can see that while the difference between the predicted and true values of the different models in the training set gradually decreased with the increase in training epochs, their performance on the test set varied significantly. Without the BN layer, the error on the test set initially decreased with the increase in training epochs but then began to rise. This indicates that, in the later training sessions, the model’s predictions of the results on the test set became more different from the true results, and the model overlearned the image features in the training set. Although the model’s recognition accuracy remained almost unchanged, the increase in the loss function value revealed the presence of overfitting. In contrast, the model with the BN layer did not exhibit overfitting. This phenomenon indicates that the BN layer, by normalizing the input of each layer, reduces the impact of internal covariate shift and introduces a certain level of noise [40], thereby effectively serving a regularization function [41]. This regularization helps to mitigate the risk of overfitting on the training set and allows the model to maintain better recognition performance when encountering unseen data.

3.2.3. Comparison Experiment Before and After Optimization

In complex environments with frequent sandstorms and dramatic fluctuations in lighting conditions, the model is highly susceptible to interference during critical processing steps such as feature extraction. The trend of the two accuracy curves in Figure 8 clearly demonstrates that the addition of the BN layer and the application of data augmentation techniques effectively improve the stability of license plate recognition performance under such challenging conditions. Specifically, the overall optimized model after convergence had a recognition accuracy of more than 91%, which is much higher than the original model’s 80% recognition accuracy. This result highlights the importance of combining BN and data augmentation techniques during model training, further confirming the significant impact of these two strategies in enhancing model performance and generalization ability.

4. Discussion

4.1. Conclusions

In this paper, we created a dedicated dataset for sandstorms that may lead to the blurring of license plate recognition, reduction in contrast, and loss of some information, and used this enhanced CSCL dataset to train the AlexNetBN network. Our method proves its effectiveness through mathematical principles and model training and testing. We compared the AlexNetBN model after using the optimization method with the AlexNet model without the optimization method; the former performed extremely well on the test set, with a recognition accuracy of more than 91%, while the latter did not perform well on the test set, with a recognition accuracy of only about 80%, which shows that our method can handle the problem of license plate recognition in areas with frequent sandstorms and large variations in light intensity more effectively.

4.2. Future Work

However, some aspects of this work still need to be addressed in the future. Firstly, the proposed license plate recognition algorithm is mainly designed for Chinese license plates with a blue background and white text. Given the diverse styles of license plates worldwide, the scope of this study is relatively limited. Therefore, future research should broaden its scope to include license plate recognition across different countries and regions. Secondly, our optimization model was not compared with other current state-of-the-art models (e.g., modern YOLO algorithms) for performance in harsh environments, so this aspect will be addressed in our future work. Finally, our experiments did not include the study and optimization of license plate detection techniques under sandstorm conditions, so we will study license plate detection techniques under sandstorm conditions in detail in our future work.