Comparative Evaluation of Deep Traffic Sign Classification Models Under Visual Degradations and Interpretability Analysis

Abstract

Reliable traffic sign classification is essential for advanced driver assistance and autonomous-driving systems because recognition errors under real road conditions can directly affect navigation, warning, and safety-related decisions. This study presents a reliability-oriented comparison of BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 enhanced with the Convolutional Block Attention Module (CBAM), and knowledge-distilled MobileNetV2. All models were trained from scratch on a fixed stratified split of the German Traffic Sign Recognition Benchmark (GTSRB) using five independent random seeds. The evaluation considered clean classification performance, training stability, bootstrap confidence intervals, McNemar paired tests, probabilistic calibration, severity-wise robustness under blur, central occlusion, low-light, and Gaussian noise corruptions, external validation on 360 cropped German Traffic Sign Detection Benchmark (GTSDB) signs, computational efficiency, and Grad-CAM-based diagnostic analysis. Across five seeds, ResNet18 achieved the strongest mean clean performance, with an accuracy of 0.9856 ± 0.0093 and macro-F1 of 0.9817 ± 0.0134. MobileNetV2 remained competitive, with an accuracy of 0.9813 ± 0.0057 and macro-F1 of 0.9773 ± 0.0069, whereas BaselineCNN was substantially weaker, with an accuracy of 0.8459 ± 0.0165 and macro-F1 of 0.8384 ± 0.0189. ResNet18 also showed strong calibration, with an expected calibration error of 0.0017, and achieved the best GTSDB macro-F1 of 0.9389 in the representative Seed-42 external evaluation. Severe central occlusion was the most damaging corruption, reducing all models below 0.11 macro-F1, while low-light degradation was comparatively less harmful for the stronger classifiers. The results show that model ranking changes across accuracy, calibration, robustness, external transfer, computational cost, and visual diagnostic behavior. Therefore, traffic sign classifiers should be selected using multi-seed, multi-metric evaluation rather than clean benchmark accuracy alone.

1. Introduction

Traffic sign recognition is an important component of intelligent transportation systems, advanced driver assistance systems, and autonomous driving pipelines [1]. Traffic signs communicate speed limits, warnings, restrictions, and navigation rules; therefore, classification errors can affect safety-related downstream decisions. Although benchmark datasets have enabled substantial progress in traffic sign classification, real road conditions remain more variable than clean test images. Traffic signs may appear blurred, partially occluded, poorly illuminated, noisy, small, or captured from different viewpoints. A model that performs well on clean benchmark images may therefore still behave unreliably when visual conditions change. The German Traffic Sign Recognition Benchmark (GTSRB) [2] is a widely used benchmark for traffic sign classification and contains traffic sign images from 43 semantic classes [3]. In this study, GTSRB is used for model training, validation, clean testing, and controlled corruption testing. To examine external transfer, the German Traffic Sign Detection Benchmark (GTSDB) [4] is used only for external evaluation after cropping annotated traffic sign regions and mapping compatible labels to the GTSRB class taxonomy. No GTSDB image is used for model training, validation, calibration, or hyperparameter selection. Recent traffic sign recognition studies have examined convolutional neural networks, residual networks, lightweight mobile models, attention mechanisms, transformer-based detectors, and model compression strategies. However, many comparisons still emphasize clean accuracy or accuracy–speed trade-offs. For safety-relevant visual recognition, this is incomplete. A practical model comparison should also consider training variability, paired statistical evidence, confidence calibration, degradation sensitivity, external transfer, computational cost, and visual diagnostic behavior. Therefore, this study presents a reliability-oriented comparative evaluation of five model configurations: BaselineCNN, ResNet18 [5], MobileNetV2 [6], MobileNetV2 enhanced with CBAM [7], and knowledge-distilled MobileNetV2 [8]. The objective is not to introduce a new neural network backbone, but to compare representative model families under the same preprocessing, splitting, training, and evaluation protocol. BaselineCNN provides a shallow reference classifier, ResNet18 represents residual feature learning, MobileNetV2 represents compact mobile classification, MobileNetV2 + CBAM examines attention-based feature refinement, and MobileNetV2-KD examines teacher–student compression using ResNet18 as the teacher.

1.1. Gap Analysis and Study Objective

Existing traffic sign classification studies often report strong clean-set performance, but clean accuracy alone does not establish whether a model is stable across training runs, calibrated in its confidence estimates, robust under visual degradation, transferable to an external dataset, computationally feasible, or diagnostically interpretable. The effects of attention and knowledge distillation are also difficult to isolate when model comparisons use different training protocols, data splits, preprocessing pipelines, or evaluation metrics. This motivates a controlled comparison in which all evaluated models are trained and tested under the same experimental conditions. In this study, reliability-oriented evaluation refers to the joint assessment of clean predictive performance, training stability, paired statistical evidence, probabilistic calibration [9], robustness under controlled visual corruptions, external generalization, computational efficiency, and Grad-CAM-based diagnostic behavior. These dimensions are considered together because the model with the highest clean GTSRB accuracy may not necessarily be the most robust, best calibrated, fastest, most transferable, or most useful for diagnostic failure analysis.

1.2. Novelty and Contributions

The novelty of this study lies in the controlled, multi-dimensional evaluation protocol rather than in the proposal of a new neural network architecture. The main contributions are as follows:

• A five-model comparison is conducted using a fixed stratified GTSRB partition and five independent random seeds, allowing clean performance and training stability to be reported separately.
• Statistical and probabilistic reliability are assessed using bootstrap confidence intervals, McNemar paired tests, expected calibration error, negative log-likelihood, and Brier score.
• Robustness is evaluated under three severity levels of blur, central occlusion, low-light, and Gaussian noise corruptions.
• External generalization is evaluated using 360 cropped GTSDB signs mapped to compatible GTSRB labels, without using GTSDB for training or hyperparameter selection.
• The effects of CBAM and knowledge distillation are examined under the same training and evaluation protocol as the standard MobileNetV2 model.
• Computational efficiency is characterized through parameter count, model size, training time, inference latency, throughput, and peak memory usage.
• Grad-CAM [10] is used for diagnostic analysis of correct predictions, misclassifications, and high-confidence failure cases. Note that heatmap concentration is not necessarily a validated explanation-quality metric.

1.3. Research Questions

The study is guided by the following research questions:

• How do BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and knowledge-distilled MobileNetV2 compare in clean GTSRB classification performance, and how stable are their results across independent training seeds?
• How do the evaluated models compare in probabilistic calibration, including expected calibration error, maximum calibration error, negative log-likelihood, and Brier score?
• How robust are the evaluated models under severity-wise blur, low-light, Gaussian noise, and occlusion corruptions, and how does targeted central occlusion differ from support-matched random occlusion?
• What effects do CBAM and knowledge distillation have on clean performance, robustness, calibration, external generalization, computational efficiency, and visual-attention behavior relative to standard MobileNetV2?
• How well do the evaluated models generalize from GTSRB to the class-compatible subset of GTSDB, considering class coverage, unequal support, and per-class performance?
• What relationships exist between Grad-CAM center concentration, prediction confidence, and prediction correctness, and what limitations arise when concentration is used as a diagnostic explanation measure?

The remainder of the paper is organized as follows: Section 2 reviews related work and positions the present study. Section 3 describes the datasets, model architectures, training protocol, and evaluation metrics. Section 4 reports the experimental results. Section 5 discusses the findings, limitations, and practical implications. Section 6 concludes the paper.

2. Literature Review

Several recent studies provide direct model-comparison baselines for traffic sign recognition. Lin [11] compared CNN, ResNet-18, and MobileNetV2 on GTSRB using accuracy and inference time; Postovan and Eraşcu [12] studied compact binarized architectures across GTSRB and external Belgian/Chinese traffic sign datasets; Rahman et al. [13] compared pretrained CNNs with LIME-based explanations; and Benfaress et al. [14] combined CNN comparison, adverse condition augmentation, GTSRB validation, and Grad-CAM analysis. These works strengthen the motivation for the present study, which extends direct model comparison by jointly evaluating multi-seed stability, paired statistical testing, calibration, severity-wise corruption robustness, external GTSDB transfer, efficiency, and Grad-CAM concentration.

2.1. Deep Learning Architectures for Traffic Sign Classification

Deep learning architectures remain central to the scientific development of traffic sign recognition because they determine how visual features are extracted, represented, and separated across visually similar traffic sign classes. Obeidat et al. [15] proposed a traffic sign detection and recognition system for Jordanian road signs by combining ResNet-50 feature extraction with SVM classification. Their work is important because it shows that deep feature extractors can be effectively combined with classical classifiers for country-specific traffic sign recognition, although the reported performance also highlights the difficulty of real-world regional traffic sign datasets. Gan et al. [16] introduced Cross-domain Multi-step Thinking (CdMT) for zero-shot fine-grained traffic sign recognition in the wild. Their study is highly relevant because it explicitly addresses the cross-domain gap between clean template signs and real-world traffic sign images, which supports the need for external generalization analysis in traffic sign recognition studies. The Cui study [17] proposed a DETR-based traffic sign detection and recognition framework that combines multi-scale feature fusion with attention-enhanced representation learning. This work demonstrates the increasing role of transformer-based architectures in traffic sign perception, especially where contextual reasoning and multi-scale information are required. Sun et al. [18] proposed ERF-YOLO, which integrates adaptive sequence fusion, self-calibration feature enhancement, and wavelet-based feature reconstruction for traffic sign detection. These studies suggest that recent performance gains in traffic sign recognition depend not only on deeper backbones, but also on multi-scale feature preservation and noise-aware feature reconstruction. In short, these studies show that traffic sign recognition has progressed and shifted from conventional CNN-based classification toward hybrid, transformer-based, and feature-enhanced architectures. At the same time, many architecture-focused studies do still require stronger joint evaluation of clean accuracy, robustness, calibration, external transfer, computational cost, and interpretability.

2.2. Lightweight and Efficient Models for Embedded Traffic Sign Recognition

Since road-scene perception requires low-latency decisions on resource-constrained devices, lightweight and efficient traffic sign models are important for autonomous vehicles, embedded cameras, and advanced driver-assistance systems. Qiu et al. [19] proposed DP-YOLO for small traffic sign detection, combining modified detection layers, enhanced feature processing, and transformer-based representation to reduce parameters while improving performance on TT100K, GTSDB, and CCTSDB. Cui et al. [17] developed a lightweight real-time detection transformer for long-range traffic sign perception by integrating efficient convolutional features, small-object modules, and attention mechanisms. Huang [20] introduced a lightweight nighttime road obstacle detector that, although not traffic sign-specific, shows how MobileNet-style features, Wise-IoU loss, and parameter-free attention can support compact low-light road scene perception. Chen et al. [21] proposed YOLO-SMM, combining SlimNeck, modified coordinate attention, and MPDIoU to improve the accuracy–speed trade-off while preserving small-sign details in cluttered scenes. Together, these studies show that practical efficiency should be measured empirically through parameter count, model size, training time, inference time, and predictive performance, rather than inferred only from architectural compactness.

2.3. Robustness, Visual Degradation, and Cross-Dataset Generalization

Robustness and cross-dataset generalization are critical in traffic sign recognition because deployed models must handle changing weather, lighting, camera quality, road environments, and regional sign appearances. Wang et al. [22] introduced a harsh-environment traffic sign benchmark and an attention-based restoration method for detection under rain, snow, fog, and glare, showing that adverse conditions require condition-specific robustness evaluation rather than clean benchmarks alone. Wang et al. [23] addressed difficult illumination through adaptive enhancement and dual-branch feature fusion, supporting the need to evaluate low-light, overexposed, and underexposed traffic scenes explicitly. Asamoah et al. [24] proposed a sign-aware attention-based multitask framework for detection, classification, and retroreflectivity estimation, linking recognition with safety-oriented nighttime and adverse-weather maintenance while reporting statistical evidence across multiple runs. Du et al. [25] developed a lightweight robust detector for adverse-weather autonomous driving perception; although focused on vehicles and pedestrians, it demonstrates that weather degradation, motion blur, and poor visibility require robustness-aware model design. Collectively, these studies show that robustness depends on the degradation type rather than being a single global model property, motivating the present study’s severity-wise evaluation of blur, occlusion, low-light, and Gaussian noise corruptions, together with external GTSDB validation for cross-dataset behavior.

2.4. Attention, Interpretability, and Reliability-Aware Evaluation

Attention, interpretability, and reliability-aware evaluation are becoming increasingly important for traffic sign recognition because safety-oriented traffic sign systems must not only predict correctly but also focus on meaningful regions that provide trustworthy confidence estimates to the user, and also support transparent failure analysis in case of any failure. Santos et al. [26] critically examined limitations of Grad-CAM, including the effect of large receptive fields, feature map upsampling, feature map rectification, empty maps, and spatial correspondence assumptions. This study is important for the present work because it supports using Grad-CAM as a diagnostic tool rather than treating heatmaps as complete proof of model reasoning. Ni et al. [27] proposed an improved coordinate-attention residual network for dangerous driving behavior classification. Although the task is different from traffic sign classification, the work is relevant because it shows how coordinate attention can preserve positional information and enhance recognition of subtle traffic-related visual patterns. Noh et al. [28] proposed an efficient post hoc calibration approach for deep neural networks by reducing overly activated features that contribute to overconfident predictions. This is relevant to traffic sign recognition because safety-oriented models should not only be accurate but also calibrated, especially when incorrect high-confidence predictions may affect downstream driving decisions. Gou et al. [29] proposed a distillation strategy that transfers local relational structure from teacher to student models through intermediate features and output logits. This paper is relevant because the present study includes knowledge-distilled MobileNetV2 as a compact-model enhancement strategy and evaluates whether teacher–student learning improves performance, robustness, and reliability. Overall, the attention, explainability, calibration, and distillation literature indicates that traffic sign recognition should be evaluated not only by predictive accuracy but also by how models focus on visual regions, how reliable their confidence estimates are, and whether compact models retain useful teacher knowledge. This motivates the present study’s combined use of CBAM, Grad-CAM concentration analysis, calibration metrics, McNemar testing, and knowledge-distilled MobileNetV2 within a unified evaluation framework.

Table 1. Condensed comparison of representative traffic sign and reliability-oriented studies. The selected entries emphasize direct traffic sign model comparison, efficiency, robustness, external generalization, calibration/statistical reliability, and visual explanation.

Research Area	Study	Main Method/Focus	Dataset(s)	Clean Performance	Efficiency	Robustness	External Evaluation	Attention	Calib./Stat.	XAI
Direct model comparison	Lin [11]	Compared customized CNN, ResNet-18, and MobileNetV2 using accuracy and inference time.	GTSRB	✓	✓	✗	✗	✗	✗	✗
Direct compact model comparison	Postovan and Eraşcu [12]	Designed binarized neural-network and XNOR-style architectures for traffic sign classification.	GTSRB, Belgian traffic signs, Chinese traffic signs	✓	✓	✗	✓	✗	✗	✗
Direct XAI-based comparison	Rahman et al. [13]	Compared ResNet-34, VGG-19, DenseNet-121, and Inception V3 and used LIME to inspect prediction regions.	GTSRB	✓	✗	∼	✗	✗	✗	✓
Direct robustness and XAI comparison	Benfaress et al. [14]	Compared a proposed CNN with GoogLeNet, AlexNet, DarkNet-53, ResNet-34, VGG-16, and MicronNet-BF under adverse-condition augmentation and Grad-CAM.	TT100K, GTSRB	✓	∼	✓	✓	✗	∼	✓
Cross-domain generalization	Gan et al. [16]	Applied cross-domain multi-step reasoning with large multimodal models for zero-shot fine-grained traffic sign recognition.	GTSRB, BTSD, TT100K, Sapporo, Yokohama	✓	✗	∼	✓	∼	✗	∼
Lightweight detection and efficiency	Qiu et al. [19]	Proposed DP-YOLO based on YOLOv8s with small-object detection layers, DBBNCSPELAN4, PTCSP, and W3F_MPDIoU loss.	TT100K, GTSDB, CCTSDB	✓	✓	∼	✓	∼	✗	✗
Robustness and safety-oriented evaluation	Asamoah et al. [24]	Developed a sign-aware attention multitask framework for detection, classification, and retroreflectivity estimation.	Traffic sign and retroreflectivity datasets	✓	∼	✓	∼	✓	∼	✗
Calibration and confidence reliability	Noh et al. [28]	Proposed a post hoc calibration method for reducing overconfident deep-network predictions.	General image classification datasets	✓	∼	✗	✗	✗	✓	✗
Proposed study	Proposed study	Reliability-oriented comparison of BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and knowledge-distilled MobileNetV2.	GTSRB, cropped GTSDB	✓	✓	✓	✓	✓	✓	✓

Note: ✓ indicates that the corresponding dimension is explicitly evaluated; ∼ indicates partial or limited coverage; and ✗ indicates that the dimension is absent or not substantively evaluated.

presents a condensed comparison of representative studies selected from the four literature review themes. The complete table for literature review is displayed in

Table A1. Comparative literature review matrix highlighting the novelty of the proposed reliability-oriented traffic sign classification study. Part I: direct model-comparison, architecture, and efficiency-focused studies.

Field/Topic	Study	Main Method/Focus	Dataset(s)	Main Contribution	Clean Perf.	Efficiency	Robust.	External Eval.	Attention	Calib./Stat.	XAI	Key Limitation
Direct comparative traffic sign classification studies	Lin [11]	Compared customized CNN, ResNet-18, and MobileNetV2 for efficient GTSRB traffic sign recognition using test accuracy and inference time.	GTSRB	Reported a direct accuracy–runtime comparison: CNN achieved 95.35% accuracy in 2 s, ResNet-18 achieved 96.19% accuracy in 13 s, and MobileNetV2 achieved 87.77% accuracy in 8 s.	✓	✓	✗	✗	✗	✗	✗	Limited to clean GTSRB accuracy and inference time; does not evaluate degradation robustness, external transfer, calibration, statistical testing, or XAI.
	Postovan and Eraşcu [12]	Bottom-up design of binarized neural network and XNOR-style architectures for traffic sign classification, varying binarized convolution, max-pooling, batch normalization, and dense layer configurations.	GTSRB, Belgian traffic signs, Chinese traffic signs	Showed that compact BNN architectures can reach up to 96.45% on GTSRB and can be externally tested on Belgian and Chinese datasets while using approximately 100k to less than 2M parameters.	✓	✓	✗	✓	✗	✗	✗	Strong for compactness and cross-dataset testing but does not provide calibration analysis, paired statistical testing, severity-wise degradation robustness, or Grad-CAM/LIME-based explanation.
	Rahman et al. [13]	Compared ResNet-34, VGG-19, DenseNet-121, and Inception V3 for GTSRB classification and used LIME to inspect whether predictions relied on relevant image regions.	GTSRB	Reported near-perfect clean performance for pretrained CNNs and showed through LIME that high-scoring models may still rely on incomplete or misleading visual regions.	✓	✗	∼	✗	✗	✗	✓	Provides useful XAI-based reliability analysis but does not evaluate external transfer, probabilistic calibration, paired significance testing, computational profiling, or systematic corruption severity.
	Benfaress et al. [14]	Proposed an explainable CNN for traffic sign recognition and compared it with GoogLeNet, AlexNet, DarkNet-53, ResNet-34, VGG-16, and MicronNet-BF under augmentation-based adverse conditions.	TT100K, GTSRB	Combined model comparison, data augmentation for noise/contrast/blur/zoom effects, GTSRB validation, and Grad-CAM-based explanation for adverse-environment TSR.	✓	∼	✓	✓	✗	∼	✓	Strong robustness and XAI coverage but does not include probabilistic calibration, McNemar-style paired testing, five-seed training stability, or GTSDB-style cropped external transfer.
Deep learning architectures	Obeidat et al. [15]	ResNet-50 feature extraction with SVM classification for Jordanian traffic signs.	Jordanian traffic sign dataset	Demonstrated the usefulness of combining deep feature extraction with classical machine learning for regional traffic sign recognition.	✓	✗	∼	✗	✗	✗	✗	Focused on one regional dataset and did not provide calibration, statistical testing, or XAI analysis.
	Gan et al. [16]	Cross-domain multi-step thinking using large multimodal models for zero-shot fine-grained traffic sign recognition.	GTSRB, BTSD, TT100K, Sapporo, Yokohama	Addressed cross-domain and cross-country traffic sign recognition without task-specific training data.	✓	✗	∼	✓	∼	✗	∼	Focused on zero-shot LMM reasoning and did not evaluate model calibration, classical efficiency profiling, or degradation severity.
	Cui et al. [17]	DETR-based traffic sign detection and recognition using multi-scale feature fusion and attention.	Traffic sign detection/recognition datasets	Showed the potential of transformer-based architectures for traffic sign perception using contextual and multi-scale feature learning.	✓	∼	∼	∼	✓	✗	✗	Primarily architecture-focused, with limited reliability analysis beyond detection or recognition performance.
	Sun et al. [18]	ERF-YOLO with adaptive sequence fusion, self-calibration feature enhancement, and wavelet-based feature reconstruction.	TT100K, CCTSDB2021	Improved traffic sign detection by preserving multi-scale features and reducing noise through feature reconstruction.	✓	∼	∼	∼	∼	✗	✗	Focused on detection performance and feature enhancement without calibration or Grad-CAM-based interpretability.
Lightweight and efficient models	Qiu et al. [19]	DP-YOLO based on YOLOv8s with small-object detection layers, DBBNCSPELAN4, PTCSP, and W3F_MPDIoU loss.	TT100K, GTSDB, CCTSDB	Reduced parameters while improving small traffic sign detection across multiple datasets.	✓	✓	∼	✓	∼	✗	✗	Did not evaluate calibration, statistical reliability, or explanation quality.
	Cui et al. [17]	Lightweight real-time detection transformer for long-range traffic sign detection.	Traffic sign detection datasets	Improved long-range small-sign detection while targeting real-time autonomous driving perception.	✓	✓	∼	∼	✓	✗	✗	Focused mainly on detection efficiency and did not include calibration or XAI-based reliability assessment.
	Huang et al. [20]	Lightweight road-scene detector using compact feature extraction, Wise-IoU loss, and attention.	Nighttime road scene datasets	Demonstrated lightweight perception under low-light road conditions, supporting deployment-oriented model design.	✓	✓	✓	✗	∼	✗	✗	Not traffic sign-specific and does not provide cross-dataset traffic sign classification or calibration analysis.
	Chen et al. [21]	YOLO-SMM combining SlimNeck, modified coordinate attention, and MPDIoU for efficient traffic sign detection.	GTSRB	Improved the accuracy–speed trade-off while preserving small-sign information in cluttered scenes.	✓	✓	∼	✗	✓	✗	✗	Limited mainly to detection performance and does not jointly analyze calibration, statistical significance, or Grad-CAM behavior.

Note: ✓ indicates that the corresponding dimension is explicitly evaluated; ∼ indicates partial or limited coverage; and ✗ indicates that the dimension is absent or not substantively evaluated.

and

Table A2. Comparative literature review matrix highlighting the novelty of the proposed reliability-oriented traffic sign classification study. Part II: robustness, generalization, attention, interpretability, reliability, and positioning of the proposed study.

Field/Topic	Study	Main Method/Focus	Dataset(s)	Main Contribution	Clean Perf.	Efficiency	Robust.	External Eval.	Attention	Calib./Stat.	XAI	Key Limitation
Robustness and generalization	Wang et al. [22]	Harsh-environment traffic sign benchmark with attention-based restoration for adverse conditions.	Harsh-condition traffic sign data	Highlighted the need for evaluating traffic sign perception under rain, snow, fog, glare, and other adverse conditions.	✓	∼	✓	∼	✓	✗	✗	Strong on adverse-condition detection but not focused on classification calibration or Grad-CAM interpretability.
	Wang et al. [23]	Adaptive image enhancement and feature fusion for traffic sign detection in challenging illumination.	Traffic sign detection datasets	Improved traffic sign detection under overexposed, underexposed, and visually challenging traffic scenes.	✓	∼	✓	∼	∼	✗	✗	Primarily addresses illumination robustness and does not evaluate statistical calibration or explanation concentration.
	Asamoah et al. [24]	Sign-aware attention multitask framework for detection, classification, and retroreflectivity estimation.	Traffic sign and retroreflectivity datasets	Connected traffic sign recognition with safety-oriented nighttime and adverse-weather maintenance tasks.	✓	∼	✓	∼	✓	∼	✗	Reliability-oriented but targeted at detection and retroreflectivity rather than full classifier calibration and Grad-CAM analysis.
	Du et al. [25]	Lightweight robust detector for autonomous driving perception under adverse weather.	Adverse-weather road scene datasets	Showed that poor visibility, blur, and adverse weather require robustness-aware model design for road-scene perception.	✓	✓	✓	∼	∼	✗	✗	Focused on vehicle and pedestrian detection rather than traffic sign classification and interpretability.
Attention, interpretability, and reliability	Santos et al. [26]	Critical analysis of Grad-CAM limitations, including receptive-field effects, upsampling, and spatial correspondence assumptions.	General computer vision datasets	Showed that Grad-CAM should be interpreted carefully and used as a diagnostic rather than definitive explanation method.	✗	✗	✗	✗	✗	✗	✓	Not traffic sign-specific and does not evaluate classifier performance or robustness.
	Ni et al. [27]	Coordinate-attention residual network for dangerous driving behavior classification.	State Farm distracted driver dataset	Showed that coordinate attention can preserve spatial position information and improve traffic-related image classification.	✓	∼	∼	✗	✓	✗	✗	Traffic-related but not traffic-sign-specific; does not analyze calibration or Grad-CAM interpretability.
	Noh et al. [28]	Post hoc calibration method for reducing overconfident deep-network predictions.	General image classification datasets	Provided a reliability-oriented calibration strategy for improving confidence estimates in deep neural networks.	✓	∼	✗	✗	✗	✓	✗	General calibration study and not specific to traffic sign recognition, degradation robustness, or external traffic sign datasets.
	Gou et al. [29]	Neighborhood relation-based knowledge distillation using intermediate features and output logits.	General image classification datasets	Improved teacher–student knowledge transfer by preserving local relational structure between samples.	✓	∼	✗	✗	✗	∼	✗	General KD method and not directly evaluated for traffic sign robustness, calibration, or Grad-CAM explanation quality.
	Proposed study	Reliability-oriented comparison of BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and knowledge-distilled MobileNetV2.	GTSRB, cropped GTSDB	Provides a unified framework combining clean classification, bootstrap confidence intervals, McNemar tests, calibration metrics, severity-wise robustness, external validation, computational profiling, CBAM/KD analysis, and Grad-CAM concentration.	✓	✓	✓	✓	✓	✓	✓	Five training seeds were evaluated, although the external GTSDB subset remains small and class-imbalanced, and the explanation analysis does not include human-expert validation.

Note: ✓ indicates that the corresponding dimension is explicitly evaluated; ∼ indicates partial or limited coverage; and ✗ indicates that the dimension is absent or not substantively evaluated.

. Two studies were retained from each theme to preserve coverage while improving readability. The tick–cross columns indicate whether each study explicitly evaluates clean predictive performance, computational efficiency, visual robustness, external generalization, attention mechanisms, calibration or statistical reliability, and explainability. The comparison shows that prior studies typically address only a subset of these dimensions. The contribution of the present study therefore lies in integrating these reliability dimensions within a common five-model and five-seed evaluation framework rather than in proposing a new backbone architecture.

3. Methodology

This study adopts a controlled reliability-oriented framework for comparing traffic sign classifiers across complementary evaluation dimensions. No new traffic sign dataset was created. GTSRB was used for model development, held-out clean testing, and controlled corruption testing, while cropped and label-compatible GTSDB signs were used only for external validation. All model configurations used the same preprocessing, augmentation, optimizer, learning rate, batch size, and early-stopping protocol. The train, validation, and test partition was fixed once and reused across all models. Training was repeated with five random seeds: 42, 123, 777, 2024, and 3407. Seed 42 was used only for representative qualitative figures and diagnostic examples; the main quantitative conclusions are based on five-seed summaries unless explicitly stated otherwise. The overall workflow is shown in Figure 1.

3.1. Datasets and Experimental Design

The main experiments were conducted using the German Traffic Sign Recognition Benchmark (GTSRB), which contains traffic sign images from 43 classes. GTSRB was used for training, validation, clean testing, and corruption testing. To evaluate external generalization, cropped traffic sign samples from the German Traffic Sign Detection Benchmark (GTSDB) were used as an external test set. As GTSDB is originally structured for detection, annotated traffic sign regions were cropped and mapped to compatible GTSRB class labels. This external set contained 360 cropped signs covering 38 of the 43 GTSRB classes. No GTSDB samples were used during training, validation, calibration, or hyperparameter selection. The data partition was generated once and reused across all model configurations and seed-specific training runs. Bootstrap confidence intervals were computed by resampling the held-out test predictions and therefore estimate test sample uncertainty for a given checkpoint. Training variability is reported separately using the mean and sample standard deviation across the five independent seeds.

3.2. Reproducibility and Multi-Seed Training Protocol

To quantify variability caused by model initialization, minibatch ordering, augmentation randomness, and stochastic optimization, every principal architecture was trained independently using five predetermined random seeds: 42, 123, 777, 2024, and 3407. The same stratified data split was reused across all runs. For each model and seed, clean classification metrics, calibration metrics, corruption robustness results, external GTSDB performance, and efficiency measurements were saved. Seed 42 was retained for representative visualizations such as confusion matrices, reliability diagrams, Grad-CAM panels, and qualitative failure cases. These representative figures are not used as substitutes for the five-seed quantitative analysis. To support reproducibility, the complete experimental implementation is publicly available at https://github.com/KlausCaka1/MDPI-Traffic-Sign (accessed on 2 July 2026). The repository contains the five seed-specific notebooks used for the independent training runs, together with execution instructions, dependency information, citation metadata, and licensing information.

3.3. Model Architectures

Table 2. Architecture and implementation details of the evaluated model configurations.

Model	Architecture Details	Parameters
BaselineCNN	Conv2d(3,32), kernel 3, padding 1; ReLU; MaxPool2d(2); Conv2d(32,64), kernel 3, padding 1; ReLU; MaxPool2d(2); AdaptiveAvgPool2d(6,6); Flatten; Linear(2304,128); ReLU; Dropout(0.3); Linear(128,43).	319,979
ResNet18	Torchvision ResNet18 trained without ImageNet pretrained weights; final fully connected layer replaced with Linear(in_features,43).	11,198,571
MobileNetV2	Torchvision MobileNetV2 trained without ImageNet pretrained weights; final classifier layer replaced with Linear(in_features,43).	2,278,955
MobileNetV2 + CBAM	Torchvision MobileNetV2 feature extractor followed by CBAM. CBAM uses channel attention with reduction ratio 16 and spatial attention with kernel size 7. Features are pooled with AdaptiveAvgPool2d(1,1) and classified using Linear(1280,43).	2,483,853
MobileNetV2-KD	Same architecture as MobileNetV2. The student is trained using ResNet18 as teacher with temperature-scaled teacher–student KL divergence and hard-label cross-entropy.	2,278,955

displays the five model configurations evaluated in this study: BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and MobileNetV2-KD. These configurations were selected to represent a shallow reference model, residual learning, compact mobile classification, attention-enhanced mobile classification, and teacher–student compact model learning. Other architectures such as VGG-16, LeNet-5, TrafficSignNet, EfficientNetB7, and YOLO-family detectors were not included in the main comparison because the purpose of this study was to compare representative classification models under a controlled reliability protocol rather than to exhaust all possible traffic sign architectures. YOLO-family methods are primarily detection models and require a different scene-level detection and classification protocol.

3.4. Statistical Significance Testing

McNemar paired tests were used to compare two classifiers on the same held-out GTSRB test images. For each model pair, every test image was assigned to one of four outcomes: both models correct, both models incorrect, the first model correct and the second model incorrect, or the first model incorrect and the second model correct. The test uses only the two discordant counts because these represent images on which the paired classifiers disagree. In this study, n₀₁ denotes the number of test images misclassified by the first model but correctly classified by the second model, while n₁₀ denotes the reverse case. The null hypothesis is that the two models have equal probability of being correct on the paired test instances. McNemar testing was therefore used to determine whether observed performance differences reflected systematic paired-prediction differences rather than small numerical variation.

3.5. Calibration Analysis

The reliability of predicted probabilities was evaluated using expected calibration error (ECE), negative log-likelihood (NLL), and Brier score on the clean GTSRB test set. ECE measures the gap between predicted confidence and empirical accuracy across confidence bins. NLL evaluates the quality of the full predicted probability distribution assigned to the true class, while the Brier score measures the squared error between predicted probabilities and one-hot class labels. Lower values indicate better probabilistic calibration. Calibration is important because a traffic sign classifier should not only predict the correct class but also provide confidence estimates that are consistent with its empirical correctness.

3.6. Robustness Evaluation Under Visual Degradations

Robustness was evaluated by applying controlled synthetic corruptions to the GTSRB test images. Four corruption types were considered: Gaussian blur, central occlusion, low-light, and Gaussian noise. Each corruption was evaluated at three severity levels. Central occlusion was designed as a targeted stress test because the central region of a traffic sign often contains the discriminative symbol, digit, or directional arrow. Therefore, central occlusion should be interpreted as a worst-case corruption rather than as an unbiased estimate of average real-world obstruction. Macro-F1 was used as the main robustness indicator because it is more sensitive to class imbalance than accuracy. This protocol allows degradation sensitivity to be compared across shallow, residual, lightweight, attention-enhanced, and knowledge-distilled classifiers under the same controlled corruption settings.

3.7. Computational Efficiency Assessment

Computational efficiency was evaluated in the same execution environment for all models. The reported measures include parameter count, serialized model size, number of training epochs, total training time, mean inference time per image, frames per second, and peak GPU memory usage. Inference latency was measured with models in evaluation mode and gradient computation disabled. These measurements provide an implementation-specific comparison under the reported software and hardware environment; they should not be interpreted as hardware-independent deployment latency.

3.8. Evaluation Metrics

The evaluation framework was designed to assess not only clean classification performance but also training stability, predictive uncertainty, statistical significance, calibration, robustness, external generalization, computational efficiency, and visual interpretability.

Table 3. Evaluation dimensions, datasets, and reported metrics used in the study.

Evaluation Dimension	Dataset/Setting	Reported Metrics
Clean classification	Held-out GTSRB test set	Accuracy, macro-precision, macro-recall, macro-F1, weighted-F1
Training stability	Five independent seeds	Mean and sample standard deviation across seeds
Test sample uncertainty	Held-out GTSRB predictions	Bootstrap 95% confidence intervals
Paired significance	Same GTSRB test images	McNemar test using discordant counts $n_{01}$ and $n_{10}$
Calibration	Clean GTSRB test set	ECE, NLL, Brier score
Robustness	Corrupted GTSRB test images	Macro-F1 across three severity levels for blur, central occlusion, low-light, and Gaussian noise
External generalization	360 cropped GTSDB signs	Accuracy, macro-F1, weighted-F1, domain drop, and class-support analysis
Efficiency	Same execution environment	Parameters, model size, training time, inference time, FPS, peak memory
Visual diagnostics	Representative Seed-42 checkpoints	Grad-CAM panels and center-concentration statistics

summarizes these evaluation dimensions, together with the corresponding datasets or experimental settings and the metrics reported for each analysis.

3.9. Training Configuration

All models were trained in the Kaggle notebook environment using PyTorch 2.10.0 with GPU acceleration. The main models were trained for a maximum of 25 epochs with early stopping patience of 4 epochs based on validation performance. The knowledge-distilled MobileNetV2 was trained for a maximum of 15 epochs with a patience of 4 epochs. The no-augmentation ablation models were trained for a maximum of 12 epochs with patience of 3 epochs. Adam was used as the optimizer with a learning rate of $1\times {10}^{-3}$ and weight decay of $1\times {10}^{-4}$. The batch size was set to 64, and mixed-precision training was used to reduce training time and memory usage. Pretrained ImageNet weights were not used so that all architectures were evaluated under the same training-from-scratch setting.

Table 4. Training configuration used in the experiments.

Setting	BaselineCNN	ResNet18	MobileNetV2	MobileNetV2 + CBAM	MobileNetV2-KD
Input size	$128\times 128$	$128\times 128$	$128\times 128$	$128\times 128$	$128\times 128$
Center crop	90	90	90	90	90
Number of classes	43	43	43	43	43
Batch size	64	64	64	64	64
Optimizer	Adam	Adam	Adam	Adam	Adam
Learning rate	$1\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$
Weight decay	$1\times {10}^{-4}$	$1\times {10}^{-4}$	$1\times {10}^{-4}$	$1\times {10}^{-4}$	$1\times {10}^{-4}$
Maximum epochs	25	25	25	25	15
Early stopping patience	4	4	4	4	4
Pretrained weights	No	No	No	No	No
Mixed precision	Yes	Yes	Yes	Yes	Yes
Knowledge distillation	No	Teacher	No	No	Student

displays the training configurations used in this study.

4. Results

Unless explicitly identified as a representative qualitative analysis, the principal quantitative results are reported as mean ± sample standard deviation over five independent training seeds. Seed 42 is used for detailed confusion matrices, reliability diagrams, Grad-CAM panels, and illustrative failure cases. Bootstrap confidence intervals quantify held-out test sample uncertainty for the representative checkpoint, whereas variability across the five training runs is reported separately and is used as the primary measure of training stability.

4.1. Clean GTSRB Classification Performance and Statistical Reliability

Table 5. Clean GTSRB classification performance across five independent training seeds. Values are reported as mean ± sample standard deviation. Bold values indicate the best performing model for the particular metric.

Model	Accuracy	Macro-Precision	Macro-Recall	Macro-F1	Weighted-F1
BaselineCNN	$0.8459\pm 0.0165$	$0.8781\pm 0.0135$	$0.8199\pm 0.0195$	$0.8384\pm 0.0189$	$0.8437\pm 0.0177$
ResNet18	$\mathbf{0}.\mathbf{9856}\pm \mathbf{0}.\mathbf{0093}$	$\mathbf{0}.\mathbf{9819}\pm \mathbf{0}.\mathbf{0131}$	$\mathbf{0}.\mathbf{9830}\pm \mathbf{0}.\mathbf{0118}$	$\mathbf{0}.\mathbf{9817}\pm \mathbf{0}.\mathbf{0134}$	$\mathbf{0}.\mathbf{9857}\pm \mathbf{0}.\mathbf{0092}$
MobileNetV2	$0.9813\pm 0.0057$	$0.9795\pm 0.0082$	$0.9767\pm 0.0066$	$0.9773\pm 0.0069$	$0.9813\pm 0.0056$
MobileNetV2 + CBAM	$0.9801\pm 0.0107$	$0.9779\pm 0.0138$	$0.9755\pm 0.0141$	$0.9759\pm 0.0145$	$0.9802\pm 0.0107$
MobileNetV2-KD	$0.9743\pm 0.0088$	$0.9749\pm 0.0076$	$0.9719\pm 0.0099$	$0.9726\pm 0.0087$	$0.9743\pm 0.0089$

reports clean GTSRB classification performance as mean ± sample standard deviation over five independent training seeds. ResNet18 achieved the strongest mean clean performance, with an accuracy of 0.9856 ± 0.0093 and macro-F1 of 0.9817 ± 0.0134. MobileNetV2 was slightly lower but more stable, reaching 0.9813 ± 0.0057 accuracy and 0.9773 ± 0.0069 macro-F1. MobileNetV2 + CBAM and MobileNetV2-KD did not provide consistent mean improvements over standard MobileNetV2 across the five runs, although all four advanced models substantially outperformed BaselineCNN. BaselineCNN achieved 0.8459 ± 0.0165 accuracy and 0.8384 ± 0.0189 macro-F1, confirming that the shallow reference model was insufficient for fine-grained GTSRB classification.

Table 6. Representative McNemar paired tests on Seed-42 clean GTSRB predictions. Here, $n_{01}$ denotes cases misclassified by the first model but correctly classified by the second model, whereas $n_{10}$ denotes the reverse case.

Model Pair	${\mathit{n}}_{01}$	${\mathit{n}}_{10}$	p-Value
BaselineCNN vs. ResNet18	642	16	$8.42\times {10}^{-167}$
BaselineCNN vs. MobileNetV2	610	24	$5.39\times {10}^{-148}$
BaselineCNN vs. MobileNetV2 + CBAM	636	7	$4.83\times {10}^{-178}$
ResNet18 vs. MobileNetV2	17	57	$3.40\times {10}^{-6}$
ResNet18 vs. MobileNetV2 + CBAM	21	18	0.7493
MobileNetV2 vs. MobileNetV2 + CBAM	52	9	$1.80\times {10}^{-8}$
MobileNetV2 vs. MobileNetV2-KD	47	43	0.7520
MobileNetV2 + CBAM vs. MobileNetV2-KD	11	50	$4.59\times {10}^{-7}$

reports representative paired McNemar tests on the Seed-42 clean GTSRB predictions. These tests compare models on exactly the same test images and therefore evaluate whether the observed paired disagreement pattern is statistically meaningful. The BaselineCNN was significantly different from all stronger architectures, confirming that its lower clean performance is not a small numerical fluctuation. Differences among the stronger architectures were smaller and should be interpreted together with the five-seed mean and standard-deviation results.

These results refine the clean-performance interpretation. Rather than simply stating that a deeper model performs best, the evidence shows that ResNet18 and MobileNetV2 + CBAM form the strongest clean performance group, that standard MobileNetV2 remains a competitive compact model, and that statistical testing is required before interpreting small numerical differences as meaningful.

4.2. Calibration Reliability and Confidence Quality

To evaluate whether the models are not only accurate but also reliable in their confidence estimates, calibration metrics were computed on the clean GTSRB test set.

Table 7. Calibration results on the clean GTSRB test set.

Model	ECE	NLL	Brier Score
BaselineCNN	0.0852	0.5376	0.2453
ResNet18	0.0017	0.0154	0.0084
MobileNetV2	0.0054	0.0575	0.0261
MobileNetV2 + CBAM	0.0025	0.0182	0.0079
MobileNetV2-KD	0.0104	0.0629	0.0263

reports expected calibration error (ECE), negative log-likelihood (NLL), and Brier score. Lower values indicate better probabilistic reliability.

The calibration results show that ResNet18 and MobileNetV2 + CBAM are not only highly accurate but also well calibrated. ResNet18 achieved the lowest ECE of 0.0017, while MobileNetV2 + CBAM achieved the lowest Brier score of 0.0079. BaselineCNN had the weakest calibration behavior, with an ECE of 0.0852 and Brier score of 0.2453. The knowledge-distilled MobileNetV2 did not improve calibration compared with standard MobileNetV2 in this run, suggesting that distillation should not be assumed to automatically improve confidence reliability.

4.3. Severity-Wise Robustness Under Visual Degradations

The robustness analysis evaluates model behavior under four controlled degradation types: blur, occlusion, low-light, and Gaussian noise. Each degradation was applied at three severity levels, and macro-F1 was used as the main robustness indicator because it is more sensitive to class imbalance than accuracy. The full severity-wise robustness curves are shown in Figure 2.

Table 8. Macro-F1 under the highest degradation severity level.

Model	Blur S3	Occlusion S3	Low-Light S3	Gaussian Noise S3
BaselineCNN	0.3490	0.1093	0.4150	0.5010
ResNet18	0.4944	0.0506	0.8710	0.8309
MobileNetV2	0.3052	0.0499	0.9138	0.7747
MobileNetV2 + CBAM	0.4356	0.0349	0.9507	0.7230
MobileNetV2-KD	0.3852	0.0441	0.8435	0.8044

summarizes the macro-F1 scores at the most severe corruption level.

The results show that occlusion is the most damaging degradation for all models. At severity level 3, all macro-F1 scores fall below 0.11, and the advanced architectures do not maintain their clean-set advantage. This finding indicates that partial information loss remains a critical weakness for traffic sign classification and cannot be solved by clean-set training alone. Blur also causes substantial degradation, although ResNet18 remains the most robust model under severe blur with a macro-F1 of 0.4944. Low-light is comparatively less harmful for the stronger architectures, with MobileNetV2 + CBAM maintaining the highest macro-F1 of 0.9507 at severity level 3. Gaussian noise produces an intermediate robustness challenge, where ResNet18 and MobileNetV2-KD retain the strongest performance.

These results provide a more detailed interpretation than the earlier single-level robustness analysis. The practical implication is that robustness strategies should be degradation-specific: occlusion requires stronger occlusion-aware training or detection-before-classification strategies, blur requires blur augmentation or restoration-based preprocessing, low-light degradation can be mitigated through brightness and contrast augmentation, and Gaussian noise can be addressed through noise augmentation or denoising mechanisms.

4.4. Effect of CBAM and Knowledge Distillation

The five-seed results indicate that CBAM and knowledge distillation should be interpreted as condition-dependent modifications rather than uniformly superior enhancements. Across the five principal runs, MobileNetV2 + CBAM did not improve mean clean macro-F1 over standard MobileNetV2. However, the no-augmentation ablation in

Table 9. CBAM ablation without data augmentation. The final row reports the absolute difference between MobileNetV2 + CBAM and MobileNetV2 under the same no-augmentation setting.

Variant	Clean Acc.	Clean Macro-F1	Blur S2	Occlusion S2	Low-Light S2
MobileNetV2, no augmentation, no CBAM	0.9561	0.9369	0.6267	0.0666	0.9181
MobileNetV2, no augmentation, with CBAM	0.9712	0.9584	0.7038	0.0762	0.9052
Difference	+0.0151	+0.0215	+0.0771	+0.0096	−0.0129

shows whether CBAM changes performance when augmentation is removed from the comparison. Knowledge-distilled MobileNetV2 remained compact and training efficient, but it did not consistently outperform standard MobileNetV2 across clean performance, calibration, external transfer, and Grad-CAM diagnostics.

To separate the effect of CBAM from data augmentation, an additional no-augmentation ablation was conducted. As shown in Table 9, CBAM improved no-augmentation MobileNetV2 clean accuracy from 0.9561 to 0.9712 and macro-F1 from 0.9369 to 0.9584. Under severity-2 blur, CBAM also improved macro-F1 from 0.6267 to 0.7038. However, under low-light severity 2, the CBAM variant slightly decreased macro-F1 from 0.9181 to 0.9052, and under occlusion severity 2 both variants remained weak.

These ablation results directly clarify the role of CBAM. CBAM improves clean classification and blur robustness in this setting, provides only marginal benefit under occlusion, and does not consistently improve low-light robustness. Therefore, CBAM should be interpreted as a selective feature-refinement mechanism rather than as a universal robustness solution. Knowledge distillation also provides compact model enhancement, but the results show that the distilled MobileNetV2 does not consistently outperform standard MobileNetV2 or MobileNetV2 + CBAM across all criteria.

4.5. External Generalization on GTSDB

To evaluate whether the models generalize beyond GTSRB, the trained models were tested on cropped traffic sign samples extracted from GTSDB. This external evaluation used 360 cropped sign samples. The results are shown in Figure 3 and in

Table 10. Representative Seed 42 external GTSDB generalization and macro-F1 domain drop.

Model	GTSDB Accuracy	GTSDB Macro-F1	GTSRB Clean Macro-F1	Macro-F1 Domain Drop
BaselineCNN	0.7333	0.6305	0.8240	0.1936
ResNet18	0.9778	0.9389	0.9949	0.0560
MobileNetV2	0.9583	0.9080	0.9820	0.0741
MobileNetV2 + CBAM	0.9167	0.8344	0.9949	0.1605
MobileNetV2-KD	0.9250	0.8269	0.9868	0.1599

.

ResNet18 achieved the strongest external generalization, with 0.9778 accuracy and 0.9389 macro-F1 on GTSDB. MobileNetV2 also generalized well, reaching 0.9583 accuracy and 0.9080 macro-F1. In contrast, MobileNetV2 + CBAM showed a larger domain drop, decreasing from 0.9949 macro-F1 on clean GTSRB to 0.8344 macro-F1 on GTSDB. This result is important because it shows that CBAM improves in-domain GTSRB performance but does not necessarily improve external transfer. MobileNetV2-KD also showed a relatively large domain drop, indicating that the distillation configuration did not fully close the generalization gap.

Therefore, the external validation changes the interpretation of the clean-set results. Although ResNet18 and MobileNetV2 + CBAM were nearly tied on clean GTSRB performance, ResNet18 was more stable under external GTSDB evaluation. This suggests that the best model for clean benchmark classification is not necessarily the best model for cross-dataset generalization.

4.6. Computational Efficiency and Model Complexity

The computational-efficiency analysis evaluates parameter count, model size, training time, inference time, frames per second, and peak GPU memory. The results are summarized in

Table 11. Computational efficiency profile of the evaluated models.

Model	Parameters	Size (MB)	Epochs	Training Time (s)	Inference (s/Image)	FPS
BaselineCNN	319,979	1.22	25	1798.14	0.000366	2735.34
ResNet18	11,198,571	42.76	25	1682.78	0.002059	485.61
MobileNetV2	2,278,955	8.82	21	1534.61	0.005068	197.32
MobileNetV2 + CBAM	2,483,853	9.61	25	1644.16	0.005412	184.77
MobileNetV2-KD	2,278,955	8.82	15	1011.97	0.004881	204.87

and Figure 4.

BaselineCNN was the fastest and smallest model, but this efficiency was achieved at the cost of substantially weaker predictive performance. ResNet18 had the largest parameter count and model size, but it remained faster at inference than the MobileNetV2 variants in the measured environment. This confirms that nominally lightweight architecture does not always translate into the lowest observed latency under a particular software and hardware configuration. MobileNetV2-KD required the fewest training epochs among the advanced models and had the lowest total training time, which supports its value as a compact-model alternative. However, its predictive and external-generalization results did not consistently exceed those of standard MobileNetV2.

The efficiency results therefore support a practical model selection interpretation. ResNet18 is the strongest overall option when clean performance, calibration, external generalization, and measured inference speed are considered together. MobileNetV2 remains a useful compact baseline, MobileNetV2 + CBAM is most attractive when in-domain clean performance and low-light robustness are prioritized, and MobileNetV2-KD is useful when shorter training and compactness are important.

4.7. Grad-CAM-Based Interpretability and Explanation Concentration

Grad-CAM was used to examine whether the models focused on visually meaningful traffic sign regions during prediction. In the analysis, Grad-CAM is not treated only as a qualitative visualization tool. Instead, representative heatmaps are complemented by a quantitative center-concentration score that estimates how strongly the explanation is concentrated around the central sign region. The representative Grad-CAM examples are shown in Figure 5 and Figure 6, and the quantitative summary is given in

Table 12. Grad-CAM center concentration diagnostics for the representative Seed 42 checkpoints. Pearson and Spearman coefficients measure the association between center concentration and prediction confidence. ${\Delta }_{\mathrm{C}-\mathrm{I}}$ denotes the mean concentration for correct predictions minus that for incorrect predictions.

Model	Mean	SD	Pearson r	Spearman $\mathit{\rho }$	Correct	Incorrect	${\mathbf{\Delta }}_{\mathbf{C}-\mathbf{I}}$
ResNet18	0.6110	0.0248	$-0.2106$	$-0.1597$	0.6104	0.6338	$-0.0235$
MobileNetV2	0.5868	0.0339	$-0.0469$	$-0.1733$	0.5866	0.5946	$-0.0080$
MobileNetV2 + CBAM	0.5483	0.0564	0.0435	$-0.0400$	0.5492	0.5135	0.0357
MobileNetV2-KD	0.4694	0.0281	0.1291	0.3888	0.4694	0.4710	$-0.0016$

.

Table 12 shows that center concentration was not consistently associated with either prediction confidence or correctness. ResNet18 produced the highest mean concentration, but its Pearson and Spearman correlations with confidence were both negative. Moreover, incorrect ResNet18 predictions showed greater mean concentration than correct predictions. MobileNetV2 exhibited a similarly weak negative relationship, while MobileNetV2 + CBAM showed only a small positive Pearson correlation and a near-zero negative Spearman correlation. MobileNetV2-KD produced the strongest positive rank correlation, although its mean concentration was the lowest among the evaluated architectures and its correct–incorrect difference was negligible. These results indicate that stronger center concentration should not be interpreted as evidence of greater predictive reliability or explanation quality. The score is therefore retained only as a descriptive measure of spatial activation distribution.

To further examine whether spatial concentration is associated with model confidence, Figure 7 presents the relationship between the Grad-CAM center concentration score and prediction confidence for the representative Seed 42 checkpoints. The observed associations are weak and model dependent. ResNet18 and MobileNetV2 exhibit negative Pearson and Spearman relationships, indicating that increasingly central activation does not necessarily correspond to greater confidence. MobileNetV2 + CBAM shows only a weak positive Pearson association and an approximately negligible rank-based relationship, whereas MobileNetV2-KD exhibits a more noticeable positive Spearman correlation but a substantially weaker Pearson correlation. These inconsistencies indicate that center concentration is not a reliable general surrogate for prediction confidence. Accordingly, the measure is interpreted only as a descriptive indicator of spatial attention distribution rather than as a validated measure of explanation quality or predictive reliability.

The Grad-CAM concentration results show that ResNet18 produced the most concentrated explanations on average, followed by MobileNetV2 + CBAM and MobileNetV2. The CBAM variant showed a modest increase in explanation concentration relative to standard MobileNetV2, which supports the interpretation that CBAM can refine visual focus. However, the improvement is moderate rather than decisive, and the external GTSDB results show that stronger in-domain attention concentration does not automatically imply better cross-dataset generalization. MobileNetV2-KD produced the lowest mean concentration score among the analyzed advanced models, suggesting that the distillation setup used in this study did not improve Grad-CAM localization.

The qualitative examples further show that correct predictions are generally associated with focused activation around the sign region, while errors and difficult cases often show more diffuse or misplaced attention. These findings support the use of Grad-CAM as a diagnostic tool for identifying hidden visual-reasoning failures. At the same time, the results also show that Grad-CAM should not be interpreted as a complete proof of model reliability; it is most useful when combined with quantitative performance, robustness, calibration, and external generalization evidence. Overall, the qualitative cases in Figure 5 and Figure 6, the summary statistics in Table 12, and the correlation analysis in Figure 7 jointly show that visually concentrated activation should not be interpreted as evidence of correct, confident, or faithful model reasoning.

5. Discussion

The results show that traffic sign classifier selection depends strongly on the evaluation dimension. ResNet18 achieved the strongest overall balance in this study because it combined high clean GTSRB performance, strong calibration, good external GTSDB transfer, and competitive measured inference speed. However, no single model dominated every criterion. MobileNetV2 remained a strong compact classifier, MobileNetV2 + CBAM provided condition-dependent gains, and MobileNetV2-KD offered a compact teacher–student alternative without consistently surpassing the standard MobileNetV2. These findings support the central argument that traffic sign classifiers should not be selected using clean accuracy alone.

5.1. Clean Classification Performance, Statistical Significance, and Calibration

The comparison of BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and knowledge-distilled MobileNetV2 on the clean GTSRB test set showed that deeper and better structured models produced substantially stronger performance than the BaselineCNN. It also shows that residual and lightweight deep architectures are better able to capture fine-grained sign structures than a shallow convolutional baseline. However, the analysis provides a more careful interpretation because clean performance was supported by bootstrap confidence intervals and McNemar paired statistical tests. The strongest clean-set models were ResNet18 and MobileNetV2 + CBAM. Their performance was very close, and the McNemar test did not show a statistically significant difference between them. This result is important because it prevents overinterpreting small numerical differences in accuracy or macro-F1 score. In contrast, the performance differences between the BaselineCNN and the stronger models were statistically significant, confirming that the baseline architecture was clearly less suitable for fine-grained traffic sign classification in the present setting. MobileNetV2 remained a strong compact alternative, while the knowledge-distilled MobileNetV2 achieved competitive clean performance but did not significantly outperform the standard MobileNetV2. The calibration results further refine the interpretation of model reliability. ResNet18 and MobileNetV2 + CBAM were not only accurate but also well calibrated, meaning that their predicted probabilities were more reliable than those of the BaselineCNN. This is relevant for safety-oriented traffic sign recognition because an overconfident wrong prediction may be more problematic than an uncertain prediction. The BaselineCNN showed weaker calibration, which suggests that shallow models may not only misclassify more frequently but also provide less reliable confidence estimates. The knowledge-distilled MobileNetV2 did not improve calibration over the standard MobileNetV2 in the present run, indicating that distillation should not be assumed to automatically improve probabilistic reliability.

5.2. Robustness Under Visual Degradation

The robustness results show that model performance depends strongly on the type and severity of visual degradation. This directly supports the central argument that clean benchmark accuracy alone is not sufficient for evaluating traffic sign classifiers. The severity-wise analysis revealed that occlusion was the most damaging condition across all models. At the highest occlusion severity, even the stronger architectures suffered severe performance degradation, indicating that partial information loss remains a critical weakness in traffic sign classification. This finding is practically important because real traffic signs may be partially blocked by vehicles, poles, vegetation, road infrastructure, or other environmental objects. Blur also caused substantial degradation, although the models did not fail as severely as under occlusion. ResNet18 was comparatively more stable under severe blur, suggesting that residual feature extraction may help preserve discriminative structure when image sharpness is reduced. However, the performance drop under blur still indicates that motion-related or sensor-related image distortion can strongly affect recognition reliability. This supports the need for blur-aware training, multi-scale representation learning, or image restoration mechanisms in future traffic sign recognition systems.

Low-light degradation was comparatively less harmful for the stronger architectures. MobileNetV2 + CBAM maintained particularly strong performance under severe low-light conditions, suggesting that attention-based feature refinement can help the model preserve relevant sign structures when illumination is reduced. However, this benefit should be interpreted carefully because CBAM did not improve every robustness condition. Gaussian noise produced an intermediate challenge, whereas ResNet18 and MobileNetV2-KD retained comparatively strong performance. These results show that robustness should be discussed separately for each degradation type rather than reported as a single average score. From a practical perspective, the results suggest different mitigation strategies for different failure modes. Occlusion may require occlusion-aware augmentation, random erasing, detection-before-classification pipelines, or temporal information from video sequences. Blur may be addressed through blur augmentation, deblurring preprocessing, or multi-scale features. Low-light conditions may be improved through brightness and contrast augmentation, while Gaussian noise may require noise augmentation or denoising-based preprocessing. Therefore, robustness improvement should be degradation specific rather than treated as a general property of a model.

5.3. Role of CBAM and Knowledge Distillation

The experiments indicate that CBAM provides condition-dependent effects rather than a uniform improvement. The clean GTSRB results showed that MobileNetV2 + CBAM improved over standard MobileNetV2, and the no-augmentation ablation further confirmed that CBAM can improve clean classification and blur robustness even when the effect of data augmentation is separated. This supports the view that channel and spatial attention can help lightweight models focus on more informative visual regions. However, the effect of CBAM was condition dependent rather than universally beneficial. The ablation results showed that CBAM provided only a marginal benefit under occlusion and did not consistently improve low-light robustness. More importantly, the external GTSDB evaluation showed that MobileNetV2 + CBAM suffered a larger domain drop than ResNet18 and standard MobileNetV2. This indicates that stronger in-domain attention focusing does not automatically guarantee stronger external transfer. Therefore, CBAM should be interpreted as a useful feature-refinement mechanism for certain conditions, especially clean GTSRB classification and blur robustness, but not as a complete solution for robustness or generalization. Knowledge distillation was included as a compact-model enhancement strategy. The distilled MobileNetV2 achieved competitive clean performance and required fewer training epochs than some of the other advanced models. This suggests that distillation can be useful when compactness and training efficiency are important. However, the distilled model did not consistently outperform standard MobileNetV2 across calibration, external generalization, and interpretability metrics. Therefore, in the present study, knowledge distillation is best understood as a promising but not uniformly superior compact model strategy. Future work may explore different teacher–student combinations, distillation temperatures, loss weightings, and robustness-aware distillation objectives.

5.4. External Generalization from GTSRB to GTSDB

The external GTSDB evaluation tests whether strong GTSRB performance transfers to a related but distinct traffic sign dataset. The results showed that ResNet18 achieved the strongest external generalization, followed by standard MobileNetV2. This finding changes the interpretation of the clean-set results: although ResNet18 and MobileNetV2 + CBAM were nearly tied on clean GTSRB performance, ResNet18 generalized more reliably to GTSDB. The larger domain drop observed for MobileNetV2 + CBAM and MobileNetV2-KD suggests that model enhancements that improve in-domain performance may not always improve cross-dataset robustness. This is an important practical finding because traffic sign recognition systems are expected to operate across changing camera conditions, sign appearances, backgrounds, and acquisition settings. A model that learns dataset-specific visual cues may perform strongly on one benchmark while losing performance on another dataset. Therefore, external validation should be treated as an essential component of traffic sign classifier evaluation rather than an optional extension. At the same time, the external evaluation remains limited because the GTSDB experiment used cropped traffic sign samples rather than a full detection-and-classification pipeline. Therefore, the results should be interpreted as evidence of cross-dataset classification transfer, not as complete real-world deployment validation. Nevertheless, the GTSDB analysis provides stronger evidence than a GTSRB-only evaluation and helps identify which models are more stable beyond the original benchmark.

5.5. Computational Efficiency and Practical Model Selection

The relationship between prediction quality and computational efficiency is not as simple as separating models into “deep” and “lightweight” categories. The BaselineCNN achieved the fastest inference and smallest model size, but its weaker accuracy, macro-F1 score, calibration, and robustness limit its suitability when recognition reliability is the main requirement. Therefore, while the baseline model is computationally attractive, it is not the strongest practical choice for reliable traffic sign classification. ResNet18 had the largest parameter count and model size, but it provided the strongest overall balance across clean performance, calibration, external generalization, and measured inference speed in the present environment. This is an important finding because it shows that a nominally larger architecture may still be practically competitive when evaluated using actual runtime measurements. MobileNetV2 and MobileNetV2 + CBAM remained valuable compact alternatives, but their measured inference times were not lower than ResNet18 in this implementation. This confirms that architectural compactness does not always translate directly into lower observed latency, since runtime can depend on hardware, library optimization, batch configuration, and implementation details. MobileNetV2-KD offered a useful training efficiency advantage because it required fewer epochs and lower total training time than several other advanced models. However, its predictive and generalization performance did not consistently exceed standard MobileNetV2. This suggests that model selection should be based on the intended deployment priority. If clean accuracy, calibration, and external generalization are most important, ResNet18 is the strongest option in this study. If compactness and in-domain performance are prioritized, MobileNetV2 + CBAM is attractive. If shorter training time and compact architecture are more important, MobileNetV2-KD may be useful.

5.6. Grad-CAM Interpretability and Visual Attention Behavior

Grad-CAM analysis provided useful insight into how the evaluated models arrived at correct and incorrect predictions. The analysis strengthens this interpretation by adding a quantitative center concentration score, which provides an approximate measure of how strongly the Grad-CAM activation is concentrated around the central sign region. The results showed that ResNet18 produced the highest mean Grad-CAM concentration, followed by standard MobileNetV2 and MobileNetV2 + CBAM. This supports the interpretation that ResNet18 not only performed strongly in classification and external generalization but also produced comparatively focused visual explanations. The CBAM variant showed comparable results in explanation concentration compared with standard MobileNetV2, suggesting that attention refinement can be comparable for visual focus to some extent. The qualitative Grad-CAM examples further showed that correct predictions were generally associated with focused activation on the traffic sign region, including sign borders, symbols, and central iconography. In contrast, incorrect predictions and failure cases were more likely to show diffuse or misplaced attention on background objects, poles, sky regions, or surrounding structures. This confirms that accuracy scores alone may hide important visual-reasoning failures. At the same time, Grad-CAM should not be treated as complete proof of interpretability or causal reasoning. It is more reliable when used as a diagnostic tool alongside performance, robustness, calibration, and external validation results.

5.7. Significance of This Study

The significance of this study lies in its numerical evaluation and methodological framing. Rather than presenting a new backbone architecture, the study provides a reliability-oriented evaluation framework for traffic sign classification. This framework compares baseline, residual, lightweight, attention-enhanced, and knowledge-distilled models across multiple criteria that are relevant for practical use. These criteria include clean predictive performance, statistical significance, calibration reliability, robustness under controlled degradation severity, external generalization, computational efficiency, and Grad-CAM based explanation behavior. The study therefore contributes to the traffic sign recognition literature by showing that model evaluation should not be based on clean benchmark accuracy alone. The findings demonstrate that different models can be preferred under different priorities. ResNet18 is the strongest overall model when clean performance, calibration, external generalization, and measured inference speed are considered together. MobileNetV2 remains a competitive compact model. MobileNetV2 + CBAM improves clean and some robustness results but is not universally superior, especially under external transfer. MobileNetV2-KD offers a compact and training-efficient alternative, although its benefits are not consistent across all metrics. Together, these findings provide practical guidance for researchers and engineers developing traffic sign classification systems. The study highlights that robust traffic sign recognition requires a balanced evaluation of accuracy, confidence reliability, degradation sensitivity, transferability, computational cost, and visual reasoning. In this sense, the contribution of the study is not only the comparison of several models but also the demonstration of a more complete evaluation protocol for reliable and interpretable traffic sign classification.

5.8. Limitations and Future Directions

Several limitations remain. First, although five independent training seeds quantify variability due to stochastic training, five runs provide limited power for seed-level non-parametric significance tests. The reported mean, standard deviation, paired differences, and seed-wise win counts should therefore be interpreted together rather than relying only on binary significance thresholds. Second, the external evaluation contains 360 cropped GTSDB signs covering 38 of the 43 target classes. Several represented classes have very low support, and the experiment does not evaluate a complete scene-level detection-and-classification system. The external metrics therefore provide limited cross-dataset evidence rather than comprehensive real-world validation. Third, the corruption experiments use controlled synthetic degradations. Central occlusion intentionally represents a targeted worst-case condition, while random occlusion provides a less targeted comparison; neither protocol fully captures natural obstruction by vehicles, vegetation, poles, weather, or complex backgrounds. Fourth, the knowledge distillation analysis considers one teacher–student architecture and one prespecified main hyperparameter configuration. The sensitivity analysis is local and does not establish a globally optimal temperature or loss weighting. Distillation also did not provide a consistent calibration advantage across seeds. Fifth, Grad-CAM center concentration is a descriptive spatial measure and was not validated against human expert judgments or perturbation-based faithfulness tests. It should not be interpreted as a direct measure of explanation quality. Finally, computational measurements depend on the software and hardware environment and may not transfer directly to embedded processors, automotive accelerators, or edge devices. Future work should include additional geographic datasets, naturally degraded traffic scenes, hardware-specific profiling, broader distillation and attention configurations, and human- or perturbation-based explanation validation.

6. Conclusions

This study presented a reliability-oriented comparative evaluation of BaselineCNN, ResNet18, MobileNetV2, MobileNetV2 + CBAM, and knowledge-distilled MobileNetV2 for traffic sign classification. The models were evaluated on GTSRB, with additional external validation using cropped traffic sign samples from GTSDB. Unlike a conventional clean-accuracy comparison, the proposed framework considered statistical significance, calibration reliability, severity-wise robustness, external generalization, computational efficiency, and Grad-CAM based interpretability. The results showed that ResNet18 and MobileNetV2 + CBAM achieved the strongest clean GTSRB performance, while their difference was not statistically significant. ResNet18 also provided the strongest external generalization to GTSDB and showed highly reliable calibration. MobileNetV2 remained a competitive compact model, whereas BaselineCNN was weaker in both predictive performance and confidence reliability. The robustness analysis showed that occlusion was the most damaging degradation across all models, followed by blur, while low-light degradation was comparatively more manageable for the stronger architectures. These findings confirm that robustness should be evaluated separately for different degradation types rather than treated as a single general model property. The CBAM and knowledge-distillation results provided a more balanced view of model enhancement strategies. CBAM improved clean performance and some robustness behavior, particularly under blur, but its benefits were condition-dependent and did not automatically improve external transfer. Knowledge-distilled MobileNetV2 offered a compact alternative, but it did not consistently outperform standard MobileNetV2 across all evaluation dimensions. Grad-CAM analysis showed that correct predictions were generally associated with focused activation on traffic sign regions, while failure cases often involved diffuse or misplaced attention toward background structures. Overall, the study demonstrates that traffic sign classifiers should be evaluated using a multi-dimensional reliability framework that jointly considers accuracy, statistical evidence, calibration, robustness, generalization, efficiency, and interpretability. Future work should extend this framework to additional traffic sign datasets, naturally degraded images, more diverse weather and viewpoint conditions, edge-device deployment, and stronger quantitative explanation metrics.