RPF-MAD: A Robust Pre-Training–Fine-Tuning Algorithm for Meta-Adversarial Defense on the Traffic Sign Classification System of Autonomous Driving

Abstract

Traffic sign classification (TSC) based on deep neural networks (DNNs) plays a crucial role in the perception subsystem of autonomous driving systems (ADSs). However, studies reveal that the TSC system can make dangerous and potentially fatal errors under adversarial attacks. Existing defense strategies, such as adversarial training (AT), have demonstrated effectiveness but struggle to generalize across diverse attack scenarios. Recent advancements in self-supervised learning (SSL), particularly adversarial contrastive learning (ACL) methods, have demonstrated strong potential in enhancing robustness and generalization compared to AT. However, conventional ACL methods lack mechanisms to ensure effective defense transferability across different learning stages. To address this, we propose a robust pre-training–fine-tuning algorithm for meta-adversarial defense (RPF-MAD), designed to enhance the sustainability of adversarial robustness throughout the learning pipeline. Dual-track meta-adversarial pre-training (Dual-MAP) integrates meta-learning with ACL methods, which improves the generalization ability of the upstream model to different adversarial conditions. Meanwhile, adaptive variance anchoring robust fine-tuning (AVA-RFT) utilizes adaptive prototype variance regularization to stabilize feature representations and reinforce the generalizable defense capabilities of the downstream model. Leveraging the meta-adversarial defense benchmark (MAD) dataset, RPF-MAD ensures comprehensive robustness against multiple attack types. Extensive experiments across eight ACL methods and three robust fine-tuning (RFT) techniques demonstrate that RPF-MAD significantly improves both standard accuracy ($SA$) by 1.53% and robust accuracy ($RA$) by 2.64%, effectively enhances the lifelong adversarial resilience of TSC models, achieves a 13.77% improvement in the equilibrium defense success rate ($EDSR$), and reduces the attack success rate ($ASR$) by 9.74%, outperforming state-of-the-art (SOTA) defense methods.

1. Introduction

With the growing maturity of autonomous driving technology, ensuring the robustness of ADS has become a key research focus [1,2]. As a core multimodal interface, the TSC module performs hierarchical visual–semantic parsing via DNNs to interpret regulatory instructions and guide real-time driving policies [1,2,3,4]. Enhanced by adaptive feature disentanglement mechanisms [5], TSC balances robustness and decision granularity. However, numerous studies have demonstrated that subtle adversarial examples (AEs) [6] can mislead TSC models, posing potentially fatal risks [7,8,9]. This challenge has motivated researchers to prioritize the development of robust and effective defense mechanisms for TSC systems.

Pavlitska et al. [10] offered a comprehensive overview of attacks on TSC systems, while highlighting a significant research gap in adversarial defense strategies. Our analysis of recent defense approaches reveals that most of them focus on input transformation [11,12,13,14,15,16,17] and detection-based methods [18,19,20]. For digital defenses, Wu et al. [13] utilized 5G capabilities and mobile edge computing with singular value decomposition to mitigate perturbations from traditional attacks such as CW [21], DeepFool [22], and I-FGSM [23]. Inspired by attention mechanisms, Li et al. [14] employed a CNN to extract affine coordinates and filter pixels to counter various attacks. Salek et al. [15] explored the use of GANs for AE denoising, though it suffers from poor transferability. Suri et al. [16] proposed a bit-plane isolation system, utilizing a voting mechanism across models for defense, though it demands significant resources. Huang et al. [20] used a deep image prior combined with U-Net for image reconstruction, employing a multi-stage decision-making mechanism to defend against physical AEs targeting traffic signs. Collectively, these studies indicate that current defense strategies are predominantly inclined to image reconstruction, neglecting the model’s inherent robustness. In contrast, AT [24] has emerged as a widely used strategy for directly enhancing model robustness. Well-designed AT schemes have demonstrated considerable success in balancing robustness and accuracy [25,26,27,28]. Therefore, a comprehensive TSC system should achieve intrinsic robustness and strong generalization through AT, while maintaining minimal resource consumption.

Inspired by the cost-effective success of parameter-efficient fine-tuning [29] in large-scale models such as GPT-3 [30,31]. The robust pre-training–fine-tuning paradigm [32] has garnered increasing attention, which consists of two stages, namely, robust pre-training (RPT) and RFT. Within this context, recent innovations in AT-based RPT techniques, such as ACL methods [33,34,35,36], have made notable contributions to robust representation learning [37,38]. Correspondingly, the following three representative RFT strategies have emerged: standard linear fine-tuning (SLF), adversarial linear fine-tuning (ALF), and adversarial full fine-tuning (AFF). These techniques have been successfully leveraged in TSC systems, yielding outstanding performance in both standard and adversarial settings [39].

Nevertheless, challenges remain, particularly with the rise of upstream attacks [40,41], where AEs generated during pre-training compromise the robustness of downstream models. To provide a more holistic evaluation of defense capabilities in TSC systems, we categorize traditional attacks such as the fast gradient sign method (FGSM) [6] and projected gradient descent (PGD) [24] as downstream attacks. Additionally, our previous work, MAD [42], introduced a benchmark dataset specifically designed to assess robustness against unknown attacks, further broadening the adversarial threat landscape considered in this study.

Motivation. Through a review of the existing defense methods against these diverse and complex attacks, we identify the following drawbacks in current research: (1) SSL-based defense benchmarks tailored to upstream attacks, such as genetic evolution-nurtured adversarial fine-tuning (Gen-AF) [39], suffer drastic performance drops when deployed on downstream tasks or unseen attack scenarios. (2) ACL methods remain vulnerable to unknown attacks, revealing limitations in robustness-oriented encoder pre-training, as shown in Figure 1. (3) Current robust pre-training–fine-tuning paradigm fails to address comprehensive security-performance trade-offs in TSC systems.

To address these limitations, we propose RPF-MAD, a robust pre-training–fine-tuning method aimed at providing comprehensive defense for TSC systems, with a focus on mitigating fundamental digital attacks. Following the widely adopted and efficient two-stage paradigm in Robust-SSL [43], we introduce targeted innovations in both the upstream and downstream stages. In the RPT phase, we proposed a Dual-MAP algorithm, which optimizes ACL strategies to reinforce SSL-based defenses. In the RFT stage, we proposed AVA-RFT, designed as an implicit manifold alignment mechanism to preserve feature space integrity and enhance the inheritance of meta-generalized defense knowledge from the upstream model. Furthermore, by integrating meta-adversarial learning during pre-training and enforcing structure-preserving fine-tuning, RPF-MAD is inherently designed to improve adaptability and resilience against future novel adversarial attacks, thereby ensuring the sustainable robustness of TSC systems even in the presence of emerging threats. The detailed framework for RPF-MAD is depicted in Figure 2. Experimental results demonstrate that the RPF-MAD optimized TSC model achieves comprehensive outstanding performance across upstream, downstream, and unknown attacks, with an average improvement of 1.53% in $SA$, 2.64% in $AA$, and 13.77% in equilibrium defense success rate ($EDSR$), significantly outperforming existing methods. In summary, the specific contributions of this study are as follows:

• We propose RPF-MAD as a comprehensive SSL-based defense algorithm for TSC systems, which is capable of defending against upstream, downstream, and unknown attacks.
• We introduce the Dual-MAP algorithm, which empowers upstream models to maintain inherent robustness while ensuring generalized defense capabilities.
• We propose the AVA-RFT algorithm, which preserves the structural integrity of the feature space, ensuring effective transfer of meta-learned robustness to downstream models. Extensive experiments validate the sustainability of the proposed defense mechanism and its adaptability across different attacks.

Figure 2. Overview of RPF-MAD and classification of adversarial attacks in the pre-training–fine-tuning paradigm.

Figure 2. Overview of RPF-MAD and classification of adversarial attacks in the pre-training–fine-tuning paradigm.

The remainder of this article is structured as follows: Section 2 reviews related work on adversarial attacks and defense strategies in TSC systems. Section 3 introduces the theoretical foundation. Section 4 details the design of RPF-MAD and key components, while Section 5 presents experimental evaluations and ablation studies. Section 6 discusses broader implications, and Section 7 concludes the study with future research directions.

2. Related Work

The following section begins by reviewing the implementation of adversarial attacks in TSC systems, highlighting their risks and providing a defense-oriented analysis. Next, we present a more detailed examination of AT-based approaches, contrasting them with earlier defense strategies to demonstrate their effectiveness and versatility. Finally, both attack and defense techniques for TSC systems are then explored through the lenses of supervised and self-supervised learning, highlighting the limitations of existing methods.

2.1. Adversarial Attacks in TSC Systems

Attacks on the supervised model. In terms of digital attacks, Papernot et al. [44] improved the white-box FGSM [6] to attack Own-CNN, while Li et al. [45] achieved black-box adversarial attacks on CNN-ST [46] by enhancing the square attack [47] and SimBA [48]. For physical attacks, Eykholt et al. [49] proposed the foundational RP2 attack, inspiring numerous subsequent methods, while Liu et al. [50] utilized GANs to create physical adversarial patches effectively. Overall, digital attacks in TSC systems primarily leverage traditional classification methods, aligning with the core principles of AEs. In contrast, physical attacks focus on crafting adversarial-sensitive patches, often exhibiting visible defects but with practical research significance.

Attacks on the self-supervised model. The rise of SSL and the widespread adoption of the pre-training–fine-tuning paradigm provided by cloud servers have shifted adversarial attacks from task-specific targets to widely used upstream pre-trained models, introducing new challenges to system robustness. A significant advancement in this area is the development of downstream-agnostic adversarial examples (DAEs) [40], designed to achieve cross-fine-tuning transferability and compromise downstream tasks. Among the SOTA methods, pre-trained adversarial perturbations (PAPs) [41] and AdvEncoder [40] have emerged as prominent techniques, demonstrating high $ASR$ in downstream classification tasks, particularly on the TSC systems. These sophisticated attacks underscore the urgency of broadening research into defense strategies to fortify upstream models and ensure secure downstream deployments.

2.2. Adversarial Defenses in TSC Systems

Defenses on the supervised model. Defense strategies based on AT for TSC systems have received significantly less attention than adversarial attack research. Aung et al. [51] first explored digital defenses on the GTSRB dataset using FGSM and Jacobian-based saliency map attack (JSMA) [52], showing that combining AT with defensive distillation improves robustness. However, their study lacked focus on the same network and advanced attack validation. Notably, their work indirectly supports our approach, highlighting the effectiveness of offline AEs from MAD in AT. Metzen et al. [53] proposed meta-adversarial training (MAT), integrating AT with meta-learning to dynamically generate adversarial patches, and improving robustness against universal patch attacks. However, MAT requires substantial resources, relies on online patch generation, and does not address generic digital attacks. Our goal is to establish a comprehensive robustness model for TSC systems against digital attacks, laying the groundwork for defenses against physical attacks.

Defenses on the self-supervised model. Renowned for enhancing the robustness of SSL, ACL [33] has evolved into numerous variants, including RoCL [36], AdvCL [34], DynACL [54], and ACL-RCS [55]. Despite these advancements, the impact of upstream attacks and the robustness of SSL-based TSC systems remain underexplored. Frameworks such as Gen-AF have been proposed to defend SSL models against upstream attacks. As shown in

Table 1. Performance comparison between SimCLR (Vanilla), Gen-AF (Baseline), and RPF-MAD (Our) under upstream, downstream, and unknown attacks.

Category	Attack	Metric	SimCLR [37]	Gen-AF [39]	RPF-MAD
Upstream	AdvEncoder [40]	$R{A}_d$ (↑)	15.99	88.58	88.34
			79.64	2.24	1.29
	PAP [41]		0.00	89.81	90.26
			100.00	0.89	−0.86
	SSP [57]	$ASR$ (↓)	0.18	89.05	86.04
			99.77	1.73	3.85
	UAPGD [58]		0.05	89.35	89.29
			99.94	1.40	0.23
Downstream	AutoAttack [56]	$S{A}_d$ (↑)	78.54	90.62	94.70
		$AA$ (↑)	0.00	7.01	78.20
Unknown	2, 10, 11, 27, 28, 29 [42]	$EDSR$ (↑)	2.33	3.69	30.78

Arrows (↑/↓) indicate the preferred direction for each metric. Bold font marks the best performance.

, Gen-AF achieves promising performance under upstream attack scenarios. However, models fine-tuned under such frameworks still exhibit significant vulnerabilities to sophisticated downstream attacks, such as AutoAttack (AA) [56], and to unknown threats. To address these gaps, we propose a holistic defense algorithm, RPF-MAD, which not only achieves comparable performance to Gen-AF under upstream attacks but also substantially improves robustness against downstream and unknown attacks while effectively preserving task-specific performance.

3. Materials and Methods

This section provides the theoretical background and essential methodologies for robust model learning in both supervised and self-supervised modes. We first introduce the formulation of supervised RFT, followed by a detailed explanation of ACL-based RPT and corresponding RFT strategies.

3.1. Supervised Robust Fine-Tuning

Recent advancements in supervised transfer learning have increasingly concentrated on RFT, with notable methods such as TWINS [59] and AutoLoRa [60] paving the way. Additionally, Hua et al. [61] provided a comprehensive survey of various parameter-efficient fine-tuning strategies. The core principle remains consistent: fine-tuning robust pre-trained models with AT for downstream tasks [32]. The context of RFT is shown in (1):

$$\underset{\widehat{\theta }}{min}{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[\underset{{\parallel \delta \parallel }_r\le ϵ}{max}{\mathcal{L}}_{\mathrm{SL}}(x+\delta ,y;\theta \cup \widehat{\theta })\right].$$

(1)

where ${\mathcal{L}}_{\mathrm{SL}}$ represents the loss function of supervised learning (SL), typically chosen as cross-entropy loss. $\theta$ and $\widehat{\theta }$ represent the frozen and tunable parameters, respectively. $(x,y)$ is the training pair sampled from the downstream dataset $\mathcal{D}$, and $\delta$ denotes an adversarial perturbation constrained by its r-norm with a bound of $ϵ$. Normally, adversarial perturbations are generated using the PGD-10 with $ϵ=8/255$, step size $\alpha =2/255$, and $r=\infty$.

3.2. Self-Supervised Robust Pre-training–Fine-tuning

Contrastive learning (CL) is a self-supervised paradigm that learns generalizable feature representations by maximizing positive sample similarity and minimizing negative sample similarity, reducing reliance on labeled data. ACL methods incorporate AEs into CL methods, enhance the robustness of pre-trained models, and facilitate the transferability of adversarial resilience to downstream tasks. Analogous to the role of AT in SL, the fundamental formulation of ACL methods is as follows:

$$\underset{\theta }{min}{\mathbb{E}}_{x\in \mathcal{X}}\left[\underset{{\parallel \delta \parallel }_r\le ϵ}{max}\sum _i\sum _{j\in {\mathcal{D}}_p\left(i\right)}{\mathcal{L}}_{\mathrm{ACL}}\left(x_i,x_j;\theta \right)\right],{\mathcal{D}}_p\left(i\right)\leftarrow {\mathcal{D}}_p\left(i\right)\cup \left\{{\widehat{x}}_i+\delta \right\}.$$

(2)

Here, $\mathcal{X}$ denotes the dataset from which samples x are drawn, while ${\mathcal{D}}_p(·)$ represents the positive sample set. AEs are incorporated into ${\mathcal{D}}_p(·)$, where ${\widehat{x}}_i$ represents a view of $x_i$ used to generate AEs. Variations among ACL methods primarily lie in the choice of contrastive loss (${\mathcal{L}}_{\mathrm{ACL}}$), while other AE-related settings align with vanilla AT.

The pre-training–fine-tuning paradigm in Robust-SL involves loading AT-trained checkpoints derived from large-scale datasets [32], with innovations predominantly concentrated in RFT techniques. In contrast, Robust-SSL emphasizes enhancing downstream robustness through refined RPT, ensuring seamless adversarial resilience transfer while integrating diverse RFT strategies.

SLF trains a linear classifier ($f_c(·)$) atop a frozen pre-trained feature extractor ($g_e(·)$) using clean examples. By freezing the encoder, SLF preserves the pre-trained representations and adheres to SSL principles by restricting gradient updates to the classifier. The objective is to minimize the supervised loss ${\mathcal{L}}_{\mathrm{SL}}$ with respect to the true labels y, as shown below:

$$\mathrm{SLF}:\underset{{\theta }_c}{min}{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left\{{\mathcal{L}}_{\mathrm{SL}}\left[f_c(g_e\left(x\right);{\theta }_c),y\right]\right\},$$

(3)

ALF extends SLF by training $f_c(·)$ on AEs leveraging principles of AT while keeping $g_e(·)$ remains frozen. The optimization objective is defined as follows:

$$\mathrm{ALF}:\underset{{\theta }_c}{min}{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left\{\underset{{\parallel \delta \parallel }_r\le ϵ}{max}{\mathcal{L}}_{\mathrm{SL}}\left[f_c(g_e(x+\delta );{\theta }_c),y\right]\right\},$$

(4)

AFF generalizes ALF by jointly updating both $f_c(·)$ and $g_e(·)$ on AEs. This approach enhances robustness and generalization, making it one of the most effective fine-tuning strategies. The training objective is expressed as follows:

$$\mathrm{AFF}:\underset{{\theta }_c,{\theta }_e}{min}{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left\{\underset{{\parallel \delta \parallel }_r\le ϵ}{max}{\mathcal{L}}_{\mathrm{SL}}\right[f_c(g_e(x+\delta ;{\theta }_e);{\theta }_c),y\left]\right\}.$$

(5)

Standard full fine-tuning (SFF), which updates both the classifier and the feature extractor on clean examples, is excluded due to its inferior performance in robust settings [62].

The use of robust weight initialization derived from SSL models significantly accelerates convergence and enhances model performance [63]. Moreover, robustness evaluation is typically conducted on the complete model $f_c\circ g_e$ under adversarial attacks such as PGD or AA, ensuring a comprehensive assessment of model resilience.

4. Robust Pre-training–Fine-tuning Algorithm for Meta-Adversarial Defense

MAD augments AT by introducing broadly adaptable defensive capabilities while maintaining classification accuracy. However, its potential applicability and effectiveness within the domain of Robust-SSL are underexplored. To address this, we propose the RPF-MAD, as depicted in Figure 2. The approach consists of two stages, as follows: (1) Dual-MAP, which enhances the robustness and defense generalization of the upstream model, and (2) Meta-RFT, which facilitates the effective transfer of upstream robustness to the downstream model, thereby ensuring comprehensive defense against various adversarial attacks.

4.1. Dual-Track Meta-Adversarial Pre-Training Algorithm (Dual-MAP)

Given an upstream MAD dataset ${\mathcal{D}}_m$ and an ACL pre-trained encoder $g_e(·)$, we construct meta-task $\mathcal{T}$ from a batch of AE pairs $\left\{(x^{\prime },x,y)\in {\mathcal{D}}_m\mid x^{\prime }=x+\delta \right\}$. During training, each $\mathcal{T}$ comprises the following:

• Support set (S): $k\times n\times \mathcal{A}$ AEs with k-shot samples from $\mathcal{A}$ attack types across n classes.
• Query set (Q): $m\times n\times {\mathcal{A}}_Q$ AEs generated using ${\mathcal{A}}_Q$ distinct attack types.

The validation/testing phase adopts an “$\mathcal{A}$-way, K-shot” protocol with non-overlapping sets $S^{\prime }$ and $Q^{\prime }$ following equivalent sizing rules. Distinct from the vanilla Meta-AT, Dual-MAP introduces a dual-track optimization strategy. We apply differentiated learning rates to the $f_c(·;{\theta }_c)$ and $g_e(·;{\theta }_e)$, explicitly formulated as follows:

$${\theta }_c\leftarrow {\theta }_c-{\beta }_c{\nabla }_{{\theta }_c}{\mathcal{L}}_{\mathrm{MAP}},{\theta }_e\leftarrow {\theta }_e-{\beta }_e{\nabla }_{{\theta }_e}{\mathcal{L}}_{\mathrm{MAP}}\left({\beta }_c=\lambda {\beta }_e,\lambda <1\right).$$

(6)

where $\lambda$ governs the learning rate decay for the encoder to prevent catastrophic forgetting of generic features. Consequently, the upstream model $f_c\circ g_e$ is optimized to satisfy the following:

$$\underset{\theta }{min}{\mathbb{E}}_{\mathcal{T}\sim \mathcal{P}\left(\mathcal{T}\right)}\left[{\displaystyle \frac{1}{s}}\sum _{j=1}^s\underset{{∥{\delta }_b∥}_r<ϵ}{max}{\mathcal{L}}_{\mathrm{MAP}}(\mathcal{T};\theta )\right].$$

(7)

where $\mathcal{P}\left(\mathcal{T}\right)$ denotes the distribution of tasks, and during the training process, term s denotes the number of episodes, multiple norms ($r\in (0,1,2,\infty )$) of perturbations are learned. Furthermore, the upstream meta-adversarial pre-training loss is the objective function that incorporates task-specific cross-entropy loss with a smoothness-inducing adversarial regularizer (SAR) as below.

$${\mathcal{L}}_{\mathrm{MAP}}\left(\theta \right)={\mathcal{L}}_{\mathrm{SL}}\left(\theta \right)+{\eta }_1{\mathcal{L}}_{\mathrm{SAR}}\left(\theta \right),$$

(8)

${\eta }_1>0$ is a tuning parameter, and SAR is applied during the meta-update phase to effectively control the complexity of the model. These implementations are based on the TRADES framework [64]. We define the ${\mathcal{L}}_{\mathrm{SL}}$ and ${\mathcal{L}}_{\mathrm{SAR}}$ as follows:

$${\mathcal{L}}_{\mathrm{SL}}({\theta }_c,{\theta }_e)={\ell }_{ce}\left[f_c(g_e(x_j+{\delta }_b;{\theta }_e);{\theta }_c),y_j\right],$$

(9)

$${\mathcal{L}}_{\mathrm{SAR}}({\theta }_c,{\theta }_e)={\ell }_k{f}_c(g_e(x_j+{\delta }_b;{\theta }_e);{\theta }_c),y_j],[f_c(g_e(x_j;{\theta }_e);{\theta }_c),y_j\left]\right\}.$$

(10)

Specifically, where ${\ell }_k$ is chosen as the symmetrized KL-divergence. We seek to determine the optimal parameter set ${\theta }^{*}$ of the parameterized function $\mathrm{\Phi }\left(\theta \right)$ at the point of minimal loss, such that the fine-tuned model $\varphi (\theta ,{\mathcal{T}}_a)$ under the ath attack generalizes effectively to a new task ${\mathcal{T}}_b$ under the bth attack. The meta-update process is then expressed as follows:

$$f_c\left(g_e(x;{\theta }^{*})\right)={\mathrm{\Phi }}_{\varphi \left(\theta ,{\mathcal{T}}_a\right)}\left(x\right).$$

(11)

The detailed pre-training phase of Dual-MAP is outlined in Algorithm 1.

Algorithm 1 Dual-MAP (pre-training phase).

Input:  Pre-trained encoder $g_e(·)$, meta-learning classifier $f_c(·)$, training dataset ${\mathcal{D}}_m$, number of attacks $\mathcal{A}$ in S and ${\mathcal{A}}_Q$ in Q, learning rates ${\beta }_e$, ${\beta }_c$, total pre-train epochs $E_P$, number of classes n. Output:  Robustly pre-trained model $\mathrm{\Phi }\left({\theta }^{\prime }\right)$ with meta-knowledge. 1: Initialize parameters ${\theta }_e$ and ${\theta }_c$ randomly; 2: for $epoch=1$ to $E_P$ do 3:     Sample a batch of tasks ${\left\{{\mathcal{T}}_i\right\}}_{i=1}^{\mathcal{A}}\sim \mathcal{P}\left(\mathcal{T}\right)$, where ${\mathcal{T}}_i=\left\{{\left({\mathcal{T}}_i^j\right)}_S,{\left({\mathcal{T}}_{i^{\prime }}^j\right)}_Q\mid j\in [1,s],i^{\prime }\in [1,{\mathcal{A}}_Q]\right\}$; 4:     for each ${\mathcal{T}}_i$ do 5:        Sample $k\times n$ datapoints ${\mathcal{D}}_S={\left\{\left(x_i^j,y_i^j\right)\right\}}_{d=1}^n$ from ${\left({\mathcal{T}}_i^j\right)}_S$; 6:        Compute loss ${\mathcal{L}}_{\mathrm{MAP}}$ on ${\mathcal{D}}_S$ using Equation (8); 7:        Update classifier: ${\theta }_c\leftarrow {\theta }_c-{\beta }_c{\nabla }_{{\theta }_c}{\mathcal{L}}_{\mathrm{MAP}}$; 8:        Update encoder: ${\theta }_e\leftarrow {\theta }_e-{\beta }_e{\nabla }_{{\theta }_e}{\mathcal{L}}_{\mathrm{MAP}}$; 9:        Sample $K\times n$ datapoints ${\mathcal{D}}_Q={\left\{\left(x_{i^{\prime }}^j{,}_{i^{\prime }}^j\right)\right\}}_{d=1}^n$ from ${\left({\mathcal{T}}_{i^{\prime }}^j\right)}_Q$; 10:        Compute meta-update loss ${\mathcal{L}}_{\mathrm{MAP}}$ on ${\mathcal{D}}_Q$ using Equation (8); 11:     end for 12:     Update global parameters: ${\theta }^{\prime }\leftarrow \theta -{\displaystyle \frac{{\beta }_e}{s}}{\sum }_{{\mathcal{T}}_i}{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{MAP}}$. 13: end for

4.2. Adaptive Variance Anchoring Robust Fine-Tuning Algorithm (AVA-RFT)

Traditional fine-tuning distorts the latent space geometry of pre-trained models, compromising adversarial robustness and weakening the rapid adaptation capability of Dual-MAP against unknown attacks. To mitigate this, we propose the AVA-RFT algorithm, which serves as an implicit manifold alignment mechanism. It preserves intra-class compactness and enhances inter-class separability, ensuring consistency between downstream optimization and latent space evolution. AVA-RFT introduces an adaptive variance anchoring (AVA) term ${\mathcal{L}}_{\mathrm{anchor}}$ into the traditional fine-tuning loss function. The overall fine-tuning loss ${\mathcal{L}}_{\mathrm{total}}$ is defined as follows (using AFF as an example):

$${\mathcal{L}}_{\mathrm{total}}=\underset{\mathrm{TRADES}}{\underbrace{{\mathcal{L}}_{\mathrm{SL}}+{\eta }_2·{\mathcal{L}}_{\mathrm{SAR}}}}+\underset{\mathrm{AVA}}{\underbrace{{\eta }_3·{\mathcal{L}}_{\mathrm{anchor}}}}.$$

(12)

AFF follows the TRADES paradigm. The pre-trained encoder $g_e(x;{\theta }^{*})$ maps input data to a high-dimensional implicit differentiable manifold $\mathcal{M}$. The geometric properties of this manifold encode meta-knowledge. The geometric structure of $\mathcal{M}$ encodes meta-knowledge, which AVA leverages to enforce structure-preserving fine-tuning. This process comprises three key components, mathematically formulated as follows:

$${\mathcal{L}}_{\mathrm{anchor}}=\left({\mathcal{V}}_{\mathrm{intra}}-{\mathcal{V}}_{\mathrm{inter}}\right),$$

(13)

Intra-class compactness and local manifold linearization. Minimizing intra-class variance ${\mathcal{V}}_{\mathrm{intra}}$ forces features of the same class to contract toward their class centroid on manifold $\mathcal{M}$. This ensures feature compactness in the local tangent space:

$${\mathcal{V}}_{\mathrm{intra}}={\displaystyle \frac{1}{\parallel \mathcal{B}\parallel }}\sum _c\sum _{x_i\in {\mathcal{B}}_c}{∥G\left(x_i\right)-{\mu }_c∥}^2,$$

(14)

Using the Euclidean distance approximation (L2-norm), this assumption implies that the manifold is locally flat, aligning with the low-curvature property of pre-trained models. Here, $\parallel \mathcal{B}\parallel$ represents the size of the current mini-batch, ${\mathcal{B}}_c$ denotes the subset of $\mathcal{B}$ belonging to class c, $G\left(x\right)$ denote the feature representations, and ${\mu }_c$ denotes the class centroid.

Inter-class separability and global topology preservation. Maximizing inter-class variance ${\mathcal{V}}_{\mathrm{inter}}$ increases the geodesic distance between different class centroids, preventing feature collapse and maintaining global manifold distinguishability:

$${\mathcal{V}}_{\mathrm{inter}}={\displaystyle \frac{2}{C(C-1)}}\sum _{i<j}{∥{\mu }_i-{\mu }_j∥}^2$$

(15)

Here, C is the number of classes, and ${\mu }_i$, ${\mu }_j$ represent different class centroids. This is equivalent to maximizing the deviation of all centroids from the global mean under Euclidean assumptions.

AVA achieves adaptability by computing variance within each mini-batch, ensuring independence from batch size while dynamically adjusting feature distributions to align with local manifold structures. This mechanism prevents bias accumulation in global statistics and enhances feature separation. To validate the effectiveness of AVA, we analyze the dynamic behavior of ${\mathcal{V}}_{\mathrm{intra}}$ and ${\mathcal{V}}_{\mathrm{inter}}$. From a qualitative perspective, the reduction of ${\mathcal{V}}_{\mathrm{intra}}$ encourages intra-class samples to cluster within the tangent space, while the increase of ${\mathcal{V}}_{\mathrm{inter}}$ enhances inter-class separability, thereby optimizing feature distribution on the manifold. This attraction-repulsion mechanism, jointly optimized with the task loss, ensures both compactness and discriminability of feature representations. Furthermore, we quantitatively track the variance changes throughout training, as illustrated in Figure 3. It is evident that ${\mathcal{V}}_{\mathrm{intra}}$ exhibits a relatively decreasing trend, while ${\mathcal{V}}_{\mathrm{inter}}$ gradually increases. Despite some fluctuations, their overall trends align with theoretical expectations. These results demonstrate that AVA effectively regulates feature distribution as a structural constraint, thereby enhancing model robustness and generalization. Overall, the AVA-RFT procedure is outlined in Algorithm 2, taking AFF as an example. To further address unknown attacks, we provide the pseudo-code for the complete model evaluation process at the testing stage in Algorithm 3.

Algorithm 2 AVA-RFT (Fine-tuning phase).

Input:  Pre-trained model $\mathrm{\Phi }\left(\theta \right)$, downstream dataset $D_d$, learning rate ${\beta }_d$, regularization parameters $({\eta }_2,{\eta }_3)$, batch size $\parallel \mathcal{B}\parallel$, total fine-tune epochs $E_F$. Output:  Fine-tuned model $\mathrm{\Psi }\left(\widehat{\theta }\right)$. 1: Initialize optimizer; 2: for $epoch=1$ to $E_F$ do 3:     Shuffle dataset $D_d$ and partition into mini-batches ${\left\{{\mathcal{B}}_t\right\}}_{t=1}^{|D_d|/|\mathcal{B}|}$; 4:     for each mini-batch ${\mathcal{B}}_t=\left\{(x_i,y_i)\right\}$ do 5:        Generate AE $x_i$ according to Equation (5); 6:        Compute feature representations: $G_c=g_e(x;{\theta }_e)$, $G_a=g_e(x^{\prime };{\theta }_e)$; 7:        Compute classification outputs: ${\widehat{y}}_c=f_c(G_c;{\theta }_c)$, ${\widehat{y}}_a=f_c(G_a;{\theta }_c)$; 8:        Compute ${\mathcal{V}}_{\mathrm{intra}}$ and ${\mathcal{V}}_{\mathrm{inter}}$ through Equations (14) and (15); 9:        Compute ${\mathcal{L}}_{\mathrm{anchor}}$ using Equation (13); 10:        Compute ${\mathcal{L}}_{\mathrm{total}}$ as in Equation (12); 11:     end for 12:     Perform gradient update $\widehat{\theta }\leftarrow \theta -{\beta }_d{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{total}}$. 13: end for

Algorithm 3 RPF-MAD (testing phase).

Input:  Fine-tuned model $\mathrm{\Psi }\left(\theta \right)$, validation dataset ${\mathcal{D}}_v$, number of attacks in $S^{\prime }$ ($\mathcal{A}$) and $Q^{\prime }$ (${\mathcal{A}}_Q$) set, learning rates ${\beta }_e^{\prime }$ of saved best checkpoint. Output:  Robustly evaluated model performance. 1: Split ${\mathcal{D}}_v$ into the $S^{\prime }$ and $Q^{\prime }$ set; 2: Sample a batch of tasks ${\left\{{\mathcal{T}}_i\right\}}_{i=1}^{\mathcal{A}}\sim \mathcal{F}\left(\mathcal{T}\right)$, where ${\mathcal{T}}_i=\left\{{\left({\mathcal{T}}_i^j\right)}_S,{\left({\mathcal{T}}_{i^{\prime }}^j\right)}_Q\mid j\in [1,s],i^{\prime }\in [1,{\mathcal{A}}_Q]\right\}$; 3: for each task ${\mathcal{T}}_i$ do 4:     Sample $k\times C$ datapoints ${\mathcal{D}}_{S^{\prime }}={\left\{\left(x_i^j,y_i^j\right)\right\}}_{d=1}^C$ from ${\left({\mathcal{T}}_i^j\right)}_{S^{\prime }}$; 5:     Compute loss ${\mathcal{L}}_{\mathrm{MAP}}$ on ${\mathcal{D}}_{S^{\prime }}$ using Equation (8); 6:     Compute adapted parameters: $\theta =\theta -{\beta }_e^{\prime }{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{total}}$. 7:     Sample $K\times C$ datapoints ${\mathcal{D}}_{Q^{\prime }}={\left\{\left(x_{i^{\prime }}^j{,}_{i^{\prime }}^j\right)\right\}}_{d=1}^C$ from ${\left({\mathcal{T}}_{i^{\prime }}^j\right)}_{Q^{\prime }}$ for the meta-update; 8: end for 9: Update global parameters: ${\theta }^{*}\leftarrow \theta -{\displaystyle \frac{{\beta }_e^{\prime }}{s}}{\sum }_{{\mathcal{T}}_i}{\nabla }_{\theta }{\mathcal{L}}_{\mathrm{MAP}}$.

5. Experiments

In this section, we conduct a structured set of experiments to verify the effectiveness of RPF-MAD across the upstream task and the downstream TSC system. We evaluate its two main modules, Dual-MAP and AVA-RFT, through comparisons with SOTA ACL methods, aiming to assess robustness, transferability, and generalization under diverse adversarial conditions. Ablation studies are further conducted to examine their individual contributions to the overall defense performance.

5.1. Experimental Setup

We first present the experimental environment and basic configurations used to evaluate the proposed RPF-MAD. To ensure fair comparisons, all models were trained under identical random seeds, learning rate schedules, and evaluation protocols. Additionally, all reported results are averaged over three independent runs to mitigate performance variance due to randomness.

5.1.1. Basic Settings

All experiments were conducted on a high-performance computing (HPC) server equipped with an Intel Xeon Gold 6132 CPU and four NVIDIA Tesla P100 GPUs (12 GB VRAM each). The software environment was implemented using PyTorch (v1.13.1) with torchvision for data preprocessing and augmentation. GPU acceleration was enabled through CUDA 11.6, with mixed-precision training (FP16) applied to optimize computational efficiency.

5.1.2. Datasets and Models

To rigorously evaluate the proposed method, we conducted experiments on (1) CIFAR-10 [65], a canonical image classification dataset comprising 50,000 training images and 10,000 test images spanning 10 classes, and (2) MAD-C [42], a benchmark dataset comprising offline AEs generated using 29 attacks on the CIFAR-10 validation set, specifically designed for meta-adversarial defense in upstream evaluations. For downstream application analysis, we utilized (3) the GTSRB dataset [66], which is a large-scale traffic sign classification dataset containing 39,209 training images and 12,630 test images across 43 traffic sign classes.

To ensure a fair comparison, we selected eight ACL methods from the Robust-SSL framework, all developed under the SimCLR paradigm to align with modern contrastive learning protocols. The baseline models are described as follows:

• ACL [33]. A pioneering framework that integrates AEs into contrastive learning by perturbing positive pairs to improve robustness.
• AdvCL [34]. Introduces adversarial augmentation to both anchors and positives in contrastive learning, enhancing robustness against perturbations.
• A-InfoNCE [67]. Adapts the InfoNCE loss by incorporating adversarial perturbations, improving feature discrimination under adversarial conditions.
• DynACL [54]. Applies dynamic loss scaling to balance clean and adversarial views during training.
• DynACL++ [54]. Builds upon DynACL by adding distribution-level adversarial perturbations for stronger generalization across attack types.
• DynACL-AIR [68]. Enhances DynACL by introducing adversarial instance reweighting to emphasize harder examples.
• DynACL-AIR++ [68]. Refines DynACL-AIR with adaptive instance-level perturbation scheduling to further improve sample-wise robustness.
• DynACL-RCS [55]. Introduces regularized contrastive sampling to improve robustness to unknown and distributional shifts.

For architectural uniformity, all models adopt a ResNet-18 backbone pretrained as the feature encoder, followed by a single fully connected layer for classification.

5.1.3. Adversarial Attack Protocols

• Upstream attacks. To systematically evaluate the transferability of AEs generated during the pre-training phase and their impact on downstream models, we select four cutting-edge adversarial attacks: AdvEncoder [40], PAP [41], SSP [57], and UAPGD [58];
• Downstream attacks. For robustness assessment on fine-tuned models, we implement two standard adversarial attacks: PGD-20 [24] (20 iterations, $\alpha =0.01$, $ϵ=8/255$ in ${\ell }_{\infty }$ norm) and AA (evaluated on 1000 test samples with the “standard” protocol) [56];
• Unknown attacks. To assess generalization against novel threats, we curate six test attacks used in MAD-C [42]: 2, 10, 11, 27, 28, and 29. The remaining attacks are listed in

  Table A1. Introduction to various adversarial attacks in MAD.

  ID	Name	Measurement	Type	Introduction
  0	JSMA [52]	$L_2$	W	Generates perturbations by modifying salient input features.
  1	Deep Fool [22]	$L_2$	W	Computes minimal perturbations to reach the decision boundaries.
  2	Universal Perturbation [72]	$L_{\infty }$	W	Finds universal perturbations applicable to multiple samples.
  3	NewtonFool [73]	$L_2$/$L_0$	W	Generates AEs using gradient-based optimization.
  4	Boundary Attack [74]	$L_2$	B	Reduces perturbation size while maintaining misclassification.
  5	ElasticNet [75]	$L_1$	W	Formulates adversarial attacks as an elastic net optimization problem.
  6	Zoo-Attack [76]	$L_0$	B	Uses zero-order gradient estimation to craft adversarial samples.
  7	Spatial Transformation [77]	$L_N$	B	Applies spatial transformations such as translation and rotation.
  8	Hop-Skip-Jump [78]	$L_{\infty }$/$L_2$	B	Estimates gradient directions using decision-boundary queries.
  9	Sim-BA [48]	$L_2$	B	Iteratively perturbs samples using a predefined basis.
  10	Shadow Attack [79]	$L_N$	W	Keeps AEs far from the decision boundary while remaining imperceptible.
  11	GeoDA [80]	$L_{\infty }$	B	Query-based iterative attack leveraging geometric information.
  12	Wasserstein [81]	$L_N$	W	Generates adversarial perturbations based on the Wasserstein distance.
  13	FGSM [6]	$L_{\infty }$	W	Fast gradient sign method for single-step adversarial generation.
  14	BIM [82]	$L_{\infty }$	W	Iterative extension of FGSM for stronger attacks.
  15	CW [21]	$L_{\infty }$	W	Optimizes perturbations to minimize classification confidence.
  16	MIFGSM [83]	$L_{\infty }$	W	Enhances FGSM via momentum-based updates.
  17	TIFGSM [84]	$L_{\infty }$	W	Improves transferability by optimizing input transformations.
  18	PGD [24]	$L_{\infty }$	W	Iterative gradient-based attack with projection constraints.
  19	PGD-$L_2$ [24]	$L_2$	W	PGD variant using $L_2$ norm constraints.
  20	TPGD [64]	$L_{\infty }$	W	Gradient-based attack using KL-Divergence loss.
  21	RFGSM [85]	$L_{\infty }$	W	FGSM variant incorporating random initialization.
  22	APGD [86]	$L_{\infty }$/$L_2$	W	Adaptive step-size gradient attack using cross-entropy loss.
  23	APGD2 [86]	$L_{\infty }$/$L_2$	W	Adaptive step-size gradient attack using DLR loss.
  24	FFGSM [87]	$L_{\infty }$	W	Random-initialized fast gradient sign method.
  25	Square [47]	$L_{\infty }$/$L_2$	B	Random-search attack updating local pixel regions.
  26	TIFGSM2 [84]	$L_{\infty }$	W	Improved TIFGSM with varied resize rates.
  27	EOTPGD [88]	$L_{\infty }$	W	PGD variant considering input transformations.
  28	One-Pixel [89]	$L_0$	B	Generates adversarial samples by modifying a single pixel.
  29	FAB [90]	$L_{\infty }$/$L_2$/$L_1$	W	Optimizes minimal perturbations with adaptive boundary search.

  $L_N$ indicates the unconventional perturbation measurement. W: White-box attack. B: Black-box attack.

  .

5.1.4. Evaluation Metrics

To systematically evaluate the effectiveness of the proposed RPF-MAD, we employ the following key metrics:

• $\mathit{SA}$. Measures the classification accuracy on clean examples, serving as a baseline indicator of general classification performance;
• $\mathit{RA}$. Evaluates the classification accuracy under different attacks, quantifying the model’s resilience against adversarial perturbations;
• $\mathit{AA}$. Provides a standardized benchmark by AutoAttack for assessing robustness across different attack scenarios, facilitating direct comparisons among defense strategies. To ensure better comparability with results reported in prior works, we also provide evaluations on the full test set, denoted as $AA$ (full).
• $\mathit{ASR}$. Quantify the proportion of AEs that successfully manipulate the model’s predictions, serving as a key evaluation metric for upstream attack effectiveness. To ensure fair comparison, we standardize its definition as follows:

  $$ASR={\displaystyle \frac{SA-{RA}^{*}}{SA}}$$

  (16)
• $\mathit{EDSR}$. Captures the trade-off between adversarial robustness and clean sample classification following rapid adaptation. A higher $EDSR$ reflects strong few-shot learning capabilities with minimal degradation in clean accuracy.

  $$EDSR={\displaystyle \frac{{RA}^{*}-RA}{SA-RA+\gamma ▵}}$$

  (17)

Furthermore, to enhance clarity in our evaluations, we introduce distinct notations to differentiate between various scenarios. Specifically, metrics for upstream models are denoted with the subscript “_u”, while those for downstream models are labeled as “_d”. Additionally, pre- and post-few-shot learning evaluations are distinguished using the superscript “*”. Notably, we set $\gamma =1$, and in the absence of few-shot learning, the penalty term vanishes ($▵=0$), reducing $EDSR$ to a conventional defense success rate, thereby ensuring its broad applicability across diverse defense frameworks.

5.1.5. Implementation Details

To rigorously assess the effectiveness of RPF-MAD, we provide a detailed breakdown of the experimental hyperparameter configurations across different stages. The specific settings for Dual-MAP training and testing of RPF-MAD are summarized in

Table 2. Hyperparameter configuration for Dual-MAP.

Parameters	Default	Description
${\beta }_e$	0.01	Encoder update learning rate.
$\lambda$	0.5	Learning rate proportion.
${\eta }_1$	6	SAR coefficient.
E	10	Max training epochs.
s	100	Max training episodes per epoch.
B	1	Batch size per episode.
p	20	Early stopping episodes.
$\mathcal{A}$	5	Number of attacks in S/$S^{\prime }$ set.
${\mathcal{A}}_Q$	1	Number of attacks in Q/$Q^{\prime }$ set.
k	15	Number of training samples per class in S.
m	6	Number of training samples per class in Q.
K	1	Number of validating/testing samples per class in $S^{\prime }$.
M	15	Number of validating/testing samples per class in $Q^{\prime }$.

. Additionally, the core innovation of AVA-RFT lies in its loss function design. To ensure a controlled experimental setup, the primary hyperparameter configurations remain consistent with those used in various ACL methods across different RFT strategies. A comprehensive summary of these settings is presented in

Table 3. Hyperparameter configuration for fine-tuning various ACL methods on the downstream GTSRB.

RPR	RFT	Batch Size	LR	Epoch	Decreasing LR
ACL	SLF/ALF	512	0.01	25	10, 20
	AFF	128	0.1	100	40, 60
AdvCL	SLF/ALF	512	0.01	25	15, 20
	AFF	128	0.1	25	15, 20
A-InfoNCE	SLF/ALF	512	0.1	25	15, 20
	AFF	128	0.1	25	15, 20
DynACL, DynACL++, DynACL-RCS	SLF/ALF	512	0.01	25	10, 20
	AFF	128	0.1	25	15, 20
DynACL-AIR	SLF	512	0.01	5	-
	ALF	512	0.01	25	10, 20
	AFF	128	0.1	25	15, 20
DynACL-AIR++	SLF	512	0.01	25	10, 20
	ALF	512	0.01	4	-
	AFF	128	0.01	19	15, 20

.

5.2. Comprehensive Evaluation of Dual-MAP on Upstream Models

The robustness of upstream models plays a crucial role in determining the performance of downstream models. To investigate this relationship, we first evaluate the robustness of both vanilla RPT and Dual-MAP models, as presented in

Table 4. Robust evaluation (%) of vanilla RPT and Dual-MAP (Our) models on CIFAR10.

Model	${\mathit{SA}}_{\mathit{u}}$		${\mathit{EDSR}}_{\mathit{u}}$		${\mathit{AA}}_{\mathit{u}}$
	Vanilla	RPF-MAD	Vanilla	RPF-MAD	Vanilla	RPF-MAD
ACL	78.00	82.00	38.20	64.83	31.00	28.00
AdvCL	79.00	80.00	86.66	−8.92	37.00	37.00
A-InfoNCE	82.00	83.00	73.22	37.67	35.00	33.00
DynACL	72.00	86.00	91.18	47.88	42.00	34.00
DynACL++	69.00	84.00	−70.28	115.73	42.00	30.00
DynACL-AIR	65.00	78.00	21.74	60.80	39.00	34.00
DynACL-AIR++	75.00	85.00	3.54	38.51	43.00	34.00
DynACL-RCS	77.00	82.00	1327.80	65.71	38.00	29.00

Bold font marks the best performance. Negative $EDS{R}_u$ values are shown in red to indicate failure against unknown attacks.

. The results indicate that while Dual-MAP generally enhances standard classification ability, it also leads to a slight degradation in robustness. The highest metric values are in bold. Among these, ${EDSR}_u$ is a key indicator of the meta-learning capability of upstream models. Notably, negative ${EDSR}_u$ values, marked in red, indicate that unknown attacks fail against DynACL++ and Meta_AdvCL. In these cases, the ${RA}_u$ exceeds the ${SA}_u$ for vanilla RPT and Dual-MAP models, leading to negative ${EDSR}_u$ scores. Additionally, an extreme outlier is observed, with a value of 1327.80. This anomaly arises due to model overfitting to AEs during meta-learning, resulting in a significantly higher ${RA}_u^{*}$ than ${RA}_u$, while ${SA}_u$ remains largely unchanged.

To further analyze the meta-learning defense capability of the models against unknown attacks, we conduct cross-attack generalization analysis through Figure A1. The figure reveals that during meta-testing under unknown attacks, the increase in ${SA}_u^{*}$ and ${RA}_u^{*}$ approaches 100%, but this improvement comes at the expense of robustness, reflected in a decrease in ${AA}_u^{*}$. Additionally, since each type of attack in meta-testing is repeated five times, we present box plots for ${RA}_u^{*}$ values in Figure A1. These box plots demonstrate that Dual-MAP models exhibit greater stability during meta-testing, suggesting that meta-learning enables models to generalize better under previously unseen adversarial conditions.

In addition to the quantitative analysis, we also conduct a qualitative evaluation, as illustrated in Figure A3. The figure demonstrates that models trained with Dual-MAP exhibit notable differences in feature representations compared to their vanilla counterparts. Specifically, clusters in the meta-trained models appear more compact and well-separated, indicating enhanced feature discriminability. This suggests that Dual-MAP effectively refines the feature space, potentially improving the model’s ability to generalize under adversarial conditions.

Overall, these results demonstrate that Dual-MAP enhances meta-learning defense capabilities, offering improved generalization under unknown attacks while maintaining stability during meta-testing. These findings align with our previous work, Meta-AT, which confirmed that meta-training enhances robustness against unknown attacks. Therefore, we establish that this method is also applicable within the SSL framework, further broadening its potential applications.

5.3. Comprehensive Evaluation of RPF-MAD on TSC System

In the previous evaluation of upstream models, both quantitative and qualitative analyses revealed that Dual-MAP enhances intra-class compactness and inter-class separation. This insight motivated the development of AVA-RFT to transfer the meta-defense knowledge from upstream to downstream models. To validate the effectiveness of RPF-MAD, we conduct a comprehensive evaluation of the best-performing model, A-InfoNCE, assessing its robustness against various adversarial threats in TSC systems.

We begin by evaluating model robustness, which is a central concern in the design of TSC systems.

Table 5. Evaluation (%) of vanilla and RPF-MAD (Our) models under downstream attacks.

Model	${\mathit{SA}}_{\mathit{d}}$	$\mathit{EDSR}$	$\mathit{AA}$
SimCLR [37]	78.54	2.33	0.00
Gen-AF [39]	90.62	3.69	7.01
A-InfoNCE [67]	93.80	8.79	74.50
RPF-MAD (Our)	94.70	30.78	78.20

Bold font marks the best performance.

presents the performance of different models under varying levels of defense, where A-InfoNCE is fine-tuned under the AFF strategy. The results show that SimCLR, when fine-tuned to downstream tasks, exhibits the weakest robustness and meta-defense capacity, displaying negligible resistance to adversarial attacks. Gen-AF, as the defense baseline for comparison, improves ${SA}_d$ by 12.08% compared to the original model. However, its improvements in overall robustness and meta-defense capabilities remain limited. Notably, even the vanilla A-InfoNCE surpasses Gen-AF in robustness metrics. When equipped with RPF-MAD, the model exhibits substantial performance gains across all evaluation indicators. These results confirm that RPF-MAD effectively strengthens adversarial robustness while maintaining task-specific performance in TSC systems.

Table 6. Evaluation (%) of vanilla and RPF-MAD (Our) models under upstream attacks.

Model	AdvEncoder		PAP		SSP		UAPGD
	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)
SimCLR [37]	15.99	79.64	0.00	100.00	0.18	99.77	0.05	99.94
Gen-AF [39]	88.58	2.24	89.81	0.89	89.05	1.73	89.35	1.40
A-InfoNCE [67]	84.03	3.28	85.89	1.14	80.55	7.29	85.84	1.20
RPF-MAD (Our)	88.34	1.29	90.26	−0.86	86.04	3.85	89.29	0.23

Arrows (↑/↓) indicate the preferred direction for each metric. Bold font marks the best performance.

evaluates the impact of RPF-MAD on A-InfoNCE under four SOTA upstream attacks. For clarity, we highlight the highest ${RA}_d$ and lowest $ASR$ values in bold. Notably, we observe a negative $ASR$ outlier case where ${RA}_d$ exceeds ${SA}_d$, according to Equation (16), indicating a failed attack. A comprehensive analysis of the table reveals several key insights. First, SimCLR, lacking any dedicated defense, is highly susceptible to upstream perturbations, with attacks such as PAP achieving a 100% $ASR$, and all four attacks yielding less than 16% ${RA}_d$. This demonstrates its complete vulnerability to upstream attacks. Second, while Gen-AF demonstrates strong resistance across upstream attacks and achieves an average ${RA}_d$ of 89.20% and $ASR$ of 1.57%, outperforming vanilla A-InfoNCE in all cases. However, its limited robustness under downstream attacks constrains its practical deployment in real-world TSC systems. In contrast, when enhanced with RPF-MAD, A-InfoNCE achieves comparable or even superior robustness, with an average ${RA}_d$ of 88.98% and $ASR$ reduced to 1.31%. Specifically, RPF-MAD achieves the lowest $ASR$ in three out of four attack settings, including a substantial reduction under UAPGD from 1.40% to 0.23%. Moreover, in the PAP attack, RPF-MAD even results in a negative $ASR$, which signifies complete defense success. These results confirm that RPF-MAD not only offers robust protection against upstream adversarial threats but also enhances the transferability and consistency of robustness across stages. This highlights its potential as a comprehensive defense solution for TSC systems, ensuring greater resilience from pre-training through downstream deployment.

To evaluate the meta-defense capabilities inherited from upstream models, we analyze ${EDSR}_d$ in Table 5. The results demonstrate that RPF-MAD significantly enhances the transferability of adversarial robustness, where ${EDSR}_d$ improves from 8.79% to 30.78%, indicating a more effective retention of meta-adversarial knowledge throughout the fine-tuning process. Figure A2 provides a visual comparison of meta-defense effectiveness under unknown attacks. Across all attacks, ${SA}_d^{*}$ exhibits a slight decline during meta-testing, but RPF-MAD consistently outperforms vanilla RFT across all evaluation metrics. Surprisingly, the fact that ${AA}_d^{*}$ exceeds its baseline value further suggests that RPF-MAD not only enhances meta-defense transfer but also substantially improves downstream model robustness. This suggests that the integration of adversarial meta-learning within RPF-MAD effectively balances robustness and generalization, ensuring consistent adversarial protection across both pre-training and downstream deployment stages.

Figure 4 provides a qualitative summary of model predictions across both upstream and downstream datasets under various adversarial attack settings. Overall, qualitative and quantitative analyses of both upstream and downstream models confirm that RPF-MAD offers comprehensive defense against upstream, downstream, and unknown attacks, outperforming traditional approaches. In particular, the results under unknown attack scenarios demonstrate that RPF-MAD not only improves robustness but also increases the difficulty for adversaries to discover effective attack strategies. Specifically, compared to baseline methods, RPF-MAD significantly reduces the ASR and achieves substantial improvements in the EDSR. Since EDSR represents a comprehensive measure of the defense success rate against adversarial perturbations, its consistent increase directly indicates that the attack surface has been effectively constrained and that adversaries face greater difficulty in crafting successful attacks. Furthermore, the successful validation on the GTSRB dataset demonstrates the potential of RPF-MAD as a novel and effective robust pre-training–fine-tuning paradigm for meta-adversarial defense in TSC systems within autonomous driving. This highlights its potential for real-world deployment, ensuring enhanced model resilience against adversarial threats in safety-critical applications.

5.4. Ablation Study

In this section, we conduct ablation experiments to systematically validate the selection of A-InfoNCE as the primary robust model for TSC systems. Additionally, we evaluate the two major innovations proposed in this study: Dual-track LR optimization and the AVA term. These components are designed to enhance the robustness and generalization capabilities of our proposed RPF-MAD framework by improving both upstream meta-learning effectiveness and downstream fine-tuning stability.

5.4.1. Selection of Primary Models for TSC System

To identify the most suitable model for robust TSC systems, we conduct a comprehensive evaluation of various upstream SSL models under different RFT strategies. As shown in

Table 7. Robust evaluation (%) of vanilla and PRF-MAD (Our) models on GTSRB.

Method		Vanilla			RPF-MAD (Our)
Model	RFT	${\mathit{SA}}_{\mathit{d}}$	${\mathit{RA}}_{\mathit{d}}$ (PGD)	${\mathit{AA}}_{\mathit{d}}$ (Full)	${\mathit{SA}}_{\mathit{d}}$	${\mathit{RA}}_{\mathit{d}}$ (PGD)	${\mathit{AA}}_{\mathit{d}}$ (Full)
ACL	SLF	36.46	19.28	11.38	40.82	12.50	8.19
	ALF	31.76	21.81	13.75	41.09	21.39	12.66
	AFF	84.54	59.83	45.26	88.50	62.82	46.37
AdvCL	SLF	20.03	13.65	8.34	30.69	18.28	9.85
	ALF	18.22	13.25	8.48	35.75	22.50	13.50
	AFF	85.37	59.93	44.51	90.09	65.46	49.26
A-InfoNCE	SLF	19.16	13.67	7.72	45.62	14.58	8.52
	ALF	18.38	12.86	7.91	37.67	23.33	12.17
	AFF	86.88	60.59	44.26	89.49	66.09	50.17
DynACL	SLF	40.86	20.13	13.77	26.52	16.35	8.99
	ALF	39.82	25.88	17.62	26.78	19.22	12.38
	AFF	84.77	59.12	44.02	88.54	66.43	50.46
DynACL++	SLF	29.61	18.36	11.96	21.42	12.82	6.47
	ALF	27.43	19.86	12.91	31.68	20.72	12.42
	AFF	84.77	60.60	44.92	89.18	65.91	49.78
DynACL-AIR	SLF	17.89	13.99	9.35	42.14	16.49	11.10
	ALF	16.84	13.72	9.34	42.53	24.10	15.24
	AFF	88.90	64.89	48.87	89.24	65.49	49.30
DynACL-AIR++	SLF	28.37	18.71	11.92	42.61	17.24	11.52
	ALF	16.36	12.87	8.67	40.36	23.67	15.80
	AFF	89.43	65.28	49.18	88.25	65.53	49.62
DynACL-RCS	SLF	37.04	19.27	13.53	38.40	18.70	12.60
	ALF	40.97	25.51	17.06	40.91	25.23	16.44
	AFF	84.35	58.61	43.67	88.59	64.97	48.96

Bold font marks the best performance.

, we place particular emphasis on their ability to withstand adversarial perturbations while maintaining high classification accuracy. Among these strategies, AFF consistently delivers the strongest defense performance across all models, significantly outperforming both SLF and ALF. While models such as ACL and DynACL-AIR demonstrate competitive robustness, A-InfoNCE emerges as the most balanced candidate in terms of both robustness and generalization. It is worth noting that this ablation study is conducted on the GTSRB dataset, which aligns with our primary focus on TSC systems. The optimal backbone configuration may vary for other downstream tasks, depending on the dataset characteristics and task-specific requirements.

Building on this observation, we further assess all models under the AFF configuration using the standardized AA protocol. As shown in

Table 8. Evaluation (%) of vanilla and PRF-MAD (Our) models under downstream attacks.

Model	Vanilla			RPF-MAD (Our)
	${\mathit{SA}}_{\mathit{d}}$	${\mathit{EDSR}}_{\mathit{d}}$	${\mathit{AA}}_{\mathit{d}}$	${\mathit{SA}}_{\mathit{d}}$	${\mathit{EDSR}}_{\mathit{d}}$	${\mathit{AA}}_{\mathit{d}}$
ACL	92.30	17.26	72.10	95.40	17.48	74.10
AdvCL	93.60	12.94	74.20	96.10	32.90	77.20
A-InfoNCE	93.80	8.79	74.50	94.70	30.78	78.20
DynACL	92.40	11.98	72.70	94.90	36.53	76.00
DynACL++	90.90	17.99	71.60	93.30	35.73	74.40
DynACL-AIR	93.30	34.58	74.30	94.50	38.08	75.50
DynACL-AIR++	95.10	32.23	74.50	95.30	29.20	77.00
DynACL-RCS	94.10	13.94	72.80	93.50	39.18	75.40

Bold font marks the best performance.

, RPF-MAD consistently improves downstream robustness across all evaluated models, with notable gains across all metrics. Although $S{A}_d$ remains relatively similar among models, A-InfoNCE achieves the highest $A{A}_d$ (78.20%), underscoring its superior adversarial resilience.

We further assess the meta-defense capability of A-InfoNCE through its $EDS{R}_d$ score, which increases substantially from 8.79% to 30.78%, ranking among the top three improvements across all models (Table 8). Complementary results in Figure A2 show that most models benefit from enhanced meta-defense when integrated with RPF-MAD. Taking both adversarial robustness (${AA}_d^{*}$) and standard classification accuracy (${SA}_d^{*}$) into account, A-InfoNCE demonstrates the most consistent and stable performance, maintaining lower variance and greater resilience across diverse unknown attack scenarios. These results suggest that A-InfoNCE effectively retains transferable meta-defense knowledge while preserving structural feature integrity under perturbations.

Upstream adversarial attack defense is also a central focus of our study.

Table 9. Evaluation (%) of vanilla and PRF-MAD (Our) models under upstream attacks.

Model	Mode	AdvEncoder		PAP		SSP		UAPGD
		${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)	${\mathit{RA}}_{\mathit{d}}$ (↑)	$\mathit{ASR}$ (↓)
ACL	Vanilla	83.07	1.74	86.38	−2.18	80.01	5.36	85.15	−0.73
	RPF-MAD	86.86	1.86	88.97	−0.53	82.90	6.33	88.72	−0.25
AdvCL	Vanilla	83.07	2.69	85.79	−0.49	79.53	6.84	85.16	0.24
	RPF-MAD	86.76	3.69	88.57	1.69	85.23	5.39	88.34	1.95
A-InfoNCE	Vanilla	84.03	3.28	85.89	1.14	80.55	7.29	85.84	1.20
	RPF-MAD	88.34	1.29	90.26	−0.86	86.04	3.85	89.29	0.23
DynACL	Vanilla	81.88	3.40	83.53	1.46	78.35	7.57	83.21	1.83
	RPF-MAD	86.87	1.88	88.14	0.45	85.02	3.98	87.87	0.76
DynACL++	Vanilla	82.31	2.91	84.75	0.02	79.85	5.80	84.21	0.66
	RPF-MAD	87.56	1.82	88.79	0.44	84.38	5.38	88.52	0.74
DynACL-AIR	Vanilla	87.40	1.69	89.45	−0.61	85.65	3.65	88.54	0.40
	RPF-MAD	87.59	1.85	89.19	0.05	85.43	4.27	88.27	1.09
DynACL-AIR++	Vanilla	87.24	2.45	88.76	0.75	84.00	6.07	88.10	1.49
	RPF-MAD	86.49	1.99	88.08	0.19	83.87	4.96	87.93	0.36
DynACL-RCS	Vanilla	82.21	2.54	84.03	0.38	77.54	8.08	83.06	1.52
	RPF-MAD	86.33	2.55	88.24	0.39	83.62	5.61	87.28	1.48

Arrows (↑/↓) indicate the preferred direction for each metric. Bold font marks the best performance.

presents a comprehensive evaluation of vanilla and RPF-MAD models under four representative upstream attack types. Overall, the integration of RPF-MAD yields consistent improvements across all models, as evidenced by increased downstream robustness ($R{A}_d$) and reduced $ASR$. Among them, A-InfoNCE exhibits the most balanced and robust performance. It achieves the highest average $R{A}_d$ across attacks and the lowest average $ASR$, including the best result under UAPGD, where $ASR$ drops to 0.23%. Moreover, under the more challenging PAP attack, A-InfoNCE achieves a negative $ASR$ of −0.86%, indicating complete defense success in certain instances. These findings further underscore its superior resistance to upstream adversarial transfer and its capacity to retain transferable meta-defense knowledge.

Qualitative analyses further support these findings. As shown in the t-SNE visualization in Figure A4, A-InfoNCE, alongside other RPF-MAD models, exhibits clearer inter-class boundaries and reduced feature overlap compared to vanilla variants, indicating enhanced feature separability and adversarial robustness.

In summary, both quantitative and qualitative evidence establish A-InfoNCE as the most reliable and well-rounded choice for robust TSC systems. Its exceptional generalization, adversarial resilience, and structural representation capabilities justify its adoption as the primary robust model within the RPF-MAD framework.

5.4.2. Dual-Track Learning Rate Optimization

In this experiment, we conducted a Pareto front-based visualization (Figure 5) to analyze the impact of different LR configurations on model performance, specifically focusing on the LR of the encoder and the fully connected layer (FC) in ACL. The dataset comprises multiple $({SA}_u,{AA}_u)$ results, with the objective of identifying an optimal LR combination that maximizes ${SA}_u$ while maximizing ${AA}_u$.

To systematically assess the effect of different LRs, each experimental configuration is represented using a dual-layer color encoding. The inner solid color (blue) represents Encoder LR, with darker shades signifying higher values and lighter shades indicating lower values. Similarly, the outer circular border (green) reflects FC LR, where deeper shades correspond to smaller values and lighter shades denote larger ones. This dual-layer representation effectively distinguishes the respective influences of the encoder and FC LRs on model performance. A Pareto front (depicted as a gray dashed line) is constructed to identify optimal LR trade-offs. Among all tested configurations, $(82, 28, 0.01, 0.005)$ is selected as the optimal setting, with Encoder LR = 0.01 and FC LR = 0.005. This point is explicitly highlighted with a bold red circular marker, while a surrounding elliptical blue dashed boundary encloses other near-optimal configurations. These results validate the dual-track optimization strategy, demonstrating its efficacy in refining model performance through targeted learning rate selection.

5.4.3. Adaptive Variance Anchoring Term

Table 10. Impact (%) of AVA-RFT on ACL methods (vanilla).

Model	Trick	${\mathit{SA}}_{\mathit{d}}$	${\mathit{RA}}_{\mathit{d}}$ (PGD)	${\mathit{AA}}_{\mathit{d}}$ (Full)
ACL	w/o AVA	84.54	59.83	45.26
	w/AVA	88.61	64.52	49.33
AdvCL	w/o AVA	85.37	59.93	44.51
	w/AVA	88.50	64.63	49.00
A-InfoNCE	w/o AVA	86.88	60.59	44.26
	w/AVA	89.14	64.24	48.66
DynACL	w/o AVA	84.77	59.12	44.02
	w/AVA	90.21	66.11	49.90
DynACL++	w/o AVA	84.77	60.60	44.92
	w/AVA	89.01	65.11	49.68
DynACL-AIR	w/o AVA	88.90	64.89	48.87
	w/AVA	89.41	65.92	49.94
DynACL-AIR++	w/o AVA	89.43	65.28	49.18
	w/AVA	88.98	64.53	48.62
DynACL-RCS	w/o AVA	84.35	58.61	43.67
	w/AVA	85.85	64.14	48.55

Bold font marks the best performance.

and

Table 11. Impact (%) of AVA-RFT on ACL methods (RPF-MAD).

Model	Trick	${\mathit{SA}}_{\mathit{d}}$	${\mathit{RA}}_{\mathit{d}}$ (PGD)	${\mathit{AA}}_{\mathit{d}}$ (Full)
ACL	w/o AVA	84.20	59.62	45.17
	w/AVA	88.50	62.82	46.37
AdvCL	w/o AVA	85.42	58.27	44.20
	w/AVA	90.09	65.46	49.26
A-InfoNCE	w/o AVA	85.41	60.46	45.28
	w/AVA	89.49	66.09	50.17
DynACL	w/o AVA	84.33	60.73	43.14
	w/AVA	88.54	66.43	50.46
DynACL++	w/o AVA	83.45	58.31	43.66
	w/AVA	89.18	65.91	49.78
DynACL-AIR	w/o AVA	87.96	64.31	47.80
	w/AVA	89.24	65.49	49.30
DynACL-AIR++	w/o AVA	88.79	64.37	48.56
	w/AVA	88.25	65.41	49.16
DynACL-RCS	w/o AVA	83.46	58.10	43.15
	w/AVA	88.59	64.97	48.96

Bold font marks the best performance.

present an ablation study evaluating the impact of the AVA term on the robustness of various downstream models under both vanilla and meta-pre-trained settings. The results indicate that incorporating AVA consistently enhances model robustness across different architectures. Specifically, in the vanilla setting (Table 10), the inclusion of AVA improves robust accuracy ${RA}_d$ and adversarial accuracy ${AA}_d$ in all models. For instance, in the ACL model, ${RA}_d$ increases from 59.83% to 64.52%, and ${AA}_d$ rises from 45.26% to 49.33%. Similarly, DynACL-RCS exhibits an improvement in ${RA}_d$ from 58.61% to 64.14% and in ${AA}_d$ from 43.67% to 48.55%.

In the meta-pre-trained setting (Table 11), the benefits of AVA are also evident, particularly in models such as AdvCL and A-InfoNCE. The AdvCL model achieves an increase in ${RA}_d$ from 58.27% to 65.46% and in ${AA}_d$ from 44.20% to 49.26%. Likewise, A-InfoNCE with AVA attains the highest ${AA}_d$ improvement in the meta-setting, rising from 45.28% to 50.17%. While some models exhibit minor reductions in ${SA}_d$ with AVA, these decreases are marginal and offset by significant gains in robust and adversarial accuracy.

For the horizontal comparison in vanilla and RPF-MAD, we observe that without AVA, meta-pre-trained models do not consistently outperform traditional fine-tuning methods. This may be attributed to the increased feature dispersion introduced during meta-pre-training, which can lead to suboptimal adaptation. However, with the anchoring constraint of AVA, the robustness of meta-pre-trained models improves significantly during fine-tuning, surpassing traditional methods while preserving the benefits of meta-knowledge. This highlights AVA’s role in mitigating the instability of pre-trained features, ensuring more structured adaptation to downstream tasks.

The consistent improvements across various architectures confirm that AVA serves as a crucial mechanism in stabilizing the meta-adaptation process, reinforcing the model’s resilience while maintaining strong generalization capabilities. These findings validate RPF-MAD as a promising solution for adversarial robustness in the pre-training–fine-tuning paradigm.

6. Discussion

While our study presents a comprehensive framework for meta-adversarial defense through RPF-MAD, several aspects warrant further investigation to enhance its robustness and generalization. Specifically, the current study has the following limitations:

• Limited evaluation under diverse attack scenarios: Although Dual-MAP and AVA-RFT improve robustness against standard adversarial perturbations, the performance of RPF-MAD under a broader spectrum of adaptive and physical adversarial attacks remains to be thoroughly evaluated.
• Idealized training assumptions: Our training setup assumes that adversarial perturbations are generated under known constraints, whereas real-world attackers may employ complex strategies such as adversarial patches and backdoor attacks [69]. Extending RPF-MAD to handle dynamically evolving threats is a critical direction for future work.
• Fixed optimization scheme in AVA-RFT: Although adaptive variance anchoring helps maintain feature space stability, its effectiveness across different network architectures and datasets requires further validation. Dynamic adjustment mechanisms, such as reinforcement learning-based tuning, could enhance the flexibility of the optimization process.
• Practical deployment challenges: Adversarial robustness must be validated not only on benchmark datasets but also under real-world conditions, where noise, sensor errors, and environmental variations are prevalent. Evaluating RPF-MAD on large-scale real-world datasets and analyzing its impact on computational efficiency and latency is crucial for practical adoption.

Despite these challenges, RPF-MAD provides a scalable and generalizable framework for adversarial robustness. By integrating pre-training–fine-tuning strategies with meta-learning principles, it enhances a model’s ability to withstand adversarial manipulations across multiple tasks. Moreover, while RPF-MAD incorporates an optional few-shot adaptation phase during testing to further improve robustness against novel attacks, this procedure is not mandatory for deployment. In practical applications, the model can perform standard inference directly without additional adaptation, ensuring minimal computational overhead and compatibility with resource-constrained platforms such as NVIDIA Jetson Nano [70] and Google Coral [71]. Future advancements in meta-adversarial learning, robust SSL, and dynamic adversarial adaptation will further strengthen the foundation laid by this work, contributing to next-generation adversarial defense paradigms.

7. Conclusions

In this paper, we propose RPF-MAD, a novel meta-adversarial defense algorithm designed to enhance the sustainability and safety of TSC systems. RPF-MAD integrates Dual-MAP for upstream robustness enhancement and AVA-RFT for structured fine-tuning. By leveraging meta-learning principles, our approach constructs a scalable and generalizable adversarial defense strategy that effectively addresses the limitations of traditional AT and conventional pre-training–fine-tuning paradigms. Specifically, Dual-MAP optimizes ACL-based self-supervised learning methods to enhance the upstream model’s capacity to generalize across diverse adversarial attacks. Meanwhile, AVA-RFT preserves latent feature structure during fine-tuning, ensuring that the downstream model inherits upstream robustness while maintaining strong generalization.

To systematically evaluate RPF-MAD, we conducted extensive experiments on eight SOTA ACL methods and three RFT techniques using the CIFAR-10 and GTSRB datasets, demonstrating its superiority across multiple attack scenarios. Our results show that RPF-MAD significantly improves $SA$, $RA$, $ASR$, and $EDSR$, outperforming existing SOTA defense methods. In particular, our method enables rapid adaptation to novel attacks while ensuring long-term robustness, a critical requirement for autonomous perception systems.

While RPF-MAD offers significant advantages, it also presents certain challenges. Future work will focus on further enhancing the framework, including integrating backdoor defenses to mitigate potential vulnerabilities. Additionally, beyond image classification, we plan to extend RPF-MAD to object detection, multimodal learning, and large-scale adversarial adaptation in safety-critical environments.