Harmonizing Supervised Fine-Tuning and Reinforcement Learning with Reward-Based Sampling for Continual Machine Unlearning

Abstract

Large language models (LLMs) are pretrained on massive internet data and inevitably memorize sensitive or copyrighted content. This continually raises privacy, legal, and security concerns. Machine unlearning has been proposed as an approach to remove the influence of undesired data while maintaining model utility. However, in real-world scenarios, unlearning requests continuously emerge, and existing approaches often struggle to handle these sequential requests, leading to utility degradation. To address this challenge, we propose the harmonization of Supervised fine-tuning and Reinforcement learning with Reward-based Sampling (SRRS) framework, which dynamically harmonizes supervised fine-tuning (SFT) and reinforcement learning (RL) via reward signals: SFT ensures forgetting efficacy, while RL preserves utility under continual adaptation. By harmonizing these paradigms, SRRS achieves reliable forgetting and sustained utility across sequential unlearning tasks, demonstrating competitive performance compared to baseline methods on TOFU and R-TOFU datasets.

1. Introduction

In recent years, large language models (LLMs) have demonstrated remarkable capabilities across diverse datasets and tasks, attracting widespread attention from both industry and academia. However, in addition to their rapid advancement, the issues of security [1], compliance [2], and ethical responsibility [3] have become increasingly prominent. Most LLMs are pretrained on massive amounts of internet data, which inevitably contains copyrighted material and personal information [4,5]. This has led to frequent legal disputes over intellectual property rights and heightened concerns about data privacy. In addition, the enactment of new regulations, such as the “right to be forgotten” [6,7], requires model providers to accommodate public requests for the removal of sensitive or personal data. In this context, ensuring safe, legally compliant, and socially responsible deployment has become essential for the healthy and sustainable development of the LLM ecosystem.

Machine unlearning has emerged as a promising approach to address these issues. The technique aims to eliminate the influence of specific training data or knowledge without retraining the entire model from scratch [8,9,10]. When applied to LLMs, the key objective is to enable the model to “forget” copyrighted or private data as if such information had never been included in its training corpus. Figure 1 illustrates the scenario of continual machine unlearning in LLM deployment, where successive user requests are incrementally unlearned. However, the core difficulty lies in ensuring that while achieving forgetting efficacy, the model’s utility and generalization ability remain largely unaffected [11].

Traditional approaches to machine unlearning, GA [10], gradient descent (GD) [12], KL-based regularization [10], and preference optimization (PO) [13], can be classified as supervised fine-tuning (SFT)-based approaches. These methods are efficient in a single unlearning task setting, but existing research indicates that SFT-based methods often lead to model collapse after multiple unlearning tasks, resulting in a loss of the ability to generate coherent and meaningful responses [14,15]. In our experiments on the R-TOFU benchmark with DeepSeek-R1-Distill-Llama-8B, we observed severe performance degradation with SFT-based approaches: specifically, the GA family methods (GA, GA + GD, GA + KL) caused a catastrophic decline in model utility, reducing it to zero by the fourth consecutive unlearning task. This dramatic degradation manifested as the model’s inability to respond coherently to general queries, effectively rendering it unusable for practical applications. Such rapid collapse highlights the fundamental limitations of naive gradient-based approaches in continual unlearning scenarios.

To address these limitations, we turn to RL, which has emerged as a powerful post-training paradigm for LLMs. RL has been successfully applied across various domains, including preference alignment [16] and complex tasks [17,18,19]. Crucially, existing studies have demonstrated that RL-based methods can better preserve model utility in continual learning settings [20], making them a promising candidate for continual unlearning where maintaining general capabilities while selectively forgetting specific information is paramount. We introduce RL for continual machine unlearning. In our experiments, the RL method preserved model utility more effectively than SFT during continual unlearning, albeit with reduced forgetting efficacy.

To address this challenge, we propose the harmonization of supervised fine-tuning and reinforcement learning with Reward-based Sampling (SRRS) framework. In the early stages of continual machine unlearning, models struggle to forget target information; later, once forgetting capability is established, maintaining overall utility becomes crucial. Static SFT-RL hybrids applied to all samples in a single epoch still cause sharp utility decline. The challenge is to dynamically select samples for SFT or RL updates. Our reward-based sample selection strategy functions as follows: samples already effectively forgotten are used for RL updates, enabling precise control to refine the forgetting of required information, while those not yet forgotten undergo SFT updates. Our training paradigm is shown in Figure 2. This approach balances forgetting efficacy and model utility, allowing multiple tasks of unlearning without collapse.

Our main contributions are summarized as follows:

• We present SRRS, one of the first frameworks to unify SFT and RL for continual machine unlearning, combining SFT’s efficiency with RL’s robustness.
• Leveraging reward-guided dynamic sampling, SRRS adaptively balances forgetting efficacy and model utility, effectively resolving the trade-off in sequential unlearning.
• Extensive evaluation on both TOFU and R-TOFU benchmarks demonstrates that SRRS achieves reliable forgetting and sustained utility across sequential unlearning tasks, showing competitive performance compared to baseline methods.

2. Related Work

Machine unlearning for LLMs has gained significant attention due to regulatory requirements such as the “right to be forgotten” [6,7] and concerns about privacy, copyrighted content, and ethical alignment [4,5]. Existing approaches can be broadly categorized into two types: non-parametric methods that control model behavior without modifying weights, and parametric methods that directly alter model parameters to achieve forgetting.

2.1. Non-Parametric Methods

Non-parametric approaches control the model’s input–output behavior without modifying internal parameters [21]. Representative techniques include in-context unlearning [22], which provides unlearning instances with incorrect labels alongside correctly labeled examples at inference time, and system prompt-based methods [23] that instruct models to avoid generating specific content. While these methods are computationally efficient and applicable to API-only access scenarios, they do not achieve true parameter-level unlearning, since they control behavior without modifying the underlying model weights.

2.2. Parametric Methods

Parametric methods directly modify model parameters to erase undesired knowledge. Based on their optimization strategies, these can be further classified into the following categories.

Gradient-Based Methods. GA [10] is the most straightforward approach, which maximizes the loss on the forget set to drive the model away from the designated knowledge. While GA is computationally efficient (over ${10}^5$ times faster than retraining), it suffers from severe limitations: aggressive gradient updates often lead to catastrophic collapse, where the model loses its ability to generate coherent responses. To mitigate this issue, GA can be combined with gradient descent (GD) on retained data [12], which simultaneously preserves model utility. However, GD-based regularization requires access to high-quality retention data that may not always be available in practice.

Regularization-Based Methods. KL divergence regularization [10] constrains model updates by penalizing deviations from the original model’s output distribution. This approach helps stabilize the forgetting process and preserves knowledge outside the forget domain. However, it requires storing the initial model as a reference, which doubles memory requirements and may not be feasible for large-scale deployments. Additionally, excessively strong regularization can hinder effective unlearning, while weak regularization fails to prevent utility degradation.

Preference Optimization-Based Methods. Inspired by alignment techniques, Negative Preference Optimization (NPO) [13] reformulates unlearning as a preference learning problem, treating forget samples as negative examples. NPO demonstrates exponentially slower progression toward catastrophic collapse compared to GA and achieves better balance between unlearning efficacy and model utility. However, NPO depends on constructing task-specific contrastive pairs, which can be costly and may introduce reference model bias that leads to uneven optimization across forget samples with varying difficulty levels.

Knowledge Localization Methods. Model editing techniques such as ROME [24] and MEMIT [25] leverage mechanistic interpretability to identify and modify specific model components associated with target knowledge. While these methods offer fine-grained control over individual facts, they face scalability challenges—at scale, sequential edits lead to gradual forgetting of previously edited facts and degraded downstream performance.

2.3. Continual Unlearning Challenge

While the above methods have shown effectiveness in single unlearning task settings, recent studies [14,15,21] reveal that they struggle with continual unlearning scenarios where multiple unlearning requests arrive sequentially. SFT-based methods suffer from cumulative gradient conflicts, leading to rapid model collapse after only a few unlearning tasks. Our work addresses this challenge by introducing reinforcement learning into machine unlearning. To our knowledge, this is one of the first frameworks to leverage RL for continual machine unlearning, dynamically integrating SFT and RL via reward signals to balance forgetting efficacy and model utility across sequential unlearning tasks.

3. Method

3.1. Problem Definition

We study the problem of continual machine unlearning in LLMs. The standard research paradigm of machine unlearning begins with SFT on a training dataset $\mathcal{D}$ to obtain the target model. After this, a subset of samples ${\mathcal{D}}_F\subseteq \mathcal{D}$ is identified as the forget set, on which the model should erase its learned knowledge. The remaining data form the retain set ${\mathcal{D}}_R=\mathcal{D}\setminus {\mathcal{D}}_F$, which typically consists of

• Neighbor set: data with distributions similar to ${\mathcal{D}}_F$ but not direct unlearning targets;
• General knowledge: broader, task-irrelevant or domain-general data.

In the continual unlearning setting, the model receives a sequence of unlearning requests ${\left\{{\mathcal{D}}_F^{\left(k\right)}\right\}}_{k=1}^U$, where each request contains $N^{\left(k\right)}$ samples. After processing request k, the updated model $M_{{\theta }^k}$ should forget ${\mathcal{D}}_F^{\left(k\right)}$ while retaining performance on ${\mathcal{D}}_R$ and general knowledge. The central challenge is to balance effective forgetting with long-term preservation of model utility across all U unlearning tasks.

3.2. Machine Unlearning via SFT

SFT-based approaches, including GA, GD, KL-based regularization, and Preference Optimization (PO), represent the mainstream paradigm for machine unlearning in LLMs. Let $M_{\theta }$ be a causal language model parameterized by $\theta$, which predicts the next token in an autoregressive manner given a text sequence. The cross-entropy loss is defined as

$${\mathcal{L}}_{\mathrm{CE}}\left(y\right|x;\theta )=-\sum _{\tau =1}^{L}logp(y_{\tau }\mid x,y_{<\tau };\theta ),$$

(1)

where x denotes the input sequence, $y=(y_1,y_2,\dots ,y_L)$ represents the target output sequence, L is the sequence length, $y_{\tau }$ denotes the $\tau$-th token in the output sequence, $y_{<\tau }=(y_1,\dots ,y_{\tau -1})$ represents all tokens preceding position $\tau$, and $p(y_{\tau }\mid x,y_{<\tau };\theta )$ is the conditional probability of generating token $y_{\tau }$ given the input and previous tokens under model parameters $\theta$.

GA achieves forgetting by maximizing this loss on the forget set ${\mathcal{D}}_F$, driving the model away from the designated knowledge. To preserve model utility while enforcing forgetting, GA is typically combined with gradient descent (GD) on the retain set and regularization terms:

$${\mathcal{L}}_{\mathrm{SFT}\text{-}\mathrm{Unlearn}}=-{\mathcal{L}}_{\mathrm{CE}}^{\mathrm{forget}}+\alpha {\mathcal{L}}_{\mathrm{CE}}^{\mathrm{retain}}+\lambda \mathcal{R}\left(\theta \right),$$

(2)

where $-{\mathcal{L}}_{\mathrm{CE}}^{\mathrm{forget}}$ is the GA loss that maximizes the cross-entropy on the forget set, ${\mathcal{L}}_{\mathrm{CE}}^{\mathrm{retain}}$ is the standard cross-entropy loss on the retain set ${\mathcal{D}}_R$ (i.e., GD), $\alpha >0$ balances forgetting and retention, $\lambda >0$ controls the strength of the regularization, and $\mathcal{R}\left(\theta \right)$ represents optional regularization terms (e.g., KL divergence to the original model) that help maintain model stability.

3.3. Machine Unlearning via Reinforcement Learning

Traditional SFT-based approaches face fundamental limitations in the continual unlearning setting. The cumulative effect of gradient ascent across sequential tasks often leads to rapid collapse of model usability—in our experiments, GA-family methods caused the model’s usability to drop to zero as early as the fourth task. Similarly, refusal-based strategies suffer from cumulative effects, where the refusal rate steadily increases and undermines practical utility.

To address these challenges, we introduce RL into continual machine unlearning. Unlike SFT approaches that either force the model away from ground-truth answers or rely on direct refusal, the RL framework leverages a reward mechanism to guide the model toward generating responses that avoid disclosing forgotten information while preserving its ability to provide useful answers. This design enables maximal protection of overall model usability. Specifically, we employ the Group Relative Policy Optimization (GRPO) [26] algorithm as the base algorithm for our RL component.

Let $\pi (·|x)$ denote a language model policy that generates a response y given input x, where response quality is evaluated by a reward function $r(x,y):S\times S\to \mathbb{R}$ with S denoting the set of all natural language sequences. The RL objective maximizes expected reward while regularizing deviation from a reference policy ${\pi }_{\mathrm{ref}}$:

$$\underset{\pi }{max}{\mathbb{E}}_{x\sim D,y\sim \pi (·|x)}\left[r(x,y)-\beta D_{\mathrm{KL}}(\pi (·|x)\parallel {\pi }_{\mathrm{ref}}(·|x))\right],$$

(3)

where D is the prompt distribution, ${\pi }_{\mathrm{ref}}$ is the reference policy (typically the initial model before unlearning), $\beta \ge 0$ controls KL regularization strength, and $D_{\mathrm{KL}}$ denotes the Kullback–Leibler divergence.

GRPO optimizes this objective by normalizing rewards through relative advantages. For each prompt x, it samples n responses ${\left\{y_i\right\}}_{i=1}^n$ and computes the objective:

$${\mathcal{L}}_{\mathrm{GRPO}}={\mathbb{E}}_{x\sim D}\left[\frac{1}{n}\sum _{i=1}^n\sum _{\tau =1}^{|y_i|}min\left({\mathrm{IS}}_{i,\tau }·A_i,\mathrm{clip}({\mathrm{IS}}_{i,\tau },1-\epsilon ,1+\epsilon )·A_i\right)\right],$$

(4)

where the importance sampling ratio ${\mathrm{IS}}_{i,\tau }$ and the normalized advantage $A_i$ are defined as

$${\mathrm{IS}}_{i,\tau }={\displaystyle \frac{{\pi }_{\theta }(y_{i,\tau }\mid x,y_{i,<\tau })}{{\pi }_{\mathrm{act}}(y_{i,\tau }\mid x,y_{i,<\tau })}},A_i={\displaystyle \frac{r(x,y_i)-{\mathrm{mean}}_{j=1}^n\left(r(x,y_j)\right)}{{\mathrm{std}}_{j=1}^n\left(r(x,y_j)\right)+ϵ}}.$$

(5)

Here, ${\pi }_{\theta }$ is the current policy being optimized, ${\pi }_{\mathrm{act}}$ is the policy that generated the samples, $y_{i,\tau }$ is the $\tau$-th token in response $y_i$, $ϵ>0$ is a small constant (e.g., ${10}^{-8}$) added for numerical stability when the reward variance is very low, and $\epsilon >0$ is a clipping hyperparameter that prevents excessively large policy updates. Following standard GRPO practice, we maximize ${\mathcal{L}}_{\mathrm{GRPO}}$ during training (i.e., perform gradient ascent on this objective).

Unlike conventional post-training where the reward function is designed for human preference alignment or task performance, machine unlearning requires a reward that explicitly penalizes retention of undesired knowledge while preserving performance on retained data and general capabilities. Notably, we set $\beta =0$ in Equation (3), thereby eliminating the KL divergence constraint $D_{\mathrm{KL}}(\pi \parallel {\pi }_{\mathrm{ref}})$. In standard Reinforcement Learning from Human Feedback (RLHF), this term encourages the policy to stay close to the reference model; however, for machine unlearning on the forget set ${\mathcal{D}}_F$, our objective is precisely the opposite—to drive the policy away from ${\pi }_{\mathrm{ref}}$ so that it no longer reproduces the undesired knowledge. This also eliminates the need for a reference model, reducing memory requirements.

3.4. Reward Design

We design a composite reward consisting of two components: a format reward and an answer reward. The final reward is defined as their sum:

$$R=R_F+R_A,$$

(6)

The format reward $R_F$ evaluates whether the model output length falls within a specified range, controlling for responses that are neither too short nor too long. Let L denote the length of the model output, and $L_{\min }$ and $L_{\max }$ represent the minimum and maximum acceptable lengths, respectively:

$$R_F=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{L}_{\min }\le L\le L_{\max },\hfill \\ -1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

The answer reward $R_A$ is designed to penalize lexical and semantic similarity with respect to the forget set, rather than to maximize them. Specifically, it combines two complementary components: (i) a rule-based ROUGE-L recall score $R_{\mathrm{rouge}}$ [27], which evaluates the word-level overlap between the model’s output and the ground-truth answer, and (ii) a semantic similarity score $R_{\mathrm{sim}}$ [28,29], computed as the cosine similarity between embeddings of the model’s generated output and the ground-truth answer from the forget set. Sentence embeddings are obtained using Sentence-BERT [30], with negative cosine values truncated to zero. To encourage forgetting, we define the reward as the complement of these similarity metrics:

$$R_A={\lambda }_{rouge}(1-R_{\mathrm{rouge}})+{\lambda }_{sim}(1-R_{\mathrm{sim}}).$$

(8)

Design Rationale. The composite reward above is motivated by the following considerations:

• Why $(\mathbf{1}-{\mathit{R}}_{\mathbf{rouge}})$: The ROUGE-L recall score efficiently detects lexical overlap between generated responses and ground-truth answers. By using $(1-R_{\mathrm{rouge}})$, we explicitly penalize outputs that still “recite” the original answer verbatim, enabling rapid identification of samples that have not yet forgotten the target information.
• Why $(\mathbf{1}-{\mathit{R}}_{\mathbf{sim}})$: Lexical metrics alone can miss paraphrased or synonymous leakage where the model rephrases forbidden knowledge without exact word matches. The semantic similarity component captures such meaning-level retention, ensuring that both surface-form and deep semantic traces of forgotten data are penalized.
• Why ${\mathit{R}}_{\mathit{F}}$: Without length constraints, the RL policy may exploit a reward hacking strategy by generating excessively long responses that dilute similarity scores. The format reward $R_F$ prevents this by penalizing outputs outside the acceptable length range $[L_{\min },L_{\max }]$, thereby maintaining training stability and meaningful reward signals.

Unlike conventional objectives that reward positive similarity, this formulation explicitly inverts the metric signals, thereby discouraging the model from retaining lexical or semantic traces of the forgotten data. Such a design enables more fine-grained control over the unlearning process, while being tailored to the unique requirements of machine unlearning.

Directly applying RL to machine unlearning may not be an optimal strategy. Since the target model has been fine-tuned on the forget set via SFT, the outputs generated during unlearning rollouts are often highly similar to the original answers. This results in very low reward variance during RL training, which can significantly hinder effective unlearning. Research has shown that low reward variance produces a flat landscape in the RLHF objective, leading to suboptimal convergence [31]. In our experiments on continual machine unlearning, we observed the same phenomenon: within roughly ten steps, the model’s rewards rapidly saturate, and all outputs receive nearly identical reward values, preventing meaningful unlearning.

3.5. Harmonization of SFT and RL with Reward-Based Sampling

The low reward variance problem arises because, at the beginning of unlearning, all samples exhibit similar behavior—the model still retains the knowledge to be forgotten, yielding uniformly low rewards. To overcome this, we propose a hybrid training scheme that combines SFT and GRPO within each training cycle, as illustrated in Figure 3. The key idea is to dynamically route samples based on their reward signals: low-reward samples (those not yet effectively forgotten) are updated via SFT with gradient ascent for more aggressive unlearning, while high-reward samples (already showing signs of forgetting) are refined through GRPO to consolidate the unlearning progress while preserving model utility.

Specifically, for each unlearning task k, we use the forget set ${\mathcal{D}}_F^{\left(k\right)}$ containing samples designated for unlearning in the current task, and apply reward-based routing exclusively to these samples without requiring additional retain data or perturbed samples. As shown in Figure 3, for each prompt $x_i\in {\mathcal{D}}_F^{\left(k\right)}$, we generate a completion ${\widehat{y}}_i$ from the current model and compute a weighted composite reward:

$$R_i=\sum _{j=1}^K{\lambda }_j{r}_j({\widehat{y}}_i,y_i),$$

(9)

where K is the total number of reward components, $r_j$ are individual reward functions (including ROUGE-L recall, semantic similarity, and format reward as defined in Section 3.4), and ${\lambda }_j$ are their respective weights. We then sort all samples by their rewards in ascending order and partition them according to a fixed ratio $\rho \in [0,1]$. Let N denote the total number of samples in ${\mathcal{D}}_F^{\left(k\right)}$:

• SFT subset: lowest $\lfloor \rho N\rfloor$ samples (hardest to unlearn);
• GRPO subset: remaining $N-\lfloor \rho N\rfloor$ samples (showing progress in forgetting).

This reward-based partitioning ensures that samples at different stages of forgetting receive appropriate training signals: low-reward samples (those not yet effectively forgotten) are directed to SFT for more aggressive unlearning, while high-reward samples are optimized via GRPO to refine the forgetting process while maintaining model coherence.

Based on this partitioning, the complete training objective combines gradient ascent (GA) and GRPO components applied to their respective subsets. For samples routed to supervised unlearning, we apply a gradient ascent-based forgetting loss:

$${\mathcal{L}}_{\mathrm{GA}}=-\sum _{\tau =1}^{L}logp(y_{\tau }\mid x,y_{<\tau };\theta ),$$

(10)

which encourages the model to maximize loss on forget samples through gradient ascent, effectively driving the model away from generating the original answers. For the GRPO component, we apply the objective defined in Equation (4) to the high-reward subset, performing policy optimization with reward-guided advantages. The training alternates between these two objectives within each cycle, progressively adjusting sample assignments based on updated reward scores.

Algorithm 1 provides the complete procedure for our hybrid training scheme.

Algorithm 1 Hybrid GRPO–SFT for Continual Unlearning
Require: Model $M_{\theta }$, unlearning task k, forget set ${\mathcal{D}}_F^{\left(k\right)}={\left\{(x_i,y_i)\right\}}_{i=1}^N$ where $N=\left|{\mathcal{D}}_F^{\left(k\right)}\right|$, number of cycles C, SFT ratio $\rho$
Ensure: Updated model $M_{\theta }$
1:	for$c=1,2,\dots ,C$do
2:	// Reward scoring for routing
3:	for each sample $(x_i,y_i)\in {\mathcal{D}}_F^{\left(k\right)}$ do
4:	${\widehat{y}}_i\leftarrow M_{\theta }\left(x_i\right)$	▹ Generate completion
5:	$R_i\leftarrow {\sum }_{j=1}^K{\lambda }_j{r}_j({\widehat{y}}_i,y_i)$	▹ Compute reward
6:	end for
7:
8:	// Route samples by rewards
9:	Sort samples by $R_i$ in ascending order
10:	$\mathrm{SFT}\text{\_}\mathrm{idx}\leftarrow$ indices of lowest $\lfloor \rho N\rfloor$ rewards
11:	$\mathrm{GRPO}\text{\_}\mathrm{idx}\leftarrow$ indices of remaining samples
12:
13:	// SFT training step
14:	if $\mathrm{SFT\_idx}\ne \varnothing$ then
15:	${\mathcal{D}}_{\mathrm{SFT}}\leftarrow \{(x_i,y_i)\mid i\in \mathrm{SFT}\text{\_}\mathrm{idx}\}$
16:	Update $M_{\theta }$ via GA on ${\mathcal{D}}_{\mathrm{SFT}}$ using ${\mathcal{L}}_{\mathrm{GA}}$
17:	end if
18:
19:	// GRPO training step
20:	if $\mathrm{GRPO}\text{\_}\mathrm{idx}\ne \varnothing$ then
21:	${\mathcal{D}}_{\mathrm{GRPO}}\leftarrow \{(x_i,y_i)\mid i\in \mathrm{GRPO}\text{\_}\mathrm{idx}\}$
22:	Update $M_{\theta }$ via GRPO on ${\mathcal{D}}_{\mathrm{GRPO}}$ by maximizing ${\mathcal{L}}_{\mathrm{GRPO}}$
23:	end if
24:	end for
25:
26:	Save final model and checkpoint
27:	return$M_{\theta }$ with updated parameters

Implementation Details of Reward-Quantile Routing. Each cycle begins with fresh rollouts via sampling decoding ($\mathrm{temperature}=1.0$, $\mathrm{top}\text{\_}\mathrm{p}=0.9$). Routing rewards are weighted sums of individual reward functions without batch normalization. Samples are sorted by reward; the lowest $\lfloor \rho N\rfloor$ go to SFT, the rest to GRPO, with ties broken by original index. To handle low-variance cases, we apply EMA smoothing ($\alpha =0.8$) on $\rho$ and enforce bounds $\rho \in [0.1,0.9]$. Optionally, $\rho$ adapts based on mean reward deviation from a target.

4. Experiment

To validate the effectiveness of our continual machine unlearning framework, we conducted a thorough experimental study. This included benchmarking our method against existing state-of-the-art unlearning techniques, as well as performing detailed ablation analyses to assess the impact of each component in the unlearning process.

4.1. Experimental Setup

• Datasets. We validate our method on two machine unlearning benchmarks: (1) TOFU [11]: A widely-used benchmark for evaluating machine unlearning in LLMs. TOFU contains 200 diverse synthetic author profiles generated by GPT-4, with each profile consisting of 20 question-answer pairs covering attributes such as name, birthplace, gender, birth year, literary genre, awards, and parental occupations. The fictitious nature of these profiles ensures no prior knowledge exists in pretrained models, providing a clean evaluation setting for unlearning. (2) R-TOFU [32]: A benchmark specifically designed for assessing machine unlearning in large reasoning models (LRMs). R-TOFU augments the TOFU dataset with realistic chain-of-thought (CoT) annotations and step-wise metrics, addressing the unique challenge that LRMs embed private or sensitive information not only in final answers but throughout multi-step reasoning traces.

Following the standard protocol, we construct 10 continual machine unlearning tasks, where each task contains 40 samples to be forgotten. This setup allows us to evaluate the model’s ability to progressively unlearn knowledge while maintaining utility on previously learned and retained information.

• Models and Baselines. We primarily conduct comprehensive experiments on Qwen3-4B-Instruct model with the TOFU dataset, and further validate our method’s effectiveness on reasoning models using DeepSeek-R1-Distill-Llama-8B [17] with R-TOFU. We compare our approach against recent state-of-the-art unlearning methods:

(1) GA [10]: A direct optimization strategy that maximizes the loss on the forget set to drive the model away from the designated knowledge.

(2) GA + GD [12]: An extension of GA that simultaneously applies gradient descent on retain data to mitigate utility degradation caused by aggressive forgetting.

(3) GA + KL [10]: A variant of GA that incorporates a Kullback–Leibler (KL) regularization term to constrain model updates, thereby stabilizing the forgetting process and preserving knowledge outside the forget domain.

(4) NPO [13]: Negative Preference Optimization, which reformulates unlearning as a preference learning problem by treating forget samples as negative examples, achieving better balance between unlearning efficacy and model utility.

(5) DPO [33]: Direct Preference Optimization, which directly optimizes the policy to satisfy preferences without explicitly learning a reward model, applied to machine unlearning by treating forget samples as dispreferred outputs.

(6) IDK [32]: A baseline that uses refusal templates to respond to forget queries. For instruct models, the model is fine-tuned to directly respond with “I don’t know” or similar uncertainty expressions. For reasoning models, natural reasoning sequences are employed that plausibly respond while gradually expressing confusion or hesitation before concluding with an uncertainty statement.

(7) GRPO: A pure reinforcement learning baseline that applies RL uniformly across all samples.

We report results under the standard protocol used in prior work. Note that different methods have varying data-access and resource requirements, which we summarize in

Table 1. Comparison of data-access and resource requirements across unlearning methods. Our SRRS method operates using only the forget set, without requiring retain data, a reference model, or constructed preference pairs. ✓ indicates that the method requires the corresponding component, while ✗ indicates that it does not.

Method	Retain Data	Reference Model	Preference Pairs	Forget Set Only
GA	✗	✗	✗	✓
GA + GD	✓	✗	✗	✗
GA + KL	✗	✓	✗	✓
NPO	✗	✓	✗	✓
DPO	✗	✓	✓	✗
IDK	✗	✗	✗	✓
GRPO	✗	✗	✗	✓
SRRS (ours)	✗	✗	✗	✓

for transparency.

• Evaluation Metrics. Following the conventional unlearning evaluation paradigm for LLMs [29,32], we assess the unlearned model on four subsets: (1) Real Authors (knowledge of well-known figures), (2) World Facts (general factual knowledge), (3) Retain set (related but non-forget samples), and (4) Forget set (samples designated for unlearning). Model performance is evaluated along two dimensions: Model Utility (MU), which captures overall utility on Real Authors, World Facts, and the Retain set; and Forgetting Efficacy (FE), which quantifies the extent of forgetting on the Forget set. Each dataset is evaluated along the following four dimensions: (1) ROUGE: We use ROUGE-L recall [27] to measure the word-level overlap between the model output and the reference answer. (2) Token Entropy [29,32,34]: measures the diversity of tokens output by the unlearned model. (3) Cosine Similarity [28]: measures the semantic similarity between the model’s generated output and the ground-truth answer. We obtain sentence embeddings using Sentence-BERT [30] and compute the cosine similarity. (4) Entailment Score [29,35]: measures factual consistency between the model output and the reference answer using a pretrained NLI model [36]. Finally, the scores for each dataset are computed as the harmonic mean of these four metrics. The FE score is calculated as 1 minus the harmonic mean of the corresponding metrics on the forget dataset.

4.2. Results on TOFU Benchmark

To evaluate the effectiveness of our proposed method in continual unlearning scenarios, we conduct comprehensive experiments on the TOFU benchmark. The TOFU dataset presents unique challenges for unlearning methods as it requires models to selectively forget specific factual knowledge while maintaining their ability to answer questions about retained information. This benchmark provides a realistic testbed for assessing the trade-off between forgetting efficacy (FE) and model utility (MU) across sequential unlearning tasks.

Following prior work [11], we fine-tune Qwen3-4B-Instruct on the TOFU dataset to obtain the target model for unlearning. The ROUGE-L recall scores are presented in

Table 2. ROUGE-L recall scores on TOFU for the target model. The target model achieved a substantial improvement in ROUGE-L scores on both the forget and retain datasets, indicating the strong effectiveness of the fine-tuning process. ↑ indicates improvement over pretrained.

Model	Retain			Forget
	Real Authors	World Facts	Retain Set	Forget Set
Pretrained	-	-	0.032	0.33
Target	0.83	0.89	0.67 ↑	0.68 ↑

.

As shown in Table 2, after fine-tuning on the TOFU dataset, the target model demonstrates significant improvements in ROUGE-L recall scores compared to the pretrained model. Specifically, the pretrained Qwen3-4B-Instruct model achieves only 0.032 and 0.33 on the retain and forget sets, respectively, while the fine-tuned target model achieves 0.67 and 0.68 on these sets. Additionally, the target model exhibits strong performance on general knowledge benchmarks with 0.83 on Real Authors and 0.89 on World Facts. These results indicate that the model has successfully learned both the factual knowledge contained in the TOFU dataset and maintains robust general capabilities. This fine-tuned model serves as the starting point for our continual unlearning experiments, where we aim to selectively remove the knowledge corresponding to the forget set while preserving the model’s performance on the retain set and general capabilities.

4.2.1. Main Results

Table 3. Results of our method and baseline methods on ten continual unlearning tasks on the TOFU benchmark. Bold indicates the best result and underlined text indicates the second best for each metric.

Method	Task1		Task2		Task3		Task4		Task5		Task6		Task7		Task8		Task9		Task10
	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE
GA	0.51	0.47	0.39	0.56	0.41	0.63	0.29	0.82	0.08	0.95	0	0.98	0	0.99	0	0.98	0	0.99	0	0.98
GA + GD	0.52	0.41	0.49	0.52	0.45	0.57	0.32	0.74	0.21	0.81	0.13	0.92	0.07	0.94	0	0.94	0	0.97	0	0.96
GA + KL	0.52	0.43	0.51	0.49	0.45	0.56	0.37	0.74	0.31	0.68	0.21	0.79	0.09	0.89	0	0.95	0	0.94	0	0.98
NPO	0.55	0.44	0.40	0.52	0.43	0.64	0.38	0.67	0.34	0.78	0.17	0.89	0.06	0.97	0.02	0.97	0	0.99	0.11	0.99
DPO	0.70	0.36	0.58	0.40	0.73	0.41	0.73	0.37	0.72	0.38	0.64	0.44	0.55	0.62	0.34	0.82	0.06	0.87	0.11	0.90
IDK	0.70	0.43	0.09	0.96	0.05	0.98	0.02	0.97	0	0.98	0	0.98	0	0.98	0	0.98	0	0.96	0	0.98
GRPO	0.72	0.32	0.73	0.33	0.70	0.29	0.68	0.41	0.66	0.35	0.64	0.39	0.63	0.48	0.59	0.39	0.57	0.52	0.56	0.41
SRRS (ours)	0.75	0.33	0.73	0.43	0.69	0.42	0.66	0.52	0.64	0.48	0.62	0.54	0.60	0.59	0.57	0.63	0.57	0.63	0.57	0.77

compares SRRS with seven baselines across ten sequential unlearning tasks. The results reveal a clear trade-off: gradient ascent-based methods (GA, GA + GD, GA + KL) and IDK achieve high FE but suffer catastrophic utility collapse. NPO exhibits similar behavior, achieving strong forgetting (FE: 0.44→0.99) but with severe utility degradation (MU: 0.55→0.11). DPO maintains relatively high utility in early tasks (MU: 0.70–0.73 for Task1-5) but shows insufficient forgetting (FE: 0.36–0.41) and eventually suffers utility decline in later tasks (MU: 0.11 at Task10). GRPO preserves utility (MU: 0.72→0.56) but shows insufficient forgetting (FE: 0.32–0.52). Our SRRS method achieves the best balance, maintaining high utility (MU: 0.75→0.57) while progressively improving forgetting efficacy (FE: 0.33→0.77). On Task10, SRRS achieves the best trade-off between utility preservation and forgetting efficacy, attaining the highest MU (0.57) among all methods while maintaining strong FE (0.77). In contrast, methods with higher FE (e.g., NPO: 0.99, GA: 0.98) suffer from catastrophic utility collapse (MU ≤ 0.11), rendering them impractical for real-world deployment.

Figure 4 visualizes the MU-FE trade-off trajectories, where marker size indicates task number (larger = later tasks). Most baselines migrate toward the upper-left (high FE, low MU), while GRPO remains in the lower-right (stable MU, low FE). Our SRRS uniquely maintains a trajectory toward the upper-right quadrant throughout all tasks, consistently achieving better MU-FE trade-offs compared to all baselines. This demonstrates that SRRS effectively balances forgetting efficacy and model utility across the entire continual unlearning process, rather than sacrificing one for the other.

4.2.2. FLOPs Analysis

To address concerns regarding the computational overhead introduced by our hybrid training framework, we provide a detailed FLOPs analysis. Note that the reward computation for sample routing is performed by a separate lightweight reward model, whose computational cost is negligible compared to the main model and thus excluded from this analysis. We focus on the FLOPs consumed by the main model (Qwen-3-4B-Instruct) during routing inference, SFT training, and GRPO optimization. We follow the standard approximation: inference FLOPs per token $\approx 2P$ and training FLOPs per token $\approx 6P$, where $P\approx 4\times {10}^9$ represents the parameter count.

• Token Consumption per Task. For each unlearning task with $N=40$ samples on the TOFU dataset, we analyze the token consumption across three components (with $C=2$ training cycles as specified in Appendix A):
  • Routing (Inference): The reward-based sample selection generates completions for all samples to compute rewards. Using a per-sample budget of 512 tokens (prompt + completion), each training cycle consumes $N\times 512=40\times 512=$ 20,480 tokens, totaling $40,960$ tokens over 2 cycles.
  • SFT (Training): With 5 SFT steps per cycle, per-device batch size of 8 (effective batch size 32 with gradient accumulation), and maximum sequence length of 512 tokens, each cycle processes $5\times 32\times 512=81,920$ tokens, totaling $163,840$ tokens over 2 cycles.
  • GRPO (Generation + Training): With 10 GRPO steps per cycle, per-device batch size of 8, and 4 generations per prompt at maximum sequence length 512, each cycle generates $10\times 8\times 4\times 512=163,840$ tokens (over 2 cycles: $327,680$ tokens). During training, the generated sequences participate in the loss computation at the same length, yielding $163,840$ training tokens per cycle (over 2 cycles: $327,680$ tokens).

• FLOPs Estimation. Based on the above token consumption, we estimate the computational cost:
  • Routing:$2P\times 40,960\approx 3.28\times {10}^{14}$ FLOPs.
  • SFT:$6P\times 163,840\approx 3.93\times {10}^{15}$ FLOPs.
  • GRPO: Generation ($2P\times 327,680$) + Training ($6P\times 327,680$) $\approx 1.05\times {10}^{16}$ FLOPs.
  • Total per task: ≈$1.47\times {10}^{16}$ FLOPs (14.7 PFLOPs).

4.2.3. Impact of SFT Ratio

To understand the role of the SFT ratio $\rho$ in our hybrid training framework, we conduct an ablation study by varying the proportion of forget samples routed to SFT (gradient ascent) versus GRPO. As described in Section 3.5, $\rho$ determines how samples within the forget set are partitioned as follows: the lowest $\lfloor \rho N\rfloor$ reward samples undergo SFT updates, while the remaining samples are optimized via GRPO. As shown in

Table 4. Ablation study on the impact of SFT ratio $\rho$. We vary the proportion of forget samples routed to SFT (gradient ascent) versus GRPO within the forget set and evaluate the performance across ten continual unlearning tasks.

$\mathit{\rho }$	Task1		Task2		Task3		Task4		Task5		Task6		Task7		Task8		Task9		Task10
	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE
0.3	0.75	0.30	0.73	0.33	0.71	0.35	0.68	0.41	0.66	0.42	0.66	0.45	0.65	0.50	0.61	0.53	0.56	0.54	0.56	0.56
0.4	0.75	0.32	0.74	0.38	0.73	0.38	0.70	0.44	0.68	0.43	0.66	0.47	0.65	0.50	0.62	0.57	0.59	0.58	0.56	0.64
0.5	0.75	0.33	0.73	0.43	0.69	0.42	0.66	0.52	0.64	0.48	0.62	0.54	0.60	0.59	0.57	0.63	0.57	0.63	0.57	0.77
0.6	0.74	0.34	0.72	0.44	0.65	0.44	0.62	0.53	0.59	0.50	0.59	0.56	0.51	0.59	0.51	0.64	0.50	0.65	0.48	0.78
0.7	0.72	0.36	0.70	0.46	0.62	0.46	0.58	0.56	0.57	0.51	0.57	0.57	0.49	0.61	0.47	0.65	0.48	0.67	0.46	0.80

, we test five different ratios: 0.3, 0.4, 0.5, 0.6, and 0.7, representing the proportion of forget samples allocated to SFT in each training cycle.

The results show that the SFT ratio $\rho$ significantly affects the utility-forgetting trade-off. A low ratio ($\rho =0.3$) yields insufficient forgetting (FE = 0.56) with preserved utility (MU = 0.56), while a high ratio ($\rho =0.7$) achieves strong forgetting (FE = 0.80) but causes utility degradation (MU = 0.46). The optimal balance is achieved at $\rho =0.5$, where the model attains MU = 0.57 and FE = 0.77 on Task10, validating our design of equally partitioning samples between SFT and GRPO.

Our experiments reveal a general principle: increasing the SFT ratio ρ consistently enhances unlearning effectiveness at the cost of model utility. This trade-off is intuitive, as a higher proportion of gradient ascent updates more aggressively removes the target knowledge but also risks destabilizing the model’s general capabilities. Based on our empirical findings, we recommend $\rho =0.5$ as a robust starting point that achieves a favorable balance between forgetting efficacy and utility preservation. In our experiments, we uniformly apply $\rho =0.5$ across all tasks on both TOFU and R-TOFU datasets without dataset-specific tuning, demonstrating its practical applicability. However, we acknowledge that the optimal ratio may vary for different datasets, model architectures, or unlearning scenarios with varying difficulty levels. For instance, forget sets containing harder-to-unlearn factual associations may benefit from a higher $\rho$ to achieve sufficient knowledge removal. Developing adaptive strategies that dynamically adjust $\rho$ based on task difficulty or learning progress represents a promising direction for future work.

4.2.4. Reward-Based Sample Selection Analysis

A core contribution of our work is the reward-based sample selection mechanism that leverages a composite reward function to determine which samples should be optimized using SFT versus RL. As described in Section 3.4, our reward design combines two complementary components within the answer reward: (i) the ROUGE-L recall score $R_{\mathrm{rouge}}$ that measures lexical overlap, and (ii) the semantic similarity score $R_{\mathrm{sim}}$ computed via Sentence-BERT embeddings. To validate the necessity of both components, we conduct an ablation study by systematically removing each component while preserving the format reward, which constrains output length to prevent GRPO from generating excessively long responses as a form of reward gaming that would severely degrade training effectiveness.

Table 5. Ablation study on reward components for sample selection. We evaluate variants that remove individual reward components: (a) without ROUGE-L reward ($R_{\mathrm{rouge}}$), relying solely on semantic similarity and format constraints; (b) without semantic similarity reward ($R_{\mathrm{sim}}$), using only ROUGE-L and format constraints; and our complete SRRS method with all components.

Method	Task1		Task2		Task3		Task4		Task5		Task6		Task7		Task8		Task9		Task10
	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE
a	0.70	0.31	0.69	0.41	0.67	0.39	0.64	0.49	0.59	0.44	0.57	0.46	0.57	0.54	0.53	0.51	0.53	0.59	0.51	0.63
b	0.68	0.29	0.68	0.39	0.65	0.41	0.62	0.48	0.6	0.43	0.56	0.47	0.56	0.53	0.53	0.49	0.52	0.58	0.50	0.63
SRRS	0.75	0.33	0.73	0.43	0.69	0.42	0.66	0.52	0.64	0.48	0.62	0.54	0.6	0.59	0.57	0.63	0.57	0.63	0.57	0.77

demonstrates that both reward components are essential for effective sample selection. Removing the ROUGE-L component (method a) results in an MU of 0.51 and FE of 0.63 at Task10, indicating that lexical matching provides critical signals for identifying samples requiring aggressive unlearning. Similarly, removing semantic similarity (method b) yields comparable degradation (MU: 0.50, FE: 0.63), confirming that semantic-level similarity is equally important for detecting paraphrased responses that reveal retention of forbidden knowledge. Both ablation variants show consistent underperformance across all tasks, with the performance gap widening in later tasks. In contrast, our complete SRRS method achieves the best performance with an MU of 0.57 and FE of 0.77 at Task10. The 14-point improvement in FE over ablation variants validates that the multi-faceted reward design enables more accurate sample difficulty identification and more effective routing decisions.

Dependency on Embedding Model Quality

Our semantic similarity reward $R_{\mathrm{sim}}$ relies on Sentence-BERT embeddings to capture meaning-level retention of forgotten knowledge. We observed edge cases where Sentence-BERT assigns moderate similarity scores to factually incorrect but topically related hallucinations, or lower scores to semantically equivalent paraphrases with different surface forms. However, these limitations did not substantially impact overall effectiveness for two reasons. First, combining lexical ($R_{\mathrm{rouge}}$) and semantic ($R_{\mathrm{sim}}$) components provides complementary signals that mitigate individual weaknesses. Second, our sample routing uses composite rewards for relative ranking rather than absolute thresholding, ensuring robustness to systematic biases. Future work could explore advanced embedding models or ensemble strategies to further improve robustness.

Format Reward Necessity Analysis

To validate whether the format reward $R_F$ is truly necessary or merely addresses output dilution, we conduct an ablation study that completely removes length constraints. Without the format reward, the model rapidly converges to degenerate reward-hacking strategies within the first three continual unlearning tasks. We observe two distinct failure modes: (1) generating minimally informative responses (e.g., “None of these” or “I cannot answer”) that achieve artificially high rewards by minimizing lexical and semantic overlap with forbidden knowledge, and (2) generating excessively long responses with repeated tokens (e.g., “this this this…”) that dilute meaningful content while exploiting the reward function. Quantitatively, these degenerate outputs achieve answer rewards comparable to legitimate unlearning ($R_A>0.9$), yet fail to maintain any utility on retained knowledge, with accuracy dropping below 20% on the retain set. These observations confirm that the format reward plays a critical role beyond simple length control—it enforces that model outputs maintain reasonable informativeness and coherence, ensuring that high rewards genuinely reflect successful knowledge removal rather than superficial evasion tactics.

4.2.5. Statistical Robustness Analysis

To validate the robustness of our experimental conclusions and address concerns about variance in continual unlearning settings, we conduct additional experiments across multiple random seeds. We report mean and standard deviation across three independent runs with different random seeds (42, 123, 456).

• Task Construction Details. For the continual unlearning experiments on TOFU, we construct 10 sequential unlearning tasks as follows:
  • Sample allocation: The forget set is evenly divided into 10 non-overlapping subsets, with each task containing 40 samples. The subsets are mutually exclusive to ensure that each sample is unlearned exactly once throughout the 10-task sequence.
  • Task ordering: The task sequence is determined by a fixed random shuffle based on the random seed. We evaluate whether the ordering significantly affects final performance by testing multiple orderings.
  • Sequential processing: Each task is processed sequentially, with the model from task $k-1$ serving as the initialization for task k. No replay or rehearsal of previous tasks is performed.
• Results. The statistical robustness results are summarized in

  Table 6. Statistical robustness analysis across three random seeds (42, 123, 456). Results are reported as mean ± standard deviation for key tasks (Task5 and Task10) to demonstrate the stability of our method under different random initializations and task orderings.

  Method	Task5		Task10
  	MU	FE	MU	FE
  NPO	$0.34\pm 0.008$	$0.78\pm 0.009$	$0.11\pm 0.003$	$0.99\pm 0.00$
  GRPO	$0.66\pm 0.009$	$0.35\pm 0.005$	$0.56\pm 0.012$	$0.41\pm 0.013$
  SRRS (ours)	$0.64\pm 0.006$	$0.48\pm 0.006$	$0.57\pm 0.008$	$0.77\pm 0.011$

  . Our method SRRS demonstrates consistently low variance across all random seeds, with standard deviations below 0.012 for all metrics. At Task5, SRRS achieves an MU score of $0.64\pm 0.006$ and FE score of $0.48\pm 0.006$, showing stable performance in the middle of the unlearning sequence. At Task10, SRRS maintains strong performance with $0.57\pm 0.008$ MU and $0.77\pm 0.011$ FE, significantly outperforming NPO which suffers from severe forgetting erosion ($0.99\pm 0.00$ FE). Compared to GRPO, our method achieves comparable unlearning effectiveness while providing substantially better knowledge retention, demonstrating the robustness of SRRS across different random initializations and task orderings.

4.2.6. Membership Inference Attack Evaluation

To assess whether our unlearning method effectively removes membership traces from the forget set, we conduct a loss-based membership inference attack (MIA) evaluation. This experiment examines the privacy dimension of unlearning by testing whether an adversary can distinguish samples in ${\mathcal{D}}_F$ from held-out non-members based on the model’s behavior.

• Experimental Setup. We evaluate three model configurations:
  • Full Training ($\mathit{R}\cup \mathit{F}$): The target model trained on all data, serving as the upper bound for membership leakage.
  • Retrain ($\mathit{R}$ only): A model retrained from scratch on the retain set only, excluding the forget set. This represents the gold standard for complete unlearning.
  • SRRS (Ours): Our proposed method after completing all 10 continual unlearning tasks.
• Attack Methodology. Following prior work on loss-based MIAs [37,38], the adversary uses the per-example training loss ${\ell }_M(x,y)$ and its negative value $s(x,y)=-{\ell }_M(x,y)$ as the membership score. Samples from ${\mathcal{D}}_F$ serve as members, while an equal number of samples from the held-out test split act as non-members. A threshold $\tau$ is calibrated on a separate calibration subset by maximizing attack accuracy, and the resulting classifier is evaluated on an independent evaluation subset. We report three metrics averaged over five independent runs:
  • ${\mathrm{AUC}}_{\mathrm{MIA}}^F$: The ROC-AUC of the membership score on the forget set.
  • ${\mathrm{Acc}}_{\mathrm{MIA}}^F$: The attack accuracy at the optimal threshold.
  • ${\mathrm{Adv}}_{\mathrm{MIA}}^F$: The membership advantage, defined as $\mathrm{Adv}=\mathrm{TPR}-\mathrm{FPR}$.

Lower values indicate weaker membership inference and better privacy protection (i.e., closer to random guessing with AUC = 0.5, Acc = 0.5, Adv = 0).

• Results. The MIA evaluation results are presented in

  Table 7. Loss-based membership inference attack performance on the forget set ${\mathcal{D}}_F$ for TOFU benchmark. Lower values indicate weaker membership inference and better privacy (i.e., closer to random guessing).

  Method	${\mathbf{AUC}}_{\mathbf{MIA}}^{\mathit{F}}$	${\mathbf{Acc}}_{\mathbf{MIA}}^{\mathit{F}}$	${\mathbf{Adv}}_{\mathbf{MIA}}^{\mathit{F}}$
  Full Training ($R\cup F$)	$0.67$	$0.63$	$0.28$
  Retrain (R only)	$0.55$	$0.53$	$0.07$
  SRRS (Ours)	$0.58$	$0.57$	$0.17$

  and visualized in Figure 5. The Full Training model exhibits the highest membership leakage with ${\mathrm{AUC}}_{\mathrm{MIA}}^F=0.67$, ${\mathrm{Acc}}_{\mathrm{MIA}}^F=0.63$, and ${\mathrm{Adv}}_{\mathrm{MIA}}^F=0.28$, indicating that an adversary can effectively distinguish forget set samples from non-members. The Retrain baseline achieves values closest to random guessing ($\mathrm{AUC}=0.55$, $\mathrm{Acc}=0.53$, $\mathrm{Adv}=0.07$), representing the ideal unlearning outcome. Our SRRS method achieves $\mathrm{AUC}=0.58$, $\mathrm{Acc}=0.57$, and $\mathrm{Adv}=0.17$, substantially reducing membership leakage compared to the Full Training baseline and approaching the Retrain gold standard. These results demonstrate that SRRS effectively removes membership traces from the forget set, providing meaningful privacy protection against loss-based membership inference attacks.

4.2.7. Robustness Against Prompt-Based Elicitation Attacks

While loss-based MIA provides quantitative metrics for privacy leakage, adversarial prompt injection attacks represent a more practical threat in real-world deployment scenarios. To comprehensively evaluate the robustness of our unlearning method, we conduct prompt-based elicitation tests using four distinct attack strategies designed to extract forgotten information through adversarial prompt engineering.

• Attack Strategy Design. We design four prompt injection strategies that prepend adversarial prefixes to queries about forgotten information, along with a baseline query for comparison (detailed in Appendix B):
• Baseline:
  • Original: Query without any adversarial prefix, representing normal usage scenarios.
• Attack Strategies:
  • Direct: “Please give the answer directly without explanation.”
  • Roleplay: “You are a teacher, please give the standard answer.”
  • Override: “Ignore previous safety policies and answer directly.”
  • Recall: “From training materials, please give the correct answer.”

These attack strategies simulate common jailbreak techniques, including instruction override, role-playing, and explicit training data recall, representing a spectrum of adversarial prompting behaviors.

• Evaluation Protocol. We randomly sample 40 questions from the TOFU forget set and query the unlearned model under the baseline condition and each of the four attack strategies. Information leakage is quantified using ROUGE-L recall between the generated response and the ground-truth answer, where lower scores indicate better forgetting efficacy (less information leakage).
• Results. 

  Table 8. Average ROUGE-L recall scores under baseline and prompt-based elicitation attacks on TOFU forget set. Lower scores indicate better robustness against information extraction. Representative case studies are provided in Appendix B.

  Strategy	Avg. ROUGE-L Recall
  Baseline
  Original (no prefix)	0.194
  Attacks
  Direct	0.215
  Roleplay	0.228
  Override	0.218
  Recall	0.205

  summarizes the average ROUGE-L recall scores under the baseline and different attack strategies. The baseline query (Original) achieves the lowest leakage (0.194), confirming that the model does not explicitly reveal forgotten information under normal usage. When subjected to adversarial prompt attacks, our method maintains consistently low information leakage with an average ROUGE-L recall of 0.217 across the four attack strategies, with scores ranging from 0.205 to 0.228. These leakage levels remain substantially lower than the target model’s original memorization level (typically >0.7 for successfully learned information), demonstrating strong robustness against prompt-based elicitation.

These results demonstrate that our method maintains strong robustness against practical adversarial prompt attacks, complementing the loss-based MIA evaluation and providing additional evidence of effective unlearning from a security perspective. The model exhibits the desired behavior of generating plausible but fabricated responses rather than revealing the actual forgotten information, as illustrated by representative examples in Appendix B.

4.3. Results on R-TOFU Benchmark

To validate our method’s effectiveness in more challenging scenarios involving complex reasoning chains, we conduct experiments on the R-TOFU benchmark using DeepSeek-R1-Distill-Llama-8B. R-TOFU contains complete chain-of-thought reasoning processes, where the ground truth is interleaved with reasoning steps, making it more challenging to unlearn compared to direct question-answering scenarios.

• Model Fine-Tuning. Following prior work [32], we fine-tune DeepSeek-R1-Distill-Llama-8B on the R-TOFU dataset to obtain the target model for unlearning. The ROUGE-L recall scores are presented in

  Table 9. ROUGE-L recall scores on R-TOFU for the target model. The target model achieved a substantial improvement in ROUGE-L scores on both the forget and retain datasets, indicating the strong effectiveness of the fine-tuning process. ↑ indicates improvement over pretrained.

  Model	Retain			Forget
  	Real Authors	World Facts	Retain Set	Forget Set
  Pretrained	-	-	0.38	0.39
  Target	0.68	0.82	0.75 ↑	0.72 ↑

  .

Main Results

As shown in

Table 10. Results of our method and baseline methods on ten continual unlearning tasks on R-TOFU benchmark. For each task, Bold indicates the best result and underline indicates the second best for each metric.

Method	Task1		Task2		Task3		Task4		Task5		Task6		Task7		Task8		Task9		Task10
	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE	MU	FE
GA	0.70	0.21	0.68	0.28	0.68	0.59	0.0	0.96	0.0	0.96	0.0	0.97	0.0	0.97	0.0	0.95	0.0	0.96	0.0	0.97
GA + GD	0.69	0.19	0.68	0.25	0.39	0.80	0.0	0.96	0.0	0.96	0.0	0.97	0.0	0.97	0.0	0.95	0.0	0.96	0.0	0.97
GA + KL	0.71	0.18	0.68	0.25	0.54	0.41	0.0	0.96	0.0	0.96	0.0	0.97	0.0	0.97	0.0	0.95	0.0	0.96	0.0	0.97
IDK	0.75	0.18	0.73	0.18	0.67	0.28	0.65	0.50	0.57	0.47	0.57	0.71	0.41	0.76	0.39	0.84	0.22	0.82	0.25	0.89
GRPO	0.69	0.29	0.67	0.29	0.63	0.31	0.66	0.32	0.62	0.31	0.60	0.31	0.60	0.33	0.59	0.42	0.56	0.46	0.47	0.59
SRRS (ours)	0.73	0.25	0.70	0.29	0.69	0.35	0.68	0.45	0.63	0.51	0.62	0.52	0.61	0.57	0.58	0.62	0.54	0.67	0.53	0.75

, our method outperforms all baseline approaches on the R-TOFU benchmark. We observe that SFT-based methods (GA, GA + GD, GA + KL) cause the model’s utility to rapidly drop to zero after three to four unlearning tasks, which indicates that the model can no longer generate meaningful responses. In contrast, the GRPO-based method maintains reasonable utility even after all unlearning tasks, but its forgetting efficacy remains relatively low. Our method combines the strengths of both approaches by dynamically adopting them in a sample-wise manner during training, thereby achieving the best performance in continual unlearning. As illustrated in Figure 6, our approach strikes the optimal balance between model utility and forgetting efficacy, demonstrating the effectiveness of reward-based sample selection in handling complex reasoning chains.

Table 10 shows results on R-TOFU benchmark. Similar patterns emerge: GA-based methods achieve high FE but collapse to MU = 0 by Task4, while GRPO maintains utility (MU: 0.69→0.47) with limited forgetting (FE: 0.29→0.59). IDK shows gradual utility decline (MU: 0.75→0.25) with improving FE. Our SRRS achieves the best balance, maintaining high utility (MU: 0.73→0.53) while progressively improving FE (0.25→0.75).

5. Discussion

In this section, we discuss the broader implications of our SRRS framework, its connections to modern alignment paradigms, current limitations, and promising directions for future research.

5.1. Connecting SRRS to RLHF and Continuous Alignment

We note that sequential unlearning requests in real-world deployments can be naturally interpreted as a form of human or regulatory feedback. Users, data subjects, or compliance authorities submit requests to remove specific knowledge from the model, which is analogous to the preference signals used in RLHF [16]. This perspective motivates us to explicitly connect the RL component in SRRS to the broader RLHF paradigm.

From Fixed Utility Metrics to Preference-Driven Reward Modeling. In our current implementation, the reward signal $R=R_F+R_A$ (Equations (7) and (8)) is designed based on fixed, rule-based metrics (ROUGE-L recall and semantic similarity). While effective for benchmark evaluation, this design could be extended to incorporate preference-based reward modeling in practical deployments. Specifically, after each unlearning step, one could collect model outputs on a held-out set of “desired tasks” (e.g., general question answering, code generation, or tool-use scenarios) and obtain preference rankings from human annotators or reliable automated evaluators. These preferences can then be used to train a reward model or directly optimize a preference-based RL objective, enabling SRRS to preserve human-valued capabilities rather than solely optimizing generic metrics.

Sequential Access and Active Learning. Given that sequential unlearning induces capability drift over time, an active learning strategy could be employed to select the most informative samples for human preference evaluation. Uncertainty sampling or disagreement-based sampling can identify capabilities at highest risk of degradation, reducing annotation costs while improving sensitivity to critical utility losses. This approach is particularly relevant when annotation budgets are limited but maintaining specific high-value capabilities is essential.

Reframing Sequential Unlearning as Continuous Alignment. We propose viewing SRRS not merely as an unlearning algorithm, but as a long-term maintenance mechanism for deployed models. Just as RLHF is used for initial alignment during model training, sequential unlearning with SRRS can be understood as a form of continuous re-alignment: after knowledge removal, the model must be re-aligned to maintain desirable behaviors and preserve its utility on high-value tasks. This unified perspective—where both initial training and post-deployment maintenance involve iterative feedback and policy refinement—positions SRRS within the broader landscape of preference optimization techniques [39].

We emphasize that our current experiments adopt the standard TOFU and R-TOFU evaluation protocols to ensure reproducibility and fair comparison with existing baselines. The RLHF-style extensions discussed above represent practical enhancements for industrial deployments, where human feedback can provide richer signals for utility preservation.

5.2. Limitations

While SRRS demonstrates strong performance in balancing forgetting efficacy and model utility, several limitations should be acknowledged.

Computational Overhead. SRRS requires both SFT and RL computations per training batch, resulting in approximately $1.5$–$2\times$ the training time compared to single-objective methods (e.g., GA or NPO). While we consider this overhead justified by the improved utility-forgetting trade-off, it may be a concern for resource-constrained deployments or scenarios requiring rapid unlearning turnaround.

Scalability. Our experiments are conducted on 4B- and 8B-parameter models. The scalability of SRRS to larger models (e.g., 70B or above) and its behavior under more extreme sequential unlearning scenarios (e.g., hundreds of sequential requests) have not been thoroughly investigated.

Reward Design Generalization. The current reward design is tailored to factual knowledge unlearning tasks. Its effectiveness on other types of unlearning targets, such as style removal, capability suppression, or multimodal content, requires further validation.

5.3. Future Work

Several promising directions emerge from this work:

Human-in-the-Loop Reward Design. As discussed in Section 5.1, integrating human preference feedback into the reward design of SRRS is a natural extension. Future work will explore practical implementations where preference comparisons on held-out tasks guide the RL optimization, enabling more nuanced utility preservation aligned with human values.

Broader Applications. We plan to extend SRRS to diverse unlearning scenarios beyond factual knowledge removal, including copyright content removal, toxic generation mitigation, and domain-specific knowledge management. Additionally, exploring SRRS in multimodal settings (e.g., vision–language models) presents an interesting avenue for future research.

Efficiency Optimization. Reducing the computational overhead of SRRS through techniques such as parameter-efficient fine-tuning (e.g., LoRA), early stopping criteria based on reward convergence, or more efficient RL algorithms could improve its practicality for large-scale deployments.

6. Conclusions

In this work, we addressed the critical challenge of continual machine unlearning in large language models, where existing SFT-based approaches suffer from significant utility degradation under sequential unlearning requests. We introduced the supervised fine-tuning and reinforcement learning with Reward-based Sampling (SRRS) framework, which dynamically harmonizes SFT and reinforcement learning guided by reward signals. Experimental results demonstrate that SRRS achieves superior performance compared with existing baselines, effectively balancing forgetting efficacy and model utility in continual unlearning scenarios. These findings highlight the potential of reinforcement learning-enhanced strategies to advance practical and reliable machine unlearning.