Balancing Specialization and Generalization Trade-Off for Speech Recognition Models

Abstract

Recently, using foundation models pretrained on massive volumes of data that are finetuned for the downstream task has become a standard practice in many machine learning applications, including automatic speech recognition (ASR). In some scenarios, we are interested in optimizing performance for the target domain (specialization)while preserving the general capabilities of the pretrained model. In this work, we study this effect for various finetuning strategies that aim to preserve pretrained model capabilities. We identify model merging as a promising strategy that performs well across diverse scenarios. However, our findings show that leveraging a small number of data points from the task we are interested in preserving the accuracy of significantly improves the balance between specialization and generalization. In this context, we demonstrate that combining a simplest finetuning strategy with a memory buffer yields highly competitive results, surpassing other more complicated approaches. Our analysis highlights the need for further research into methods that effectively utilize memory buffers, especially in low-resource scenarios. To encourage further exploration in this area, we have open-sourced our code.

1. Introduction

Most recent progress in deep learning is based on large foundation models [1] that were pretrained on large-scale data and finetuned for the target downstream tasks. These models, often based on transformer architectures [2], have revolutionized natural language processing (NLP), computer vision, and speech-related tasks by leveraging vast amounts of diverse training data. Those advances were also brought to automatic speech recognition: a Whisper [3] model was trained on 680,000 h collected from the internet and has demonstrated automatic speech recognition (ASR) capabilities in 96 languages. Such models possess impressive general capabilities: they are effective learners from scarce data (few-shot or zero-shot learning) and are robust to different types of distribution shift [1,3]. Their adaptability and ability to transfer knowledge across tasks and languages make them particularly useful in scenarios where labeled data is limited.

As with all foundation models, they usually require some finetuning to the downstream task. For example, the Whisper training data is heavily English-centric, with about 65% of the labeled data being in English, which influences the Whisper model’s performance: it performs best in English but still provides reasonable accuracy in other supported languages. However, they still stand as a strong starting point for further finetuning; they generally perform better on low-resource languages than monolingual models trained from scratch, since similarities between languages can be leveraged.

Interestingly, when finetuning those models, their desirable properties (i.e., robustness, zero-shot performance) are forgotten [4,5]. A standard strategy to mitigate this issue is a data replay. However, this is very costly and often not feasible as the data used to train foundation models is kept private [3,6]. Yet, such a problem has many use cases. For example, when finetuning the Whisper model for some low-resource languages, we would like this model to still excel in some commonly used languages (i.e., English). Similarly, when improving ASR performance on some specialized domain, for example in the medical domain which includes challenges such as medical jargon and terminology, we would like to maintain the general recognition capabilities learned during pretraining.

The aforementioned problem of forgetting previously acquired knowledge, also known as catastrophical forgetting [7,8], has been long studied in the Continual Learning (CL) domain. The most popular approaches include some kind of regularization, either by constraining changes in model weights [9], changes in model features (DER [10]) or outputs (e.g., LwF [11]). Recently, weight interpolation between pretrained and finetuned models, known as model merging, was also used in this area [12].

However, this problem has not been extensively studied in the ASR domain. In this work, we investigate it in two scenarios: one where no retention task data (i.e., data from the language whose recognition capabilities we aim to preserve) is available, and another where some of this data is accessible. In the first, more restrictive scenario, we focus on understanding the problem of forgetting in isolation. We find that out of existing methods, model merging is the method that performs consistently the best; however, it did not resolve the forgetting problem for the low-resource adaptation scenario. In the second, we explore potential solutions to mitigate this issue by using small data for experience replay. In this scenario, we show that merging does not benefit much from additional data, and as more data is available for the replay, standard approaches, including simple finetuning, are very competitive.

Overall, our work offers numerous findings, and the contributions are as follows:

• Analysis of the specialization–generalization trade-off in ASR—We systematically study how finetuning affects a model’s ability to retain general ASR capabilities while adapting to new tasks, particularly in low-resource and domain-specific scenarios.
• Evaluation of model merging as a forgetting mitigation strategy—We find that model merging effectively balances adaptation and knowledge retention, especially when no prior task data is available, but struggles for tasks with low relatedness.
• Demonstration of experience replay as a simple yet powerful baseline—Our results show that even a small memory buffer significantly improves retention, often outperforming more complex regularization methods.
• Code release—We release our code to enable reproduction of our work and further investigation.

2. Related Work

ASR. Progress in ASR has been accelerated by the development of self-supervised (SSL) pretraining techniques [13,14,15] and scaling them to larger datasets and models. Large speech models trained on large datasets have been studied previously in both monolingual [16] and multilingual contexts [3,17,18,19]. The availability of pretrained multilingual models in ASR has enabled transfer learning approaches for domains with limited data. This has been especially beneficial for improving speech recognition for non-standard speech and low-resource languages [20,21,22]. Self-supervised learning (SSL) has also been explored to adapt initially trained ASR models to new domains without using extra labels, leveraging unlabeled speech data to improve domain robustness [23]; however, in our work, we also focus on scenarios where some labels are available for the adaptation.

Robust finetuning. Naively finetuning the model to the downstream task will cause forgetting of the pretrained model knowledge [12,24]. Wortsman et al. [12] proposed weight averaging as an effective strategy that balances the accuracy of the target distribution and robustness to distribution shifts. Weight averaging uses a linear interpolation between the pretrained and finetuned model. It works well when the two models are connected by mode connectivity, which is often the case when finetuning pretrained models [25]. This strategy was then followed by others [26]. Zheng et al. [4] additionally observed the degradation of zero-shot performance during the finetuning stage. Their mitigation strategy includes weight averaging combined with knowledge distillation from external datasets. In [24], the authors proposed minimizing changes in feature space (instead of the weight space) to preserve knowledge of the pretrained model. Alternatively, it is possible to preserve pretrained model knowledge by changing only selected model parameters [27]. Some other strategies include training only adapters for different tasks [28]. However, this requires knowing or predicting task identity during inference, which is not always feasible.

Forgetting in ASR. The problem of forgetting was also studied from the ASR perspective. To prevent this issue, Pekarek-Rosin and Wermter proposed combining data replay with selective model freezing [29]. This problem was extended to the continual learning setup [30,31] when the model is sequentially trained on disjoint tasks (i.e., ASR for different languages), and the goal is to achieve high accuracy for all languages. Combining weight averaging and knowledge distillation was shown to work well in such a setting [32]; however, as we show, the results of such a method can be surpassed by simple finetuning, even with a very small memory budget. Another strategy included increasing the model size for new tasks, which can limit the forgetting [33]. However, this increases the model size and requires the model to predict the task identity during inference, which we want to avoid.

Continual Learning (CL). In CL, regularization-based approaches are most common, either in the output space [11] or in the weight space [9]. Using some buffer from previous task data, known as Experience Replay (ER) [34,35], is a very strong baseline in continual learning. While many works in the CL focus on scenarios without any access to exemplars [36,37], we consider both scenarios and compare how existing methods work in such settings.

Comparative Analysis of Related Work. In our work, we are interested in analyzing the generality–specificity in ASR, which occurs when learning different languages or specialized terminology. A similar trade-off was analyzed by Diwan et al. [38], who trained speaker-specific ASR systems, observing a decrease in performance for other speakers. Their strategy involves learning user (task) specific modules, with the aim of personalization for real-word embedded devices. Our work is complementary to theirs; here, we focus on tasks that can be in different languages. Additionally we assume that one model should be able to handle all tasks, as we do not assume the task identity to be known. In contrast to approaches that train only adapters for different tasks [28], which require task identity to be known or predicted during inference, our work avoids this constraint. Moreover, strategies such as increasing model capacity to accommodate new tasks, as proposed in [33] to mitigate forgetting, introduce additional complexity and still rely on task identity prediction during inference. This assumption is not feasible in our setting.

3. Method

Model. Whisper is an end-to-end (E2E) encoder–decoder model [39] designed for automatic speech recognition (ASR). The architecture consists of a Transformer encoder and decoder [2], optimized for transcribing audio input into text tokens. The encoder is a multi-layer Transformer network that processes audio features extracted from the input waveform. It employs self-attention mechanisms to capture temporal dependencies in the audio sequence. The decoder is also a multi-layer Transformer network. It utilizes cross-attention to the encoder outputs and autoregressively generates text tokens, translating the audio representation into a text sequence.

Let $\theta \in {\mathbb{R}}^N$ represent the model parameters, where N is the total number of parameters in the model. The function $f(X;\theta )$ denotes the outputs (after applying softmax) of the model, for an input utterance $X\in {\mathbb{R}}^{F\times f}$ consisting of F frames of dimension f. The Whisper model is trained by cross-entropy loss comparing the model output $f(X;\theta )$ for utterance X with y—the ground truth transcription:

$$L_{CE}(X,y;\theta )=-{\displaystyle \frac{1}{T}}\sum _{t=1}^T\sum _{i=1}^V{y}_{t,i}log\left({\widehat{y}}_{t,i}\right)$$

(1)

where T is the number of tokens, V is the vocabulary size, $y_{t,i}$ is the true label (one-hot encoded) and $y_{t,i}^{\ast }$ is the predicted probability for the ground-truth class.

Problem formulation. During finetuning, we are interested in preserving the existing knowledge of the current model (general capabilities) while adapting to the new task (specialization); e.g., when improving recognition of the pretrained Whisper model in specialized medical jargon and terminology, we would like to maintain the general recognition capabilities learned during pretraining. This is an important problem which was considered from both transfer learning [12] and continual learning perspectives [4,24]. The problem arises because, in many scenarios, we do not have access to the data on which the pretrained model was trained either because the data is large and kept private [3], or we want to limit used computational resources. Let $D_{pre},D_{downstream}$ represent the labeled pretraining and downstream datasets. $T=0$ specifically denotes the pretraining task. The objective of the downstream task $T=1$ is to learn a new model ${\theta }^t$ that satisfies

$${\theta }^T=\underset{\theta }{\mathrm{argmin}}\sum _{t=0}^T\sum _{(X,y)\in D_t}L(X,y;\theta )$$

(2)

However, when learning downstream tasks, the previous task’s data $D_{pre}$ is not accessible (e.g., no access to pretraining data) or limited. Therefore, ${\theta }^T$ cannot be directly computed from Equation (2). In transfer learning, we are learning only one downstream task ($T=1$), while in continual learning, the number of tasks can be larger.

4. Experimental Setup

4.1. Experiments

We focus on a transfer learning setup for downstream tasks, with the additional goal of preserving certain knowledge from the pretrained model, such as its recognition capabilities in the English (EN) language. Throughout this paper, we refer to this as the retention task, emphasizing our interest in maintaining the knowledge associated with this task. More specifically, we consider the following experiments:

• Adaptation to Polish (PL) language while preserving the pretrained model recognition capabilities in the English (EN) language.
• Similar to the above, but adaptation is performed to the Slovak (SK) language. Here, we are experimenting with a very low-resource language adaptation, with an order of magnitude smaller number of utterances than for the PL language.
• Adaptation to specialized medical jargon in the PL language while investigating forgetting in the PL language. This experiment allows us to study the generalization–specialization trade-off when there is a similarity between the two tasks.

We have used PL language as it will be also used in experiments with medical data, and SK language was chosen because it is an example of a very low-resource language.

4.2. Data

Datasets. As a data source for PL, we use CommonVoice [40] (version 11.0) and LibriSpeech Multilingual [41] datasets (69 k utterances in total). For the SK language, we use the CommonVoice 11.0 dataset (3 k training utterances). To monitor forgetting during finetuning, we measure model performance on the English (EN) language, by using CommonVoice and LibriSpeech datasets. Summary is given in

Table 1. Details on data used in this paper. EN data is used for memory buffer (train split) and for evaluating forgetting (test split).

	Speakers	Source	Train Split Size	Test Split Size
PL	2823	CV and LS	41.5 k	8.8 k
SK	137	CV	3.01 k	2.24 k
EN	55,574	CV and LS	216.0 k	19.0 k
MED-PL	181	Private	42.9 k	4.8 k

.

MED-PL. We additionally experiment with our in-house Polish Medical corpus (MED-PL) dataset, collected for the development of a specialized ASR system for medical applications in Poland, in the context of supporting medics [42]. MED-PL consists of short speech commands (up to 30 s) recorded by Polish speakers. Overall 181 people recorded 48 k utterances, for a total of 99 h. While most of the transcripts are close to the ground truth, some speakers deviated by repeating, skipping, or adding words.

4.3. Models

Whisper. As our model, we use a small version of the Whisper pretrained model, which contains 244 M parameters [3]. The model contains 12 transformer encoders and 12 decoders, with attention dimension 768 and feedforward dimension 3072. For all of the experiments, all model parameters are shared across all of the task. This is in contrast to existing works [32,33], where some parts of the network are task-specific. As a result our inference is task-agnostic; that is, we do not need to pass the task identifier (language or dialect) to the model, which is more difficult, but also more realistic.

Baselines. In our work we consider the following baselines.

(1)

Finetuning—naively adapts the model to the target. This is a highly plastic model that will perform well on new tasks but forget much of the previous knowledge.

(2)

Frozen—we additionally report the results of the frozen model which favors stability (knowledge preservation).

(3)

Learning without forgetting (LwF) [11]—it is a popular Continual Learning method, which was previously successfully applied also in ASR [43,44]. During finetuning it adds a distillation loss to minimize the KL divergence between outputs for the ($\theta$) and the model from the previous task (${\theta }^{\ast }$):

$$L(X,y;\theta )=L_{CE}(X,y;\theta )+\lambda KL(f(\theta ,X),f({\theta }^{\ast },X))$$

(3)

We run a parameter search for the optimal $\lambda$ value in Section 5.3.

(4)

EMA_teacher. The LwF is a strong Continual Learning approach; however, it can limit learning of the new task [36]. Therefore, to improve its flexibility, we also experiment with a variant where we allow the teacher to slightly adapt towards the target task. Specifically, we present a EMA_teacher strategy, which is a similar strategy to the LwF, but here the teacher is set to be an exponential moving average (EMA) of the current model:

$${\theta }_{teacher}=\beta \ast {\theta }_{teacher}+(1-\beta )\ast \theta$$

(4)

The EMA was first proposed by Tarvainen & Valpola [45] in the semi-supervised setting. The same strategy was later used in self-supervised learning [46,47] and more recently to online continual learning [48], which inspired us to use it in our setting. Following popular approaches in this area, we set $\beta =0.999$ [48,49].

(5)

Model Merging—we also experiment with merging approaches [12,50]. More specifically, in the first stage, a simple finetuning procedure is applied, and then the weights of the new model are computed as follows:

$${\theta }_{new}=\alpha \ast {\theta }_t+(1-\alpha )\ast {\theta }_{t-1},$$

(5)

where in our case, $t=1$ refers to the downstream task, and $t=0$ refers to the pretraining stage. Similar, as in other works [32], we set $\alpha =0.5$. We also experiment with Model Merging combined with LwF regularization, following [32].

(6)

Experience Replay (ER)—we additionally optionally equip all methods with a small budget of memory of previous task in Section 5.3, as commonly used in Continual Learning [29,31]. The specific type of data used for experience replay varies depending on the experiment and is detailed later in the paper.

4.4. Methodology

Implementation details. To provide a fair comparison, all methods follow the same training procedure: number of training steps, learning rates, batch size, etc. The optimizer is AdamW, with a batch size of 64 and weight decay of 0.1, following [29]. The learning rate and number of steps were determined empirically by running the finetuning procedure until the validation loss converged. Each experiment was repeated 3 times and mean results are reported.

Metrics. Our main metric is the Word Error Rate (WER). For transfer learning experiments, we report WER on target tasks (PL, SK or MED-PL) and for previous knowledge (EN or PL). We are interested in finding a good balance between accuracies in downstream and pretrained tasks. We use geometric mean contour lines to visualize the trade-off between forgetting and downstream accuracy, as this metric emphasizes balanced performance and penalizes extreme imbalances more than an arithmetic mean.

Training Time. The experiments were conducted on an NVIDIA A100 40 GB GPU. All evaluated methods have similar computational requirements. For example, finetuning for the PL experiment takes 4 h 21 min 34 s, while LwF and ${\mathrm{EMA}}_{\mathrm{teacher}}$ increase the training time by approximately 10%. The additional cost of Model Merging is negligible.

5. Results

5.1. Forgetting in ASR Models

In the first part, we seek to understand how finetuning the model on the target domain affects the pretrained model knowledge in two scenarios:

• The target and retention tasks are disjointed. We analyze finetuning in low-resource downstream tasks (PL and SK) and evaluate forgetting on retention tasks (EN).
• The target and retention tasks are similar. For this purpose, we finetune the model on a specialized dataset of Polish medical speech and evaluate forgetting in the PL language (retention task).

Results are presented in Figure 1. For PL language, the model was finetuned for 2.3 epochs, and for both SK and MED, it was finetuned for 5 epochs, corresponding to 1.5 k, 0.23 k and 3.35 k steps, respectively, to provide the most optimal results.

Effect of Downstream Tasks on Forgetting. We can see that the level of forgetting on the retention task is highly dependent on the downstream task. While the forgetting is excessive for the PL and SK downstream tasks (WER for the retention task reaches 63.8% and 75.7%, respectively), it is less severe in the case of the MED-PL experiment, where there is more similarity between downstream and retention tasks.

Generality–specificity trade off. The finetuning strategy favors accuracy on the target task, losing the general recognition capabilities of the model, compared to the frozen model. Regularization-based approaches (LwF, ${\mathrm{EMA}}_{\mathrm{teacher}}$), in general, struggle to find significantly better solutions. On the other hand, Merging is the strategy that works consistently well, significantly outperforming other approaches, also when combined with LwF.

However, we can see that the efficiency of this method is again highly task-dependent. In particular, for the SK language, where the pretrained model struggles (WER on SK equals to 66.3%), the Merging allows us to find some balance, but the resulting model is only mediocre in both EN and SK languages.

Forgetting is a relevant problem. Without any memory buffer, most methods are affected by forgetting, as indicated by results on general knowledge (EN) and previous tasks, compared to the frozen model. Although Model Merging seems to perform quite well, the forgetting is far from being solved. For example, in the PL experiment, WER for EN increases from 17.6% to 26.4%, and even more for the SK language. In the medical experiment, forgetting was minimal; however, adaptation to the downstream task was more compromised in this scenario.

5.2. Impact of Experience Replay

In this section, we assume some access to the data of the task for which we aim to preserve recognition. While the data used to train pretrained models is often private, it is practical to assume that some data for the task of interest can be gathered. Here, we revisit the scenarios discussed in the previous subsection:

• When finetuning PL and SK languages, we also used additional data for EN language.
• When finetuning on the Medical dataset, we also used PL data.

In both scenarios we use Common Voice and Libri Speech as a source for memory buffer from previous tasks. Following [29], instead of including a fixed number of samples from the retention task domain in each batch, we include a fixed percentage of samples from the retention task spread out over all batches. Results in this section are reported with a memory budget of 5%, following prior work [29].

Experience Replay is a strong baseline. We can observe that adding memory buffer to the finetuning is a strong baseline in this setup (Figure 2). More advanced methods, such as (LwF, ${\mathrm{EMA}}_{\mathrm{teacher}}$) do not yield substantial improvements over this baseline. This observation aligns with prior Continual Learning studies, which report that memory buffers often constitute a strong baseline in similar scenarios [51]. On the other hand, Merging approach is still very competitive; however, it does not consistently improve over the simple finetuning approach.

Varying size of the memory buffer. We also take a detailed look at how using replay data affects different methods’ performance. Specifically, we look at the most promising methods (Finetuning, LwF, Merging) under different sizes of the memory, where the memory budget is within {1, 5, 10, 20}% of the retention task data. Figure 3 examines the performance of various methods with different memory buffer sizes. This allows us to study how the size of the memory buffer affects different methods. Overall, we can observe that when a very small memory budget is used (i.e., 1%), Merging still works the same as or better than competing methods. However, as soon as reasonable amounts of data buffer are available, other approaches (LwF, ER) find better solutions. Interestingly, we can observe that Merging benefits very little from adding the memory buffer.

5.3. Detailed Analysis

5.3.1. Corruption Robustness

Here, we evaluate how the robustness of the ASR models is affected during finetuning. More specifically, we use robustness to synthetic Gaussian noise, as recently introduced in the Speech Robust Bench [52]. We show the evaluation in Figure 4, where all the testing utterances are modified using the Gaussian noise distortions. First, we can observe that adding Gaussian noise decreases model recognition capabilities, WER increases from 17.5% and 17.2% to 60.1% and 88.0% for EN and PL languages, respectively. The decrease in quality is significantly more severe for the PL language, which shows that robustness to noise is also language-dependent. As the pretrained model has seen much more data for the EN language, presumably with varying quality, it is also more robust to noise distortion in that language. Finally, we can observe that our previous findings also transfer to this robustness setting. While Merging excels when there is no memory buffer available, we can see that it is outperformed by other methods when some data for the retention task is available.

5.3.2. Similarity Analysis

Throughout our experiments we have observed that the following simple strategies work well:

• Merging models, when there is a limited access to the data of the pretrained task,
• Experience Replay when sufficient buffer of data is available.

We are interested in understanding how those different approaches result in their solutions, i.e., whether the solutions found by Model Merging are similar to those found when using experience replay. To understand this, we compute the Euclidean distance of the final solution with regard to the pretrained model and compute Centered Kernel Alignment (CKA [53]). CKA is a similarity metric used to compare two sets of high-dimensional representations. It evaluates the agreement between two matrices, each representing pairwise similarities of all samples in a dataset, where the matrices are derived from the representations (features) produced by a model. In our case, we compute the CKA with regard to the pretrained model on the retention dataset (EN for PL and SK experiments and PL for MED-PL experiment). The results can be found in

Table 2. Word error rates for the transfer learning experiments, along with the Euclidean distance and CKA alignment relative to the pretrained model (using control task data). Model Merging achieves effective solutions by remaining close to the pretrained model and maintaining high CKA alignment. Interestingly, adding experience replay has only a minimal impact on these metrics. Bolded font marks best results.

Method	PL				SK				MED-PL
	PL	EN	L2-Dist	CKA	SK	EN	L2-Dist	CKA	MED-PL	PL	L2-Dist	CKA
Finetuning	$\mathbf{12}.\mathbf{71}\pm 0.24$	$63.81\pm 6.78$	11.25	0.77	$\mathbf{27}.\mathbf{34}\pm 0.68$	$75.71\pm 2.48$	4.38	0.82	$\mathbf{7}.\mathbf{72}\pm 0.33$	$38.46\pm 2.03$	17.83	0.64
Merging	$13.09\pm 0.07$	$\mathbf{26}.\mathbf{4}\pm 0.07$	5.63	0.84	$54.79\pm 4.27$	$\mathbf{56}.\mathbf{79}\pm 0.53$	2.19	0.85	$14.50\pm 0.06$	$\mathbf{18}.\mathbf{79}\pm 0.29$	8.91	0.81
LwF	$16.26\pm 1.42$	$45.79\pm 5.14$	11.55	0.77	$\mathbf{27}.\mathbf{31}\pm 0.68$	$72.21\pm 2.48$	4.38	0.80	$9.52\pm 0.01$	$37.33\pm 0.09$	17.55	0.59
Finetuning + ER	$\mathbf{13}.\mathbf{15}\pm 0.31$	$16.26\pm 2.11$	10.89	0.81	$\mathbf{27}.\mathbf{08}\pm 0.14$	$\mathbf{54}.\mathbf{85}\pm 3.87$	4.48	0.77	$\mathbf{8}.\mathbf{56}\pm 0.85$	$24.19\pm 0.51$	18.8	0.66
Merging + ER	$14.02\pm 0.32$	$\mathbf{15}.\mathbf{25}\pm 0.09$	5.45	0.86	$42.83\pm 2.85$	$\mathbf{54}.\mathbf{85}\pm 1.42$	2.24	0.83	$15.22\pm 0.08$	$\mathbf{15}.\mathbf{63}\pm 0.18$	8.91	0.80
LwF + ER	$15.19\pm 2.56$	$16.26\pm 0.97$	11.11	0.76	$\mathbf{27}.\mathbf{09}\pm 0.30$	$\mathbf{54}.\mathbf{83}\pm 0.19$	4.51	0.78	$8.93\pm 0.68$	$24.41\pm 0.49$	18.56	0.64

. We can observe that Model Merging achieves effective solutions by remaining close to the pretrained model and maintaining high CKA alignment. Surprisingly, adding memory buffer replay has small impact on these metrics, despite the fact that memory buffer significantly boosts the retention task accuracy.

5.3.3. Hyperparameter Sensitivity

In Figure 5, we analyze the effect of the interpolation factor $\alpha$ in the merging experiments. The interpolation curves are smoother when the target and retention tasks are more similar. In low-resource scenarios, the optimal $\alpha$ can deviate significantly from the commonly used value of 0.5. For the SK language in particular, it appears that a better value could be found, though determining this would require validation sets, which can be challenging in very low-resource scenarios.

LwF uses additional hyperparameters that balance between adapting to the new task and preserving previous task accuracy. To determine the optimal value, we run a hyperparameter sensitivity check and visualize those results in Figure 6. We set $\lambda =0.8$ as it offers a good trade-off between downstream and retention task accuracies, which is also a value commonly used in the community [54,55].

Further, we investigate how varying the length of the finetuning stage affects the merging performance (

Table 3. Impact of finetuning duration on Model Merging performance in the PL task. Results remain stable with shorter training, but excessive finetuning reduces accuracy, highlighting the importance of early stopping.

Train time	50%	75%	100%	125%	150%
Downstream Task WER (PL)	13.4	12.8	13.3	13.3	18.1
Pretrained Task WER (EN)	26.1	25.4	26.4	29.9	38.8

). Overall, we find that Model Merging is quite robust to the training length, i.e., the results did not change much as we reduced the training time. However, training too long can reduce the effectiveness of Merging; therefore, it is important to use cross-validation during finetuning.

6. Discussion

Throughout this article, we conducted a series of experiments to explore the balance between generalization and specialization in ASR models. Overall, the key findings can be summarized as follows:

• Model Merging consistently performs well across tasks, data quantity, and quality.
• When some memory buffer is available, simple finetuning achieves competitive results.
• Forgetting is a key issue, influenced by the pretrained model’s initial difficulty.
• Model Merging improves CKA alignment and reduces L2 distance; memory buffer slightly impacts these metrics but significantly boosts accuracy.

While Model Merging did not always yield optimal results (e.g., for the SK language), it outperformed competing approaches in most cases. These findings align with recent research on robust finetuning and continual learning, which also highlights the effectiveness of this technique [12,50].

However, when a reasonable memory buffer was allowed, other methods, including the straightforward Experience Replay, demonstrated superior performance. The relatively suboptimal performance of Model Merging in such scenarios appears underexplored in the literature, likely because many studies assume no access to prior task data, especially in continual learning [36,37]. While this assumption may hold in certain cases, such as those involving privacy concerns, there are many scenarios where it is feasible to gather some data, as it is our case. From an ASR perspective, even if access to data from the pretrained model’s original tasks is restricted, it is often possible to collect data specific to the task for which knowledge preservation is required.

We have also identified the initial difficulty of the task for the pretrained model as a key factor influencing the potential to achieve a good balance in accuracy between downstream and retention tasks, particularly when using Model Merging. As illustrated in Figure 5, the interpolation curves are smooth when the two tasks are related (right image). In contrast, for SK language adaptation (center image), there is a significant loss barrier. This suggests that finetuning has substantially altered the model’s parameters to adapt to the new language, thereby increasing the difficulty of merging.

Limitations. Overall, while Model Merging performs well across various scenarios, it is important to acknowledge its limitations, particularly from a practitioner’s perspective. First, the approach struggles when tasks are unrelated, as seen in the SK experiment. In such scenarios, more sophisticated merging approaches could be used (e.g., [56,57]). Additionally, although Merging is relatively robust to hyperparameter changes, careful validation of the $\alpha$ interpolation factor (Figure 5) and training duration (Table 3) remains important. Scaling this approach to a larger number of tasks also presents challenges [26]. In such cases, incorporating exemplars or adopting more specialized merging techniques, may be necessary.

Our study focuses on task-agnostic inference, which we consider a more realistic and challenging setting compared to the commonly used task-aware inference [28,33]. In preliminary experiments with larger models from the Whisper family, we found that reliable training in the task-agnostic setting was not feasible, likely due to the limited size of the available datasets. As a result, we restricted our investigation to the small model variant.

7. Conclusions

In this paper, we examine the challenge of balancing the acquisition of new knowledge (specialization) with the retention of prior knowledge (generality) when adapting a pretrained model to a downstream task. This issue holds significant practical importance; for instance, when adapting a pretrained model to specialized medical data, it is crucial to preserve the model’s general capabilities, particularly given that access to pretraining data is often unavailable.

We structure our analysis around two scenarios: one assuming limited access to previous task data and the other assuming no access at all. Our findings reveal that in the absence of prior task data, Model Merging emerges as a consistently effective strategy, outperforming more sophisticated regularization approaches. However, when even a small memory buffer is permitted, this advantage diminishes, and simple experience replay can achieve superior performance. These results highlight a gap in the community’s focus, as much of the current research prioritizes scenarios without memory buffers, whereas practical applications often allow for some degree of data retention.

A limitation of our paper is using a relatively small set of target tasks and running experiments on one pretrained model. Looking ahead, we identify significant potential in combining Model Merging with memory buffer strategies, presenting a promising avenue for future exploration.